Saturday, August 17, 2013

Leap Motion: Towards a New Linguistics

TL;DR Methods for exporting data from our brains are woefully slow and incomplete. Tools such as the new device from Leap Motion may provide an intriguing ability to extend language itself with color and other attributes.

Humans Are Not as Fast as Computers as Communicators

The laptop I am using to write this post can export data to the world at 1,000,000,000 bits of data per second - which is quite slow compared to the speed it can transmit data internally. But what about the speed of getting the data from my brain to the computer?

If you speak fast, you might output about one hundred sixty words per minute. At five characters per word plus a space that adds up to 960 bytes of data per minute. Typing is generally half that speed.

The world record for piano playing is currently at 765 keys played in a minute. A very good guitar player might hit 600 notes per minute.

Of course there is much meta data as well. Voice has pitch and timbre. The piano has acceleration and duration. So there's more data but not that much more

Using a mouse it may well be possible to click several hundred times in a minute. A game controller is likely to produce even more but a computation of an estimate of the data output per minute is beyond my skill set. But it can be no more than a few thousand bytes per minute.

Is there a pattern here? If there is then it is really simple: The human being, using current methods, is able to produce only a very small amount of new digital data per minute.

You can read fast, listen fast and view incredibly fast but in terms of creating or generating new information that you can share you are painfully slow when compared to the technology you have in your hand.

And, frankly, even the receiving of data into your brain is not that fast. Reading 300 words per minute - or 1,500 bytes per minute is considered to be speed reading.

These speeds have certainly improved over the centuries. We have been reading and writing for three or so thousand years and speaking and listening to words for perhaps a hundred times that long. Certainly we can gather and disseminate faster than our ancestors. Trying to do speed reading with stone tablets was probably not self-evident.

Nevertheless the improvements in the speed of reading and writing comprehension have not improved significantly in in my lifetime. For example, Any improvements the abilities for our brains to process data has been nowhere near the gain in the ability to transmit the data that the Internet brought about. In other words your grandmother could read a letter about as fast as you can read an email.

Perhaps there is some kind of an asymptotic limit to the speed at which our brains can import and export data and we seem to be approaching it. At least we seem to be approaching some limit using the tools we have become accustomed to using over the last millennium.

Emerging Alternative Methods of Writing

But are letters and spoken words the only means we have at our disposal? Are there other ways/methods/vehicles that we could use to communicate?

Let's consider some ways.

I am intrigued by the differences between the way Westerners and East Asians tend to read. Westerners use a phonetic alphabet and East Asians ten to use ideographs. Both have their strong points.

And I begin to see a blend occurring. The East Asians have been dragged into learning the Roman alphabet. And Westerners have begun to learn a new series of ideographs that range from the home icon, reload, mute, go back and other icons as well as smileys and emoji.

I also see much increased visual complexity in the data being sent out, Web pages and text and imagery and sophisticated graphic design. Whether codes such as 'lol', 'bte', and 'rofl' speed up your data output is up for grabs.

Full Body Data

Can we switch gears? Is there a new 'communiKid' on the block. Could we import and export data to and from our brains at 5K bytes per minute or more without inserting tubes and connecting wires into our brains?

If so, how would we do this?
When people develop computer games, they need to simulate body movements in order too create animated characters. They use a technology dubbed 'motion capture' to do this. A typical method of motion capture is for an actor to have a number of dots attached to their body, be filmed going through a series of moments and the film decoded in such a way that the movement of the dots can be saved as X, Y and Z coordinates.

In this manner, hundreds of thousands of bytes of intentional data can be exported from a human being per minute.

This may be 'poetry in motion' but the data itself is not a a mere poem. Every byte of data was caused by an intentional action that occurred in that actor's brain. This is a huge amount of engaged data being created and logged per minute.

When we look at ballerinas, golfers, tennis players, ju-jitsu practitioners and others we can see virtually every aspect of the body brought under control of the mind and dedicated to communicating.

Thus, conceivably, not only can we record huge amounts of data but in the right bodies much of that data could be termed significant or intentional data.

We cannot, however, all be Tiger Woods. Nor do we have access to Hollywood motion data recording studios. Are there, perhaps, other ways to capture the body's kinetic motions and transforms that motion into digital data?

Dandy New Device

Golly gee. It's just arrived on my desk.

I am using the device recently released by Leap Motion. It's tiny device, no bigger than a pack of chewing gum, that records the X, Y and Z position of every finger you move as well as the three rotation angles of the fingers and the velocity. It is also doing the same and a bit more with the palm of the two hands. Whether this is 98 data points or more or less is up for debate, nevertheless the data is coming in at over 100 times per second or more than 6,000 events per minute. So is this half a megabyte of data being created per minute? Who knows? All I know is that the more days I code for this thing the more data I am getting out of it.

Is the data as erudite as a Shakespeare sonnet or as elegant as an Einstein equation? I don't need to answer that do I? But then again the first human grunts or scribbling were also probably nothing to write home about.

Thus you can probably feel safe that you will go to your grave with a keyboard and microphone. But what about the children and the grandchildren?

My intuition is that using such devices and their successors we will build new, extras layers of communication with such devices. The price is already cheap enough for it to be attached to all phones or laptops.

Eventually you (or your grand children) are typing in thin air - perhaps more like playing a multi-keyboard organ or air guitar - and what is coming out is some kind of multi-dimensional information stream which in turn builds into a new lexicon. A way of communicating that is phonetic, ideographic, vocal, gestural, 3D and even colorful.

A New Linguistics

If this sounds phantasmagorical, please do remember:

We use gestures to write down musical notation to record the details of the gestures used to create music. And then read that notation and turn the notes back into gestures.
Writing is using gestures to record speech by manipulating pen and ink or by frenetic tapping on plastic keys.

We use gesturing all the time - without thinking. But we subjugate these gestures, we make them the servants of the oral and the aural.

We all know how to wave hello, make the OK sign, show a thumbs up or give the finger. The Italians seem to be able to recognize 250 gestures. These are all the most simplistic and basic events. But they are a start and indicate that our brains are wired to communicate using our hands - just as dogs and cats wired to communicate with their tails and ears. And the use of appendages to communicate with other beings predates visual and auditory communication and is thus perhaps part of our oldest and deepest thought processes.

Perhaps it's time we let gestures act in their own right. Using the Leap Motion device - and its eventual successors and competitors - we will have methods of recording, editing and playing back gestures without referencing or mediation of any other device or instrument. [Explanations of these aspects will be provided soon.]

The changes will not happen overnight. It may even take several generations. Today's babies are learning to swipe on tablets. Tomorrow's babies may learn to swipe in the air.

And the changes may start to occur on several fronts. We are all beginning to use voice recognition. This enables us to get up from our desks and be more healthy. Since we are standing and away from keyboards, I can see gesturing and voice recognition working closely together. In the beginning by adding line breaks and formatting the text while we speak but in the future I can see you speaking while gently moving your hands and fingers. The movements are altering the pitch and timbre of what you are writing much like the gestures of the conductor shape the music of the orchestra. What appears on the screen or in our glass is a writing that we today would hardly recognize. The text is full of colors and devices that emphasize or modify the tempo. Diacritical marks indicate the importance of particular aspects to the reader or the writer. The final output is a gushing of sound, music, spoken words, gesture symbols, images. And did I mention that you will need a 3D printer in order to read stuff in the future?

Too Frightening Maybe? Consider the Possible Happiness

Actually, it all does sound quite frightful doesn't it? Life is already complex enough. Do we really need all this more stuff happening more fast all at once? Not likely.

But then consider this. People may knit and talk at the same time. Drive a car and chat on the phone - not. Take a shower and sing simultaneously - that's good. And when you are doing this, my guess is that you tend to be happy. The more you use the more parts of your brain, the happier your brain. The more your brain is fully engaged the more fulfilled you feel.

This writing - or even the linguistics - of the future may well embody more of your body in service of more comprehensive and faster and more fulfilling methods of communicating.

As I mentioned before, the Leap Motion device is what is opening up these thoughts. It can recognize movements that are a fraction of a millimeter. it's small and its successors will be embedded in phones, laptops and wearables. If it only costs $80 today how little will it cost on five years?
The Leap Motion device is not a game-changer. For example, it won't significantly alter this years' holiday season technology sales. But it will change the game. In ten years or so the rules of the game we call life will be different than they are today. Or is that just me waving my hands at you?

Links

https://www.leapmotion.com/

And here is a link to some of the code I have been working on that provoked these thoughts. This code could not have been written without the support of Leap Motion and the wonderful Three.js 3D JavaScript library that enables me to access the WebGL in your browser:
http://jaanga.github.io/gestification/

If you do not have a Leap Motion device you can get a glimpse of the apps using the links on this page
http://jaanga.github.io/gestification/no-leap-view-only.html