Skin and Bones for Leap Motion Devices ~ Update

Please see the previous post on this topic:

http://www.jaanga.com/2013/09/so-close-yet-still-so-far-skin-and.html

This morning I built and posted Phalanges R5 - a great improvement over the previous release:

http://jaanga.github.io/gestification/work-in-hand/phalanges/r5/phalanges.html

with info here:

https://github.com/jaanga/gestification/tree/gh-pages/work-in-hand/phalanges

The interesting issue in all this is the difference between the methods Leap Motion uses to expose its data and the methods normally used in character animation.

In character animation, all 'bones' are connected. If you move the upper arm then all the bones below move as well.

The Leap provides individual positions and angles data for all the fingers and palms.

Quite frequently you do not have information for all the fingers.

In normal character animation, this is not much of an issue because if you move the palm then any unaccounted fingers will move along with palm automatically.

But with the Leap Motion data, fingertips seen previously may end up sitting frozen in space disjointed from the hand or they may simply disappear. For some people this may be a disconcerting series of events.

[Disclosure: my left hand disappeared a number of years ago never to return, so this sort of thing is no big issue for me. ;-]

The first releases of of Phalanges relied on the fingertips, finger bases and palms all moving and controlled separately. This made for lots of fingers disappearing. The more recent releases followed the idea of all bones being connected and this caused fingertips to move in all sorts of inhuman ways.

The current release is a hybrid. The palm and the finger bases are connected - move the palm and the bases move with it. The fingertips all move independently from each other and from the palm.  This works just fine - until the Leap Motion device decides that a fingertip no longer exists.

So what looks like the next solution to investigate is a hybrid-hybrid solution. When Leap Motion fingertip data is available use the hybrid solution. When Leap Motion data is not available make the Leap fingertips invisible and make a completely connected finger visible. When the Leap finger data is again available, switch out the fingers.

Now all this may seem a wee bit complicated and you would think that sticking just a single joint between tip and palm would be no big deal. And you would be quite right. And you would be really, really smart because your brain would know how to crawl in and out and all over things like inverse kinematics and be prepared to lots more code and include more libraries

But that sort of thing is way beyond my skill level. My brain starts to fatigue when an app is over 300 lines. The current app is at 222 lines. With a bit of luck we can have a skinnable phalanges release that even my little brain may grasp...

Link:

https://github.com/jaanga/gestification/tree/gh-pages/work-in-hand/phalanges

So Close / Yet Still So Far: Skin and Bones for Leap Motion Devices - A Progress Report

Hand image from Leap Motion documentation

2013-09-22: See also update post that discusses much improved Phalges R5:
http://www.jaanga.com/2013/09/skin-and-bones-for-leap-motion-devices.html  


The above image is from the documentation for the Leap Motion device. Questions relating as how to produce such images or how to access the 'raw data' that produces such images are some of the most frequently asked questions in the Leap Motion forums. The bad news is that there is no source code or coding examples currently provided by Leap Motion for producing such a display.

The good news is: Wow! What an excellent coding challenge...

This post is a progress report on the current status to produce realistic-looking and behaving hands that can be controlled by the Leap Motion device.

The most exciting recent development is certainly this recent post by Roman Liutikov:

http://blog.romanliutikov.com/post/60899246643/manipulating-rigged-hand-with-leap-motion-in-three-js

With demo file here:

http://demo.romanliutikov.com/three/10/

Roman provides very clear guidance as how to export skin and bones from Blender as a JSON file that can be read by Three.js and used to display arbitrary, real-time finger movements generated by a Leap Motion device.

An interesting side note is that the code uses a BVH-like structure to control the movement of the fingers. I recently wrote about the importance and efficacy of BVH here:

http://www.jaanga.com/2013/09/bvh-format-to-capture-motion-simply.html

The unfortunate aspect of this work is that there are a number of issues with the movement of the hand and fingers.

Nevertheless, this code is an important step forward and well worth inspecting.  I did so myself and have re-written Roman's code in my own (admittedly somewhat simplistic) style:

Demo: http://jaanga.github.io/gestification/work-in-hand/phalanges/liutikov/liutikov.html

With information and background here:

https://github.com/jaanga/gestification/tree/gh-pages/work-in-hand/phalanges/liutikov

My own work, since the publication of the post on BVH, has been involved with building up a notion of the best methods for positioning and angling the 'bones' inside the fingers. There are a host of issues - too many to list here - including: hands that sometimes have five fingers, or two fingers or no fingers; finger 2 easily switches places with finger 3; the order of the fingers is 4, 2, 0, 1, 3 and so on.

The latest demo (R4) is here:

http://jaanga.github.io/gestification/work-in-hand/phalanges/r4/phalanges.html

Previous releases, source code and further information is available here:

https://github.com/jaanga/gestification/tree/gh-pages/work-in-hand/phalanges

Much is working: the hand generally moves and rotates appropriately, fingers stay in the same position and don't disappear. But it is readily apparent that the tips of the fingers are still quite lost in space.

Not to worry. Eventually the light bulb will turn on. Actually the more likely thing is that a search on Google will turn up an answer or some person very smart in the ways of vectors will respond on Stackoverflow.

Also worth noting is that the people at Leap Motion gave a demo of routines at the recent developer's conference in San Francisco that may provide a satisfactory response. The interesting thing will be to see which code come out first and which code is the more hackable.

BVH: A format to capture motion - simply, quickly and economically

One of the reasons that Android phones have such good voice recognition is because, under Peter Norvig's guidance, Google has acquired an immense corpora or database of what and how people speak. It is my contention that gestures and other non-verbal means of communication will eventually regain some of the primacy that they had before primate evolved verbal communication. If this is to happen to gesturing, then we need some fast, cool effective methods for recording the many gestures people make.

This is not a new thought. Personally and over the years, I have spent some fascinating moments exploring dance notation. And sign language is the codification of gesturing. But coming back to computers, we have all the methods used by computer games to record and replay the movements of game characters. Collada, FBX and the new glTF come to mind.

Here's the thing: gesturing can generate huge amounts of data per second. It's nearly as good ( or bad - depending on your outlook) as video - if nothing else because the data gathering usually is via video. Secondly, if data scientists are ever to be able to parse our gestures they will need the data in digital format. The concept represented by letters 'donut' is far smaller than the audio file of the sound bit let alone the object in question.

Because of my joy in exploring the Leap Motion device, I have spent the last month or so looking into ways of registering gestures.

One of my experiments is to record all the messages sent out by the Leap Motion device and save then in JSON format. The messages are used by software developers and for testing. In normal coding such messages are typically short and sweet (or not). But even a short gesture may generate a JSON file of over a megabyte. If you have a Leap device, you can have a look at the app here:

http://jaanga.github.io/gestification/cookbook/jest-play/r1/jest-play.html

With source code and more details here:

https://github.com/jaanga/gestification/tree/gh-pages/cookbook/jest-play

Thus, as helpful as this app should be to developers and testers (especially as none of the example apps in the Leap Motion examples site can do this), this is not an app that should be used to recorded and replay a corpora of thousands or millions of gestures because the files sizes are too large.

In July I wrote a paper using Google Docs about gesture recording. You can have a look at the paper here:

Skeleton API Considerations for Leap Motion Devices R2

In this document I recommend looking at the BVH format. This is not my first encounter with BVH. Last I wrote a five post tutorial on getting animations into Three.js by importing BVH files into Blender. I have yet to hear or see anybody else that was able to follow - successfully - the tortuous path I proposed that you should dance down. And, in the mean time, there have been so changes, that half the stuff no longer works.

Anyway, because of the paper and because of the Leap device, I decided to write a BVH reader based on code I had found [only after many searches over a long period of time] including these two examples:

https://code.google.com/p/papervision3d/source/browse/trunk/as3/trunk/src/org/papervision3d/objects/parsers/mocap/BVH.as
https://github.com/sinisterchipmunk/bvh

Even though I code a lot, I am not really a programmer and it soon all started to get a bit daunting. When that sort of thing happens I tend to go into denial and whatever. And I did a Google search on 'Three.js BVH reader' and up came this:

http://saqoo.sh/a/labs/perfume/2/

I nearly fell out of my chair. Here was everything I wanted: A simple Three.js app that reads BVH files. And more than that, the code itself is fascinating. The methods the author uses to do 'if/then' within a 'for' loop were totally new to me.

Saqoosha: you are amazing! And thank you for your kind permission to build upon your code. Here's Saqoosha's web site:

http://saqoo.sh/a/

So within quick order I had several demos up and running - each accessing slightly different dialects of BVH. The links are at the end of this post.  And now I have had several days, reading and thinking about BVH and comparing it with other methods.

And the TL;DR is that the BVH format is awesome. Accept no substitute.

You can read about BVH here and here and here.

Thing #1. The main thing is that the main data part of the format is about as sparse as you can get in uncompressed ASCII. It's just numbers and spaces. And, the most important, it's only the numbers you actually need.

Let me try and explain. To position something like a hand or foot in space you to specify it X, Y and  as well as the pitch, roll and yaw angles. That's six numbers - the 'six degrees of freedom'. But the BVH files only records pitch, roll and yaw - three numbers. It assumes you can fill in the X, Y and Z yourself at runtime. How? Because the header tells you the offset distances for all the body bits. In essence, for the purpose of this app,  the length of an arm or a leg is a constant not a variable, so you don't need repeat these values endlessly and the actual position is calculated in real-time frame by frame. Of course, all of this is recursive which short circuits my tiny brain.

Anyway, the main about BVH is that it is not possible to come up with a smaller method of recording motion than BVH. [I say this in the context of being a person often in the midst of people who understand mathematics - so wait and see awhile before accepting this assertion.]

Thing #2. Since the X, Y and Z information is all in the header. You can change this at any time. Even run time - and make the character morph as it's moving.  Thus you can fairly easily adapt a BVH file to different character sizes.

Thing #3. All the movement data is in an array of strings which contain the relevant angles. At runtime you can easily splice, pull or shift the array and update the character to have a new series of motions. So you could have a character moving about for twenty minutes but be, say, just twenty seconds ahead in terms of data that needs to be loaded.

Thing #4. The BVH is supported by Blender, Daz,  MakeHuman, Mixamo, FreeMocap and probably a number of other suppliers off 3D stuff. It's a fairly safe format. And the only commonly accepted format dedicated to motion.

Thing #5. The format is quite flexible. It can handle all the bones in the toes and fingers, or creatures with seven tentacles or just a robot arm with three moving parts. This does mean that there are a number of BVH 'dialects' out there, but my guess is that a good parser will eventually be able to identify the major types and adjust accordingly.

Thing #6. BVH data may be generated either via motion capture devices or by algorithm - and you can mix the two easily.

So is BVH perfect? Perhaps it is, but there is an issue. If BVH is the 'verb' - the thing that gets things moving, then what about the 'noun' the data that needs to be moved about? That is the subject of a whole story in itself and I will talk about this in an upcoming post.

In the mean, please enjoy the code that Saqoosha wrote to get your screen to dance:

Live demo: http://jaanga.github.io/cookbook/bvh-reader/r1/bvh-reader-saqoosha.html
Live demo: http://jaanga.github.io/cookbook/bvh-reader/r1/bvh-reader-saqoosha-cmu-daz.html
Live demo: http://jaanga.github.io/cookbook/bvh-reader/r1/bvh-reader-saqoosha-truebones.html

Details and source code here:
https://github.com/jaanga/cookbook/tree/gh-pages/bvh-reader

JavaScript App to Record, Save and Replay Leap Motion Gestures in 3D

The Leap Motion device leads the way into new computer interfaces designed for specifically for our hands. Why? Because all other devices (mice, pianos, steering wheels whatever) require that the hands touch, hold or manipulate some other things. The Leap Motion device is the first device that captures your hand and finger movements in a completely free and unfettered manner.

Being the first device of this kind has its issues. There is not a lot of software for the device. There are not many good tools for designing software. And there really isn't even a good idea as to what the best tools should be or should do.

Frankly, I think this is amazing. This is one of the very rare occasions when we have a 'green field site' that doesn't even have a green field.

So what fun things need too be addressed first? Well, one of the main things is that there is no way of recording the movement of your hands and fingers and then replying the gestures and being to read the numeric data as well as view a representation in 3D. And perhaps more interesting there is no simple, easy-to-understand FOSS method for recording motions.  Or maybe there is but I haven't seen it.

There was, however, a great first attempt. Kai Chung put together two code examples, 'Leap Motion Recorder' and 'Leap Motion Replayer', available from here:

http://js.leapmotion.com/examples.

There are issues with both the apps. The recorder app provides no method for saving data and the replayer app only replays just the one file it is hardwired to replay and ut has no method for selecting and opening files. And in terms of helping programmers, these apps are coded to work with an early beta version of the Leap Motion software and do not seem to work when linked to the current version of the JavaScript Leap library.

But now, as of today, there is 'jestPlay'. If you have a Leap Motion device, try out the app here:

http://jaanga.github.io/gestification/cookbook/jest-play/r1/jest-play.html

There is a replay only version in the works - so that people without the device can replay gestures. It should be available shortly.

The jestPlay app enables you to record your hand movements by saving data from the device to your computer as JSON files. Once saved, you can open these files and watch a full 3D replay of the movements.

The app is a 'cookbook' style app. It is not a fully-featured or even a finished app. It does, however, provide you with a simple working example in just over two hundred lines of very simple JavaScript code that you can use to start developing your own code.

The app provides full access to your operating systems file save and file open dialog boxes
- which are features not normally found in JavaScript as they were recently introduced in HTML 5.

Based on the Three.js library, the jestPlay app allows you to zoom, pan and and rotate the views of the replays - so from another person's point of view - you can see your handiwork .

Source Code
https://github.com/jaanga/gestification/tree/gh-pages/cookbook/jest-play

The Barfolina Pavilion: Towards a Procedural Architecture

The past week or so has been exhilarating because I have been able to churn out so much interesting code. But now I have material for a dozen blog posts. So there is going to be a battle between the coding fingers and the writing fingers. For the moment, the writing fingers are in charge.

Nico B wants an app that would allow people to 'fly'' over the harbor in Iquique Chile and he wants to do this using the motion controller from Leap Motion. How do you fly in an imaginary way - hundreds of feet over the harbor and down through the buildings - while using just one hand twirling and swirling inches above the Leap Motion device? Neither the Leap Motion sample software nor Three.js examples have anything that does exactly this. So we needed to come with the flying app ourselves. In order to build the app, we need landscape and building to practice with. The actual physical project in Iquique has issues, so we needed to come up with our own imaginary buildings.

The first simulation I came  up with was an imaginary landscape:
http://jaanga.github.io/gestification/projects/flying-leap-3d/r1/flying-leap-3d.html

In many ways, this was just fine. But really it was just too good. All you do is float around. It's actually quite difficult to get anywhere specific.

So then Nico found the data for this castle:
http://jaanga.github.io/gestification/projects/flying-leap-3d/castle/load-castle.html 

It may take a number of seconds for it to load. There were many issues here. The biggest issue has been getting the flying speed right, when you are in the castle it's too fast and when you are flying around it's too slow. And then the walls only have textures on the outside. When you go inside, the walls are invisible. So you think you are still outside. Therefore you keep on going and then you are outside without having known you were inside. And so on.

So the Nico came up with the Barcelona Pavilion - and he even sourced a Blender 3D file for it. Conceptually the Pavilion is a perfect place to lean to fly.  You can start in the landscape, move to the courtyard than try to negotiate the narrow passages. It was a perfect fit.
But the Blender file was missing its textures. I found two other Blender files. Again there were issues.

What to do? Build my own models using Blender or SketchUp or whatever? No way. I don't build stuff using tools I did not have a hand in building myself. So I built a 3D model of the pavilion using Three.js. Every floor, wall, window is a procedure.  The project took half a day. It's about 500 lines of code and is about 28K in file size. Simple, tiny and fast. Perfection.

Not really. As you fly through the building you'll see a dozen or so mistakes. That's because I mostly did things by eye and feel. It will take an hour or two to fix these, but I have no fear. Unlike the old-school CAD programs, there won't be broken walls that have to be 'fixed' or dozens of items that need changing because a height has changed.

The pavilion - which I call the Barfolina Pavilion - is viewable here:
http://jaanga.github.io/gestification/projects/flying-leap-3d/barfolina-pavillion/r1/barfolina-pavillion.html

It's all still at a Release 1.0 stage, but it was so much fun thus far that there will be many more releases with things like transparent roofs and people visiting the pavilion and maybe even en exhibit or two.

But working on this project make me think a lot of the buildings of the future. These building, as we all know, will be built and edited and updated continuously by robots. "The grandparents are coming. We need to add a guestroom." "Saturday is Tammy's birthday. Take down all the walls so there's room for the party." The robots will not want static databases of dimensions. The robots will want to know the program.

So in coding this 1929 building, perhaps I was designing for 2129...

Source Code:
https://github.com/jaanga/gestification/tree/gh-pages/projects/flying-leap-3d

Folding Polygons the Naughty Way: in 3D, with Intersections and a Video Veneer

My recent post on displaying video on wobbly surfaces amused ArtS of Meno Park CA and to make a long story short we had a delightful lunch together today. It's not often that I find anybody like Art in the meatworld that likes to talk about 3D.

We talked about many aspects of 3D. For example, a great introdution to JavaScript coding in 3D is here: http://www.mrdoob.com/projects/htmleditor/. Look for the word 'wireframeLinewidth' ( at the end of the long sentence in the middle) and change the '2' to an '8'. Bingo! You are a programmer.

And we talked a lot about tween.js - a brilliant way of morphing all matter of stuff in 3D and 3D.

But mostly we talked about displaying video on folding polygons, Folding polygons are things that look like the images in the link provide by Art to this book: http://graphics.berkeley.edu/papers/Iben-RPP-2006-06/. Basically, if you like origami then you like folding polygons.

Many of the discussions on folding polygons relate to morphing the polygons on a 2D plane
such that no vertex is 'naughty' and crosses over or intersects anybody else's line. This is certainly fun stuff. But even more fun - or fun in a different way - is the exploration of 'naughty' folding and 3D folding.

After lunch I built some demos that begin to explore the naughty bits.

Demo:
http://jaanga.github.io/cookbook/video-folding-polygons/r1/video-folding-polygons-5x5.html

This first demo is a version of the Three.js demo:
http://mrdoob.github.io/three.js/examples/#webgl_materials_video

The code is greatly simplified, and made suitable for being used as boilerplate for further apps/

The next demo is here:
http://jaanga.github.io/cookbook/video-folding-polygons/r1/video-folding-polygons-pixelated.html

Question: can you make a video with holes in it? This app shows the answer is 'yes!'

The fun thing here is the array that is used to layout the position of the holes. See below - if you look carefully you can see the word 'Art' spelled out. Now is that the Art I had lunch with or is it that thing that people do with chemicals and brushes? Who knows.

You can see that the array is laid out as 20 x10 grid - just as  the cubes in the grid are laid out. A 1 indicates inserting the cube. A 0 indicates leaving the cube out. I enjoyed this cute, ever so simple 'Art'istic method for creating a 'pixelated' video.

var pixels = [
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,
1,1,1,1,0,1,1,1,0,0,0,1,1,0,0,0,0,0,1,1,
1,1,1,0,1,0,1,1,0,1,1,0,1,1,1,0,1,1,1,1,
1,1,0,1,1,1,0,1,0,1,1,0,1,1,1,0,1,1,1,1,
1,1,0,1,1,1,0,1,0,1,1,0,1,1,1,0,1,1,1,1,
1,1,0,0,0,0,0,1,0,0,0,1,1,1,1,0,1,1,1,1,
1,1,0,1,1,1,0,1,0,1,1,0,1,1,1,0,1,1,1,1,
1,1,0,1,1,1,0,1,0,1,1,0,1,1,1,0,1,1,1,1,
1,1,0,1,1,1,0,1,0,1,1,0,1,1,1,0,1,1,1,1,
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1
];

Demo:
http://jaanga.github.io/cookbook/video-folding-polygons/r1/video-folding-polygons-deformed-planes.html

This is the actual 'naughty' folded polygon demo. You will note that the 'teeth; are splayed out in 3D
but if they were laid out flat the teeth would intersect. In other words, you could not cut this thing out of a single sheet of paper.

And then again, even if you could, you might also have some trouble displaying video on the impossibly cut sheet of paper.

If you asked me a year ago if a script-kiddie of my ability could code the display of a video on an impossible origami fold, I would have laughed. Actually, I am still laughing because the demo kind of sucks. Looking at the video from the back or from the side is vaguely interesting - for about four seconds. And sometimes the video feels a bit 3D-like.  But, frankly, I am happiest when it's all 3D through and through. But if you do have any good "Can you do this?" challenges, I would be delighted to hear about them. Art, I am looking at you.

Source Code:
https://github.com/jaanga/cookbook/tree/gh-pages/video-folding-polygons