So Close / Yet Still So Far: Skin and Bones for Leap Motion Devices - A Progress Report

Hand image from Leap Motion documentation

2013-09-22: See also update post that discusses much improved Phalges R5:
http://www.jaanga.com/2013/09/skin-and-bones-for-leap-motion-devices.html  


The above image is from the documentation for the Leap Motion device. Questions relating as how to produce such images or how to access the 'raw data' that produces such images are some of the most frequently asked questions in the Leap Motion forums. The bad news is that there is no source code or coding examples currently provided by Leap Motion for producing such a display.

The good news is: Wow! What an excellent coding challenge...

This post is a progress report on the current status to produce realistic-looking and behaving hands that can be controlled by the Leap Motion device.

The most exciting recent development is certainly this recent post by Roman Liutikov:

http://blog.romanliutikov.com/post/60899246643/manipulating-rigged-hand-with-leap-motion-in-three-js

With demo file here:

http://demo.romanliutikov.com/three/10/

Roman provides very clear guidance as how to export skin and bones from Blender as a JSON file that can be read by Three.js and used to display arbitrary, real-time finger movements generated by a Leap Motion device.

An interesting side note is that the code uses a BVH-like structure to control the movement of the fingers. I recently wrote about the importance and efficacy of BVH here:

http://www.jaanga.com/2013/09/bvh-format-to-capture-motion-simply.html

The unfortunate aspect of this work is that there are a number of issues with the movement of the hand and fingers.

Nevertheless, this code is an important step forward and well worth inspecting.  I did so myself and have re-written Roman's code in my own (admittedly somewhat simplistic) style:

Demo: http://jaanga.github.io/gestification/work-in-hand/phalanges/liutikov/liutikov.html

With information and background here:

https://github.com/jaanga/gestification/tree/gh-pages/work-in-hand/phalanges/liutikov

My own work, since the publication of the post on BVH, has been involved with building up a notion of the best methods for positioning and angling the 'bones' inside the fingers. There are a host of issues - too many to list here - including: hands that sometimes have five fingers, or two fingers or no fingers; finger 2 easily switches places with finger 3; the order of the fingers is 4, 2, 0, 1, 3 and so on.

The latest demo (R4) is here:

http://jaanga.github.io/gestification/work-in-hand/phalanges/r4/phalanges.html

Previous releases, source code and further information is available here:

https://github.com/jaanga/gestification/tree/gh-pages/work-in-hand/phalanges

Much is working: the hand generally moves and rotates appropriately, fingers stay in the same position and don't disappear. But it is readily apparent that the tips of the fingers are still quite lost in space.

Not to worry. Eventually the light bulb will turn on. Actually the more likely thing is that a search on Google will turn up an answer or some person very smart in the ways of vectors will respond on Stackoverflow.

Also worth noting is that the people at Leap Motion gave a demo of routines at the recent developer's conference in San Francisco that may provide a satisfactory response. The interesting thing will be to see which code come out first and which code is the more hackable.