Skeletal Tracking with JavaScript and the Leap Motion

When it comes to using JavaScript with next generation technology, the Leap Motion is one of my favourite devices to work with. With the Leap Motion, JavaScript is not an after thought. It isn’t a third party module hacked together by enthusiasts – it is developed with care and consideration by the Leap Motion team with each new API release. Their latest release of V2.1, currently in beta, is pretty darn exciting. In this article, I’ll give an overview of the new capabilities available to JavaScript developers.

Key Takeaways

• The Leap Motion API V2.1, currently in beta, provides JavaScript developers with new capabilities for skeletal tracking, including additional hand data and finger data.
• The API includes ‘hand.confidence’ which provides a value to indicate how certain the Leap Motion device is of its data. It also includes ‘hand.type’ to determine if it’s a left or right hand, ‘hand.pinchStrength’ to measure how close together the thumb and finger are, and ‘hand.grabStrength’ to represent whether the hand is open or closed.
• The Leap Motion API provides detailed finger data, including the type of each finger and whether it’s extended or not. It also provides data on the four finger bones in each finger, ordered from wrist to the tip of each finger.
• While the V2 Leap Motion API also supports VR headsets like the Oculus VR and provides access to the actual image data from the Leap Motion, these features are not yet available in the JavaScript API.

Keep Track of Those Hands with Greater Confidence

The API provides additional data for each overall hand. Hand data is available in frame.hands just as it was in the previous API, however in the updates we get a wonderful collection of new data that will make life easier for Leap Motion devs.

Hand Confidence

One of my favorite inclusions in the new API is the ability to know how certain the Leap Motion is of its data. hand.confidence provides a value from 0 to 1 which lets you know how certain what it is seeing matches the model of a hand it understands. Low values could mean it is not certain of overall hand data or it could mean it is uncertain of finger/bone positions. This allows you to check how sure the Leap Motion is before taking an action. I’d love to see this lead to much smoother interactions and hopefully encourage users to be very clear in their gestures.

Hand Type

hand.type works out whether the Leap Motion thinks it is seeing a left hand or a right hand. If your hand is in a fist, then it’s gonna struggle a bit with this distinction.

Pinch Strength

hand.pinchStrength is a wonderful new property which gives a value from 0 to 1 which represents how close together the thumb and finger are to each other. This could open up some interesting possibilities for gestures and controlling apps as a pinch is a nice and natural human gesture.

Grab Strength

hand.grabStrength returns a value from 0 to 1 just as pinch strength does, but it represents whether your hand is open flat or closed into a fist. The possibilities here are equally as exciting – Get your users to squeeze a virtual stress ball or grab and toss something across a virtual room.

New Finger Data

In the new skeletal API changes, every hand has always got five fingers and you can get a whole new set of data for each finger the Leap Motion can see.

Finger Type

You can read in what sort of finger each bit of data represents using finger.type. This object will be a value from zero to four:

• 0 is a thumb
• 1 is an index finger
• 2 is a middle finger
• 3 is a ring finger
• 4 is a pinky finger

Each time Leap Motion reads in hand data, it reads it in with all five fingers.

Finger Extended

If some fingers aren’t pointing out above the Leap Motion, the device records them as not extended using finger.extended = false.

As per the name “Skeletal”, we get data down to the bone

I’m not well versed in the bone structure of the hand, so I had the chance to learn something new here. My doctor would be proud of my expanded knowledge. Here’s hoping the metacarpus comes into casual conversation someday.

The Leap Motion provides data on the four finger bones in each finger, ordered from wrist to the tip of each finger: metacarpal, proximal phalanges, intermediate phalanges and distal phalanges. That makes no sense to most of society, so here’s a lovely diagram I found on Wikipedia Commons:

Finger bone details

It goes into more detail with each bone (I’ve arranged these from the wrist to the tip of the finger):

• finger.carpPosition – The x, y and z position of the base of the metacarpal bone.
• finger.mcpPosition – The position of the finger’s knuckle – the spot between the metacarpal and proximal phalanx bone.
• finger.pipPosition – The position of the spot between the proximal and the intermediate phalanges.
• finger.dipPosition – The position of the base of the distal bone where it connects to the intermediate phalanx.

It also has objects for each finger bone with more detail – finger.metacarpal, finger.proximal, finger.intermediate and finger.distal. Each one of these is represented in the Leap Motion API as a bone with the following details:

• bone.type – a value from 0 to 4 representing the type of bone:

• 0 – Metacarpal

• 1 – Proximal phalanx
• 2 – Intermediate phalanx
• 3 – Distal phalanx

• 4 – Forearm

• bone.basis – provides a two dimensional array that represents the vector of the bone. You can do some pretty crazy math with this to detect particular finger positions. I’ll need to fit in some mathematics practice before I attempt to use these values but it’s brilliant the potential is there!

• bone.prevJoint and bone.nextJoint – the coordinates of the end of the bone closest to the wrist and the end of the bone closest to the finger tip respectively.
• bone.length – the length of that bone in millimeters.
• bone.width – Leap Motion describes this best in their docs as “the average width of the fleshy part surrounding this bone”.
• bone.left – whether the bone is on the left hand (if not, then it’s on the right hand, there doesn’t seem to be a bone.right)
• bone.center – the midpoint of the bone
• bone.direction – the direction vector that the bone is in

Basically, that’s a good amount of data right there.

That Data in a Demo with Lots of Numbers

I threw together a quick demo to show just how much co-ordinate data is streaming through this API. The following demo shows the co-ordinates for each of the finger.carpPosition, finger.mcpPosition, finger.pipPosition and finger.dipPosition, along with basic data that is retrieved for the hand and fingers. It’s only a small fraction of the data available too!

The demo looks a bit like this:

You can try it out in action with your Leap Motion here.

There’s More to Come!

There is more to the V2 Leap Motion API than just these features, it also has support for VR headsets like the Oculus VR and also provides access to the actual image data from the Leap Motion. Sadly, these features aren’t available in the JavaScript API just yet but hopefully we’ll see them soon before the API leaves beta!

Other Resources

If you’re looking for more details on getting started with these APIs, the best place to go is the Leap Motion developer site:

• JavaScript SDK Documentation
• A gallery of Leap Motion JavaScript demos, check out the source code for some good insight!

Frequently Asked Questions about Skeletal Tracking with JavaScript and Leap Motion

How does Leap Motion work with JavaScript for skeletal tracking?

Leap Motion is a hardware device that captures the movement of the human hand in 3D space. It works with JavaScript through the LeapJS library, which provides a simple and intuitive API for accessing the Leap Motion data. This allows developers to create interactive applications that can track and respond to hand movements. The LeapJS library provides methods for tracking the position, velocity, and orientation of the hands, fingers, and tools, as well as for detecting gestures such as swipes, circles, and taps.

What are the system requirements for using Leap Motion with JavaScript?

To use Leap Motion with JavaScript, you need a computer with a USB 2.0 port, running Windows 7 or later, Mac OS X 10.7 or later, or Linux. You also need the Leap Motion software, which includes the LeapJS library. The software can be downloaded from the Leap Motion website. In addition, you need a web browser that supports WebGL, such as Chrome, Firefox, or Safari.

How can I install and set up Leap Motion for use with JavaScript?

To install and set up Leap Motion for use with JavaScript, first download and install the Leap Motion software from the Leap Motion website. Then, connect the Leap Motion controller to your computer with a USB cable. The Leap Motion software will automatically detect the controller and start tracking. To use Leap Motion with JavaScript, include the LeapJS library in your HTML file with a script tag.

What kind of applications can be developed using Leap Motion and JavaScript?

Leap Motion and JavaScript can be used to develop a wide range of applications, from games and virtual reality experiences to educational tools and scientific visualizations. The ability to track hand movements in 3D space opens up new possibilities for user interaction, allowing users to control applications in a more natural and intuitive way.

How accurate is the skeletal tracking provided by Leap Motion and JavaScript?

Leap Motion provides highly accurate skeletal tracking, with a precision of up to 0.01mm. It can track the position and orientation of each finger and the palm of the hand, as well as detect gestures such as swipes, circles, and taps. The LeapJS library provides a simple and intuitive API for accessing this data, making it easy to incorporate into JavaScript applications.

Can I use Leap Motion with other programming languages besides JavaScript?

Yes, Leap Motion provides SDKs for several programming languages, including C , C#, Java, Python, and JavaScript. This makes it a versatile tool for developers, allowing them to choose the language that best suits their needs and skills.

What are the limitations of using Leap Motion with JavaScript?

While Leap Motion and JavaScript provide a powerful tool for developing interactive applications, there are some limitations to be aware of. The Leap Motion controller requires a clear line of sight to the hands, and can be affected by bright light or reflective surfaces. In addition, while the LeapJS library provides a simple and intuitive API, it may not offer the same level of control and flexibility as the SDKs for other languages.

How can I troubleshoot issues with Leap Motion and JavaScript?

If you’re having trouble with Leap Motion and JavaScript, there are several steps you can take. First, make sure the Leap Motion controller is connected and the Leap Motion software is running. If the controller is not being detected, try unplugging it and plugging it back in. If you’re having trouble with the LeapJS library, make sure you’re using a compatible web browser and check the JavaScript console for errors.

Can I use Leap Motion with JavaScript on mobile devices?

Currently, Leap Motion does not support mobile devices. The Leap Motion controller requires a USB connection, which is not typically available on mobile devices. However, Leap Motion is continually developing and improving their technology, so this may change in the future.

Where can I find more resources and tutorials on using Leap Motion with JavaScript?

There are many resources available online for learning how to use Leap Motion with JavaScript. The Leap Motion website provides a wealth of information, including a developer guide, API documentation, and a community forum. In addition, there are numerous tutorials and examples available on sites like GitHub and Stack Overflow.

The above is the detailed content of Skeletal Tracking with JavaScript and the Leap Motion. For more information, please follow other related articles on the PHP Chinese website!