Looking Glass Documentation
Buy Looking GlassDiscord
  • Looking Glass User Guide
  • NEW! โœจLooking Glass Goโœจ
  • Displays
    • ๐Ÿ”ฎLooking Glass Go
      • โšกGet Started with Looking Glass Go
      • ๐Ÿ“ฑUsing the Go Mobile App
      • ๐Ÿ”†Liteformsโ„ข
      • ๐Ÿ–ฅ๏ธStandalone and Desktop Modes
      • ๐Ÿ”ŒPower and LED States
      • ๐Ÿช„Default Playlist
      • โญ๏ธWhat's Next?
      • ๐Ÿ‘œAccessories
      • โš™๏ธAdvanced Playlist Creation
      • โš™๏ธManual Firmware Update
      • โ“Common Troubleshooting Issues
    • ๐Ÿ”ฎLooking Glass 16" Light Field Displays
      • ๐Ÿช„Looking Glass 16" Light Field Display (Landscape)
      • ๐Ÿช„Looking Glass 16" Light Field Display (Portrait)
      • โ“Common Troubleshooting Issues
    • ๐Ÿ”ฎLooking Glass 32" Light Field Displays
      • ๐Ÿช„Looking Glass 32" Light Field Display (Landscape)
      • ๐Ÿช„Looking Glass 32" Light Field Display (Portrait)
      • โ“Common Troubleshooting Issues
    • ๐Ÿ”ฎLooking Glass 65" Light Field Display
    • ๐Ÿ”ฎLooking Glass Portrait
      • โšกGet Started with Looking Glass Portrait
      • ๐ŸŽ‡Demo Holograms
      • โ“Common Troubleshooting Issues
  • Software Tools
    • ๐Ÿ“šOverview
    • ๐ŸŒ‰Looking Glass Bridge
      • โœจUsing Looking Glass Bridge
      • ๐ŸชŸDisplay Settings on Windows
      • ๐ŸŽDisplay Settings on macOS
      • ๐ŸงDisplay Settings on Linux
      • ๐Ÿ‘ทTroubleshooting
    • ๐Ÿ“ฝ๏ธLooking Glass Studio
      • ๐Ÿ“šUsing Looking Glass Studio
      • RGB-D Photo / Video
      • Quilt Images / Video
      • iPhone Portrait Photos
      • Android Depth Photos
        • Samsung Portrait Mode
      • Light Field Photosets
      • Exporting Holograms
      • Troubleshooting
    • ๐ŸงŠLooking Glass Blocks
      • โœจConvert any image into a hologram
      • โฌ†๏ธUploading, editing and sharing Quilts
      • ๐ŸฅฝViewing your holograms on different platforms
    • ๐ŸŽฌHologram Video
    • ๐Ÿ”†Liteformsโ„ข
      • ๐ŸชŸGet Started on Windows
      • ๐ŸŽGet Started on MacOS
      • ๐Ÿง™Making Custom Liteforms
      • ๐ŸชConfigure Liteforms for Kiosk Setup
    • ๐Ÿ‘€Looking Glass Model Viewer
    • ๐ŸงชExperiments
      • ๐Ÿ“ฝ๏ธMedia Encoder
      • โฉLooking Glass Media Encoder
  • Key Concepts
    • ๐Ÿ”‘Hologram 101
      • ๐ŸŽ‰Making great holograms
      • ๐ŸŒŽHolograms in the real world
      • ๐Ÿ‘พHolograms, but virtual
    • ๐Ÿ”How does Looking Glass Work?
    • ๐Ÿ“ธVirtual Camera
    • ๐ŸงŠ3D Design Guidelines
      • Lighting
      • Post-Processing
      • Scene Composition
      • Materials & Textures
      • Performance
    • ๐Ÿ’กQuilts
      • ๐ŸŽฌQuilt Video Encoding
    • ๐Ÿ’กLight Fields
      • โšกLinear Light Field Capture
    • ๐Ÿ“ฝ๏ธFilming a Looking Glass
  • Creator Tools
    • ๐ŸŽจOverview
    • ๐Ÿ”ŒUnity
      • โšกUsing Unity with Looking Glass
      • Setting up your Development Environment
      • Example Scenes
      • Prefabs
        • Hologram Camera
        • 3D Cursor
        • Dual Monitor Application
      • Script Reference
        • HologramCamera.cs
        • LKGDisplaySystem.cs
        • QuiltCapture.cs
        • Cursor3D.cs
        • OrbitControls.cs
        • BlockUploader.cs
        • MobileDMAController.cs
        • iOS Sample Scene Scripts
          • ChangeInteractionModality.cs
          • XYSlider.cs
          • XYSpotlight.cs
          • AudioRotateControl.cs
          • ModelController.cs
          • DemoIOSUIController.cs
          • PageHandler.cs
          • PageConnectHandler.cs
          • PageCalibrateHandler.cs
          • PageTestHandler.cs
          • DynamicAlcove.cs
      • Package Integrations
        • Built-in Render Pipeline Post-Processing
      • Developing for iOS
      • Depth of Field and Other Post-Processing Effects
      • Quilt Video Audio and Playback
      • Developing for Ultraleap Hand Tracking on Looking Glass
      • Looking Glass Pro (Legacy)
      • Additional Support
      • Change Log
    • ๐ŸนBlender
    • ๐Ÿ”ŒUnreal Engine
      • ๐Ÿ—’๏ธChange Log and Old Versions
      • ๐Ÿ’ฝInstalling the Unreal Engine plugin
      • ๐ŸŽฅUsing the Unreal Engine Plugin
        • Using the HoloPlay Capture
        • Exporting Quilt Images and Video with Sequencer
      • Accessing HoloPlay Data with Blueprint Scripting
      • Packaging the Project
      • Unreal Engine 4 (Legacy Support)
        • Unreal Engine 4 Quick Start
          • Using the HoloPlay Capture
          • Exporting Quilt Images and Video with Sequencer
    • ๐ŸŒWebXR
      • ๐Ÿ”ถPlayCanvas
      • 3๏ธโƒฃ3๏ธโƒฃ Three.JS
      • โš›๏ธReact-Three-Fiber
      • ๐ŸŒˆSpline
  • Tutorials
    • ๐Ÿ“ฑRunning Looking Glass Displays with iOS Devices
    • โ“‚๏ธCapturing Light Fields in Maya
    • ๐Ÿง‘โ€๐Ÿ”ฌUsing ShaderToy with Looking Glass
    • ๐ŸŽจSculpting with Adobe Medium and Looking Glass
    • ๐Ÿ”Turn Videogames Holographic with Refract
    • ๐Ÿ–๏ธBuilding with Leap Motion
  • Developer Docs
    • โš™๏ธLooking Glass Bridge SDK
      • ๐Ÿช›Web Application Integration
      • ๐Ÿ”ŒIntegrating Native Applications
      • ๐Ÿ““Native Function Reference
    • ๐ŸฆLooking Glass Core Migration Guide
  • Third Party Software
    • ๐Ÿ—ก๏ธ3D Slicer
    • ๐Ÿ”ฌVTK (Visualization Toolkit)
    • ๐Ÿ—บ๏ธParaView
    • ๐Ÿ”ฉCommunity Made Tools & Projects
  • Legacy Software
    • ๐Ÿ“ก3D Model Importer
      • 3D Model Importer App Guide
      • 3D Model Importer App Reference
    • ๐ŸŽญDiorama
      • Change Log
    • ๐ŸŽž๏ธDepth Recorder
    • ๐Ÿ“ธLightfield Photo App
    • ๐ŸLooking Glass Core Overview
    • ๐Ÿฝ๏ธLooking Glass Core SDK
      • C API Overview
        • HoloPlayCore.h
          • State Functions
          • General-Query-Functions
          • Device-Query-Functions
        • libHoloPlayCore.h
        • HoloPlayShaders.h
    • ๐Ÿ”ฐLooking Glass Core JS
      • API
        • HoloPlayCore.Client
        • HoloPlayCore.Message
        • HoloPlay Service Related
        • examples
  • Legacy Hardware
    • ๐Ÿ”ปLooking Glass 8.9"
    • ๐Ÿ”ปLooking Glass 15.6"
    • ๐Ÿ”ปLooking Glass 8K
    • ๐Ÿ”ปLooking Glass 16"
    • ๐Ÿ”ปLooking Glass 32"
    • ๐Ÿ”ปLooking Glass Pro
  • ๐Ÿ’ฌContact Us
  • ๐Ÿ› ๏ธFAQ & Troubleshooting
Powered by GitBook

Follow us

  • Twitter
  • Instagram
  • LinkedIn
On this page
  • How does the Leap Motion Controller work?
  • What is Ultraleap hand tracking good for in the Looking Glass?
  • The LEAP runtime
  • The LEAP SDK
  • LeapServiceProvider/Controller
  • The tracking space
  • The hands
  • Graphical hands:
  • Physics hands:
  • Attachment hands:
  • Pinching:
  • Final notes

Was this helpful?

  1. Creator Tools
  2. Unity

Developing for Ultraleap Hand Tracking on Looking Glass

PreviousQuilt Video Audio and PlaybackNextLooking Glass Pro (Legacy)

Last updated 1 year ago

Was this helpful?

This tutorial is provided for your convenience/rapid experimentation, but we do not regularly update the samples. As such, we highly recommend you follow the for production.

First, make sure you have all the right software:

How does the Leap Motion Controller work?

Unlike depth cameras like the Kinect or RealSense, the Leap Motion Controller doesn't actually generate a depth map. It does also use infrared emitters and receivers, but to get an isolated, high-contrast image of the hands. It then feeds the last few frames into some pretty amazing computer vision operations to get a feasible idea of what all your hand bones are doing. That happens in the continuous LEAP service (runtime), which the LEAP Unity SDK then queries for tracking information. The SDK itself is responsible for higher-level tasks like generating the colliders, meshes, pinch detection and so on.

What is Ultraleap hand tracking good for in the Looking Glass?

The most straightforward use case for hand tracking may also be the most powerful: using the collider hands to interact with simple 3D physics never gets old. Sophisticated interaction is possible with (relatively solid) pinch detection and contextual UI's that pop out and attach to your hands.

The LEAP runtime

If it doesn't appear to be running in the tray, you can search for a "Leap Motion Control Panel" program, which should start it up. The icon turns green when the LEAP is detected, and you can double click it to open the control panel. If your Leap Motion Controller has been sitting around since 2013 (or if you just want to make sure it's performing optimally) I'd advise using the calibrator on the troubleshooting tab. If you ever want to check the version, that's in the About tab.

The LEAP SDK

I'd strongly recommend downloading , which I've configured to be a pretty optimal starting point for LKG + Ultraleap. You could also start from one of the example scenes that come with the LEAP SDK, but most are set up for VR, with some stuff we don't want. Just don't even try to set it all up from scratch, that's asking for trouble!

To get it running, import the LEAP SDK, HoloPlay Unity Plugin, as well as the example scene all into one Unity project. Then open the LEAP+LKG scene. You should see this:

LeapServiceProvider/Controller

The central component of the LEAP SDK is LeapServiceProvider.cs, which usually lives on a "Leap Motion Controller" gameobject. Note this is distinct from LeapXRServiceProvider.cs, which is for headset-mounted tracking instead. The service provider draws a cone gizmo with white lines representing the tracking space of the LEAP, up to 2-3 feet away or so in meatspace. Scaling the controller object scales the cone and thus the range of motion of the hands in the scene. If the hands are a child of this object then everything will scale uniformly allowing you to make the hands fit your scene.

The tracking space

As for positioning and scaling the tracking space, we generally let the visible area of the Looking Glass correspond to about a foot of physical height, starting at about 4 inches from the LEAP (the minimum for solid tracking). A more subtle consideration is the angle relative to the Looking Glass, which sits at a 61 degree angle to the table. Picture it this way: when moving your meat hands around on a plane parallel to your table, do you want the 3D hands to look like they're moving parallel to the table as well, or parallel to the bottom of the Looking Glass block? A conceptually accurate -29 degree offset always feels like too much for some reason, so I usually set it to somewhere in between, -15 degrees or so.

Angledย 0ย degrees

Angledย 15ย degrees

Angledย 30ย degrees

If you're moving or animating the HoloPlay Capture, you can make the controller a child of it to have the hands stay localized in-frame as the camera moves around. Keep in mind that changing the capture size will require you to manually scale and reposition the controller (and the pinch distance, see below).

The hands

The hands in the example scene are made up of a few separate components:

  • Graphical (meshes, renderers)

  • Physical (colliders, rigidbodies)

  • Attachments (transforms for connecting things to joints)

  • Pinch detector (pinchy pinch)

Graphical hands:

There are basic Capsule Hands made up of primitives, and the somewhat nicer rigged meshy ones called LoPoly Rigged Hands, which I use for everything.

Physics hands:

There are hands that use box colliders and hands that use capsule colliders. I think capsules are almost always superior, unless you happen to have some particularly boxy fingers? Some of the prefabs contain wrist/arm colliders too, but only the capsule hands can actually draw meshes for the wrist/arm. I'd suggest simply deleting the "Forearm" collider objects if you're using the LoPoly Rigged Hands. NOTE: If you do ever want to use these alternate prefabs, pull one of each hand (L/R) into the scene and replace the old references on the HandModels object.

Attachment hands:

These are very useful if you want to have something following or pinned onto a specific hand joint. ALWAYS USE THIS instead of trying to attach something to the graphics or physics hands, sometimes it re-generates those so any attachments might get lost.

The Attachment Hands component has a nice little window where you can toggle which bones have attachments, which summons new objects into existence. Any children you add to these will then be attached, and will pivot around the attachment point unless you zero out their local position and rotation.

In many cases, the palm transform point is the smoothest/most reliable, but there's plenty of times when you want something to follow or interact with only a fingertip. For that sort of thing, It's usually way easier to set up specific colliders and functionality on an attached object rather than on a part of the hand prefab itself.

There's also a script called AttachmentHandEnableDisable.cs which is critical in many cases- it enables/disables the attachments to match the state of the hand they're on. It's most often preferable to turn off the attachment when the hand disappears, rather than having it stick to the last tracked position. You need one for each hand, and they each need a reference to a hand. Yes, I do think this simple functionality should just be a bool in AttachmentHands.cs instead.

Pinching:

PinchDetector.cs references a HandModel (either physics or graphics) and triggers UnityEvents upon pinch and release. The trigger distance is in world space and you'll need to change it if you scale the controller. If the "ControlsTransform" bool is true, the PinchDetector object will update to move around relative to the hand. This is a useful way to get the pinch position, the midpoint between the thumb and index fingertips.

Final notes

  • The attachment and pinch scripts don't actually need to be their own objects if you prefer to condense the whole setup.

  • There's a key binding in the LEAP SDK that will override the HoloPlay Capture's ctrl+E toggle binding.

  • We've occasionally seen the Leap Motion Controller struggle when going through USB hubs or questionable cables.

  • It's also sensitive to infrared blasts from other equipment (vive lighthouses, depth cameras, roombas) and from direct sunlight, so keep that in mind.

  • When installing the Leap Motion Controller on a desk or horizontal surface, in order to prevent malfunction, check "Auto-orient Tracking" in the Leap Motion Control Panel. This setting will assume the LED is at the front. Also, from the Visualizer, ensure that you are in "Desktop mode".

๐Ÿ”Œ
documentation provided by Leap Motion
The LEAP runtime installed (or updated to Gemini 5.0.0)
LEAP SDK (aka core assets)
Example scene including the full configuration of features we'll go over
the example scene
leap and lkg
hand attachments
15 degrees
30 degrees
0 degrees