1. Oculus Rift DK2 + Leap Motion Unity Tutorial
Chris

2. Content
Leap Motion VR
Hand tracking using Leap SDK V2 (Skeletal Tracking)
Object interaction
Movement using Rift DK2 positional tracking

3. Introduces skeletal tracking
Leap Motion SDK V2
Currently in beta
Introduces skeletal tracking

4. Leap Motion VR Normally, need Leap mount to mount Leap to DK1/DK2.
Can also use blue tac or other methods.

5. Augmented Reality Image API Infrared & Night Vision
Enable in Leap Motion Control Panel > Settings > General > Allow Images
Leap Oculus Passthrough Unity Package
Next Leap Motion may have colour camera
Leap Oculus Passhthrough Unity package allows integration of the infrared image passthrough in Unity. Can use to blend or show real world in VR.

6. Requirements IDE Leap Rift DK2 Unity Pro Leap Motion Driver
Leap Motion V2 Tracking Beta (v )
Leap Motion V2 Skeletal Assets (beta)
Rift DK2
Unity 4 Pro Integration
Need to use Unity Pro instead of free due to Leap integration using external plugins.
Leap Motion V2 Skeletal Assets (beta) – found on Unity Assets store.

7. Scope
Map physical hand and finger movements to 3D hand model and physics
Grab physics objects (rigidbodies) by pinching
Movement and jump using DK2’s positional tracker

8. Leap Motion - Unity File Structure
Materials
Blinn2.mat
Gloves.mat
Hands_Alt_blinn1.mat
Models
HandsRealistic.fbx
Prefabs
Hand Graphics
RigidLeftHand.prefab
RigidRightHand.prefab
Hand Physics
ThickRigidHand.prefab
HandController.prefab
Pick these files from the Leap Unity Assets package.
Folders are bolded.
Even some files included here you may not need.

9. LeapMotion … Scripts Hand Tools Utils HandController.cs FingerModel.cs
HandModel.cs
RiggedFinger.cs
RiggedHand.cs
RigidFinder.cs
RigidHand.cs
SkeletalFinger.cs
SkeletalHand.cs
Tools
ToolModel.cs
Utils
LeapRecorder.cs
LeapUnityExtensions.cs
MagneticPinch.cs
HandController.cs

10. Plugins
[Include all files]

11. Attach hands to player
Assumed that you’ve already integrated Rift with player
Is compatible with Sixense’s Razer Hydra integration via script (see LeapHandExtendController.cs in tutorial)

12. Create empty GameObject called “LeapHandController” and place under OvrCameraController.
Either attach the HandController prefab to game object or attach HandController.cs script.
Fill out variables with the following:

13. Setup RigidLeftHand + RigidRightHand prefabs
Check if both prefabs are set as follows or change accordingly.
Red underline are suggested variables where the prefab’s may have different values than we want.
Green underline is a suggestion.
MagneticPinch.cs script allows user to
grab rigidbodies by pinching fingers

14. Each finger must have the following variables, based on type of finger (i.e. index, pinky) and if it’s left (starts with L) or right (starts with R):
For right hand, choose the [..]Arm3 mesh

15. Setup ThickRigidHand prefab
Gives the hand collisions and physics
Can therefore push objects

16. Magnetic Pinch Script
Allows user to grab closest object which has a rigidbody
Force Spring Constant sets the elasticity of the grip to object
Magnetic Distance determines how close an object must be from a hand for it to be grabbed
The magnetic pinch and hand physics conflict is due to the magnetic pinch script programmed to find the closest rigidbody which isn’t the hand’s graphics model which happens to be the physics hand model in this case.
Warning: There is a conflict between Magnetic Pinch and enabling the Left/Right Physics Models. You’d need to fix this conflict by making the magnetic script ignore the hand’s physics

17. Grab Hand and Grabbable scripts
As an alternative to Magnetic Pinch script
Assign GrabHand.cs to hand graphic prefab
Assign Grabbable.cs to rigidbody object to be grabbed
Haven’t yet experimented much with this one so I can’t add much detail yet.

18. Hand Physics Interact with objects in a realistic way
Grab small objects with one hand
Grab larger objects with multiple hands
Push/Pull other objects
Other possibilities
Grabbable objects must have colliders
Currently, grasping objects with hands is still quite jittery. Future SDK or Leap hardware updates should improve on this hopefully.

20. Handle conflicts with other hand trackers
This approach will hide/disable Sixense hands if leap motion detects leap hands in scene
Start by extending Leap Motion’s HandController.cs

21. LeapHandExtendController.cs (1)
using UnityEngine; using System.Collections; using Leap; public class LeapHandExtendController : HandController { protected Controller leap_controller_; protected void Awake () { leap_controller_ = new Controller(); }

22. LeapHandExtendController.cs (2)
protected void LateUpdate () { if (leap_controller_ == null) return; CheckIfHandsEnabled (); }
Currently we can’t override HandController’s Update() function, so we’ll use the LateUpdate() function instead.

23. LeapHandExtendController.cs (3)
protected void CheckIfHandsEnabled () { Frame frame = leap_controller_.Frame(); HandList hands = frame.Hands; int num_hands = hands.Count; if (num_hands > 0) { TriggerSixenseHands (false); } else { TriggerSixenseHands (true); }
Check the amount of leap hands visible, and if there are more than zero then disable Sixense (or any other) hand trackers.

24. Movement - DK2 Positional Tracking
Move/Jump by using DK2’s positional tracking
Move forward or backward by moving head in those directions
Either rotate or strafe sideways by moving head left/right
Jump by popping head upwards
Could crouch too by popping head downwards
User should normally be positioned directly in front of DK2’s tracker

25. Logic - Positional Tracking Movement
Create/modify your player input controller script, attached to player
Head position is calculated by subtracting the initial local position of the main camera, of when scene is loaded, from its current position
Create configurable Vector3 variables for:
Sensitivity – strength multiplier of movement or jump actions
Minimum – The minimum position that the Rift must be away from the centre position, for the action to actually be triggered
Movement will use the X (-left/+right) and Z axis (+forward/-backward) in the direction of the camera as a variable sent to the Character Motor’s inputMoveDirection variable
Jump will use the Y axis and will be mapped to the Character Motor’s inputJump variable
Vector3 mapping to movements:
Left = negative X
Right = positive X
Forward = positive Z
Backward = negative Z
Jump = positive Y
(Additionally) Crouch = negative Y

27. Code – Positional Track Movement
[RequireComponent(typeof(CharacterMotor))] public class FPSInputController : MonoBehaviour { … public bool ovrMovement = false; // Enable move player by moving head on X and Z axis public bool ovrJump = false; // Enable player jumps by moving head on Y axis upwards public Vector3 ovrControlSensitivity = new Vector3(1, 1, 1); // Multiplier of positiona tracking move/jump actions public Vector3 ovrControlMinimum = new Vector3(0, 0, 0); // Min distance of head from centre to move/jump public enum OvrXAxisAction { Strafe = 0, Rotate = 1 } public OvrXAxisAction ovrXAxisAction = OvrXAxisAction.Rotate; // Whether x axis positional tracking performs strafing or rotation private GameObject mainCamera; // Camera where movement orientation is done and audio listener enabled private CharacterMotor motor; // OVR positional tracking, currently works via tilting head private Vector3 initPosTrackDir; private Vector3 curPosTrackDir; private Vector3 diffPosTrackDir;
Note – entire code for class is not here, only relevant code. If you’d like some help on this, feel free to contact
mainCamera needs to be declared as either the only camera following the player, or one of the 2 Oculus Rift cameras.

28. Start + Update
void Start() { … initPosTrackDir = mainCamera.transform.localPosition; } void Update() { // Get the input vector from OVR positional tracking if (ovrMovement || ovrJump) { curPosTrackDir = mainCamera.transform.localPosition; diffPosTrackDir = curPosTrackDir - initPosTrackDir;
Set current tracking position as the camera’s local positon, then calculate the difference position by subtracting the initial camera position from the current.

29. Update (continued)
… if (ovrMovement) { if (diffPosTrackDir.x <= -ovrControlMinimum.x || diffPosTrackDir.x >= ovrControlMinimum.x) { if (ovrXAxisAction == OvrXAxisAction.Strafe) { diffPosTrackDir.x *= ovrControlSensitivity.x; } else { transform.Rotate(0, diffPosTrackDir.x * ovrControlSensitivity.x, 0); diffPosTrackDir.x = 0; }
If head position is further than the minimum from the centre, then set the direction to move multiplied by sensitivity, otherwise do not move in that direction.

30. Update (continued)
… if (diffPosTrackDir.z <= -ovrControlMinimum.z || diffPosTrackDir.z >= ovrControlMinimum.z) { diffPosTrackDir.z *= ovrControlSensitivity.z; } else { diffPosTrackDir.z = 0; } directionVector = new Vector3(diffPosTrackDir.x, 0, diffPosTrackDir.z);
directionVector is used afterwards to set the movement directions and strength.

31. Update (continued)
if (ovrJump) { if (diffPosTrackDir.y > ovrControlMinimum.y) { motor.inputJump = true; } else { motor.inputJump = false; } motor.inputJump = Input.GetButton ("Jump"); … motor.inputMoveDirection = mainCamera.transform.rotation * directionVector; // performs actual movement
If jump via positional tracking :
Enabled, then trigger jump action if camera’s difference position is above the set minimum Y axis position
Disabled, then map jump keyboard key to jump action

32. }