1. Gesture recognition using salience detection and concatenated HMMs Ying Yin yingyin@csail.mit.edu Randall Davis davis@csail.mit.edu Massachusetts Institute of Technology

2. System overview Depth & RGB images Gesture spotting & recognition Hand tracking Hand movement segmentation Xsens data Feature vector sequence Feature vector sequence with movement

3. System overview Depth & RGB images Gesture spotting & recognition Hand tracking Hand movement segmentation Xsens data Feature vector sequence Feature vector sequence with movement

4. Hand tracking Kinect skeleton tracking is less accurate when the hands are close to the body or move fast We use both RGB and depth information Skin Gesture salience (motion and closeness to the observer)

5. Hand tracking

7. Input to recognizer Feature vector x t From the Kinect data and hand tracking Relative position of the gesturing hand with respect to shoulder center in world coordinate (R 3 ) From the Xsens unit on the hand Linear acceleration (R 3 ) Angular velocity (R 3 ) Euler orientation (yaw, pitch, roll) (R 3 )

8. System overview Depth & RGB images Gesture spotting & recognition Hand tracking Hand movement segmentation Xsens data Feature vector sequence Feature vector sequence with movement

9. Hand movement segmentation Part of gesture spotting Train Gaussian models for rest and non-rest positions During recognition, an observation x t is first classified as a rest or a non-rest position It is a non-rest position if

10. System overview Depth & RGB images Gesture spotting & recognition Hand tracking Hand movement segmentation Xsens data Feature vector sequence Feature vector sequence with movement

11. Temporal model of gestures

12. Continuous gesture models Pre- stroke Nucleus Post- stroke Rest End

13. Continuous gesture models Pre- stroke Nucleus Post- stroke Rest End

14. Continuous gesture models Pre- stroke Nucleus Post- stroke Rest End

15. Pre-stroke & post-stroke phases

16. Bakis model for nucleus phase start s1s1 s2s2 s3s3 s6s6 p(s1)p(s1) p(END|s 6 ) 6 hidden states per nucleus phase in the final model Emission probability: mixture of Gaussians with 6 mixtures s4s4 s5s5

17. Concatenated HMMs Train an HMM for each phase for each gesture Model termination probability for each hidden state s as p(END|s) EM parameter estimation

18. Concatenated HMMs After training, concatenate HMMs for each phase to form one HMM for each gesture Compute transition probability from the previous phase to the next phase Ensure

19. Detect rest vs non-rest segments Find concatenated HMM that gives the highest probability Find most probable hidden state sequence using Viterbi Assign hidden states to corresponding phases Identify segment without nucleus phase Gesture spotting & recognition no nucleus phase

20. Gesture recognition result visualization

21. Hand position only Xsens onlyHand position & Xsens F1 score0.677 (0.04)0.890 (0.02)0.907 (0.01) ATSR score0.893 (0.02)0.920 (0.01)0.923 (0.02) Final score0.710 (0.03)0.895 (0.01)0.912 (0.01) Gesture recognition result 10 users and 10 gestures and 3 rest positions 3-fold average

22. Gesture recognition result User independent training and testing 3-fold average Latent Dynamic CRF Concatenated HMMs F1 score0.820 (0.03)0.897 (0.03) ATSR score0.923 (0.02)0.907 (0.02) Final score0.828 (0.02)0.898 (0.02) Training time18hr7min

23. Contributions Employed novel gesture phase differentiation using concatenated HMMs Used hidden states to identify movements with no nucleus phases accurately detect start and end of nucleus phases Improved hand tracking when the hand is close to the body or moving fast by gesture salience detection