1. 3D Fingertip and Palm Tracking in Depth Image Sequences
Hui Liang, Junsong Yuan and Daniel Thalmann
Proceedings of the 20th ACM international conference on Multimedia 2012

2. Outline Introduction Related Work Proposed Method Experimental Results
Conclusion

3. Introduction

4. Introduction
Human hand is an essential body part for human-computer interaction.
The positions of tracked fingertips: gesture estimation
Difficulty in fingertip tracking:
Anaglyph 紅藍眼鏡
Polarization 偏光眼鏡 –
在一般LCD TV前面貼上一層微相位差膜(Micro-retarder)，利用光的偏振方向 來將左眼與右眼的影像分離
Shutter 快門眼鏡 - 更新頻率120Hz以上播放左、右眼視角畫面，藉由快速切換左右眼資訊，使得左右眼分別看到正確的左眼與右眼畫面，經過視覺暫留與大腦融合後，即可呈現出具
深度感的立體影像。
此技術所顯示的3D畫面解析度不會下降，且立體效果非常的好。
然而由於要提升3D影像品質，因此左眼與右眼觀看螢幕的時間非常短，使得整體亮度會下降許多，亦是目前研發上需克服的重點。
Side-by-side
Bending
Nearby

5. Introduction Many previous methods: In this paper:
Only focus on extracting 2D fingertips
Cannot track fingertips robustly for a freely moving hand
In this paper:
Present a robust fingertip and palm tracking scheme
With the input of depth images (KINECT)
Track the 3D fingertip positions quite accurately
Anaglyph 紅藍眼鏡
Polarization 偏光眼鏡 –
在一般LCD TV前面貼上一層微相位差膜(Micro-retarder)，利用光的偏振方向 來將左眼與右眼的影像分離
Shutter 快門眼鏡 - 更新頻率120Hz以上播放左、右眼視角畫面，藉由快速切換左右眼資訊，使得左右眼分別看到正確的左眼與右眼畫面，經過視覺暫留與大腦融合後，即可呈現出具
深度感的立體影像。
此技術所顯示的3D畫面解析度不會下降，且立體效果非常的好。
然而由於要提升3D影像品質，因此左眼與右眼觀看螢幕的時間非常短，使得整體亮度會下降許多，亦是目前研發上需克服的重點。

6. Related Work

7. Related work Focus only on 2D fingertips:[4][5][6][9]
Based on contour analysis of the extracted hand region:[2][4][5][6]
Usually can track the fingertips for only stretched fingers.

8. Related work
In [6],
Fingertips are tracked for infrared image sequences.
It utilizes a template matching strategy
Fingertip tracking : Kalman filter
In [2],
Stereoscopic vision is adopted
Maximize the distance center of gravity of the hand & the boundary

9. Related work In [9] (Kinect), Minimum depth Depth < Threshold
[9] J. L. Raheja, A. Chaudhary, and K. Singal. Tracking of fingertips and centres of palm using kinect, Proc. Of the 3rd Int’l Conf. on Computational Intelligence, Modelling and Simulation.
Related work
In [9] (Kinect),
Minimum depth
Depth < Threshold
Circular filter

10. Related work
[2] S. Consei1, S. Bourennane, and L. Martin. Three dimensional fingertip tracking in stereovision, Proc. of the 7th Int’l Conf. on Advanced Concepts for Intelligent Vision Systems.
[4] K. Hsiao, T. Chen, and S. Chien. Fast fingertip positioning by combining particle filtering with particle random diffusion, Proc. IEEE Int’l Conf. on Multimedia and Expo.
[5] I. Katz, K. Gabayan, and H. Aghajan. A multi-touch surface using multiple cameras, Proc. of the 9th Int’l Conf. on Advanced concepts for intelligent vision systems.
[6] K. Oka, Y. Sato, and H. Koike. Real-time tracking of multiple fingertips and gesture recognition for augmented desk interface systems, Proc. IEEE Int’l Conf. on Automatic Face and Gesture Recognition.
[9] J. L. Raheja, A. Chaudhary, and K. Singal. Tracking of fingertips and centres of palm using kinect, Proc. Of the 3rd Int’l Conf. on Computational Intelligence, Modelling and Simulation.

11. Proposed Method

12. Foreground Segmentation
Overview
Stereo images
(Depth map)
Foreground Segmentation
Palm
Localization
Hand
Segmentation
Fingertip Detection
Fingertip Tracking
Fingertips

13. Hand and Palm Detection
1) Assume the hand is the nearest object
2) Constrain global hand rotation by:
: global rotation angle of the hand
Foreground
Segmentation
Palm
Localization
Hand

14. Hand and Palm Detection
Foreground Segmentation 
Palm
Localization
Threshold the depth frame to obtain the foreground F :
p : a pixel coordinate
z(p) : depth value (of point p )
z0 : the minimum depth value
zD : depth threshold
Hand
Segmentation
0.2m
foreground F

15. Hand and Palm Detection
Foreground Segmentation
Palm
Localization
The palm region is approximated with a circle:
pp : the palm center (of point p )
rp : the radius
Assume that hand palm forms a globally largest blob
Cp equals to the largest inscribed circle of the contour of F .
2D Kalman filter : reduce computational complexity
Hand
Segmentation

16. Hand and Palm Detection
Foreground Segmentation
Palm
Localization
Separate hand and forearm by a line:
1) Tangent to Cp
2) Perpendicular to the orientation of the forearm
Orientation of the forearm :
The Eigenvector that corresponds to the largest Eigenvalue of the covariance matrix of the contour pixel coordinates of F
Hand
Segmentation
Hand region : FV(2D) → FD(3D)

17. Fingertip Detection and Tracking
Constraints on possible fingertip locations:
1) Only in depth discontinuous region ( in contour Fv)
2) | Depth(one point) – Depth(neighborhoods) | are important.
3) Utilize the 3D geodesic shortest path (GSP)
Fingertip vs. Non-fingertip
Nearby Fingertips
Fingertip
Detection
Tracking

18. Fingertip Detection
Fingertip detection
Fingertip tracking
Goal: detect all five fingertips in the depth image
Based on three depth-based features
Build a graph G :
Vh : contains of all points within FV (hand contour)
Eh : for each pair of vertices(p,q), ) they are in the 8-neighborhood of each other ) their 3D distance 𝑑 𝑝,𝑞 = 𝑝−𝑞 2 is within threshold τ
Edge weight : 3D Euclidean distance

19. Fingertip Detection Calculate Geodesic distance dg(p):
Fingertip tracking
Calculate Geodesic distance dg(p):
From palm center pp for each vertex ∈ Vh
Dijkstra graph search on Gh
GSP point set Ug(p):
The set of vertices on the shortest path from pp to p
Rectangle local feature RL(p):
Describe the neighborhood of a point p in FV
𝜂(𝑝) : ratio of 1s 1 X
1cm

20. Fingertip Detection Calculate Geodesic distance dg(p): 𝜂(𝑝) dg(p)
Fingertip tracking
Calculate Geodesic distance dg(p):
𝜂(𝑝)
0.4
dg(p)
fingertips

21. Fingertip Detection Fingertip labeling:
Fingertip tracking
Fingertip labeling:
Nmax=6
j : label 𝚪 1 2 3 4 5
𝜞(𝒊,𝒋) :estimate the probability that 𝑝 𝑓 𝑖 has the label lj
𝑵 : number of GSP points
𝒌 : kth GSP point
i : fingertip

22. Fingertip Detection Fingertip labeling: Fingertip detection
Fingertip tracking
Fingertip labeling:

23. Fingertip Tracking Build a particle filter for each fingertip
Fingertip detection
Fingertip tracking
Build a particle filter for each fingertip
(x, ω) denote a particle
x : 2D position in FV
ω : the particle weight
Constrain the positions of each particle to the border point set UB to reduce the search space

24. Fingertip Tracking Likelihood function : / / pre now 𝑹𝑳(𝒑) Metric
Fingertip detection
Fingertip tracking
Likelihood function :
pre
now
𝑹𝑳(𝒑)
Metric
parameters
difference
Hausdorff difference
feature difference
Geodesic
distance /
dg(p)
GSP points
Neighbor
depth /

25. Fingertip Tracking Likelihood function : Fingertip detection

26. Experimental Results

27. Experimental Results Quantitative results on synthetic sequences:
‧Error : Euclidean distance
‧Ground truth : phalanx end point
Quantitative results on synthetic sequences:
Seq. No.
Motion
Seq. 1
grasping
Seq. 4
flexion
Seq. 2
adduction/abduction
Seq. 5
global rotation
Seq. 3
successive single finger
Seq. 6
combination of grasping and global rotation

28. Experimental Results

29. Experimental Results
Virtual object grasping:

30. Conclusion

31. Conclusion
Using multiple depth-based features for accurate fingertip localization
Adopting a particle filter to track the fingertips over successive frames
Track the 3D positions of fingertips robustly
Great potential for extension to other HCI applications