Fingertip Tracking Based Active Contour for General HCI Application Proceedings of the First International Conference on Advanced Data and Information Engineering, 2014 Kittasil Silanon and Nikom Suvonvorn Department of Computer Engineering, Faculty of Engineering, Prince of Songkala University Speaker: Yi-Ting Chen

Outline Introduction Flowchart Proposed Method –Initial Hand Segmentation –Finger Detection and Tracking Experimental Result Conclusions 2

Introduction 3 Hand gesture recognition provides more natural human-computer interaction. Many real-time system are proposed: –Using trajectories of hand motion [3,4,5] –Contours-based method for 2D fingertips tracking [7,8,9] –Using stereo vision to analyze the 3D fingertip positions [13,14,15,16]

Reference [3] Feng-Sheng Chen, Chih-Ming Fu, Chung-Lin Huang: Hand gesture recognition using a realtime tracking method and hidden Markov models. Image and Vision Computing (2003) [4] Elmezain M., Al-Hamadi: Gesture Recognition for Alphabets from Hand Motion Trajectory Using Hidden Markov Model. In: IEEE International Symposium on Signal Processing and information Technology, pp (2007) [5] Kittasil Silanon, Nikom Suvonvorn: Hand Motion Analysis for Thai Alphabet Recognition using HMM. In: International Journal of Information and Electronics Engineering (2011) [7] Antonis A. Argyros, Manolis I. A. Lourakis: Vision-based interpretation of hand gestures for remote control of a computer mouse. In: Computer Vision in Human-Computer Interaction, pp (2006) [8] Ko-Jen Hsiao, Tse-Wei Chen, Shao-Yi Chien: Fast fingertip positioning by combining particle filtering with particle random diffusion. In: IEEE International Conference on Multimedia and Expo, pp (2008) [9] J. Ravikiran, Mahesh Kavi, Mahishi Suhas, R. Dheeraj, S. Sudheender, Pujari Nitin V.: Finger Detection for Sign Language Recognition In: International MultiConference of Engineers & Computer Scientists, pp. 489 (2009) 4

Reference [13] Dung Duc Nguyen, Thien Cong Pham, Jae Wook Jeon: Fingertip detection with morphology and geometric calculation. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, pp (2009) [14] M. Do, T. Asfour, R. Dillman: Partical filter-based fingertips tracking with circular hough transform feature. In: Proceedings of the 12th IAPR Conference on Machine Vision Application (2011) [15] Raheja, J.L., Chaudhary, A., Singal, K.: Tracking of Fingertips and Centers of Palm Using KINECT Computational Intelligence. Third International Conference on Modeling and Simulation (CIMSiM), pp (2011) [16] Hui Liang, Junsong Yuan, Daniel Thalmann: 3D fingertip and palm tracking in depth image sequences. In: Proceedings of the 20th ACM international conference on Multimedia (MM '12). ACM, New York, NY, USA, pp (2012) 5

Introduction 6 In previous work [5], we proposed hand detector by using object detection method. But failed when other parts of body move close to hand’s region. [5] Kittasil Silanon, Nikom Suvonvorn: Hand Motion Analysis for Thai Alphabet Recognition using HMM. In: International Journal of Information and Electronics Engineering (2011)

Main Contributions The most problems: –Failure when the fingertips are bending into the palm. –Failure when the fingertips are overlapped each other. Therefore, this paper presented system to deal with such situations by using the depth image from Kinect. In addition, we develop an HCI application based on the fingertips tracking result. 7

The Application 8

Method Overview Using the depth image to segment the hand region. Detecting initial hand features. Tracking the 3D fingertips. –By calculating the internal and external energy. 9

Flowchart 10 Initial Hand Segmentation Finger Detection and Tracking

Hand Segmentation We proposed hand detector by using object detection method [5]. –Used to search hand’s region in image. But failed when other parts of body move close to hand’s region. 11

Using the Depth Image Therefore, depth image is used in the system. 12

Feature Extraction After extraction hand’s region, the initial hand’s features will be estimated. 13 Hand CenterFingertips Position Palm Size

Hand Center Point We obtained the center point of hand’s region. –Computed from the moments of pixels. 14 Hand Center

Palm Size The palm size is defined as the distance between the center point and the closest pixel on hand contour. 15 Palm Size

Fingertips Position Using the polygon approximation method [18] to extract key point P1,…,Pn Michael Kass, Andrew Witkin, Demetri Terzopoulos: Snakes: Active contour models. In: International journal of computer vision, vol. 1, no. 4, pp (1988)

Fingertips Position Each key point Pi has two parameters. –Angle (θ) By two vectors [P(i-k)P(i)] and [P(i)P(i+k)] –Slope. 17

Fingertips Position Satisfied: –1. Curvature value is in the threshold –2. Slope is positive So the key point is an initial fingertip 18

Flowchart 19 Initial Hand Segmentation Finger Detection and Tracking

Finger Location Most movements in hand gesture are finger movements. For a stretching finger, we defined two conditions. –Distance Condition : –Depth Condition : 20

Candidate Fingertips We define searching area to locate the candidate fingertip positions (Cfi). Fingertip positions should be points on hand contour. –Polygon approximation algorithm is used again. Using depth to find point which has minimum depth to be candidate fingertips. 21

Fingertip Tracking The possible candidate fingertips of each fingertip will be assigned energy. The maximum energy point is chosen to be the fingertip in the next frame. 22

23

24

25

26

Experimental Results We evaluate the fingertip tracking precision between the tracked fingertips and the ground truth. We have defined the ground truth using the end point contour of each finger. Testing 5 sequences, each sequence is tested at 10 rounds. 27

Fingertip Tracking Precision 28

Fingertip Tracking Precision 29

Human-Computer Interaction Application 30

Conclusion In this paper, we present the method. –Dealing with some issues by using depth data. –Apply concept of active contour to track fingertips over finger movement. It shows good performance in term of real-time and also has capability to expansion to HCI application. 31

Thanks for your listening! 32