A Fast and Robust Fingertips Tracking Algorithm for Vision-Based Multi-touch Interaction Qunqun Xie, Guoyuan Liang, Cheng Tang, and Xinyu Wu th IEEE International Conference on Control and Automation (ICCA)

Outline Introduction Related Work Proposed Method Hand localization Fingertips tracking The Multi-touch system Experimental Results Conclusion 2

Introduction 3

Multi-touch technology: Sensor Based Directly receive finger touch as input High cost → limits its application to some extent Computer Vision Based Good scalability as well as good performance Introduction 4 Image: Oka, K, Sato, Y, Koike, H. "Real-time fingertip tracking and gesture recognition", IEEE Computer Graphics and Applications, 2012

Introduction In this paper: Propose a robust fingertip tracking algorithm: Real-time Stereovision-based 3D multi-touch interaction system Skin / Depth / Geometry structure 5 Hand Detection Fingertip Tracking Multi-touch System

Related Work 6

Related work Geometry properties: Curvature Edge or shape Build a model Image Analysis Template matching Color Segmentation 7 L. Jin, D. Yang, L. Zhen, and J. Huang. A novel vision based finger-writing character recognition system. Journal of Circuits, Systems, and Computers (JCSC), 16(3):421–436, D. Lee and S. Lee. Vision-based finger action recognition by angle detection and contour analysis. Electronics and Telecommunications Research Institute Journal, 33(3):415–422, 2011.

Related work Palm Center: Fingertip Detection 8 [a] [b] [c] [d] Geodesic distance GSP points Neighbor depth

Related work [a] Hui Liang, Junsong Yuan, and Daniel Thalmann, "3D Fingertip and Palm Tracking in Depth Image Sequences", Proceedings of the 20th ACM international conference on Multimedia, 2012 [b]Chia-Ping Chen, Yu-Ting Chen, Ping-Han Lee, Yu-Pao Tsai, and Shawmin Lei, "Real-time Hand Tracking on Depth Images", IEEE Visual Communications and Image Processing (VCIP), 2011 [c] Ziyong Feng, Shaojie Xu, Xin Zhang, Lianwen Jin, Zhichao Ye, and Weixin Yang, “Real- time Fingertip Tracking and Detection using Kinect Depth Sensor for a New Writing-in-the Air System”, Proceedings of the 4th International Conference on Internet Multimedia Computing and Service, 2012 [d] Zhichao Ye, Xin Zhang, Lianwen Jin, Ziyong Feng, Shaojie Xu, "FINGER-WRITING-IN- THE-AIR SYSTEM USING KINECT SENSOR", IEEE International Conference on Multimedia and Expo Workshops (ICMEW),

Proposed Method 10

Hand Segmentation Skin Color filter YCbCr color space Gaussian Mixture Model Describe the skin-color distribute Single Gaussian Model: Gaussian Mixture Model: 11 Training data: Weight of each Gaussian model: color vector

Hand Segmentation Skin Color filter : how skin-like the color is Expectation Maximization(EM) algorithm 12

Hand Segmentation Depth Cue: The points with minimum depth are picked as seeds Region grow algorithm 13 skin depthskin + depth

Hand Segmentation 14 [18] Z. Mo and U. Neumann, “Real-time hand pose recognition using low-resolution depth images,” in Computer Vision and Pattern Recognition, 2006 IEEE Computer Society Conference on, vol. 2. r : row index c : column index z(r,c) : depth value, range threshold (related to palm size) boundary

Palm Region Extraction Observation : Palm is a rectangle-like region Method : Project the hand region in all directions 15

Palm Region Extraction 16 Intersection

Palm Center Localization 17 X

Palm Center Localization Palm Center: The point with maximum distance from the closest palm boundary [18]. The size of palm R: 18 palm region palm boundary

Fingertip Localization Fingertip : The point with maximum distance to the palm center (on the contour of each finger) Candidate set F: 19 P : contour point C 0 : palm center d 2 : distance R : palm size 1.2 F

Fingertip Localization Assign an index to each point in candidate set: Sort candidate set by 20

Fingertip Localization 21

Multi-touch system TUIO (Tangible User Interface Object) 22 [24] M. Kaltenbrunner, T. Bovermann, R. Bencina, and E. Costanza, “Tuio:A protocol for table-top tangible user interfaces,” in Proc. of the The 6th Intl Workshop on Gesture in Human-Computer Interaction and Simulation, 2005.

Experimental Results 23

Experimental Results Xeon 3.07Ghz workstation frame rate : 20Hz on average (real-time) Modules Fingertip tracking TUIO server TUIO client 24 [10] C. Shan, Y. Wei, T. Tan, F. Ojardias, ”Real Time Hand Tracking by Combining Particle Filtering and Mean Shift”, In: International Conference on Automatic Face and Gesture Recognition, 2004, pp

Conclusion 27

Conclusion Fast and robust fingertip tracking Without pressuring sensing device & extra marks Hand Segmentation Depth / Skin Fingertip Detection Palm region projection Palm center distance from the boundary Fingertip : assign index (angle) 28