A Robust Method of Detecting Hand Gestures Using Depth Sensors Yan Wen, Chuanyan Hu, Guanghui Yu, Changbo Wang Haptic Audio Visual Environments and Games (HAVE), 2012 IEEE International Workshop on 1

Outline  Introduction  Related Works  The Proposed Method  Experimental Results  Conclusion 2

Introduction 3

 In human-computer interaction(HCI) system, recognizing hand and finger gestures are significant.  Medical system, computer games, and human-robot  Depth-sensing camera(Kinect, Xtion) add a dimension to increase accuracy.  Goal: detect hand gestures with color and depth information 4

Related Works 5

 Body [4][5] V.S. Hand  Hand  Superiority: simple  Inferiority: small scale, low resolution  Strict condition: cluttered background, lighting variation 6

Related Works  Hand gesture recognition  Only color [12]  Data glove [7]  Training process [9][10]  Earth Mover’s Distance(EMD) [11] 7

References  [7] R. M. Satava, “Virtual reality surgical simulator,” Surgical Endoscopy, vol. 7, pp. 203–205,  [9] C. Keskin, F. Kirac, Y. Kara, and L. Akarun, “Real time hand pose estimation using depth sensors,” in Computer Vision Workshops (ICCV Workshops), 2011 IEEE International Conference on, nov  [10] P. Doliotis, A. Stefan, C. McMurrough, D. Eckhard, and V. Athitsos, “Comparing gesture recognition accuracy using color and depth information,” in Proceedings of the 4th International Conference on Pervasive Technologies Related to Assistive Environments, ser. PETRA ’11. New York, NY, USA: ACM, 2011, pp. 20:1–20:7.  [11] Z. Ren, J. Yuan, and Z. Zhang, “Robust hand gesture recognition based on finger- earth mover’s distance with a commodity depth camera,” in Proceedings of the 19th ACM international conference on Multimedia, ser. MM ’11. New York, NY, USA: ACM, 2011, pp. 1093–1096.  [12] A. Argyros and M. Lourakis, “Real-time tracking of multiple skincolored objects with a possibly moving camera,” in Computer Vision -ECCV 2004, ser. Lecture Notes in Computer Science. Springer Berlin/ Heidelberg, 2004, vol. 3023, pp. 368–379. 8

The Proposed Method 9

10 Gesture Representation Finger Recognition Find convex hullDetect fingertip and direction Hand Segmentation Find hands through color Separate hands by k-means Find palm center

Hand Segmentation (I) 11

 Hand Segmentation (I) -- Find hands through color 12 RGB imageDepth image L*a*b color space where b = 2 L*a*b color space where b = 3 Skin color images after AND operation Binary image of hand segmentation

Hand Segmentation (II)  Separate hands by k-means  k=2  Assignment:  Update:  Threshold of distance between 2 clusters 13

Hand Segmentation (II) -- Separate Hands By K-means 14

Hand Segmentation (III)  Find palm center  Inscribed circle  Minimum inner distance  Maximum element of inner distances set 15

Finger Recognition (I)  Find convex hull  Graham’s scan algorithm  P: the lowest y-coordinate  Sort in increasing order of angle  Point to point is left/right turn  Left-turn: O ; right-turn: X 16

Finger Recognition (II)  Detect fingertip and direction  Fingers are long and narrow  Find an isosceles triangle with V  V: Every vertex on the convex hull  Set a maximum threshold to the vertex angle  The direction vector is paralleled with the median length of an isosceles triangle 17

Finger Recognition 18 Blue point : cluster centroid Green point : palm center Red points : fingertips Yellow curves : hand contour Long lines : finger directions Structures around the hand : convex hull

The Proposed Method 19 Gesture Representation Finger Recognition Find convex hullDetect fingertip and direction Hand Segmentation Find hands through color Separate hands by k-means Find palm center

Gesture Representation  All information about hands  Palm center location  Finger number  Fingertips location  Finger direction vectors  Gestures  Rock-paper-scissors game  Drag images  Grasping, releasing 20

Experimental Results 21

Experimental Results  Use Kinect as input of depth and color images  The detection successful rate can reach 95%.  No matter the hand is horizontally or vertically placed. 22

23

Experimental Results 24

Experimental Results --Shadow Puppetry 25

Conclusion 26

Conclusion  Present a new method to detect hands’ positions and gestures  NO training, NO database  Future works  Set a threshold to the distance between the palm center and the fingers  Add additional sensor devices to overcome no palm detection  Shadow Puppetry project 27