Wrist Recognition and the Center of the Palm Estimation Based on Depth Camera Zhengwei Yao ; Zhigeng Pan ; Shuchang Xu Virtual Reality and Visualization (ICVRV), 2013 International Conference on 1

Outline  Introduction  Related work  Proposed method  Experimental results  Conclusion 2

Introduction 3

 Problem: Can not separate a hand from a forearm using color and depth information  Solution: Find wrist to recognize hand 4

Related Work 5

 Hand segmentation and extraction  Color [11,12]  Depth threshold [13,14]  The location of other body parts [15~17]  Wrist  Wear wristband[14]  Palm detection[18] 6

Reference [13] D. Uebersax, J. Gall, and M. Van den Bergh, and L. Van Gool, “Realtime sign language letter and word recognition from depth data”. International Conference on Computer Vision Workshops (ICCV Workshops), 2011 [14] Z. Ren, J. Yuan, and Z. Zhang, “Robust hand gesture recognition based on finger- earth mover's distance with a commodity depth camera”. ACM international conference on Multimedia, 2011 [15] T. I. Cerlinca and S. P. Pentiuc, “Robust 3D Hand Detection for Gestures Recognition”. Proc. the 5th International Symposium on Intelligent Distributed Computing, Delft, 2012 [16] M. Van den Bergh and L. Van Gool, “Combining RGB and ToF cameras for real-time 3D hand gesture interaction”. Workshop on Applications of Computer Vision (WACV), Kona, 2011 [17] K. Fujimura and L. Xia, “Sign recognition using depth image streams”. Automatic Face and Gesture Recognition, 2006 [18] U. Lee and J. Tanaka, “Hand Controller: Image Manipulation Interface Using Fingertips and Palm Tracking with Kinect Depth Data”. Proc. of 10th Asia Pacific Conference on Computer Human Interaction(APCHI), Matsue,

Related Work  Candescent NUI(Natural User Interface) project  Hand and finger tracking  Develop by Stefan Stegmueller, Swiss  Open source: Use the OpenNI framework with the Kinect sensor “Finger direction detection” Blue : cluster centroid Green : palm center Red : fingertips Yellow : hand contour Long lines : finger directions ※ “A Robust Method of Detecting Hand Gestures Using Depth Sensors”, Yan Wen; Chuanyan Hu; Guanghui Yu; Changbo Wang, 2012 IEEE International Workshop on HAVE 8

Proposed Method 9

 Hand segmentation and palm estimation  Wrist recognition  The center of the palm estimation 10

Hand Segmentation and Palm Estimation (1/5)  a. Cluster the hand data  K-means clustering algorithm  Specify the depth range: 0.5~0.8m 11

Hand Segmentation and Palm Estimation (2/5)  b. Compute the Convex hull of the hands  The Graham scan algorithm 12

Hand Segmentation and Palm Estimation (3/5)  c. Detect the hand contours  Moor-Neighbor tracking algorithm 13

Hand Segmentation and Palm Estimation (4/5)  d. Detect the fingertips  Find all candidate points that are both on the convex hull and the contour  The distance of P 0 and P’> threshold 14

Hand Segmentation and Palm Estimation (5/5)  e. Estimate the center of the palm  The biggest circle inside the hand contour 15

Wrist Recognition  Wrist: pit points  Find an obvious pit point in the contour of hand  Create an appropriate to find another wrist point. inscribed rectangle 16

Wrist Recognition (1/4)  Step 1: Find candidate lines of wrist  The ends of the candidate line should not be both fingertips.  The distance of the candidate line should not be less than a specific value. 17

Wrist Recognition (2/4)  Step 2: Find the corresponding candidate contours whose ends are the ends of the candidate lines. 18

Wrist Recognition (3/4)  Step 3: Find one of the wrist points  Calculate the maximum distance between the candidate line and the corresponding candidate contour.  The largest distance from these maximum distances.  The point with the largest distance is one of the wrist points 19

Wrist Recognition (4/4)  Step 4: Find another wrist point  Connect this wrist point to each point in the hand contour, and take these connecting lines as the diagonals of rectangles.  If the rectangle is not inside the hand contour, the corresponding point in the contour is not another wrist point.  Find out the point with the shortest rectangle diagonal as another wrist point. 20

Wrist Recognition  Candidate lines  Corresponding contour  Find one of the wrist points  Find another wrist point 21

Estimating the Center of the Palm (1/4)  Step 1: Select three points from the hand contour  The three points (P 1, P 2, P 3 ) form an acute triangle. 22

Estimating the Center of the Palm (2/4)  Step 2: Find circumcenter O j of the triangle  The O j coordinate  The radius of circle : 23

 Step 3: Determine the center  Calculate the distances from each point in the hand contour to the center Formula.  Condition A: The number of distance R ji > R j is bigger than the threshold  Condition B: R j >minR (minR=> the minimum radius of the palm )  If A or B is not satisfied, Step 4 Estimating the Center of the Palm (3/4) 24

 Step 4: Find another appropriate palm center  One end-point of these two intersectant chords is replaced by point P min  Repeat step 2 to step 4 until the ending condition is true Estimating the Center of the Palm (3/4) 25

Proposed Method  Hand segmentation and palm estimation  Wrist recognition  The center of the palm estimation Fingertips detection 26

Experimental Results 27

Experimental Results  Device: AMD Athlon(tm)Formula Dual Core Processor Formula CPU, 4GB RAM, NVIDIA GeForce 9600GT Graphics card and Window7 32bit OS  Threshold setting  Depth: m  Minimum distance of line: 50  Minimum radius of the palm: 33  #hand contour inside circle: 25~50 28

Experimental Results Before After 29

Experimental Results  Divide into three groups based on the number of the points inside the hand contour. #contour#inside contour a b c d e f g h i

Experimental Results 31

Experimental Results  Improved original algorithm: every 8 th point  The new algorithm: proposed method 32

Conclusion 33

Conclusion  Propose the wrist recognition algorithm to separate the hand from the forearm,  Propose a new algorithm of estimating the center of the palm to reduce the computing time.  Without Kinect SDK 34