Vision Based hand tracking for Interaction The 7th International Conference on Applications and Principles of Information Science (APIS2008) Dept. of Visual Contents, Dongseo University Seung Il Han, Jeon Yeong Mi, Jung Hye Kwon, Hwang-Kyu Yang, Byung Gook Lee

Outline APIS Vision Based hand tracking for Interaction, Han Seung Il, Dongseo Univ., 1. Purpose 2. System overview 3. Flow chart 4. Segmentation of a hand region 5. Matching stereo data 6. Conclusion 7. Future work

APIS Vision Based hand tracking for Interaction, Han Seung Il, Dongseo Univ., 1. Introduction to hand tracking for interaction

1.1 Purpose APIS Vision Based hand tracking for Interaction, Han Seung Il, Dongseo Univ.,  Any user can interact anytime without any condition of space.  More comfortable to use digital-device on public place.

1.2 Optical vs. computational reconstruction APIS Vision Based hand tracking for Interaction, Han Seung Il, Dongseo Univ.,  Hardware  Bumblebee stereo camera  Display device  PC  Software  Image processing  Stereo matching

APIS Vision Based hand tracking for Interaction, Han Seung Il, Dongseo Univ., 2. System overview

2.1 System architecture APIS Vision Based hand tracking for Interaction, Han Seung Il, Dongseo Univ.,

2.2 Flow chart APIS Vision Based hand tracking for Interaction, Han Seung Il, Dongseo Univ.,

2.3 The segmentation of a hand region APIS Vision Based hand tracking for Interaction, Han Seung Il, Dongseo Univ., I f = ½ 255 ; j I i ¡ I b j > T h res h o ld 0 ; o t h erw i se [Background] [Captured] [Foreground] [Subtraction]

2.3 The segmentation of a hand region APIS Vision Based hand tracking for Interaction, Han Seung Il, Dongseo Univ.,  Detection algorithm  Peer and Solina’s 2003 [1]  Hsu and Mottaleb Jain’s Central 2002 [2]  Hsu and Mottaleb Jain’s Ellipse 2002 [2]  M. Jones and J. Rehg’s Color Histogram [3, 4] [3] M. Jones, J. Rehg. “Statistical color models with application to skin”, In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Vol.1, pp , [1] Kovac, J. Peer, P. Solina, F. “Human Skin Color Clustering for Face Detection”, The IEEE Region, vol.2, pp , [2] R.L.Hsu, M.Abdel-Mottaleb, A.K.Jain, “Face Detection in Colour Images”, IEEE transactions, on Pattern Analysis and Machine Intelligence, vol.24, no.5, pp , [4] Shahzad Malik, “Real-time Hand Tracking and Finger Tracking for Interaction”, CSC2503F Project Report, December 18, 2003.

2.3 The segmentation of a hand region APIS Vision Based hand tracking for Interaction, Han Seung Il, Dongseo Univ., OR * Uniform daylight illumination R > 95 ; G > 40 ; B > 20 an d max f R ; G ; B g ¡ m i n f R ; G ; B g > 15 an d j R ¡ G j > 15 an d R > G an d R > B * Flashlight or light daylight R > 220 ; G > 210 ; B > 170 an d j R ¡ G j · 15 an d R > B an d G > B  Peer and Solina’s 2003

2.3 The segmentation of a hand region  Hsu and Mottaleb Jain’s Central APIS Vision Based hand tracking for Interaction, Han Seung Il, Dongseo Univ., W C i ( Y ) = 8 > < > : W LC i + ( Y ¡ Y m i n )( W C i ¡ W LC i ) K l = Y m i n ; Y < K l W HC i + ( Y max ¡ Y )( W C i ¡ W HC i ) Y max ¡ K h ; K h < Y W C i ;e l se ¹ C b ( Y ) = 8 > < > : ( K l ¡ Y ) K l ¡ Y m i n ; Y < K l ( Y ¡ K h ) Y max ¡ K h ; K h < Y 108 ;e l se C r = 0 : 5 R ¡ 0 : 418 G ¡ 0 : 081 B C b = ¡ 0 : 168 R ¡ 0 : 331 G + 0 : 5 B RGB t o Y = 0 : 299 R + 0 : 587 G + 0 : 144 B YC b C r ¹ C r ( Y ) = 8 > < > : 154 ¡ 10 ( K l ¡ Y ) K l ¡ Y m i n ; Y < K l ( Y ¡ K h ) Y max ¡ K h ; K h < Y 108 ;e l se

2.3 The segmentation of a hand region  Hsu and Mottaleb Jain’s Central 2002 ¹ C b ( Y ) ¡ ® W C b ( Y ) < C b < ¹ C b ( Y ) + ® W C b ( Y ) ; ¹ C r ( Y ) ¡ ® W C r ( Y ) < C r < ¹ C r ( Y ) + ® W C r ( Y ) APIS Vision Based hand tracking for Interaction, Han Seung Il, Dongseo Univ., : a) : b) a)b)

2.3 The segmentation of a hand region  Hsu and Mottaleb Jain’s Ellipse 2002 · x y ¸ = · cos µ s i n µ ¡ s i n µ cos µ ¸· C b ¡ c x C r ¡ c y ¸ [2] R.L.Hsu, M.Abdel-Mottaleb, A.K.Jain, “Face Detection in Colour Images”, IEEE transactions, on Pattern Analysis and Machine Intelligence, vol.24, no.5, pp , e cx = 1 : 6 ; e cy = 2 : 41 ; a = 25 : 39 ; b = 14 : 03 c x = 109 : 38 ; c y = 152 : 02 ; µ = 2 : 53 ra d i an APIS Vision Based hand tracking for Interaction, Han Seung Il, Dongseo Univ., ( x ¡ e cx ) 2 a 2 + ( y ¡ e cy ) 2 b 2 · 1

2.3 The segmentation of a hand region  M. Jones and J. Rehg’s color histogram APIS Vision Based hand tracking for Interaction, Han Seung Il, Dongseo Univ., P ( s k i n ) + P ( ¡ s k i n ) = 1 P ( s k i n ) = T s T s + T n : The skin histogram : The non-skin histogram : The pixel count in bin rgb of : The total counts contained in n [ rg b ] H s T n H n s [ rg b ] T s H s H s H n H n P ( rg b j ¡ s k i n ) = n [ rg b ] T n P ( rg b j s k i n ) = s [ rg b ] T s P ( s k i n j rg b ) = P ( rg b j s k i n ) P ( s k i n ) P ( rg b j s k i n ) P ( s k i n ) + P ( rg b j ¡ s k i n ) P ( ¡ s k i n ) P ( s k i n j rg b ) ¸ ¾ s

2.3 The segmentation of a hand region APIS Vision Based hand tracking for Interaction, Han Seung Il, Dongseo Univ., [Color image of background subtraction]  Technique of region extraction [Blob analysis of binary image] [Binary image of background subtraction] [Binary morphological opening]

2.4 Detection of fingers APIS Vision Based hand tracking for Interaction, Han Seung Il, Dongseo Univ., [5] Image Processing Program by Visual C++, Published by HANBIT Media, pp , Vol. 10, No. 1779,  Contour tracing with clockwise algorithm

2.4 Detection of fingers [Result of binary morphological opening][Contour tracing result of feature extraction] APIS Vision Based hand tracking for Interaction, Han Seung Il, Dongseo Univ.,  Result of contour tracing

2.4 Detection of fingers  Detection of peak point  At each pixel in a hand contour  Compute k-curvature  Angle between the two vectors and  Peak point and valley point  Peak point  Valley point µ k < µ m i n µ k > µ max µ C i ( j ) APIS Vision Based hand tracking for Interaction, Han Seung Il, Dongseo Univ., j i [ C i ( j ) ; C i ( j ¡ k )][ C i ( j ) ; C i ( j + k )] w h ere ; k i scons t an t µ k C i ( j ¡ k ) C i ( j + k ) [4] Shahzad Malik, “Real-time Hand Tracking and Finger Tracking for Interaction”, CSC2503F Project Report, December 18, w h ere ; µ m i n = 90 ± ; µ max = 200 ±

2.4 Detection of fingers  Detection of peak point APIS Vision Based hand tracking for Interaction, Han Seung Il, Dongseo Univ., [Peak point detection][Valley point detection]

2.4 Detection of fingers  Detection of midpoint  Define as:  Midpoints represent the axis of the finger.  is a point that is k points to the left along a hand contour  is a point to the right  Let is the index of a finger tip cnorm ( x ) = 8 < : x + N :x < 0 x ¡ N : ( x + 1 ) > N x:o t h erw i se : w h ere N represen tt h enum b ero f po i n t sa l ong t h econ t ourper i me t er : APIS Vision Based hand tracking for Interaction, Han Seung Il, Dongseo Univ., cnorm ( x ) P ( cnorm ( i + k )) P ( cnorm ( i ¡ k )) i f t [4] Shahzad Malik, “Real-time Hand Tracking and Finger Tracking for Interaction”, CSC2503F Project Report, December 18, 2003.

2.4 Detection of fingers  Detection of midpoint  A midpoint  Compute for Q ( k ) Q ( k ) = P ( cnorm ( i f t + k )) + P ( cnorm ( i f t ¡ k )) 2 Q ( k ) k m i n < k < k max APIS Vision Based hand tracking for Interaction, Han Seung Il, Dongseo Univ.,

2.5 Matching stereo data APIS Vision Based hand tracking for Interaction, Han Seung Il, Dongseo Univ.,  Calculation of depth value xy P y x r B D f [6] Tomase Poggio, “Introductory Techniques for 3-D Computer Vision”, Massachussetts Institute of Technology, pp , P x = x l + x r 2 ; x r l x l x : Distance using centimeter : Resolution ratio, : CCD Sensor size R C P y = y l + y r 2 = y ; P z = f B j x l ¡ x r j D D = P z RC

2.5 Matching stereo data  Result of depth value  Real distance ≒ 66 Cm  Calculated depth value = 67.5 Cm APIS Vision Based hand tracking for Interaction, Han Seung Il, Dongseo Univ.,

APIS Vision Based hand tracking for Interaction, Han Seung Il, Dongseo Univ., 3. Conclusion

3.1 Conclusion  The hardware costs very low.  Hand tracking using an enhanced algorithm  Interaction with system using user’s hands  Free of conditions of time & space APIS Vision Based hand tracking for Interaction, Han Seung Il, Dongseo Univ.,

3.2 Future Work  To perform a robust segmentation with ambient environment while detection the skin color.  The determination of peak-point count and decision of location.  Gesture recognition performance  To enhance the accuracy and speed of the system.  Interaction using hands APIS Vision Based hand tracking for Interaction, Han Seung Il, Dongseo Univ.,

References 1. Kovac, J. Peer, P. Solina, F. “Human Skin Color Clustering for Face Detection”, The IEEE Region, vol.2, pp , R.L.Hsu, M.Abdel-Mottaleb, A.K.Jain, “Face Detection in Colour Images”, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol.24, no.5, pp , M. Jones, J. Rehg. “Statistical color models with application to skin”, In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Vol.1, pp , Shahzad Malik, “Real-time Hand Tracking and Finger Tracking for Interaction”, CSC2503F Project Report, December 18, “Image Processing Program by Visual C++”, Published by HANBIT Media, pp , Vol. 10, No.1779, Tomase Poggio, “Introductory Techniques for 3-D Computer Vision”, Massachussetts Institute of Technology, pp , George V. Paul, Glenn J. Beach, Charlesn J.Cohen, “A Real-time Object Tracking System using a Color Camera”, pp , J. Segen, S. Kumar. “3D hand pose estimation using a single camera”, In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Vol.1, pp , APIS Vision Based hand tracking for Interaction, Han Seung Il, Dongseo Univ.,

Thank You APIS Vision Based hand tracking for Interaction, Han Seung Il, Dongseo Univ.,