Finger Gesture Recognition through Sweep Sensor Pong C Yuen 1, W W Zou 1, S B Zhang 1, Kelvin K F Wong 2 and Hoson H S Lam 2 1 Department of Computer Science Hong Kong Baptist University 2 World Fair International Ltd

Outline Motivations Design Criteria Proposed Method Experimental results Conclusions

Motivations Vision-based interface Insert some images using face, expression, body movement… Sensor-based interface There should be a video about the body movement interface Common objective: natural input to replace traditional physical input devices

Motivations While many sensor-based gesture input have been developed, there is no algorithm/system using sweep sensor Why Sweep Sensor? low cost No latency problem (fingerprint recognition) popularity

Design Criteria User friendliness easily performed by a user. intuitive and easy to understand. User independent Generic for all users. Robustness diversity of patterns captured. Efficiency Real-time application mobile devices

t > t 0 left right No left tick right tick Yes feature vector D > 0.5 left right D > 1.3 left tick right tick D > 1/1.3 D < -0.5 noise reduction envelope enhancement input image direction estimation direction index D = D left /D right yiyi i y Characteristics Formulate the noise Proposed Method Characteristics Formulate the noise noise reduction envelope enhancement input image direction estimation direction index D = D left /D right yiyi i y t > t 0 left right No left tick right tick Yes feature vector D > 0.5 left right D > 1.3 left tick right tick D > 1/1.3 D < -0.5 Characteristics Formulate the noise noise reduction envelope enhancement input image direction estimation direction index D = D left /D right yiyi i y t > t 0 left right No left tick right tick Yes feature vector D > 0.5 left right D > 1.3 left tick right tick D > 1/1.3 D < -0.5

Input Image Characteristics Different sensor characteristics Noise level

Figure 2. The block diagram of feature extraction Segmentation Owing to different sensor characteristics, the gesture images obtained, even the gesture is the same, will be different Segmentation by estimating the sweeping time noise reduction vertical gradient thresholding horizontal projection

Segmentation (cont.) blank part sweeping part noise reduction vertical gradient TH thresholding horizontal projection

Feature Extraction Time information t (sweeping time) Finger motion information d (direction) Left and right Left diagonal and right diagonal

Feature Extraction (left / right) noise reduction Left Right direction enhancement input image direction estimation direction index D = P left - P right i-th fingerprint texture AB CD

Feature Extraction (left tick / right tick) noise reduction envelope enhancement input image direction estimation direction index D = D left /D right yiyi i y

Classification A very simple rule based on a combination of movement Classification tree (decision tree) left right No left tick right tick Yes feature vector t > t 0 D > 1.3 left tick right tick D > 1/1.3 D > 0.5 left right D < -0.5

Designed Gestures Left and RightLeft tick and Right tick

Experimental Results 2 testing groups 3 technical users – Engineers, and technical managers, research staff (95.0%) 3 Non-technical users – secretary, clerk (86.87%) Test on different sensors 4 different sensors manufacture at different period of time

Experimental Results Evaluation interface There should be a video here

Experimental Results Results by 3 non-technical staff with 4 different sensors

Experimental Results Integrated application with an image viewer There should be a video here

Conclusions

Thank You