Hand Gesture for Taking Self Portrait Shaowei Chu and Jiro Tanaka University of Tsukuba Japan 12th July 15 minutes talk

2 Introduction self portrait is to take a picture of ourself self portrait pictures are widely used. photo album, web-blog, twitter, passport, etc... shaowei, in IPLAB

3 approaches to take self portraits ask others for help photographers not always at hand set camera's timer means time, right? run back and forth to guarantee a good shot use remote controller make the hand not free computer vision face and smile detection functions are limitied out of user's control Timer: 5...

4 vision-based hand gesture interaction computer vision-based approach has more possibilities more interface could be developed hand gestures contain plenty of actions and meanings more dynamic than timer and smile face detection, plentiful interaction could be done for digital camera, additional devices are not required remote controller or sensors are not needed

5 dispaly, projector proposed approach hand gesture to make interaction with the camera hand gesture display(large-display, projector or camera frontal screen) to give camera live view feedback and GUI Timer: 3... pan & tilt platform pan & tilt platform makes camera focus on user automatically

6 prototype system IPLAB

7 demo 57 seconds

8 hand detection skin-color based hand detection skin-color segment [1] original image segmented hand [1]. Kovač, J., Peer, P., Solina, F.: Human Skin Colour Clustering for Face Detection. [2]. Malik, S., Laszlo, J.: Visual Touchpad: A Two-handed Gestural Input Device k-curvature [2] detected fingertip

9 hover button interface trigger camera shutter activate the hover button

10 cross motion interface recognize hand moving directions control camera pan & tilt motion tracking algorithm optical-flow [1] measurements analysis motion gesture sequence in time line cross motion cross motion interface [1]. Beauchemin, S.S., Barron, J.L.: The Computation of Optical Flow. ACM Computing Surveys

11 motion gesture recognize procedure (1/2) 29 feature points feature points movementmoving direction layout feature points inside interface area video frame line no motion no motion motion detection on frames motion gesture

12 motion gesture recognize procedure (2/2) time: t time: t + 1 time: t + 2 time: t + 3

13 GUI

14 merit and novel aspects of our system take self-portrait pictures by yourself (do not need other help) original motivation vision-based gesture interface has more possibilities plenty of interfaces could be developed for digital camera real-time hand detection in dynamic background cross motion interface novel way of recognize hand motion gestures Timer: 5...

15 discussion computer vision based hand detection is lighting condition sensitive and distance sensitive (within 2 meters) indoor self-portraits taking scenario works OK cross motion interface has about 80% accuracy within 2 meters distance from camera algorithm running in real-time hand detection: 180fps cross motion interface: 300fps

16 related works(1/2) sony camera robot pan & tilt platform, face and smile detection automatical shutter no interface to control no interaction

17 related works(2/2) Visual Touchpad: A Two-handed Gestural Input Device Shahzad Malik, Joe Laszlo ACM International Conference on Multimodal Interfaces 2004 using fingers as input device It needs to put the hand on a black board to segment the hand our algorithm can detect the hand in dynamic background

18 conclusion and future works proposed hand gesture approach to take self portrait pictures showed our self-portrait system with demo introduced the hand gesture recognition algorithm and GUI in the future zooming interface small screen or screen-less gesture interface head gestures based interaction

Thank you! Shaowei Chu and Jiro Tanaka University of Tsukuba, Japan Hand Gesture for Taking Self Portrait 12th July

20 Casio Camera TRYX CP+ February 2011 wave gesture