1. Cindy Song Sharena Paripatyadar

2. Use vision for HCI Determine steps necessary to incorporate vision in HCI applications Examine concerns & implications of such applications

3. Background and Trends  In Today’s world: Many devices with integrated cameras Many personal webcams  Our Goal: To understand how to take advantage of these one camera systems

4. Literature Research Idea Generation Media Player Wizard of Oz Media Player Implementation Re-evaluate Approach Literature Research Revised Implementation Learning Application User Study Evaluation

5.  Freeman and Roth  For hand posture analysis  Creates histograms of local orientation using feature vectors from pixel intensity  Recognizes 10 gestures in real time

6.  Triesch and von der Malsburg  Based on Elastic Graph Matching  Extended for skin color feature detection  Recognizes 12 gestures

7.  Freeman  Uses one open hand to control onscreen display  Real time application  Hand may not be prominent in image

8.  Original Idea: Video control for WMPlayer  Wizard of Oz for a media player  Implementation of gesture recognition Shortcoming: real time issues  New approach: Teaching Chinese hand gestures

9.  5 participants, various technical backgrounds, age 20-27  Using computer with remote control  Used alternate monitor to show user video captured

10.  Small set of user-intuitive gestures are easy to remember, but need some menu reminder  Show rationale behind gestures  Visual feedback to show recognized command before execution  Concerns with: Low-light condition Camera field of view Webcam configuration Responsiveness Accuracy

11.  Pros Don’t have to search for remote Don’t have to touch remote while eating No battery to run down  Cons Doesn’t have as many features as remote Doesn’t work in dark environments More ambiguous than remote, more errors possible – know what each button will do

13. Gesture Determination: Using surrounding frames Pattern Matching: Compare regions with gesture patterns Finger Region Detection: Find connected regions (fingers/palm) Hand Location: Create bounding box from skin color Calibration: Learn lighting and coloring Skin Color Training: With images of various lighting

14.  Train based on skin color  Locate hand in image  Analyze hand gesture Finger region detection Pattern matching  Not real-time

15.  Skin Color Training Trained on 20+ images Different lighting & people Uses “Lab” color space

16.  Calibration Short training based on person’s hand and lighting conditions - < 1 sec needed Determines correct lighting & with skin color data Learns specific hand features

17.  Hand Location Determines hand position in image using skin color Fill in missing portions of hand Create bounding box

18.  Finger Region Detection Examine bounding box Find connected regions Remove small regions

19.  Pattern Recognition Created set patterns based on 10 gestures Counts number of finger regions for gestures 1-5 For gestures 6-10, based on number regions detected, looks at other patterns  i.e. for 6 determine ratio of finger width to space between fingers

20.  Gesture Determination 20 frames needed to recognize the gesture Avoids recognizing accidental gestures

23.  Complex Backgrounds First skin color analysis Then find large connected regions of fingers and hand

24.  Motion Static gestures & frame by frame analysis Allow for moving camera Gesture determination corrects obscurities or out of frame hand positioning

25.  5 participants, various technical backgrounds, age 20-27  Taught users 2-4 gestures  Quizzed users on gestures learned  Ran gesture recognition algorithm to provide feedback  Asked several follow up questions

26.  Useful for learning sign language, teaching kids to count  Instant feedback necessary  Nice to know how to correct gesture  Needs high accuracy  Other applications Some said Media Player application more useful Or use as security system (hand gestures as a password)

27.  Of implementation Real time is difficult Pattern recognition for specific gestures vs. technique for all types of gestures Complex/moving backgrounds important for real world applications  Of user studies Video is valuable avenue for many applications Accuracy and responsiveness are important In one camera systems, there is a tradeoff between convenience and clarity

28.  Real-time  More user studies  Mobile devices  Gesture learning application i.e. Chinese cultural gestures  Media Player plug-in application