1. Knowledge Systems Lab JN 8/24/2015 A Method for Temporal Hand Gesture Recognition Joshua R. New Knowledge Systems Laboratory Jacksonville State University

2. Knowledge Systems Lab JN 8/24/2015 Outline Terminology Motivation Current Research and Applications System Overview Implementation Approach Demonstration

3. Knowledge Systems Lab JN 8/24/2015 Terminology Image Processing - Computer manipulation of images. Some of the many algorithms used in image processing include convolution (on which many others are based), edge detection, and contrast enhancement. Computer Vision - A branch of artificial intelligence and image processing concerned with computer processing of images from the real world. Computer vision typically requires a combination of low level image processing to enhance the image quality (e.g. remove noise, increase contrast) and higher level pattern recognition and image understanding to recognize features present in the image.

4. Knowledge Systems Lab JN 8/24/2015 Motivation

5. Knowledge Systems Lab JN 8/24/2015 Motivation Gesturing is a natural form of communication –Gesture naturally while talking –Babies gesture before they can talk Interaction problems with the mouse –Have to locate cursor –Hard for some to control (Parkinsons or people on a train) –Limited forms of input from the mouse

6. Knowledge Systems Lab JN 8/24/2015 Motivation (2) Interaction Problems with the Virtual Reality Glove –Reliability –Always connected –Encumbrance –Sanitation

7. Knowledge Systems Lab JN 8/24/2015 System Overview

8. Knowledge Systems Lab JN 8/24/2015 System Overview Standard Web Camera Rendering User Interface Display Hand Movement User Gesture Recognition System Image Capture Update Object Image Input

9. Knowledge Systems Lab JN 8/24/2015 System Overview (2) System: OpenCV and IPL libraries (from Intel) Input: 640x480 video image Hand calibration measure Output: Rough estimate of centroid Refined estimate of centroid Number of fingers being held up Manipulation of 3D skull in QT interface in response to gesturing

10. Knowledge Systems Lab JN 8/24/2015 System Overview (3) Hand Calibration Measure: Max hand size in x and y orientations in # of pixels

11. Knowledge Systems Lab JN 8/24/2015 System Overview (4) Saturation Channel Extraction (HSL space): Original Image Hue Lightness Saturation

12. Knowledge Systems Lab JN 8/24/2015 Proposed Approach

13. Knowledge Systems Lab JN 8/24/2015 Proposed Approach

14. Knowledge Systems Lab JN 8/24/2015 Proposed Approach (2)

15. Knowledge Systems Lab JN 8/24/2015 Proposed Approach (3) The finger-finding function sweeps out a circle around the rCoM, counting the number of white and black pixels as it progresses A finger is defined to be any 10+ white pixels separated by 17+ black pixels (salt/pepper tolerance) Total fingers is number of fingers minus 1 for the hand itself

16. Knowledge Systems Lab JN 8/24/2015 Proposed Approach (4) Temporal Recognition System for hand movement patterns Input feature-vector creation: based on calculations from 12 centroid locations (hand movement during 3 seconds – average length of temporal gesture) Learning system training and recognition: SFAM classification used for ability to interactively add new, user-defined gestures

17. Knowledge Systems Lab JN 8/24/2015 Proposed Approach (5) Temporal Gesture Recognition System

18. Knowledge Systems Lab JN 8/24/2015 Proposed Approach (6) Input Feature-Vector Creation Twelve centroids collected (x,y) Find top-left centroid so that gestures are not start-point dependent (a square is a square whether you start at the top left or the bottom right) Compute centroid differences to recognize movement, not position (a square whether the hand is at the precise 12 points or in between those points) Normalize using the perimeter to recognize percentage of total movement since users are inaccurate in repeating a gesture (a square whether large or small) Note: System still sensitive to clockwise vs. counter-clockwise square (undo-like feature)

19. Knowledge Systems Lab JN 8/24/2015 Proposed Approach (7) Learning System Training and Classification 1.Feature vector is formatted for use by SFAM 2.Movements of the hand were recorded and assembled into one file for training the SFAM system -25 examples each of circle, square, left/right, up/down 3.System classification of hand gesture every 3 seconds 4.Train New Gesture button provided, stores gesture under the label entered in the box (5 is the default since 1-4 are already taken)

20. Knowledge Systems Lab JN 8/24/2015 Proposed Approach

21. Knowledge Systems Lab JN 8/24/2015 Demonstration System Configuration System GUI Layout

22. Knowledge Systems Lab JN 8/24/2015 Demonstration (2) Gesture to Interaction Mapping Number of Fingers: 2 – Roll Left 3 – Roll Right 4 – Zoom In 5 – Zoom Out