Knowledge Systems Lab JN 1/15/2016 Facilitating User Interaction with Complex Systems via Hand Gesture Recognition MCIS Department Knowledge Systems Laboratory Jacksonville State University Joshua R. New, Erion Hasanbelliu, and Mario Aguilar

Knowledge Systems Lab JN 1/15/2016 Outline Motivation System Architecture Implementation Overview Proposed Approach Demonstration Future Directions

Knowledge Systems Lab JN 1/15/2016 Motivation Gesturing is a natural form of communication Interaction problems with the mouse –Have to locate cursor –Hard for some to control (Parkinsons or people on a train) –Limited forms of input from the mouse

Knowledge Systems Lab JN 1/15/2016 Motivation (2) Interaction Problems with the Virtual Reality Glove –Reliability –Always connected –Encumbrance

Knowledge Systems Lab JN 1/15/2016 System Architecture Standard Web Camera Rendering User Interface Display Hand Movement User Gesture Recognition System Image Capture Update Object Image Input

Knowledge Systems Lab JN 1/15/2016 Implementation Overview System: 1.6 Ghz AMD Athlon OpenCV and IPL libraries (from Intel) Input: 640x480 video image Hand calibration measure Output: Rough estimate of centroid Refined estimate of centroid Number of fingers being held up Manipulation of 3D skull in QT interface in response to gesturing

Knowledge Systems Lab JN 1/15/2016 Implementation Overview (2) Hand Calibration Measure: Max hand size in x and y orientations in # of pixels

Knowledge Systems Lab JN 1/15/2016 Implementation Overview (3) Saturation Channel Extraction (HSL space): Original Image Hue Lightness Saturation

Knowledge Systems Lab JN 1/15/2016 Proposed Approach

Knowledge Systems Lab JN 1/15/2016 Proposed Approach (2)

Knowledge Systems Lab JN 1/15/2016 Proposed Approach (3) The finger-finding function sweeps out a circle around the rCoM, counting the number of white and black pixels as it progresses A finger is defined to be any 10+ white pixels separated by 17+ black pixels (salt/pepper tolerance) Total fingers is number of fingers minus 1 for the hand itself

Knowledge Systems Lab JN 1/15/2016 Proposed Approach (4) System Runtime: Current time – 41 ms for one image from camera Processing Capability on 1.6 Ghz Athlon: 24 fps Process Steps Time (ms) Athlon XP 1900 (1.6 Ghz) 1) Extract Sat Channel9 2) Threshold3 3) Connected Contour Fill 14 4) Centroid2 5) Segment Hand From Arm 9 6) Refined Centroid4 7) Count Number of Fingers 0 Total Time41

Knowledge Systems Lab JN 1/15/2016 Demonstration System Configuration System GUI Layout

Knowledge Systems Lab JN 1/15/2016 Demonstration (2) Gesture to Interaction Mapping Number of Fingers: 2 – Roll Left 3 – Roll Right 4 – Zoom In 5 – Zoom Out

Knowledge Systems Lab JN 1/15/2016 Demonstration (3)

Knowledge Systems Lab JN 1/15/2016 Demonstration (4)

Knowledge Systems Lab JN 1/15/2016 Future Directions Optimization Calibration Phase Defining Hand Orientation Learning System Interface Extensions For additional information, please visit