1. Paul D. Varcholik ACTIVE Laboratory Institute for Simulation and Training University of Central Florida pvarchol@ist.ucf.edu James L. Merlo LTC, US Army US Military Academy West Point james.merlo@usma.edu

2. Gestural Communication

3. Problem Statement  Reliable communications between military personnel is critical  Hand gestures are used when vocal means are inadequate  Line of sight issues may cause visual signals to be unreliable

4. Our Research  Purpose To determine the usefulness of a computer mediated gesturing recognition system for non- visual communication  Scope Provide a proof of concept which would lay the groundwork for future research  Research Questions Have we developed a recognition system capable of accurately converting and transmitting a visual communication mode into a non-visual form? Do computer-mediated gestures provide a viable form of non-visual communication?

5. Input Device Nintendo Wiimote  Nintendo Wiimote 3-axis accelerometer Wireless (Bluetooth) Inexpensive COTS 100Hz Sampling Rate

6. Output Devices Tactile Belt  Auditory (headphones)  Tactile Display Wireless (Bluetooth) 1.2 lbs (w/o battery) Elastic belt 8 tactors at 45-degree increments

7. Tactile Patterns Emulating Standard Army Hand Signals (FM 21-60)

8. Gesture Recognition  Machine Learning Algorithms (3 implemented for evaluation) Linear Classifier AdaBoost Artificial Neural Network (evolved w/ NEAT)  29 Features Based on work by Rubine (1991) on 2D symbol recognition Example features: ○ Bounding Volume Length ○ Min, Max, Median, Mean (X, Y, Z) ○ Starting Angle, Total Angle Traversed, Total Gesture Distance

9. Training & Visualization UI  Arbitrary gesture set  Left-hand, right-hand, both-hands  3D animated soldier  Text label display  Sound display  Tactile display  Wiimote visualization  Data serialization  UI Independent of recognition API

10. Experiments & Results  Several experiments run to date Different algorithms and gesture sets Accuracy > 94% Classification time < 10ms / gesture Linear classifier best performer (for training time and classification considered together) AdaBoost (highest accuracy, but slower training time than linear classifier) ANN w/ NEAT (worst performer – requires more training data)

11. Discussion  Proved Concept System capable of accurately converting and transmitting a visual communication mode into a non-visual form.  Wiimote is a convenient and inexpensive device for experimentation. Technology transfers to more robust hardware (e.g. instrumented glove).  Wiimote produces some ambiguous data (e.g. static poses). Additional attachment (e.g. gyroscopes) required for more accuracy.  Experiments indicate promising form of communication – more experiments are needed.

12. Future Work  Determine the maximum number of gestures that can be accurately recognized  Gesture rejection  Dynamic mapping between gesture, sound, and tactile sequence  Scenario development for realistic experimentation (establishing context)  Transmitting signal data via RF (currently sent to local device or via UDP/IP)

13. USMA Collaboration  CDT Robert Darket, CDT Zachary Schaeffer (Principal Investigators)  Application: Training Collect exemplar gestures from SMEs Validate less-experienced soldier’s gestures against exemplars

14. Video Demonstration

15. Paul D. Varcholik ACTIVE Laboratory Institute for Simulation and Training University of Central Florida pvarchol@ist.ucf.edu James L. Merlo LTC, US Army US Military Academy West Point james.merlo@usma.edu