1. E-Gesture: A Collaborative Architecture for Energy-efficient Gesture Recognition with Hand-worn Sensor and Mobile Devices Taiwoo Park 1, Jinwon Lee 1, Inseok Hwang 1, Chungkuk Yoo 1, Lama Nachman 2, Junehwa Song 1 KAIST 1, Intel Corporation 2 ACM SenSys 2011

2. Hand Gesture-Based Mobile Interaction

3. Smartphone Mobile Gesture Interaction Framework … Applications Motion Sensor(s)

4. Design Challenges Providing Energy-efficient Gesture Processing Accurately Detecting and Classifying Hand Gestures Mobility Noises (Segmenting)

5. Design Challenges Providing Energy-efficient Gesture Processing Accurately Detecting and Classifying Hand Gestures Mobility Noises 20hrs Sensor, 250mAh 24hrs  17hrs Smartphone Over 90% False detections Only 70% Classification  (Segmenting)

6. Design Challenges Providing Energy-efficient Gesture Processing Accurately Detecting and Classifying Hand Gestures Mobility Noises 20hrs Sensor, 250mAh 24hrs  17hrs Smartphone Over 90% False detections Only 70% Classification  (Segmenting)

7. E-Gesture Architecture Wristwatch DeviceSmartphone ① Sensing ② Detection ③ Classification ④ Notification (Segmentation) 1 2 3 4

8. Collaboration between devices Detection on wristwatch, classification on smartphone Collaboration between sensors Accel triggers gyro for energy efficiency Gyro adapts accel to mobility changes Why ‘collaborative’?

9. Approaches Investigated characteristics of Accel and Gyro – Accelerometer: Mobility-Sensitive, Energy-Efficient – Gyroscope: Mobility-Robust, Energy-Hungry Designed energy-efficient, mobility-robust gesture detection architecture – Triggering Gyroscope by analyzing Accelerometer Signal – Adjusting Accelerometer sensitivity by Gyroscope Validation Suggested two gesture classification architectures considering users’ mobilities (based on HMM) (threshold)

10. Approaches Investigated characteristics of Accel and Gyro – Accelerometer: Mobility-Sensitive, Energy-Efficient – Gyroscope: Mobility-Robust, Energy-Hungry Designed energy-efficient, mobility-robust gesture detection architecture – Triggering Gyroscope by analyzing Accelerometer Signal – Adjusting Accelerometer sensitivity by Gyroscope Validation Suggested two gesture classification architectures considering users’ mobilities (based on HMM) (threshold)

11. Approaches Investigated characteristics of Accel and Gyro – Accelerometer: Mobility-Sensitive, Energy-Efficient – Gyroscope: Mobility-Robust, Energy-Hungry Designed energy-efficient, mobility-robust gesture detection architecture – Triggering Gyroscope by analyzing Accelerometer Signal – Adjusting Accelerometer sensitivity by Gyroscope Validation Suggested two gesture classification architectures considering users’ mobilities (based on HMM) (threshold)

12. Energy-Performance Tradeoff of Accel and Gyro Energy Consumption Mobility Robustness Segmentation Accuracy Accel-based LowPoorPassable Gyro-based High (9x accel)Good

13. Example of Mobility Noises Standing stillWalkingRunning

14. Waveform of real mobile gesture Accel Gyro RMS of force Waveform of ‘Touching ear’ while running RMS of rotation Mobility Noise

15. Accel threshold should fit to mobility Lower fixed threshold  False-positives on high mobility Higher fixed threshold  False-negatives on low mobility

16. Mobility Situation RIDESTANDWALKRUN Accel-based0.15G 0.2G0.35G Gyro-based25 degree/sec Optimal threshold for Accel and Gyro (minimizes FPs without incurring FNs) False-Positives per mobility with optimal threshold

17. However gyroscope is energy-hungry.. Sensor-side Energy Profile (Atmega128L, CC2420, Accel and Gyro) Accel Gyroscope

18. However gyroscope is energy-hungry.. Accel Gyro Accel and Gyro are complementary in terms of performance and energy

19. Approaches Investigated characteristics of Accel and Gyro – Accelerometer: Mobility-Sensitive, Energy-Efficient – Gyroscope: Mobility-Robust, Energy-Hungry Designed energy-efficient, mobility-robust gesture detection architecture – Triggering Gyroscope by analyzing Accelerometer Signal – Adjusting Accelerometer threshold by Gyroscope Validation Suggested two gesture classification architectures considering users’ mobilities (based on HMM) (threshold)

20. Closed-loop Collaborative Segmentation Gyro-based Detector Gyro-based Detector Gyro-based Detector Accurate, High energy

21. Closed-loop Collaborative Segmentation Gyro-based Detector Gyro-based Detector Trigger Gyro-based Detector Gyro-based Detector Accel-based Detector Accel-based Detector Gyro-based Detector Open-loop Detector Accurate, High energy

22. Closed-loop Collaborative Segmentation Gyro-based Detector Gyro-based Detector Trigger Gyro-based Detector Gyro-based Detector Accel-based Detector Accel-based Detector Gyro-based Detector Open-loop Detector Accurate, High energy (because of mobility)

23. Closed-loop Collaborative Segmentation Accurate, Low energy Trigger Gyro-based Detector Gyro-based Detector Accel-based Detector Accel-based Detector Feedback Closed-loop Collaborative Detector

24. Closed-loop Collaborative Segmentation Accurate, Low energy Trigger Gyro-based Detector Gyro-based Detector Accel-based Detector Accel-based Detector Feedback Closed-loop Collaborative Detector Performance-preserving, Energy-saving Collaborative Sensor Fusion

25. Approaches Investigated characteristics of Accel and Gyro – Accelerometer: Mobility-Sensitive, Energy-Efficient – Gyroscope: Mobility-Robust, Energy-Hungry Designed energy-efficient, mobility-robust gesture detection architecture – Triggering Gyroscope by analyzing Accelerometer Signal – Adjusting Accelerometer sensitivity by Gyroscope Validation Suggested two gesture classification models considering users’ mobilities (based on HMM) (threshold)

26. Basic HMM Trained with samples collected in stationary setting Classification accuracy drops in mobile situation Model for Gesture A Model for Gesture B Model for Garbages Gesture Candidate Sample Gesture Candidate Sample Gesture A or Gesture B or … or Garbage Gesture A or Gesture B or … or Garbage Raw, Delta, Integral (18 features) Probabilities … 8-state left-right HMMs

27. Basic HMM Trained with samples collected in stationary setting Classification accuracy drops in mobile situation Model for Gesture A Model for Gesture B Model for Garbages Candidate Gesture Sample Candidate Gesture Sample Gesture A or Gesture B or … or Garbage Gesture A or Gesture B or … or Garbage Raw, Delta, Integral (18 features) Probabilities … Design alternatives: 1) Adapt models to mobility changes (in run-time) 2) Train several different models for predefined set of mobility situations Design alternatives: 1) Adapt models to mobility changes (in run-time) 2) Train several different models for predefined set of mobility situations 8-state left-right HMMs

28. Adaptive HMM Update the models with gesture samples – Negative update scheme of uWave [PerCom09] – By MLLR (Maximum Likelihood Linear Regression) adaptation On-line adaptation with users’ gesture samples Model for Gesture A Model for Gesture B Model for Garbages Candidate Gesture Sample Candidate Gesture Sample Gesture A or Gesture B or … or Garbage Gesture A or Gesture B or … or Garbage … Wrong!!! Raw, Delta, Integral (18 features) Probabilities

29. Multi-Situation HMM Train models separately for representative mobility situations – e.g. Riding a car, Standing, Walking, Running Classify by evaluating all models Candidate Gesture Sample Candidate Gesture Sample Gesture A or Gesture B or … or Garbage Gesture A or Gesture B or … or Garbage Model for Gesture A Model for Gesture B Model for Garbages … Model for Gesture A Model for Gesture B … Model for Gesture A Model for Gesture B … Trained for RIDE Trained for STAND Trained for WALK Raw, Delta, Integral (18 features) Probabilities

30. Brief Comparison between Classification Architectures BasicAdaptive Multi- Situation Adaptation cost (Users’ burden) nonelargenone Training cost# of gestures # of gestures x # of mobile situations Evaluation cost (Processing) # of gestures # of gestures x # of mobile situations

31. Brief Comparison between Classification Architectures BasicAdaptive Multi- Situation Adaptation cost (Users’ burden) nonelargenone Training cost# of gestures # of gestures x # of mobile situations Evaluation cost (Processing) # of gestures # of gestures x # of mobile situations

32. IMPLEMENTATION

33. Sensor node – Atmega128L MCU – CC2420 Zigbee Radio – Sensors 3-Axis Accelerometer (ADXL335) 3-Axis Gyroscope (3 XV-3500CB) 40Hz Sensing – Vib motor Smartphones – Nokia N96, Google Nexus One – Bluetooth Radio Bridge node to convert Zigbee  Bluetooth Implementation

34. Ported HTK to Mobile OSs – Google Android 2.1 with NDK rev.3 – Nokia S60 3.2.0 with Open C++ library Application interface – Helps application developers easily define and develop gesture UIs

35. Sample Applications Swan Boat [Ubicomp09][MM09][ACE09] – Collaborative boat-racing exertion game – Utilizes hand gestures as additional game input Punching together, flapping together Mobile Music Player, Phone Call Manager – Featuring eye-free, touch-free controls – User can control the application by hand gestures

36. EVALUATION Gesture Data Workload Threshold(Sensitivity) Adaptation Energy Efficiency Classification Accuracy

37. Gesture Data Workload 4 Representative mobility situations – Riding a car, Standing, Walking, Running 8 Intuitive gestures Data Collection – 4 situations × 8 gestures × 30 samples × 7 participants = Collected 6720 gesture samples in total – Also collected non-gestures to generate test workloads Workload configuration (for energy efficiency) – Ratio of gestures: 10% of total time – Mobility mixture: 75% from stationary (50% STAND, 25% RIDE) 25% from mobile (12.5% WALK, 12.5% RUN)

38. Higher Threshold in Higher Mobility Lower Threshold in Lower Mobility Threshold adaptation of Closed-loop detector Q: Does the closed-loop collaborative detector adapt accelerometer threshold well?

39. Performance of Closed-loop Detector False positives from accel detector False negatives from accel detector Q: How much does the closed-loop detector suppress false-positives and false-negatives from the accel-based detector?

40. Next two questions: How much energy does the E-Gesture save? – Sensor – Smartphone Does the suggested classification models provide sufficient accuracy for user interaction?

41. Sensor-side Energy Savings from Closed-loop Architecture 46mW 39mW (↓15%) 19mW (↓59%) 59% less energy consumption, 2.4x longer lifetime 250mAh Li-ion Battery transmit raw sensing data: 20 hrs transmit only detected gestures (no sensor control) 23.7 hrs (1.2x) transmit only detected gestures (closed-loop detection): 48.7 hrs (2.4x) Energy Consumption

42. Mobile-side Energy Savings from Sensor-side gesture detection All processing on mobile: 42.1hrs Sensor-side Gesture Detection: 74hrs (1.8x) 122mW 70mW(↓43%) 1400mAh Li-ion Battery 3G/WiFi on Energy Consumption

43. Gesture Classification Performance Adaptation Cost 4x Gesture Models (Precision)

44. Conclusion A mobile gestural interaction platform A collaborative gesture interaction architecture – Collaborative sensor-mobile gesture data processing 1.8x longer battery lifetime for Smartphone – Closed-loop collaborative sensor-side segmentation 2.4x longer battery lifetime for Hand-worn Sensor + preserving gyroscope’s detection performance Experiments under different representative mobile situations – Up to 94.6% of classification accuracy on mobile usage by mobility-considered classification architecture design Thanks! Questions?

46. Conclusion 2.4x longer battery lifetime for Hand-worn Sensor + preserving gyroscope’s detection performance by sensor-wise collaboration 1.8x longer battery lifetime for Smartphone by device-wise collaboration Up to 94.6% of classification accuracy on mobile usage by mobility-considered classification architecture design Achievements: Opportunities for further energy savings – Early filtering on sensor devices Improvement on classification architecture Future Work:

47. Accuracy breakdown by mobility situation

48. Sensor node – Atmega128L MCU – CC2420 Zigbee Radio – Sensors 3-Axis Accelerometer (ADXL335) 3-Axis Gyroscope (3 XV-3500CB) 40Hz Sensing – Vib motor Smartphones – Nokia N96, Google Nexus One – Bluetooth Radio Bridge node to convert Zigbee  Bluetooth Implementation Detail (Hardware) We carefully chose gyroscope with very small latency – <30ms: XV-3500CB (Epson) Drops one sample  Same accuracy with complete samples – >50ms: ADXRS620 (Analog Devices), IDG-500 (Invensense) Drops two+ samples  0.4%+ of accuracy loss OUR CHOICE

49. Accel-based trigger alone is not effective: Mobility noises trigger gyroscope very frequently FN with a High Threshold (Accel-based Segmentor) FP with a Low Threshold (Accel-based Segmentor) Accel Gyro Trigger hard to use higher threshold because of FNs

50. If we take in account the bridge node: 51hrs (1.2x)

51. Accel’s sensitivity should fit to mobility Mobility Situation RIDESTANDWALKRUN Accel-based0.15G 0.2G0.35G Gyro-based25 degree/sec Optimal threshold for Accel and Gyro (minimizing false detections without ignorance) Higher Fixed Sensitivity  False Detections on Higher mobility Lower Fixed Sensitivity  Ignorance on Lower mobility Precision ↓ Recall ↓

52. Closed-loop Collaborative Segmentation Gyro-based Detector Gyro-based Detector Accurate, High energy or Insensitive, Low energy Accurate, Low energy Trigger Gyro-based Detector Gyro-based Detector Accel-based Detector Accel-based Detector Trigger Gyro-based Detector Gyro-based Detector Accel-based Detector Accel-based Detector Feedback Gyro-based Detector Open-loop Detector Closed-loop Collaborative Detector