1. Activity Recognition Taiwoo Park May 7, 2013
Bao, Ling, and Stephen S. Intille. "Activity recognition from user-annotated acceleration data." Pervasive Computing. Springer Berlin Heidelberg,
Park, Taiwoo, et al. "E-gesture: a collaborative architecture for energy-efficient gesture recognition with hand-worn sensor and mobile devices."Proceedings of the 9th ACM Conference on Embedded Networked Sensor Systems. ACM, 2011.
Some slides are from CSCI 546 course materials by James Reinebold, USC

2. Activity? Higher level activities Lower level activities
Giving a lecture, having a breakfast, playing soccer…
Lower level activities
Lying on a bed, standing still, running, walking, …

3. An easy example Assumptions: How can we recognize activities?
Sensors on smartphones are only available
Accelerometer, compass, gyroscope, light, …
You can attach smartphones on your body
Only three target activities to recognize
Running
Standing still
Lying on a bed
How can we recognize activities?

4. An easy example (cont’d)
Is the phone being shaken?
Phone orientation
Activity No
Upright
Yes
Lying down
Standing still
Running y
Lying on a bed
Nothing z x
Average value of accelerometer y-axis sensor signals for the last 3 seconds
Variance of accelerometer sensor signal for the last 3 seconds

5. Activity recognition pipeline
Bader, Sebastian, and Thomas Kirste. "A Tutorial Introduction to Automated Activity and Intention Recognition." (2011).

6. An easy example (revisited)
Is the phone being shaken?
Phone orientation
Activity No
Upright
Standing still
Yes
Running
Lying down
Lying on a bed …?
Classification
Average value of accelerometer y-axis sensor signals for the last 3 seconds
Feature extraction
Windowing
Variance of accelerometer sensor signal for the last 3 seconds

7. Data collection Semi-Naturalistic, User-Driven Data Collection
Obstacle course / worksheet
No researcher supervision while subjects performed the tasks
Timer synchronization
Discard data within 10 seconds of start and finish time for activities
12:30 – 12:50
Walking …
Bao, Ling, and Stephen S. Intille. "Activity recognition from user-annotated acceleration data." Pervasive Computing. Springer Berlin Heidelberg,

8. Activities Walking Walking Carrying Items Sitting and Relaxing
Working on Computer
Standing Still
Eating or Drinking
Watching TV
Reading
Running
Bicycling
Stretching
Strength-training
Scrubbing
Vacuuming
Folding Laundry
Lying down & relaxing
Brushing Teeth
Climbing stairs
Riding Elevator
Riding escalator

9. Data collection
Source: Bao 2004

10. Sensors Used
Five ADXL210E accelerometers (manufactured by Analog Devices)
Range of +/- 10g
5mm x 5mm x 2mm
Low Power, Low Cost
Measures both static and dynamic acceleration
Sensor data was stored in a memory card by using “Hoarder Board”
Source:

11. Example Signals
Source: Bao 2004

12. Activity recognition pipeline
Bader, Sebastian, and Thomas Kirste. "A Tutorial Introduction to Automated Activity and Intention Recognition." (2011).

13. Classification
Question: What is the most similar samples to the current one?
Data sample ?
‘Running’
‘Standing still’
‘Lying on a bed’
‘Walking’
Collected data samples (in advance)
Methods: Naïve bayes, nearest neighbor, decision table/tree, HMM (Hidden Markov Models), …

14. Is the phone being shaken?
Decision Table
Is the phone being shaken?
Phone orientation
Activity No
Upright
Yes
Lying down
Standing still
Running y
Lying on a bed
Nothing z x

15. Decision Trees
Make a tree where the non-leaf nodes are the features, and each leaf node is a classification. Each edge of the tree represents a value range of the feature.
Move through the tree until you arrive at a leaf node
Generally, the smaller the tree the better.
Finding the smallest is NP-Hard
Source:

16. Not lying down: Is the phone being shaken?
Decision Tree Example
Phone Orientation?
Lying down
Upright
Lying on a bed
Not lying down: Is the phone being shaken?
Yes No
Running
Standing still

17. Nearest Neighbor
Split up the domain into various dimensions, with each dimension corresponding to a feature.
Classify an unknown point by having its K nearest neighbors “vote” on who it belongs to.
Simple, easy to implement algorithm. Does not work well when there are no clusters.
Source:

18. Nearest Neighbor Example
Running
Variance of accel
Lying on a bed
Standing still
Average value of accel Y

19. Naïve Bayes Classifier
Multiplies the probability of an observed datapoint by looking at the priority probabilities that encompass the training set.
P(B|A) = P(A|B) * P(B) / P(A)
Assumes that each of the features are independent.
Relatively fast.
Stochastic method
Source: cis.poly.edu/~mleung/FRE7851/f07/naiveBayesianClassifier.pdf

20. Activity recognition pipeline
Bader, Sebastian, and Thomas Kirste. "A Tutorial Introduction to Automated Activity and Intention Recognition." (2011).

21. Feature Extraction Time-domain features [Maurer 2006]
Mean (average), Root Mean Square, Variance, …
FFT-based feature computation [Bao 2004]
Sample at Hz
512 sample windows (about 6.71 sec)
Extract mean energy, entropy, and correlation features
Maurer, Uwe, et al. "Activity recognition and monitoring using multiple sensors on different body positions." Wearable and Implantable Body Sensor Networks, BSN International Workshop on. IEEE, 2006.

22. Source: Bao 2004

23. Source: Bao 2004

24. Classification Accuracy (%, Leave-one-subject-out Training)
Results
Decision tree was the best performer, but…
Classifier
Classification Accuracy (%, Leave-one-subject-out Training)
Decision Table /
Nearest Neighbor /
Decision Tree /
Naïve Bayes /

25. Per-activity accuracy breakdown

26. Trying With Less Sensors
Accelerometer (s) Left In
Difference in Recognition Activity
Hip /
Wrist /
Arm /
Ankle /
Thigh /
Thigh and Wrist /
Hip and Wrist /
With only two accelerometers we can get good performance

27. Lessons
Accelerometers can be used to affectively distinguish between everyday activities.
Decision trees and nearest neighbor algorithms are good choices for activity recognition.
Some sensor locations are more important than others.
Selecting a feature set is important to increase recognition accuracy.

28. E-Gesture: A Collaborative Architecture for Energy-efficient Gesture Recognition with Hand-worn Sensor and Mobile Devices
I am really happy to present this work, and thank you for being here.
I am Taiwoo Park and the title of the work I am going to present is E-Gesture. It is an energy-efficient gesture recognition architecture which utilizes hand-worn sensor and mobile devices in a collaborative manner.
This work was done in collaboration with my collagues Jinwon, Inseok and Chungkuk in KAIST, Lama Nachman in Intel Research, and my advisor Junehwa Song.
I’ll first show you a demo video showing the goal of this work.

29. Mobile Gesture Interaction Framework
Motivation
Mobile applications using hand gestures …
Mobile Gesture Interaction Framework
Wristwatch-type
motion sensor
(Accelerometer,
Gyroscope)
Smartphone
We envision that these kind of mobile applications utilizing hand gesture interaction will be frequently used in the near future. <Click> To facilitate the development of these applications, we designed and implemented a framework supporting hand gesture interaction considering users’ mobility. The framework uses wristwatch-type motion sensor and smartphone, because these devices are quite acceptable even in mobile situation.
Mobility!!!!!

30. Challenge: Energy and Accuracy
• Conventional gesture processing pipeline (for gesture recognition in stationary setting)
Sensor
Mobile Device
Data Sensing
Gesture Segmentation
(Button, Algorithms)
Classification
(HMM, DTW)
Continuous Raw Data
Gesture
Samples
(Candidate)
Result
Accel
Gyro
Gesture ‘A’ or ‘B’ or non-gesture
The major design challenges of the mobile gesture interaction framework are energy and accuracy. To help understanding, I’ll show you a figure, showing the conventional gesture processing pipeline for stationary gesture recognition. <Click> First, the sensor continuously takes hand motion sensor data, and transmit the data to mobile device. <Click> In mobile device, the segmentation module detects hand movements which could be a intended user gestures. <Click> The detected hand movement is sent to the classification stage, in a form of time sequence of motion signals, and classified to one of the predefined gestures or non-gesture garbage by machine learning algorithms such as Hidden Markov Model or Dynamic Time Warping or so. Again, the conventional gesture processing architecture has three stages, sensing, segmentation, and classification.

31. Challenge: Energy and Accuracy
Sensor: 20hrs (250mAh)
Smartphone: 24hrs  17hrs
Sensor
Mobile Device
Data Sensing
Gesture Segmentation
(Button, Algorithms)
Classification
(HMM, DTW)
Continuous Raw Data
Gesture
Samples
(Candidate)
Result
Accel
Continuous data transmission
Energy
Accuracy
Mobility noises
Accuracy
Mobility noises
Energy
Energy-hungry
Gyroscope
(56%)
Gyro
This processing pipeline gives several challenges in mobile use. <Click> First, energy consumption is quite high, because the sensor and the mobile device continuously communicate to send and receive hand motion data. <Click> Also, energy-hungry gyroscope should be always turned on, and it consumes significant amount of energy in the sensor device, more than 50% of total sensor energy consumption. As a result, <Click> we observed that a battery lifetime of a sensor device is 20 hours with 250 mAh battery. Also, the gesture processing shortened smartphones’ lifetime from 24 hours to 17 hours. <Pause and Click> Second, this system fails to detect and classify hand gestures with mobility noises. Users’ mobility such as running and walking produces body noises, which makes it hard to accurately recognize hand gestures. As a result, <click> In gesture segmentation, it shows over 90% of false positive errors in segmentation, that is, falsely detected hand motions. Also, gesture classification shows only 70% of accuracy in mobile settings. Note that conventional gesture processing pipeline does not address mobile use. We designed E-Gesture architecture to address these two challenges, by redesigning this pipeline.
Mobility Noises
Over 90% False segmentation
Only 70% Classification

32. E-Gesture Architecture
1. Device-wise collaboration
Detection on wristwatch, classification on smartphone
Collaborative Gesture Sensing and Segmentation
Classification
(Adaptive and Multi-Situation HMM)
Gesture Samples
(Candidate)
Result
(e.g. laydown)
Accel
Gyro
Trigger
Adaptation
Wristwatch Sensor Device
Mobile Device
2. Sensor-wise collaboration
Accel turns on gyro for energy efficiency
Gyro adapts accel’s sensitivity for mobility changes
Accelerometer: (+)Energy-efficient, (-)Mobility-vulnerable
Gyroscope: (-)Energy-hungry, (+)Mobility-robust
This is an overall architecture of e-gesture. I showed you the conventional gesture processing architecture with three stages, sensing, segmentation, and classification. We carefully redesigned and distributed the stages for our framework, considering energy efficiency and mobility noises. The key design concept is two kind of collaboration. <Click> One is the collaboration between sensor and mobile device, that is, we moved gesture segmentation into sensor device. By this, we could selectively transmit gesture candidate samples only, and reduced energy consumption from continuous data transmission. <Click> The another one is the collaboration between two types of motion sensors, accelerometer and gyroscope. We found that accelerometer and gyroscope have very interesting characteristics in terms of energy and mobility robustness. That is, accelerometer consumes small amount of energy, but it is vulnerable to mobility noise. However, Gyroscope is energy-hungry but robust to mobility. We utilized these complementary characteristics. In brief, accelerometer monitors hand motion signal, and turns on gyroscope when it detects significant hand motion. For this we used simple threshold-based detection algorithm. Then, gyroscope immediately wake up and validates motion signal whether the user is really performing significant hand motion. We also designed feedback adaptation, that is, if the accelerometer turns on gyroscope too frequently because of mobility noise, gyroscope controls the sensitivity of accelerometer by feedback adaptation loop.

33. Sensor-side Energy Savings
46mW
Continuous sensing + transmission:
20 hrs
39mW (↓15%)
Device-wise collaboration
(reduced transmission)
23.7 hrs (1.2x)
250mAh Li-ion Battery
Device-wise, Sensor-wise collaboration
(gyroscope power control,
reduced transmission)
48.7 hrs (2.4x)
19mW (↓59%)
We evaluated the energy efficiency of our architecture on sensor-side. We used a customly-designed sensor mote which is compatible with MicaZ sensor mote, and utilized 250mAh Li-ion battery for operation. The detailed evaluation settings can be found on our SenSys paper. <Click> First, continuous sensing and transmission of accelerometer and gyroscope samples requires all components in sensors to be turned on, so consumes maximum energy. In our setting, the sensor runs up to 20 hours. However, <Click> if it transmits detected gesture samples, we can save energy consumption for radio transmission and the lifetime gets 1.2 times longer. Finally, <Click> if we selectively turn on gyroscope by sensor-wise collaboration, it reduces 59% of energy consumption and shows 2.4 times longer battery lifetime, more than 48 hours.
Energy Consumption
59% less energy consumption, 2.4x longer lifetime

34. Mobile-side Energy Savings
122mW
All processing on mobile:
42.1hrs
Nexus One
1400mAh Li-ion Battery
3G/WiFi on
70mW(↓43%)
Device-wise collaboration
(reduced transmission)
74hrs (1.8x)
Next, we evaluated mobile device-side energy savings. We used nexus one with 14 hundred mAh battery. If we always turn-on radio transmission component in mobile device, the device lasts only 42 hours. However, through our device-wise collaboration, we could turn-off the radio component while user does not perform hand gestures. As a result, we could reduce 43% of energy consumption, and increase the battery lifetime to 74 hours.
Energy Consumption

35. Implementation Sensor node Smartphones Atmega128L MCU
Bluetooth, ZigBee
Sensors
3-Axis Accelerometer (ADXL335)
3-Axis Gyroscope (3 XV-3500CB)
40Hz Sensing
Vib motor
Smartphones
Nokia N96, Google Nexus One
Bluetooth Radio
We implemented our framework on a sensor node containing atmega128L, CC2420 for radio, and two types of motion sensors. Also we used two types of smartphones, Nokia N96 and Google Nexus One. As the smartphones do not have zigbee-compatible radio, so we used a bridge node to convert zigbee signals to bluetooth signal. It receives gesture data from the sensor via zigbee, and forward the data to the smartphones via bluetooth. <pause>
For your information, we used nexus one for evaluation.

36. Sample Applications Swan Boat [Ubicomp09][MM09][ACE09]
Collaborative boat-racing exertion game
Utilizes hand gestures as additional game input
Punching together, flapping together
Mobile Music Player, Phone Call Manager
Featuring eye-free, touch-free controls
User can control the application by hand gestures
Using E-Gesture, we have made several sample applications. One is Swan Boat, a collaborative boat-racing exertion game. Some of you may have seen the demonstration last Tuesday. Two runners steer one boat together by the difference of running speeds. For details on game play, you can easily find the papers and videos describing this game and it’s really fun. To enhance the game interaction, we tried to utilize hand gestures as additional game input, and E-Gesture greatly helped the gesture processing in the game. <pause>
I showed the music player and the phone call manager. So I’ll pass the detail.

37. Conclusion Mobile gestural interaction platform
Collaborative gesture processing
1.8x longer battery lifetime for Smartphone
2.4x longer battery lifetime for Hand-worn Sensor + preserving gyroscope’s detection performance
Mobility-robust gesture classification using HMM
Up to 94.6% of classification accuracy on mobile usage by mobility-considered classification architecture design
It will greatly facilitate gesture-based mobile applications
Provided a novel sensor fusion scheme
Serial fusion + feedback control
Saves energy + preserves detection accuracy
I’ll conclude by talking about the key points. First, We designed and implemented a mobile gesture interaction platform to support applications using hand gestures. We proposed collaborative gesture processing schemes, and it achieved 2.4 times and 1.8 times longer battery lifetime on the sensor and the smartphone, while preserving gyroscope’s better performance of gesture detection. Also we achieved up to about 95% of classification accuracy even in the mobile usage. The time is somewhat limited so I passed the classification detail, and you can easily find them in the paper. Please note that we provided a novel sensor fusion scheme. In our work, we utilized accelerometer and gyroscope which have complementary characteristics on energy consumption and detection performance. By using serial fusion and feedback control, we saved much energy while preserving detection accuracy as well. We hope that there would be another way to utilize this kind of sensor fusion. <pause> That’s it. Thank you.