Presentation is loading. Please wait.

Presentation is loading. Please wait.

Activity Recognition Taiwoo Park May 7, 2013

Similar presentations

Presentation on theme: "Activity Recognition Taiwoo Park May 7, 2013"— Presentation transcript:

1 Activity Recognition Taiwoo Park May 7, 2013
Bao, Ling, and Stephen S. Intille. "Activity recognition from user-annotated acceleration data." Pervasive Computing. Springer Berlin Heidelberg, Park, Taiwoo, et al. "E-gesture: a collaborative architecture for energy-efficient gesture recognition with hand-worn sensor and mobile devices."Proceedings of the 9th ACM Conference on Embedded Networked Sensor Systems. ACM, 2011. Some slides are from CSCI 546 course materials by James Reinebold, USC

2 Activity? Higher level activities Lower level activities
Giving a lecture, having a breakfast, playing soccer… Lower level activities Lying on a bed, standing still, running, walking, …

3 An easy example Assumptions: How can we recognize activities?
Sensors on smartphones are only available Accelerometer, compass, gyroscope, light, … You can attach smartphones on your body Only three target activities to recognize Running Standing still Lying on a bed How can we recognize activities?

4 An easy example (cont’d)
Is the phone being shaken? Phone orientation Activity No Upright Yes Lying down Standing still Running y Lying on a bed Nothing z x Average value of accelerometer y-axis sensor signals for the last 3 seconds Variance of accelerometer sensor signal for the last 3 seconds

5 Activity recognition pipeline
Bader, Sebastian, and Thomas Kirste. "A Tutorial Introduction to Automated Activity and Intention Recognition." (2011).

6 An easy example (revisited)
Is the phone being shaken? Phone orientation Activity No Upright Standing still Yes Running Lying down Lying on a bed …? Classification Average value of accelerometer y-axis sensor signals for the last 3 seconds Feature extraction Windowing Variance of accelerometer sensor signal for the last 3 seconds

7 Data collection Semi-Naturalistic, User-Driven Data Collection
Obstacle course / worksheet No researcher supervision while subjects performed the tasks Timer synchronization Discard data within 10 seconds of start and finish time for activities 12:30 – 12:50 Walking Bao, Ling, and Stephen S. Intille. "Activity recognition from user-annotated acceleration data." Pervasive Computing. Springer Berlin Heidelberg,

8 Activities Walking Walking Carrying Items Sitting and Relaxing
Working on Computer Standing Still Eating or Drinking Watching TV Reading Running Bicycling Stretching Strength-training Scrubbing Vacuuming Folding Laundry Lying down & relaxing Brushing Teeth Climbing stairs Riding Elevator Riding escalator

9 Data collection Source: Bao 2004

10 Sensors Used Five ADXL210E accelerometers (manufactured by Analog Devices) Range of +/- 10g 5mm x 5mm x 2mm Low Power, Low Cost Measures both static and dynamic acceleration Sensor data was stored in a memory card by using “Hoarder Board” Source:

11 Example Signals Source: Bao 2004

12 Activity recognition pipeline
Bader, Sebastian, and Thomas Kirste. "A Tutorial Introduction to Automated Activity and Intention Recognition." (2011).

13 Classification Question: What is the most similar samples to the current one? Data sample ? ‘Running’ ‘Standing still’ ‘Lying on a bed’ ‘Walking’ Collected data samples (in advance) Methods: Naïve bayes, nearest neighbor, decision table/tree, HMM (Hidden Markov Models), …

14 Is the phone being shaken?
Decision Table Is the phone being shaken? Phone orientation Activity No Upright Yes Lying down Standing still Running y Lying on a bed Nothing z x

15 Decision Trees Make a tree where the non-leaf nodes are the features, and each leaf node is a classification. Each edge of the tree represents a value range of the feature. Move through the tree until you arrive at a leaf node Generally, the smaller the tree the better. Finding the smallest is NP-Hard Source:

16 Not lying down: Is the phone being shaken?
Decision Tree Example Phone Orientation? Lying down Upright Lying on a bed Not lying down: Is the phone being shaken? Yes No Running Standing still

17 Nearest Neighbor Split up the domain into various dimensions, with each dimension corresponding to a feature. Classify an unknown point by having its K nearest neighbors “vote” on who it belongs to. Simple, easy to implement algorithm. Does not work well when there are no clusters. Source:

18 Nearest Neighbor Example
Running Variance of accel Lying on a bed Standing still Average value of accel Y

19 Naïve Bayes Classifier
Multiplies the probability of an observed datapoint by looking at the priority probabilities that encompass the training set. P(B|A) = P(A|B) * P(B) / P(A) Assumes that each of the features are independent. Relatively fast. Stochastic method Source:

20 Activity recognition pipeline
Bader, Sebastian, and Thomas Kirste. "A Tutorial Introduction to Automated Activity and Intention Recognition." (2011).

21 Feature Extraction Time-domain features [Maurer 2006]
Mean (average), Root Mean Square, Variance, … FFT-based feature computation [Bao 2004] Sample at Hz 512 sample windows (about 6.71 sec) Extract mean energy, entropy, and correlation features Maurer, Uwe, et al. "Activity recognition and monitoring using multiple sensors on different body positions." Wearable and Implantable Body Sensor Networks, BSN International Workshop on. IEEE, 2006.

22 Source: Bao 2004

23 Source: Bao 2004

24 Classification Accuracy (%, Leave-one-subject-out Training)
Results Decision tree was the best performer, but… Classifier Classification Accuracy (%, Leave-one-subject-out Training) Decision Table / Nearest Neighbor / Decision Tree / Naïve Bayes /

25 Per-activity accuracy breakdown

26 Trying With Less Sensors
Accelerometer (s) Left In Difference in Recognition Activity Hip / Wrist / Arm / Ankle / Thigh / Thigh and Wrist / Hip and Wrist / With only two accelerometers we can get good performance

27 Lessons Accelerometers can be used to affectively distinguish between everyday activities. Decision trees and nearest neighbor algorithms are good choices for activity recognition. Some sensor locations are more important than others. Selecting a feature set is important to increase recognition accuracy.

28 E-Gesture: A Collaborative Architecture for Energy-efficient Gesture Recognition with Hand-worn Sensor and Mobile Devices I am really happy to present this work, and thank you for being here. I am Taiwoo Park and the title of the work I am going to present is E-Gesture. It is an energy-efficient gesture recognition architecture which utilizes hand-worn sensor and mobile devices in a collaborative manner. This work was done in collaboration with my collagues Jinwon, Inseok and Chungkuk in KAIST, Lama Nachman in Intel Research, and my advisor Junehwa Song. I’ll first show you a demo video showing the goal of this work.

29 Mobile Gesture Interaction Framework
Motivation Mobile applications using hand gestures Mobile Gesture Interaction Framework Wristwatch-type motion sensor (Accelerometer, Gyroscope) Smartphone We envision that these kind of mobile applications utilizing hand gesture interaction will be frequently used in the near future. <Click> To facilitate the development of these applications, we designed and implemented a framework supporting hand gesture interaction considering users’ mobility. The framework uses wristwatch-type motion sensor and smartphone, because these devices are quite acceptable even in mobile situation. Mobility!!!!!

30 Challenge: Energy and Accuracy
• Conventional gesture processing pipeline (for gesture recognition in stationary setting) Sensor Mobile Device Data Sensing Gesture Segmentation (Button, Algorithms) Classification (HMM, DTW) Continuous Raw Data Gesture Samples (Candidate) Result Accel Gyro Gesture ‘A’ or ‘B’ or non-gesture The major design challenges of the mobile gesture interaction framework are energy and accuracy. To help understanding, I’ll show you a figure, showing the conventional gesture processing pipeline for stationary gesture recognition. <Click> First, the sensor continuously takes hand motion sensor data, and transmit the data to mobile device. <Click> In mobile device, the segmentation module detects hand movements which could be a intended user gestures. <Click> The detected hand movement is sent to the classification stage, in a form of time sequence of motion signals, and classified to one of the predefined gestures or non-gesture garbage by machine learning algorithms such as Hidden Markov Model or Dynamic Time Warping or so. Again, the conventional gesture processing architecture has three stages, sensing, segmentation, and classification.

31 Challenge: Energy and Accuracy
Sensor: 20hrs (250mAh) Smartphone: 24hrs  17hrs Sensor Mobile Device Data Sensing Gesture Segmentation (Button, Algorithms) Classification (HMM, DTW) Continuous Raw Data Gesture Samples (Candidate) Result Accel Continuous data transmission Energy Accuracy Mobility noises Accuracy Mobility noises Energy Energy-hungry Gyroscope (56%) Gyro This processing pipeline gives several challenges in mobile use. <Click> First, energy consumption is quite high, because the sensor and the mobile device continuously communicate to send and receive hand motion data. <Click> Also, energy-hungry gyroscope should be always turned on, and it consumes significant amount of energy in the sensor device, more than 50% of total sensor energy consumption. As a result, <Click> we observed that a battery lifetime of a sensor device is 20 hours with 250 mAh battery. Also, the gesture processing shortened smartphones’ lifetime from 24 hours to 17 hours. <Pause and Click> Second, this system fails to detect and classify hand gestures with mobility noises. Users’ mobility such as running and walking produces body noises, which makes it hard to accurately recognize hand gestures. As a result, <click> In gesture segmentation, it shows over 90% of false positive errors in segmentation, that is, falsely detected hand motions. Also, gesture classification shows only 70% of accuracy in mobile settings. Note that conventional gesture processing pipeline does not address mobile use. We designed E-Gesture architecture to address these two challenges, by redesigning this pipeline. Mobility Noises Over 90% False segmentation Only 70% Classification

32 E-Gesture Architecture
1. Device-wise collaboration Detection on wristwatch, classification on smartphone Collaborative Gesture Sensing and Segmentation Classification (Adaptive and Multi-Situation HMM) Gesture Samples (Candidate) Result (e.g. laydown) Accel Gyro Trigger Adaptation Wristwatch Sensor Device Mobile Device 2. Sensor-wise collaboration Accel turns on gyro for energy efficiency Gyro adapts accel’s sensitivity for mobility changes Accelerometer: (+)Energy-efficient, (-)Mobility-vulnerable Gyroscope: (-)Energy-hungry, (+)Mobility-robust This is an overall architecture of e-gesture. I showed you the conventional gesture processing architecture with three stages, sensing, segmentation, and classification. We carefully redesigned and distributed the stages for our framework, considering energy efficiency and mobility noises. The key design concept is two kind of collaboration. <Click> One is the collaboration between sensor and mobile device, that is, we moved gesture segmentation into sensor device. By this, we could selectively transmit gesture candidate samples only, and reduced energy consumption from continuous data transmission. <Click> The another one is the collaboration between two types of motion sensors, accelerometer and gyroscope. We found that accelerometer and gyroscope have very interesting characteristics in terms of energy and mobility robustness. That is, accelerometer consumes small amount of energy, but it is vulnerable to mobility noise. However, Gyroscope is energy-hungry but robust to mobility. We utilized these complementary characteristics. In brief, accelerometer monitors hand motion signal, and turns on gyroscope when it detects significant hand motion. For this we used simple threshold-based detection algorithm. Then, gyroscope immediately wake up and validates motion signal whether the user is really performing significant hand motion. We also designed feedback adaptation, that is, if the accelerometer turns on gyroscope too frequently because of mobility noise, gyroscope controls the sensitivity of accelerometer by feedback adaptation loop.

33 Sensor-side Energy Savings
46mW Continuous sensing + transmission: 20 hrs 39mW (↓15%) Device-wise collaboration (reduced transmission) 23.7 hrs (1.2x) 250mAh Li-ion Battery Device-wise, Sensor-wise collaboration (gyroscope power control, reduced transmission) 48.7 hrs (2.4x) 19mW (↓59%) We evaluated the energy efficiency of our architecture on sensor-side. We used a customly-designed sensor mote which is compatible with MicaZ sensor mote, and utilized 250mAh Li-ion battery for operation. The detailed evaluation settings can be found on our SenSys paper. <Click> First, continuous sensing and transmission of accelerometer and gyroscope samples requires all components in sensors to be turned on, so consumes maximum energy. In our setting, the sensor runs up to 20 hours. However, <Click> if it transmits detected gesture samples, we can save energy consumption for radio transmission and the lifetime gets 1.2 times longer. Finally, <Click> if we selectively turn on gyroscope by sensor-wise collaboration, it reduces 59% of energy consumption and shows 2.4 times longer battery lifetime, more than 48 hours. Energy Consumption 59% less energy consumption, 2.4x longer lifetime

34 Mobile-side Energy Savings
122mW All processing on mobile: 42.1hrs Nexus One 1400mAh Li-ion Battery 3G/WiFi on 70mW(↓43%) Device-wise collaboration (reduced transmission) 74hrs (1.8x) Next, we evaluated mobile device-side energy savings. We used nexus one with 14 hundred mAh battery. If we always turn-on radio transmission component in mobile device, the device lasts only 42 hours. However, through our device-wise collaboration, we could turn-off the radio component while user does not perform hand gestures. As a result, we could reduce 43% of energy consumption, and increase the battery lifetime to 74 hours. Energy Consumption

35 Implementation Sensor node Smartphones Atmega128L MCU
Bluetooth, ZigBee Sensors 3-Axis Accelerometer (ADXL335) 3-Axis Gyroscope (3 XV-3500CB) 40Hz Sensing Vib motor Smartphones Nokia N96, Google Nexus One Bluetooth Radio We implemented our framework on a sensor node containing atmega128L, CC2420 for radio, and two types of motion sensors. Also we used two types of smartphones, Nokia N96 and Google Nexus One. As the smartphones do not have zigbee-compatible radio, so we used a bridge node to convert zigbee signals to bluetooth signal. It receives gesture data from the sensor via zigbee, and forward the data to the smartphones via bluetooth. <pause> For your information, we used nexus one for evaluation.

36 Sample Applications Swan Boat [Ubicomp09][MM09][ACE09]
Collaborative boat-racing exertion game Utilizes hand gestures as additional game input Punching together, flapping together Mobile Music Player, Phone Call Manager Featuring eye-free, touch-free controls User can control the application by hand gestures Using E-Gesture, we have made several sample applications. One is Swan Boat, a collaborative boat-racing exertion game. Some of you may have seen the demonstration last Tuesday. Two runners steer one boat together by the difference of running speeds. For details on game play, you can easily find the papers and videos describing this game and it’s really fun. To enhance the game interaction, we tried to utilize hand gestures as additional game input, and E-Gesture greatly helped the gesture processing in the game. <pause> I showed the music player and the phone call manager. So I’ll pass the detail.

37 Conclusion Mobile gestural interaction platform
Collaborative gesture processing 1.8x longer battery lifetime for Smartphone 2.4x longer battery lifetime for Hand-worn Sensor + preserving gyroscope’s detection performance Mobility-robust gesture classification using HMM Up to 94.6% of classification accuracy on mobile usage by mobility-considered classification architecture design It will greatly facilitate gesture-based mobile applications Provided a novel sensor fusion scheme Serial fusion + feedback control Saves energy + preserves detection accuracy I’ll conclude by talking about the key points. First, We designed and implemented a mobile gesture interaction platform to support applications using hand gestures. We proposed collaborative gesture processing schemes, and it achieved 2.4 times and 1.8 times longer battery lifetime on the sensor and the smartphone, while preserving gyroscope’s better performance of gesture detection. Also we achieved up to about 95% of classification accuracy even in the mobile usage. The time is somewhat limited so I passed the classification detail, and you can easily find them in the paper. Please note that we provided a novel sensor fusion scheme. In our work, we utilized accelerometer and gyroscope which have complementary characteristics on energy consumption and detection performance. By using serial fusion and feedback control, we saved much energy while preserving detection accuracy as well. We hope that there would be another way to utilize this kind of sensor fusion. <pause> That’s it. Thank you.


Download ppt "Activity Recognition Taiwoo Park May 7, 2013"

Similar presentations

Ads by Google