Presentation on theme: "Activity Recognition Taiwoo Park May 7, 2013"— Presentation transcript:
1 Activity Recognition Taiwoo Park May 7, 2013 Bao, Ling, and Stephen S. Intille. "Activity recognition from user-annotated acceleration data." Pervasive Computing. Springer Berlin Heidelberg,Park, Taiwoo, et al. "E-gesture: a collaborative architecture for energy-efficient gesture recognition with hand-worn sensor and mobile devices."Proceedings of the 9th ACM Conference on Embedded Networked Sensor Systems. ACM, 2011.Some slides are from CSCI 546 course materials by James Reinebold, USC
2 Activity? Higher level activities Lower level activities Giving a lecture, having a breakfast, playing soccer…Lower level activitiesLying on a bed, standing still, running, walking, …
3 An easy example Assumptions: How can we recognize activities? Sensors on smartphones are only availableAccelerometer, compass, gyroscope, light, …You can attach smartphones on your bodyOnly three target activities to recognizeRunningStanding stillLying on a bedHow can we recognize activities?
4 An easy example (cont’d) Is the phone being shaken?Phone orientationActivityNoUprightYesLying downStanding stillRunningyLying on a bedNothingzxAverage value of accelerometer y-axis sensor signals for the last 3 secondsVariance of accelerometer sensor signal for the last 3 seconds
5 Activity recognition pipeline Bader, Sebastian, and Thomas Kirste. "A Tutorial Introduction to Automated Activity and Intention Recognition." (2011).
6 An easy example (revisited) Is the phone being shaken?Phone orientationActivityNoUprightStanding stillYesRunningLying downLying on a bed…?ClassificationAverage value of accelerometer y-axis sensor signals for the last 3 secondsFeature extractionWindowingVariance of accelerometer sensor signal for the last 3 seconds
7 Data collection Semi-Naturalistic, User-Driven Data Collection Obstacle course / worksheetNo researcher supervision while subjects performed the tasksTimer synchronizationDiscard data within 10 seconds of start and finish time for activities12:30 – 12:50Walking…Bao, Ling, and Stephen S. Intille. "Activity recognition from user-annotated acceleration data." Pervasive Computing. Springer Berlin Heidelberg,
8 Activities Walking Walking Carrying Items Sitting and Relaxing Working on ComputerStanding StillEating or DrinkingWatching TVReadingRunningBicyclingStretchingStrength-trainingScrubbingVacuumingFolding LaundryLying down & relaxingBrushing TeethClimbing stairsRiding ElevatorRiding escalator
10 Sensors UsedFive ADXL210E accelerometers (manufactured by Analog Devices)Range of +/- 10g5mm x 5mm x 2mmLow Power, Low CostMeasures both static and dynamic accelerationSensor data was stored in a memory card by using “Hoarder Board”Source:
12 Activity recognition pipeline Bader, Sebastian, and Thomas Kirste. "A Tutorial Introduction to Automated Activity and Intention Recognition." (2011).
13 ClassificationQuestion: What is the most similar samples to the current one?Data sample?‘Running’‘Standing still’‘Lying on a bed’‘Walking’Collected data samples (in advance)Methods: Naïve bayes, nearest neighbor, decision table/tree, HMM (Hidden Markov Models), …
14 Is the phone being shaken? Decision TableIs the phone being shaken?Phone orientationActivityNoUprightYesLying downStanding stillRunningyLying on a bedNothingzx
15 Decision TreesMake a tree where the non-leaf nodes are the features, and each leaf node is a classification. Each edge of the tree represents a value range of the feature.Move through the tree until you arrive at a leaf nodeGenerally, the smaller the tree the better.Finding the smallest is NP-HardSource:
16 Not lying down: Is the phone being shaken? Decision Tree ExamplePhone Orientation?Lying downUprightLying on a bedNot lying down: Is the phone being shaken?YesNoRunningStanding still
17 Nearest NeighborSplit up the domain into various dimensions, with each dimension corresponding to a feature.Classify an unknown point by having its K nearest neighbors “vote” on who it belongs to.Simple, easy to implement algorithm. Does not work well when there are no clusters.Source:
18 Nearest Neighbor Example RunningVariance of accelLying on a bedStanding stillAverage value of accel Y
19 Naïve Bayes Classifier Multiplies the probability of an observed datapoint by looking at the priority probabilities that encompass the training set.P(B|A) = P(A|B) * P(B) / P(A)Assumes that each of the features are independent.Relatively fast.Stochastic methodSource: cis.poly.edu/~mleung/FRE7851/f07/naiveBayesianClassifier.pdf
20 Activity recognition pipeline Bader, Sebastian, and Thomas Kirste. "A Tutorial Introduction to Automated Activity and Intention Recognition." (2011).
21 Feature Extraction Time-domain features [Maurer 2006] Mean (average), Root Mean Square, Variance, …FFT-based feature computation [Bao 2004]Sample at Hz512 sample windows (about 6.71 sec)Extract mean energy, entropy, and correlation featuresMaurer, Uwe, et al. "Activity recognition and monitoring using multiple sensors on different body positions." Wearable and Implantable Body Sensor Networks, BSN International Workshop on. IEEE, 2006.
26 Trying With Less Sensors Accelerometer (s) Left InDifference in Recognition ActivityHip/Wrist/Arm/Ankle/Thigh/Thigh and Wrist/Hip and Wrist/With only two accelerometers we can get good performance
27 LessonsAccelerometers can be used to affectively distinguish between everyday activities.Decision trees and nearest neighbor algorithms are good choices for activity recognition.Some sensor locations are more important than others.Selecting a feature set is important to increase recognition accuracy.
28 E-Gesture: A Collaborative Architecture for Energy-efficient Gesture Recognition with Hand-worn Sensor and Mobile DevicesI am really happy to present this work, and thank you for being here.I am Taiwoo Park and the title of the work I am going to present is E-Gesture. It is an energy-efficient gesture recognition architecture which utilizes hand-worn sensor and mobile devices in a collaborative manner.This work was done in collaboration with my collagues Jinwon, Inseok and Chungkuk in KAIST, Lama Nachman in Intel Research, and my advisor Junehwa Song.I’ll first show you a demo video showing the goal of this work.
29 Mobile Gesture Interaction Framework MotivationMobile applications using hand gestures…Mobile Gesture Interaction FrameworkWristwatch-typemotion sensor(Accelerometer,Gyroscope)SmartphoneWe envision that these kind of mobile applications utilizing hand gesture interaction will be frequently used in the near future. <Click> To facilitate the development of these applications, we designed and implemented a framework supporting hand gesture interaction considering users’ mobility. The framework uses wristwatch-type motion sensor and smartphone, because these devices are quite acceptable even in mobile situation.Mobility!!!!!
30 Challenge: Energy and Accuracy • Conventional gesture processing pipeline (for gesture recognition in stationary setting)SensorMobile DeviceData SensingGesture Segmentation(Button, Algorithms)Classification(HMM, DTW)Continuous Raw DataGestureSamples(Candidate)ResultAccelGyroGesture ‘A’ or ‘B’ or non-gestureThe major design challenges of the mobile gesture interaction framework are energy and accuracy. To help understanding, I’ll show you a figure, showing the conventional gesture processing pipeline for stationary gesture recognition. <Click> First, the sensor continuously takes hand motion sensor data, and transmit the data to mobile device. <Click> In mobile device, the segmentation module detects hand movements which could be a intended user gestures. <Click> The detected hand movement is sent to the classification stage, in a form of time sequence of motion signals, and classified to one of the predefined gestures or non-gesture garbage by machine learning algorithms such as Hidden Markov Model or Dynamic Time Warping or so. Again, the conventional gesture processing architecture has three stages, sensing, segmentation, and classification.
31 Challenge: Energy and Accuracy Sensor: 20hrs (250mAh)Smartphone: 24hrs 17hrsSensorMobile DeviceData SensingGesture Segmentation(Button, Algorithms)Classification(HMM, DTW)Continuous Raw DataGestureSamples(Candidate)ResultAccelContinuous data transmissionEnergyAccuracyMobility noisesAccuracyMobility noisesEnergyEnergy-hungryGyroscope(56%)GyroThis processing pipeline gives several challenges in mobile use. <Click> First, energy consumption is quite high, because the sensor and the mobile device continuously communicate to send and receive hand motion data. <Click> Also, energy-hungry gyroscope should be always turned on, and it consumes significant amount of energy in the sensor device, more than 50% of total sensor energy consumption. As a result, <Click> we observed that a battery lifetime of a sensor device is 20 hours with 250 mAh battery. Also, the gesture processing shortened smartphones’ lifetime from 24 hours to 17 hours. <Pause and Click> Second, this system fails to detect and classify hand gestures with mobility noises. Users’ mobility such as running and walking produces body noises, which makes it hard to accurately recognize hand gestures. As a result, <click> In gesture segmentation, it shows over 90% of false positive errors in segmentation, that is, falsely detected hand motions. Also, gesture classification shows only 70% of accuracy in mobile settings. Note that conventional gesture processing pipeline does not address mobile use. We designed E-Gesture architecture to address these two challenges, by redesigning this pipeline.Mobility NoisesOver 90% False segmentationOnly 70% Classification
32 E-Gesture Architecture 1. Device-wise collaborationDetection on wristwatch, classification on smartphoneCollaborative Gesture Sensing and SegmentationClassification(Adaptive and Multi-Situation HMM)Gesture Samples(Candidate)Result(e.g. laydown)AccelGyroTriggerAdaptationWristwatch Sensor DeviceMobile Device2. Sensor-wise collaborationAccel turns on gyro for energy efficiencyGyro adapts accel’s sensitivity for mobility changesAccelerometer: (+)Energy-efficient, (-)Mobility-vulnerableGyroscope: (-)Energy-hungry, (+)Mobility-robustThis is an overall architecture of e-gesture. I showed you the conventional gesture processing architecture with three stages, sensing, segmentation, and classification. We carefully redesigned and distributed the stages for our framework, considering energy efficiency and mobility noises. The key design concept is two kind of collaboration. <Click> One is the collaboration between sensor and mobile device, that is, we moved gesture segmentation into sensor device. By this, we could selectively transmit gesture candidate samples only, and reduced energy consumption from continuous data transmission. <Click> The another one is the collaboration between two types of motion sensors, accelerometer and gyroscope. We found that accelerometer and gyroscope have very interesting characteristics in terms of energy and mobility robustness. That is, accelerometer consumes small amount of energy, but it is vulnerable to mobility noise. However, Gyroscope is energy-hungry but robust to mobility. We utilized these complementary characteristics. In brief, accelerometer monitors hand motion signal, and turns on gyroscope when it detects significant hand motion. For this we used simple threshold-based detection algorithm. Then, gyroscope immediately wake up and validates motion signal whether the user is really performing significant hand motion. We also designed feedback adaptation, that is, if the accelerometer turns on gyroscope too frequently because of mobility noise, gyroscope controls the sensitivity of accelerometer by feedback adaptation loop.
33 Sensor-side Energy Savings 46mWContinuous sensing + transmission:20 hrs39mW (↓15%)Device-wise collaboration(reduced transmission)23.7 hrs (1.2x)250mAh Li-ion BatteryDevice-wise, Sensor-wise collaboration(gyroscope power control,reduced transmission)48.7 hrs (2.4x)19mW (↓59%)We evaluated the energy efficiency of our architecture on sensor-side. We used a customly-designed sensor mote which is compatible with MicaZ sensor mote, and utilized 250mAh Li-ion battery for operation. The detailed evaluation settings can be found on our SenSys paper. <Click> First, continuous sensing and transmission of accelerometer and gyroscope samples requires all components in sensors to be turned on, so consumes maximum energy. In our setting, the sensor runs up to 20 hours. However, <Click> if it transmits detected gesture samples, we can save energy consumption for radio transmission and the lifetime gets 1.2 times longer. Finally, <Click> if we selectively turn on gyroscope by sensor-wise collaboration, it reduces 59% of energy consumption and shows 2.4 times longer battery lifetime, more than 48 hours.Energy Consumption59% less energy consumption, 2.4x longer lifetime
34 Mobile-side Energy Savings 122mWAll processing on mobile:42.1hrsNexus One1400mAh Li-ion Battery3G/WiFi on70mW(↓43%)Device-wise collaboration(reduced transmission)74hrs (1.8x)Next, we evaluated mobile device-side energy savings. We used nexus one with 14 hundred mAh battery. If we always turn-on radio transmission component in mobile device, the device lasts only 42 hours. However, through our device-wise collaboration, we could turn-off the radio component while user does not perform hand gestures. As a result, we could reduce 43% of energy consumption, and increase the battery lifetime to 74 hours.Energy Consumption
35 Implementation Sensor node Smartphones Atmega128L MCU Bluetooth, ZigBeeSensors3-Axis Accelerometer (ADXL335)3-Axis Gyroscope (3 XV-3500CB)40Hz SensingVib motorSmartphonesNokia N96, Google Nexus OneBluetooth RadioWe implemented our framework on a sensor node containing atmega128L, CC2420 for radio, and two types of motion sensors. Also we used two types of smartphones, Nokia N96 and Google Nexus One. As the smartphones do not have zigbee-compatible radio, so we used a bridge node to convert zigbee signals to bluetooth signal. It receives gesture data from the sensor via zigbee, and forward the data to the smartphones via bluetooth. <pause>For your information, we used nexus one for evaluation.
36 Sample Applications Swan Boat [Ubicomp09][MM09][ACE09] Collaborative boat-racing exertion gameUtilizes hand gestures as additional game inputPunching together, flapping togetherMobile Music Player, Phone Call ManagerFeaturing eye-free, touch-free controlsUser can control the application by hand gesturesUsing E-Gesture, we have made several sample applications. One is Swan Boat, a collaborative boat-racing exertion game. Some of you may have seen the demonstration last Tuesday. Two runners steer one boat together by the difference of running speeds. For details on game play, you can easily find the papers and videos describing this game and it’s really fun. To enhance the game interaction, we tried to utilize hand gestures as additional game input, and E-Gesture greatly helped the gesture processing in the game. <pause>I showed the music player and the phone call manager. So I’ll pass the detail.
37 Conclusion Mobile gestural interaction platform Collaborative gesture processing1.8x longer battery lifetime for Smartphone2.4x longer battery lifetime for Hand-worn Sensor + preserving gyroscope’s detection performanceMobility-robust gesture classification using HMMUp to 94.6% of classification accuracy on mobile usage by mobility-considered classification architecture designIt will greatly facilitate gesture-based mobile applicationsProvided a novel sensor fusion schemeSerial fusion + feedback controlSaves energy + preserves detection accuracyI’ll conclude by talking about the key points. First, We designed and implemented a mobile gesture interaction platform to support applications using hand gestures. We proposed collaborative gesture processing schemes, and it achieved 2.4 times and 1.8 times longer battery lifetime on the sensor and the smartphone, while preserving gyroscope’s better performance of gesture detection. Also we achieved up to about 95% of classification accuracy even in the mobile usage. The time is somewhat limited so I passed the classification detail, and you can easily find them in the paper. Please note that we provided a novel sensor fusion scheme. In our work, we utilized accelerometer and gyroscope which have complementary characteristics on energy consumption and detection performance. By using serial fusion and feedback control, we saved much energy while preserving detection accuracy as well. We hope that there would be another way to utilize this kind of sensor fusion. <pause> That’s it. Thank you.