A Practical Approach to Recognizing Physical Activities Jonathan Lester Tanzeem Choudhury Gaetano Borriello

The Idea What am I doing?

Requirements, and intriguing questions Single point of location on the body and location independence – Does it matter? Work for any person Personalization only increases accuracy – How much variation across users? Cost Effective – How many sensors are needed?

Sensors & Locations - Single versus Multiple Majority of research with single sensor modality, at multiple locations Sensor placement and data collection becomes cumbersome Use of single sensor (single location) reduced accuracy by 35% [1] Compensate for accuracy lost using multiple sensor modalities [2] More comfortable for user to wear at single location Can integrate into existing mobile platforms [1] Bao, L., Intille, S.: Activity Recognition from User-Annotated Acceleration Data. In: Proc. Proc. Pervasive (2004) 1-17 [2] Choudhury, T., Lester, J., Kern, N., Borriello, G., Hannaford, B.. A Hybrid Discriminative Discriminative/Generative Approach for Modeling Human Activities.

Components Three main components Sensing Module  Gathers low level information about activities  microphone, accelerometer, light sensors, etc Feature Processing and Selection Module  Process raw data from sensors into features  Features discriminate activities Classification Module  Tags the activity being performed from predefined types

The Experimental System A Multi-Mode Sensor Board (MSB) Bluetooth Intel Mote (iMote) USB rechargeable battery board

Methodology Data collected across 12 individuals performing 8 activities 8 in mid twenties, 4 in their thirties 2/3 of data collected in a computer science building, 1/3 collected in office building Data collected from 3 MSBs located at shoulder strap, side of waist, right wrist Given a sequence of activities - sitting followed by climbing stairs followed by brushing teeth Data collected on laptop, annotated by an observer using iPaq

Feature Extraction About 18,000 samples per second How to bring out the important details – compute features Features – linear, log scale FFT frequency coefficients, spectral entropy, band pass filter coefficients, correlations, integrals, means, etc Not every feature for every sensor Total of 651 features computed Need to pick few important features to avoid confusing classification algorithm

Training the Classifiers Have all the data with the features extracted Need to separate the data into training and testing sections Data separated into 4 folds 3 folds used for training the classifier 1 fold used for testing Done with all combinations of 3 of 4 folds Temperature and humidity sensor data not used

Classifier Structure Modified version of AdaBoost [1] used to select best features and rank them based on classification performance Most discriminative sub-set of features selected to learn static classification of activities Usage of Hidden Markov model (HMM) as a second layer using the class probabilities Discriminative classifier tuned to make activities more distinguishable, while HMM ensures temporal smoothness [1] Viola, P., Jones, M.: Rapid Object Detection using a Boosted Cascade of Simple Features. In: Proc. Computer Vision and Pattern Recognition (2001)

Confusion Matrix – The Results Table: Confusion matrix for the static and HMM classifier trained using a single stream of sensor data from all three locations on the body

Location Sensitivity Sensors at different locations on body give better classification Want to make the sensors invisible to the user Ideally classification should work accurately with data from different locations on the body Allows user to carry at any convenient location How to determine which location is best to place the sensor board? Train four sets of classifiers – (1) data from all 3 locations (2) data from shoulder (3) data from waist (4) data from wrist

Location Sensitivity Results Precision = True Positive / (True Positive + False Positive) Recall = True Positive / (True Positive + False Negative) Overall accuracy = (True Positive + True Negative) / Total number of example Table: Overall precision/recall for the static and HMM classifiers trained/tested on all locations (top row) and a single location (bottom rows).

Variation across users Is there any need for training period for particular end user? User would want it to work immediately upon purchase Good if it gives better results with more usage but reasonable performance to begin with Train the classifier with lot of data collected from diverse group of people Use the classifier for new individual – No need to collect any training data, no need to retrain the classifier

User Variation – Test 1 Data collected from 12 users. Select N (N = 1 to 12) users’ data for training. Test on all 12 users’ data Idea – To show that accuracy improves with increase in training data

User Variation – Test 2 Data collected from 12 users. Select N (N = 1 to 12) users’ data for training. Test on remaining 12 – N users’ data Idea – To show improvement in accuracy due to general classifier and not increased training data size

Sensors Necessary Do not necessarily need all sensors on MSB to accurately classify Results shown here do not use temperature and humidity sensors Reducing number of sensors makes system less susceptible to environmental changes Increasing number of sensors increases complexity, power and computational requirements, thus higher costs Fewer sensors means smaller sizes, more practical to use 3 important sensors – accelerometer, audio and barometric pressure sensor

Single Versus Multiple Sensors Table: Classifiers trained using a single sensor. Overall accuracies approximately 65% Table: Classifiers trained using three sensors. Overall accuracies approximately 90%

Conclusion Accurate recognition of range of activities can be achieved Have answered the three questions  Single-board activity recognition system generalizes well. No need to learn location specific activity models  For the data set, no need for customization to specific individuals  Although 7 different modalities on board, but three main – audio, barometric pressure and accelerometer Participants were young healthy individuals How to handle activities that do not fall into predefined classes? How to handle ambiguities associated with compound activities?