July 25, 2010 SensorKDD Activity Recognition Using Cell Phone Accelerometers Jennifer Kwapisz, Gary Weiss, Samuel Moore Department of Computer & Info. Science Fordham University

July 25, SensorKDD 2010 We are Interested in WISDM WISDM: WIreless Sensor Data Mining WISDM: WIreless Sensor Data Mining Powerful portable wireless devices are becoming common and are filled with sensors Powerful portable wireless devices are becoming common and are filled with sensors Smart phones: Android phones, iPhone Smart phones: Android phones, iPhone Music players: iPod Touch Music players: iPod Touch Sensors on smart phones include: Sensors on smart phones include: Microphone, camera, light sensor, proximity sensor, temperature sensor, GPS, compass, accelerometer Microphone, camera, light sensor, proximity sensor, temperature sensor, GPS, compass, accelerometer

July 25, SensorKDD 2010 Accelerometer-Based Activity Recognition The Problem: use accelerometer data to determine a user’s activity The Problem: use accelerometer data to determine a user’s activity Activities include: Activities include: Walking and jogging Walking and jogging Sitting and standing Sitting and standing Ascending and descending stairs Ascending and descending stairs More activities to be added in future work More activities to be added in future work

July 25, SensorKDD 2010 Applications of Activity Recognition Health Applications Health Applications Generate activity profile to monitor overall type and quantity of activity Generate activity profile to monitor overall type and quantity of activity Parents can use it to monitor their children Parents can use it to monitor their children Can be used to monitor the elderly Can be used to monitor the elderly Make the device context-sensitive Make the device context-sensitive Cell phone sends all calls to voice mail when jogging Cell phone sends all calls to voice mail when jogging Adjust music based on the activity Adjust music based on the activity Broadcast (Facebook) your every activity Broadcast (Facebook) your every activity

July 25, SensorKDD 2010 Our WISDM Platform Platform based on Android cell phones Platform based on Android cell phones Android is Google’s open source mobile computing OS Android is Google’s open source mobile computing OS Easy to program, free, will have a large market share Easy to program, free, will have a large market share Unlike most other work on activity recognition: Unlike most other work on activity recognition: No specialized equipment No specialized equipment Single device naturally placed on body (in pocket) Single device naturally placed on body (in pocket)

July 25, SensorKDD 2010 Our WISDM Platform Current research was conducted off-line Current research was conducted off-line Data was collected and later analyzed off-line Data was collected and later analyzed off-line In future our platform will operate in real-time In future our platform will operate in real-time In June we released real-time sensor data collection app to Android marketplace In June we released real-time sensor data collection app to Android marketplace Currently collects accelerometer and GPS data Currently collects accelerometer and GPS data

July 25, SensorKDD 2010 Accelerometers Included in most smart phones & other devices Included in most smart phones & other devices All Android phones, iPhones, iPod Touches, etc. All Android phones, iPhones, iPod Touches, etc. Tri-axial accelerometers that measure 3 dimensions Tri-axial accelerometers that measure 3 dimensions Initially included for screen rotation and advanced game play Initially included for screen rotation and advanced game play

July 25, SensorKDD 2010 Examples of Raw Data Next few slides show data for one user over a few seconds for various activities Next few slides show data for one user over a few seconds for various activities Cell phone is in user’s pocket Cell phone is in user’s pocket Earth’s gravity is registered as acceleration Earth’s gravity is registered as acceleration Acceleration values relative to axes of the device, not Earth Acceleration values relative to axes of the device, not Earth In theory we can correct this given that we can determine orientation of the device In theory we can correct this given that we can determine orientation of the device

July 25, SensorKDD 2010 Standing

July 25, SensorKDD 2010 Sitting

July 25, SensorKDD 2010 Walking

July 25, SensorKDD 2010 Jogging

July 25, SensorKDD 2010 Descending Stairs

July 25, SensorKDD 2010 Ascending Stairs

July 25, SensorKDD 2010 Data Collection Procedure User’s move through a specific course User’s move through a specific course Perform various activities for specific times Perform various activities for specific times Data collected using Android phones Data collected using Android phones Activities labeled using our Android app Activities labeled using our Android app Data collection procedure approved by Fordham Institutional Review Board (IRB) Data collection procedure approved by Fordham Institutional Review Board (IRB) Collected data from 29 users Collected data from 29 users

July 25, SensorKDD 2010 Data Preprocessing Need to convert time series data into examples Need to convert time series data into examples Use a 10 second example duration (i.e., window) Use a 10 second example duration (i.e., window) 3 acceleration values every 50 ms (600 total values) 3 acceleration values every 50 ms (600 total values) Generate 43 total features Generate 43 total features Ave. acceleration each axis (3) Ave. acceleration each axis (3) Standard deviation each axis (3) Standard deviation each axis (3) Binned/histogram distribution for each axis (30) Binned/histogram distribution for each axis (30) Time between peaks (3) Time between peaks (3) Ave. resultant acceleration (1) Ave. resultant acceleration (1)

July 25, SensorKDD 2010 Final Data Set

July 25, SensorKDD 2010 Data Mining Step Utilized three WEKA learning methods Utilized three WEKA learning methods Decision Tree (J48) Decision Tree (J48) Logistic Regression Logistic Regression Neural Network Neural Network Results reported using 10-fold cross validation Results reported using 10-fold cross validation

July 25, SensorKDD 2010 Summary Results

July 25, SensorKDD 2010 J48 Confusion Matrix Predicted Class WalkJogUpDownSitStand ActualClassActualClass Walk Jog Up Down Sit Stand

July 25, SensorKDD 2010 Conclusions Able to identify activities with good accuracy Able to identify activities with good accuracy Hard to differentiate between ascending and descending stairs. To limited degree also looks like walking. Hard to differentiate between ascending and descending stairs. To limited degree also looks like walking. Can accomplish this with a cell phone placed naturally in pocket Can accomplish this with a cell phone placed naturally in pocket Accomplished with simple features and standard data mining methods Accomplished with simple features and standard data mining methods

July 25, SensorKDD 2010 Related Work At least a dozen papers on activity recognition using multiple sensors, mainly accelerometers At least a dozen papers on activity recognition using multiple sensors, mainly accelerometers Typically studies only users Typically studies only users Activity recognition also done via computer vision Activity recognition also done via computer vision Actigraphy uses devices to study movement Actigraphy uses devices to study movement Used by psychologists to study sleep disorders, ADD Used by psychologists to study sleep disorders, ADD A few recent efforts use cell phones A few recent efforts use cell phones Yang (2009) used Nokia N95 and 4 users Yang (2009) used Nokia N95 and 4 users Brezmes (2009) used Nokia N95 with real-time recognition Brezmes (2009) used Nokia N95 with real-time recognition One model per user (requires labeled data from each user) One model per user (requires labeled data from each user)

July 25, SensorKDD 2010 Future Work Add more activities and users Add more activities and users Add more sophisticated features Add more sophisticated features Try time-series based learning methods Try time-series based learning methods Generate results in real time Generate results in real time Deploy higher level applications: activity profiler Deploy higher level applications: activity profiler

July 25, SensorKDD 2010 Other WISDM Research Cell Phone-Based Biometric identification 1 Cell Phone-Based Biometric identification 1 Same accelerometer data and same generated features but added 7 users (36 in total) Same accelerometer data and same generated features but added 7 users (36 in total) If we group all of the test examples from one cell phone and apply majority voting, achieve 100% accuracy If we group all of the test examples from one cell phone and apply majority voting, achieve 100% accuracy Can be used for security or automatic personalization Can be used for security or automatic personalization Interested in GPS spatio-temporal data mining Interested in GPS spatio-temporal data mining 1 Kwapisz, Weiss, and Moore, Cell-Phone Based Biometric Identification, Proceedings of the IEEE 4 th International Conference on Biometrics: Theory, Applications, and Systems (BTAS-10), September 2010.

July 25, 2010 SensorKDD Thank You Questions?