1. Gary M. Weiss Comp & Info Science Dept Fordham University gweiss@cis.fordham.edu www.cis.fordham.edu/wisdm or wisdmproject.com

2.  Data Mining:  Extraction of knowledge from data via automated methods  Smart phone sensor mining:  Extraction of useful knowledge from the data generated by smart phone sensors 1/11/2012Gary M. Weiss ICCS 20122

3.  What sensors are found on smart phones?  Audio sensor (microphone)  Image sensor (camera, video recorder)  Tri-Axial Accelerometer  Location sensor (GPS, cell tower, WiFi)  Infrared proximity sensor; Light sensor  Magnetic compass; Temperature sensor; Touch sensor  Virtual/calculated sensors: ▪ Proximity (via light), gravity, orientation, gyroscope 1/11/2012Gary M. Weiss ICCS 20123

4.  Learning about smart phone users  Security requires understanding how devices used  Main focus of talk not on security but on what can be learned about smart phone users  Smart phone based biometric identification  Can be considered a security application  Many news stories about abuses  Apps to spy on your spouse; iPhone location fiasco 1/11/2012Gary M. Weiss ICCS 20124

5.  Activity recognition (what are you doing)?  Are you walking, jogging, sitting, standing, etc?  Biometric Identification (who are you)?  Are you John Smith?  Trait Identification (who are you at diff. level)?  Are you male? Are you tall? What do you weigh? 1/11/2012Gary M. Weiss ICCS 20125

6.  Data miners want to learn everything about you  Somehow that info will be useful  Develop useful apps, marketing leads, etc.  Many positive uses ▪ That is why NSF provided WISDM with funding for activity recognition from “Health and Well Being” program  But obviously issues with privacy and abuse 1/11/2012Gary M. Weiss ICCS 20126

7.  Approach to Predictive Data Mining 1. Collect labeled (sensor) training data 2. Apply data mining method to build predictive model 3. Apply predictive model to future unlabelled data 1/11/2012Gary M. Weiss ICCS 20127

8. 1/11/2012Gary M. Weiss ICCS 20128

9.  Why is it useful?  Context-sensitive applications ▪ Context influences handling of phone calls or music to play  Health applications ▪ Track activity levels or detect falls in elderly  Approaches to activity recognition  Uses multiple accelerometers  Use custom devices (pedometer, FitBit)  Our approach: use existing smart phones 1/11/2012Gary M. Weiss ICCS 20129

10.  Accelerometer data from Android phone  Walking  Jogging  Climbing Stairs  Lying Down  Sitting  Standing Gravity included 1/11/2012Gary M. Weiss ICCS 201210

11. 1/11/2012Gary M. Weiss ICCS 201211

12. 1/11/2012Gary M. Weiss ICCS 201212

13. 1/11/2012Gary M. Weiss ICCS 201213

14. 1/11/2012Gary M. Weiss ICCS 201214

15. 1/11/2012Gary M. Weiss ICCS 201215 Impersonal (Universal) Model Single Model trained and used for everyone Data Mining Method: Instance Based Learning (WEKA IB3) 72.4% Accuracy Predicted Class WalkingJoggingStairsSittingStanding Lying Down Actual Class Walking 220946789240 Jogging 451656148100 Stairs 41254869310 Sitting 1004755330241 Standing 805764483 Lying Down 51730113131

16. 1/11/2012Gary M. Weiss ICCS 201216 Personal Model: Model Build per User Data Mining Method: Instance Based Learning (WEKA IB3) 98.4% accuracy Predicted Class WalkingJoggingStairsSitting Standing Lying Down Actual Class Walking 30331240 00 Jogging 4178840 00 Stairs 42412921 00 Sitting 004870 26 Standing 50111 5090 Lying Down 40870442

17. 1/11/2012Gary M. Weiss ICCS 201217

18.  Identification based on physical/behavioral traits  Fingerprints, DNA, iris, gait, etc.  Biometrics for everyone  Equipment smaller & cheaper (sensors + processing) ▪ Laptops currently perform face recognition  Gait-based recognition  Most work is camera-based  Some applications  device security, customization & personalization 1/11/2012Gary M. Weiss ICCS 201218

19.  Used for identification and authentication  Identification means predicting identity from pool of users (36 in initial study and 200 in recent study)  Authentication is a binary class prediction ▪ Is it you or an imposter?  We evaluate walking and other activities as well as unclassified activities  Predictions made on individual 10 sec. samples but also combine “votes” to exploit larger samples 1/11/2012Gary M. Weiss ICCS 201219

20. UnclassifiedWalkJogUpDown J4872.284.083.065.861.0 Neural Net69.590.992.263.354.5 Straw Man4.34.25.06.54.7 1/11/2012Gary M. Weiss ICCS 201220 UnclassifiedWalkJogUpDown J4836/36 31/3231/3128/31 Neural Net36/36 32/3228.5/3125/31 Based on 10 second test samples Based on most frequent prediction for 5-10 minutes of data Recent unpublished results demonstrate 100% accuracy with 200 users! Authentication results even better (~90% with 10 sec samples)

21. 1/11/2012Gary M. Weiss ICCS 201221

22.  Soft biometrics: traits can aid with biometrics  As data miners we want to know everything about a person  Marketing applications: ads based on sex  Inferred weight to predict calories burned 1/11/2012Gary M. Weiss ICCS 201222

23.  Normally think about traits as being:  Unchanging: race, skin color, eye color, etc.  Slow changing: Height, weight, etc.  But want to know everything about a person:  What they wear, how they feel, if they are tired, etc.  Have never seen this goal for mobile sensor mining 1/11/2012Gary M. Weiss ICCS 201223

24.  Work in early stages  Data initially collected from ~70 people, now 200  Accelerometer and survey data  Survey data includes anything we could think of that might somehow be predictable ▪ Sex, height, weight, age, race, handedness, disability ▪ Shoe size, footwear type, size of heels, type of clothing ▪ # hours academic work, # hours exercise  Too few subjects investigate all factors ▪ Many were not predictable (maybe with more data) 1/11/2012Gary M. Weiss ICCS 201224

25. Accuracy 71.2% MaleFemale Male317 Female1216 1/11/2012Gary M. Weiss ICCS 201225 Accuracy 83.3% ShortTall Short155 Tall220 Accuracy 78.9% LightHeavy Light137 Heavy217 Results for IB3 classifier. For height and weight middle categories removed.

26. 1/11/2012Gary M. Weiss ICCS 201226

27.  Security policies vary widely by OS & platform  Symbian requires properly signed keys to remove restrictions on using certain APIs  iPhone apps have relatively strict oversight  Android OS has few restrictions and Marketplace has essentially no oversight or restrictions ▪ WISDM project has had no problem tapping into sensors and transmitting results. Just pay $25 for account. 1/11/2012Gary M. Weiss ICCS 201227

28.  Android notifies user of services  SYSTEM PERMISSIONS FOR WISDM SensorCollector ▪ Coarse location, fine location, internet access, keep from sleeping, modify/delete USB storage  Applications routinely access sensitive services  Fandango : fine GPS location, read phone state & identity, modify/delete USB storage, internet access  Angry Birds: identical permissions!  Notifications probably next to useless given this! 1/11/2012Gary M. Weiss ICCS 201228

29.  Even legitimate applications have to be concerned with privacy & security  WISDM will encrypt data in transit, encrypt on phone, include secure accounts & passwords, etc.  Need to ensure than any aggregated info is made public only if cannot be traced to individual 1/11/2012Gary M. Weiss ICCS 201229

30.  Good Policies:  Make it clear what you are monitoring and storing  Provide application level control for the user ▪ Allow user to turn on/off monitoring of specific sensors ▪ If they use an option to upload the information to Facebook then little privacy!  Since legitimate and illegitimate apps function alike, no easy way to distinguish them  Could try to use only certified apps, but quite limiting 1/11/2012Gary M. Weiss ICCS 201230

31.  WISDM is building & deploying the actitracker service to track your activities real-time and display them via a web-based interface  Useful health information and thus supported by NSF Grant & Google faculty research award  Actitracker.com online and should have basic functionality shortly 1/11/2012Gary M. Weiss ICCS 201231

32.  WISDM research group  Current Members ▪ Anthony Alcaro, Alex Armero, Shaun Gallagher, Andrew Grosner, Margo Flynn, Jeff Lockhart, Paul McHugh, Luigi Patruno, Tony Pulickal, Greg Rivas, Priscilla Twum, Bethany Wolff, Zach Wyhowanec, Jack Xue  Key Former Members ▪ Jennifer Kwapisz, Sam Moore, Shane Skowron, Alvan Wong  Funders: NSF, Google, and Fordham 1/11/2012Gary M. Weiss ICCS 201232

33. 1. J.R. Kwapisz, G.M. Weiss, and S.A. Moore. 2010. Activity recognition using cell phone accelerometers, in Proceedings of the Fourth International Workshop on Knowledge Discovery from Sensor Data, 10-18. 2. J. R. Kwapisz, G.M. Weiss, and S.A. Moore, 2010. Cell phone-based biometric identification, in Proceedings of the IEEE Fourth International Conference on Biometrics: Theory, Applications and Systems. 3. J.W. Lockhart, G.M. Weiss, J.C. Xue, S.T. Gallagher, A.B. Grosner, T.T. Pulickal. 2011. Design considerations for the WISDM smart phone-based sensor mining architecture, in Proceedings of the Fifth International Workshop on Knowledge Discovery from Sensor Data, San Diego, CA. 4. G.M. Weiss, and J.W. Lockhart, 2011. Identifying user traits by mining smart phone accelerometer data, in Proceedings of the 5 th International Workshop on Knowledge Discovery from Sensor Data., San Diego, CA. 1/11/2012Gary M. Weiss ICCS 201233

34. For more information go to wisdmproject.comGary Weiss gweiss@cis.fordham.edu 1/11/2012Gary M. Weiss ICCS 201234