1. Activity, Audio, Indoor/Outdoor classification using cell phones Hong Lu, Xiao Zheng Emiliano Miluzzo, Nicholas Lane CS 185 Final Project presentation

2. Outline Motivation Results Conclusion Future work Demo

3. Motivation In our lab we are working on a project where we use sensors to determine the sensing presence of people, i.e.: –Activity –Audio analysis –Location (Indoor/Outdoor)‏ We are interested in using off-the-shelf devices like cell phones (with the camera, microphone, and accelerometer)‏

4. Requirements The activity and audio inference must be done on the phone (to improve scalability and limit the amount of bytes sent over the air)‏ Downside: limited hardware (computation resources) and software (programmability) capabilities

5. Let’s start with sound

6. Experimental methodology Over a period of one week we collected 250 audio samples of conversations and 250 audio samples w/out conversation 8 (>) people involved in different conversation settings The phone used to collect the samples was placed at different places (inside/outside pocket) and at different distances from the audio source The samples w/out conversation are taken indoors (office environment), outdoor in a quite location, and outdoor on the sidewalk (to catch the sound of cars, etc.)‏

7. Observing the FFT Conversation No Conversation

8. Sound feature vector We focus on the 250 – 750 Hz portion of the spectrum We focus on the –Mean: because we see that the power differs in the 2 cases in this portion of the spectrum –Std Dev: we also see that the power oscillates more in the conversation case than in the no conversation case

9. Mean and Std dev

10. Mean and Std dev distribution

12. Total Error

13. ROC curves

14. Misclassification = 31%Misclassification = 28.4% K-meansDiscriminant analysis

15. Now location…

16. Problem Detecting when the cell phone is indoor or outdoors. Use the limited sensing capabilities of the phone. Why try to do this? 1.Health Sensing Simple less accurate UV and pollution exposure with value in the potential for wide spread sampling 2.Context-dependant sensing queries Indoor/outdoor classification is one useful criteria of describing conditions when (and when not) sense. 3.It seemed interesting to try

17. Intuition of an approach Sensor values will exhibit different characteristics depending on if the phone is indoors or outdoors. A key problem == Limited Sensors Available: –Different sensors would have made this simpler –Available: WiFi, GPS, Sound, Camera, GPRS etc. We tried feature vectors formed by using: –GPS + Radio Patterns (from WiFi) Various features tried that were based on: –WiFi AP Counting (assumption: lower density outdoors vs indoors) –Sum of WiFi AP Neighborhood RSS (assumption: lower signal strengths seen outdoors) –GPS Satellite Signal Lock (assumption: signal lock, which is effected by LOS issues, will occur less often indoors than outside)

18. Hope from weak discriminators? KEY: GREEN == OUTDOORSBLUE == INDOORS Data Set: 1 user, 2,000 data points collected; 1 sample per minute. Sampling WiFi and GPS based measurements.

19. Groping for a model  Technique Used: Logistic regression  Tested many variants of the inputs to find effective feature vectors.  Windowed Variables – [RSS Sum, AP Count, GPS Flag]  Variance of Windowed Variables – [RSS Sum, AP Count, GPS Flag]  Multiple variants of model inputs failed to provide sufficient discriminative power.  ANOVA test showed little reduction in deviance.  We favored a simpler model over the complex one.  Checked that normal distribution existed in feature vectors

20. The Model classifier_label_param ~ ap_count + gps + rss_sum (Intercept) ap_count gps rss_sum -0.23107255 2.49738687 0.46060383 -0.03293664 Significance of the co-efficients? (Are they really different from zero?) ap_count < 2e-16 *** gps 0.00358 ** rss_sum < 2e-16 ***

21. Kicking the tires of the model

22. Empirical performance  dubious hypothesis  results not impressive but interesting  focused on other parts. Mini-conclusion OutdoorIndoor Outdoor163694 Indoor259468 Accuracy 85.6%

23. Now activity…

24. Data collection 3-axis accelerometer embedded in the Nokia Sport Phone (in the future many phones will have an accelerometer)‏ Sampling rate = 37Hz The phone can be clipped on the belt or carried inside the pocket Data labeled by the user

25. The raw data X-axis – BLUE Y-axis – GREEN Z-axis – RED From the 3 axis of the accelerometer

26. Feature Extraction Simply the mean and standard deviation of each axis Window size = 128 data points Lightweight for running on the phone

27. PCA Use centerized PCA to get principal component projection matrix: Each column contains the coefficients for one principal component. The columns are sorted by decreasing component variance. -0.0150 -0.0862 0.0762 -0.0316 -0.0801 0.9895 -0.0086 0.3733 0.4937 -0.7741 0.1314 -0.0197 0.0898 0.6502 -0.7286 -0.1438 0.0665 0.1149 -0.8425 -0.2966 -0.3070 -0.3274 0.0082 -0.0248 -0.4889 0.5694 0.3288 0.4243 -0.3854 -0.0008 -0.2067 0.1352 0.1310 0.3031 0.9074 0.0817

28. The 2 dimensional example Use the first 2 components, leave out the others.

29. Conclusion We wanted to do more than we could do Collecting the data took long time Programming the phones can be challenging The phone’s hardware and software platform limit the applicability of off-the- shelf data mining techniques

30. Now the demo!!!

31. Future work Increase the number of activities we can detect (including sitting, cycling, climbing stairs, etc)‏ Augment the audio classifier to be able to do speaker recognition, party detection, type of noise (car, wind, etc)‏ Use the cell phone’s camera and light detector to make the outdoor/indoor classifier more robust

32. Thank you!