1. Introduction to Mobile Sensing with Smartphones Uichin Lee April 22, 2013

2. iPhone 4 - Sensors

3. Applications Transportation – Traffic conditions (MIT VTrack, Nokia/Berkeley Mobile Millennium) Social Networking – Sensing presence (Dartmouth CenceMe) Environmental Monitoring – Measuring pollution (UCLA PIER) Health and Well Being – Promoting personal fitness (UbiFit Garden)

4. Citysense MacroSense CabSense

5. Eco-system Players Multiple vendors – Apple AppStore – Google Play (Android Market) – Microsoft Mobile Marketplace Developers – Startups – Academia – Small Research laboratories – Individuals Critical mass of users

6. Scale of Mobile Sensing

7. Sensing Paradigm Participatory: active sensor data collection by users – Example: managing garbage cans by taking photos – Advantages: supports complex operations – Challenges: Quality of data is dependent on participants Opportunistic: automated sensor data collection – Example: collecting location traces from users – Advantages: lowers burden placed on the user – Challenges: Technically hard to build – people underutilized Phone context problem (dynamic environments)

8. SENSE LEARN INFORM, SHARE AND PERSUASION Mobile Sensing Architecture Mobile Computing Cloud

9. Sense Programmability – Managing smartphone sensors with system APIs – Challenges: fine-grained control of sensors, portability Continuous sensing – Resource demanding (e.g., computation, battery) – Energy efficient algorithms – Trade-off between accuracy and energy consumption Phone context – Dynamic environments affect sensor data quality – Some solutions: Collaborative multi-phone inference Admission controls for removing noisy data SENSE LEARN INFORM, SHARE, PERSUASION

10. Learn Integrating sensor data – Data mining and statistical analysis Learning algorithms – Supervised: data are hand-labeled (e.g., cooking, driving) – Semi-supervised: some of the data are labeled – Unsupervised: none of the data are labeled Human behavior and context modeling Activity classification Mobility pattern analysis (place logging) Noise mapping in urban environments SENSE LEARN INFORM, SHARE, PERSUASION

11. Learn: Scaling Models Scaling model to everyday uses – Dynamic environments; personal differences – Large scale deployment (e.g., millions of people) Models must be adaptive and incorporate people into the process If possible, exploit social networks (community guided learning) to improve data classification and solutions Challenges: – Lack of common machine learning toolkits for smartphones – Lack of large-scale public data sets – Lack of public repository for sharing data sets, code, and tools SENSE LEARN INFORM, SHARE, PERSUASION

12. Inform, Share, Persuasion Sharing – Data visualization, community awareness, and social networks Personalized services – Profile user preferences, recommendations, persuasion Persuasive technology – systems that provide tailored feedback with the goal of changing user’s behavior – Motivation to change human behavior (e.g., healthcare, environmental awareness) – Methods: games, competitions, goal setting – Interdisciplinary research combining behavioral and social psychology with computer science SENSE LEARN INFORM, SHARE, PERSUASION

13. Privacy Issues Respecting the privacy of the user is the most fundamental responsibility of a mobile sensing system Current solutions – Cryptography – Privacy-preserving data mining – Processing data locally versus cloud services – Group sensing applications is based on user membership and/or trust relationships

14. Privacy Issues Reconstruction type attacks – Reverse engineering collected data to obtain invasive information Second-hand smoke problem – How can the privacy of third parties be effectively protected when other people wearing sensors are nearby? – How can mismatched privacy policies be managed when two different people are close enough to each other for their sensors to collect information? Stronger techniques for protecting people’s privacy are needed

15. Understanding Smartphone Sensors: accelerometer, compass, gyroscope, location, etc

16. Smart Phone/Pad Sensors Galaxy Nexus iPhone4iPhone5 Samsung Galaxy S3 Samsung Galaxy S4 Galaxy Tab/ iPad2 Accelerometer OOOOOO Magnetometer (Compass) OOOOOO Gyroscope OOOOOO Light OOOOOO Proximity OOOOOO Camera OOOOOO Voice OOOOOO Pressure (Barometer) OOO Humidity/IR/Temperature O

17. Smartphone Sensors Accelerometer Magnetometer (digital compass) Gyroscope Light Pressure (Barometer) Proximity Camera Voice Temperature/Humidity/IR Gesture (Galaxy S4)

18. Android APIs Package: android.hardware Classes: – SensorManager – android service – Sensor – specific sensor – SensorEvent – specific event of the sensor = data

19. SensorManager Most sensors interfaced through SensorManager or LocationManager – Obtain pointer to android service using Context.getSystemService(name) – For name, use constant defined by Context class SENSOR_SERVICE for SensorManager LOCATION_SERVICE for LocationManager Check for available sensors using List getSensorList(int type) – Type constants provided in Sensor class documentation

20. SensorManager Use getDefaultSensor(int type) to get a pointer to the default sensor for a particular type Sensor accel = sensorManager.getDefaultSensor( Sensor.TYPE_ACCELEROMETER); Register for updates of sensor values using registerListener(SensorEventListener, Sensor, rate) – Rate is an int, using one of the following 4 constants SENSOR_DELAY_NORMAL (delay: 200ms) SENSOR_DELAY_UI (delay: 60ms) SENSOR_DELAY_GAME (delay: 20ms) SENSOR_DELAY_FASTEST (delay: 0ms) – Use the lowest rate necessary to reduce power usage Registration will power up sensor: mSensorService.enableSensor(l, name, handle, delay);

21. SensorManager Unregister for sensor events using unregisterListener(SensorEventListener, Sensor) or unregisterListener(SensorEventListener) Undregistering will power down sensors: mSensorService.enableSensor(l, name, handle, SENSOR_DISABLE) Perform register in OnResume() and unregister in OnPause() to prevent using resources while your activity is not visible SensorListener is deprecated, use SensorEventListener instead – See documentation for Sensor, SensorManager, SensorEvent and SensorEventListener

22. SensorEventListener Must implement two methods onAccuracyChanged(Sensor sensor, int accuracy) onSensorChanged(SensorEvent event) SensorEvent – int accuracy – Sensor sensor – long timestamp Time in nanoseconds at which event happened – float[] values Length and content of values depends on sensor type

23. API – Setup public class MainActivity extends Activity implements SensorEventListener {.. private SensorManager sm = null; … public void onCreate (Bundle savedInstanceState) {.. sm = (SensorManager) getSystemService(SENSOR_SERVICE); } protected void onResume () {.. List typedSensors = sm.getSensorList(Sensor.TYPE_LIGHT); // also: TYPE_ALL if (typedSensors == null || typedSensors.size() <= 0) … error… sm.registerListener(this, typedSensors.get(0), SensorManager.SENSOR_DELAY_GAME); // Rates: SENSOR_DELAY_FASTEST, SENSOR_DELAY_GAME, // SENSOR_DELAY_NORMAL, SENSOR_DELAY_UI }

24. API – Processing Events It is recommended not to update UI directly! public class MainActivity extends Activity implements SensorEventListener {.. private float currentValue; private long lastUpdate; … public void onSensorChanged(SensorEvent event) { currentValue = event.values[0]; lastUpdate = event.timestamp; }.. }

25. API – Cleanup public class MainActivity extends Activity implements SensorEventListener { … protected void onPause () { … sm. unregisterListener (this); } … protected void onStop () { … sm. unregisterListener (this); }.. }

26. Accelerometer Mass on spring GravityFree FallLinear AccelerationLinear Acceleration plus gravity 1g = 9.8m/s 2 -1g 1g

27. Accelerometer STMicroelectronics STM331DLH three-axis accelerometer (iPhone4)

28. Accelerometer Sensor.TYPE_ACCELEROMETER Values[3] = m/s 2, measure the acceleration applied to the phone minus the force of gravity (x, y, z) – GRAVITY_EARTH, GRAVITY_JUPITER, – GRAVITY_MARS, GRAVITY_MERCURY, – GRAVITY_MOON, GRAVITY_NEPTUNE

29. Compass Magnetic field sensor (magnetometer) Z X Y X Y Z 3-Axis Compass? Magnetic inclination Horizontal Gravity Magnetic field vector Magnetic declination Magnetic north Geographic north

30. Compass Hall Effect 3-Axis Compass

31. Compass Sensor.TYPE_MAGNETIC_FIELD values[3] = in micro-Tesla (uT), magnetic field in the X, Y and Z axis SensorManager’s constants – MAGNETIC_FIELD_EARTH_MAX: 60.0 – MAGNETIC_FIELD_EARTH_MIN: 30.0

32. Orientation Sensor Sensor.TYPE_ORIENTATION – A fictitious sensor: orientation is acquired by using accelerometer and compass Deprecated – Now use getOrientation (float[] R, float[] result) Values[3] – (Azimuth, Pitch, Roll) – Azimuth, rotation around the Z axis 0 <= azimuth <= 360, 0 = North, 90 = East, 180 = South, 270 = West – Pitch, rotation around the X axis -180 <= pitch <= 180 0 = sunny side up 180, -180 = up side down -90 = head up (perpendicular) 90 = head down (perpendicular) – Roll, rotation around the Y axis -90<=roll <= 90 Positive values when the z-axis moves toward the x-axis.

33. Orientation: Why Both Sensors? We need two vectors to fix its orientation! (gravity and magnetic field vectors) Tutorial: http://cache.freescale.com/files/sensors/doc/app_note/AN4248.pdfhttp://cache.freescale.com/files/sensors/doc/app_note/AN4248.pdf

34. Gyroscope Angular velocity sensor – Coriolis effect – “fictitious force” that acts upon a freely moving object as observed from a rotating frame of reference

35. Gyroscope

36. Sensor.TYPE_GYROSCOPE Measure the angular velocity of a device Detect all rotations, but few phones have it – iPhone4, iPad2, Samsung Galaxy S, Nexus S – Values[] – iPhone 4 gives radians/sec, and makes it possible to get the rotation matrix

37. Accelerometer vs. Gyroscope Accelerometer – Senses linear movement, but worse rotations, good for tilt detection, – Does not know difference between gravity and linear movement Shaking, jitter can be filtered out, but the delay is added Gyroscope – Measure all types of rotation – Not movement – Does not amplify hand jitter A+G = both rotation and movement tracking possible

38. How to Use the Data – Example float[] matrixR = new float[9]; float[] matrixI = new float[9]; SensorManager.getRotationMatrix( matrixR, matrixI, matrixAccelerometer, matrixMagnetic); float[] lookingDir = MyMath3D.matrixMultiply(matrixR, new float[] {0.0f, 0.0f, -1.0f}, 3); float[] topDir = MyMath3D.matrixMultiply(matrixR, new float[] {1.0f, 0.0f, 0.0f}, 3); GLU.gluLookAt(gl, 0.4f * lookingDir[0], 0.4f * lookingDir[1], 0.4f * lookingDir[2], lookingDir[0], lookingDir[1], lookingDir[2], topDir[0], topDir[1], topDir[2]);

39. Accelerometer Noise - Simple Exponential moving average const float kFilteringFactor = 0.1f; //play with this value until satisfied float accel[3]; // previous iteration //acceleration.x,.y,.z is the input from the sensor accel[0] = acceleration.x * kFilteringFactor + accel[0] * (1.0f - kFilteringFactor); accel[1] = acceleration.y * kFilteringFactor + accel[1] * (1.0f - kFilteringFactor); accel[2] = acceleration.z * kFilteringFactor + accel[2] * (1.0f - kFilteringFactor); result.x = acceleration.x - accel[0]; result.y = acceleration.y - accel[1]; result.z = acceleration.z - accel[2]; Return result;

40. Accelerometer Noise – Notes If it is too slow to adapt to sudden change in position, do more rapid changes when angle(accel, acceleration) is bigger You can throw away single values that are way out of average. |acc| does not have to be equal to |g| ! Kalaman filters – too complicated?

41. Other sensors Light sensor Proximity sensor Pressure sensor Temperature/Humidity/IR Gesture (Galaxy S4)

42. Light sensor Sensor.TYPE_LIGHT values[0] = ambient light level in SI lux units SensorManager’s constants – LIGHT_CLOUDY: 100 – LIGHT_FULLMOON: 0.25 – LIGHT_NO_MOON: 0.001 – LIGHT_OVERCAST: 10000.0 (cloudy) – LIGHT_SHADE: 20000.0 – LIGHT_SUNLIGHT: 110000.0 – LIGHT_SUNLIGHT_MAX: 120000.0 – LIGHT_SUNRISE: 400.0

43. Proximity sensor Sensor.TYPE_PROXIMITY values[0]: Proximity sensor distance measured in centimeters (sometimes binary near-far)

44. Temperature sensor Sensor.TYPE_TEMPERATURE values[0] = temperature

45. Pressure sensor Sensor.TYPE_PRESSURE values[0] = pressure no constants