1. ECE 494 Capstone Design Final Design Presentation Smartphone Based Human Behavioral Analysis Andrew Jackson Michael Armstrong Robbie Rosati Andy McWilliams Aaron Stewart April 18, 2014 Advisor: Dr. Fei Hu

2. Outline 4/18/2014Smartphone Based Human Behavioral Analysis2  Project Recap & Goal  System Diagram  Subsystem Breakdown & Team Roles  Sensor Data Extraction & Analysis  App Development & Activity Recognition  DTW Algorithm  Environment Sensors  NN Algorithm  Raw Memory Extraction & Analysis  HMM Algorithm  Administration

3. Project Recap & Goal 4/18/2014Smartphone Based Human Behavioral Analysis3  Create a system for tracking and detecting a user’s behavioral patterns though the phone’s sensors and internal memory logs.  Can be used in multiple areas:  Healthcare  Activity Monitoring  Homeland Security

4. System Diagram 4/18/20144Smartphone Based Human Behavioral Analysis

5. Behavior Recognition Phone Data Sensor Data Extraction & Analysis Accelerometer Light Sound Proximity GPS Raw Memory Extraction & Analysis dc3dd Call Logs Text Messages Machine Learning Algorithms Hidden Markov Model Neural Networks Dynamic Time Warping Subsystem Breakdown & Team Roles 4/18/20145Smartphone Based Human Behavioral Analysis Andrew Jackson Android Sensor App DTW Algorithm Aaron Stewart Raw Memory Analysis Team Leader Robbie Rosati Machine Learning Algorithms Raw Memory Extraction Michael Armstrong Raw Memory Extraction HMM Algorithm Andy McWilliams HMM Algorithm Raw Memory Analysis

6. Sensor Data Extraction & Analysis Aaron Stewart

7. Android, iOS, or Windows Phone We chose to go with Android. No development license fees Majority of market Many sensors available Open file system Previous Android programming experience Cheaper phone prices 4/18/20147Smartphone Based Human Behavioral Analysis

8. Samsung Galaxy S4 Mini 4/18/2014Smartphone Based Human Behavioral Analysis8  New phone with many sensors  Fairly inexpensive for an unlocked phone  Smaller size great for testing in pockets

9. Accelerometer LightSoundProximityGPS 4/18/2014 Smartphone Based Human Behavioral Analysis 9 Sensors

10. Sensor Types 4/18/2014Smartphone Based Human Behavioral Analysis10 Accelerometer Measures acceleration with gravity value Reports data in 3 values, one for each axis Example: Could tell if user is walking or stationary GPS Reports GPS coordinates See changes in location Example: Could tell if user is driving

11. Sensor Types 4/18/2014Smartphone Based Human Behavioral Analysis11 Light Intensity of light Reports data in a single value Example: Indoors/Outdoors, Weather, Time of day Sound Detects sound waves Reports loudness Example: Voice recognition Proximity Changes depending on surrounding objects Boolean value Example: Device is in tight space such as pocket or purse

12. Behavior Analysis App Version 1.0 4/18/2014Smartphone Based Human Behavioral Analysis12  Shows values from each sensor in real-time as the sensor updates  Can save the data to a file on phone  Move file to computer to analyze with MATLAB  Graph the data  Validate that it makes sense

13. Testing with App 3/10/2014Smartphone Based Human Behavioral Analysis13  Most sensor data comes in a set of 3 points  Few have single values  Tests related to typical smartphone user behaviors  Used MATLAB for data graphing and filtering

14. Light Sensor Test (Different Exposure) 4/18/2014Smartphone Based Human Behavioral Analysis14

15. Accelerometer Test (Answer Call) 4/18/2014Smartphone Based Human Behavioral Analysis15 XYZXYZ

16. App Development and Activity Recognition Andrew Jackson

17. 4/18/2014 Smartphone Based Human Behavioral Analysis 17

18. Behavior Analysis App Version 2.0 4/18/2014 Smartphone Based Human Behavioral Analysis 18  Integrates DTW machine learning algorithm  Allows training of the algorithm from the phone – no computer required  Once trained, will recognize where the phone is with almost 100% accuracy  Visual Upgrades  All features from previous app rolled over

19. Further App Improvements 4/18/2014Smartphone Based Human Behavioral Analysis19  Increased accuracy of activity recognition by  Increasing sensor sensitivity  Polling each sensor more times a second  Tweaking DTW algorithm code  Using multiple sensors at the same time  Continued adding new activities  Added speech recognition By demo day:  Add in GPS coordinates  Further explore light, sound, and proximity sensor possibilities

20. Accelerometer, Light, and Proximity 4/18/2014 Smartphone Based Human Behavioral Analysis 20  Chose to work with accelerometer first because it can give some of the most useful data about the phone  Applying DTW on accelerometer data  Next, applied light and proximity sensors to get better results

21. Recognized Activities Last Presentation Now 4/18/2014 Smartphone Based Human Behavioral Analysis 21  Walking (with phone in hand)  Talking on phone  Sitting on table  Holding in hand  Walking (with phone in pocket)  Running  Stairs  Sitting  Driving

22. 4/18/2014 Smartphone Based Human Behavioral Analysis 22 App Demonstration

23. Dynamic Time Warping Andrew Jackson

24. Dynamic Time Warping  Measures similarity between two sequences which may vary in time or speed  Calculates an optimal match between the two given sequences or time series  A distance-like quantity is measured between the two series Smartphone Based Human Behavioral Analysis4/18/201424

25. DTW Time Series 4/18/2014Smartphone Based Human Behavioral Analysis25  Time series can be accelerated/decelerated as much as necessary to give an optimal match. Cost: 3.3084e+05Cost: 2.7239e+06

26. Applying DTW to the Project 4/18/2014Smartphone Based Human Behavioral Analysis26  Allows training for different activities with disregard to time  Supports three-dimensional data  Provides extremely accurate results  Runs in O(n 2 ) time

27. FastDTW 4/18/2014Smartphone Based Human Behavioral Analysis27  Open-source algorithm based on original DTW  Runs in O(n) time  Slightly less accurate  Easy to port to Android

28. Environmental Sensors Robbie Rosati

29. Proximity Sensor 4/18/2014Smartphone Based Human Behavioral Analysis29  Gives proximity in cm  Most phones only return binary values “near” and “far”  Ability to check whether phone is pressed against ear, in pocket, etc. neural networks

30. Light Sensor 4/18/2014Smartphone Based Human Behavioral Analysis30  Detects ambient luminosity in lux  Useful for indoor/outdoor detection  Could be included in gestures, used with DTW

31. GPS 4/18/2014Smartphone Based Human Behavioral Analysis31  Can get user’s latitude and longitude  Could improve detection for if user is driving  Worst sensor with battery life  Need to only use it occasionally

32. Microphone 4/18/2014Smartphone Based Human Behavioral Analysis32  Speaker recognition via NN algorithm  Passive or active detection  Could use to detect loudness of rooms  Also see things in sound waveform – like snoring

33. Neural Networks Robbie Rosati

34. Transition from Support Vector Machine to Neural Networks 4/18/2014Smartphone Based Human Behavioral Analysis34  Implemented SVM into our Behavior Analysis app  Ran too inefficiently for phone hardware  Difficult to train with the sound sensor Therefore, decided to use an alternative algorithm that would fit our needs.  Decided on Neural Networks, a popular machine learning algorithm for speech recognition

35. Neural Networks 4/18/2014Smartphone Based Human Behavioral Analysis35  Algorithm used for machine learning and pattern recognition  Inspired by the way the brain recognizes objects and sound  Presented as systems of interconnected neurons that can compute values  Difficult to train in a short time so used an API from Google to offload processing from the phone  Integrates speech recognition into the app

36. Google Speech API 4/18/2014Smartphone Based Human Behavioral Analysis36  Open source API for speech recognition  Could be coded into our Behavior Analysis app  Uses NN to interpret speech  Can be presented in text with further coding  However, the app requires an internet connection for this function since it streams audio to remote servers

37. 4/18/2014 Smartphone Based Human Behavioral Analysis 37 App Demonstration

38. Raw Memory Extraction & Analysis Michael Armstrong

39. 4/18/2014 Smartphone Based Human Behavioral Analysis 39

40. Overview 4/18/2014Smartphone Based Human Behavioral Analysis40  Retrieving the physical image of a device is our goal.  Immense variety of phones with an array of OS and applications.  Current solutions are time consuming and/or very expensive.  Access to deleted data  A logical image is easier to obtain, but it omits deleted data, and logical extraction interfaces usually enforce access rules and may modify data upon access.

41. Samsung Galaxy S3 Mini 4/18/2014Smartphone Based Human Behavioral Analysis41  Relatively inexpensive  Compatible with team’s SIM cards  SD card slot  Has some sensors in case we need it as back up for sensor testing

42. Testing Options 4/18/2014Smartphone Based Human Behavioral Analysis42  Flashing is interpreted as a dump of the phones memory into a format that is either hexadecimal or binary.  Backup Software  Flashing box  Linux Forensics Software  dc3dd

43. dc3dd 4/18/2014Smartphone Based Human Behavioral Analysis43  Terminal based utility for Linux  Parses a partition bit-by-bit and creates binary image  Advantages: Exactly what we need, easy to use  Disadvantage: Parses through empty space and fills it with zeroes, creating a very large image file full of nothing however, there is a workaround for this issue

44. Steps for Extraction 4/18/2014Smartphone Based Human Behavioral Analysis44  Gain root access to Android using simple utility  Extract a copy of the database file  Shrink the partition of a USB flash drive to the smallest possible size and copy the database file to the partition  Use dc3dd to parse the partition and create the binary image for the database file

45. Viewing the Image File 4/18/2014Smartphone Based Human Behavioral Analysis45  Any hex viewer application should able to view the data inside the image  Chose GHex because it is easy to obtain with Ubuntu and has search functions for finding the data we are looking for

46. Finding Desired Data 4/18/2014Smartphone Based Human Behavioral Analysis46  Messages and numbers are encoded as ASCII  Locate corresponding hex data using the search function in Ghex  Verify by comparing the located hex data to ASCII values  Process should work for any type of file because the ASCII data will there regardless of file type

47. Example: Text Message 4/18/2014Smartphone Based Human Behavioral Analysis47

48. ‘Strings’ and ‘Grep’ Commands 4/18/2014Smartphone Based Human Behavioral Analysis48  Strings - Captures groups of letters separated by spaces in a file  Grep - Searches for an expression matching the user’s input expression

49. Hidden Markov Model Andy McWilliams

50. Hidden Markov Model Smartphone Based Human Behavioral Analysis4/18/201450  Way of predicting the next possible output in a sequence of events  Equivalent to recognizing or interpreting that particular sequence  Slightly complicated because there can be multiple ways to produce the same observed output

51. Hidden Markov Model 4/18/2014Smartphone Based Human Behavioral Analysis51  Accomplished through MATLAB  There will be different matrices such as the transition matrix and emission matrix  Transition matrix – probability of transitioning from one state to the next  Emission matrix – probability that one state emits the next state

52. Hidden Markov Model 4/18/2014Smartphone Based Human Behavioral Analysis52 1. Model Evaluation Problem  What is the probability of the observation?  Given an observed sequence and an HMM, how probable is that sequence?  Forward algorithm 2. Path Decoding Problem  What is the best hidden state sequence for the observation?  Given an observed sequence and an HMM, what is the most likely state sequence that generated it?  Viterbi algorithm 3. Model Training Problem  How to estimate the model parameters?  Given an observation, can we learn an HMM for it?

53. Example with Binary 4/18/2014 Smartphone Based Human Behavioral Analysis 53

54. HMM MATLAB 4/18/2014Smartphone Based Human Behavioral Analysis54  Answers received contain new transition and emission matrices with probabilities  There are three files in which we work with to execute the code  Multiple Assumptions must be made in order to create the three files

55. Three Files of MATLAB 4/18/2014Smartphone Based Human Behavioral Analysis55  1. Markovsource is a data file that contains multiple fields of phone numbers, dates, etc.  2. The transfer file contains matlab code to convert the data into the necessary values  3. The markovTEmatrix file contains the transition and emission matrices  These three files are used to receive the state probabilities for each matrix

56. Assumptions 4/18/2014Smartphone Based Human Behavioral Analysis56  There are three kinds of fields within the data file such as phone numbers with 10 digits, dates, and other non- numerical alphabetic fields  Within the matrices, the first ten rows represent the date field’s probabilities  The next ten rows represent the phone number fields  There is an extra 1 or 2 rows that are used for the non- numerical alphabetic fields

57. Transition and Emission Matrices 4/18/2014Smartphone Based Human Behavioral Analysis57

58. Multiple MATLAB Functions 4/18/2014Smartphone Based Human Behavioral Analysis58  Hmmdecode - used to calculate the state probabilities of a sequence of emissions  Hmmviterbi – used to calculate the most probable state path for a hidden Markov model  Hmmtrain – used to calculate likelihood estimates for the transition and emission matrices from a certain sequence  Hmmestimate – can be used if the sequence is already known, but it has something missing

59. State Probabilities 4/18/2014Smartphone Based Human Behavioral Analysis59

60. Administration Andy McWilliams

61. Validation Plan 4/18/2014Smartphone Based Human Behavioral Analysis61  Sensor Data Extraction & Analysis Retrieve real-time data from all chosen sensors Graph and analyze each set of data by performing tests  Raw Memory Extraction & Analysis Test each raw memory extraction option Retrieve binary image of databases  Machine Learning Algorithms Use algorithms to recognize patterns in user’s behavior Use HMM algorithm to retrieve deleted data

62. Budget 4/18/201462Smartphone Based Human Behavioral Analysis Also received $400 from Dr. Hu’s budget to purchase Samsung Galaxy S4 Mini

63. Schedule Smartphone Based Human Behavioral Analysis4/18/201463

64. Impacts  Healthcare, Activity Monitoring, Forensics  National Science Foundation  Dr. Hu will use project findings to help teach CPS at rural schools in a virtual environment 4/18/201464Smartphone Based Human Behavioral Analysis

65. Thank you! Questions?