1. AsiaCCS 2016
Smartwatch-Based Keystroke Inference Attacks and Context-Aware Protection Mechanisms
Anindya Maiti, Oscar Armbruster, Murtuza Jadliwala, Jibo He
{axmaiti, oxarmbruster, murtuza.jadliwala,

2. Keystrokes and Privacy
We type more than ever before.
It is important to protect from eavesdropping, because often the typed information is sensitive.
Typing has become an integral part of our lives. But there’s always someone watching!
Friday, February 22, 2019

3. Keystrokes and Privacy
Credit Card Information
Friday, February 22, 2019

4. Keystrokes and Privacy
Tax Filing
Friday, February 22, 2019

5. Keystrokes and Privacy
s/Messages
And so on…
Friday, February 22, 2019

6. Wi-Fi Channel State Information
Side-Channel Attacks
Channel Type
Common Limitation
Berger et al. [CCS’06]
Acoustic
Change in position of either keyboard or eavesdropping device renders previous training data useless!
Marquardt et al. [CCS’11]
Surface Vibration
Ali et al. [MobiCom’15]
Wi-Fi Channel State Information
For this talk let’s focus on eavesdropping attacks similar to our work, also know as side-channel attacks. They are called side-channel because the primary purpose of the eavesdropping device/sensor was not intended for eavesdropping.
Friday, February 22, 2019

7. Day 1
The fundamental principle behind such attacks is that each key produces unique characteristics, for example, due to the distance from the eavesdropping device. Here, we look at Marquardt et al.’s setup using smartphone motion sensors to capture surface vibration. Keys near… Keys far…will produce different characteristics in surface vibration.
Day 2
Friday, February 22, 2019

8. Smartwatches Wristwatch with functionality well beyond timekeeping.
Miniaturized computer.
Friday, February 22, 2019

9. Behind the Scenes Sensors Motion Microphone GPS Camera Ambient Light
Temperature …
Sensors such as GPS and camera are user manageable, not motion sensors.
Friday, February 22, 2019

10. Problems
We can’t turn off motion sensors. All applications have access to motion sensors by default.
Permissions allows control of access to data directly sensed by the sensors, but not to information that can be inferred indirectly from the sensors!
Smartwatch motions sensors are more difficult to manage than on smartphone. Similar to smartphones motion sensors are nor user manageable.
Friday, February 22, 2019

11. Our Previous Work “(Smart)Watch Your Taps” ISWC’15
Friday, February 22, 2019

12. New Target: QWERTY Keyboards
Friday, February 22, 2019

13. Linear Accelerometer Readings
We Asked Ourselves
Is it Possible to Infer What is Being Typed on the Keyboard Based on the Wrist Movements Observable by the Smartwatch Motion Sensors?
We observed that there was more motion when ever a key on the left side of the keyboard was pressed. Smartwatch worn on left hand. Q M
Linear Accelerometer Readings
Friday, February 22, 2019

14. Dividing the Keyboard
Thus, an attacker can infer which side of the keyboard was pressed based on the level of activity during a key press.
Friday, February 22, 2019

15. Assuming watch is worn on left hand
Further Observations
We can also categorize the direction of movement for the watch wearing hand.
Normally we type multiple characters at a time, such as words or sentences. Therefore, analyzing pairs of keystrokes can fetch us more information than treating them individually. So, we categorize key pairs based on geographical directions such as N, S….And O represents the same key presses consecutively.
Assuming watch is worn on left hand
Friday, February 22, 2019

16. Forming “Word-Profiles”
Word-profile for the word “boards”:
bo oa ar rd ds
R X R . R X L . L E L . L S L . L W L
X represent unknown transitions, where right hand alphabets are involved in the character pair. Mention about this challenge, where we don’t have information from one hand.
Friday, February 22, 2019

17. Learning Phase One of the authors typed the training words.
Friday, February 22, 2019

18. Attack Phase
Friday, February 22, 2019

19. Samsung Gear Live smartwatch
Evaluation
25 participants aged between years.
Matlab and PyBrain
Matlab for feature extraction. PyBrain for machine learning.
Samsung Gear Live smartwatch
Anker A
Bluetooth keyboard
Friday, February 22, 2019

20. Results: Basic Text Recovery
Dictionary: Ten sentences in List 6 of Harvard sentences
Typed: The same ten sentences above
L-R classifier misclassifications: 0%
N-E-S-W-O classifier misclassifications: ~5%
Word Recovery Error: Out of 48 words of four letters or more, only 3 were not recovered correctly (93.75% success in recovery)
Friday, February 22, 2019

21. Similarity Score Closest Matching Word-Profile Frequency of Use
Based on number of matching features
Frequency of Use
As in Dictionary Pool or English Literature
Similarity score is checked against all words in the dictionary.
Friday, February 22, 2019

22. Results: Contextual Dictionary
Participants typed a paragraph of 40 words (of length four or more) that appear in a National Public Radio (NPR) news article on Greece debt crisis, and this experiment simulates eavesdropping on a reporter typing the NPR news article.
The dictionary is formed with words that appear in six other news articles related to Greece debt crisis, that were published a week before the target article.
Friday, February 22, 2019

23. Results: Contextual Dictionary
Contextual Dictionary: Percentage of words recovered per participant, presented in descending order of typing speed of the participants.
Friday, February 22, 2019

24. Results: Typing Behavior and Speed
We observed that in many instances participants did not follow our assumed layout. Some of the participants frequently used their left hand to press a key on the right side of the keyboard, and vice versa.
We also found that participant who typed slower, were less likely to follow the left and right division of the keyboard.
Friday, February 22, 2019

25. Results: Typing Behavior and Speed
Talk about attacker’s speed compared to participants.
Contextual Dictionary: Percentage of words recovered per participant, presented in descending order of typing speed of the participants.
Friday, February 22, 2019

26. Results: Large Dictionary
38 English words typed by participants.
English dictionary of 60,000 words, sorted by frequency of use in English literature.
Problem of colliding word-profiles:
Show: LXR . RXR . RXL
Sums: LXR . RXR . RXL
Explain colliding word-profiles.
Friday, February 22, 2019

27. Results: Large Dictionary
This is similar to Google search (or Baidu). If the desired search result is on the first page, it is more useful than having to search among 20 pages. We know no one goes to Google second page! Talk about drawbacks in experimental setup of Marquardt et al. [surface vibration] and Berger et al. [acoustic emanation].
A comparison of accuracy of our attack with Marquardt et al. [surface vibration] and Berger et al. [acoustic emanation]. Note that in spite of not having wrist movement information available from the non-watch-wearing hand, our results are roughly comparable for a large (60,000 words) dictionary.
Friday, February 22, 2019

28. Limitations Ambient Wrist Movement Left and Right Handedness. But…
Inferring Non-Dictionary Text
Friday, February 22, 2019

29. Smart Mitigation
Access control over seemingly innocuous sensors required.
But should not be the old-fashioned way.
Must be context-aware in order to automatically manage sensor permissions, without having the user to manually change these settings repetitively.
Explain poor usability in ACL. Can we cut-off the motion sensors when the user is typing? Why not? So we used the same motion sensors to develop a typing activity detection framework.
Friday, February 22, 2019

30. Proposed Protection Framework
(i) a real-time typing activity detection (rTAD) and (ii) a motion sensor access-controller (MSAC)
Energy: Activity measured in terms of cumulative linear accelerometer readings. An unworn watch lying on a table has zero energy, while an athlete's watch has high energy. Typing activity typically results in low but nonzero energy. We apply a low pass filter over the linear accelerometer to eliminate high-frequency noise caused by environmental factors.
Turnarounds: Major positive to negative (or vice versa) changes on linear accelerometer readings signify the turnarounds adjoining transitional movements between key presses. Multiple turnarounds in close time proximity can be associated with many activities, such as brushing teeth, eating, playing drums, etc. As a result, we need additional features to distinguish typing from other similar activities.
Magnetic Field Change: Wrists are not rotated significantly when a user types on a QWERTY keyboard, while sitting in front of a stationary desk. Rapid change in north, east and nadir vectors implies non-typing activity.
Direction of Gravity: Gravity generally remains dominant on z-axis of accelerometer while typing on a horizontally placed keyboard. Any major fluctuations or gravity on x-axis or y-axis implies other activities.
Step Count: We assume that the user will be stationary while typing on a computer keyboard. Thus, whenever step count increases, we rule out typing activity.
Friday, February 22, 2019

31. rTAD Parameters
Energy: Activity measured in terms of cumulative linear accelerometer readings. An unworn watch lying on a table has zero energy, while an athlete's watch has high energy. Typing activity typically results in low but nonzero energy. We apply a low pass filter over the linear accelerometer to eliminate high-frequency noise caused by environmental factors.
Turnarounds: Major positive to negative (or vice versa) changes on linear accelerometer readings signify the turnarounds adjoining transitional movements between key presses. Multiple turnarounds in close time proximity can be associated with many activities, such as brushing teeth, eating, playing drums, etc. As a result, we need additional features to distinguish typing from other similar activities.
Magnetic Field Change: Wrists are not rotated significantly when a user types on a QWERTY keyboard, while sitting in front of a stationary desk. Rapid change in north, east and nadir vectors implies non-typing activity.
Direction of Gravity: Gravity generally remains dominant on z-axis of accelerometer while typing on a horizontally placed keyboard. Any major fluctuations or gravity on x-axis or y-axis implies other activities.
Step Count: We assume that the user will be stationary while typing on a computer keyboard. Thus, whenever step count increases, we rule out typing activity.
Friday, February 22, 2019

32. Motion Sensor Access-Controller (MSAC)
Complete Blocking
Reduced Sampling Rate
Random Out of Order Blocks
Explain positives and negatives of each. We were not able to implement it, but should be easy task for OS developers to implement.
Friday, February 22, 2019

33. rTAD Evaluation: High Sensitivity
Friday, February 22, 2019

34. rTAD Evaluation: Low Sensitivity
Friday, February 22, 2019

35. rTAD Evaluation Results
Results can be further improved with new features.
Friday, February 22, 2019

36. Conclusion
A new keystroke inference attack which utilizes wrist-motion data gathered from a smartwatch as side-channel information.
A smart protection framework to detect typing activity and automatically regulate sensor access, aimed to improve privacy without degrading utility of the device.
Thank You!
Friday, February 22, 2019