Communication via the Skin: The Challenge of Tactile Displays Lynette Jones Department of Mechanical Engineering, Massachusetts Institute of Technology Cambridge, MA

Spectrum of Tactile Displays Sensory substitutionHuman-computer interactionsNavigation/orientation Visual impairments Hearing impairments Vestibular (balance) impairments TSAS (Rupert et al.) MIT tactile display CyberTouch Data Glove EST Tactile mouse Tactile belt DataGlove

Tactile Displays - CTA ADA Focus Utilize a relatively underused sensory channel to convey information that is private and discreet Assist in navigation or threat location in the battlefield Increase SA in virtual environments used for training Enhance the representation of information in displays

Torso-based Tactile Displays Abdomen Finger Forearm Function as an alert Orientation and direction information Sequential activation of array – vector conveys “movement” in environment Effective in environments with reduced visibility – enhances situation awareness Vibrotactile Sensitivity

Development of Tactile Display Actuator (tactor) selection and characterization Development of body-based system (configuration of display, power, wireless communication) Perceptual studies – optimize design of the display in terms of human perceptual performance Develop a framework for creating a tactile vocabulary – tactons Field studies – measure the efficacy of display for navigation, identifying location of environmental events, and examine robustness of system (e.g. impact of body armor)

Characteristics of the Actuators Evaluated Cylindrical Motor Pancake Motor R1 Rototactor Length Diameter (mm) Mass (g) Peak frequency (Hz) 110 (at 4 V) 103 (at 8.8 V) 200 (at 10 V) Peak accel. (ms -2 ) Voltage (V) Rated: 3 Range: Rated maximum current (mA) Current (mA) at 3.3 V Cylindrical motor Rototactor Pancake motor Cylindrical motor (Jones, Lockyer & Piateski, 2006) C2 tactor Tactaid

Prototypes

Tactile Display - Final Elements Core components - Pancake motors, Wireless Tactile Control Unit Contact area - ~ 300 mm 2 (encased in plastic) Input signal – 130 Hz at 3.3 V, sinusoidal waveform Power – 9 V battery or 7.2 V Li-ion rechargeable, 2200 mAh Display – vest, waist band, sleeve Visual Basic GUI

Actuator Evaluation – Frequencies and Forces Mechanical properties not affected by encasing motors (Jones & Held, 2008)

Actuator Evaluation – Tactor spacing and Intensity Mechanical testing of skin Skinsim with accelerometers (Jones & Held, 2008)

Transitions – MIT Tactile Display ARL  Investigated the efficacy of tactile and multimodal alerts on decision making by Army Platoon Leaders (Krausman et al., 2005, 2007)  Analyzed the effectiveness of tactile cues in target search and localization tasks and when controlling robotic swarms (Hass, 2009)  Evaluated Soldiers’ abilities to interpret and respond to tactile cues while they navigated an Individual Movement Techniques (IMT) course (Redden et al., 2006)  Measured the effects of tactile cues on target acquisition and workload of Commanders and Gunners and determined the detectability of vibrotactile cues while combat assault maneuvers were being performed (Krausman & White, 2006; White et al., in press). The MIT tactile displays have also been incorporated into multi-modal platforms developed by the University of Michigan, ArtisTech in the CTA test bed, and Alion MA&D for a robotics control environment.

Questions addressed – MIT Research Can tactile signals be used to provide spatial cues about the environment that are accurately localized? How does the location and configuration of the tactile display influence the ability of the user to identify tactile patterns? What is the maximum size of a tactile vocabulary that could be used for communication? Which characteristics of vibrotactile signals are optimal for generating a tactile vocabulary? Can a set of Army Hand and Arm Signals be translated into tactile signals that are accurately identified when the user is involved in concurrent tasks?

Localization of Tactile Cues for Navigation and Orientation Navigation –Way-finding –Location of events – real and simulated environments –Control of robots WaistBack Experiments 10 subjects in each experiment Each tactor activated 5 times (randomly) Subject indicate location of tactor vibrated

Navigation – Tactile Belt – One-dimensional Display Navel Spine Left Right Identification of tactor location Eight locations – 98% correct (Inter-tactor spacing: mm) Twelve locations – 74% correct (Spacing: mm) (Jones & Ray, 2008) Inter-tactor distance mm

Localization – Two-dimensional Display Identification of tactor location 16 locations – 59% correct (40-82%) Within 1 tactor location: 95% Inter-tactor spacing: 40 mm vertical 60 mm horizontal Darker the shading, the more accurate the localization (Jones & Ray, 2008)

Results Spatial localization becomes more difficult as the number of tactors increases and the inter-tactor distance decreases Two-dimensional 16-tactor array on the back is unable to support precise spatial mapping, for example between tactile location and visual target –driving or to highlight on-screen information One-dimensional array is very effective for conveying directions

Questions addressed Can tactile signals be used to provide spatial cues about the environment that are accurately localized? How does the location and configuration of the tactile display influence the ability of the user to identify tactile patterns? What is the maximum size of a tactile vocabulary that could be used for communication? Which characteristics of vibrotactile signals are optimal for generating a tactile vocabulary? Can a set of Army Hand and Arm Signals be translated into tactile signals that are accurately identified when the user is involved in concurrent tasks?

Location and Configuration of Tactile Display Tested vibrotactile pattern recognition on forearm and back Fabricated 3x3 (arm) and 4x4 (torso) arrays both controlled by Wireless Tactile Control Unit (WTCU) Tactile patterns varied with respect to spatial cues (location), amplitude (number of tactors simultaneously active) and spatio- temporal sequence.

Group mean percentage of correct responses – averaged across tactors – 89% Results (Piateski & Jones, 2005) Tactile Patterns

A 2, 4 1, 3 2, 4 1, 3 1,2, 3,4 BC D E FGH UpDown Right Left Left, right, left, right Top, bottom, top, bottom Blink corners 4 times Blink single motor 4 times 100% 99% 97% Back -Tactile Pattern Recognition (Piateski & Jones, 2005)

,4 N M 2,4 1,3 4,8 3,7 2,6 1,5 1,3 2,4 1,3 2,4 Top, bottom, top, bottom Bottom, top, bottom, top Right, left, right, left Each corner vibrates twice Up H Down ED 1,2 5,6 7,8 1 Right Left Left, right, left, right Single tactor vibrates four times AB F C G I I 1,3 2, K Corners vibrate together four times Middle two rows Outer corners then inner twice L Diagonal vibrates four times O Two corners vibrate in turn twice Tactile Vocabulary – Tactons? Mean: 96% J (Jones, Kunkel, & Torres, 2007)

Experiment 1A Experiment 1B Mean correct response rate: 62% in Expt 1A 85% in Expt 1B IT: 1.48 bits IT: 2.15 bits Confusion matrix (Expt 1A): A misidentified as F, whereas F mis- identified as D – errors not symmetrical. Tactile patterns that “moved” across the arm more accurately perceived than those that “moved” along the arm Tactile Pattern Recognition – Effect of Stimulus Set (Jones, Kunkel, & Piateski, 2009) A C D E F G H B Up Down Right Left Blink center 3 times Blink X-shape 3 times , 2, 3 1,3 1,3 1,3 2 Top, bottom, top Left, right, left ,3

Summary of Findings  Arm vs back – both provide effective substrates for communication  Array dimensions – marked effect on spatial localization  Asymmetries in spatial processing on the skin  Need to evaluate patterns in the context of the “vocabulary” used  Tactile vocabulary size – absolute identification vs communication  Interceptor Body Armor - no effect on performance Direction and orientation Saltation Tap on shoulder

Navigation path Field Experiments Five subjects participated Eight patterns with five repetitions Familiarization with visual analog initially Brief training period outdoors Navigation using only tactile cues, without feedback 100% accuracy for 7/8 patterns presented Single error on 8th pattern Demonstrated that navigation is accurate using only tactile cues as directions (Jones, Lockyer & Piateski, 2006)

Questions addressed Can tactile signals be used to provide spatial cues about the environment that are accurately localized? How does the location and configuration of the tactile display influence the ability of the user to identify tactile patterns? What is the maximum size of a tactile vocabulary that could be used for communication? Which characteristics of vibrotactile signals are optimal for generating a tactile vocabulary? Can a set of Army Hand and Arm Signals be translated into tactile signals that are accurately identified when the user is involved in concurrent tasks?

Tactons (tactile icons) Structured tactile messages that can be used to communicate information. These tactons must be intuitive and salient. Navigation tactons Communication tactons Assemble/rally 1,23,4 5,67,8

Tactons for hand-based communication Frequency Duration, repetition rate Waveform complexity (Jones & Sarter, 2008)

FrequencyIntensityWaveformDurationLocation Range: Hz. Optimal sensitivity: Hz 1 Absolute thresholds across body sites:  m at 200 Hz 3 Relatively insensitive to waveforms: sinusoidal, triangular, square wave 5 Burst duration: ms (typical) Differential thresholds: 7- 50% 5 Localization accuracy varies with body site 7 Body site influences perceived frequency Changes with increased voltage to a single tactor and with number of tactors activated Amplitude modulation of sinusoids effective for varying roughness of signals 6 Pulse repetition rate (create temporal patterns - rhythms) Inter-tactor spacing and array configuration important Differential thresholds: 18-50% 2 Differential thresholds: 5- 30% 4 Number of pulses: 1-5 (typical) Localization superior near anatomical points of reference (elbow, spine) 7 Tacton building blocks: Relevant properties of each variable (Jones, Kunkel, & Piateski, 2009)

Arm and Hand Signals for Ground Forces Identify a set of structured tactile messages (tactons) that can be used to communicate information. 1, , 6 3, Take cover Increase speed Danger area

Advance or Move Out Halt , 4 1, 3 Attention Each tactile hand signal was designed to keep some of the iconic information of the matching visual hand signal

Mean (N=10) percentage of correct responses (35 trials per subject) when identifying the hand signal with both the illustration and schematic available (black - 98% correct) and with only the illustration available (red – 75% correct). Hand and Arm Signals – Tactile-visual mapping (Jones, Kunkel, & Piateski, 2009)

Questions addressed Can tactile signals be used to provide spatial cues about the environment that are accurately localized? How does the location and configuration of the tactile display influence the ability of the user to identify tactile patterns? What is the maximum size of a tactile vocabulary that could be used for communication? Which characteristics of vibrotactile signals are optimal for generating a tactile vocabulary? Can a set of Army Hand and Arm Signals be translated into tactile signals that are accurately identified when the user is involved in concurrent tasks?

Field Experiments – Concurrent activities (Jones, Kunkel, & Piateski, 2009) Nuclear, biological and chemical attack AssembleAttention Increase speed Take cover Advance to left Danger area Halt 91% 93%

Conclusions  Vibrotactile patterns easily perceived on torso with little training and single stimulus exposure  Demonstrated feasibility of using sites that are non- intrusive and body movements are not impeded  Shown that the ability to perceive tactile patterns is not affected by concurrent physical and cognitive activities  Directional patterns are intuitive and can readily be used as navigational and instructional cues  Two-dimensional arrays provide greater capabilities for communication, but one-dimensional arrays are effective for simple commands

Acknowledgements Edgar Torres Amy Lam David Held Christa Margossian Katherine Ray Brett Lockyer Mealani Nakamura Erin Piateski Jacquelyn Kunkel Research was supported through the Advanced Decision Architectures Collaborative Technology Alliance sponsored by the U.S. Army Research Laboratory under Cooperative Agreement DAAD

References Jones, L.A., Kunkel, J. & Piateski, E. (2009). Vibrotactile pattern recognition on the arm and back. Perception, 38, Jones, L.A. & Held, D.A. (2008). Characterization of tactors used in vibrotactile displays. Journal of Computing and Information Sciences in Engineering, Jones, L.A. & Ray, K. (2008). Localization and pattern recognition with tactile displays. Proceedings of the Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems, Jones, L.A. & Sarter, N. (2008). Tactile displays: Guidance for their design and application. Human Factors, 50, Jones, L.A., Kunkel, J., & Torres, E. (2007). Tactile vocabulary for tactile displays. Proceedings of the Second Joint Eurohaptics Conference and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems, Jones, L.A., Lockyer, B., & Piateski, E. (2006). Tactile display and vibrotactile pattern recognition on the torso. Advanced Robotics, 20, Piateski, E. & Jones, L.A. (2005). Vibrotactile pattern recognition on the arm and torso. Proceedings of the First Joint Eurohaptics Conference and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems,