Presentation on theme: "Designing ‘Robo-Buddies': Technologies to Mediate Communication in the Wild Julie A. Hengst, Laura S. DeThorne, Hillary Valentino & Maeve McCartin University."— Presentation transcript:
Designing ‘Robo-Buddies': Technologies to Mediate Communication in the Wild Julie A. Hengst, Laura S. DeThorne, Hillary Valentino & Maeve McCartin University of Illinois Champaign, IL, USA Oral Session – 1530 Platinum Lounge
“Robo-Buddies” Project Participatory Design Project Team – Engineers, computer scientists, communication specialists, participant-users, disability specialists Collaborating to develop communication technologies that function as pseudo-intelligent mediators of interactions – to improve communication between diverse communicators ( e.g., students on campus with and without disabilities) – by blending strengths of human mediators with features of AAC devices.
Augmentative & Alternative Communication (AAC) /images.html Forms of communication other than oral speech e.g., facial expressions, gestures, symbols, pictures, or writing; people with communication disorders may rely on AAC AAC Devices specialized assistive technologies used by people to supplement or replace impaired speech abilities. e.g., communication boards, dedicated computer systems sorders/AAC.htm palsy/details.aspx?id=92
Activity-based Approach to AAC Prosthetic Approach Grounded in transmission models of communication – Design technology to replace or augment impaired body part or function – For individual user – To produce linguistic messages Proven, but limited, approach for… – speech production in pre-planned, predictable, controlled activities Activity Approach Grounded in CHAT theories – Design technology to mediate interactions and support activities – For functional system – To support interactional alignments in routine activities Novel approach aimed at… – supporting interaction in dynamic, emergent, less-controlled activities
Collecting Interactional Data Participants 13 primary participants (7 AT users with disabilities; 6 comparisons) 7 secondary participants; 51 incidental participants Data Collection—42 video recorded sessions 13 semi-structured interviews (one with each primary participant) 24 On-campus observations (e.g., scavenger hunts; working with PA in dorm; navigating campus) 5 lab trials (e.g., working with trial technologies)
Analyzing Interactional Patterns 17 on-campus observations – 4 with “David”; 3 w/ “Artemesia” & “Iris”; 2 w/ “Chip”, “Jester” & “Jesse”; 1 w/ “Izzy”. 11 hrs 20 mins, video data. Analysis: – Transcribing all sessions. – Coding all sessions for interactional patterns. – Completing situated analysis of selected interactions. “Artemesia” “Iris” “David”
Coding Interactional Patterns Conversational Features: – Conversational trouble source (TS) disruptions in conversational flow, may or may not lead to a breakdown. – Conversational repetition, or reformulation (R) marked repetition, reported speech, and replays, within or across modalities. Interactional Discourse Resources: – Humor, or playful episodes (PE) utterances sharing a common playful theme. – Conversational narratives (CN) Verbal or nonverbal telling of event(s) displaced from the moment of telling, plus evaluation. – Procedural discourse (PD) telling how to do something (e.g., giving directions). Object Specific Codes: – Contextual orchestration with objects (CO) objects within activity. – Fidgets (F), fidgeting with objects, unrelated to communicative activity.
Coding Summary: Average #/10mins of observation by participant. (Note: Total counts across participants in grey.) Participant et al. Time (mins) TSRPECNPDCOF Artemesia Iris David Chip Jester Jessie Izzy Total Counts
Interactional Profiles Average of All (60.0*):R>CO>PE>CN>TS>PD>F Chip (96.5):R, CO, PE, CN, TS, F, PD Jester (76.1):R, CO, PD, CN, TS, PE, F Iris (64.9):R, PE, CN, CO, PD, TS, F Jessie (49.4):R, CO, CN, TS, PD, PE, F David (48.7): R, CO, TS, PD, PE, CN, F Artem. (46.9):R, CO, PE, TS, CN, PD, F Izzy (37.8):R, CO, CN, PE, PD, TS, F Listed in order from most to least. Overall frequency of codes varies (e.g., /10 min). Frequency of specific codes varies (e.g., PD varied from 3 rd - 7 th ) *#codes/10 minutes of observation
Interactional Profiles Average of All (60.0*):R>CO>PE>CN>TS>PD>F Chip (96.5):R, CO, PE, CN, TS, F, PD Jester (76.1):R, CO, PD, CN, TS, PE, F Iris (64.9):R, PE, CN, CO, PD, TS, F Jessie (49.4):R, CO, CN, TS, PD, PE, F David (48.7): R, CO, TS, PD, PE, CN, F Artem. (46.9):R, CO, PE, TS, CN, PD, F Izzy (37.8):R, CO, CN, PE, PD, TS, F Data Excerpts: Artemesia & Jessie Service Encounters, with PA’s included Examples of: Repetition Contextual use of objects Trouble source Mediational roles of the PAs
“Artemesia” 38-year-old history major; – Employs multiple PA’s to assist 24hrs/day with physical needs due to CP 2 on-campus observations – Dining hall with a PA; – Outside dining all, with a PA Profile: R, CO, PE, TS, CN, PD, F Dining Hall: “selecting meal items with PA” Dining Hall: “PA interpreting story”
Service encounter in dining hall with familiar partner. Repetition for confirming. Artemesia directing PA to selecting items; PA questions some choices.
“Jessie” 30-yr-old, doctoral student in Community Health & Disability Studies; – Employs 6 PA’s to assist with physical needs due to CP 2 on-campus observations – Student Union bowling alley; – w/ his PA, Tech Store in Union Profile:R, CO, CN, TS, PD, PE, F At SHS Building: “story of first meeting” At Tech Store: “clickers”
Service encounter, shopping in store with unfamiliar clerk. Repetition helps to resolve trouble source. PA is co-shopper.
CHAT Approaches to AAC Shift in unit of analysis – from “replacing” individual abilities to mediating activities of functional systems Interactional data and profiles – empirically grounded discussions – distributed, flexible, and persistent nature of functional systems Imagining mediational roles for technologies – connecting people to histories and broader contexts – reformulating words/actions in the moment – layering ongoing activities with interpretive frames
Acknowledgements Grant Support: M. Hasegawa-Johnson (PI), J.A. Hengst, L.S. DeThorne, T. Huang, P. Malik, and T. Gunderson. Pseud-intelligent mediators ("Robo-Buddies") to improve communication between students with and students without disabilities. UIUC IN3 Grant. Funded August August 2014; Amount: $199,898. Speech and Hearing Science Sara Small, Katie Lester, Maeve McCartin, Hillary Valentino, Monique Kammo, Shanthi Sivasankaran, Carissa Ernat, Caroline Eichelberger, Gena Carpenter, Mariana Aparicio, Suma Devanga, Christina Bronson-Lowe, Human Computer Interface Karrie Karahalios, Jennifer Kim, Ha Kyung Kong Disability Resource Educational Services Electrical & Computer Engineering Xuesong Yang, Yang Zhang, Dennis Lin, Pooya Khorrami Research Team and Lab Assistants: Participants: A total of 71 individuals as primary, secondary, or incidental participants.