1. Technology for the Aging A Collaborative Effort Edward Riseman Allen Hanson Roderic Grupen Erik Learned-Miller Phebe Sessions Julie Abramson Mary Olson Candy Sidner UMassAmherst Supported by the National Science Foundation MERL

2. Collaborative Effort Collaborative Research Effort Collaborative Research Effort Social scientists and computer scientists Social scientists and computer scientists Elderly representatives and their stakeholders Elderly representatives and their stakeholders caregivers caregivers family family local and state agencies, etc. local and state agencies, etc. Two Part Talk: Social Science and Technology Two Part Talk: Social Science and Technology

3. Enhancing Control and Empowerment for the Elderly through Assistive Technology Social Science Components November 4, 2005 Washington, D.C. Dr. Phebe Sessions Smith College School for Social Work

4. Outline Issues in collaboration Issues in collaboration Theoretical framework: ecosystemic and Theoretical framework: ecosystemic and social constructionist social constructionist Gerontological literature and traditions Gerontological literature and traditions Research plan Research plan

5. Issues in Collaboration UMass Computer Sciences/Smith SSW Why social work? Pragmatism Pragmatism Commitment to underserved populations Commitment to underserved populations Community-building skills Community-building skills Challenges of collaboration Novelty and not knowing Novelty and not knowing Different languages Different languages

6. Theoretical framework Ecosystemic and Social Constructionist Ecosystemic and Social Constructionist Strengths perspective Strengths perspective Collaboration, dialogue Collaboration, dialogue Qualitative methods in research Qualitative methods in research

7. Gerontological literature and traditions Aging and the deficit perspective: “declinism” Aging and the deficit perspective: “declinism” Positive aging and core significance of control Positive aging and core significance of control Learning issues and aging Learning issues and aging Social networks and aging Social networks and aging Ethical considerations Ethical considerations

8. Research plan Qualitative methods Qualitative methods Year One: Focus groups with elderly, Year One: Focus groups with elderly, families, and service providers families, and service providers Year Two: Implementation of technology Year Two: Implementation of technology in common room in common room Year Three: Revision of technology based Year Three: Revision of technology based on Year Two findings on Year Two findings

9. Technology Components Umass Vision and Robotics groups Umass Vision and Robotics groups Professor Emeritus Edward Riseman (PI) Professor Emeritus Edward Riseman (PI) Professor Allen Hanson (co-PI) Professor Allen Hanson (co-PI) Professor Roderic Grupen (co-PI) Professor Roderic Grupen (co-PI) Mitsubishi Electronic Research Laboratory Mitsubishi Electronic Research Laboratory Dr. Candace Sidner Dr. Candace Sidner

10. Technology Overview ISSUES: privacy access client control Graphics Scripts Vision Speech Synthesis Tracking Speech Understanding Lifestyle/Event Coordination Activity Modeling Conversational Interaction Assisted-Living Clientele Family Staff Professionals Care-Givers Legal Medical Structure Interface Family/Community Interface Human Services Interface MERLUMASS SMITH

11. Interaction: Appointment Management IA: Hi, Maisie. Senior: Hi. IA: later this morning at 11 am you have a dentist appt. Senior: Ok IA: You will have to take the bus so you need to leave here an hour before and walk to the bus stop. That means leaving about 10 am. Senior: I want to leave later. IA: Well, the bus takes 20 minutes. How long do you need to get to the bus stop? Senior: 15 minutes IA: The bus comes every 15 minutes, so when do you think you should leave? Senior: 10:15 IA: Okay. How about I remind you at 10 am to get ready? Senior: Good. IA: Good bye for now.

12. Technical Goals An effective API to the client’s environment: An effective API to the client’s environment: Independent Living Independent Living Social/family Social/family Medical Medical Financial Financial Social services Social services Legal Legal ….. ….. Plug and play for rapid development and deployment Plug and play for rapid development and deployment Interface driven user interactions Interface driven user interactions System interfaces System interfaces Nimble, reconfigurable, robust, minimally intrusive, controllable Nimble, reconfigurable, robust, minimally intrusive, controllable

13. The Client-Care Sandbox Nimble & Reconfigurable

14. Component Technologies Fault tolerant script based interfaces: Collagen plus GUI Fault tolerant script based interfaces: Collagen plus GUI Minimal intrusion Minimal intrusion Robust Robust Speech and speech understanding Speech and speech understanding Command and control Command and control Voice Analysis - e.g. stress Voice Analysis - e.g. stress Visual interfaces Visual interfaces Virtual Reality Virtual Reality Tracking Tracking Change Detection Change Detection Environment Environment Object locations Object locations Motion/expected tracks, deviation from normal activity Motion/expected tracks, deviation from normal activity People People Gait and postural models Gait and postural models Speech patterns Speech patterns

15. Collagen: Collaborative Agents Principals: Candace L. Sidner and Charles Rich Java middleware for collaborative interface agents User and agent collaborate to complete predefined tasks Speech or text-based discourse with agent mouse, touch screen, others Provides friendlier, more familiar interface for inexperienced users Many existing applications http://www.merl.com/projects/collagen Principals: Candace L. Sidner and Charles Rich Java middleware for collaborative interface agents User and agent collaborate to complete predefined tasks Speech or text-based discourse with agent mouse, touch screen, others Provides friendlier, more familiar interface for inexperienced users Many existing applications http://www.merl.com/projects/collagen Mitsubishi Electric Research Laboratory

16. Collaborative Interface Agent * SharedPlans per Grosz, Sidner, Lochbaum, Kraus, et al. communicate interact observe plan tree focus stack * Collagen

17. Collagen Example: Calendar Application Dialog Fragment 1. User: “Let’s create a new appointment.” 2. Agent: activates Create Appointment window. 3. User: enters the appointment’s date and description. 4. User: “What’s Next?” 5. Agent: “What time is the appointment?” 6. User: “11 AM.” 7. Agent: sets the time of the appointment to 11 AM. 8. Agent: “I’m going to save this appointment.” 9. User: “OK.” 10. Agent: “We have created a new appointment!”

18. Speech Recognition Available engines Available engines IBM Viavoice IBM Viavoice Microsoft Speech SDK Microsoft Speech SDK CMU Sphinx Engine 2 and 4 CMU Sphinx Engine 2 and 4 Mode of application Mode of application Continuous speech recognition mode Continuous speech recognition mode Speaker dependent Speaker dependent Low accuracy Low accuracy Voice commands mode Voice commands mode Speaker independent Speaker independent Higher Accuracy Higher Accuracy Still susceptible to noise Still susceptible to noise

19. Managing Speech Recognition Technology Reduce noise Reduce noise Pressure-gradient/Noise-canceling Microphone Pressure-gradient/Noise-canceling Microphone Sound card/USB pod Sound card/USB pod Continuous recognition Continuous recognition Training is essential Training is essential Do not expect high accuracy Do not expect high accuracy Approach Approach Design dialog script carefully Design dialog script carefully Build in a fault tolerant mechanism Build in a fault tolerant mechanism Always have a “take me to an operator” option Always have a “take me to an operator” option Constrained uses: Constrained uses: voice command mode voice command mode voice stress analysis voice stress analysis

20. Local Experimental Environment “Smart” Room “Smart” Room Distributed sensor network Distributed sensor network Human and robotic residents Human and robotic residents Complex environment Complex environment : camera locations

21. Tracking Distributed sensor network Distributed sensor network Sensor management Sensor management Selection, reasoning across sensors Selection, reasoning across sensors Real-Time fault-tolerant approach Real-Time fault-tolerant approach Fault containment units Fault containment units Hardware, Software Hardware, Software Hierarchical Hierarchical Wrappers around a resource policy Wrappers around a resource policy Augmented by fault and context Augmented by fault and context reporting mechanisms reporting mechanisms Dynamic restructuring Dynamic restructuring Supports Supports Activity modeling Activity modeling Identification Identification Unusual event detection Unusual event detection

22. Tracking Supports Virtual Worlds

23. Immersive Virtual Worlds Provide several levels of interaction styles Provide several levels of interaction styles Client control over interaction style and level of access of remote visitors (privacy) Client control over interaction style and level of access of remote visitors (privacy) Avatar plus virtual world Avatar plus virtual world Avatar plus real world Avatar plus real world Person plus virtual world Person plus virtual world Person plus real world Person plus real world (Registered) Transparent Objects

24. Rapid Viewpoint Changes Camera context switches e.g. surveillance Camera context switches e.g. surveillance Disconcerting, difficult for user Disconcerting, difficult for user “Snap” Transitions “VR Smoothed” Transitions

25. Mixing Virtual and Real Worlds Events in real world mapped to virtual world Events in real world mapped to virtual world Temporal events synchronized to virtual world Temporal events synchronized to virtual world Transparency - look through objects Transparency - look through objects Form of access control Form of access control

26. Conclusions Starting up Starting up Conjectures on the future Conjectures on the future Empowering the elderly: they decide Empowering the elderly: they decide Social etiquette enforced through virtual worlds Social etiquette enforced through virtual worlds Client controls level of access and degree of privacy Client controls level of access and degree of privacy Built on Built on Modern social science theory Modern social science theory Existing technology base Existing technology base Client-care Sandbox model for rapid prototyping Client-care Sandbox model for rapid prototyping Particulars determined through focus groups Particulars determined through focus groups