1 Assisted Cognition Henry Kautz, Oren Etzioni, & Dieter Fox University of Washington Department of Computer Science & Engineering

2 An Epidemic of Alzheimer’s 4.6 million people in the US with Alzheimer’s 16 million people by 2050 Today costs $100 year for care Additional $61 billion in lost productivity from family members $ ½ Trillion total cost by 2050! Projections even worse for Japan and Europe

3

4 Cognition in Context Can often compensate for physical disabilities by change in environment Wheelchairs Redesigned appliances Cognitive competence also depends on environment Can you cook dinner, given a dead animal, a stone knife, and set of flints?

5 Problem Caregiver burnout ½ of all family caregivers suffer depression “The 36 Hour Day” Far too few professional caregivers to provide constant 1-on-1 help in institutional settings Already a nationwide shortage of good staff

6 Vision Understanding human behavior from low-level sensory data Using commonsense knowledge Learning individual user models Actively offering prompts and other forms of help as needed Alerting human caregivers when necessary Computer systems that improve the independence and safety of people suffering from cognitive limitations by…

7 Data Interpretation Food Chain Movement IntentionsBehaviorInterventions

8 Model-Based Interpretation General approach: build a probabilistic model of –Common user goals –Plans (complex behaviors) that achieve those goals Feasibility constraints Temporal constraints Failure (abnormality) modes – How simple behaviors are sensed Run model “backwards” to interpret sensed data

9 Activity Compass Zero-configuration personal guidance system Learns model of user’s travel on foot, by public transit, by bike, by car Predicts user’s next destination, offers proactive help if lost or late Integrates user data with external constraints Maps, bus schedules, calendars, … EM approach to clustering & segmenting data The Activity CompassThe Activity Compass Don Patterson, Oren Etzioni, and Henry Kautz (2003)

10 Scenario I Joe gets off the bus on the way to the community center. He can’t remember which way to walk. He consults his Activity Compass. It has predicted his destination based on past behavior, and guides him to the community center.

11 Scenario II Joe has a regular physical therapist’s appointment at 2 pm. Compass alerts Joe that he will need to leave home in 10 minutes to catch the usual bus there. Joe fails to leave in time. Compass asks whether goal of going to physical therapist is valid. If answer is affirmative, Compass creates alternative plan to get Joe there as soon as possible.

12 Minimalist User Interface

13 Data Source: Elite Care Residences Elite Care is the realization of our dream to provide a fundamentally different approach to assisted living. Extended Family Residences promote a family lifestyle of close staff-resident relationships, meaningful community- building activities, and physical and mental achievement while providing assistance when needed… Technology helps us maintain our family environment by allowing us to the focus on residents… - Lydia Lundberg & Bill Reed

14 Activity of daily living monitor & prompter Foundations of Assisted Cognition Systems.Foundations of Assisted Cognition Systems. Kautz, Etzioni, Fox, Weld, and Shastri, 2003

15 Recognizing unexpected events using online model selection User errors, abnormal behavior Select model that maximizes likelihood of data: Generic model User-specific model Corrupt (impaired) user model Neurologically-plausible corruptions Repetition Substitution Stalling Fox, Kautz, & Shastri (forthcoming) fill kettle put kettle on stove fill kettle put kettle on stove put kettle in closet

16 Technical Foundations Hidden Markov models –Mathematical framework for describing processes with hidden state that must be inferred from observations Hierarchical plan networks –Represents how a task can be broken down into subtasks Hierarchical hidden Markov models* –Key to climbing food-chain! * Precisely speaking: factorial hierarchical hidden semi-Markov models

17 General Architecture Ubiquitous computing infrastructure Sensors – position, motion, sound, vision Output – speech, graphics, robots Portable wireless computing devices