1 Assisted Cognition Henry Kautz Don Patterson, Nan LI Oren Etzioni, Dieter Fox University of Washington Department of Computer Science & Engineering

2 Cognition in Context Can often compensate for physical disabilities by change in environment Wheelchairs Redesigned appliances Cognitive competence also depends on environment Can you cook dinner, given a dead animal, a stone knife, and set of flints?

3 Social Context The context for cognition involves both the physical and social environments Stability & organization of physical environment may reduce cognitive load Other people (e.g. a spouse) can actively assist in problem solving How can I make coffee? Which way is home?

4 The $80 Billion Question Can we build computer systems that (like a caregiver) actively assist a person with Alzheimer’s perform the tasks of day-to-day living? Enhance quality of life Prolong aging in place Lessen burden on other caretakers Depression affects 20% of Alzheimer’s patients, but 50% of Alzheimer’s caregivers Crisis in demographics – shortage of caretakers

5 The Assisted Cognition Project University of Washington Computer Science & Engineering UW Medical Center Alzheimer’s Disease Research Center (ADRC) UW Institution on Aging Outside Collaborators: Intel Research – Seattle and Jones Farm OGI/OHSU Elite Care

6 Vision Understanding human behavior from low-level sensory data Using commonsense knowledge Learning individual user models Actively offering prompts and other forms of help as needed Alerting human caregivers when necessary Computer systems that improve the independence and safety of people suffering from cognitive limitations by…

7 Example: Activity Compass Help user move between home and community Walking, riding in a car, public transport Predicts where user is going Offers simple directions Detects potential problems Is user on the wrong bus? Is user wandering?

8 Example: ADL Prompter 1. Joe enters bathroom at 9:00 am. 2. He turns on water, and picks up toothbrush. 3. Nothing happens for 30 seconds. AC system recognizes “tooth brushing” activity has stalled. 4. Prompts Joe to pick up toothpaste. Joe does so and completes task. 5. Joe leaves bathroom with water still running. AC system gently encourages Joe to go back and turn it off.

9 Common Architecture

10

11 Technical Approach GSP equipped Palm monitors location & velocity, communicates with server Dynamic Bayesian Net determines current mode of transportation Learned Markov Model predicts most likely activity path – i.e., user trajectory through time and space Each segment is a different mode of transport History, time of day, appointment calendar, bus schedules Guide user along activity path

12 Minimalist User Interface

13 User Feedback User may deviate from predicted path because System is wrong – need to update model User is in error – confused, forgetful System may ask for user for confirmation “Tap if you’re okay” Balance cost of annoying user vs. probability that user is in danger

14 Deciding When to Intervene (Horvitz 98) G = prediction that help is needed

15 Gathering Data

16 Velocity Histogram

17 Dynamic Bayesian Net M S V T1 T2 M S V T1 T2 TT+1 Mode Speed Timing BusStop Velocity BB

18 Mode Prediction

19

20 Current Work Measure accuracy of Markov Model for predicting activity path Compare other approaches Employ Relational Markov Model Less training data Increased power Planning algorithms for “error correction” E.g., once user has missed bus, find new path to achieve same goal

21 ADL Prompter General approach: build a probabilistic model of Common user goals “Plans” (complex behaviors) that achieve those goals Including failure modes How simple behaviors are sensed Run model “backwards” to interpret sensed data

22 Badge SensorDoor SensorGPS Location Get out of bed Walk to bathroom Flush Walk to bedroom Get into bed Night bathroom run Get out of bed Walk to bathroom

23 Badge SensorDoor SensorGPS Location Get out of bed Walk to kitchen Get crackers Walk to bedroom Get into bed Night snack run

24 Badge SensorDoor SensorGPS Location Night bathroom run Night pattern Night snack run Sleep

25 Timing Constraints Walk to bedroom Get into bed < 10 min Night bathroom run active [9 pm – 7 am] Night wandering violation

26 Summary: ADL Prompter Commonsense knowledge base of “significant” behaviors Hierarchically organized Probabilistic at all levels Several parallel ongoing activities possible Absolute and relative timing constraints Probabilities “tuned” by machine learning techniques for individual users Failure modes – points of possible intervention

27 Conclusions Growing research area combining AI, ubiquitous computing, and assistive technology NIST, AAAI, Ubicomp Workshops RESNA Gerontechnology Key idea: Patient and computer as a problem-solving team

28 End

29 Technical Foundations Hidden Markov models Mathematical framework for describing processes with hidden state that must be inferred from observations Hierarchical plan networks Represents how a task can be broken down into subtasks Hierarchical hidden Markov models Key to climbing food-chain!

30 Key Issue How to go from noisy and incomplete sensor measurements to A meaningful description of what a person is doing “Trying to brush teeth” “Trying to get home” A decision by the system about whether or not to intervene … in a principled and scalable manner!

31 Interventions Framework allows AC system to predict when a “failure” is likely Different failures have different costs Wandering in bedroom Wandering outside Forgetting to take medicine Forgetting to flush Must avoid:

32 Advertisement UbiCog 2002 – Workshop on Ubiquitous Computing for Cognitive Aids September 29, 2002 Gothenberg, Sweden Part of UBICOMP-2002, the major ubiquitous computing conference Some space still available, Henry Kautz

33 green – GPS readings (10 sec), yellow – location estimation (probability distribution)

34

35 Creating the User Model Training Data: 20,000 GPS readings Predicting mode 98% accuracy (10 FCV) Predicting next mode transition 97% accuracy (10 FCV)