Assisted Cognition Systems Henry Kautz Department of Computer Science

Vision Understanding human behavior from sensor data Actively prompting, warning, and advising Alerting caregivers as necessary Computer systems that improve the independence and safety of people suffering from cognitive limitations by…

Foundations Advances in sensors Wearables: acceleration, RFID, GPS, physiological measures Environmental: vision, audio, indoor localization Machine learning Integrate sensor data with commonsense knowledge

Effective Interfaces Multimodal – GUI, touch, gesture, speech, … Mobile – Portable, networked, wearable Intuitive – Easy to learn, use, maintain Adaptive Proactive

University of Rochester Center for Future Health – Interdisciplinary center for proactive healthcare technology – Researchers from Strong Medical Center, UR Electrical & Computer Engineering, UR Computer Science Laboratory for Assisted Cognition Environments (LACE) – New (2007) effort focuses on applying AI and machine learning to technology to help cope with cognitive disabilities – Ties with Intel Research and University of Washington

ADL Monitoring Automating ADL logging / assessment Long term trends Immediate help Intel Research wearable RFID reader Image Courtesy Intel Research

ACCESS Help persons with cognitive disabilities travel safely in their community and employ public transit – Huge issue for quality of life for millions of people GPS cell phone-based system – User carries phone during daily routine E.g. with job coach or family member – Automatically learns pattern of behavior Infers public transportation use – System notes breaks from ordinary routine Provides proactive help

Integrated Cueing & Sensing PEAT: handheld-based activity cueing system for persons with executive function impairment (Attention Control Systems Inc) Problem: requires frequent input from user Solution: use sensor to detect activities – Reduce user interaction – Reduce “learned dependency” – Enable context-dependent cues

© Copyright 2007 ACS Adding Sensors to PEAT Sensors to detect user’s location (GPS and floor mats) Sensors to detect which objects user is touching (RFID – Radio Frequency ID) State Estimation and Activity Recognition software infers current state and current activity State Estimator Cellphone Wireless connections GPS, Bluetooth (BT), WiFi State DB Daily Activity Planning & Cueing PEAT User Interface PEAT RFID reader bracelet WiFi Relay BT ZigBee WiFi A wireless cognitive aid with sensing, planning and cueing Pressure mats User

© Copyright 2007 ACS Example Scenarios Demonstration scenarios PEAT detects user touches cereal bowl & inhibits “Start Breakfast” cue PEAT detects user steps outside & inhibits “get mail” cue PEAT detects user steps outside without cane & generates cue to use cane

Current / Future Directions More detailed sensor data – Machine vision fused with direct sensing More detailed models of cognitive state – Affect (emotional display) recognition – Aid for improved self-awareness Interactive design/testing with various target populations – TBI – Alzheimers – Autism