1. Transforming Standard Homes Into Cognitive Prosthesis Pervasive Assistance for Cognitively Impaired People Sylvain Giroux

2. Plan Context Objectives and Approach Pervasive and mobile computing Tangible User interface From homes… Hardware level >>> networks, sensors, effectors … to smart homes … Middleware level >>> pervasive infrastructure … to smart care ! Application level >>> cognitive assistance & tele-monitoring Validation Usability and clinical studies Conclusion

3. Context People suffering of cognitive impairments in Quebec Alzheimer disease : 5.1% of people over 65 years old Head trauma : 3000 new cases each year Schizophrenia : 1% of the population In many cases, they would be able to stay at home if light assistance was provided. But healthcare resources are scarce. So relatives have to take responsibility for care. It then turns to an exhausting burden. Hence relatives and caregivers urge for help.

5. Objectives Provide adapted and personalized environmental cues to Foster the autonomy of cognitively impaired people Reduce risks and hazards Pervasive computing & Tangible user interfaces Keep ensuring cognitive assistance outside people’s home Mobile computing & Location-based services Help relatives and caregivers to stay in touch at distance with cognitively impaired people

6. From homes… Smart homes are augmented environments Heterogeneous networks Sensors networks Embedded processors in appliances, clothes, jewels… Information devices Networked communicating objects

7. DOMUS: an augmented apartment

8. … smart homes … Smart homes are augmented environments Heterogeneous networks Wireless : WiFi, Bluetooth, RFID, UWB... Wired : Ethernet, Electrical wires, X10, power line… Servers Full control over audio and video streams

9. … smart homes … Smart homes are augmented environments Sensors networks Identification and localization of objects and people Ubisense tags, UWB Smart tags (RFID)

10. … smart homes … Smart homes are augmented environments Embedded processors in devices and clothes

11. … smart homes … Smart homes are augmented environments Information devices Fixed: smart boards, Icebox… Mobile: laptop, wireless screen, PDAs…

12. … smart homes … Smart homes are augmented environments Networked communicating objects Sight Hearing Smell (not yet investigated) Touch (not yet investigated) Taste (is it possible ?)

13. From homes to smart homes

14. Smart home : the middleware Some issues investigated at DOMUS Spontaneous networking Heterogeneous networks Autonomic computing Mobile code and agents Location and context awareness Security and privacy (not yet) Some prototypes towards a pervasive infrstructure A pervasive reminder system Multi-channel delivery of services

15. A Pervasive Reminder System for Smart Homes How to localize a user from simple sensors information ? How to achieve pervasiveness? Follow-me : transparent user friendly migration of sessions How to use spontaneous networking and service discovery to build zero-configuration system ? How to cope with heterogeneity of devices, networks, and OS ? How to keep the system clean?

16. Multi-channel delivery of services On-the-fly generation of user interfaces from raw code Means to control complex interactions with a user Ready to use service delivery infrastructure

17. … to smart care Smart homes can assist cognitively impaired people foster their autonomy The whole home then becomes a true cognitive prosthesis At distance, smart homes can help caregivers to grant better care give a sense of security to residents and their relatives

18. Cognitive assistance in smart environments What is the available information ? Identification and localization of people and objects Objects involved in an activity Primitive actions performed by the resident Who the resident is ? Personalization What is the user doing ? Activity recognition How to assist the resident ? Taditional user interfaces Tangible user interfaces

19. Activity recognition I Based on plan recognition Hierarchical descriptions of tasks Epitalk, Markov models, etc. Monitoring of one activity by one person The assistance system reacts after the user “error” Lattice-based models Manage concurrent activities recognition Predict the user behaviour (and ideally anticipating his “errors”) Disambiguate activities from observed events Identify opportunities for assistance

20. Activity recognition II Based on involved objects Perkowitz et al., Mining Models of Human Activities from the Web, WWW 2004, May 17-22, 2004, New York, NY USA. …enhanced with contextual information

21. Personalization User modelling Health status, cognitive deficits to address… Tasks to monitor Relatives and caregivers network etc. Cognitive modelling based on episodic memory To know from life habits, how one usually performs an activity method usually used to achieve a task estimations on time (average time of completion…) most likely location to perform a task

22. Towards interactive environments Interactive maps 9

23. Towards interactive environments >> Show-me objects + « Follow-me » applied to objects The lamp turns off when the object is too far away The lamp turns on to highlight the searched red book

24. Prototype I Pervasive cognitive assistance In collaboration with France Telecom Jean-Pierre Savary, Projet Synapse Patent pending

25. Cognitive Assistance What kind of cognitive deficits can we address ? Initiation : inactive periods whereas the person is supposed to perform actions Memory : difficulties to remember the activity to perform, the steps, the locations of tools and materials involved Planning : difficulties to perform an appropriate sequence of actions in the right order to achieve a goal Attention : shifts of attention from the activity under progress to a stimulus causing interference How to interact with the resident ? Traditional GUI vs Tangible user interfaces : Transforming the whole house into a cognitive prosthesis Who initiate the interactions? Resident vs Assistant Telemonitoring Asynchronous vs Synchronous

26. Prototype II Using a PDA-based personal agenda as a tool for cognitive assistance tele-monitoring medical assessment coordination of caregivers

27. Architecture

28. Using a personal agenda for cognitive assistance: the user perspective Reminding activities of daily living (ADLs) Confirming a given ADL is performed

29. Using a personal agenda for medical assessment: the user perspective Monitoring side-effects of medication

30. Using a personal agenda for medical assessment: the user perspective Asking for help from caregivers

31. Using a personal agenda for remote monitoring: the caregiver perspective Monitoring many patients Monitoring one patient activities Monitoring one activity of a patient & caregiver coordination

32. Activities of daily living: the caregiver perspective Specifying activities of daily living (ADLs) Assigning to patients ADLs to monitor

33. Usability and clinical studies Fernand-Séguin Research Center (June 2006) L-H Lafontaine psychiatric hospital, Montreal Gathering ecological data Monitoring ADLs 3 schizophrens + 3 caregivers (occupational therapists or social workers) DOMUS Laboratory (September 2006) Pervasive cognitive assistance Intellectual disabilities Center for rehabilitation Estrie (November 2006) Gathering ecological data Monitoring ADLs Assistance outside the home using location-based services Head trauma

34. Conclusion Pervasive computing and tangible user interfaces can help to transform home into smart homes adapted to cognitively impaired people Going beyond the usual view of computing as “PC-based” Pervasive computing enables a seamless integration of assistance in residents’ everyday life Going beyond traditional human computer interfaces TUI helps to turn the whole house into a cognitive prosthesis Smart environments and mobile applications can Foster people’s autonomy Lead to smarter care

35. Future works Radical changes are needed transform and adapt the environment for cognitively impaired people as our society did in the early 80s for physically disabled people Change the healthcare approach by providing continuous feedback to people to foster prevention and better life habits Cognitive assistance and telemonitoring can be the core part of the solution

36. Initiation deficits Initiation deficits leads to inactive periods whereas the person is supposed to perform actions Wandering for a long time could be attributed to an initiation deficit >> Prompt the resident

37. Planning deficits Difficulties to perform an appropriate sequence of actions in the rightorder to achieve a goal. >> Prepare_coffee = {take_milk, take_cup…} >> Show to the user where to perform the next action

38. Attention deficits Shifts of attention from the activity under progress to a stimulus causing interference The current activity may be forgotten and never completed >> Remind the activity under progress to the resident

39. Our team Researchers from the Faculties of science, engineering, and administration Sylvain Giroux, Ph. D. in Computer science Hélène Pigot, Ph. D. in Computer science and B. in occupational therapy André Mayers, Ph. D. in Computer science and M. inpsychology Philippe Mabilleau, Ph.D. in engineering Claude Caron (geo-business) Analyst Francis Bouchard Students 6 Ph. D. students 12 M. Sc. students 6 B.Sc. students, 2 international trainees (M. Sc. level) Some collaborations CRE, Centre de réadaptation Estrie Centre de recherches Fernand Séguin, Computer science Université Joseph Fourier, Grenoble France Telecom Ariane Controls, Canada

40. Thank you for your attention http://domus.usherbrooke.ca/