1. Spatio-Temporal and Context Reasoning in Smart Homes Sook-Ling (Linda) Chua Stephen Marsland, Hans W. Guesgen School of Engineering and Advanced Technology Massey University, New Zealand COSIT 2009

2. T The world is aging - have we noticed? Source: United Nations (2007) h e S i t u a t i o n...

3. The populations of the world are aging Source: United Nations (2007) T h e S i t u a t i o n...

4. The populations of the world are aging Source: United Nations (2007) T h e S i t u a t i o n...

5. The populations of the world are aging Source: United Nations (2007) T < 10% > 25% h e S i t u a t i o n...

6. T People choose to stay in their own homes as long as possible and remain independent h e S i t u a t i o n...

7. T Aging Physical disabilityCognitive impairment diminished sense and touch slower ability to react poor vision, hearing problems memory problems leads to People choose to stay in their own homes as long as possible and remain independent h e S i t u a t i o n...

8. Supporting inhabitants daily activities T h e S i t u a t i o n... Smart Homes Figure extracted from: http://www.dreamhomesmagazine.com /

9. T h e S i t u a t i o n... To react intelligently, the smart home needs to: (1) recognise inhabitants behaviour (2) perform reasoning spatio-temporal information contextual information

10. sensor output Tokens B Figure extracted from: http://www.dreamhomesmagazine.com / The Smart Home e h a v i o u r R e c o g n i t i o n

11. Tokens The direct representation of current sensor states being triggered E.g. of a sequence of tokens from the sensors Date ActivationTime ActivationRoomObject TypeSensor State 16/6/200818:05:23Living roomTelevisionOff 16/6/200818:08:19Living roomCurtainClosed 16/6/200818:09:48KitchenLightOn 16/6/200818:10:35KitchenCupboardOpen 16/6/200818:25:06KitchenFridgeOpen 16/6/200819:00:02LaundryWashing Machine On B e h a v i o u r R e c o g n i t i o n

12. Tokens The direct representation of current sensor states being triggered E.g. of a sequence of tokens from the sensors Date ActivationTime ActivationRoomObject TypeSensor State 16/6/200818:05:23Living roomTelevisionOff 16/6/200818:08:19Living roomCurtainClosed 16/6/200818:09:48KitchenLightOn 16/6/200818:10:35KitchenCupboardOpen 16/6/200818:25:06KitchenFridgeOpen 16/6/200819:00:02LaundryWashing Machine On B e h a v i o u r R e c o g n i t i o n

13. 19th Jan 2009 18:03:16 19th Jan 2009 18:07:56 19th Jan 2009 18:20:27 19th Jan 2009 18:33:44 19th Jan 2009 18:50:12 19th Jan 2009 19:01:08 19th Jan 2009 19:37:21 19th Jan 2009 19:41:26............ … Kitchen Dining/Living Room Bedroom Laundry Office/Study Figure extracted from: The Aware Home, 2002 B e h a v i o u r R e c o g n i t i o n

14. 19th Jan 2009 18:03:16 19th Jan 2009 18:07:56 19th Jan 2009 18:20:27 19th Jan 2009 18:33:44 19th Jan 2009 18:50:12 19th Jan 2009 19:01:08 19th Jan 2009 19:37:21 19th Jan 2009 19:41:26............ … Kitchen Dining/Living Room Bedroom Laundry Office/Study Figure extracted from: The Aware Home, 2002 B e h a v i o u r R e c o g n i t i o n

15. 19th Jan 2009 18:03:16 19th Jan 2009 18:07:56 19th Jan 2009 18:20:27 19th Jan 2009 18:33:44 19th Jan 2009 18:50:12 19th Jan 2009 19:01:08 19th Jan 2009 19:37:21 19th Jan 2009 19:41:26............ … Kitchen Dining/Living Room Bedroom Laundry Office/Study Q: How do we recognise behaviours? Figure extracted from: The Aware Home, 2002 B e h a v i o u r R e c o g n i t i o n

16. B e h a v i o u r R e c o g n i t i o n Kitchen Dining/Living Room Bedroom Laundry Office/Study 19th Jan 2009 18:03:16 19th Jan 2009 18:07:56 19th Jan 2009 18:20:27 19th Jan 2009 18:33:44 19th Jan 2009 18:50:12 19th Jan 2009 19:01:08 19th Jan 2009 19:37:21............ Figure extracted from: The Aware Home, 2002 Challenges: (a) Exact activities are not directly observed, only the sensor observations...

17. B e h a v i o u r R e c o g n i t i o n Kitchen Dining/Living Room Bedroom Laundry Office/Study 19th Jan 2009 18:03:16 19th Jan 2009 18:07:56 19th Jan 2009 18:20:27 19th Jan 2009 18:33:44 19th Jan 2009 18:50:12 19th Jan 2009 19:01:08 19th Jan 2009 19:37:21............... Figure extracted from: The Aware Home, 2002 Challenges: (a) Exact activities are not directly observed, only the sensor observations

18. B e h a v i o u r R e c o g n i t i o n Kitchen Dining/Living Room Bedroom Laundry Office/Study 19th Jan 2009 18:03:16 19th Jan 2009 18:07:56 19th Jan 2009 18:20:27 19th Jan 2009 18:33:44 19th Jan 2009 18:50:12 19th Jan 2009 19:01:08 19th Jan 2009 19:37:21............ Figure extracted from: The Aware Home, 2002 Challenges: (a) Exact activities are not directly observed, only the sensor observations ?...

19. B e h a v i o u r R e c o g n i t i o n Challenges: (b) Same sensor activations will be involved in multiple behaviours

20. B e h a v i o u r R e c o g n i t i o n Challenges: (b) Same sensor activations will be involved in multiple behaviours Figure extracted from: www.rebecca-waring.com, www.cyh.com, www.chow.com

21. B e h a v i o u r R e c o g n i t i o n Challenges: (c) No. of observations can vary between activities Making breakfastMaking dinner Fridge Toaster Cupboard Stove Microwave Oven Tap Drawer

22. B e h a v i o u r R e c o g n i t i o n Challenges: (d) Behaviours are rarely identical on each use E.g. Making a cup of tea With / withoutMilk / water first?How long? components can be present/absent the order of individual components happen can change length of time each piece takes can change

23. B e h a v i o u r R e c o g n i t i o n Challenges: (d) Behaviours are rarely identical on each use E.g. Making a cup of tea With / withoutMilk / water first?How long? components can be present/absent the order of individual components happen can change length of time each piece takes can change Stochastic Approach

24. B e h a v i o u r R e c o g n i t i o n The Hidden Markov Model (HMM) probabilistic graphical model Source: Rabiner, L. (1989) uses probability distributions to determine the states for a sequence of observations over time

25. B e h a v i o u r R e c o g n i t i o n The Hidden Markov Model (HMM) probabilistic graphical model Source: Rabiner, L. (1989) Observations We know this.. uses probability distributions to determine the states for a sequence of observations over time

26. B e h a v i o u r R e c o g n i t i o n The Hidden Markov Model (HMM) probabilistic graphical model Source: Rabiner, L. (1989) Observations We know this.. …… States But, not this uses probability distributions to determine the states for a sequence of observations over time

27. … … States Observations Markov property: The probability of transition to a state (S t+1 ) depends only on the current state (S t ) [represented by solid line] The observation at O t depends only on the state S t at that time slice [represented by dashed line] B e h a v i o u r R e c o g n i t i o n … The Hidden Markov Model (HMM) Source: Rabiner, L. (1989)

28. … … States Observations Markov property: The probability of transition to a state (S t+1 ) depends only on the current state (S t ) [represented by solid line] The observation at O t depends only on the state S t at that time slice [represented by dashed line] B e h a v i o u r R e c o g n i t i o n … The Hidden Markov Model (HMM) Source: Rabiner, L. (1989)

29. … … States Observations Markov property: The probability of transition to a state (S t+1 ) depends only on the current state (S t ) [represented by solid line] The observation at O t depends only on the state S t at that time slice [represented by dashed line] B e h a v i o u r R e c o g n i t i o n … The Hidden Markov Model (HMM) Source: Rabiner, L. (1989)

30. B e h a v i o u r R e c o g n i t i o n Kitchen Dining/Living Room Bedroom Laundry Office/Study 19th Jan 2009 18:03:16 19th Jan 2009 18:07:56 19th Jan 2009 18:20:27 19th Jan 2009 18:33:44 19th Jan 2009 18:50:12 19th Jan 2009 19:01:08............

31. B e h a v i o u r R e c o g n i t i o n Kitchen Dining/Living Room Bedroom Laundry Office/Study 19th Jan 2009 18:03:16 19th Jan 2009 18:07:56 19th Jan 2009 18:20:27 19th Jan 2009 18:33:44 19th Jan 2009 18:50:12 19th Jan 2009 19:01:08............ Observations

32. B e h a v i o u r R e c o g n i t i o n Kitchen Dining/Living Room Bedroom Laundry Office/Study 19th Jan 2009 18:03:16 19th Jan 2009 18:07:56 19th Jan 2009 18:20:27 19th Jan 2009 18:33:44 19th Jan 2009 18:50:12 19th Jan 2009 19:01:08............ ? Observations Cupboard Coffee Machine Fridge States

33. To use HMM to recognise behaviours: (1) Segmentation break the token sequence into appropriate pieces that represent individual behaviours... Observations start end B e h a v i o u r R e c o g n i t i o n

34. ... (2) Classification identify the behaviours using the HMM Observations To use HMM to recognise behaviours: Behaviour ABehaviour B B e h a v i o u r R e c o g n i t i o n

35. Behaviour Recognition using HMM Our approach: Use a set of HMMs that each recognise different behaviours Making coffeeShowering These HMMs will compete to explain the current observations Model selection is based on maximum likelihood Making lunch... B e h a v i o u r R e c o g n i t i o n Source: Chua, Marsland and Guesgen (2009)

36. Experiment: Competition between HMMs Datasets MIT PlaceLab Designed a set of simply installed state-change sensors that were placed in two different apartments with real people living in them Source: Tapia (2004) B e h a v i o u r R e c o g n i t i o n

37. Experiment: Competition between HMMs Datasets The subjects kept a record of their activities that form a set of annotations for the data Ground-truth segmentation of the dataset We used the dataset from the first subject 77 sensors collected for 16 consecutive days B e h a v i o u r R e c o g n i t i o n

38. Datasets Activities take place in one room (kitchen) Location of the sensors is known a priori Behaviours: Prepare breakfast (toaster) Prepare breakfast (cereal) Prepare beverage Prepare lunch Do the laundry Experiment: Competition between HMMs B e h a v i o u r R e c o g n i t i o n

39. Based on 727 observations (using 11 days testing and 5 days training set) B e h a v i o u r R e c o g n i t i o n

40. Based on 727 observations (using 11 days testing and 5 days training set) B e h a v i o u r R e c o g n i t i o n

41. Based on 727 observations (using 11 days testing and 5 days training set) B e h a v i o u r R e c o g n i t i o n

42. Experimental Results Method works effectively performs segmentation and detects changes of activities B e h a v i o u r R e c o g n i t i o n MicrowaveFridge Coffee Machine Drawer... observation FridgeCupboard

43. Experimental Results Method works effectively performs segmentation and detects changes of activities B Preparing lunch Preparing a beverage MicrowaveFridge Coffee Machine Drawer... observation FridgeCupboard e h a v i o u r R e c o g n i t i o n

44. Discussion Lack of spatio-temporal information Misclassification: The end of one behaviour contains observations that should be the start of the next MicrowaveCupboardFridge Coffee Machine Drawer … observation Preparing lunch Preparing a beverage B e h a v i o u r R e c o g n i t i o n

45. Discussion Lack of spatio-temporal information Misclassification: The end of one behaviour contains observations that should be the start of the next MicrowaveCupboardFridge Coffee Machine Drawer … observation Preparing lunch Preparing a beverage B e h a v i o u r R e c o g n i t i o n

46. Discussion Lack of spatio-temporal information Misclassification: The end of one behaviour contains observations that should be the start of the next MicrowaveCupboardFridge Coffee Machine Drawer … observation Preparing lunch Preparing a beverage B e h a v i o u r R e c o g n i t i o n

47. MicrowaveCupboardFridge Coffee Machine Drawer... observation Preparing lunch Preparing a beverage Preparing lunch Preparing a beverage Discussion Lack of spatio-temporal information Misclassification: The end of one behaviour contains observations that should be the start of the next B e h a v i o u r R e c o g n i t i o n

48. A: Augment current algorithm to include spatio-temporal information Q: How to reduce misclassification? Discussion Lack of spatio-temporal information B e h a v i o u r R e c o g n i t i o n

49. NOT directly interested in the exact coordinates Spatial information (Where?) S So, what are we interested in? Room location e.g. Figures extracted from: www.istockphoto.com, www.clubjam.jammag.com, www.nancilea.blogspot.com p a t i o - t e m p o r a l

50. Spatial information (Where?) S p a t i o - t e m p o r a l Current study used very basic spatial information (just the kitchen!) In the future,...

51. B e d r o o mK i t c h e n B a t h r o o m D i n i n g R o o m L i v i n g R o o m Preparing a beverage Cooking Showering Grooming Computing Sleeping Washing dishes Eating Reading Watching TV Exercising Sitting around fireplace Resting S p a t i o - t e m p o r a l

52. B e d r o o mK i t c h e n B a t h r o o m D i n i n g R o o m L i v i n g R o o m Preparing a beverage Cooking Showering Grooming Computing Sleeping Washing dishes Eating Reading Watching TV Exercising Sitting around fireplace Resting S p a t i o - t e m p o r a l

53. S p a t i o - t e m p o r a l... observation MicrowaveCupboardFridgeDrawer Preparing lunch Fridge Coffee Machine Drawer Preparing a beverage FanShower Showering KitchenBathroomKitchen Spatial information (Where?)

54. S p a t i o - t e m p o r a l Spatial information (Where?)... is this sufficient for reasoning? WITHOUT temporal, the system cannot differentiate: Bathroom 3 am 8 am Vs. Figure extracted from: http://hazard.com/graphics

55. S p a t i o - t e m p o r a l Temporal information When does a behaviour occur? Source: Guesgen and Marsland (2009) How long does behaviour take? How often does behaviour occur? Mapping to time scale Duration Frequency e.g. Mary vacuums every Sunday e.g. Microwave used for a dangerously long time e.g. Peter showers 3 times a day

56. S p a t i o - t e m p o r a l Temporal information (When) 3.03 pm weekends vs. weekdays winter vs. summer ½ hour after shower having breakfast 2 hours before meeting am vs. pm Absolute time Relative time........

57. S p a t i o - t e m p o r a l Time scales Yearly (e.g. Christmas, New Year, Easter, etc.) Weekly (e.g. vacuuming, visit from health worker, etc.) Daily (e.g. showering, eating, etc.)

58. S p a t i o - t e m p o r a l Temporal information (a) segment the behaviours (b) generate a sequence of behavioural patterns tells us when, for how long and how frequent behaviour occurs Source: Guesgen and Marsland (2009)

59. S p a t i o - t e m p o r a l a mp m E v e n i n g N i g h t Preparing a beverage Cooking Showering GroomingComputing Washing dishes Eating Reading Watching TV Exercising Resting Sleeping Watching TV Cooking Washing dishes Computing Preparing a beverage Sitting around fireplace Exercising Reading Eating Preparing a beverage Showering

60. S p a t i o - t e m p o r a l a mp m E v e n i n g N i g h t Preparing a beverage Cooking Showering GroomingComputing Washing dishes Eating Reading Watching TV Exercising Resting Sleeping Watching TV Cooking Washing dishes Computing Preparing a beverage Sitting around fireplace Exercising Reading Eating Preparing a beverage Showering

61. S p a t i o - t e m p o r a l a mp m E v e n i n g N i g h t Preparing a beverage Cooking Showering GroomingComputing Washing dishes Eating Reading Watching TV Exercising Resting Sleeping Watching TV Cooking Washing dishes Computing Preparing a beverage Sitting around fireplace Exercising Reading Eating Preparing a beverage Showering

62. S p a t i o - t e m p o r a l Temporal information (a) segment the behaviours (b) generate a sequence of behavioural patterns tells us when, for how long and how frequent behaviour occurs

63. S p a t i o - t e m p o r a l Time Cooking Reading Resting Preparing a beverage Washing dishes Eating Watching TV Computing Exercising Cooking Watching TV Behaviour

64. Time Cooking Reading Resting Preparing a beverage Washing dishes Eating Watching TV Computing Exercising Cooking Watching TV...... Behaviour S p a t i o - t e m p o r a l Dining Room Eating Reading Kitchen Preparing a beverage Cooking Washing dishes Living Room Watching TV Exercising Sitting around fireplace Resting Space Competition among HMMs Bedroom Grooming Computing Sleeping

65. Time Cooking Reading Resting Preparing a beverage Washing dishes Eating Watching TV Computing Exercising Cooking Watching TV Behaviour S p a t i o - t e m p o r a l Space Competition among HMMs...... Dining Room Eating Reading Kitchen Preparing a beverage Cooking Washing dishes Living Room Watching TV Exercising Sitting around fireplace Resting Bedroom Grooming Computing Sleeping

66. Time Cooking Reading Resting Preparing a beverage Washing dishes Eating Watching TV Computing Exercising Cooking Watching TV Behaviour S p a t i o - t e m p o r a l Space Competition among HMMs...... Dining Room Eating Reading Kitchen Preparing a beverage Cooking Washing dishes Living Room Watching TV Exercising Sitting around fireplace Resting Bedroom Grooming Computing Sleeping

67. Time Cooking Reading Resting Preparing a beverage Washing dishes Eating Watching TV Computing Exercising Resting Watching TV Behaviour S p a t i o - t e m p o r a l Space Competition among HMMs...... Dining Room Eating Reading Kitchen Preparing a beverage Cooking Washing dishes Living Room Watching TV Exercising Sitting around fireplace Resting Bedroom Grooming Computing Sleeping

68. What happens if the person is late one day and makes lunch at 3 pm? The system may make mistakes, particularly with time! Fuzzy logic system S p a t i o - t e m p o r a l

69. C How was the current situation is reached ? Contextual information What else is happening ? What is the state of the environment ?... needs to be considered! o n t e x t u a l R e a s o n i n g

70. C... is this normal? John is boiling water in the middle of the night o n t e x t u a l R e a s o n i n g

71. C John is boiling water in the middle of the night Spatial: Kitchen Temporal: Middle of the night Is the information sufficient for reasoning? o n t e x t u a l R e a s o n i n g

72. C John is boiling water in the middle of the night after watching late night movie Contextual information Spatial: Living room Kitchen Temporal: Middle of the night and is Saturday... he stays up longer !!! o n t e x t u a l R e a s o n i n g

73. C Competition between HMMs a possible mechanism for behaviour recognition and segmentation Spatio-temporal and context awareness play an important role in interpreting behaviour o n c l u s i o n

74. A Stephen Marsland, Hans Guesgen Massey University Smart Environment (MUSE) members School of Engineering and Advanced Technology (SEAT) Massey University c k n o w l e d g e m e n t s

75. Thank you! (Merçi!) F i n a l l y...