Understanding Human Behavior from Sensor Data Henry Kautz University of Washington

Trends

Growing Ubiquitous Sensing Infrastructure GPS Wi-Fi localization RFID tags Wearable sensors

Advances in Artificial Intelligence Graphical models Particle filtering Belief propagation Statistical relational learning

Crisis in Caring for the Cognitively Disabled Epidemic of Alzheimer’s Community integration of 7.5 million citizens with MR 100,000 @ year disabled by TBI Post-traumatic stress syndrome Caregiver burnout

...An Opportunity Create methods for modeling and interpreting human behavior from sensor data In order to develop assistive technologies to support independent living by people with cognitive disabilities Help people perform activities of daily living Monitor behavior to prevent health crises

Outline Learning and reasoning about transportation routines ACCESS personal guidance system Understanding activities of daily living CARE monitoring system Further directions

Assisted Cognition in Community, Employment, & Social Settings ACCESS Assisted Cognition in Community, Employment, & Social Settings

Motivation: Community Access for the Cognitively Disabled

The Problem Using public transit is cognitively challenging Learning bus routes and numbers Transfers Recovering from mistakes Point to point shuttle service impractical Slow Expensive Current GPS units hard to use Require extensive user input Error-prone near buildings, inside buses No help with transfers, timing

Goal A personal guidance system that Requires no explicit programming by user or caregiver Proactively assists user in completing transportation plans Recognizes user errors, and helps user recover

Technical Problem Given a data stream from a wearable GPS unit... Infer the user’s location and mode of transportation (foot, car, bus, bike, ...) Predict the user’s destination Detect user errors

GPS Receivers GeoStats GPS logger Today the GPS receivers become much smaller and cheaper than a few years ago. GeoStats GPS logger Data capacity: 3 weeks @ 0.5 second frequency Battery life: 72 hours Nokia 6600 Cell Phone Java Bluetooth GPS unit Internet connectivity for off-board processing Battery life: 8 hours

GIS Data Street map Bus routes & stops Business / residential areas Census 2000 Bus routes & stops Seattle Metro Business / residential areas MapPoint The second related system is …. The kernel of such systems is the database about geographic information. GIS technology has been used for tens of years and today there are a large number of different kinds of geographic data available. In this project we rely on two kinds of geographic data. The first data is the street map. The left picture shows…. The second is the bus route and bus stop data. The picture on the right…

Challenges of GPS Data Many dead zones in urban areas Sparse measurements inside cars and buses Systematic error  10 meters Multi-path propagation Mapping errors When we talk about GPS-based location tracking, many people will expect it is trivial.

Representation Map is a directed graph G=(V,E) Location: Movement: Edge (block) Distance along edge Actual GPS reading Movement: Mode { foot, car, bus, indoors } influences velocity Change edges at intersections Change mode at bus stops, parked car, buildings Tracking (filtering): Given some prior estimate, Update position & mode according to motion model Correct according to next GPS reading So far, I have described the general picture of our system and some applications. In the rest of the presentation, I will focus on the technical approach of probabilistic reasoning. There are three key things in our system. For the .., we use…. Because of the time issue, I cannot discuss those techniques in details. So I will only give you a very rough picture for each of them and show the results of our experiments.

Motion Model for Mode

Dynamic Bayesian Network I mk-1 mk Transportation mode xk-1 xk Edge, velocity, position qk-1 qk Data (edge) association zk-1 zk GPS reading Time k-1 Time k

Rao-Blackwellised Particle Filtering Evolve approximation to state distribution using samples (particles) Supports multi-modal distributions and discrete variables (mode, edge) Rao-Blackwellisation Particles include distributions over variables Each particle is a Kalman filter (Gaussian along edge) Improved accuracy with fewer particles After we have the DBN, one central task is to estimate the …….. This is called inference. For example, in our DBN, the inference means to infer the current goal, trip segment and location and velocities from the GPS measurements up to now. Many ways to do inference. Particle filter is one of them.

Learning to Predict User Prior knowledge – general constraints on transportation use Vehicle speed range Bus stops Learning – specialize model to particular user 30 days GPS readings of one user, logged every second Unlabeled data Learn edge and mode transition parameters using expectation-maximization (EM)

Mode & Location Tracking Measurements Projections Bus mode Car mode Foot mode Green Red Blue Explain why it switches from foot to bus, not car: because of the bus stops If people ask why no car particle at all (there are car particles, just their weights are too small that can not be displayed).

How to improve the model’s predictive power? Predictive Accuracy How to improve the model’s predictive power? Probability of correctly predicting the future 5 blocks 50% accuracy City Blocks

Transportation Routines Home A B Workplace Goal: intended destination Workplace, home, friends, restaurants, … Trip segments: <start, end, mode> Home to Bus stop A on Foot Bus stop A to Bus stop B on Bus Bus stop B to workplace on Foot Now the system could infer location and transportation modes, but wee still want to know more. We first show an example. Goal In order to get to the goal, the user may switch transportation mode. Thus we introduce the concept of trip segments.

Dynamic Bayesian Net II gk-1 gk Goal tk-1 tk Trip segment mk-1 mk Transportation mode xk-1 xk Edge, velocity, position qk-1 qk Data (edge) association zk-1 zk GPS reading Time k-1 Time k

Unsupervised Hierarchical Learning Use previous model to infer: Goals - locations where user stays for long periods of time Transition points - locations with high mode transition probability Trip segments – paths connecting transition points or goals Perform EM learning on the hierarchical model Learn transition parameters: Between goals Between trip segments, given the goal Between edges & modes, given the trip segment

High Probability Trip Segments Conditioned on Goal Goal = Workplace Goal = Home We visualize the learned street transitions. Left picture, the goal is the workplace. Conditioned on the goal, we draw the trip segment and the very likely street transitions in the trip segments. we can clearly see the different trajectories to the workplace in different transportations Right picture, similar, different destination : bus : car : foot

Predicting Goal and Path Predicted goal Predicted path

Improvement in Predictive Accuracy 45 blocks 50% accuracy

Detecting User Errors Learned model represents typical correct behavior Model is a poor fit to user errors We can use this fact to detect errors! Cognitive Mode Normal: model functions as before Error: switch in prior (untrained) parameters for mode and edge transition After we learn the model, detect potential errors.

Dynamic Bayesian Net III zk-1 zk xk-1 xk qk-1 qk Time k-1 Time k mk-1 mk tk-1 tk gk-1 gk ck-1 ck Cognitive mode { normal, error } Goal Trip segment Transportation mode Edge, velocity, position Data (edge) association GPS reading

Goal Clamping The user’s goal may be explicitly known Ask the user to confirm highest-probability goal Appointment calendar Incorporating such information clamps the goal Distinguishes novelty from errors

Detecting User Errors Untrained Trained Clamped In this episode, the person takes the bus home but missed the usual bus stop. The black disk is the goal and the cross mark is where the system think the person should get off the bus. The probability of normal and abnormal activities are represented using a bar here. The black part is the probability of being normal while the red part is the probability of being abnormal. At here, the person missed the bus stop and the probability of abnormal activity jump quickly. Untrained Trained Clamped

ACCESS Prototype Cell phone with GPS, camera, high-speed internet access Prompts when it infers that user is…

ACCESS Prototype Cell phone with GPS, camera, high-speed internet access Prompts when it infers that user is… About to begin a transportation plan Confirm destination? Here is your route!

ACCESS Prototype Cell phone with GPS, camera, high-speed internet access Prompts when it infers that user is… About to change mode This is your bus! Your stop is next!

ACCESS Prototype Cell phone with GPS, camera, high-speed internet access Prompts when it infers that user is… Making an error You missed your stop! Here is how to get back on track …

ACCESS Prototype Cell phone with GPS, camera, high-speed internet access Prompts when it infers that user is… Visiting a new destination Please take a picture!

Status Medical partnerships Extension to indoor navigation Funding by National Institute of Disability & Rehabilitation Research (NIDRR) Partnership with UW Center for Technology and Disability Studies User & caregiver needs studies (TBI & MR) Data collection by job coaches Extension to indoor navigation Hospitals, nursing homes, assisted care communities Wi-Fi localization Multi-modal interface Speech, graphics, text Guidance strategies Proactive / Just in time Coordinates / Landmarks WOZ design study

WOZ

Papers Patterson, Liao, Fox, & Kautz, UBICOMP 2003 Inferring High Level Behavior from Low Level Sensors Patterson et al, UBICOMP-2004 Opportunity Knocks: a System to Provide Cognitive Assistance with Transportation Services Liao, Fox, & Kautz, AAAI 2004 (Best Paper) Learning and Inferring Transportation Routines

Cognitive Assistance in Real-world Environments CARE Cognitive Assistance in Real-world Environments

Goal A home monitoring system that Assists user in performing activities of daily living Tracks activities, and provides prompts and warnings as needed Can be deployed in an ordinary home Does not require the user to learn a different way to perform the activities – the system adapts, not the user

Short-Term Application Accurate, automated ADL logs Changes in routine often precursor to illness, accidents Human monitoring intrusive & inaccurate Image Courtesy Intel Research

Technical Requirements Sensor hardware that can be practically deployed in a ordinary home Methods for activity tracking from sensor data Methods for automated prompting that consider Probability of user errors Probability of system errors Cost / benefit tradeoffs

Object-Based Activity Recognition Activities of daily living involve the manipulation of many physical objects Kitchen: stove, pans, dishes, … Bathroom: toothbrush, shampoo, towel, … Bedroom: linen, dresser, clock, clothing, … We can recognize activities from a time-sequence of object touches

Sensing Object Manipulation RFID: Radio-frequency identification tags Small Semi-passive Durable Cheap Near future: use products’ own tags

Wearable RFID Readers Designed by Intel Research Seattle Will be shared with other Intel partners later this year 13.56MHz reader, radio, power supply, antenna 12 inch range, 12-150 hour lifetime

Experiment: Morning Activities 10 days of data from the morning routine in an experimenter’s home 61 tagged objects 11 activities Often interleaved and interrupted Many shared objects Use bathroom Make coffee Set table Make oatmeal Make tea Eat breakfast Make eggs Use telephone Clear table Prepare OJ Take out trash

Methodology Goal: simplest model that can robustly track activities Comparison Hidden Markov Model Dynamic Bayesian Network with aggregate features DBN with aggregation and abstraction smoothing

Hidden Markov Model Trained on labeled data 10-fold cross validation 88% accuracy 9.4 errors per 20 minute episode

Cause of Errors Observations were types of objects Spoon, plate, fork … Typical errors: confusion between activities Using one object repeatedly Using different objects of same type Critical distinction in many ADL’s Eating versus setting table Dressing versus putting away laundry

Aggregate Features HMM with individual object observations fails No generalization! Solution: add aggregate variables Bit-vector maintains history of objects touched Aggregate distribution nodes sum number of distinct instances Aggregate nodes treated as pseudo-observations when an activity transitions DBN encoding avoids explosion of HMM

DBN with Aggregation Average number of errors reduced from 9.4 to 6.5 (31%) Deterministic nodes add minimal computational overhead

Improving Robustness Both HMM and DBN fail if novel (but reasonable) objects are used Solution: smooth parameters over abstraction hierarchy of object types

Abstraction Smoothing Methodology: Train on 10 days data Test where one activity substitutes one object Change in error rate: Without smoothing: 26% increase With smoothing: 1% increase

Experiment: ADL Form Filling Tagged real home with 108 tags 14 subjects each performed 12 of 14 ADLs in arbitrary order Used glove-based reader Given trace, recreate activities

Results: Detecting ADLs Activity Prior Work SHARP Personal Appearance 92/92 Oral Hygiene 70/78 Toileting 73/73 Washing up 100/33 Appliance Use 100/75 Use of Heating 84/78 Care of clothes and linen 100/73 Making a snack 100/78 Making a drink 75/60 Use of phone 64/64 Leisure Activity 100/79 Infant Care 100/58 Medication Taking 100/93 Housework 100/82 Legend Point solution General solution RFID Inferring ADLs from Interactions with Objects Philipose, Fishkin, Perkowitz, Patterson, Hähnel, Fox, and Kautz IEEE Pervasive Computing, 4(3), 2004

Summary: CARE Activities of daily living can be learned and robustly tracked using RFID tag data Simple, direct sensors can often replace (or augment) general machine vision Works for essentially all ADLs defined in healthcare literature

Next Steps

Key Next Problems Decision-theoretic control of user interfaces Prompts helpful or distracting? User error, or user model error? Context-dependent costs Decision-theoretic natural language processing

Key Next Steps Measures of effectiveness ACCESS: user studies with potential clients (controlled conditions) this spring CARE: Intel / UW / Wash Dept Social Services developing trial for caregiver evaluation Goal: Long-term deployment in naturalistic settings Homes, nursing homes, assisted care facilities

Future Research Physiological sensors Heterogeneous sensors Heart rate, respiration, temperature Heterogeneous sensors Environmental + wearables + machine vision Smart homes Systems for improving self-awareness Emotional self-regulation Social pragmatics Target populations Autism spectrum disorders Traumatic brain injury

General Architecture common- sense knowledge decision making user cognitive state decision making intentions user profile activities physical behavior user interface caregiver alerts machine learning sensors

Conclusion: Why Now? An early goal of AI was to create programs that could understand ordinary human experience This goal proved elusive Missing tools for probabilistic inference Systems not grounded in real world Lacked compelling purpose Today we have the mathematical tools, the sensors, and the motivation

Other Research

real-time audio feature extraction Building Social Network Models from Sensor Data NSF Human Social Dynamics multi-modal sensor board Coded Database code identifier real-time audio feature extraction audio features WiFi strength

Modal Markov Logic Applied to Dialog Understanding DARPA CALO ASK_IF(S, H, P)  B(S, B(H,P) v B(H,~P)) B(p v q) B p B q B ~q B ~p B~p & B(p v q) => Bq ~B~p v ~Bp ~B~q v ~Bq B~q & B(p v q) => Bp

Planning as Satisfiability NSF Intelligent Systems Unless P=NP, no polynomial time algorithm for SAT But: great practical progress in recent years 1980: 100 variable problems 2005: 100,000 variable problems Can we use SAT as a engine for planning? 1996 – competitive with state of the art ICAPS 2004 Planning Competition – 1st prize, optimal STRIPS planning Inspired research on bounded model-checking

Efficient Model Counting NSF Intelligent Systems SAT – can a formula be satisfied? #SAT – how many ways can a formula be satisfied? Compact translation discrete Bayesian networks  #SAT Efficient model counting (Sang, Beame, & Kautz 2004, 2005) Formula caching Component analysis New branching heuristics Cachet – fastest model counting algorithm

Credits Graduate students: Colleagues: Funders: Don Patterson, Lin Liao, Ashish Sabharwal, Yongshao Ruan, Tian Sang, Harlan Hile, Alan Liu, Bill Pentney, Brian Ferris Colleagues: Dieter Fox, Gaetano Borriello, Dan Weld, Matthai Philipose, Tanzeem Choudhury Funders: NIDRR, Intel, NSF, DARPA