Understanding Human Behavior from Sensor Data

Understanding Human Behavior from Sensor Data
Henry Kautz University of Washington

Trends

Growing Ubiquitous Sensing Infrastructure
GPS Wi-Fi localization RFID tags Wearable sensors

Advances in Artificial Intelligence
Graphical models Particle filtering Belief propagation Statistical relational learning

Crisis in Caring for the Cognitively Disabled
Epidemic of Alzheimer’s Community integration of 7.5 million citizens with MR year disabled by TBI Post-traumatic stress syndrome Caregiver burnout

...An Opportunity Create methods for modeling and interpreting human behavior from sensor data In order to develop assistive technologies to support independent living by people with cognitive disabilities Help people perform activities of daily living Monitor behavior to prevent health crises

Outline Learning and reasoning about transportation routines
ACCESS personal guidance system Understanding activities of daily living CARE monitoring system Further directions

Assisted Cognition in Community, Employment, & Social Settings
ACCESS Assisted Cognition in Community, Employment, & Social Settings

Motivation: Community Access for the Cognitively Disabled

The Problem Using public transit is cognitively challenging
Learning bus routes and numbers Transfers Recovering from mistakes Point to point shuttle service impractical Slow Expensive Current GPS units hard to use Require extensive user input Error-prone near buildings, inside buses No help with transfers, timing

Goal A personal guidance system that
Requires no explicit programming by user or caregiver Proactively assists user in completing transportation plans Recognizes user errors, and helps user recover

Technical Problem Given a data stream from a wearable GPS unit...
Infer the user’s location and mode of transportation (foot, car, bus, bike, ...) Predict the user’s destination Detect user errors

GPS Receivers GeoStats GPS logger
Today the GPS receivers become much smaller and cheaper than a few years ago. GeoStats GPS logger Data capacity: second frequency Battery life: 72 hours Nokia 6600 Cell Phone Java Bluetooth GPS unit Internet connectivity for off-board processing Battery life: 8 hours

GIS Data Street map Bus routes & stops Business / residential areas
Census 2000 Bus routes & stops Seattle Metro Business / residential areas MapPoint The second related system is …. The kernel of such systems is the database about geographic information. GIS technology has been used for tens of years and today there are a large number of different kinds of geographic data available. In this project we rely on two kinds of geographic data. The first data is the street map. The left picture shows…. The second is the bus route and bus stop data. The picture on the right…

Challenges of GPS Data Many dead zones in urban areas
Sparse measurements inside cars and buses Systematic error  10 meters Multi-path propagation Mapping errors When we talk about GPS-based location tracking, many people will expect it is trivial.

Representation Map is a directed graph G=(V,E) Location: Movement:
Edge (block) Distance along edge Actual GPS reading Movement: Mode { foot, car, bus, indoors } influences velocity Change edges at intersections Change mode at bus stops, parked car, buildings Tracking (filtering): Given some prior estimate, Update position & mode according to motion model Correct according to next GPS reading So far, I have described the general picture of our system and some applications. In the rest of the presentation, I will focus on the technical approach of probabilistic reasoning. There are three key things in our system. For the .., we use…. Because of the time issue, I cannot discuss those techniques in details. So I will only give you a very rough picture for each of them and show the results of our experiments.

Motion Model for Mode

Dynamic Bayesian Network I
mk-1 mk Transportation mode xk-1 xk Edge, velocity, position qk-1 qk Data (edge) association zk-1 zk GPS reading Time k-1 Time k

Rao-Blackwellised Particle Filtering
Evolve approximation to state distribution using samples (particles) Supports multi-modal distributions and discrete variables (mode, edge) Rao-Blackwellisation Particles include distributions over variables Each particle is a Kalman filter (Gaussian along edge) Improved accuracy with fewer particles After we have the DBN, one central task is to estimate the …….. This is called inference. For example, in our DBN, the inference means to infer the current goal, trip segment and location and velocities from the GPS measurements up to now. Many ways to do inference. Particle filter is one of them.

Learning to Predict User
Prior knowledge – general constraints on transportation use Vehicle speed range Bus stops Learning – specialize model to particular user 30 days GPS readings of one user, logged every second Unlabeled data Learn edge and mode transition parameters using expectation-maximization (EM)

Mode & Location Tracking
Measurements Projections Bus mode Car mode Foot mode Green Red Blue Explain why it switches from foot to bus, not car: because of the bus stops If people ask why no car particle at all (there are car particles, just their weights are too small that can not be displayed).

How to improve the model’s predictive power?
Predictive Accuracy How to improve the model’s predictive power? Probability of correctly predicting the future 5 blocks 50% accuracy City Blocks

Transportation Routines
Home A B Workplace Goal: intended destination Workplace, home, friends, restaurants, … Trip segments: <start, end, mode> Home to Bus stop A on Foot Bus stop A to Bus stop B on Bus Bus stop B to workplace on Foot Now the system could infer location and transportation modes, but wee still want to know more. We first show an example. Goal In order to get to the goal, the user may switch transportation mode. Thus we introduce the concept of trip segments.

Dynamic Bayesian Net II
gk-1 gk Goal tk-1 tk Trip segment mk-1 mk Transportation mode xk-1 xk Edge, velocity, position qk-1 qk Data (edge) association zk-1 zk GPS reading Time k-1 Time k

Unsupervised Hierarchical Learning
Use previous model to infer: Goals - locations where user stays for long periods of time Transition points - locations with high mode transition probability Trip segments – paths connecting transition points or goals Perform EM learning on the hierarchical model Learn transition parameters: Between goals Between trip segments, given the goal Between edges & modes, given the trip segment

High Probability Trip Segments Conditioned on Goal
Goal = Workplace Goal = Home We visualize the learned street transitions. Left picture, the goal is the workplace. Conditioned on the goal, we draw the trip segment and the very likely street transitions in the trip segments. we can clearly see the different trajectories to the workplace in different transportations Right picture, similar, different destination : bus : car : foot

Predicting Goal and Path
Predicted goal Predicted path

Improvement in Predictive Accuracy
45 blocks 50% accuracy

Detecting User Errors Learned model represents typical correct behavior Model is a poor fit to user errors We can use this fact to detect errors! Cognitive Mode Normal: model functions as before Error: switch in prior (untrained) parameters for mode and edge transition After we learn the model, detect potential errors.

Dynamic Bayesian Net III
zk-1 zk xk-1 xk qk-1 qk Time k-1 Time k mk-1 mk tk-1 tk gk-1 gk ck-1 ck Cognitive mode { normal, error } Goal Trip segment Transportation mode Edge, velocity, position Data (edge) association GPS reading

Goal Clamping The user’s goal may be explicitly known
Ask the user to confirm highest-probability goal Appointment calendar Incorporating such information clamps the goal Distinguishes novelty from errors

Detecting User Errors Untrained Trained Clamped
In this episode, the person takes the bus home but missed the usual bus stop. The black disk is the goal and the cross mark is where the system think the person should get off the bus. The probability of normal and abnormal activities are represented using a bar here. The black part is the probability of being normal while the red part is the probability of being abnormal. At here, the person missed the bus stop and the probability of abnormal activity jump quickly. Untrained Trained Clamped

ACCESS Prototype Cell phone with GPS, camera, high-speed internet access Prompts when it infers that user is…

ACCESS Prototype Cell phone with GPS, camera, high-speed internet access Prompts when it infers that user is… About to begin a transportation plan Confirm destination? Here is your route!

ACCESS Prototype Cell phone with GPS, camera, high-speed internet access Prompts when it infers that user is… About to change mode This is your bus! Your stop is next!

ACCESS Prototype Cell phone with GPS, camera, high-speed internet access Prompts when it infers that user is… Making an error You missed your stop! Here is how to get back on track …

ACCESS Prototype Cell phone with GPS, camera, high-speed internet access Prompts when it infers that user is… Visiting a new destination Please take a picture!

Status Medical partnerships Extension to indoor navigation
Funding by National Institute of Disability & Rehabilitation Research (NIDRR) Partnership with UW Center for Technology and Disability Studies User & caregiver needs studies (TBI & MR) Data collection by job coaches Extension to indoor navigation Hospitals, nursing homes, assisted care communities Wi-Fi localization Multi-modal interface Speech, graphics, text Guidance strategies Proactive / Just in time Coordinates / Landmarks WOZ design study

Papers Patterson, Liao, Fox, & Kautz, UBICOMP 2003
Inferring High Level Behavior from Low Level Sensors Patterson et al, UBICOMP-2004 Opportunity Knocks: a System to Provide Cognitive Assistance with Transportation Services Liao, Fox, & Kautz, AAAI 2004 (Best Paper) Learning and Inferring Transportation Routines

Cognitive Assistance in Real-world Environments
CARE Cognitive Assistance in Real-world Environments

Goal A home monitoring system that
Assists user in performing activities of daily living Tracks activities, and provides prompts and warnings as needed Can be deployed in an ordinary home Does not require the user to learn a different way to perform the activities – the system adapts, not the user

Short-Term Application
Accurate, automated ADL logs Changes in routine often precursor to illness, accidents Human monitoring intrusive & inaccurate Image Courtesy Intel Research

Technical Requirements
Sensor hardware that can be practically deployed in a ordinary home Methods for activity tracking from sensor data Methods for automated prompting that consider Probability of user errors Probability of system errors Cost / benefit tradeoffs

Object-Based Activity Recognition
Activities of daily living involve the manipulation of many physical objects Kitchen: stove, pans, dishes, … Bathroom: toothbrush, shampoo, towel, … Bedroom: linen, dresser, clock, clothing, … We can recognize activities from a time-sequence of object touches

Sensing Object Manipulation
RFID: Radio-frequency identification tags Small Semi-passive Durable Cheap Near future: use products’ own tags

Wearable RFID Readers Designed by Intel Research Seattle
Will be shared with other Intel partners later this year 13.56MHz reader, radio, power supply, antenna 12 inch range, hour lifetime

Experiment: Morning Activities
10 days of data from the morning routine in an experimenter’s home 61 tagged objects 11 activities Often interleaved and interrupted Many shared objects Use bathroom Make coffee Set table Make oatmeal Make tea Eat breakfast Make eggs Use telephone Clear table Prepare OJ Take out trash

Methodology Goal: simplest model that can robustly track activities
Comparison Hidden Markov Model Dynamic Bayesian Network with aggregate features DBN with aggregation and abstraction smoothing

Hidden Markov Model Trained on labeled data 10-fold cross validation
88% accuracy 9.4 errors per 20 minute episode

Cause of Errors Observations were types of objects
Spoon, plate, fork … Typical errors: confusion between activities Using one object repeatedly Using different objects of same type Critical distinction in many ADL’s Eating versus setting table Dressing versus putting away laundry

Aggregate Features HMM with individual object observations fails
No generalization! Solution: add aggregate variables Bit-vector maintains history of objects touched Aggregate distribution nodes sum number of distinct instances Aggregate nodes treated as pseudo-observations when an activity transitions DBN encoding avoids explosion of HMM

DBN with Aggregation Average number of errors reduced from 9.4 to 6.5 (31%) Deterministic nodes add minimal computational overhead

Improving Robustness Both HMM and DBN fail if novel (but reasonable) objects are used Solution: smooth parameters over abstraction hierarchy of object types

Abstraction Smoothing
Methodology: Train on 10 days data Test where one activity substitutes one object Change in error rate: Without smoothing: 26% increase With smoothing: 1% increase

Experiment: ADL Form Filling
Tagged real home with 108 tags 14 subjects each performed 12 of 14 ADLs in arbitrary order Used glove-based reader Given trace, recreate activities

Results: Detecting ADLs
Activity Prior Work SHARP Personal Appearance 92/92 Oral Hygiene 70/78 Toileting 73/73 Washing up 100/33 Appliance Use 100/75 Use of Heating 84/78 Care of clothes and linen 100/73 Making a snack 100/78 Making a drink 75/60 Use of phone 64/64 Leisure Activity 100/79 Infant Care 100/58 Medication Taking 100/93 Housework 100/82 Legend Point solution General solution RFID Inferring ADLs from Interactions with Objects Philipose, Fishkin, Perkowitz, Patterson, Hähnel, Fox, and Kautz IEEE Pervasive Computing, 4(3), 2004

Summary: CARE Activities of daily living can be learned and robustly tracked using RFID tag data Simple, direct sensors can often replace (or augment) general machine vision Works for essentially all ADLs defined in healthcare literature

Next Steps

Key Next Problems Decision-theoretic control of user interfaces
Prompts helpful or distracting? User error, or user model error? Context-dependent costs Decision-theoretic natural language processing

Key Next Steps Measures of effectiveness
ACCESS: user studies with potential clients (controlled conditions) this spring CARE: Intel / UW / Wash Dept Social Services developing trial for caregiver evaluation Goal: Long-term deployment in naturalistic settings Homes, nursing homes, assisted care facilities

Future Research Physiological sensors Heterogeneous sensors
Heart rate, respiration, temperature Heterogeneous sensors Environmental + wearables + machine vision Smart homes Systems for improving self-awareness Emotional self-regulation Social pragmatics Target populations Autism spectrum disorders Traumatic brain injury

General Architecture common- sense knowledge decision making user
cognitive state decision making intentions user profile activities physical behavior user interface caregiver alerts machine learning sensors

Conclusion: Why Now? An early goal of AI was to create programs that could understand ordinary human experience This goal proved elusive Missing tools for probabilistic inference Systems not grounded in real world Lacked compelling purpose Today we have the mathematical tools, the sensors, and the motivation

Other Research

real-time audio feature extraction
Building Social Network Models from Sensor Data NSF Human Social Dynamics multi-modal sensor board Coded Database code identifier real-time audio feature extraction audio features WiFi strength

Modal Markov Logic Applied to Dialog Understanding DARPA CALO
ASK_IF(S, H, P)  B(S, B(H,P) v B(H,~P)) B(p v q) B p B q B ~q B ~p B~p & B(p v q) => Bq ~B~p v ~Bp ~B~q v ~Bq B~q & B(p v q) => Bp

Planning as Satisfiability NSF Intelligent Systems
Unless P=NP, no polynomial time algorithm for SAT But: great practical progress in recent years 1980: 100 variable problems 2005: 100,000 variable problems Can we use SAT as a engine for planning? 1996 – competitive with state of the art ICAPS 2004 Planning Competition – 1st prize, optimal STRIPS planning Inspired research on bounded model-checking

Efficient Model Counting NSF Intelligent Systems
SAT – can a formula be satisfied? #SAT – how many ways can a formula be satisfied? Compact translation discrete Bayesian networks  #SAT Efficient model counting (Sang, Beame, & Kautz 2004, 2005) Formula caching Component analysis New branching heuristics Cachet – fastest model counting algorithm

Credits Graduate students: Colleagues: Funders:
Don Patterson, Lin Liao, Ashish Sabharwal, Yongshao Ruan, Tian Sang, Harlan Hile, Alan Liu, Bill Pentney, Brian Ferris Colleagues: Dieter Fox, Gaetano Borriello, Dan Weld, Matthai Philipose, Tanzeem Choudhury Funders: NIDRR, Intel, NSF, DARPA

Understanding Human Behavior from Sensor Data

Similar presentations

Presentation on theme: "Understanding Human Behavior from Sensor Data"— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

Understanding Human Behavior from Sensor Data

Similar presentations

Presentation on theme: "Understanding Human Behavior from Sensor Data"— Presentation transcript:

Similar presentations

About project

Feedback