Recognizing Activities of Daily Living from Sensor Data Henry Kautz Department of Computer Science University of Rochester

Activity Recognition Much recent interest in recognizing human activity from heterogeneous sensor data Motion sensors GPS RFID Video Compelling applications Military / security operations (e.g. ASSIST) Smart homes & offices

Gathering data on indoor activities

Interpreting RFID Data (using Switching HMM)

Gathering Multi-view Video

Interpreting Video Computing scene statistics Ai = activity Oi = object Si = scene statistic Di = object statistics Ri = RFID label (for training) Computing object statistics

Gathering data on outdoor activities Raw GPS

Discovering significant places Conditional Random Field

Predicting transportation goals Dynamic Bayesian Network

Issue Previous work on activity recognition has used a wide variety of probabilistic models for different tasks and kinds of data HMMs, DBNs, CRFs, … Background knowledge is implicitly encoded in the structure of the models E.g.: Relation between transportation goals, plans, actions Increasingly difficult to scale & integrate

Markov Logic Markov logic will provide common modeling language & inference tools, enabling Easier integration of multiple sensors Easier generalization From one activity at a time to multiple ongoing activities From one individual to multiple individuals Easier modification of background knowledge Add / modify library of plans and goals

Example Scenario John goes into his kitchen (video) He takes out a jug from the refrigerator, and a bowl from the cabinet (RFID) He leaves his apartment, and walks to a convenience store (GPS) He returns carrying a box (video) He pours the box into the bowl (accelerometer) and the contents of the jug (accelerometer & RFID) Why did John leave the apartment? What did he do?

UR Contributions to MURI: Scenario Development & Data Collection Develop set of activity recognition scenarios of increasing complexity Activities in the home Outdoor activities Enact and gather sensor data Heterogeneous: GPS, RFID, video, motion, … Intermittent and noisy Make dataset available to team Including feature sequences extracted from video and acceleration data Ground truth 1 st data set mid-Year One, then ongoing

UR Contributions to MURI: Unified ML Model of Daily Activities Recast our previous work on recognition using HMMs, DBNs, CRFs in Markov Logic Integrate and generalize earlier results Year One: HMM  ML Generalize to multiple ongoing activities Handle novel observations using similarity Representing actions, intentions, and goals Extend ML to include “modal operators” Distinguish beliefs of observer from beliefs of subject Ability to model imperfect agents, whose plans are flawed

From HMMs to ML Hidden Markov models describe the world as probabilistic state machine ML encoding of HMM can be relaxed to allow subject to be in multiple states (multiple activities) by making “unique state” constraint soft

From HMMs to ML Novel observations can be handled by applying background knowledge about similarity

Modal Operators Most previous work on probabilistic activity recognition does not distinguish What system infers is true about the world What the subject believes is true about the world What the system predicts will happen What the subject intends to happen Modal operators relate agents to attitudes Bel( John, contains(jug, gasoline) )  But system may know jug is empty Goal( John, ignite(jug) )  Knowledge of subject’s goal can drive cooperative system to help subject, or antagonistic system to block user

Semantic Inference Modal operators do not work like ordinary predicates or logical connectives Modal proof theory is hard to automate However: Modal operators have well-understood “possible world” semantics Agent believes P in possible world W iff P is true in all worlds W’ such that reachable(W,W’) ML’s inference engine works at the semantic level (direct search over possible worlds) Promising approach: semantic inference for modal constructs in ML Explicitly model reachability relationships for each attitude and agent

Idea Alchemy searches over models (truth assignments) Modal formulas are evaluated over structures Structure = set of models and accessibility relationships over the models Structures are too big to explicitly search Modify Alchemy to search over samples drawn from structures