1. Derek Hao Hu, Qiang Yang Hong Kong University of Science and Technology

2. Activity Recognition (Goal Recognition) Input: Sensor Readings (Action sequences) Output: Inferred action sequences (Inferred goal sequences) We won’t disambiguate these two notations later since they basically discuss the same thing. Traditional Assumption Users achieve one goal / one activity within each time slice. Goals are achieved consecutively. 7/16/2008 Physically Grounded AI: Activity Recognition2

3.  Goal composition types in activity sequences 7/16/2008 Physically Grounded AI: Activity Recognition3 Previous works only try to tackle with Type 1 and Type 2 (Single Goal and Non-interleaving, Non- concurrent multiple Goals) The major objective of our algorithm is to provide accurate recognition for all the five basic composition types of activity sequences.

4. Real world situations are often the case that goals are achieved in a concurrent and interleaving way. (multiple-goal recognition problem). Example 1: Concurrent Goals Goal_1: Print research papers Goal_2: Go to the meeting room Goals are pursued concurrently Example 2: Interleaving Goals Goal_1: Having breakfast Goal_2: Dealing with boiling water Goals are pursued interleavingly 7/16/2008 Physically Grounded AI: Activity Recognition4

5.  Logic-based approaches ◦ [Kautz 1987] A Formal Theory of Plan Recognition ◦ Keep track of all logically consistent explanations of the observed activities  Probabilistic-based approaches ◦ State-space models for inferring hidden states, given observations ◦ Aggregate DBN (Patterson et al. 2005) and CRF (Vail et al. 2007)  Multiple Goal Recognition ◦ (Chai and Yang, 2005), deterministic model for handling interleaving goals ◦ (Wu, Lian and Hsu 2007), factorial CRF model for handling multiple concurrent goals ◦ No model can handle both interleaving and concurrent goals at the same time ◦ An interesting trivia: an ICML 2008 paper for dealing with interleaving activities.  Niels Landwehr, Modeling Interleaved Hidden Processes, ICML 2008  The experimental setting is activity recognition. 7/16/2008 Physically Grounded AI: Activity Recognition5

6.  Decomposition of interleaving goals  Analogy: Multi-class classification -> Binary Classification 7/16/2008 Physically Grounded AI: Activity Recognition6

7.  Skip-Chain CRF Model ◦ Advantages  [1] Relatedness of multiple-goal recognition with NER (Named Entity Recognition)  [2] Able to model uncertainty  [3] Flexibility of adding skip edges (In our case, we add skip edges between nodes if prior probability larger than $\theta$) ◦ Hence, skip-chain CRF model is a natural graphical model we use for modeling interleaving goals in our multiple-goal recognition problem. 7/16/2008 Physically Grounded AI: Activity Recognition7

8. 7/16/2008 Physically Grounded AI: Activity Recognition8 For more technical details, check CRF papers like [Sutton et al. 2007], Dynamic Conditional Random Fields: Factorized Probabilistic Models for Labeling and Segmenting Sequence Data

9.  Similarities and relatedness of different goals ◦ Observation: The relations (positive & negative) between different goals are possible to help ◦ We need to know how similar and correlated two goals are ◦ Examples  Example 1: Academic-related goals and sports-related goals (Negative correlations)  Example 2: Watching TV and Having breakfast ◦ Objective: To build a correlation graph of goals, where two goals are related by an edge with large positive weight in [0,1] if they have strong positive correlations. ◦ Discussions of negative correlations are omitted in this paper  Possible for future work 7/16/2008 Physically Grounded AI: Activity Recognition9

10.  Our goal is to build a similarity matrix S, where S_{ij} denotes the positive correlation between goals G_i and G_j. ◦ Initialization: S_{ij} = P(G_i|G_j) (posterior probability, maybe sparse)  This is S_0. ◦ Iteration: S_{k+1} = A{S_k}A^T + A^T{S_k}A  $A$ is the adjacency matrix of the similarity graph (A_{i,j}=0 if P(G_i|G_j=0, A_{i,j}=1 otherwise).  Convergence property can be proved. 7/16/2008 Physically Grounded AI: Activity Recognition10

11.  Quadratic Programming Formulation 7/16/2008 Physically Grounded AI: Activity Recognition11 We are trying to make optimizations s.t. 1.Minimize the differences between strong correlated goals 2.Minimize the differences between adjusted probability and the initially inferred probabilities

12. 7/16/2008 Physically Grounded AI: Activity Recognition12 Note that, we have $m$ CRFs, each with $T$ nodes. However, if we use Factorial CRF, we may need to build a CRF with $mT$ nodes, which exponentially increases the worst-case training complexity. Therefore, the complexity of our algorithm is more scalable than a general CRF model.

13.  Baseline Descriptions ◦ MG-Recognizer: Deterministic model in [Chai and Yang, 2005]  For handling interleaving activities / goals ◦ FCRF: Factorial CRF in [Wu, Lian, Hsu 2007], mainly for modeling concurrent activities. ◦ SCCRF with different parameters of $\theta$: Only model with interleaving activities, no QP part. ◦ CIGAR with different parameters of $\theta$ and \$mu$: the main algorithm with different parameter settings.  Criteria ◦ Recognition accuracy: The percentage of correctly recognized goals over all goals across all time slices for all the activity sequences.  All datasets downloadable from my homepage: http://www.cse.ust.hk/~derekhh/ http://www.cse.ust.hk/~derekhh/ 7/16/2008 Physically Grounded AI: Activity Recognition13

14.  Dataset 1 ◦ Departmental Office in HKUST ◦ Have both single-goal and multiple-goal sequences 7/16/2008 Physically Grounded AI: Activity Recognition14

15.  Dataset 2 ◦ Morning activities dataset in [Patterson et al. 2005] ◦ No concurrent activities in this dataset 7/16/2008 Physically Grounded AI: Activity Recognition15

16.  Dataset 3 ◦ MIT PlaceLab PLIA1 dataset ◦ Goals clustered into six “hypergoals” for testing interleaving and concurrency 7/16/2008 Physically Grounded AI: Activity Recognition16

17.  The nature of real-world activity recognition calls for online inference algorithms.  A unified CRF-like model that can deal with both concurrent and interleaving goals? ◦ General CRF model can do this, but speed is a major concern ◦ How to speed up the 2D CRF-like model under such a situation?  Other CRF training methods for better accuracy?  Taking negative correlation into consideration  Comparison with other models? (e.g. the one mentioned in ICML 2008?) 7/16/2008 Physically Grounded AI: Activity Recognition17

18. Any Questions?