Human Activity Analysis 吴心筱 wuxinxiao@bit.edu.cn Based on the CVPR 2011 Tutorial: Human Activity Analysis. http://cvrc.ece.utexas.edu/mryoo/cvpr2011tutorial/

Introduction

Understanding People in Video Goal Find the people Infer their poses Recognize what they do

Level of People Understanding Object-Level Understanding Locations of persons and objects Tracking-Level Understanding Object trajectories --correspondence

Level of People Understanding Pose-Level Understanding Human body parts Activity-Level Understanding Recognition of human activities and events

Object Detection Pedestrian (i.e. human) detection Detect all the persons in the video 找个行人检测的video

Object Tracking Person tracking Tracking the person in every frame 找个行人跟踪的video

Pose Estimation Human Pose Joint locations or angles of a person measured per frame Video as a sequence of poses 找个行人检测的video

Human Activity Recognition A collection of human/object movements with a particular semantic meaning Activity Recognition Finding of video segments containing such movements 找个行人检测的video

Levels of Human Activities Categorized based on their complexity Hierarchy # of participants Gestures Actions Interactions Group Activities 找个行人检测的video

Gestures Atomic components Single body-part movements 找个行人检测的video

Actions Single actor movement Composed of multiple gestures organized temporally 找个行人检测的video

Interactions Human-human interaction Human-object interaction 增加:Human-scene interaction

Group Activities Multiple persons or objects 找个行人检测的video

Applications 找个行人检测的video

Surveillance Monitor suspicious activities for real-time reactions. (e.g.,‘Fighting’, ‘stealing’) Currently, surveillance systems are mainly for recording. Activity recognition is essential for surveillance and other monitoring systems in public places 找个行人检测的video

Intelligent Environments (HCI) Intelligent home, office, and workspace Monitoring of elderly people and children. Recognition of ongoing activities and understanding of current context is essential. 找个行人检测的video

Sports Play Analysis 找个行人检测的video

Web-based video retrieval YouTube 20 hours of videos uploaded every minute Content-based search Search based on contents of the video, instead of user-attached keywords Example: search ‘kiss’ from long movies 找个行人检测的video

Challenges 找个行人检测的video

Robustness Environment variations Background Moving backgrounds Pedestrians Occlusions View-points – moving camera 找个行人检测的video

Motion Style Each person has his/her own style of executing an activity Who stretches his hand first? How long does one stay his hand stretched? 找个行人检测的video

Various Activities There are various types of activities The ultimate goal is to make computer recognize all of them reliably. 找个行人检测的video

Learning Insufficient amount of training videos Traditional setting: Supervised learning Human efforts are expensive! Unsupervised learning Interactive learning Interactive learning?

Overview 找个行人检测的video

Activity Classification Simple task of identifying videos Classify given videos into their types. Known, limited number of classes Assumes that each video contains a single activity 找个行人检测的video

找个行人检测的video

Activity Detection Search for the particular time interval <starting time, ending time> Video segment containing the activity 找个行人检测的video

Activity Detection by Classification Binary classifier Yes, Puch ! Sliding window technique Classify all possible time intervals 找个行人检测的video

Recognition Process Represent videos in terms of features Captures properties of activity videos Classify activities by comparing video representations Decision boundary 找个行人检测的video

Approaches 找个行人检测的video

Approach based taxonomy 找个行人检测的video

Single-layered vs. hierarchical Single-layered approaches Hierarchical approaches

Single-layered approaches 找个行人检测的video

Two different types Space-time approaches (data-orientated) Activities as video observations 3D space-time volume (3D XYT volume) A set of features extracted from the volume 找个行人检测的video

Two different types Sequential approaches (semantic-oriented) Activities as human movements A sequence of particular observations(feature vectors) 找个行人检测的video

Space-time approaches Space-time volumes Space-time local features Space-time trajectories 找个行人检测的video

Space-time volumes Problem: matching between two volumes 找个行人检测的video

Motion history images Bobick and J. Davis, 2001 Motion history images (MHIs) Weighted projection of a XYT foreground volume Template matching Bobick and J. Davis, The recognition of human movement using temporal templates,IEEE T PAMI 23(3),2001

Segments Ke, Suktankar, Herbert 2007 Volume matching based on its segments. Segment matching scores are combined. [ Ke, Y., Sukthankar, R., and Hebert, M., Spatio-temporal shape and flow correlation for action recognition. CVPR 2007]

Space-time local features Local descriptors / interest points From 2D to 3D ; Sparse Low-level: which local features to be extracted Mid-level: How to represent the activity using local features High-level: What method to classify activities 找个行人检测的video

Cuboid Dollar et al., Cuboid, VS-PETS 2005 2D Gaussian smoothing kernel: 1D Gabor filters applied temporally: 加入公式 [Dollar, P., Rabaud, V., Cottrell, G., and Belongie, S., Behavior recognition via sparse spatio-temporal features, VS-PETS 2005]

Cuboid Dollar et al., Cuboid, VS-PETS 2005 Appearances of local 3-D XYT volumes Raw appearance Gradients Optical flows Captures salient periodic motion. 加入公式 [Dollar, P., Rabaud, V., Cottrell, G., and Belongie, S., Behavior recognition via sparse spatio-temporal features, VS-PETS 2005]

STIP interest points Laptev and Linderberg 2003 Introduced the KTH dataset Local descriptor based on Harris corner detector Simple periodic actions 加入公式 [Schuldt, C., Laptev, I., and Caputo, B., Recognizing human actions: A local SVM approach, ICPR 2004]

STIP interest points Laptev and Linderberg 2003 加入公式 [Schuldt, C., Laptev, I., and Caputo, B., Recognizing human actions: A local SVM approach, ICPR 2004]

Bag-of-words Representation 找个行人检测的video

SVMs classification SVMs classifier Shake hands ! Multiple kernel learning 找个行人检测的video …… [Schuldt, C., Laptev, I., and Caputo, B., Recognizing human actions: A local SVM approach, ICPR 2004]

pLSA models pLSA from text recognition Probabilistic latent semantic analysis Reasoning the probability of features originated from a particular action video. 加入公式 [Niebles, J. C., Wang, H., and Fei-Fei, L., Unsupervised learning of human action categories using spatial-temporal words, BMVC 2006]

pLSA models 加入公式 [Niebles, J. C., Wang, H., and Fei-Fei, L., Unsupervised learning of human action categories using spatial-temporal words, BMVC 2006]

Space-time trajectories Trajectory patterns Yilmaz and Shah, 2005 – UCF Joint trajectories in 3-D XYT space. [[Yilmaz, A. and Shah, M., Recognizing human actions in videos acquired by uncalibrated moving cameras, ICCV 2005]

Space-time trajectories Trajectory patterns Yilmaz and Shah, 2005 – UCF Compared trajectory shapes to classify human actions. [[Yilmaz, A. and Shah, M., Recognizing human actions in videos acquired by uncalibrated moving cameras, ICCV 2005]

Space-time approaches Summary Space-time volumes A straightforward solution Difficult in handling speed and motion variations 找个行人检测的video

Space-time approaches Summary Space-time local features Robust to noise and illumination changes Recognize multiple activities without background subtraction or body-part modeling Difficult to model more complex activities 找个行人检测的video

Space-time approaches Summary Space-time trajectories Perform detailed-level analysis View-invariant in most cases Difficult to extract the trajectories 找个行人检测的video

Two different types Sequential approaches (semantic-oriented) Activities as human movements A sequence of particular observations(feature vectors) 找个行人检测的video

Sequential approaches Exemplar-based approaches State model-based approaches

Exemplar-based approaches Matching between the input sequence of feature vectors and the template sequences Problem: how to match two sequences in different styles/different rates

Dynamic Time warping Match two sequences with variations Find an optimal nonlinear match 找个行人检测的video [[Yilmaz, A. and Shah, M., Recognizing human actions in videos acquired by uncalibrated moving cameras, ICCV 2005]

State model-based approaches A human activity as a model composed of a set of states Each class has a corresponding model Measure the likelihood between the model an the input image sequence HMMs 找个行人检测的video

HMMs Given observations V (a sequence of poses), find the HMM Mi that maximizes P(V|Mi). Transition probabilities aij and observations probabilities bik are pre-trained using training data. 找个行人检测的video

HMMs for Actions Each hidden state is trained to generate a particular body posture. Each HMM produces a pose sequence: action 找个行人检测的video

HMMs for Hand Gestures HMMs for gesture recognition American Sign Language (ASL) Sequential HMMs shapes and position of hands 找个行人检测的video [Starner, T. and Pentland, A., Real-time American Sign Language recognition from video using hidden Markov models. International Symposium on Computer Vision, 1995.]

Sequential approaches summary Designed for modeling sequential dynamics Markov process Motion features are extracted per frame Limitations Feature extraction Assumes good observation models Complex human activities? Large amount of training data 找个行人检测的video

exemplar-based vs. state model-based Provide more flexibility for the recognition system: multiple sample sequences Less training data State model-based make a probabilistic analysis of the activity model more complex activity 找个行人检测的video

Hierarchical approaches 找个行人检测的video

Hierarchy Hierarchy implies decomposition into subparts 找个行人检测的video

Hierarchical approaches Statistical approaches Syntactic approaches Description-based approaches 找个行人检测的video

Statistical approaches Statistical state-based models for recognition multiple layers of state-based models low-level: a sequence of feature vectors mid-level: a sequence of atomic action labels high-level: label of activity 找个行人检测的video

Layered hidden Markov models (LHMMs) Oliver et al. 2002 Bottom layer HMMs recognize atomic actions of a single person Upper layer HMMs treat recognized atomic actions as observations …… 找个行人检测的video [Oliver, N., Horvitz, E., and Garg, A. 2002. Layered representations for human activity recognition. In Proceedings of the IEEE International Conference on Multimodal Interfaces (ICMI). IEEE, Los Alamitos, CA, 3-8.

Statistical approaches summary Reliably recognize activities in the case of noisy inputs with enough training data Inability to recognize activities with complex temporal structures (e.g., concurrent subevents) subevent A Subevent B 找个行人检测的video subevent A subevent A subevent B subevent B

Syntactic approaches Model activity as a string of symbols, each symbol corresponds to an atomic-level action Parsing techniques from the field of programming languages Context-free grammars (CFG) and stochastic context-free grammars (SCFGs)

CFG for human activities Given the recognized atomic actions Production rules

Parse tree 找个行人检测的video

Gesture analysis with CFGs 找个行人检测的video

Primitive recognition with HMMs

Parse Tree

Syntactic approaches summary Difficult in the recognition of concurrent activities Require a set of production rules fro all possible events Directions: how to learn grammar rules from observations automatically? 找个行人检测的video

Description-based approaches Represents a high-level human activity in terms of simpler activities Describe various relationships between subevents (atomic actions) temporal relationship spatial relationship logical relationship

Description-based approaches Recognize activities using semantic matching. Hand shake = “two persons do shake-action (stretches, stays stretched, withdraw) simultaneously, while touching”. Recognition by finding observations satisfying the definition.

Hierarchical approaches summary