1. Activity Recognition Ram Nevatia Presents work of
F. Lv, P. Natarajan and V. Singh
Institute of Robotics and Intelligent Systems
Computer Science Department
Viterbi School of Engineering
University of Southern California

2. Activity Recognition: Motivation
Is the key content of a video (along with scene description)
Useful for
Monitoring (alerts)
Indexing (forensic, deep analysis, entertainment…)
HCI ..

3. Issues in Activity Recognition
Inherent ambiguities of 2-D videos
Variations in image/video appearance due to changes in viewpoint, illumination, clothing (texture)….
Variations in style: different actors or even the same actor at different times
Reliable detection and tracking of objects, especially those directly involved in activities
Temporal segmentation
Most work assumes single activity in a given clip
“Recognition” of novel events

4. Match video signals directly
Possible Approaches
Match video signals directly
Dynamic time warping
Extract spatio-temporal features, classify based on them
Bag of words, histograms, “clouds”…
Work of Laptev et al
Most earlier work assumes action segmentation (detection vs classification)
Andrew’s talk on use of localization and tracking
Structural Approach
Based on detection of objects, their tracks and relationships
Requires ability to perform above operations

5. Primitive events: those we choose not to decompose, e.g. walking
Event Hierarchy
Composite Events
Compositions of other, simpler events.
Composition is usually, but not necessarily, a sequence operation, e.g. getting out of a car, opening a door and entering a building.
Form a natural hierarchy (or lattice)
Primitive events: those we choose not to decompose, e.g. walking
Recognized directly from observations,
Graphical models, such as HMMs and CRFs are natural tools for recognition of composite events.

6. Only need a few primitive actions in any domain.
Key Ideas
Only need a few primitive actions in any domain.
Sign Language: Moves and Holds.
Human Pose Articulation: Rotate, Flex and Pause.
Rigid Objects (cars, people): Translate, Rotate, Scale.
Can be represented symbolically using formal rules.
Composite Actions can be represented as combinations of the primitive actions.
Handle uncertainty and error in video by mapping rules to a Graphical Models.
HMM.
DBN.
CRF.

7. Graphical Models
A network, normally used to represent temporal evolution of a state
Next state depends only on previous state; observation depends only on current state, single state variable
Typical task is to estimate most likely state sequence given an observation sequence- Viterbi algorithm
An HMM
A CRF

8. Mid vs Near Range Mid-range Near-range
Limbs of human body, particularly the arms, are not distinguishable
Common approach is to detect and track moving objects and make inferences based on trajectories
Near-range
Hands/arms are visible; activities are defined by pose transitions, not just the position transitions
Pose tracking is difficult; top-down methods are commonly used

9. Mid-Range Example Example of abandoned luggage detection
Based on trajectory analysis and simple object detection/recognition
Uses a simple Bayesian classifier and logical reasoning about order of sub-events
Tested on PETS, ETISEO and TRECVID data 9

10. Simultaneous Tracking and Action Recognition (STAR)
Top-Down Approaches
Bottom-up methods remain slow, are not robust; many methods are based on use of multiple video streams
An alternative is of top-down approaches where processing is driven by event models
Simultaneous Tracking and Action Recognition (STAR)
In analogy with SLAM in robotics
Provides action segmentation, in addition to recogntion
Closed-world assumption
Current work limited to single actor actions

11. Activity Recognition w/o Tracking
Input sequence …
Action segments
check watch
punch
kick
pick up
throw
Our objective is very clear: given a single video sequence, we want to segment the whole sequence into many parts.
Each part contains only one instance of some human action and we want to recognize the action in each part.
We are interested in basic human actions such as walking, sitting down, punching, kicking. In our dataset, there are 15 different actions.
Although we do not explicitly estimate 3D human body poses, but our result will provide such poses as a by product. +
3D body pose …

12. Difficulties
Viewpoint change & pose ambiguity (with a single camera view)
Spatial and temporal variations (style, speed)

13. Key Poses and Action Nets
Key poses are determined from MoCap data by an automatic method that computes large changes in energy; key poses may be shared among different actions

14. Experiments: Training Set
Let me show our experimental results.
We have 15 action models, including stand, check watch, cross arm, scratch head, sit down, stand up, turn around, walk in a circle, wave hand, punch, kick, point, pick up and throw. We also include some variations for some action classes. For example, we include wave left hand v.s. wave right hand, kick gently v.s. kick vigorously.
In total we have one hundred and seventy seven key poses. Given the tilt angle, we render them from 36 viewpoints with different pan angles.
And that results in more than six thousand rendered key poses in the action net.
15 action models
177 key poses
6372 nodes in Action Net

15. Action Net: Apply constraints
Because key poses are rendered from many viewpoints, we repeat the same action net for each viewpoint.
We use these magenta links to connect action models rendered from adjacent viewpoints. These links allow us to model smooth change in the actor’s orientation.
(The major difference between the Action Net and other graph models is that information such as camera viewpoint and action connectivity is explicitly modeled in the Action Net, while these graph models use parameters to encode such information.)
10o …

16. 50 clips, average length 1165 frames
Experiments: Test Set
Our testing set contains 50 video clips. In each clip, one actor is performing all 15 actions. The actors can freely choose orientation and position. The order and the number of action instances in each clip are also variable.
These clips are shot from 5 different viewpoints.
There are 5 male actors and 5 female actors.
The large number of action classes and the large variation in viewpoints and actors make action recognition on this dataset a challenging task
50 clips, average length 1165 frames
5 viewpoints
10 actors (5 men, 5 women)

17. A Video Result extracted blob & ground truth original frame
without action net
with action net

18. Working with Natural Environments
Reduce reliance on good foreground segmentation
Key poses may not be discriminative enough w/o accurate segmentation; include models for motion between key poses
More general graphical models that include
Hierarchy
Transition probabilities may depend on observations
Observations may depend on multiple states
Duration models (HMMs imply an exponential decay)
Remove need for MoCap data to acquire models

19. Composite Event Representation
CE: Sequence(P1,P2)
P1: Rotate( Right, Arm, 90o,z-axis)
P2: Rotate( Right, Arm, 90o,-z-axis)

20. Learning Event Models
Primitive Event P1
Primitive Event P2
Composite Event = Sequence(P1,P2)

21. Dynamic Bayesian Action Network
Map action models to a Dynamic Bayesian Network
Decompose a composite action into a sequence of primitive actions
Each primitive is expressed in a function form fpe(s,s’,N).
Maps current state s to next state s’ given parameters N.
Assume a known, finite set of functions f for primitives.

22. For each current state:
Inference Overview
Given a video, obtain initial state distribution with start key pose for all composite actions
For each current state:
Predict the primitive based on the current duration
Predict a 3D pose given the primitive and current duration
Collect the observation potential of the pose using foreground overlap and difference image
Obtain the best state sequence using dynamic programming (Viterbi Algorithm)
Features used to match models with observations
If “foreground” can be extracted reliably, then we can use blob shape properties; otherwise, use edge and motion flow matching

23. Pose Tracking & Action Recognition
Obtain state distributions by matching poses sampled from action models
Infer the action by finding the max likelihood state sequence,
Inference Algorithm

24. Observations
Foreground overlap with full body model,
Difference Image overlap with body parts in action
Grid-of-centroids to match foreground blob with pose

25. Results
From CVPR08 paper

26. Action Learning
Involves two problems
Model Learning: Learning parameters N in the primitive event definition fpe(s,s’,N).
Key Pose Annotation and Lifting.
Pose Interpolation
Feature Weight Learning: Learning the weights wk of the different potentials.

27. KeyPose Annotation and 3D Lifting

28. Pose Interpolation
All limb motions can be expressed in terms of Rotate(part,axis,q).
We need to learn axis and q.
Simple to do given the start and end joints of part.

29. Feature Weight Learning
Feature weight estimation as minimization of a log-likelihood error function.
Learn the weights using Voted Perceptron Algorithm
Requires fully labeled training data -> not available.
We propose an extension to deal with partial annotations.
Latent State Voted Percepton

30. Tested method on 3 datasets
Experiments
Tested method on 3 datasets
Weizmann dataset
Gesture set with arm gestures
Grocery Store set with full body actions
Dataset
Train:Test
Ratio
Action Recognition
(% accuracy)
2D Tracking
(% error)
Speed
(fps)
Weizmann
3:6
99.5 --
Gesture
3:5
90.18
5.25 8
Grocery Store
1:7
100.0
11.88
1.6

31. Weizmann Dataset Popular dataset for action recognition
10 full body actions from 9 actors
Each video has multiple instance of one action
Train:Test
Recognition
Accuracy
Jhuang et al [9]
6:3
98.8
Space-Time Shapes [6]
8:1
100.0
Fathi Et al [5]
Sun et al [20]
3:6
87.3
DBAN
1:8
96.7
99.5

32. Gesture Dataset 5 instances of 12 gestures from 8 actors.
Indoor lab setting.
500 instances of all actions.
852x480 pixel resolution, person height: pix.

33. Grocery Store Dataset
Videos of 3 actions collected from a static camera.
16 videos from 8 actors, performed at pan angles.
Actor height varies from pixels, in 852x480 resolution videos.

34. Incorporating Better Descriptors
Previous work based on weak lower-level analysis
We can also evaluate 2D part models
Dynamic Bayesian Action Network
with Part Model

35. Experiments Hand gesture dataset in an Indoor lab
5 instances of 12 gestures from 8 actors, total of 500 action segments
Evaluation metrics
Recognition rate over all action segments
2D pose tracking as average 2D part accuracy over 48 randomly selected instances
Dataset
Train:Test
Ratio
Recognition
(% accuracy)
2D Tracking
DBAN-FGM
1:7
78.6
75.67 (89.94)
DBAN-Parts
84.52
91.76 (92.66)

36. Summary and Conclusions
Structural approach to activity recognition offers many attractions and challenges
Results are descriptive but detecting and tracking objects is challenging
Hierarchical representation is natural and can be used to reduce complexity
Good bottom-up analysis remains a key to improved robustness
Concept of “novel” or “anomalous” events remains difficult to formalize