1. Human Action Recognition across Datasets by Foreground-weighted Histogram Decomposition
Waqas Sultani, Imran Saleemi
CVPR 2014

2. Motivation

3. Dense STIP for cross dataset recognition
Training
Testing
Accuracy (avg)
UCF50
70 %
UCF50
HMDB51
55.7 %
Olympic Sports
71.8 %
UCF50
Olympic Sports
16.67 %
Recognition Accuracy drops across the datasets!
Training and Testing is done on similar actions across the datasets

4. Is the background responsible for this drop in accuracy?
UCF50
HMDB51
Is the background responsible for this drop in accuracy?

5. Do action classifiers learn background?

6. Experiment Two recent challenging datasets:
UCF YouTube, 1100 Videos, 11 Actions
UCF Sports, 150 Videos, Actions
Actor Bounding Boxes are available for these datasets
Extract STIP (HOG, HOF)
50% Spatial-Temporal overlaps.
Single scale
Foreground Features: 50% overlap with bounding box
Background Features: Less than 50% overlap with bounding box
Dense Features: All features

7. Experiment Experimental Setup: Leave one group out for UCF YouTube
Leave one actor out for UCF Sports

8. STIP Features UCF YouTube UCF Sports
UCF YouTube Biking Video
UCF Sports Running Video
Foreground Features
STIP Features
UCF YouTube
UCF Sports
Foreground
59.80 %
71.92 %
Features with more than 50% overlap with actor bounding box

9. STIP Features UCF YouTube UCF Sports
UCF YouTube Biking Video
UCF Sports Running Video
Dense Features
STIP Features
UCF YouTube
UCF Sports
Dense
60.6%
75.34 %
Foreground
59.80 %
71.92 %
All features

10. Background 55.27 % 73.97 % UCF YouTube Biking Video
UCF Sports Running Video
Background Features
Comparable performance with only background features
without even observing the actor
STIP Features
UCF YouTube
UCF Sports
59.80 %
71.92 %
Dense
60.6 %
75.34 %
Background
55.27 %
73.97 %
Foreground
Features with less than 50% overlap with actor bounding box

11. Background should be diverse but not discriminative !

12. As action datasets are becoming large and more complex, their background may become more discriminative!

13. Action class discriminatively using GIST
Experiment 1
Compute GIST descriptor for KTH, UCF50, HMDB51 datasets
Cluster GIST descriptor in ‘k’ clusters using K-means clustering
Estimate point-wise mutual information between each cluster and
action class
𝑃𝑀𝐼 𝑥;𝑦 =𝑙𝑜𝑔 𝑝 𝑥,𝑦 𝑝 𝑥 𝑝(𝑦) =𝑙𝑜𝑔 𝑝(𝑥|𝑦) 𝑝(𝑥) =𝑙𝑜𝑔 𝑝(𝑦|𝑥) 𝑝(𝑦)

14. Action class discriminatively using GIST
Experiment 1 (Continue)
PMI Distance Matrices
KTH
UCF50
HMDB51
Clusters
100
200
300
KTH
5.12
8.25
10.4
HMDB51
7.15
11.07
14.06
UCF50
7.97
11.79
14.38
Small value means more interclass confusion
Based on scene information alone, KTH is harder to classify than HMDB51 and UCF50

15. Action class discriminatively using GIST
Experiment 2
Compute GIST descriptor for every 50th frame in KTH, UCF50, HMDB51 datasets
Construct a Graph, 𝐺=(𝑉,𝐸)
V= 𝑣 𝑖 , i ∈ 1, . . .,𝑛
is the set of descriptors
E= 𝑒 𝑖𝑗 , i ∈ 1, . . .,𝑛 ,j ∈ 1, . . .,𝑛
𝑒 𝑖𝑗 = 𝑣 𝑖 − 𝑣 𝑗 2
Graph Connected Component Analysis is performed by threshold ‘E’

16. Action class discriminatively using GIST
Experiment 2 (Continue)

17. Foreground Focused Representation
Our Approach
Foreground Focused Representation

18. Foreground Focused Representation
Action localization
Binary foreground/Background Segmentation
Very challenging and difficult, akin to introducing a new problem to solve the first.
Instead
Estimate the confidence in each pixel being a part of the foreground, and use it obtain video representation

19. Motion Gradients
Color Gradients

20. Visual Saliency
Fully connected graph is built, where Edge between two pixel is given as
By computing stationary distribution of Markov chain,
new graph is build
Equilibrium distribution of chain is used as per pixel
saliency

21. Visual Saliency Due to camera motion, video saliency is noisy
Graph based Image Saliency ( NIPS 2006)

22. Coherence of Foreground Confidence
Initial aggregate of confidence map
𝑓 𝑎 = 𝑙𝑜𝑔 ( 𝑓 𝑚 𝑓 𝑐 + 𝑓 𝑠 +1)
The score is max-normalized for each frame of a video
The quality of labeling is given by:
𝐸 𝑤 = 𝜑 𝑝 𝜖 𝑉 𝐷 𝑝 𝑤 𝑝 +
(𝑝,𝑞)𝜖 𝑉 𝑉 𝑤 𝑝 − 𝑤 𝑞
𝐷 𝑝 𝑤 𝑝 = ( 𝑓 𝑎 − 𝑤 𝑝 ) 2
𝑉 𝑤 𝑝 − 𝑤 𝑞 =min⁡( ( 𝑤 𝑝 − 𝑤 𝑞 ) 2 ) ,𝛾)

23. Coherence of Foreground Confidence (continue)
Inference
The message the node p send to q is given by
𝐷 𝑝 𝑤 𝑝 +𝑉 𝑤 𝑝 − 𝑤 𝑞 + 𝑠∈ 𝑁 𝑝 \q 𝑚 𝑠→𝑝 𝑡−1 ( 𝑤 𝑝 )
𝑤 𝑝 𝑚𝑖𝑛
The belief vector of node q is given by
𝑏 𝑞 𝑡 ( 𝑤 𝑞 )= 𝐷 𝑞 𝑤 𝑞 +
𝑠∈ 𝑁 𝑞 𝑚 𝑠→𝑡 𝑡 ( 𝑤 𝑞 )

24. Spatial-Temporal Coherence using 3D-MRF
Obtain probability of each pixel being the foreground using
Spatial-Temporal Coherence using 3D-MRF
Motion Gradients
Color Gradients
Video
Saliency
Final weights

25. Examples: Final Foreground weights
UCF50 Pull up
UCF50 Basketball
Olympic Sports
Diving
HMDB51 ride-horse
UCF50 Golf swing
HMDB51 ride-bike
HMDB51 ride-bike
Olympic Sports
Pole vault

26. UCF YouTube Biking Video
Traditional Bag of words
UCF YouTube Biking Video
Histogram
Foreground words
Background words

27. Weighted k-means
To make codebook biased towards foreground features, use weights of features during clustering
The confidence of each descriptors as being on foreground in given by:
𝑤 𝑖 = (𝑥,𝑦)∈ 𝑃 𝑖 𝑓 𝑎 (𝑥,𝑦) 𝑃 𝑖
The goal of clustering is to minimize the following energy function:
𝑗=1 𝐾 𝐶(𝑖,𝑗) 𝑤 𝑖 𝑥 𝑖 − 𝑧 𝑗 2
𝐶 𝑖𝑠 𝑋×𝑘 𝑖𝑠 𝑢𝑛𝑘𝑛𝑜𝑤𝑛 𝑚𝑒𝑚𝑠ℎ𝑖𝑝 𝑚𝑎𝑡𝑟𝑖𝑥

28. Example

29. Weighted Histogram
To reduce the contribution of background features, use weights for each features of being foreground during quantization

30. UCF YouTube Biking Video
Weighted bag of words
UCF YouTube Biking Video
Weighted Histogram
Background words
Foreground words
Weighted-kmeans

31. Problem No separate foreground and background words or vocabulary
The large number of background features can sum up to be significant.

32. UCF YouTube Biking Video
Weighted bag of words
UCF YouTube Biking Video
Weighted Histogram
Background words
Foreground words
Weighted-kmeans

33. UCF YouTube Biking Video
Weighted bag of words
UCF YouTube Biking Video
Weighted Histogram
Background words
Foreground words
Weighted-kmeans

34. UCF YouTube Biking Video
Weighted bag of words
UCF YouTube Biking Video
Weighted Histogram
Background words
Foreground words
Weighted-kmeans

35. Foreground confidence based Histogram decomposition
Categorize spatiotemporal regions corresponding to different weights
in ‘𝑅’ classes
Compute Histograms for each region separately
The regions of two videos that has same foreground confidence are
compared only with each other
The kernel function becomes
∆ 𝐻 𝑖 , 𝐻 𝑗 = 𝑟=1 𝑅 𝛼 𝑟 𝜃( ℎ 𝑟 𝑖 , ℎ 𝑟 𝑗 )
𝛼 𝑖 ={1, 0.8, 0.6, 0.4, 0.2}
𝑅=5

36. Partition based weighted histograms1
Features partitions based on weights

37. UCF50 Video
Weighted Histograms
Weighted Histograms
HMDB51 Video
Histogram Intersection
Weights Partitions
Weights Partitions
Weighted Summations
Final Similarity

38. Experimental Results Datasets used: UCF50, HMDB51, Olympic Sports
Features used:
STIP
UCF50 Vs. HMDB51
10 common actions
We choose actions which are visually similar:
Biking, Golf Swing, Pull Ups, Horse Riding, Basketball
UCF50 Vs. Olympic Sport
6 common actions:
Basketball, Pole Vault, Tennis serve, Diving, Clean and Jerk,
Throw Discus

39. Qualitative Results Biking UCF 50 HMDB51
Histogram Intersection=
Weighted Histogram Intersection=0.1142
Weighted Histogram Decomposition =0.1295

40. Qualitative Results Golf Swing UCF 50 HMDB51
Histogram Intersection=
Weighted Histogram Intersection=0.2740
Weighted Histogram Decomposition =0.3089

41. Quantitative Results Pull Ups UCF 50 HMDB51
Histogram Intersection=
Weighted Histogram Intersection=0.5454
Weighted Histogram Decomposition =0.5586

42. Qualitative Results Training Testing Unweighted Weighted UCF50 70.00
74.20
77.85
HMDB51
55.70
60.00
68.70
65.30
69.30
68.00
63.3
64.00
68.67
Olympic Sports
71.80
73.95
69.79
31.25
33.33
16.67
32.29
47.91
Histogram
Decomposition

43. Quantitative Results Confusion Matrix UCF50 classifiers on HMDB51
Unweighted
Histogram Decomposition

44. Thank you