Download presentation

Presentation is loading. Please wait.

Published byCameron Chandler Modified over 2 years ago

1
**Human Action Recognition across Datasets by Foreground-weighted Histogram Decomposition**

Waqas Sultani, Imran Saleemi CVPR 2014

2
Motivation

3
**Dense STIP for cross dataset recognition**

Training Testing Accuracy (avg) UCF50 70 % UCF50 HMDB51 55.7 % Olympic Sports 71.8 % UCF50 Olympic Sports 16.67 % Recognition Accuracy drops across the datasets! Training and Testing is done on similar actions across the datasets

4
**Is the background responsible for this drop in accuracy?**

UCF50 HMDB51 Is the background responsible for this drop in accuracy?

5
**Do action classifiers learn background?**

6
**Experiment Two recent challenging datasets:**

UCF YouTube, 1100 Videos, 11 Actions UCF Sports, 150 Videos, Actions Actor Bounding Boxes are available for these datasets Extract STIP (HOG, HOF) 50% Spatial-Temporal overlaps. Single scale Foreground Features: 50% overlap with bounding box Background Features: Less than 50% overlap with bounding box Dense Features: All features

7
**Experiment Experimental Setup: Leave one group out for UCF YouTube**

Leave one actor out for UCF Sports

8
**STIP Features UCF YouTube UCF Sports**

UCF YouTube Biking Video UCF Sports Running Video Foreground Features STIP Features UCF YouTube UCF Sports Foreground 59.80 % 71.92 % Features with more than 50% overlap with actor bounding box

9
**STIP Features UCF YouTube UCF Sports**

UCF YouTube Biking Video UCF Sports Running Video Dense Features STIP Features UCF YouTube UCF Sports Dense 60.6% 75.34 % Foreground 59.80 % 71.92 % All features

10
**Background 55.27 % 73.97 % UCF YouTube Biking Video**

UCF Sports Running Video Background Features Comparable performance with only background features without even observing the actor STIP Features UCF YouTube UCF Sports 59.80 % 71.92 % Dense 60.6 % 75.34 % Background 55.27 % 73.97 % Foreground Features with less than 50% overlap with actor bounding box

11
**Background should be diverse but not discriminative !**

12
As action datasets are becoming large and more complex, their background may become more discriminative!

13
**Action class discriminatively using GIST**

Experiment 1 Compute GIST descriptor for KTH, UCF50, HMDB51 datasets Cluster GIST descriptor in ‘k’ clusters using K-means clustering Estimate point-wise mutual information between each cluster and action class 𝑃𝑀𝐼 𝑥;𝑦 =𝑙𝑜𝑔 𝑝 𝑥,𝑦 𝑝 𝑥 𝑝(𝑦) =𝑙𝑜𝑔 𝑝(𝑥|𝑦) 𝑝(𝑥) =𝑙𝑜𝑔 𝑝(𝑦|𝑥) 𝑝(𝑦)

14
**Action class discriminatively using GIST**

Experiment 1 (Continue) PMI Distance Matrices KTH UCF50 HMDB51 Clusters 100 200 300 KTH 5.12 8.25 10.4 HMDB51 7.15 11.07 14.06 UCF50 7.97 11.79 14.38 Small value means more interclass confusion Based on scene information alone, KTH is harder to classify than HMDB51 and UCF50

15
**Action class discriminatively using GIST**

Experiment 2 Compute GIST descriptor for every 50th frame in KTH, UCF50, HMDB51 datasets Construct a Graph, 𝐺=(𝑉,𝐸) V= 𝑣 𝑖 , i ∈ 1, . . .,𝑛 is the set of descriptors E= 𝑒 𝑖𝑗 , i ∈ 1, . . .,𝑛 ,j ∈ 1, . . .,𝑛 𝑒 𝑖𝑗 = 𝑣 𝑖 − 𝑣 𝑗 2 Graph Connected Component Analysis is performed by threshold ‘E’

16
**Action class discriminatively using GIST**

Experiment 2 (Continue)

17
**Foreground Focused Representation**

Our Approach Foreground Focused Representation

18
**Foreground Focused Representation**

Action localization Binary foreground/Background Segmentation Very challenging and difficult, akin to introducing a new problem to solve the first. Instead Estimate the confidence in each pixel being a part of the foreground, and use it obtain video representation

19
Motion Gradients Color Gradients

20
Visual Saliency Fully connected graph is built, where Edge between two pixel is given as By computing stationary distribution of Markov chain, new graph is build Equilibrium distribution of chain is used as per pixel saliency

21
**Visual Saliency Due to camera motion, video saliency is noisy**

Graph based Image Saliency ( NIPS 2006)

22
**Coherence of Foreground Confidence**

Initial aggregate of confidence map 𝑓 𝑎 = 𝑙𝑜𝑔 ( 𝑓 𝑚 𝑓 𝑐 + 𝑓 𝑠 +1) The score is max-normalized for each frame of a video The quality of labeling is given by: 𝐸 𝑤 = 𝜑 𝑝 𝜖 𝑉 𝐷 𝑝 𝑤 𝑝 + (𝑝,𝑞)𝜖 𝑉 𝑉 𝑤 𝑝 − 𝑤 𝑞 𝐷 𝑝 𝑤 𝑝 = ( 𝑓 𝑎 − 𝑤 𝑝 ) 2 𝑉 𝑤 𝑝 − 𝑤 𝑞 =min( ( 𝑤 𝑝 − 𝑤 𝑞 ) 2 ) ,𝛾)

23
**Coherence of Foreground Confidence (continue)**

Inference The message the node p send to q is given by 𝐷 𝑝 𝑤 𝑝 +𝑉 𝑤 𝑝 − 𝑤 𝑞 + 𝑠∈ 𝑁 𝑝 \q 𝑚 𝑠→𝑝 𝑡−1 ( 𝑤 𝑝 ) 𝑤 𝑝 𝑚𝑖𝑛 The belief vector of node q is given by 𝑏 𝑞 𝑡 ( 𝑤 𝑞 )= 𝐷 𝑞 𝑤 𝑞 + 𝑠∈ 𝑁 𝑞 𝑚 𝑠→𝑡 𝑡 ( 𝑤 𝑞 )

24
**Spatial-Temporal Coherence using 3D-MRF**

Obtain probability of each pixel being the foreground using Spatial-Temporal Coherence using 3D-MRF Motion Gradients Color Gradients Video Saliency Final weights

25
**Examples: Final Foreground weights**

UCF50 Pull up UCF50 Basketball Olympic Sports Diving HMDB51 ride-horse UCF50 Golf swing HMDB51 ride-bike HMDB51 ride-bike Olympic Sports Pole vault

26
**UCF YouTube Biking Video**

Traditional Bag of words UCF YouTube Biking Video Histogram Foreground words Background words

27
Weighted k-means To make codebook biased towards foreground features, use weights of features during clustering The confidence of each descriptors as being on foreground in given by: 𝑤 𝑖 = (𝑥,𝑦)∈ 𝑃 𝑖 𝑓 𝑎 (𝑥,𝑦) 𝑃 𝑖 The goal of clustering is to minimize the following energy function: 𝑗=1 𝐾 𝐶(𝑖,𝑗) 𝑤 𝑖 𝑥 𝑖 − 𝑧 𝑗 2 𝐶 𝑖𝑠 𝑋×𝑘 𝑖𝑠 𝑢𝑛𝑘𝑛𝑜𝑤𝑛 𝑚𝑒𝑚𝑠ℎ𝑖𝑝 𝑚𝑎𝑡𝑟𝑖𝑥

28
Example

29
Weighted Histogram To reduce the contribution of background features, use weights for each features of being foreground during quantization

30
**UCF YouTube Biking Video**

Weighted bag of words UCF YouTube Biking Video Weighted Histogram Background words Foreground words Weighted-kmeans

31
**Problem No separate foreground and background words or vocabulary**

The large number of background features can sum up to be significant.

32
**UCF YouTube Biking Video**

Weighted bag of words UCF YouTube Biking Video Weighted Histogram Background words Foreground words Weighted-kmeans

33
**UCF YouTube Biking Video**

Weighted bag of words UCF YouTube Biking Video Weighted Histogram Background words Foreground words Weighted-kmeans

34
**UCF YouTube Biking Video**

Weighted bag of words UCF YouTube Biking Video Weighted Histogram Background words Foreground words Weighted-kmeans

35
**Foreground confidence based Histogram decomposition**

Categorize spatiotemporal regions corresponding to different weights in ‘𝑅’ classes Compute Histograms for each region separately The regions of two videos that has same foreground confidence are compared only with each other The kernel function becomes ∆ 𝐻 𝑖 , 𝐻 𝑗 = 𝑟=1 𝑅 𝛼 𝑟 𝜃( ℎ 𝑟 𝑖 , ℎ 𝑟 𝑗 ) 𝛼 𝑖 ={1, 0.8, 0.6, 0.4, 0.2} 𝑅=5

36
**Partition based weighted histograms**

1 Features partitions based on weights

37
UCF50 Video Weighted Histograms Weighted Histograms HMDB51 Video Histogram Intersection Weights Partitions Weights Partitions Weighted Summations Final Similarity

38
**Experimental Results Datasets used: UCF50, HMDB51, Olympic Sports**

Features used: STIP UCF50 Vs. HMDB51 10 common actions We choose actions which are visually similar: Biking, Golf Swing, Pull Ups, Horse Riding, Basketball UCF50 Vs. Olympic Sport 6 common actions: Basketball, Pole Vault, Tennis serve, Diving, Clean and Jerk, Throw Discus

39
**Qualitative Results Biking UCF 50 HMDB51**

Histogram Intersection= Weighted Histogram Intersection=0.1142 Weighted Histogram Decomposition =0.1295

40
**Qualitative Results Golf Swing UCF 50 HMDB51**

Histogram Intersection= Weighted Histogram Intersection=0.2740 Weighted Histogram Decomposition =0.3089

41
**Quantitative Results Pull Ups UCF 50 HMDB51**

Histogram Intersection= Weighted Histogram Intersection=0.5454 Weighted Histogram Decomposition =0.5586

42
**Qualitative Results Training Testing Unweighted Weighted UCF50 70.00**

74.20 77.85 HMDB51 55.70 60.00 68.70 65.30 69.30 68.00 63.3 64.00 68.67 Olympic Sports 71.80 73.95 69.79 31.25 33.33 16.67 32.29 47.91 Histogram Decomposition

43
**Quantitative Results Confusion Matrix UCF50 classifiers on HMDB51**

Unweighted Histogram Decomposition

44
Thank you

Similar presentations

Presentation is loading. Please wait....

OK

Svetlana Lazebnik, Cordelia Schmid, Jean Ponce

Svetlana Lazebnik, Cordelia Schmid, Jean Ponce

© 2017 SlidePlayer.com Inc.

All rights reserved.

Ads by Google

Ppt on applied operational research techniques Ppt on needle stick injury images Ppt on peak load pricing economics Ppt on central limit theorem example Ppt on mass energy equation Ppt on solar distillation system Ppt on polynomials and coordinate geometry software Ppt on kinetic theory of matter/gas laws Ppt on 2 stroke engine Ppt on sickle cell disease