Presentation is loading. Please wait.

Presentation is loading. Please wait.

Attribute Learning for Understanding Unstructured Social Activity

Similar presentations


Presentation on theme: "Attribute Learning for Understanding Unstructured Social Activity"— Presentation transcript:

1 Attribute Learning for Understanding Unstructured Social Activity
Yanwei Fu, Timothy M. Hospedales, Tao Xiang, and Shaogang Gong School of EECS, Queen Mary University of London, UK Presented by Amr El-Labban VGG Reading Group, Dec 5th 2012

2 Contributions Unstructured social activity attribute (USAA) dataset
Semi-latent attribute space Topic model based attribute learning

3 Objective Automatic classification of unstructured group social activity Use an attribute based approach Start with sparse, user defined attributes Add latent ones Learn jointly

4 Dataset 1500 videos, 8 classes
69 visual/audio attributes manually labelled Weak labelling SIFT, STIP and MFCC features used Data available (features, attributes, YouTube IDs)

5 Classification Standard classification Attribute based 𝐹 : 𝑋 𝑑 𝑍
𝐹 : 𝑋 𝑑 𝑍 Attribute based 𝐹=𝑆 𝐿 . , 𝐿: 𝑋 𝑑 𝑌 𝑝 ,𝑆: 𝑌 𝑝 𝑍 Map raw data to intermediate, lower dimensional space, then to classes.

6 Semi-Latent Attribute Space
Space consisting of: User defined attributes Discriminative latent attributes Non-discriminative (background) latent attributes

7 = Topic modelling 𝑃 𝑥 𝑖 𝑑 𝑗 = 𝑘 𝑃 𝑥 𝑖 | 𝑦 𝑘 𝑃( 𝑦 𝑘 | 𝑑 𝑗 ) d y d x x y
𝑃 𝑥 𝑖 𝑑 𝑗 = 𝑘 𝑃 𝑥 𝑖 | 𝑦 𝑘 𝑃( 𝑦 𝑘 | 𝑑 𝑗 ) d y d x x y = P(x|d) P(x|y) P(y|d) x – low level features (‘words’) y – attributes (‘topics’) d – ‘documents’

8 Latent Dirichlet Allocation
y x x – low level features y – attributes (user defined and latent) θ – attribute distribution φ – word distribution α, β – Dirichlet parameters

9 Aside: Dirichlet disribution
Distribution over multinomial distributions Parameterised by α α = (6,2,2) α = (3,7,5) α = (2,3,4) α = (6,2,6)

10 Aside: Dirichlet disribution
Important things to know: α 0 = α 𝑖 𝐸 𝑋 𝑖 = α 𝑖 α peak is closer to larger α values 𝑉𝑎𝑟 𝑋 𝑖 = α 𝑖 ( α 0 − α 𝑖 ) α ( α 0 +1) - large α gives small variance α<1 gives more sparse distributions

11 Latent Dirichlet Allocation
y x x – low level features y – attributes (user defined and latent) θ – attribute distribution φ – word distribution α, β – Dirichlet parameters

12 Latent Dirichlet Allocation
y x Generative model for each document: Choose θ ~ Dir(α) Choose φ ~ Dir(β) for each word: Choose y ~ Multinomial(θ) Choose x ~ Multinomial(φ y)

13 Latent Dirichlet Allocation
y x 𝑃 𝐷 𝛼,𝛽 = 𝑘=1 𝐾 𝑃( 𝜑 𝑘 |𝛽) 𝑚=1 𝑀 𝑃( 𝜃 𝑚 |𝛼) 𝑛=1 𝑁 𝑃 𝑦 𝑚,𝑛 𝜃 𝑚 𝑃( 𝑥 𝑚,𝑛 | 𝜑 𝑦 𝑚,𝑛 )

14 Latent Dirichlet Allocation
y x EM to learn Dirichlet parameters: α,β Approximate inference for posterior: 𝑃(θ,𝑦 |𝑥,α,β)

15 SLAS User defined part Latent part Per instance prior on α.
Set to zero when attribute isn’t present in ground truth Latent part First half “class conditional” One α per class. All but one constrained to zero. Second half “background” Unconstrained

16 Classification Use SLAS posterior to map from raw data to attributes
Use standard classifier (logistic regression) from attributes to classes

17 N-shot transfer learning
Split data into two partitions – source and target Learn attribute models on source data Use N examples from target to learn attribute-class mapping

18 Zero-shot learning Detect novel class
Manually defined attribute-class “prototype” Improve with self-training algorithm: Infer attributes for novel data NN matching in user defined space against protoype For each novel class: Find top K matches Train new prototype in full attribute space (mean of top K) NN matching in the full space

19 Experiments Compare three models: Direct: KNN or SVM on raw data
SVM-UD+LR: SVM to map raw data to attributes, LR maps attributes to classes SLAS+LR: SLAS to map raw data to attributes, LR learns classes based on user-defined and class conditional attributes.

20 MASSIVE HACK “The UD part of the SLAS topic profile is estimating the same thing as the SVM attribute classifiers, however the latter are slightly more reliable due to being discriminatively optimised. As input to LR, we therefore actually use the SVM attribute classier outputs in conjunction with the latent part of our topic profile.”

21 Results - classification
SLAS+LR better as number if training data and user defined attributes decreases Copes with 25% wrong attribute bits

22 Results - classification
KNN and SVM have vertical bands – consistent misclassification

23 Results – N-shot transfer learning
Vary number of user defined attributes SVM+LR cannot cope with zero attributes

24 Results – Zero-shot transfer learning
Two cases: Continuous prototype – mean attribute profile Binary prototype – thresholded mean Tested without background latent attributes (SLAS(NF))

25 Conclusion Augmenting SVM and user defined attributes with latent ones definitely helps. Experimental hacks make it hard to say how good the model really is…


Download ppt "Attribute Learning for Understanding Unstructured Social Activity"

Similar presentations


Ads by Google