Presentation on theme: "1 Liyan Zhang et al. Liyan Zhang, Dmitri V. Kalashnikov, Sharad Mehrotra Department of Computer Science University of California, Irvine A Unified Framework."— Presentation transcript:
1 Liyan Zhang et al. Liyan Zhang, Dmitri V. Kalashnikov, Sharad Mehrotra Department of Computer Science University of California, Irvine A Unified Framework for Context Assisted Face Clustering
2 Liyan Zhang et al. Introduction Explosion of Media Data Human is Center Face Clustering Face Tagging User Feedback
3 Liyan Zhang et al. Outline Introduction to Face Clustering Traditional Approaches for Face Clustering The Proposed Context Assisted Framework Experimental Results Conclusions and Future Work
4 Liyan Zhang et al. Face Appearance based Approach Facial Features Face Similarity Graph Clustering Algorithm Detected Faces … Clustering Results
5 Liyan Zhang et al. Appearance based Face Clustering Results Good Clustering Results High Precision,High Recall Tight Clustering Threshold High Precision, Low Recall loose Clustering Threshold Low Precision, High Recall Too Much Merging Work!
6 Liyan Zhang et al. Drawbacks of Facial Similarities Same People Look Different Different PoseDifferent Expression Different IlluminationDifferent Occlusion Different People Look The Same BoyGirlBoyGirl
7 Liyan Zhang et al. Context Information Helps Common Scene: Geo Location Captured Time Image Background Social Context: People Co-occur Human Attributes: Age Ethnicity Gender Hair … Clothing: Cloth color
8 Liyan Zhang et al. Related Work  Y. J. Lee and K. Grauman. Face discovery with social context. In BMVC, 2011.  N. Kumar and et al. Describable visual attributes for face verification and image search. In IEEE TPAMI, 2011.  A. Gallagher and T. Chen. Clothing cosegmentation for recognizing people. In IEEE CVPR, 2008. People Co-occurrence  Clothing  Human Attributes  Heterogeneous Context FeatureSingle Context Type Face LevelCluster Level Context Prior work Context Heterogeneous Single
9 Liyan Zhang et al. The Framework Photo Collection Detected Faces … Initial Clusters : High Precision, Low Recall … Iterative Merging cont Common Scene People Co-occurrence Human Attributes Clothing … … Final Clusters: High Precision, High Recall
10 Liyan Zhang et al. Context Features Extraction Cluster level Common ScenePeople Co-occurrenceHuman AttributesClothing Context Similarities Context Constraints Integrate Same? Diff ?
11 Liyan Zhang et al. Common Scene Image captured time, camera model, image visual features Common Scene Same people I1I1 I2I2 I3I3 I1I1 I2I2 C1C1 C2C2
12 Liyan Zhang et al. People Co-occurrence same diff
13 Liyan Zhang et al. People Co-occurrence Cluster Co-Occurrence Graph 1 1 1 1 1 1 I2I2 I1I1 1 1 1 1 1 f 1 f 2 f 3 f 4 f 5 f 6 f 7 f 8
14 Liyan Zhang et al. Human Attributes samediff 73-D N. Kumar and et al. Describable visual attributes for face verification and image search. In IEEE TPAMI, 2011.
15 Liyan Zhang et al. Human Attributes Attri bute C5 f 1 f 2 f 3 Attri bute Similar? C5 cosine Only One Child Many Children! AGE attribute Bootstrapping: Learn Weights From Dataset! Different Attributes Different Weights
16 Liyan Zhang et al. Human Attributes FaceAttributesLabel C1C1 C1C1 C1C1 ~ C 1 Attri bute C5 f 1 f 2 f 3 Attri bute f 4 f 5 f 6 f 7 f 8 diff Attri bute Train Classifier C1 ?
17 Liyan Zhang et al. Clothing Similarity from clothes Cloth color hist similarity Time diff Time slot threshold Time Sensitive! DiffSame Time diff << S Time diff >> S
18 Liyan Zhang et al. Context Features Cluster-Level Context Similarities Common ScenePeople Co-occurrence Human Attributes Clothing
19 Liyan Zhang et al. Context Features Cluster-Level Context Constraints Diff Co-occurred people Diff Distinct Attributes Time & Location Time: t s Indoor Time: (t+1)s Outdoor Diff
20 Liyan Zhang et al. Single Context Feature Fails Common Scene People Co-occurrence Diff Same Different Attributes Different Clothing Co-occurred People Diff Same
21 Liyan Zhang et al. Integration is Required Aggregation? Set a rule? Context Similarities Context Constraints Integrate ? Context Features Merge Not Y=a +b +c +d +e +… The importance of features differ with different dataset! Learn Rules from Each Dataset!
22 Liyan Zhang et al. How to Learn Rules? Photo Collection Split Initial clusters Same Pairs Manually Label Learning Rules Apply Rules Training Dataset … Initial Clusters : High Precision, Low Recall Facial Features cont Common Scene People Co-occurrence Human Attributes Clothing … Context Similarity & Constraint Learn from data Itself! Apply rules Learning Rules Training Dataset Automatic Label Context Constraints Diff Pairs Bootstrapping
23 Liyan Zhang et al. pairsLabel Same Diff Cost-sensitive DTC Splitting Training Diff Example of Automatic Labeling same diff same diff same
24 Liyan Zhang et al. pairsLabel Same Diff Cost-sensitive DTC pairspredict Same Diff Same Diff …… ( ): 5 same ( ): 1 same Splitting Training Predicting Diff 1 st Splitting—Training--Predicting
25 Liyan Zhang et al. pairsLabel Same Diff Cost-sensitive DTC pairspredict Same Diff …… ( ): 4 same ( ): 0 same Splitting Training Predicting Diff 2 nd Splitting—Training--Predicting
26 Liyan Zhang et al. Combine results C1-C3: 5 same C2-C3: 1 same C1-C3: 4 same C2-C3: 0 same C1-C3: 9 same C2-C3: 1 same Merge C1-C3 pairspredict Same Diff Same Diff …… pairspredict Same Diff …… 1 st Time 2 nd Time
27 Liyan Zhang et al. Unified Framework Pure clusters splittingtrainingprediction splittingtrainingprediction splittingtrainingprediction … Final Decision Extracted Faces Merge Pairs? YES No Results Iterative Merging Photo Album Facial Context Features
29 Liyan Zhang et al. Evaluation Metrics B-cubed Precision and Recall
30 Liyan Zhang et al. Performance Comparison Facial Features Photo Album Context Features Pure Clusters Splitting Training Predicting Process Merge Decision Update Our Approach: Precision Recall Picasa: Cluster Threshold 50 95 Different Clusters Affinity Propagation: Context Similarities Aggregation Facial Similarities Different Parameter: p Different Clusters
36 Liyan Zhang et al. Conclusion and Future Work Heterogeneous Context Features Context Constraint Co-occur People Distinct Attributes Time & Space Context Similarity Common Scene People Co-occur Human Attributes Clothing Single Context Feature Context Similarity Prior work Our Approach Efficiency?User Feedback? Break points for precision dropping? Future work Bootstrapping Integration Iterative Merging High precision High recall