A New Method for Automatic Clothing Tagging Utilizing Image-Click-Ads Introduction Conclusion Can We Do Better to Reduce Workload?

A Tradeoff between Workload and Accuracy Low workload Low accuracy High workload High accuracy Low Workload, High Accuracy? Utilizing Image-Click-Ads Introduction Conclusion Manual Tagging Tool Assisted Tagging Tool

The Objective Automatically segment object regions in an image With intuitive tagging interface For the purpose of achieving low workload, high accuracy for the tagging task Utilizing Image-Click-Ads Introduction Conclusion

The Research Problem To design a new automatic segmentation method for: Reducing workload Producing high accuracy tagged images In the domain of clothing images Utilizing Image-Click-Ads Introduction Conclusion

Background: Image Parsing Models Utilizing Image-Click-Ads Introduction Conclusion Image Parsing (Semantic Segmentation) Unary TermPairwise Term MRF Probability of each node for each class Local neighborhood knowledge CRF Probability of each node for each class More complex prior knowledge CRF is better CRF example

Color-category labels [Liu et al 2014]: blue shirt Data driven method [Yamaguchi et al 2013]: retrieve similar images from dataset CRF based method [Yamaguchi et al 2012] Related Work: Clothing Parsing Utilizing Image-Click-Ads Introduction Conclusion fashion images == common images? Unary term Prior distribution: pairwise co-occurrence of clothing labels Prior distribution: Probability of neighboring pairs having the same label

Drawbacks (1/5) Utilizing Image-Click-Ads Introduction Conclusion Problem 1: Fails to Preserve Local Contrast Cause: Large variations in configuration and appearance, resulting in unreliable prior distribution knowledge, a common problem of CRF.

Drawbacks (2/5) Utilizing Image-Click-Ads Introduction Conclusion Problem 2: Background Spill Cause: feature similarity

Drawbacks (3/5) Utilizing Image-Click-Ads Introduction Conclusion Problem 3: Incomplete Region Prediction Cause: occlusion

Drawbacks (4/5) Utilizing Image-Click-Ads Introduction Conclusion Problem 4: Over Smoothing of Infrequent Label Mentioned in [Yamaguchi et al 2012] Cause: unbalanced labeled data (low frequency, small region) Unbalanced Labeled Data For example Probability = 0.4

Drawbacks (5/5) Utilizing Image-Click-Ads Introduction Conclusion Problem 5: High Computational Cost Cause: hierarchical segmentation method

A deviation of MRF (Re-Weighted MRF) by introducing Background Prior : addressing background spill problem Occlusion Prior: addressing occlusion problem A Re-weighted Pairwise Term: addressing over smoothing of infrequent labels problem Integrated with SLIC superpixel segmentation method Addressing high computational cost Obtains better performance of clothing parsing in Parsing accuracy Processing efficiency The Proposed Method Utilizing Image-Click-Ads Introduction Conclusion

The Proposed Method: Workflow pose estimation superpixel background prior occlusion prior Re-Weighted MRF Utilizing Image-Click-Ads Introduction Conclusion

Two New Priors Utilizing Image-Click-Ads Introduction Conclusion Background Prior Occlusion Prior NON: neighbors of neighborhood For VFor V*

Re-weighted Pairwise Term Utilizing Image-Click-Ads Introduction Conclusion Probability prediction Original Pairwise Term

Online Image-Click-Ads Tagging System: EyeDentifyIt 3.0

Evaluation: Training and Inference Training data set  Fashionista [Yamaguchi et al. 2012] in EyeDentifyIt Feature vector  RGB color [m X n X 3]  CIELAB color [m X n X 3]  Gabor feature [m X n X 4]  Absolute 2D coordinates [m X n X 2]  Relative 2D coordinates [m X n X 28] Training and inference  Logistic regress (liblinear library)  Max-flow min-cut (gco-v3.0) Utilizing Image-Click-Ads Introduction Conclusion 37

Quantitative Evaluation (1/2) Utilizing Image-Click-Ads Introduction Conclusion Compare Pixel ACC and MAGR among MRF Reweighted Pairwise Term (RW) Reweighted Pairwise Term + Background Prior (RW+BP) Reweighted Pairwise Term + Background Prior + occlusion prior (RW+BP+OP) Result 1: steady improvements with more priors Result 2: RW reaches the best MAGR 38

method Pixel AccMAGR Training Time Processing Time Re-Weighted MRF 90.5%63.0%631.8 sec5.2 sec [Yamaguchi et al 2012] 85.1%57.2% sec81.5 sec Baseline 77.6%12.8%N/A Quantitative Evaluation (2/2) Utilizing Image-Click-Ads Introduction Conclusion Compare between Re-Weighted MRF CRF [Yamaguchi et al 2012] Baseline: naively predict to be all background Result 3: 5.4% gain on pixel ACC, 5.8% gain on MAGR, 86.1% gain on training time, 93.6% on processing time 39

Qualitative Evaluation (1/3) CRF [Yamaguchi et al 2012] Re-Weighted MRFMRF Resolved P1: Fails to preserve local contrast P4: Over smoothing of infrequent label 40

Qualitative Evaluation (2/3) CRF [Yamaguchi et al 2012] RW+BPRW+BP+OP Resolved P2: Background spill P3: Incomplete region prediction 41

Contributions Developed a new automated clothing parsing method Proposed background prior, occlusion prior to resolve background spill and occlusion problem in clothing parsing Proposed re-weighted pairwise term for MRF model to justify infrequent small label prediction Demonstrated MRF performs better than CRF in conditions that the local knowledge is more trust worthy than the statistical model learned from training dataset Integrated in EyeDentifyIt 3.0, driven by Image-Click- Ads framework Utilizing Image-Click-Ads Introduction Conclusion 42

Thank You ！ 谢 谢！ 46