Visual Event Recognition in Videos by Learning from Web Data Lixin Duan†, Dong Xu†, Ivor Tsang†, Jiebo Luo¶ † Nanyang Technological University, Singapore ¶ Kodak Research Labs, Rochester, NY, USA

Outline Overview of the Event Recognition System Similarity between Videos Aligned Space-Time Pyramid Matching Cross-Domain Problem Adaptive Multiple Kernel Learning Experiments Conclusion

Overview GOAL: Recognize consumer videos Large intra-class variability; limited labeled videos Wedding Sports Picnic  

A Large Number of Web Videos Overview GOAL: Recognize consumer videos by leveraging a large number of loosely labeled web videos (e.g., from YouTube) Wedding Sports Picnic   Consumer Videos A Large Number of Web Videos

Overview Flowchart of the system Video Database Test video Classifier Output

Similarity between Videos Pyramid matching methods Temporally aligned pyramid matching, D. Xu and S.-F. Chang [1] Unaligned space-time pyramid matching, I. Laptev [2] Space-time axes Time axis Space axes

Similarity between Videos  

Similarity between Videos Aligned Space-Time Pyramid Matching Level 1 Distance    

Similarity between Videos   Distance Integer-flow Earth Mover’s Distance (EMD), Y. Rubner [3]     s.t.

Similarity between Videos Distance     Integer-flow Earth Mover’s Distance (EMD), Y. Rubner [3]     s.t.

Cross-Domain Problem Data distribution mismatch between consumer videos and web videos Consumer videos: Naturally captured Web videos: Edited; Selected Maximum Mean Discrepancy (MMD), K. M. Borgwardt [4]        

Cross-Domain Problem   Prior information    

Cross-Domain Problem            

Cross-Domain Problem Adaptive Multiple Kernel Learning (A-MKL) MMD Structural risk functional   where    

Cross-Domain Problem      

Cross-Domain Problem  

Experiments Data set 195 consumer videos and 906 web videos collected by ourselves and from Kodak Consumer Video Benchmark Data Set [5] 6 events: “wedding”, “birthday”, “picnic”, “parade”, “show” and “sports” Training data: 3 videos per event from consumer videos and all web videos Test data: The rest consumer videos

Experiments  

Experiments Aligned Unaligned Aligned Space-Time Pyramid Matching (ASTPM) vs. Unaligned Space-Time Pyramid Matching (USTPM) ASTPM is better than USTPM at Level 1

Experiments    

Experiments Comparisons of cross-domain learning methods (a) SIFT features (b) ST features (c) SIFT features and ST features “parade”: 75.7% (A-MKL) vs. 62.2% (FR)

Experiments Comparisons of cross-domain learning methods Relative improvements SVM_T: 36.9% SVM_AT: 8.6% Feature Replication (FR) [6]: 7.6% Adaptive SVM (A-SVM) [7]: 49.6% Domain Transfer SVM (DTSVM) [8]: 9.9% MKL-based methods Better fuse SIFT features and ST features Handle noise in the loose labels

Conclusion We propose a new event recognition framework for consumer videos by leveraging a large number of loosely labeled web videos. We develop a new aligned space-time pyramid matching method. We present a new cross-domain learning method A-MKL which handles the mismatch between the data distributions of the consumer video domain and the web video domain.

References [1] D. Xu and S.-F. Chang. Video event recognition using kernel methods with multi-level temporal alignment. T-PAMI, 30(11):1985–1997, 2008. [2] I. Laptev, M. Marszałek, C. Schmid, and B. Rozenfeld. Learning realistic human actions from movies. In CVPR, 2008. [3] Y. Rubner, C. Tomasi, and L. J. Guibas. The Earth mover’s distance as a metric for image retrieval. IJCV, 40(2): 99-121, 2000. [4] K. M. Borgwardt, A. Gretton, M. J. Rasch, H.-P. Kriegel, B. Schölkopf, and A. Smola. Integrating structured biological data by kernel maximum mean discrepancy. In ISMB, 2006.

References [5] F. Bach, G. R. G. Lanckriet, and M. I. Jordan. Multiple kernel learning, conic duality and the SMO algorithm. In ICML, 2004. [6] H. Daumé III. Frustratingly easy domain adaptation. In ACL, 2007. [7] L. Duan, I. W. Tsang, D. Xu, and S. J. Maybank. Domain transfer svm for video concept detection. In CVPR, 2009. [8] J. Yang, R. Yan, and A. G. Hauptmann. Cross-domain video concept detection using adaptive svms. In ACM MM, 2007. [9] D. G. Lowe. Distinctive image features from scale-invariant keypoints. IJCV, 60(2):91–110, 2004.

Thank you!