1. Coached Active Learning for Interactive Video Search Xiao-Yong Wei, Zhen-Qun Yang Machine Intelligence Laboratory College of Computer Science Sichuan University, China

2. Coached Active Learning for Interactive Video Search Xiao-Yong Wei, Zhen-Qun Yang Machine Intelligence Laboratory College of Computer Science Sichuan University, China

3. Xiao-Yong Wei, Zhen-Qun Yang Machine Intelligence Laboratory College of Computer Science Sichuan University, China Coached Active Learning for Interactive Video Search

4. Xiao-Yong Wei, Zhen-Qun Yang Machine Intelligence Laboratory College of Computer Science Sichuan University, China

6. Search “car on road” with Google

9. Similar to the Automatic Search task in TRECVID

10. Relevance Feedback Interactive Search

11. Relevance Feedback Interactive Search Query Modeling, i.e., to figure out what the searcher wants?

12. Relevance Feedback Interactive Search Query Modeling, i.e., what the searcher wants? Querying Strategy, i.e., how to keep the searcher’s patience on labeling?

13. Query Modeling is indeed a task of Space Exploration

14. Search & Space Exploration

16. Feature Space with Unlabeled Instances

21. Space Exploration for Query Modeling How to explore effectively? Querying Strategy: which instances to query next (i.e., being presented to the searcher for labeling)? Query Distribution

22. The Goal of Query Modeling in Terms of Space Exploration A reasonable querying strategy –To find the Query Distribution (i.e., the distribution of the relevant instances) ASAP –To satisfy Searchers with Relevant Samples ASAP

23. Active Learning with Uncertainty Sampling – A Popular and Effective Strategy

24. Active Learning with Uncertainty Sampling The 1 st Round – Training Classifier

25. Active Learning with Uncertainty Sampling The 1 st Round – Query the Searcher

26. Active Learning with Uncertainty Sampling The 2 nd Round – Training Classifier

27. Active Learning with Uncertainty Sampling The 2 nd Round – Query the Searcher

28. Active Learning with Uncertainty Sampling The 3 rd Round

29. Active Learning with Uncertainty Sampling The 4 th Round

30. Active Learning with Uncertainty Sampling The 5 th Round

31. Active Learning with Uncertainty Sampling

32. Active Learning with Uncertainty Sampling may not be a “kindred soul” with Video Search

34. The spirit of Active Learning (to reduce labeling efforts) The goal of Query Modeling (to harvest relevant instances) Also found by A.G. Hauptmann and et al. [MM’06] Active Learning with Uncertainty Sampling may not be a “kindred soul” with Video Search

35. The Dilemma of Exploration and Exploitation Exploration: to explore more unknown area and get more about the Query Distribution and to boost future gains Exploitation: to harvest more relevant instances and to obtain immediate rewards

36. Our Idea – The Query Distribution, even Unknown, is Predictable

37. The Idea of Coached Active Learning Estimate the Query Distribution after each round to avoid the risk of learning on a completely unknown distribution

38. The Idea of Coached Active Learning Estimate the Query Distribution pdf of the underlying Query Distribution

39. The Idea of Coached Active Learning Estimate the Query Distribution Query users with instances picked from both dense and uncertain areas of the distribution Balance the proportion of the two types of instances

40. The Idea of Coached Active Learning Modeling the Query Distribution Balancing the Exploration and Exploitation

41. Modeling the Query Distribution

42. L+ : An incomplete sampling of the Query Distribution ? ? ? ? ? ? ? ? ?

43. Modeling the Query Distribution An indirect way of estimating the pdf of the Query Distribution –Find training examples which are statistically from the same distribution as L+ –Use the pdf(s) of those training examples to estimate that of the query ? ? ? ? ? ? ? ? ?

44. Modeling the Query Distribution A practical implementation –Training: organize training examples into nonexclusive semantic groups –Testing: check L+ with each group, see whether they are statistically from the same distribution using two-sample Hotelling T-Square Test –Testing: estimate Query Distribution with GMM ? ? ? ? ? ? ? ? ?

45. Modeling the Query Distribution Rationale of the implementation –Samples from the same distribution may carry similar semantics –Semantic groups (which passed the test) then reflect the searcher’s query intention

46. Modeling the Query Distribution

47. Distribution of the semantic group “road”

48. Modeling the Query Distribution Distribution of the semantic group “road” Distribution of the semantic group “car” Distribution of the query “car on road”

49. Balancing the Exploration and Exploitation

50. The priority of selecting an instance x to query next (i.e., present to the searcher for labeling) Exploitative PriorityExplorative Priority Balancing Factor

51. Harvest(x): Exploitative Priority How likely the exploitation will be boosted when x is selected to query next Current Decision Boundary

52. How likely the exploitation will be boosted when x is selected to query next Current Decision Boundary Harvest(x): Exploitative Priority

53. How likely the exploitation will be boosted when x is selected to query next Harvest(x): Exploitative Priority

54. How likely the exploitation will be boosted when x is selected to query next Harvest(x): Exploitative Priority

55. Explore(x): Explorative Priority How likely the exploration will be improved when x is selected to query next Entropy of the prior distribution Entropy of the posterior distribution

56. How likely the exploration will be improved when x is selected to query next Explore(x): Explorative Priority

57. λ: Balancing Factor Updated by investigating the Harvest History, i.e., numbers of relevant instances found in previous rounds The expected harvest

58. Experimental Results

59. Experiments Learning pdf(s) of the semantic groups –Large Scale Concept Ontology for Multimedia (LSCOM), 449 concept labeled on TRECVID 2005 development dataset –Using concept combination to create groups –Results: 23,064 groups and their pdfs (http://www.cs.cityu.edu.hk/~xiaoyong/CAL/)

60. Experiments Comparison with Uncertainty Sampling (US) –12 volunteers (age 19~26, 3 girls and 9 boys) –TREVID 2005 test dataset (45,765 video shots in English/Chinese/Arabic) –24 TRECVID queries and ground truth –Mean Average Precision (MAP) –Concept-based search for the initial round –Interface

61. Experiments Comparison with Uncertainty Sampling (US) –12 volunteers (age 19~26, 3 girls and 9 boys) –TREVID test dataset (45,765 video shots in English/Chinese/Arabic) –24 TRECVID queries and ground truth –Mean Average Precision (MAP) –Interface

62. Experiments Comparison with Uncertainty Sampling (US)

63. Experiments Comparison with Uncertainty Sampling (US) Harvest the shots by initial search

64. Experiments Comparison with Uncertainty Sampling (US) Harvest the shots in the most dense area

65. Experiments Comparison with Uncertainty Sampling (US) Drop down little a bit to encourage exploration and then remains stable to harvest the newly found relevant shots

66. Experiments Comparison with Uncertainty Sampling (US) Most relevant shots have been harvested, more and more emphasis on exploration

67. Experiments Comparison with Uncertainty Sampling (US) –Purely explorative US causes early give-up Most shots have been harvested, more and more emphasis on exploration “difficult” queries

68. Experiments Study of user patience on labeling

69. Experiments Can GMM stimulate the query distribution well? Query “the road with one more cars”

70. Experiments Comparison with the state-of-the-art –CAL (with virtual searchers) vs. The best results reported in TRECVID 06-09

71. Conclusions

72. Advantages of CAL –Predictable Query Distribution –Fast Convergence –Balance between Exploitation and Exploration in a principled way Toys and demos available at http://www.cs.cityu.edu.hk/~xiaoyong/CAL/

73. Conclusions Future work –Replace the Harvest() and Explore() with more advanced ones –Try it on larger datasets or web videos

74. Advantages of CAL Predictable Query Distribution Fast Convergence Uncertainty Sampling CAL, the 1 st round

75. Thanks ！