1. TransRank: A Novel Algorithm for Transfer of Rank Learning Depin Chen, Jun Yan, Gang Wang et al. University of Science and Technology of China, USTC Machine Learning Group, MSRA depin.chen@mail.ustc.edu.cn Page 1 2008-12-15

2. Content Ranking for IR Paper motivation The algorithm: TransRank Results & future work 2008-12-15 Page 2

3. Ranking in IR Ranking is crucial in information retrieval. It aims to move the good results up, while the bad down. A well known example: web search engine Page 3 2008-12-15

4. Learning to rank Ranking + Machine learning = Learning to rank An early work Ranking SVM, “Support Vector Learning for Ordinal Regression”, Herbrich et al [ICANN 99]. Page 4 2008-12-15

5. Learning to rank for IR Page 5 2008-12-15

6. Existing approaches Early ones Ranking SVM, RankBoost … Recently IRSVM, AdaRank, ListNet... Tie-Yan Liu’s team at MSRA Page 6 2008-12-15

7. Content Learning to rank in IR Paper motivation The algorithm: TransRank Results & future work Page 7 2008-12-15

8. Training data shortage Learning to rank relies on the full supply of labeled training data. In real world practice … Page 8 2008-12-15 Label data Labeling data is expensive  lack of training data Learn the model Bad generalization ability operation Poor performance

9. Transfer learning Transfer learning definition Transfer knowledge learned from different but related problems to solve current problem effectively, with fewer training data and less time [Yang, 2008]. –Learning to walk can help learn to run –learning to program with C++ can help learn to program with JAVA –…–… We follow the spirit of transfer learning in this paper. Page 9 2008-12-15

10. Content Learning to rank in IR Paper motivation The algorithm: TransRank Results & future work Page 10 2008-12-15

11. Problem formulation S t : training data in target domain S s : auxiliary training data from a source domain Note that, What we want? A ranking function for the target domain Page 11 2008-12-15

12. TransRank Three steps of TransRank: Page 12 2008-12-15 Step 1: K-best query selection Step 2: Feature augmentation Step 3: Ranking SVM

13. Step 1: K-best query selection Query’s ranking direction query 11 in OHSUMED query 41 in OHSUMED Page 13 2008-12-15

14. The goal of step 1: We want to select the queries from source domain who have the most similar ranking directions with the target domain data. These queries are treated to be most like the target domain training data. 2015-8-26 Microsoft ConfidentialPage 14

15. Utility function (1) Preprocess S s : select k best queries, and discard the rest. A “best” query is the query, whose ranking direction is confidently similar with that of queries in S t. The utility function combines two parts: confidence and similarity. Page 15 2008-12-15

16. Utility function (2) Confidence is valued using a separation value. The better different classes of instances are separated, the ranking direction will be more confident. Page 16 2008-12-15

17. Utility function (3) Cosine similarity. Page 17 2008-12-15

18. Step 2: Feature augmentation Daumé implemented cross-domain classification in NLP through a method called “feature augmentation” [ACL 07]. For source-domain document vector (1, 2, 3) (1, 2, 3)  (1, 2, 3, 1, 2, 3, 0, 0, 0) For target-domain document vector (1, 2, 3) (1, 2, 3)  (1, 2, 3, 0, 0, 0, 1, 2, 3) Page 18 2008-12-15

19. Step 3: Ranking SVM Ranking SVM is the state-of-the-art learning to rank algorithm, proposed by Herbrich et al [ICANN 99]. Page 19 2008-12-15

20. Content Learning to rank in IR Paper motivation The heuristic algorithm: TransRank Results & future work Page 20 2008-12-15

21. Experimental settings Datasets: OHSUMED (the LETOR version), WSJ, AP Features: feature set defined in OHSUMED. Same features are abstracted on WSJ and AP Evaluation measures: NDCG@n, MAP For Ranking SVM, we use SVM light by Joachims. Two group of experiments WSJ OHSUMED AP OHSUMED Page 21 2008-12-15

22. Compared algorithms Baseline: run Ranking SVM on S t TransRank Directly Mix: Step 1 + Step3 2015-8-26 Microsoft ConfidentialPage 22

23. Performance comparison 40% of target labeled data, k=10 source domain: WSJ source domain: AP Page 23 2008-12-15

24. Impact of target labeled data From 5% to 100%, k=10 source domain: WSJ source domain: AP Page 24 2008-12-15

25. Impact of k 40% of target labeled data Page 25 2008-12-15

26. Future work Web scale experiments, i.e. data from search engines More integrated algorithm using machine learning techniques Theoretical study for transfer of rank learning Page 26 2008-12-15

27. Q & A Page 27 2008-12-15

28. Thanks! Page 28 2008-12-15