1. RANKING David Kauchak CS 451 – Fall 2013

2. Admin Assignment 4 Assignment 5

3. Course feedback Thanks!

4. Course feedback Difficulty

5. Course feedback Time/week

6. Course feedback Experiments/follow-up questions More interesting data sets More advanced topics I’m hoping to get to: - large margin classifiers - probabilistic modeling - Unsupervised learning/clustering - Ensemble learning - Collaborative filtering - MapReduce/Hadoop

7. Java tip for the day static ArrayList data = dataset.getData(); How can I iterate over it?

8. Java tip for the day ArrayList data = dataset.getData(); for( int i = 0; i < data.size(); i++ ){ Example ex = data.get(i) } OR // can do on anything that implements the Iterable interface for( Example ex: data ){ }

9. An aside: text classification Raw datalabels Chardonnay Pinot Grigio Zinfandel

10. Text: raw data Raw dataFeatures?labels Chardonnay Pinot Grigio Zinfandel

11. Feature examples Raw data Features (1, 1, 1, 0, 0, 1, 0, 0, …) clintonsaidcaliforniaacrosstvwrongcapital pinot Clinton said pinot repeatedly last week on tv, “pinot, pinot, pinot” Occurrence of words labels Chardonnay Pinot Grigio Zinfandel

12. Feature examples Raw data Features (4, 1, 1, 0, 0, 1, 0, 0, …) clintonsaidcaliforniaacrosstvwrongcapital pinot Clinton said pinot repeatedly last week on tv, “pinot, pinot, pinot” Frequency of word occurrences labels Chardonnay Pinot Grigio Zinfandel This is the representation we’re using for assignment 5

13. Decision trees for text Each internal node represents whether or not the text has a particular word

14. Decision trees for text wheat is a commodity that can be found in states across the nation

15. Decision trees for text The US views technology as a commodity that it can export by the buschl.

16. Printing out decision trees (wheat (buschl predict: not wheat (export predict: not wheat predict: wheat)) (farm (commodity (agriculture predict: not wheat predict: wheat) predict: wheat))

17. Ranking problems Suggest a simpler word for the word below: vital

18. Suggest a simpler word Suggest a simpler word for the word below: vital wordfrequency important13 necessary12 essential11 needed8 critical3 crucial2 mandatory1 required1 vital1

19. Suggest a simpler word Suggest a simpler word for the word below: acquired

20. Suggest a simpler word Suggest a simpler word for the word below: acquired wordfrequency gotten12 received9 gained8 obtained5 got3 purchased2 bought2 got hold of1 acquired1

21. Suggest a simpler word vital important necessary essential needed critical crucial mandatory required vital gotten received gained obtained got purchased bought got hold of acquired … training data train ranker list of synonyms list ranked by simplicity

22. Ranking problems in general ranking1ranking2ranking3 f 1, f 2, …, f n training data: a set of rankings where each ranking consists of a set of ranked examples … train ranker ranking/ordering or examples f 1, f 2, …, f n

23. Ranking problems in general ranking1ranking2ranking3 f 1, f 2, …, f n training data: a set of rankings where each ranking consists of a set of ranked examples … Real-world ranking problems?

24. Netflix My List

25. Search

26. Ranking Applications reranking N-best output lists - machine translation - computational biology - parsing - … flight search …

27. Black box approach to ranking Abstraction: we have a generic binary classifier, how can we use it to solve our new problem binary classifier +1 optionally: also output a confidence/score Can we solve our ranking problem with this?

28. Predict better vs. worse ranking1 f 1, f 2, …, f n Train a classifier to decide if the first input is better than second: - Consider all possible pairings of the examples in a ranking - Label as positive if the first example is higher ranked, negative otherwise

29. Predict better vs. worse ranking1 f 1, f 2, …, f n Train a classifier to decide if the first input is better than second: - Consider all possible pairings of the examples in a ranking - Label as positive if the first example is higher ranked, negative otherwise f 1, f 2, …, f n new examplesbinary label +1 +1

30. Predict better vs. worse binary classifier +1 Our binary classifier only takes one example as input

31. Predict better vs. worse binary classifier +1 Our binary classifier only takes one example as input a 1, a 2, …, a n b 1, b 2, …, b n f’ 1, f’ 2, …, f’ n’ How can we do this? We want features that compare the two examples.

32. Combined feature vector Many approaches! Will depend on domain and classifier Two common approaches: 1. difference: 2. greater than/less than:

33. Training f 1, f 2, …, f n new exampleslabel +1 +1 f’ 1, f’ 2, …, f’ n’ extract features train classifier binary classifier

34. Testing unranked f 1, f 2, …, f n binary classifier ranking?

35. Testing unranked f 1, f 2, …, f n extract features f’ 1, f’ 2, …, f’ n’

36. Testing f 1, f 2, …, f n extract features f’ 1, f’ 2, …, f’ n’ binary classifier +1 +1

37. Testing f 1, f 2, …, f n +1 +1 What is the ranking? Algorithm?

38. Testing f 1, f 2, …, f n +1 +1 f 1, f 2, …, f n for each binary example e jk : label[j] += f jk (e jk ) label[k] -= f jk (e jk ) rank according to label scores

39. An improvement? ranking1 f 1, f 2, …, f n new examplesbinary label +1 +1 Are these two examples the same?

40. Weighted binary classification ranking1 f 1, f 2, …, f n new examplesweighted label +1 +2 +1 -2 Weight based on distance in ranking

41. Weighted binary classification ranking1 f 1, f 2, …, f n new examplesweighted label +1 +2 +1 -2 In general can weight with any consistent distance metric Can we solve this problem?

42. Testing If the classifier outputs a confidence, then we’ve learned a distance measure between examples During testing we want to rank the examples based on the learned distance measure Ideas?

43. Testing If the classifier outputs a confidence, then we’ve learned a distance measure between examples During testing we want to rank the examples based on the learned distance measure Sort the examples and use the output of the binary classifier as the similarity between examples!

44. Ranking evaluation ranking f 1, f 2, …, f n 1234512345 prediction 1325413254 Ideas?

45. Idea 1: accuracy ranking f 1, f 2, …, f n 1234512345 prediction 1/5 = 0.2 Any problems with this? 1325413254

46. Doesn’t capture “near” correct ranking f 1, f 2, …, f n 1234512345 prediction 1/5 = 0.2 prediction 1543215432 1325413254

47. Idea 2: correlation ranking 1234512345 prediction 1543215432 1325413254 Look at the correlation between the ranking and the prediction