1. Learning to Generalize for Complex Selection Tasks Alan Ritter University of Washington Sumit Basu Microsoft Research research.microsoft.com/~sumitb/smartselection IUI 2009

2. Outline 1.Smart Selection 2.Learning to Generalize 3.User Study 4.Conclusions

3. Multiple Selection Files HTML list boxes E-mail PowerPoint objects Spreadsheets Etc… Complex selection tasks currently require programming knowledge – Unix Shell – Regular Expressions

4. Our Task: File Selection

5. Tedious Selection Tasks Files do not group together by sorting – Substring of file name (e.g. copy or backup) Users forced to click on a large number of files

6. Smart Selection Selection Classifier Label Classify

7. Related Work Text editing – LAPIS (Miller and Myers IUI 02) DOM extraction – REFORM (Toomim et. al.CHI 09) – KARMA (Tuchinda et. al. IUI 08) Other Domains – Image regions: Crayons (Fails and Olsen IUI 03) – Image search: CueFlik (Fogarty et. al. CHI 08)

8. Selection Classifier Label Classify one session Few Labels Available

9. Selection Classifier Label Classify many users, many sessions one session How to use historical tasks?

10. Our Contributions 1.Make use of many peoples historical data – Learning to Generalize 2.Flexible selection classifier – Works well for the File Browser Domain

11. Demo

12. Outline 1.Smart Selection 2.Learning to Generalize 3.User Study 4.Conclusions

13. Basic Classification Framework Selection Classifier: Boosted Decision Trees Limited depth (2) Adjustable complexity Features: File name substrings File extension Creation Date Size Selection Classifier Label Classify Limited Training Data Available

14. How can we improve? bad idea: Heuristics about users behavior? better option: Learn to generalize from Historical Data!

15. Example Behavioral Feature Foo.py Food.txt Foo2.py Bar.py FBaz.py BFoo.doc FBFoo.py BazFoo.py Foo.py Foo2.py FBaz.py FBFoo.py Bar.py Positive Evidence? Learn this from Data!

16. Selection Classifier Label Classify many users, many sessions one session How do we Learn from Behavioral Data?

17. α Selection Classifier Label Classify many users, many sessions Label Regressor Behavior Features one session Predict Labels for Unlabeled Data α

18. Extract Features Labels Training the Label Regressor Step 1Step 2 … Step n Step 1Step 2Step 3 User Applies some operation on files

19. Lots of labeled data! m files, n steps, j tasks, k users Produces labeled examples Plenty of data available for LR! – No need to manually label – Personalization

20. Training Selection Classifier Explicit Labels Implicit Labels – Label Regressor produces a label and a weight/confidence – Weight modulated by Selection Classifier Label Classify Label Regressor α α

21. Recap of Our Method Label Regressor – Features based on Users Behavior – Makes weighted predictions – Trained on historical data Selection Classifier – Predicts which items to select – Trained on: Users explicit examples Modulated predictions from Label Regressor α

22. Outline 1.Smart Selection 2.Learning to Generalize 3.User Study 4.Conclusions

23. User Study 9 User pilot study – Gathered training data for LR Full Study: 12 Participants 3 Conditions: – A: Standard shell (control) – B: Selection Classifier, but no Label Regressor – C: Selection Classifier + Label Regressor

24. Tasks 8 tasks for each condition – Isomorphic Widely varying – Half were easy with sorting/block select – Widely varying directory sizes

25. Number of Examples manual selection smart selection, explicit labels only smart selection, explicit + soft labels task # examples

26. How accurate are LR posteriors?

27. Selection Accuracy smart selection, explicit labels only smart selection, explicit + soft labels task accuracy

28. Closer to goal in early rounds smart selection, explicit labels only smart selection, explicit + soft labels step accuracy Advantages: Less Dramatic changes Switch to manual & quickly complete Advantages: Less Dramatic changes Switch to manual & quickly complete

29. Conclusions Take advantage of data from other tasks! – Lots of data – Cheap Behavior features can reliably predict selection

30. Quotes: The 2nd method (B) seemed a more "aggressive" version of method 1 (C). However the UI presentation i.e. the selection and deselection of large numbers of files strained my eyes and annoyed me. Selecting more files than desired can seem dangerous in some situations - especially when selecting files to delete or modify.