1. User-Initiated Learning (UIL) Kshitij Judah, Tom Dietterich, Alan Fern, Jed Irvine, Michael Slater, Prasad Tadepalli, Oliver Brdiczka, Jim Thornton, Jim Blythe, Christopher Ellwood, Melinda Gervasio, Bill Jarrold

2. CALO: Intelligent Assistant for the Desktop Knowledge Worker Learn to Understand Meetings Learn to Keep User Organized Learn to Manage Email Learn to Prepare information Products Learn to Schedule and Execute CALO: Learning to be an Intelligent Assistant PAL Program Focus: “Learning in the Wild” 2

3. User-Initiated Learning All of CALO’s learning components can perform Learning In The Wild (LITW) But the learning tasks are all pre-defined by CALO’s engineers: What to learn What information is relevant for learning How to acquire training examples How to apply the learned knowledge UIL Goal: Make it possible for the user to define new learning tasks after the system is deployed 3

4. Motivating Scenario: Forgetting to Set Sensitivity TIMELINETIMELINE Scientist Sets sensitivity to confidential Sends email to team Sends email to a colleague “Lunch today?” Does not set sensitivity to confidential Collaborates on a Classified project Sends email to team Forgets to set sensitivity to confidential Research Team 4

5. Motivating Scenario: Forgetting to Set Sensitivity TIMELINETIMELINE “Please do not forget to set sensitivity when sending email” Scientist Research Team Teaches CALO to learn to predict whether user has forgot to set sensitivity Sends email to team CALO reminds user to set sensitivity 5

6. User-CALO Interaction: Teaching CALO to Predict Sensitivity SAT Based Reasoning System SPARK Procedure Instrumented Outlook Instrumented Outlook Events user Integrated Task Learning user Compose new email Modify Procedure Procedure Demonstration and Learning Task Creation User Interface for Feature Guidance User Selected Features user Feature Guidance Email + Related Objects CALO Ontology Trained Classifier Feature Guidance Machine Learner Knowledge Base Training Examples Learning Legal Features SAT Based Reasoning System Class Labels 6

7. User-CALO Interaction: Teaching CALO to Predict Sensitivity SAT Based Reasoning System SPARK Procedure Instrumented Outlook Instrumented Outlook Events user Integrated Task Learning user Compose new email Modify Procedure Procedure Demonstration and Learning Task Creation User Interface for Feature Guidance User Selected Features user Feature Guidance Email + Related Objects CALO Ontology Trained Classifier Feature Guidance Machine Learner Knowledge Base Training Examples Learning Legal Features SAT Based Reasoning System Class Labels 7

8. Initiating Learning via Demonstration LAPDOG: Transforms an observed sequence of instrumented events into a SPARK procedure SPARK representation generalizes the dataflow between the actions of the workflow 8

9. Initiating Learning via Demonstration TAILOR: Supports procedure editing For UIL, it allows adding a condition to one or more steps in a procedure 9

10. Initiating Learning via Demonstration The condition becomes the new predicate to be learned 10

11. User-CALO Interaction: Teaching CALO to Predict Sensitivity SAT Based Reasoning System SPARK Procedure Instrumented Outlook Instrumented Outlook Events user Integrated Task Learning user Compose new email Modify Procedure Procedure Demonstration and Learning Task Creation User Interface for Feature Guidance User Selected Features user Feature Guidance Email + Related Objects CALO Ontology Trained Classifier Feature Guidance Machine Learner Knowledge Base Training Examples Learning Legal Features SAT Based Reasoning System Class Labels 11

12. Inferring Feature Legality HasToField HasSubject HasBody HasAttachment HasSensitivity … Description StartDate Enddate … HasToField HasSubject HasBody HasAttachment HasSensitivity … FirstName LastName Address Phone … FirstName LastName Address Phone … ToRecipient CCRecipient PrevEmailMessage Project EmailMessage Subset of Ontology Naively, the system will use all features For example, system will use HasSensitivity Dangerous to use HasSensitivity: Has one-to-one correlation with target and is present at training time Not present at test time Feature filtering removes such features at training time 12

13. Feature Guidance Interface 13

14. User-CALO Interaction: Teaching CALO to Predict Sensitivity SAT Based Reasoning System SPARK Procedure Instrumented Outlook Instrumented Outlook Events user Integrated Task Learning user Compose new email Modify Procedure Procedure Demonstration and Learning Task Creation User Interface for Feature Guidance User Selected Features user Feature Guidance Email + Related Objects CALO Ontology Trained Classifier Feature Guidance Machine Learner Knowledge Base Training Examples Learning Legal Features SAT Based Reasoning System Class Labels 14

15. Training Instance Generation Goal: autonomously generate labeled training instances for the learning component from stored user emails Problem: actions used to create emails are not stored in the CALO knowledge base, so we need to infer how email was created {defprocedure do_rememberSensitivity.... [do: (openComposeEmailWindow $newEmail)] [do: (changeEmailField $newEmail "to")] [do: (changeEmailField $newEmail "subject")] [do: (changeEmailField $newEmail "body")] [if: (learnBranchPoint $newEmail) [do: (changeEmailField $newEmail "sensitivity")]] [do: (sendEmailInitial $newEmail)].... } Specifically, we want to know: Whether an email is an instance of the procedure? Which branch was taken during creation of the email? No such inference can be drawn 15

16. Training Instance Generation Domain Axioms Domain Axioms NewComposition  ComposeNewMail ReplyComposition  ReplyToMail HasAttachment  (AttachFile  ForwardMail) … SPARK Axioms SPARK Axioms ProcInstance  (u 1  u 2  …  U n ) (  forget  label)  (C 1  C 2  …  C n ) {defprocedure do_rememberSensitivity.... [do: (openComposeEmailWindow $newEmail)] [do: (changeEmailField $newEmail "to")] [do: (changeEmailField $newEmail "subject")] [do: (changeEmailField $newEmail "body")] [if: (learnBranchPoint $newEmail) [do: (changeEmailField $newEmail "sensitivity")]] [do: (sendEmailInitial $newEmail)].... } Label Analysis Formula (LAF) Label Analysis Formula (LAF) Knowledge Base NewComposition ReplyComposition HasToField HasSubject HasBody HasAttachment … Reasoning Engine Reasoning Engine E   forget ╞ (ProcInstance  Label) E   forget ╞ (ProcInstance   Label) otherwise Positive Example Negative Example Discard email 16

17. Training Instance Generation Goal: autonomously generate labeled training instances for the learning component from stored user data Problem: actions used to create the objects are not stored in the CALO knowledge base, so we need to infer the actions SAT based reasoning system is used It employs a domain model of the UI actions and how they affect the object attributes 17

18. Training Instance Generation: The Domain Model A proposition symbol is defined for each email attribute and action Example action propositions: ComposeNewMail, ForwardMail, ReplyToMail, ModifyToField etc. Example property propositions: NewComposition, ForwardedComposition, HasCCField, HasBody etc. An action proposition is true relative to an email iff it was taken during creation of the email A property proposition is true relative to an email iff the email satisfies the corresponding property 18

19. Training Instance Generation: The Domain Model The domain model includes one axiom for each property proposition and defines the possible ways that proposition can be made true Example axioms: NewComposition  ComposeNewMail ReplyComposition  ReplyToMail HasAttachment  (AttachFile  ForwardMail) HasSubject  (ModifySubject  ReplyToMail  ForwardMail) … 19

20. Training Instance Generation: Inferring Class Labels Given email from CALO KB and SPARK procedure, we wish to assign class label to the email We construct the Label Analysis Formula (LAF) LAF contains domain propositions + three new propositions: Label: represents truth value of the branch condition Forget: whether user intended but forgot to perform conditional action ProcInstance: whether email is instance of the procedure LAF = Domain axioms + SPARK axioms 20

21. Training Instance Generation: Inferring Class Labels SPARK axioms place constraints on ProcInstance, Label and Forget: ProcInstance  (u 1  u 2  …  U n ) ( forget  label)  (C 1  C 2  …  C n ) Email is labeled as follows: Construct formula E: (1) For each property proposition P, if property is true in the email, add P to LAF else add  P; (2) Add  forget to the LAF Try to prove: E ╞ (ProcInstance  Label) If successful, email is a positive training instance Try to prove: E ╞ (ProcInstance   Label) If successful, email is a negative training instance otherwise email is not a training instance 21

22. The Learning Component Logistic Regression is used as the core learning algorithm Features Relational features extracted from ontology Incorporate User Advice on Features Apply large prior variance on user selected features Select prior variance on rest of the features through cross- validation Automated Model Selection Parameters: Prior variance on weights, classification threshold Technique: Maximization of leave-one-out cross-validation estimate of kappa 22

23. Assisting the User: Inferring Forgetting Given new email, forgetting is inferred as follows: Construct formula E: (1) For each property proposition P, if property is true in the email, add P to LAF else add  P; (2) Add label to the LAF if prediction is true else add  label to the LAF Try to prove: E ╞ (ProcInstance  forget) If successful, stop send and remind the user Try to prove: E ╞ (ProcInstance   forget) If successful, send email Otherwise, send email? 23

24. Assisting the User: Reminding 24

25. Empirical Evaluation Problems: Attachment Prediction Importance Prediction Learning Configurations Compared: No User Advice + Fixed Model Parameters User Advice + Fixed Model Parameters No User Advice + Automatic parameter Tuning User Advice + Automatic parameter Tuning User Advice: 18 keywords in the body text for each problem 25

26. Empirical Evaluation: Data Set Set of 340 emails obtained from a real desktop user 256 training set + 84 test set For each training set size, compute mean kappa (  ) using test set to generate learning curves  is a statistical measure of inter-rater agreement for discrete classes  is a common evaluation metric in cases when the classes have a skewed distribution 26

27. Empirical Evaluation: Learning Curves (A) No User Advice + Fixed Model Parameters (B) User Advice + Fixed Model Parameters (C) No User Advice + Model Selection (D) User Advice + Model Selection Attachment Prediction User advice is helpful but not critical Self-tuning is absolutely critical 27

28. Empirical Evaluation: Learning Curves Attachment Prediction 28

29. Empirical Evaluation: Learning Curves Attachment Prediction 29

30. Empirical Evaluation: Learning Curves Attachment Prediction 30

31. Empirical Evaluation: Learning Curves Attachment Prediction 31

32. Empirical Evaluation: Learning Curves Importance Prediction 32

33. Empirical Evaluation: Learning Curves Importance Prediction 33

34. Empirical Evaluation: Learning Curves Importance Prediction 34

35. Empirical Evaluation: Learning Curves Importance Prediction 35

36. Empirical Evaluation: Robustness to Bad Advice We intended to test the robustness of the system to bad advice Bad advice was generated as follows: Use SVM based feature selection in WEKA to produce a ranking of user provided keywords Replace top three words in the ranking with randomly selected words from the vocabulary 36

37. Empirical Evaluation: Robustness to Bad Advice Attachment Prediction 37

38. Empirical Evaluation: Robustness to Bad Advice Attachment Prediction 38

39. Empirical Evaluation: Robustness to Bad Advice Attachment Prediction 39

40. Empirical Evaluation: Robustness to Bad Advice Attachment Prediction 40

41. Empirical Evaluation: Robustness to Bad Advice Importance Prediction 41

42. Empirical Evaluation: Robustness to Bad Advice Importance Prediction 42

43. Empirical Evaluation: Robustness to Bad Advice Importance Prediction 43

44. Empirical Evaluation: Robustness to Bad Advice Importance Prediction 44

45. Empirical Evaluation: Prediction Utility We want to evaluate the utility of the system for the user We use a new metric called Critical Cost Ratio (CCR) Intuition: A measure of how high cost of forgetting should be compared to cost of interruption for the system to be useful Intuition : Hence, if CCR is low, the system is useful more often For example, if CCR=10, then cost of forgetting should be 10 times more than cost of interruption for net benefit 45

46. Empirical Evaluation: Prediction Utility Attachment Prediction 46

47. Empirical Evaluation: Prediction Utility Attachment Prediction 47 At size 256, cost of forgetting should be at least 5 times of cost of interruption to gain net benefit from the system

48. Empirical Evaluation: Prediction Utility Importance Prediction 48

49. Empirical Evaluation: Prediction Utility Importance Prediction 49

50. Lessons Learned User interfaces should support rich instrumentation, automation, and intervention User interfaces should come with models of their behavior User advice is helpful but not critical Self-tuning learning algorithms are critical for success 50

51. Beyond UIL: System-Initiated Learning CALO should notice when it could help the user by formulating and solving new learning tasks Additional Requirements Knowledge of user’s goals, user’s costs, user’s failure modes (e.g., forgetting, over-committing, typos) Knowledge of what is likely to be learnable and what is not Knowledge of how to formulate learning problems (classification, prediction, anomaly detection, etc.) 51

52. Thank you and Questions 52