1. D ESIGNING AN I NTERACTIVE T EACHING T OOL WITH ABML K NOWLEDGE R EFINEMENT L OOP enabling arguing to learn 1 Faculty of Education, University of Ljubljana, Slovenia 2 Faculty of Computer and Information Science, University of Ljubljana, Slovenia Matej Zapušek 1, Martin Možina 2, Ivan Bratko 2, Jože Rugelj 2, Matej Guid 2 12 th International Conference on Intelligent Tutoring Systems ITS 2014: Honolulu, Hawaii 2014

2. S OME C ONCEPTS ARE D IFFICULT TO E XPLAIN... How to distinguish edible from toxic mushrooms?

3. I NTRODUCING D OMAIN E XPERTS Even for domain experts it is hard to articulate their knowledge!

4. M ACHINE L EARNING : H OW TO I NVOLVE A D OMAIN E XPERT ? The expert can state constraints and the domain knowledge in advance... … verify, evaluate, and correct results of machine learning… … or the expert and the computer iteratively improve the model. ABML argument-based machine learning

5. A RGUMENT -B ASED M ACHINE L EARNING given set of labeled learning examples e i described with attribute values D i where C i is classification of learning example e i goal learn prediction model (hypoteshis) H IF... THEN...... H CiCi  e i : D i ai ai example e i may have argument a i

6. I T IS M UCH E ASIER TO E XPLAIN I NDIVIDUAL C ASES ! Is this mushroom toxic? Why?

7. ABML K NOWLEDGE R EFINEMENT L OOP Step 1: Learn a hypothesis with ABML Step 2: Find the “most critical” example (if none found, stop) Step 3: Expert explains the example Return to step 1 critical example learn data set Argument ABML

8. ABML K NOWLEDGE R EFINEMENT L OOP Step 1: Learn a hypothesis with ABML Step 2: Find the “most critical” example (if none found, stop) Step 3: Expert explains the example Return to step 1 Step 3a: Explaining a critical example (in a natural language) Step 3b: Adding arguments to the example Step 3c: Discovering counter examples Step 3d: Improving arguments Return to step 3c if counter example found

9. I LLUSTRATIVE E XAMPLE : L EARNING TO D IAGNOSE F LU PacientTemperatureVaccinationCoughingHeadache...Flu 1normalyesno...no 2highnoyesno...yes 3very highno yes...yes 4highyes no...no... The current model: IF Temperature < very high THEN Flu = no cannot explain well Pacient 2. The question to the expert: „What is the reason for Pacient 2 having the flu?“

10. PacientTemperatureVaccinationCoughingHeadache...Flu 1normalyesno...no 2highnoyesno...yes 3very highno yes...yes 4highyes no...no... E XPERT ‘ S E XPLANATION Expert‘s explanation: „Pacient #2 has the flue because of a high temperature.“ Expert‘s argument is attached to learning example #2. New model is built. Flu = yes BECAUSE Temperature > Normal

11. PacientTemperatureVaccinationCoughingHeadache...Flu 1normalyesno...no 2highnoyesno...yes 3very highno yes...yes 4highyes no...no... W HAT IF THE E XPERT ‘ S A RGUMENT IS NOT GOOD ENOUGH ? ML method now induced a rule consistent with argument: IF Temperature > Normal THEN Flu = yes The rule is inconsistent with data! Expert is presented with counter example: „Compare pacients #2 and #4. Why Pacient #4 doesn‘t have the flu?“

12. PacientTemperatureVaccinationCoughingHeadache...Flu 1normalyesno...no 2highnoyesno...yes 3very highno yes...yes 4highyes no...no... T HE E XPERT MAY I MPROVE THE A RGUMENT Expert finds the crucial difference between Pacients #2 and #4: „Pacient 2 didn‘t get vaccinated against the flu.“

13. PacientTemperatureVaccinationCoughingHeadache...Flu 1normalyesno...no 2highnoyesno...yes 3very highno yes...yes 4highyes no...no... I MPROVED R ULES MAY E XPLAIN U NSEEN E XAMPLES AS W ELL ML method induces a new rule: IF Temperature > Normal AND Vaccination = no THEN Flue = yes The new rule also explains diagnosis for Pacient #3: „Has flu because of a high temperature and didn‘t get vaccinated against it.“

14. ABML K NOWLEDGE R EFINEMENT L OOP : T HE I NNER L OOP Step 3a: Explaining a critical example (in a natural language) „Pacient #2 has the flue because of a high temperature.“ Step 3b: Adding arguments to the example Step 3c: Discovering counter examples Step 3d: Improving arguments with counter examples IF T EMPERATURE > N ORMAL AND V ACCINATION = NO Temperature > Normal

15. ABML R EFINEMENT L OOP & K NOWLEDGE E LICITATION IF... THEN...... ABML argument-based machine learning explain single example easier for experts to articulate knowledge “critical” examples expert provides only relevant knowledge “counter” examples detect deficiencies in explanations arguments critical examples counter examples

16. E XPERT CAN I NTRODUCE NEW C ONCEPTS (A TTRIBUTES ) Pacient...HeadacheFatigueSoreThroatAppetiteFlu 1...no yesnormalno 2...noyes lowyes 3...yes nolowyes 4...no normalno... FluSymptoms no yes no Possible rule with the new attribute: IF Temperature > Normal AND FluSymptoms = yes THEN Flu = yes...

17. E XPERT CAN C ORRECT C LASSIFICATION OF L EARNING E XAMPLE...HeadacheFatigueSoreThroatAppetiteFluSymptomsFlu...no normalnoyes PacientTemperatureVaccinationCoughing... 37normalno... The question to the expert: „What is the reason for Pacient 37 having the flu?“ Expert corrects the classification of Pacient 37: „Pacient 37 doesn‘t have the flu.“ no

18. K NOWLEDGE E LICITATION WITH ABML IF... THEN...... ABML argument-based machine learning inconsistencies in labels are detected automatically misclassificated examples are easily recognized and corrected arguments critical examples counter examples experts introduce new attributes human-understandable models suitable for teaching

19. IF... THEN...... ABML argument-based machine learning arguments critical examples counter examples How to use ABML in educational setting? I NTRODUCING S TUDENTS

20. T HE O UTER L OOP Step 1: Learn a hypothesis with ABML Step 2: Find the “most critical” example (if none found, stop) Step 3: Student explains the example Return to step 1

21. T HE O UTER L OOP & THE I NNER L OOP Step 1: Learn a hypothesis with ABML Step 2: Find the “most critical” example (if none found, stop) Step 3: Student explains the example Return to step 1 Step 3a: Explaining a critical example (in a natural language) Step 3b: Adding arguments to the example Step 3c: Discovering counter examples Step 3d: Improving arguments with counter examples Return to step 3c if counter example found USING TEACHER‘S ATTRIBUTES!

22. A RGUING TO L EARN Argumentation involves elaboration, reasoning, and reflection. These activities have been shown to contribute to deeper conceptual learning (Bransford, Brown, & Cocking, 1999) Participating in argumentation helps students learn about argumentative structures (Kuhn, 2001)

23. A N EW P ARADIGM arguing to learn with argument-based machine learning

24. Let‘s see how this works in practice

25. B ASIC OR A DVANCED ? „basic“ „advanced“

26. L EARNING D ATA S ET 121 solutions of 62 different exercises teacher labeled each solution as „basic“ or „advanced“ learn data: 91 examples test data: 30 examples

27. K NOWLEDGE E LICITATION FROM THE T EACHER 1. relevant description language: new attributes 2. consistently labeled examples TEACHER‘S GOALS:

28. R ESULTS OF K NOWLEDGE E LICITATION FROM THE T EACHER 9 iterations 9 new attributes 9 rules only 1 out of 5 initial attributes remained

29. A S TUDENT -C OMPUTER I NTERACTIVE L EARNING S ESSION STUDENT‘S TASK obtain rules for distinguishing „basic“ and „advanced“ solutions rules must consist of attributes in teacher‘s final model use teacher‘s descriptive language

30. A S TUDENT -C OMPUTER I NTERACTIVE L EARNING S ESSION RECOMMENDATIONS TO THE STUDENT use the most important features for explanations use the smallest possible number of features in a single argument try not to repeat the same arguments

31. T HE F IRST „C RITICAL “ E XAMPLE The question to the student: „Why is this solution advanced?“ Student‘s argument: „Because function zip is present and the number of rows is low.“ Solution = advanced BECAUSE Zip = True AND cRows = low

32. C OUNTER E XAMPLE IF Zip = True AND cRows = low THEN Solution = advanced The rule is inconsistent with data! Student is presented with counter example: „Compare these two solutions. Why is the second solution a basic one?“

33. I MPROVING A RGUMENT IF Zip = True AND cRows = low AND LiCom = True THEN Solution = advanced Student‘s extended argument: „Because function zip is present, the number of rows is low, and a list comprehension occurs. “ no more counter examples next iteration

34. A T THE E ND OF THE I NTERACTIVE S ESSION 5 iterations half an hour 90% accuracy on (previously unseen) testing data several suggestions of new descriptive features

35. A SSESMENT Results: experiment with 7 students 7.1 iterations 87.1% classification accuracy of obtained rule model 86.7% correctly „manually“ classified (previously unseen) examples very positive qualitative feedback from the students

36. C ONCLUSIONS New paradigm: arguing to learn with argument-based machine learning Future work: applications in several domains assessing argument‘s quality for improved immediate feedback goal-oriented extension (see our ITS 2012 paper)

37. Q UESTIONS & D ISCUSSION Thank you! slides: ailab.si/matej enabling arguing to learn