1. AutoLearn’s authoring tool € A piece of cake for teachers Martí Quixal Fundació Barcelona Media – Universitat Pompeu Fabra Co-authors: Susanne Preuß, Beto Boullosa and David García-Narbona Joint work with Toni Badia, Mariona Estrada, Raquel Navarro, John Emslie, Alice Foucart, Mike Sharwood Smith, Paul Schmidt, Isin Bengi-Öner and Nilgün Firat € Research funded by the Lifelong Learning Programme 2007-2013 (2007-3625/001-001) 1

2. Outline Motivation and goal A tool for authoring ICALL materials Using and evaluating of AutoTutor Concluding remarks 2

3. Outline Motivation and goal A tool for authoring ICALL materials Using and evaluating of AutoTutor Concluding remarks 3

4. Motivation: ICALL’s irony 4

5. Motivation: FLTL requirements 5

6. Motivation: involve FLTL practitioners 6

7. Goal: one NLP response to shape language technology to the needs of the teachers (and learners) by – allowing for feedback generation focusing both on form and on content – providing a tool and a methodology for teachers to author/adapt ICALL materials autonomously 7

8. Outline Motivation and goal A tool for authoring ICALL materials General functionalities (and GUI) Answer specifications NLP-resource generation Using and evaluating of AutoTutor Concluding remarks 8

9. Context: AutoLearn project 9

10. AutoTutor: ICALL for Moodle 10

11. AutoTutor: architecture and process 11

12. AutoTutor: functionalities (I) ERROR MODEL ANSWER MODEL 12

13. AutoTutor: functionalities (II) 13

14. AutoTutor: functionalities (III) 14

15. AutoTutor: doing activities 15

16. AutoTutor: immediate feedback 16

17. S TEP TWO : SPECIFIC LANGUAGE CHECKING S TEP ONE : GENERAL LANGUAGE CHECKING Short () on the NLP side T OKENIZER M ORPH. ANALYSIS S PELL CHECKING G RAMMAR CHECKING C ONTENT MATCHING C USTOMIZED SPECIFIC CHECKING S PECIFIC FORM CHECKING G ENERAL CONTENT EVALUATION L EXICON E XERCISE - SPECIFIC LEXICON 17

18. ATACK: answer specification (I) 18

19. ATACK: answer specification (II) 19

20. ATACK: answer specification (III) B B1B2 20

21. ATACK: answer specification (IV) A A B1 B2 E DE 21 C

22. ANSWER MODEL ATACK: NLP resource generation B LOCKS ( CHUNKS ) M ORPHOLOGICAL ANALYSIS M ATCH SETTINGS C ONTENT MATCHING E XERCISE - SPECIFIC ERROR CHECKING E XERCISE - SPECIFIC LEXICON B LOCK ORDER G LOBAL ANSWER WELL - FORMEDNESS ERROR MODEL 22

23. ATACK’s underlying NLP strategy 23

24. ATACK: underlying NLP formalism (I) 24 D ESCRIPTION PART A CTION PART M ARKERS V ARIABLES Q UANTIFIERS O PERATORS

25. ATACK: underlying NLP formalism(II) 25 B2 a1a2... a1...a2... a1a2... a1...a2... b1b2... b1...b2... b1b2... b1...b2... AB1 C D1 D2 E

26. ATACK: “info” chunker Word-level Word level with extra-stuff Lemma-level Lemma-level with extra stuff Lemma-level with stuff missing Some “key” words (concept words) Word-level Word level with extra-stuff Lemma-level Lemma-level with extra stuff Lemma-level with stuff missing Some “key” words (concept words) 26

27. ATACK: “global” well-formedness ori_chunked_gap_no_need= ?$-1a{chunked=a_G_A_C2_D2_F_E_;a_G_A_C1_D1_D1_E_; a_G_A_H_B1_C2_D2_F_E_;a_G_A_H_B1_C1_D1_D1_E_; a_C2_D2_F_E_;a_C1_D1_D1_E_}, +Aa{flagc~=_,lu~=@;&at,snr~=1000}e{flagc=g,style=no_need} ::: Ar{style=no_need,bstyle=no_need,estyle=no_need}, $-1g{style=no_need}. 27 Correct block order Correctness in within block “Blended” structures Missing blocks Correct block order Correctness in within block “Blended” structures Missing blocks

28. Outline Motivation and goal A tool for authoring ICALL materials Using and evaluating of AutoTutor Concluding remarks 28

29. AutoLearn: testing in real-life (I) 29 78% would only use ICALL materials if ready-made.

30. AutoLearn: testing in real-life (II) 30

31. AutoLearn: training ICALL developers 4-hour course (2 sessions), plus 4 control meetings (and individual work) How to plan, pedagogically speaking, a learning sequence including ICALL materials? What can NLP do for you? How do you use ATACK’s GUI? 31

32. Learnt from cooperation with teachers Designing FLT materials knowing in advance that they will be part of an ICALL system is more difficult than selecting activities from books The notion of time The notion of space The lack of expertise in using ICALL/NLP results into overdemanding or not challenging NLP tasks 32

33. Learning to restrict NLP complexity (I) 33 Which is your attitude concerning responsible consumption? How do you deal with recycling? Do you think yours is an ecological home? Are you doing your best to reduce your ecological footprint? Make a list of 10 things you could do to reduce, reuse or recycle your waste at home.

34. Learning to restrict NLP complexity (II) 1.Which is both the challenge and the opportunity of managing our waste? 2.If we do not recycle the stock of aluminium and steel in our society, where would they come from? 3.What consequence has the 1994 packaging directive on people’s behaviour? 4.For which two types of products have hazaradous substances been prohibited in their production? 5.What should we require from Europe to become a recycling society? 34

35. Analysing AutoTutor’s performance 35 QuestionInv.Tot 1st273 2nd21100

36. Building a “gold standard” QuestionCorr.Part.Incorr.Inv.Tot 1st362312273 2nd14293621100 36

37. Quantitative analysis (accuracy) MESSAGESREAL ERRORSPERCENTAGE FormContFormContFormCont CORRECT ANSWERS 311391571 48,451,1 PARTIALLY CORRECT 884742 87,550 INCORRECT ANSWERS 41303918 95,160 MESSAGESREAL ERRORSPERCENTAGE FormContFormContFormCont CORRECT ANSWERS 645820 10044,4 PARTIALLY CORRECT 291101857 62,151,8 INCORRECT ANSWERS 20932177 10082,8 37

38. Main causes of misbehaviour MISBEHAVIOURPHASE 1PHASE 2 Connection failed10 Bad use of the system11 System misleading learner42 False positive (L1-driven, OOV)2233 Inappropriate focus on form3521 Artificial separation of messages061 Poor specifications162 TOTAL64180 38

39. Misbehaviour in formal aspects 39 Inappropriate focus on form “Rare” entries

40. Artificial separation 40

41. Poor specifications 41 Semantic extension of answer Syntactic flexibility

42. Outline Motivation and goal A tool for authoring ICALL materials Using and evaluating of AutoTutor Concluding remarks 42

43. Conclusions: improve coverage 43

44. Conclusions: improve accuracy 44

45. Conclusions: general message It was Feasible to overcome ICALL’s irony Possible to meet some FLTL requirements Incredibly useful to involve real-life teachers and learners in testing NLP developers have to work closely together with FLT trainers 45

46. Martí Quixal (marti.quixal@barcelonamedia.org) Fundació Barcelona Media – Universitat Pompeu Fabra Diagonal 177, planta 10 E-08018 Barecelona Acknowledgements: thanks to Holger Wunsch, Ramon Ziai and Detmar Meurers for their very useful comments on a rehearsal of this presentation Thanks for your attention! Questions or remarks? http://autolearn.barcelonamedia.org/ http://parles.upf.edu/autolearn/ http://parles.upf.edu/autolearnTutorKit Thanks for your attention! Questions or remarks? http://autolearn.barcelonamedia.org/ http://parles.upf.edu/autolearn/ http://parles.upf.edu/autolearnTutorKit 46

47. References Amaral 2007 Amaral & Meurers, submitted J. Burstein, S. Wolff, and C. Lu, Using lexical semantic techniques to classify free-responses, 227– 244, in Breadth and depth of semantic lexicons, ed. by Evelyne Viegas, Kluwer, Dordrecht, 1997. (Carl and Schmidt-Wigger 1998) Kathleen Graves. Designing Language Courses: A Guide for Teachers. Boston, MA: Heinle & Heinle, 2000. (Heift 2001) do they read it Heift & Schulze 2007 Heift 2003 (Levy 1997, 200-203) Levy and Stockwell, 2006 (Pujolà 2001, 2002) Quixal 2006 Toole and Heift 2002 Schmidt 2004 Ziai 2009 47

48. References From Graves 2000 (p 149) When teachers are required to strictly adhere to a textbook and timetable there is little room for them to make decisions. On the other hand, the majority of teachers are not paid or do not have the time in their schedules to develop all the materials for every course they teach. 48

49. AutoLearn’s authoring tool A piece of cake for teachers Martí Quixal (marti.quixal@barcelonamedia.org) Some more slides (bonus-track) 49

50. Conclusions: user behaviour Learners do not always go through the two correction steps Even if they do not read it, what else can we do for them thanks to NLP techniques? (Heift 2001) Layered feedback (Pujolà 2001, 2002) Adaptive feedback (Heift 2003, Ziai 2009) Intelligent visual feedback (?) We want the teacher to monitor and enhance system behaviour, but also to overview the learning process! 50

51. Motivation: real life experiences 51

52. AutoTutor: functionalities (#) 52

53. AutoTutor: working process CALL Tutor- learner interaction Teacher supervision Answer specification (activity design) NLP-resource generation ATACKATACK ATAPATAP 53

54. ATACK: underlying NLP formalism (II) In ALLES KURD was extended: (Schmidt et al. 2004, Quixal et al. 2006) Text level structure, where nodes are sentences, instead of words, and rules can be applied to feature bundles associated with sentences (info percolation procedure) JUMP operator: so that rule order application could be controlled externally 54

55. ATACK: “info” chunker 55

56. ATACK: exercise-specific lexicon 56

57. ATACK: error modelling “a measure that measures”  “This is correct…” match_lit = Aa{ori=a}e{gram=gramprop}, Aa{ori=measure}e{gram=gramprop}, Aa{ori=that}e{gram=gramprop}, Aa{ori=measures}e{gram=gramprop} ::: Au{style=gramprop,gram=gramprop}r{gramprop=This is correct but check whether it is stylistically appropriate.}, $-1g{style=gramprop}. 57

58. Results from teacher work (I) 58

59. Context: AutoLearn main activities 59

60. Reviewed teacher work (I) 60