1. Bart Loosvelt - Bernard Tilleman- Piet Desmet - Bert Wylin RechtenOnline 2008 Giving useful feedback on open questions RechtenOnline K.U.Leuven Campus Kortrijk – Televic Education nv Giving useful feedback on open questions in the law faculty The development of “cases” and a “supported-open” question type

2. Bart Loosvelt - Bernard Tilleman- Piet Desmet - Bert Wylin RechtenOnline 2008 Giving useful feedback on open questions RechtenOnline K.U.Leuven Campus Kortrijk – Televic Education nv Content 1.Problem 2.Beyond closed exercises: typology of exercise types 3.Two possible solutions 4.Strategy 1 : decision tree or “case” 5.Strategy 2 : open supported 6.Quantitative & qualitative evidence 7.Work in progress

3. Bart Loosvelt - Bernard Tilleman- Piet Desmet - Bert Wylin RechtenOnline 2008 Giving useful feedback on open questions RechtenOnline K.U.Leuven Campus Kortrijk – Televic Education nv 1. Problem How can we give useful feedback on open questions (with many correct answers) in an online (language) learning environment ?

4. Bart Loosvelt - Bernard Tilleman- Piet Desmet - Bert Wylin RechtenOnline 2008 Giving useful feedback on open questions RechtenOnline K.U.Leuven Campus Kortrijk – Televic Education nv challenges: most exercise and test platforms only focus on closed questions (with a very limited number of possible answers) for which corrective feedback is offered + they also include an open question type (e.g. Blackboard: essay question) that allow for free text answers but without corrective feedback -> go beyond closed exercises but with corrective feedback

5. Bart Loosvelt - Bernard Tilleman- Piet Desmet - Bert Wylin RechtenOnline 2008 Giving useful feedback on open questions RechtenOnline K.U.Leuven Campus Kortrijk – Televic Education nv Why go beyond closed exercises? to allow for more complex cognitive activities = crucial in CAL: -go beyond drill & practice - allow for skills oriented tasks with focus on meaning (and not only on form) - essential for intermediate and advanced learners

6. Bart Loosvelt - Bernard Tilleman- Piet Desmet - Bert Wylin RechtenOnline 2008 Giving useful feedback on open questions RechtenOnline K.U.Leuven Campus Kortrijk – Televic Education nv All results have been integrated into our generic exercise and test platform used to create a considerable amount of language learning and testing environments @ K.U.Leuven, used with single sign on (SSO) webservice from within Blackboard LMS also fully SCORM compliant e.g. http://www.franel.eu a free online learning environment for French and Dutch with more than 16.000 registered users

7. Bart Loosvelt - Bernard Tilleman- Piet Desmet - Bert Wylin RechtenOnline 2008 Giving useful feedback on open questions RechtenOnline K.U.Leuven Campus Kortrijk – Televic Education nv Context of this presentation: 2 research projects granted by K.U.Leuven OI-project Jurim@tic (2005-2007) OI-project Juriflex (2007-2009) Prof. Bernard Tilleman, director, law faculty, Prof. Piet Desmet, co-director, faculty of arts/linguistics, Isabelle Demortier & Bart Loosvelt, lawyers, junior researchers, Bert Wylin, educational technology, Televic Education Eddy Demeersseman, educational scientist, educational support, K.U.Leuven Campus Kortrijk

8. Bart Loosvelt - Bernard Tilleman- Piet Desmet - Bert Wylin RechtenOnline 2008 Giving useful feedback on open questions RechtenOnline K.U.Leuven Campus Kortrijk – Televic Education nv Focus of these projects:  Electronic revolution seemed to have more or less passed by the Law Faculty  One of the reasons is that questioning in law school is far more open ended  The typical educational approach of the Law Faculty is to work with cases -> How can we reduce the complexicity of law questioning and working with cases so it could be handled in an online learning environment ? -> More generic question: How to integrate open questions into an e- learning environment

9. Bart Loosvelt - Bernard Tilleman- Piet Desmet - Bert Wylin RechtenOnline 2008 Giving useful feedback on open questions RechtenOnline K.U.Leuven Campus Kortrijk – Televic Education nv These 2 projects focus on CALI (Computer Assisted Legal Instruction), but are transposable to other domains, like CALL: 1) CLIL (Content and Language Integrated Learning): one learns a subject through the medium of a foreign language 2) TBL (Task-Based Learning): goal-oriented real-life tasks focusing on meaning (not on form) 3) Comprehension activities (reading and listening): meaning-focused questions 4) …

10. Bart Loosvelt - Bernard Tilleman- Piet Desmet - Bert Wylin RechtenOnline 2008 Giving useful feedback on open questions RechtenOnline K.U.Leuven Campus Kortrijk – Televic Education nv Parameters input from learner – number of correct answers – level of freedom – possible answers given beforehand output software – type of correction – reliability of automated correction 2. Beyond closed exercises: typology of exercise types

11. Bart Loosvelt - Bernard Tilleman- Piet Desmet - Bert Wylin RechtenOnline 2008 Giving useful feedback on open questions RechtenOnline K.U.Leuven Campus Kortrijk – Televic Education nv a. Closed vs open learner input closedopen # correct answers limited (to 1 or a few) many

12. Bart Loosvelt - Bernard Tilleman- Piet Desmet - Bert Wylin RechtenOnline 2008 Giving useful feedback on open questions RechtenOnline K.U.Leuven Campus Kortrijk – Televic Education nv b. Open: half-open vs. open software output closedhalf-opensupported open open correction type automatedautohalf- automated + manual support manual reliability automate d correction highaverage to high low

13. Bart Loosvelt - Bernard Tilleman- Piet Desmet - Bert Wylin RechtenOnline 2008 Giving useful feedback on open questions RechtenOnline K.U.Leuven Campus Kortrijk – Televic Education nv c. closed : closed vs. halfclosed learner input closedhalf-closedopen level of freedom limited free possible answers given beforehan d choice from given choice from many given answer s not given

14. Bart Loosvelt - Bernard Tilleman- Piet Desmet - Bert Wylin RechtenOnline 2008 Giving useful feedback on open questions RechtenOnline K.U.Leuven Campus Kortrijk – Televic Education nv … concrete exercise types closedhalf-closedhalf-opensupported open open Edumatic split up into parts of the sentence drag and drop text horizontal ordering drag and drop words with images vertical ordering drag and drop hot zones ordering in columns connect elements (connect) dropdown in text multipl e choice from a long list select text crossw ord puzzle fill-in dictatio n translate reformulate correct open supported essay

15. Bart Loosvelt - Bernard Tilleman- Piet Desmet - Bert Wylin RechtenOnline 2008 Giving useful feedback on open questions RechtenOnline K.U.Leuven Campus Kortrijk – Televic Education nv 3. Two possible solutions Strategy 1: reformulate an open question (e.g. case) as a decision tree with a series of closed or half-closed exercises (allows for automatic corrective feedback) Strategy 2: create a supported open question type that allows for half-automated correction and feedback

16. Bart Loosvelt - Bernard Tilleman- Piet Desmet - Bert Wylin RechtenOnline 2008 Giving useful feedback on open questions RechtenOnline K.U.Leuven Campus Kortrijk – Televic Education nv 4. Strategy 1: decision tree or “case” Could open questions like cases be seen as decision trees with branches ? – one case reformulated into multiple connected questions – answer path is clarified  The path to knowledge is as important as the knowledge itself  Get the right answer for the right reason

17. Bart Loosvelt - Bernard Tilleman- Piet Desmet - Bert Wylin RechtenOnline 2008 Giving useful feedback on open questions RechtenOnline K.U.Leuven Campus Kortrijk – Televic Education nv Example Several questions must be answered if a Belgian statute regulation is claimed to be in conflict with the Constitution: Which norm has to be tested? Which norm can serve as a basis for the examination? Which court is competent for examining this conflict? Is there a term to be observed? Which procedure has to be followed? What is the result of this examination (abeyance or nullification)? Etc. These questions are the path to the solution of the conflict.

18. Bart Loosvelt - Bernard Tilleman- Piet Desmet - Bert Wylin RechtenOnline 2008 Giving useful feedback on open questions RechtenOnline K.U.Leuven Campus Kortrijk – Televic Education nv Example (in chart) Which norm? Which court? Is there a term? Etc…

19. Bart Loosvelt - Bernard Tilleman- Piet Desmet - Bert Wylin RechtenOnline 2008 Giving useful feedback on open questions RechtenOnline K.U.Leuven Campus Kortrijk – Televic Education nv Technical solution: “case” case is not really a question type it is a complex or multiple question type it links several items together in one case one overview is provided for the entire case – the overview is an option – the overview is progressive

20. Bart Loosvelt - Bernard Tilleman- Piet Desmet - Bert Wylin RechtenOnline 2008 Giving useful feedback on open questions RechtenOnline K.U.Leuven Campus Kortrijk – Televic Education nv

21. Bart Loosvelt - Bernard Tilleman- Piet Desmet - Bert Wylin RechtenOnline 2008 Giving useful feedback on open questions RechtenOnline K.U.Leuven Campus Kortrijk – Televic Education nv

22. Bart Loosvelt - Bernard Tilleman- Piet Desmet - Bert Wylin RechtenOnline 2008 Giving useful feedback on open questions RechtenOnline K.U.Leuven Campus Kortrijk – Televic Education nv

23. Bart Loosvelt - Bernard Tilleman- Piet Desmet - Bert Wylin RechtenOnline 2008 Giving useful feedback on open questions RechtenOnline K.U.Leuven Campus Kortrijk – Televic Education nv

24. Bart Loosvelt - Bernard Tilleman- Piet Desmet - Bert Wylin RechtenOnline 2008 Giving useful feedback on open questions RechtenOnline K.U.Leuven Campus Kortrijk – Televic Education nv

25. Bart Loosvelt - Bernard Tilleman- Piet Desmet - Bert Wylin RechtenOnline 2008 Giving useful feedback on open questions RechtenOnline K.U.Leuven Campus Kortrijk – Televic Education nv “Case” allows to integrate all existing exercise types + development of 2 new half-closed exercise types: 1) connect: explicitation of logical relations 2) select text: selection of relevant text fragments

26. Bart Loosvelt - Bernard Tilleman- Piet Desmet - Bert Wylin RechtenOnline 2008 Giving useful feedback on open questions RechtenOnline K.U.Leuven Campus Kortrijk – Televic Education nv Connect connect is a half-closed exercise type (possible answers are not given in beforehand) horizontal or vertical matching in 2 stages – 1st stage: match elements – 2nd stage: define the relations with several feedback options

27. Bart Loosvelt - Bernard Tilleman- Piet Desmet - Bert Wylin RechtenOnline 2008 Giving useful feedback on open questions RechtenOnline K.U.Leuven Campus Kortrijk – Televic Education nv 1st step: Match elements

28. Bart Loosvelt - Bernard Tilleman- Piet Desmet - Bert Wylin RechtenOnline 2008 Giving useful feedback on open questions RechtenOnline K.U.Leuven Campus Kortrijk – Televic Education nv 2nd step: Define relationships

29. Bart Loosvelt - Bernard Tilleman- Piet Desmet - Bert Wylin RechtenOnline 2008 Giving useful feedback on open questions RechtenOnline K.U.Leuven Campus Kortrijk – Televic Education nv Select text half-closed (possible answer not given, limited level of freedom) mark the keyword(s) in a given text (sentence or paragraph) define ranges for selection – ranges don’t count linked keywords grouped keywords feedback

30. Bart Loosvelt - Bernard Tilleman- Piet Desmet - Bert Wylin RechtenOnline 2008 Giving useful feedback on open questions RechtenOnline K.U.Leuven Campus Kortrijk – Televic Education nv select text Terug

31. Bart Loosvelt - Bernard Tilleman- Piet Desmet - Bert Wylin RechtenOnline 2008 Giving useful feedback on open questions RechtenOnline K.U.Leuven Campus Kortrijk – Televic Education nv select text Terug

32. Bart Loosvelt - Bernard Tilleman- Piet Desmet - Bert Wylin RechtenOnline 2008 Giving useful feedback on open questions RechtenOnline K.U.Leuven Campus Kortrijk – Televic Education nv “Case”: useful, but … Level of freedom of the learner input is limited Clarify the path to knowledge is interesting in the first phase of the learning path, but less afterwards -> strategy 2: supported open

33. Bart Loosvelt - Bernard Tilleman- Piet Desmet - Bert Wylin RechtenOnline 2008 Giving useful feedback on open questions RechtenOnline K.U.Leuven Campus Kortrijk – Televic Education nv 5. Strategy 2 : supported open open question with free learner input with due date generation of feedback on the basis of:  model answer  keyword matching white list (+ score) –and –if –if then black list (0 or – score) negations (and range)

34. Bart Loosvelt - Bernard Tilleman- Piet Desmet - Bert Wylin RechtenOnline 2008 Giving useful feedback on open questions RechtenOnline K.U.Leuven Campus Kortrijk – Televic Education nv 4 functions of supported open exercise type: 1)Creation of open question with model answer, black list, white list, elaborated feedback, etc. 2)Publication of this item fix due date, select student groups, follow-up received answers, etc. 3)Half-automated correction of the answers correction proposal on the basis of the available info manual correction of scores and adaptation of black list & white list (-> update of automatic scores) 4)Generation of feedbackreport individualised feedback, fix scores, add personal comments notify all users by automatically generated mail

35. Bart Loosvelt - Bernard Tilleman- Piet Desmet - Bert Wylin RechtenOnline 2008 Giving useful feedback on open questions RechtenOnline K.U.Leuven Campus Kortrijk – Televic Education nv Supported Open: Example of frontend

36. Bart Loosvelt - Bernard Tilleman- Piet Desmet - Bert Wylin RechtenOnline 2008 Giving useful feedback on open questions RechtenOnline K.U.Leuven Campus Kortrijk – Televic Education nv Supported Open: Example of Backoffice

37. Bart Loosvelt - Bernard Tilleman- Piet Desmet - Bert Wylin RechtenOnline 2008 Giving useful feedback on open questions RechtenOnline K.U.Leuven Campus Kortrijk – Televic Education nv Supported open : Example of half-automated correction

38. Bart Loosvelt - Bernard Tilleman- Piet Desmet - Bert Wylin RechtenOnline 2008 Giving useful feedback on open questions RechtenOnline K.U.Leuven Campus Kortrijk – Televic Education nv Supported open: adapt score and generate feedback

39. Bart Loosvelt - Bernard Tilleman- Piet Desmet - Bert Wylin RechtenOnline 2008 Giving useful feedback on open questions RechtenOnline K.U.Leuven Campus Kortrijk – Televic Education nv Use of supported open exercises in three steps – Step 1 : try out as a marking and feedbacktool (aid) used by teaching staff -> human verification and improvement of the black & white list is necessary – Step 2 : learning result of scenario 1 can be used as an exercise with full automatic corrective and elaborated feedback (with human intervention!) -> human verification and e-mail feedback – Step 3 : exam simulation results of scenario 2 can be used as an exercise with full immediate automatic corrective and elaborated feedback (without human intervention!)

40. Bart Loosvelt - Bernard Tilleman- Piet Desmet - Bert Wylin RechtenOnline 2008 Giving useful feedback on open questions RechtenOnline K.U.Leuven Campus Kortrijk – Televic Education nv 6. Quantitative & qualitative evidence Effectiveness of the half-automated correction tool User statistics: Intensity of use Student satisfaction

41. Bart Loosvelt - Bernard Tilleman- Piet Desmet - Bert Wylin RechtenOnline 2008 Giving useful feedback on open questions RechtenOnline K.U.Leuven Campus Kortrijk – Televic Education nv Effectiveness of the halfautomated correction tool N= 200 Exam january 2008 One supported open question Manual correction vs (half-)automatic correction: total match for 195/200 answers = 97,5% ! mismatch for 5 answers due to incoherence of student answer (1) due to incoherence of human correction (4) -> high marginal profit (quality of correction) and low marginal cost (people & time)

42. Bart Loosvelt - Bernard Tilleman- Piet Desmet - Bert Wylin RechtenOnline 2008 Giving useful feedback on open questions RechtenOnline K.U.Leuven Campus Kortrijk – Televic Education nv Effectiveness of the halfautomated correction tool N= 200 Exam january 2008 One supported open question Feedback (by analysis of student input): 5 black list words predicted by teacher analysis 1-50: + 3 analysis 51-100: + 1 analysis 101-200: + 0

43. Bart Loosvelt - Bernard Tilleman- Piet Desmet - Bert Wylin RechtenOnline 2008 Giving useful feedback on open questions RechtenOnline K.U.Leuven Campus Kortrijk – Televic Education nv User statistics (1) Example 1: first bachelor Law (at Kortrijk) – 68 students – 1771 sessions – 554 hours activity – 26 sessions/student – Mean of 8 hours 9 minutes of activity per student – Mean length of a session: 19 minutes

44. Bart Loosvelt - Bernard Tilleman- Piet Desmet - Bert Wylin RechtenOnline 2008 Giving useful feedback on open questions RechtenOnline K.U.Leuven Campus Kortrijk – Televic Education nv User statistics (2) Example 2: first bachelor Law (Leuven) – 285 students – 8806 sessions – 3232 hours – 30 sessions/student – Mean of 11 hours 20 minutes per student – Mean length of a session: 22 minutes

45. Bart Loosvelt - Bernard Tilleman- Piet Desmet - Bert Wylin RechtenOnline 2008 Giving useful feedback on open questions RechtenOnline K.U.Leuven Campus Kortrijk – Televic Education nv Learning visits June 2006 (1 BA Political and Social sciences) User statistics (3)

46. Bart Loosvelt - Bernard Tilleman- Piet Desmet - Bert Wylin RechtenOnline 2008 Giving useful feedback on open questions RechtenOnline K.U.Leuven Campus Kortrijk – Televic Education nv The content of the cases matches with the content of the course. Student satisfaction: survey N= 135

47. Bart Loosvelt - Bernard Tilleman- Piet Desmet - Bert Wylin RechtenOnline 2008 Giving useful feedback on open questions RechtenOnline K.U.Leuven Campus Kortrijk – Televic Education nv The use of electronic cases gives an educational extra.

48. Bart Loosvelt - Bernard Tilleman- Piet Desmet - Bert Wylin RechtenOnline 2008 Giving useful feedback on open questions RechtenOnline K.U.Leuven Campus Kortrijk – Televic Education nv The environment is attractive.

49. Bart Loosvelt - Bernard Tilleman- Piet Desmet - Bert Wylin RechtenOnline 2008 Giving useful feedback on open questions RechtenOnline K.U.Leuven Campus Kortrijk – Televic Education nv 7. Work in progress Avoiding meaningless errors (e.g. typing errors): by improving input (development of e.g. generator of “law articles”) by correction of input (before meaning-focused correction)  Spellchecker (SPL)  approximate string matching: Levinsthein and Dice coefficients

50. Bart Loosvelt - Bernard Tilleman- Piet Desmet - Bert Wylin RechtenOnline 2008 Giving useful feedback on open questions RechtenOnline K.U.Leuven Campus Kortrijk – Televic Education nv this presentation is available @ contact piet.desmet@kuleuven-kortrijk.be bert.wylin@kuleuven-kortrijk.be b.wylin@televic-education.com http:// www.itec-research.eu/presentations/worldcall http://www.edumatic.be Eduma-tic authoring tool @