1. WP5: Validation of developed functionality Report on the final Experiments Anne de Roeck Diane Evans James Gray The Open University - UK

2. Outline Introduction Business hypotheses Student Scenario (QUIZ) Results Desktop Activitiy Results Tutor Scenario Results Final Conclusions

3. Overall Hypothesis The augmentation of e-learning systems (in this case ILIAS) with NLP and semantic web technologies increases the effectiveness of learning and teaching, and in particular, Increases the effectiveness of teachers and learners in locating relevant learning objects in the context of learning related tasks.

4. Introduction Changes were made to the methodology after the previous tests and review. Measuring learning Performing Quantitative tests across the scenario activities Defined Hypotheses and formulated tests to measure Introduced Mitigation controls Also included value judgements as supporting analysis Restriction to 2 scenarios (1 student & 1 tutor) Scenarios built on experiences of previous trials Data captured electronically

5. Student Hypotheses

6. Retrieval of Content (Student) 1.1 The addition of functionalities based on NLP increases the retrievability of learning objects in terms of the relevance of the content. [Quant test] 1.2 The addition of functionalities based on NLP increases the effectiveness of learners in locating relevant learning objects in the context of answering Quiz questions. [Quant test]

7. Support for Learning 2.1 The developed tools and ontology help to support learners to better, more effectively grasp the terminological and conceptual space which defines a certain domain of knowledge. [Quant test]

8. Multi-lingual study 3.1 The ability to easily retrieve content in more than one language supports learning activities of students who are studying in a Multi-lingual situation e.g. the language of study is not their native language. [Quant test]

9. Support for Learning Paths and Personalised Learning 4.1 Learners are able to build individual learning paths by entering key terms of concepts they need to learn. [Qual. test] 4.2 Learners are able to classify and order learning material (in the form of documents) they have placed within their personal desktop by using concepts and keywords linked to each document [Qual. test]

10. Cont.. 4.3 The facility for a learner to classify and order learning material supports self-guided learning. [Qual. test] 4.4 The facility for a learner to classify and order learning material supports self-guided learning by enabling the user to create a meaningful / linked path through the content. [not tested]

11. Basis of Student scenario The student scenario focussed on: Demonstrating improvement in learning (as requested by the Reviewers) The LT4eL search functions i.e. Semantic, Definition, Keyword and Ontology Browsing. Multi-lingual search and retrieval associated with these search functions The Use of the Personal Desktop to support individual learning The scenario centred around a multiple choice QUIZ

12. Structure and rationale of the student scenario

13. Target - 1st year undergraduate humanities students Topics - Information exchange: protocols and markup languages You will shortly be studying a short course covering 'Information Exchange on the Internet'. This is a subject on which you have little or no previous knowledge. Your tutor has prepared some research questions for you in the form of a quiz to enable you to gain a basic grounding in key aspects. Depending on your group you will either use internet facilities such as Google and Wikipaedia etc or you will have access to different ILIAS search facilities to assist your search, including the LT4el searches and the structured concept browser.

14. Experiment design - students To assess whether technologies "increase the effectiveness of learning", students needed to do some learning: Learning experience - Quiz to encourage students to engage with learning objects Learning experience - Quiz to encourage students to engage with learning objects

15. Assessing the learning Assessment Activity Pre-Test Assessment Activity Pre-Test Learning experience (quiz) Learning experience (quiz) Assessment Activity Post-Test Assessment Activity Post-Test The answers provided by the student at each of the 3 stages were marked

16. Measuring the effect of our technology We introduced a control group with a different learning experience: Learning experience (quiz) ILLIAS / LT4eL Learning experience (quiz) ILLIAS / LT4eL Learning experience (quiz) Internet / Google Learning experience (quiz) Internet / Google TARGET CONTROL

17. To compare NLP technologies with plain search in the same repository PreTest Quiz Part 1 - ILIAS Quiz Part 2 - LT4eL PostTest PreTest Internet/Google PostTest TARGETCONTROL

18. Test A Question Set A ILIAS Question Set A ILIAS Controlling for question differences (eg is the post-test easier than the pre-test?): Question Set B LT4eL Question Set B LT4eL Test B Question Set B ILIAS Question Set B ILIAS Question Set A LT4eL Question Set A LT4eL Test A Quiz Part 1 Quiz Part 2 PreTest Post- Test Test A Question Set A WWW Question Set A WWW Question Set B WWW Question Set B WWW Test B Question Set B WWW Question Set B WWW Question Set A WWW Question Set A WWW Test A Group 2Group 3Group 4Group 1 TARGETCONTROL

19. Pre & Post-Tests Five questions testing understanding of relationships between concepts Pre-test and Post-test questions were paired, to test the same area Pre-test and Post-test questions were alternated in different groups to ensure fairness

20. Quiz Seven questions with learning materials in the same topics as the test questions designed to encourage engagement. Each question followed by supplementary questions to ascertain: whether student already knew the answer source document (including language) which tool (if any) helped find the answer Quiz is time-limited Question groups alternate between student groups

21. Results of Student Experiments

22. Main Student Scenario Searching Learning Multilinguality

23. Student Scenario: Searching Hypothesis 1.1: The addition of functionalities using / based on NLP increases the retrievability of learning objects in terms of the relevance of their content. Hypothesis 1.2: The addition of functionalities using / based on NLP increases the effectiveness of learners in locating relevant learning objects in the context of answering Quiz questions. Student: Searching

24. Test that mean scores in Part 2 of the quiz (LT4eL) are greater than mean scores in Part 1 of quiz (plain LMS) for the target group. Student: Searching Searching - Test 1 Test 1

25. Students scored higher in the quiz when they had access to LT4eL technologies to help them. Student: SearchingTest 1 Results

26. Was it the technology that helped? Question differences? no - rotating banks of questions Order effect? might expect students to be learning, thus achieve better results in second half of quiz. Test 1a: test by looking for improvement in control group Student: SearchingTest 1a

27. Hypothesis 1.1 - Gains achieved in both groups Control group scored better in both quizzes Student: SearchingTest 1a Results

28. Searching - Conclusions so far Students scored higher in the quiz when they had access to LT4eL technologies to help them. Partly explained by order effect - students do better in second half of quiz anyway Students using resources of World Wide Web score higher than those restricted to learning objects within an SMS LMS score improvement appears slightly better than WWW score improvement Student: SearchingConclusions 1

29. Searching - test 2 Test that for any given question in Part B of the quiz, students who choose to use LT4eL functionality score more highly for that question than those who do not use LT4eL functionality. Note: In Part 2 of the quiz, students are free to use whichever search method they feel will be most effective. This test is to assess whether those who use LT4eL methods score more highly than those who do not. Student: SearchingTest 2

30. Only three questions answered better using LT4eL functionality Student: SearchingTest 2 Results

31. Student: SearchingTest 2a Results

32. (target groups only) Student: SearchingOpinions 2

33. What did they dislike about Semantic Search? It didn't return relevant results. because it doesn't find what I am searching for its too vague i didn't use it a lot as the results were chaotic i find it not much to the point for the types of research i usually do It is a bit too much to offer this much search methods the name semantic is confusing i liked this type best.. it was the easier to find the relevant information Student: SearchingComments: Semantic

34. What did they dislike about the Concept Browser? It didn't return relevant results It was too slow for my part and did not give any additional value it's a roundabout way of searching was not eay to use it. maybe this was because i did not fully understand how the concept browser worked I am not sure what the concept browser is don't know what it is. for content questions this might be a relevant search method. However, less relevant when studying a language I like that method the most but it wouldn't be useful in my studies - English philology it helps a lot to understand given topic/term Student: SearchingComments: Concept Br

35. Student: SearchingOpinions 1

36. Searching - Test 3 Test that for any given question in Part B of the quiz, students who choose to use LT4eL functionality and answer correctly, do so more quickly than those who answer the question correctly using Full Text Search Notes: In Part 2 of the quiz, students are free to use whichever search method they feel will be most effective. This test is to assess whether those who use LT4eL methods are faster than those who do not. Only consider students who searched for an answer, and answered correctly Student: SearchingTest 3

37. Students answered faster using LT4eL technologies than using full text search. Student: SearchingTest 3 result

38. Student: Searching

39. Searching - conclusions Using Semantic Search, Concept Browser and Definition Finder to find answers, students score better than using plain text search Students appreciate all functionality, especially the Definition Finder; Students' ratings of Semantic Search and Concept Browser, though favourable, showed that these techniques were less familiar to them. Students answered faster using LT4eL technologies than using full text search; speed of searching seems to improve with practice. ConclusionsStudent: Searching

40. Hypothesis 2.1: The developed tools and ontology help / support learners to better / more effectively grasp the terminological and conceptual space which shape a certain domain of knowledge. Student: LearningTest 1 Test: Assess learning by testing that the mean score in the post-test is higher than the mean score in the pre-test

41. Learning has taken place. This validates an important part of the methodology. Student: LearningTest 1 - result

42. Learning - test 2 Test that mean improvement in test scores for target sample (using ILIAS and LT4eL) is higher than mean improvement in test scores for control group (using Internet to find content). Note: This test is intended to show whether learning is more effective in the target sample than in the control sample. Student: LearningTest 2

43. Learning appears slightly greater in Control group; but they started from slightly lower base. But not statistically significant Student: LearningTest 2 - result

44. Learning - Test 3 Test whether students who use the LT4eL functionality in the LMS learn more than those who do not. Student: LearningTest 3

45. Student: LearningTest 3 - result

46. Learning - Conclusions Learning took place in target and control group Students in the target group who used LT4eL functionality extensively, learned more than those who stuck to plain text search. Student: LearningConclusions

47. Multilinguality Hypothesis: The ability to easily retrieve content in more than one language supports learning activities of students who are studying in a Multi-lingual situation. Student: Multilinguality Notes: We considered test persons who are capable to read and understand documents in other languages than their native. In normal situation we expect also students who are studying in other language that their native (exchange students) to make use of the multilingual facilities and find documents in their language For our tests this was however not replicable as we would not have been able to ensure a relevant sample size

48. For the quiz: Allow students to select most appropriate documents to answer questions, irrespective of language Students will retrieve greater linguistic diversity of documents Students who find documents in other languages will score highly Finding Documents Getting correct answers Student: MultilingualityTests

49. Multilinguality - Test 1 Searching Test: Allow students in a quiz to select the most appropriate document to answer the questions, irrespective of language. We expect those using NLP-based functionalities to select a higher proportion of documents in a non- course language Test 1Student: Multilinguality

50. Test 1 result

51. Conclusions so far Target group retrieved more non-course language documents when they had access to LT4eL functionality, showing that multilingual access supported the learning activity No significant increase in non-course language retrieval for control group, suggesting this is not an order effect Student: Multilinguality

52. Test: test that, of students who are studying in a non-native language, those who retrieve second-language documents in Quiz Part 2 (either target or control) score more highly for those questions than those who do not. Note: This is intended to show that multilingual retrieval is helpful in the context of answering quiz questions Student: MultilingualityTest 2 Multilinguality - Test 2 Getting the right answer

53. Student: MultilingualityTest 2 result

54. Those retrieving documents in a non-course language score statistically the same for those questions as those who retrieved documents in their main course language. These results depend on the quantity and quality of information in different languages available within the repository being searched (or across the Internet for control group students). It is possible that students retrieving in a second-language document are less fluent in that language, hence have more difficulty assimilating the information within the document. In 328 cases (48%) the language of retrieval was English. It is also possible that students only searched in a second language when they had failed to find the answer in their course language. Student: Multilinguality

55. Multilinguality - test 3 Does our functionality aid multilingual retrieval? Student: MultilingualityTest 3

56. Student: MultilingualityTest 3 result

57. Student Scenario Qualitative evaluation of multilingual Search Multilingual search would be useful for my studies ? 71% (157 Students) STRONGLY AGREED 12% (25 Students) DISAGREED Lower results for students in countries where large collections of materials in their native language are available (EN, DE) or in countries were usually materials (at least for the particular domain of the test) are in English (NL) Student: MultilingualityOpinions

58. Multilinguality conclusions Students retrieved more non-course language documents when they had access to LT4eL functionality, but this did not result in higher quiz scores. LT4eL functionality helped retrieval of multilingual content - particularly Concept Browser, Definition Finder and Semantic Search Student: MultilingualityConclusions

59. Overall conclusions - Students Using Semantic Search, Concept Browser and Definition Finder to find answers, students score better than using plain text search Students most appreciate Definition Finder; they found it harder to see the benefit to them of Semantic Search and Concept Browser Students answered faster using LT4eL technologies than using full text search Learning took place in target and control group Students in the target group who used LT4eL functionality extensively, learned more than those who stuck to plain text search. Students retrieved more non-course language documents when they had access to LT4eL functionality, but this did not result in higher quiz scores. LT4eL functionality helped retrieval of multilingual content - particularly Concept Browser, Definition Finder and Semantic Search StudentsOverall conclusions

60. Personal Desktop experiments

61. Create classifications and organise their content within these sections Use LT4eL functionality to locate other resources in own and course languages. Organise these into structure Student tasks required them to: Evaluate the usefulness of these features for their study

62. Students were able to classify documents into topic sections within the personal desktop. Using the concepts and keywords linked to each document helped them do the classification Students were able to order documents within the Sections RESULTS

63. Student Scenario Multilingual support with the Personal Desktop Finding, classifying and ordering of multilingual documents within the personal desktop is possible using the NLP based functionality ? 62% (10% strongly) AGREED that they were able to classify and order documents in more than one language based on key concepts 75% (24% strongly) AGREED that they found documents in more than one language using concept and topics Student: MultilingualityOpinions

64. Students agreed that: Features would help students organise their study Multiple course modules Individual directed study Revision Research Having classified and ordered documents, they have a clearer understanding of the relationships between the topics

65. Tutor Hypotheses

66. Support for Teaching in a multi-lingual context - finding resources The addition of functionalities based on NLP increases the effectiveness of tutors in identifying supplementary resources to support students who are studying in a multi-lingual context. [Quant test] The addition of functionalities based on NLP increases the effectiveness of tutors in identifying documents that are relevant, reliable, and appropriate for level.. [Quant test]

67. Content Preparation Using the Keyword Extractor based on NLP increases the quality and consistency of keywords assigned to repository content and helps tutors who are expected to perform this task. [Quant test] Using the Keyword Extractor based on NLP enables tutors to produce a set of keywords faster than those tutors who perform the task unsupported [Quant test]

68. Basis of Tutor Scenario The tutor scenario focussed on: Multi-lingual search the LT4eL functions of Semantic search and ontology browsing The Keyword Generator for new content Both in English and in native language

69. Tutor scenario design

70. The tutor scenario was based on the need to find content to support students learning and to add additional content to the repository

71. Tutor tasks Internet Resources LMS resources Locate Content Group1 - ILIAS Group2 - LT4eL

72. Search Topics Students will learn which markup languages are appropriate for holding data. Data formats on the Web Students will be able to identify ways to construct links to internal and external resources. links in markup languages Students will be able to identify which protocols are required to transfer files over the internet file transfer and internet protocols DescriptionTopic Session 1 - Set A topics Students will learn some of the different picture formats & how to select an appropriate one for different purposes. picture formats on the Web Students will be able to identify how to present data in tabular form on web pages tables in markup language Students will be able to identify which protocols are used for sending & receiving emails email and internet protocols DescriptionTopic Session 2 -Set B topics

73. Add New Content Group1 - ILIAS Group2 – LT4eL

74. Tutor Scenario - Results Searching Multilinguality Keyword addition

75. Hypothesis: The addition of functionalities based on NLP increases the effectiveness of tutors in identifying documents that are relevant, reliable, and appropriate for level. Test within the same repository Use tutors' own assessments of relevance, reliability and appropriateness for level. Tutor: SearchingTest

76. Tutor: SearchingResults

77. Searching - conclusion LT4eL functionality makes no difference to the tutors' assessments of the Relevance, Reliability or Level of the documents found when searching within the repository. Tutor: SearchingConclusion

78. Hypothesis: The addition of functionalities based on NLP increases the effectiveness of tutors in identifying supplementary resources to support students who are studying in a multi- lingual context. i.e. tutors can find content in two languages Tutor: Multilingual

79. In more detail... Tutors are better able to find second language documents using LT4eL than using Internet search / plain ILIAS search in the same repository. Tutors find location of second language documents using LT4eL is relatively faster than using Internet searches / standard ILIAS. Tutors find location of second language documents using LT4eL is relatively easier than using Internet searches / standard ILIAS in the same repository. Tutor: MultilingualTests

80. Tutors are better able to find second language documents using LT4eL Tutor: MultilingualTest 1

81. Tutors find location of second language documents using LT4eL is relatively faster than using Internet searches / standard ILIAS Tutor: MultilingualTest 2

82. Tutors find location of second language documents using LT4eL is relatively easier than using Internet searches / standard ILIAS in the same repository. Tutor: MultilingualTest 3

83. Multilingual searches are more successful on the Internet than within repository. Within the repository, LT4eL functionality does not improve ability to find multilingual content Within the repository, when two documents are found, tutors using are more likely to find both documents together (same speed, same ease). Tutor: MultilingualConclusions

84. Keywords Using the Keyword Extractor based on NLP increases the quality and consistency of keywords assigned to repository content and helps tutors who are expected to perform this task Test: Tutors using the keyword generator produce a more consistent set of keywords than those who do not Tutor: KeywordsTest 1

85. Tutor: KeywordsTest 1 results

86. Keywords - test 2 Using the Keyword Extractor based on NLP enables tutors to produce a set of keywords faster than those tutors who perform the task unsupported. Tutors assessed their own time to complete the exercise: Tutor: KeywordsTest 2

87. Tutor: KeywordsTest 2 results

88. Keywords - test 3 Using keyword extractor gives: more consistent set of keywords faster results But are tutors using the extractor just assigning fewer keywords? Tutor: KeywordsTest 3

89. Tutor: KeywordsTest 3 results

90. Tutor: KeywordsOpinions base = 23 base = 27

91. Keywords - Conclusions Tutor: Keywords Using the keyword extractor yields a much more consistent set of keywords than manual Using the keyword extractor, tutors assign keywords 30% faster than not using it. About 90% of tutors said the keyword generator helped them Conclusions

92. Overall conclusions - Tutors Within the repository, LT4eL appears to make finding multilingual documents together slightly more likely (same speed, same ease). LT4eL functionality makes no difference to the Relevance, Reliability or Level of the documents found Using the keyword extractor yields a much more consistent set of keywords than manual Using the keyword extractor, tutors assign keywords 30% faster than not using it. TutorOverall Conclusions

93. Evaluation - Conclusions

94. WP5 Participation in Experiments

95. Participants in Student Scenario 22637387675Total per group 36661311Romania 244389Portugal 31551011Poland 275499Netherlands 41771413Malta 244488Germany 244488Czechoslovakia 19 (23)2566 (*10)Bulgaria TotalGroup 4Group3Group 2Group1 8 languages and 226 students

96. Participants in Student Personal Desktop Activity 12092016182713116 TotalRomaniaPortugalPolandNetherlandsMaltaGermanCzechBulgaria 8 Languages and 120 students

97. Participants in Tutor Scenario 71Total 11Netherlands 12Romania11German 10Portugal10Czech 11Poland6Bulgaria 7 Languages and 71 tutors

98. Not a trivial task! 8 partners Pre-test, post-test, Quiz-A, Quiz-B, Desktop questionnaire, Tutor questionnaires - all translated into host languages (and verbatim answers translated back to English) 152 data files to download, clean, and load into database Analysis database contains: 18 tables, 585 different fields, 2483 records. The total number of non-blank data items is 66,725. Over 6,500 lines of code to manipulate, condition, load and analyse that data.

99. Overall Hypothesis The augmentation of e-learning systems (in this case ILIAS) with NLP and semantic web technologies increases the effectiveness of learning and teaching, and in particular, Increases the effectiveness of teachers and learners in locating relevant learning objects in the context of learning related tasks.