1. OpenACS and.LRN Conference 2008 Automatic Limited-Choice and Completion Test Creation, Assessment and Feedback in modern Learning Processes Institute for Information Systems and Computer Media (IICM) Faculty of Informatics – Graz University of Technology, Austria Author: Christian Gütl Presenter: Victor Manuel García-Barrios

2. © 2007 - Christian Gütl 14/06/2016 2 Agenda Background & Motivation Vision Solution Approach Prototype Lessons Learned

3. © 2007 - Christian Gütl 14/06/2016 3 Background & Motivation (1/2) Knowledge assessment –important for modern learning & teaching processes –face-to-face and e-learning Types of assessment over the life-cycle –Pre-knowledge assessment Enables to select proper learning content –Formative assessment Gives teachers and students continuously feedback Enables to revise and adapt learning & teaching process –Summative assessment Gives feedback about acquired knowledge at the end of a “learning unit”

4. © 2007 - Christian Gütl 14/06/2016 4 Background & Motivation (2/2) Problem 1 –Huge effort to prepare proper tests Problem 2 –Additional effort to keep up-to-date with changing learning content –Multiple effort for personalized learning content –Dealing with unpredictable learning content (student assignments, background knowledge) Problem 3 –Different levels of knowledge requires different assessment methods –Didactically speaking Low level: limited choice, completion exercise High level: free-text answers, essays  How can we overcome these problems ?

5. © 2007 - Christian Gütl 14/06/2016 5 Vision: The Big Picture Our long-term goal: move towards a computer-based assessment system –Automatic question generation (multiple choice, completion exercise, …) –Automatic assessment (challenging short free-text answer assessment) –Automatic Feedback

6. © 2007 - Christian Gütl 14/06/2016 6 Vision: Our Current Focus Automatic Limited-Choice and Completion Test Creation, Assessment and Feedback in modern Learning Processes

7. © 2007 - Christian Gütl 14/06/2016 7 Solution Approach: Method Content Objects (learning objects, student assignments, …) Automatic Text Summarization (statistic methods, sentence extraction) Concept Identification (noun, verbs, adjectives) Lexical Semantics Processing (synonyms, antonyms, related words ) Completion Exercises Limited Choice Exercises

8. © 2007 - Christian Gütl 14/06/2016 8 Solution Approach: Architecture –Different file formats from local file system, remote (Web servers) and from ext. applications –Automatic Summarizer from [Visser & Wieling] –Word extraction by GATE (General Architecture for Text Engineering) system [Sheffield NLP group] –Candidate words used for exercise creation –Open interface to be used in various application scenarios

9. © 2007 - Christian Gütl 14/06/2016 9 Prototype (1/5) Our First Prototype –Fully implemented in Java –Designed to be used as stand-alone tool for experimenting purposes –Includes a graphical user interface –Can be instantiated as application or as applet –Graphical user interface (GUI) supports Content import from file system or remote sources Content import by pasting into a text field

10. © 2007 - Christian Gütl 14/06/2016 10 Prototype (2/5) Automatic summarization –based on compression rate –based on summarization method

11. © 2007 - Christian Gütl 14/06/2016 11 Prototype (3/5) Candidate word identification –Word classes (noun, verb, adjective) –Word list (location, person, organization, …)

12. © 2007 - Christian Gütl 14/06/2016 12 Prototype (4/5) Test creation –Number of words for word types configurable –Exercises randomly crated

13. © 2007 - Christian Gütl 14/06/2016 13 Prototype (5/5) Solution of created exercise

14. © 2007 - Christian Gütl 14/06/2016 14 Lessons Learned (& Future Work) Our proposed solution –Can support modern learning environments –First experiences of our approach are promissing Possible application scenarios –Stand-alone application –Module for pre-existing learning environments Future Work –Apply more advanced summarization techniques –Use more intelligent methods for exercise creation –Support the IMS QTI standard is not fully implemented yet

15. © 2007 - Christian Gütl 14/06/2016 15 Questions & Contact Information Thank you for your Attention! Questions are welcome! Further Information: Christian Gütl http://www.iicm.edu/cguetl cguetl@iicm.edu

16. © 2007 - Christian Gütl 14/06/2016 16 Literature Visser, W.T., & Wieling, M.B. Sentence-based Summarization of Scientific Documents. The design and implementation of an online available automatic summarizer. Report, last retrieved Nov. 29th, 2007 form http://home.hccnet.nl/m.b.wieling/files/wielingvisser05automaticsummarization.pdf GATE. Overview. Natural Language Processing Research Group, University of Sheffield, UK, last retrieved Nov. 29th, 2007 form http://gate.ac.uk/sale/gate-flyer/2007/gate-flyer-4-page.pdf Gütl, C. (2007). Moving towards a Fully-Automatic Knowledge Assessment Tool. iJET International Journal of Emerging Technologies in Learning, to be published.