OpenACS and.LRN Conference 2008 Automatic Limited-Choice and Completion Test Creation, Assessment and Feedback in modern Learning Processes Institute for Information Systems and Computer Media (IICM) Faculty of Informatics – Graz University of Technology, Austria Author: Christian Gütl Presenter: Victor Manuel García-Barrios

© Christian Gütl 14/06/ Agenda Background & Motivation Vision Solution Approach Prototype Lessons Learned

© Christian Gütl 14/06/ Background & Motivation (1/2) Knowledge assessment –important for modern learning & teaching processes –face-to-face and e-learning Types of assessment over the life-cycle –Pre-knowledge assessment Enables to select proper learning content –Formative assessment Gives teachers and students continuously feedback Enables to revise and adapt learning & teaching process –Summative assessment Gives feedback about acquired knowledge at the end of a “learning unit”

© Christian Gütl 14/06/ Background & Motivation (2/2) Problem 1 –Huge effort to prepare proper tests Problem 2 –Additional effort to keep up-to-date with changing learning content –Multiple effort for personalized learning content –Dealing with unpredictable learning content (student assignments, background knowledge) Problem 3 –Different levels of knowledge requires different assessment methods –Didactically speaking Low level: limited choice, completion exercise High level: free-text answers, essays  How can we overcome these problems ?

© Christian Gütl 14/06/ Vision: The Big Picture Our long-term goal: move towards a computer-based assessment system –Automatic question generation (multiple choice, completion exercise, …) –Automatic assessment (challenging short free-text answer assessment) –Automatic Feedback

© Christian Gütl 14/06/ Vision: Our Current Focus Automatic Limited-Choice and Completion Test Creation, Assessment and Feedback in modern Learning Processes

© Christian Gütl 14/06/ Solution Approach: Method Content Objects (learning objects, student assignments, …) Automatic Text Summarization (statistic methods, sentence extraction) Concept Identification (noun, verbs, adjectives) Lexical Semantics Processing (synonyms, antonyms, related words ) Completion Exercises Limited Choice Exercises

© Christian Gütl 14/06/ Solution Approach: Architecture –Different file formats from local file system, remote (Web servers) and from ext. applications –Automatic Summarizer from [Visser & Wieling] –Word extraction by GATE (General Architecture for Text Engineering) system [Sheffield NLP group] –Candidate words used for exercise creation –Open interface to be used in various application scenarios

© Christian Gütl 14/06/ Prototype (1/5) Our First Prototype –Fully implemented in Java –Designed to be used as stand-alone tool for experimenting purposes –Includes a graphical user interface –Can be instantiated as application or as applet –Graphical user interface (GUI) supports Content import from file system or remote sources Content import by pasting into a text field

© Christian Gütl 14/06/ Prototype (2/5) Automatic summarization –based on compression rate –based on summarization method

© Christian Gütl 14/06/ Prototype (3/5) Candidate word identification –Word classes (noun, verb, adjective) –Word list (location, person, organization, …)

© Christian Gütl 14/06/ Prototype (4/5) Test creation –Number of words for word types configurable –Exercises randomly crated

© Christian Gütl 14/06/ Prototype (5/5) Solution of created exercise

© Christian Gütl 14/06/ Lessons Learned (& Future Work) Our proposed solution –Can support modern learning environments –First experiences of our approach are promissing Possible application scenarios –Stand-alone application –Module for pre-existing learning environments Future Work –Apply more advanced summarization techniques –Use more intelligent methods for exercise creation –Support the IMS QTI standard is not fully implemented yet

© Christian Gütl 14/06/ Questions & Contact Information Thank you for your Attention! Questions are welcome! Further Information: Christian Gütl

© Christian Gütl 14/06/ Literature Visser, W.T., & Wieling, M.B. Sentence-based Summarization of Scientific Documents. The design and implementation of an online available automatic summarizer. Report, last retrieved Nov. 29th, 2007 form GATE. Overview. Natural Language Processing Research Group, University of Sheffield, UK, last retrieved Nov. 29th, 2007 form Gütl, C. (2007). Moving towards a Fully-Automatic Knowledge Assessment Tool. iJET International Journal of Emerging Technologies in Learning, to be published.