TENCompetence Assessment Model, Related Tools and their Evaluation Milen Petrov, Adelina Aleksieva-Petrova, Krassen Stefanov, Judith Schoonenboom, Yongwu Miao

Content TENCompetence Assessment Model Assessment Editor Assessment Run-time system Evaluation plan Main Results Conclusions

Main goals Design of an assessment methodology for competences, through analysis of the modern assessment methods, selection of proper methods and tools and design of basic assessment activities. Outline the major principles for planning and design of effective assessment and provides a framework and guidelines for the design of the unit of assessment.

TenCompetence Assessment Model Describes the life-cycle of the assessment Specified as formal specification using UML diagrams Optimised for competence development assessments Developed through simplification of the existing OUNL/CITO Model

Assessment model components Assessment design Item construction Assessment construction Assessment run Response rating

The reasons for simplifying OUNL/CITO assessment model The OUNL/CITO model is extensive and complex, aiming for completeness in its coverage of all forms of assessment. The TENCompetence Assessment Model, as part of the TENCompetence Domain Model, needs not to duplicate it’s components

Proof-of-concept tools Needed to validate the model Two competence assessment methods demonstrated: 360 degree feedback and Portfolio assessment Assessment Authoring tool Assessment run-time tool Developed as Java Eclipse plug-ins

Relation between TENCompetence assessment model, QTI&LD specifications and first proof-of-concept-tools

TENCompetence Assessment Data Model Why needed – to provide the needed semantics for the Assessment specification Specified as a XML schema Used to store the information about any specific details regarding any stage of the assessment process Used as an output from editing and input for the run-time

Functionality of the tools

Assessment Authoring tool Covers the following three phases of the TENCompetence Assessment Model: Assessment design, Item construction and Assessment construction Implemented as 360 degree feedback competence assessment method editor Results in an XML Schema according to the TENCompetence Assessment Data Model

The Authoring tool components Design phase Editor - define the blueprint of the assessment Question editor (item construction phase) - create and edit different type of questions (demonstration items, construction items, and selection items) Assessment Test Editor - combine assessment architect and assessment items

Assessment run-time tool Runtime environment for playing any non- traditional method of assessment Supporting the TENCompetence Assessment data model Implemented to support the Portfolio Assessment method Assessment run is based on an XML Schema according to the TENCompetence Assessment Data Model

The Runtime tool components Assessment run-time simulator – loads and perform non-traditional forms of assessment (like portfolio assessment peer review, etc) Response processing – for tracking the results from the assessment

Evaluation goals Is the model able to be used for the implementation of different competence assessment methods Are the tools capable to demonstrate the applicability of the model What is the complexity of these tasks

Evaluation Methodology Evaluate the functional quality of the tools (test case review checklist) Rate the interface and usability of the systems (end user questionnaire) Evaluate the software code quality and complexity (expert evaluation checklist)

Users involved Students for the usability testing Educational technology experts for testing the functional quality of the tools Software technology experts for the evaluation of the code quality and complexity

Design principles of ISO 9241 (Part 10) suitability to task self explanatory controllability conformity with user expectations error tolerance suitability for individualisation

Part of the given user questionnaire

Evaluation procedure Step 1: Download the evaluation bundle (user guide and assessment instruments) Step 2: Download the corresponding proof-of-concept tool Step 3: Un-package and install the software Step 4: Work with the software Step 5: Fill-in the assessment instrument Step 6: Return the filled assessment instrument

Details of the evaluation process Description Number of usersNote Software Unique users downloads 52 users3 – test users unique users/after removing test-users and anonymous users 48 users + 1 anonymous1 anonymous Software non-unique users 73 DownloadsSoftware downloads and other items from site Returned assessment instruments 33 users All valid Returned feedback form20 users3-invalid; 2-blank

ISO Evaluation Results Design principles of ISO 9241 Average result Suitability to task3,69 Self explanatory3,64 Controllability3,79 Conformity to user expectations3,08 Error tolerance3,33 Suitability for individualisation3,62

Main Evaluation Results Expert software code evaluation results: good quality of the code but very high complexity if needed to go for real life tools Functional quality results: very good functional quality Usability evaluation results: overall good quality, but several errors and issues identified

Evaluation Conclusions The tools proved the applicability of the model The Model has sufficient modelling power The complexity needed to go from proof-of-concept to real-life tools is very high and risky

Next steps 1) Further research needed into improving the mapping algorithms which are used to carry out the transformation to the data model 2) Revise and extend the TENCompetence Assessment Data Model, using elements with the functionally similar to those used in IMS LD and QTI

Questions?