‘Towards a competency model for adaptive assessment to support lifelong learning’ Onjira Sitthisak, Lester Gilbert and Hugh C. Davis Learning Technologies Group ECS,University of Southampton TENCompetence workshop Service Oriented Approaches and Lifelong Competence Development Infrastructures Manchester, 11 th – 12 th January 2007

Assessment The main goal of assessment has shifted away from content-based education to intended learning outcomes. Assessment is related to the accomplishment of learning outcomes, i.e. competence. Learners expect to be able to maintain and expose their competency profiles to multiple services throughout their life.

Adaptive Assessment System Assessment is part of the process of diagnosing the learner’s knowledge level. One of the main goals includes offering personalized support according to the personal needs and ability of each learner (Brusilovsky, 1996) The “learner’s estimated knowledge level” can be used to guide the adaptation of the system.

Three problems with adaptive assessment 1. Inconsistency in estimating the learner’s knowledge level  Each system classifies ability or knowledge level and difficulty level of assessment using different approaches and techniques.  This causes interoperability and reusability problems if the learner’s knowledge level in one system needs to be used in other systems.

2. Limitation of a single numerical value for a learner’s knowledge level  Many well-known theories for selecting questions assume one numerical measure for the relationship between ability and item response.  A numerical value may be appropriate to decide who the best learner is, but educational evaluation intends to assess the learners’ readiness for further learning (Falmagne et al., 2003).  Selecting a question in adaptive assessment should therefore be multidimensional. Three problems with adaptive assessment

3. Dependency on a specific knowledge domain  In most cases, adaptive assessment systems are developed for a specific knowledge domain using particular rules and assessments without potential for knowledge reuse.  There is no standard to combine different knowledge domains with their assessments and learned capabilities (Cheniti-Belcadhi and Braham, 2004).  This highlights the problem of supporting the assessment of lifelong learning across multiple domains. Three problems with adaptive assessment

The need for competency The proposed solution is the use of “learned capability” instead of estimated “knowledge level”. Knowledge space theory suggests that a capability for solving problems can be established from achieved competencies (Doignon and Falmagne, 1985). A competency model should support  storing, organising and sharing  achieved, current, and intended performance data  all aspects of education and training  persistent and standard way.

Criteria for a competency model Competency should be defined with a rich data structure to support a learner’s competency profile throughout life. Meeting lifelong personal needs requires a highly flexible competency model. A competency model should support the selection of suitable questions in an adaptive assessment system. Competency should be concerned with specific, identifiable and measurable behaviours (Draganidis and Mentzas, 2006).

Proposed competency model

Competency model and Learning Design In IMS Learning Design (LD), prerequisites and learning objectives can be defined using an unstructured textual resource or an IMS RDCEO specification (Koper and Tattersall, 2005). This is inadequate for an instructional designer seeking to design learning activities, environments and assessments (Paquette and Rosca, 2004). IMS LD should incorporate a structured competency definition in order to implement a Unit Of Learning (UOL) with a solid instructional design foundation.

Competency model and service oriented architecture (SOA) Competency modelling should be the shared responsibility of governments, educational institutions, and businesses. SOA is currently considered the approach of choice in supporting cross-institutional cooperation and the design, build, and management of a distributed computing infrastructure. The proposed competency model is compatible with the SOA that may support a collaborative virtual teaching and learning environment (CVTLE) SOA (Gilbert et al., 2006).

Future work The proposed competency model is being developed and implemented in the JISC-funded Placement Learning and Assessment Toolkit (mPLAT) project ( at the University of Southampton. mPLAT project aims to provide a mobile learning toolkit to support practice based learning, mentoring and assessment of trainee nurses.

Conclusion A competency model is critical to successfully managing adaptive assessment and achieving the goals of resource sharing, collaboration and automation to support lifelong learning.

Thank you Questions, comments, discussion…

References Brusilovsky, P., 'Methods and Techniques of Adaptive Hypermedia', User Modeling and User-Adapted Interaction, Vol. 6, No. 2-3, pp Falmagne, J.-C., Cosyn, E., Doignon, J.-P. and Thiery, N. (2003) 'The Assessment of Knowledge, in Theory and in Practice', Integration of Knowledge Intensive Multi-Agent Systems. Cheniti-Belcadhi, L. and Braham, R. (2004) 'A generic framework for assessment in adaptive educational hypermedia', Proceedings of IADIS International Conference WWW/Internet 2004 (ICWI 2004). Madrid, Spain. Doignon, J.-P. and Falmagne, J.-C. (1985) 'Spaces for the Assessment of Knowledge', International Journal of Man-Machine Studies, No. 23, pp Draganidis, F. and Mentzas, G. (2006) 'Competency based management: a review of systems and approaches', Information Management & Computer Security, Vol. 14, No. 1, pp Koper, R. and Tattersall, C. (Eds.) (2005) Learning Design · A Handbook on Modelling and Delivering Networked Education and Training, Springer. Paquette, G. and Rosca, I. (2004) 'An ontology-based Referencing of Actors, Operations and Resources in eLearning Systems', the 2nd International Workshop on Applications of Semantic Web Technologies for E-Learning. Eindhoven, The Netherlands. Gilbert, L., Sitthisak, O., Sim, Y. W., Wang, C. and Wills, G. (2006) 'From collaborative virtual research environment to teaching and learning', In Proceedings of TENCompetence Workshop: Learning Networks for Lifelong Competence Development. Sofia, Bulgaria.