University of Piraeus Department of Technology Education and Digital Systems Centre for Research and Technology - Hellas(C.E.R.T.H.) Informatics and Telematics Institute (I.T.I.) N. Manouselis & D.G. SampsonEDMEDIA04, Lugano, Switzerland, 24th June A Multi-criteria Model to Support Automatic Recommendation of e- Learning Quality Approaches Nikos Manouselis, Demetrios G. Sampson

University of Piraeus Department of Technology Education and Digital Systems Centre for Research and Technology - Hellas(C.E.R.T.H.) Informatics and Telematics Institute (I.T.I.) N. Manouselis & D.G. SampsonEDMEDIA04, Lugano, Switzerland, 24th June short overview Quality in e-learning Variety of approaches Need for selection Recommender systems Recommendation tasks Recommendation techniques European Quality Observatory (EQO) Model for describing e-learning quality approaches Online repository with quality approaches Recommendation mechanism Example scenario conclusions - future directions

University of Piraeus Department of Technology Education and Digital Systems Centre for Research and Technology - Hellas(C.E.R.T.H.) Informatics and Telematics Institute (I.T.I.) N. Manouselis & D.G. SampsonEDMEDIA04, Lugano, Switzerland, 24th June Quality in e-learning Quality: an issue of increasing importance in e-learning Several approaches developed worldwide They address quality control, assurance, and assessment dimensions of quality in e- learning A variety of available methods, criteria, processes, instruments etc.

University of Piraeus Department of Technology Education and Digital Systems Centre for Research and Technology - Hellas(C.E.R.T.H.) Informatics and Telematics Institute (I.T.I.) N. Manouselis & D.G. SampsonEDMEDIA04, Lugano, Switzerland, 24th June Variety of approaches The various approaches have different priorities, focusing on different e-learning quality aspects How can e-learning researchers and practitioners identify and select interesting or appropriate approaches for their application scenarios? Issues: Collection / Categorization of quality approaches Recommendation of quality approaches

University of Piraeus Department of Technology Education and Digital Systems Centre for Research and Technology - Hellas(C.E.R.T.H.) Informatics and Telematics Institute (I.T.I.) N. Manouselis & D.G. SampsonEDMEDIA04, Lugano, Switzerland, 24th June Variety of approaches The various approaches have different priorities, focusing on different e-learning quality aspects How can e-learning researchers and practitioners identify and select interesting or appropriate approaches for their application scenarios? Issues: Collection / Categorization of quality approaches Recommendation of quality approaches

University of Piraeus Department of Technology Education and Digital Systems Centre for Research and Technology - Hellas(C.E.R.T.H.) Informatics and Telematics Institute (I.T.I.) N. Manouselis & D.G. SampsonEDMEDIA04, Lugano, Switzerland, 24th June Recommender systems Any system that produces individualized recommendations as output or has the effect of guiding the user in a personalized way to interesting or useful items in a large space of possible options [Burke, 2002] They help individuals in a community of users to identify items of interest from a potentially overwhelming set of choices [Resnick & Varian, 1997]

University of Piraeus Department of Technology Education and Digital Systems Centre for Research and Technology - Hellas(C.E.R.T.H.) Informatics and Telematics Institute (I.T.I.) N. Manouselis & D.G. SampsonEDMEDIA04, Lugano, Switzerland, 24th June Recommendation tasks Recommendation broadly used to support two main user tasks [Herlocker et al., 2004] : Find Good Items: providing users with ranked lists of recommended items based on generated predictions Annotation in Context: generating predictions for an item that the user is currently viewing

University of Piraeus Department of Technology Education and Digital Systems Centre for Research and Technology - Hellas(C.E.R.T.H.) Informatics and Telematics Institute (I.T.I.) N. Manouselis & D.G. SampsonEDMEDIA04, Lugano, Switzerland, 24th June Recommendation techniques Collaborative recommendation Recognizing commonalities between users on the basis of commonly rated items Demographic recommendation Based on demographic classes of users Content-based recommendation Matching descriptions of items with user profiles Utility-based recommendation Based on a utility function for each user Knowledge-based recommendation Using knowledge structures for reasoning

University of Piraeus Department of Technology Education and Digital Systems Centre for Research and Technology - Hellas(C.E.R.T.H.) Informatics and Telematics Institute (I.T.I.) N. Manouselis & D.G. SampsonEDMEDIA04, Lugano, Switzerland, 24th June European Quality Observatory European Quality Observatory (EQO) initiative Funded by the EC eLearning Programme One of the four initiatives specifically focusing on e- learning quality Aims: To develop a metadata model for describing and categorizing quality approaches To collect quality approaches and store them in an online repository To support e-learning researchers and practitioners in identifying quality approaches suitable for their application needs

University of Piraeus Department of Technology Education and Digital Systems Centre for Research and Technology - Hellas(C.E.R.T.H.) Informatics and Telematics Institute (I.T.I.) N. Manouselis & D.G. SampsonEDMEDIA04, Lugano, Switzerland, 24th June EQO Model Metadata Model for the description and categorization of quality approaches for e-learning Extending an initial model of the CEN/ISSS Learning Technologies Workshop (LTW) Quality Assurance Group Expanding description and categorization capabilities of initial CEN/ISSS model Allowing for the description and assessment of a quality approach application

University of Piraeus Department of Technology Education and Digital Systems Centre for Research and Technology - Hellas(C.E.R.T.H.) Informatics and Telematics Institute (I.T.I.) N. Manouselis & D.G. SampsonEDMEDIA04, Lugano, Switzerland, 24th June Structure of Model EQO Model v.1.2 Four main categories of elements (available from GENERAL CONTEXT METHOD EXPERIENCE

University of Piraeus Department of Technology Education and Digital Systems Centre for Research and Technology - Hellas(C.E.R.T.H.) Informatics and Telematics Institute (I.T.I.) N. Manouselis & D.G. SampsonEDMEDIA04, Lugano, Switzerland, 24th June EQO web portal Provides access to the online repository Users can search/contribute descriptions of quality approaches Users can search/contribute experiences from quality approaches

University of Piraeus Department of Technology Education and Digital Systems Centre for Research and Technology - Hellas(C.E.R.T.H.) Informatics and Telematics Institute (I.T.I.) N. Manouselis & D.G. SampsonEDMEDIA04, Lugano, Switzerland, 24th June EQO Recommendation Module Provides recommendations to the users of the EQO web portal when searching for quality approaches in the repository Utility-based recommender, calculating the suitability of a quality approach (QA) for given application context priorities Multiple criteria engaged Ranking available quality approaches (Find Good Item task)

University of Piraeus Department of Technology Education and Digital Systems Centre for Research and Technology - Hellas(C.E.R.T.H.) Informatics and Telematics Institute (I.T.I.) N. Manouselis & D.G. SampsonEDMEDIA04, Lugano, Switzerland, 24th June Recommendation Life-Cycle Quality Approach Contribution Phase Expert contributing a quality approach provides its intended priorities and an initial estimation of its suitability Users contributing an experience express how suitable they found the quality approach for priorities of their application context Quality Approach Recommendation Phase User searching for a suitable quality approach specifies the priorities of the intended application Recommendation module calculates the suitability of a quality approach

University of Piraeus Department of Technology Education and Digital Systems Centre for Research and Technology - Hellas(C.E.R.T.H.) Informatics and Telematics Institute (I.T.I.) N. Manouselis & D.G. SampsonEDMEDIA04, Lugano, Switzerland, 24th June Recommendation Life-Cycle QA Contribution Priorities of QA Suitability of QA Expert contributing Quality Approach to EQO repository

University of Piraeus Department of Technology Education and Digital Systems Centre for Research and Technology - Hellas(C.E.R.T.H.) Informatics and Telematics Institute (I.T.I.) N. Manouselis & D.G. SampsonEDMEDIA04, Lugano, Switzerland, 24th June Recommendation Life-Cycle QA Contribution Priorities of QA Suitability of QA Expert contributing Quality Approach to EQO repository Experience Contribution Priorities of application Suitability of QA User contributing Experience from the Quality Approach to EQO repository

University of Piraeus Department of Technology Education and Digital Systems Centre for Research and Technology - Hellas(C.E.R.T.H.) Informatics and Telematics Institute (I.T.I.) N. Manouselis & D.G. SampsonEDMEDIA04, Lugano, Switzerland, 24th June Recommendation Life-Cycle QA Contribution Priorities of QA Suitability of QA Expert contributing Quality Approach to EQO repository QA Recommendation Priorities of intended application User providing priorities of the application context for which Quality Approach should be suitable Experience Contribution Priorities of application Suitability of QA User contributing Experience from the Quality Approach to EQO repository

University of Piraeus Department of Technology Education and Digital Systems Centre for Research and Technology - Hellas(C.E.R.T.H.) Informatics and Telematics Institute (I.T.I.) N. Manouselis & D.G. SampsonEDMEDIA04, Lugano, Switzerland, 24th June Recommendation Life-Cycle QA Contribution Priorities of QA Assessment of QA Expert contributing Quality Approach to EQO repository QA Recommendation Priorities of intended application User providing priorities of the application context for which Quality Approach should be suitable CONTRIBUTION PHASERECOMMENDATION PHASE Experience Contribution Priorities of application Assessment of QA User contributing Experience from the Quality Approach to EQO repository

University of Piraeus Department of Technology Education and Digital Systems Centre for Research and Technology - Hellas(C.E.R.T.H.) Informatics and Telematics Institute (I.T.I.) N. Manouselis & D.G. SampsonEDMEDIA04, Lugano, Switzerland, 24th June Recommendation Life-Cycle QA Contribution Priorities of QA Assessment of QA Expert contributing Quality Approach to EQO repository QA Recommendation Priorities of intended application User providing priorities of the application context for which Quality Approach should be suitable CONTRIBUTION PHASERECOMMENDATION PHASE Experience Contribution Priorities of application Assessment of QA User contributing Experience from the Quality Approach to EQO repository

University of Piraeus Department of Technology Education and Digital Systems Centre for Research and Technology - Hellas(C.E.R.T.H.) Informatics and Telematics Institute (I.T.I.) N. Manouselis & D.G. SampsonEDMEDIA04, Lugano, Switzerland, 24th June Criteria for calculating suitability Expressing the focus of a quality approach upon several criteria In our case, adopted from [Frydenberg, 2002] the following: 1.Focus on Institutional & executive organization / commitment 2.Focus on Technological infrastructure 3.Focus on Student services 4.Focus on Instructional design and course development 5.Focus on Instruction and instructors 6.Focus on Program delivery 7.Focus on Financial management 8.Focus on Regulatory and legal compliance 9.Focus on Evaluation and assessment procedures

University of Piraeus Department of Technology Education and Digital Systems Centre for Research and Technology - Hellas(C.E.R.T.H.) Informatics and Telematics Institute (I.T.I.) N. Manouselis & D.G. SampsonEDMEDIA04, Lugano, Switzerland, 24th June Providing search priorities

University of Piraeus Department of Technology Education and Digital Systems Centre for Research and Technology - Hellas(C.E.R.T.H.) Informatics and Telematics Institute (I.T.I.) N. Manouselis & D.G. SampsonEDMEDIA04, Lugano, Switzerland, 24th June Recommendation results

University of Piraeus Department of Technology Education and Digital Systems Centre for Research and Technology - Hellas(C.E.R.T.H.) Informatics and Telematics Institute (I.T.I.) N. Manouselis & D.G. SampsonEDMEDIA04, Lugano, Switzerland, 24th June Example scenario (1/3) Two users A and B searching for suitable QAs Each user has different priorities 12 QAs are stored in the repository, they are all candidate for recommendation g User A User B

University of Piraeus Department of Technology Education and Digital Systems Centre for Research and Technology - Hellas(C.E.R.T.H.) Informatics and Telematics Institute (I.T.I.) N. Manouselis & D.G. SampsonEDMEDIA04, Lugano, Switzerland, 24th June Example scenario (2/3) U QA f or User A U QA f or User B 1. QA120,1521. QA40, QA70,0882. QA70, QA20,0783. QA60, QA30,0764. QA20, QA60,0725. QA30, QA110,0656. QA90, QA90,0677. QA120, QA40,0618. QA110, QA50,0609. QA100, QA100, QA50, QA80, QA80, QA10, QA10,000 The suitability of each QA in the EQO repository is calculated The QAs are ranked according to their suitability

University of Piraeus Department of Technology Education and Digital Systems Centre for Research and Technology - Hellas(C.E.R.T.H.) Informatics and Telematics Institute (I.T.I.) N. Manouselis & D.G. SampsonEDMEDIA04, Lugano, Switzerland, 24th June Example scenario (3/3)

University of Piraeus Department of Technology Education and Digital Systems Centre for Research and Technology - Hellas(C.E.R.T.H.) Informatics and Telematics Institute (I.T.I.) N. Manouselis & D.G. SampsonEDMEDIA04, Lugano, Switzerland, 24th June Conclusions – Future directions The recommendation module seems to be providing useful hints to the users of the EQO portal evaluation from EQO users is required A utility-based recommender technique seems appropriate for cases similar to the EQO case study where multiple criteria are required to describe items other techniques could be also examined The criteria for calculating suitability could be furthermore elaborated and validated from the users explore the effect of user role in the definition of criteria validation of criteria through user survey

University of Piraeus Department of Technology Education and Digital Systems Centre for Research and Technology - Hellas(C.E.R.T.H.) Informatics and Telematics Institute (I.T.I.) N. Manouselis & D.G. SampsonEDMEDIA04, Lugano, Switzerland, 24th June Contact Details Nikos Manouselis and Prof. Demetrios Sampson Informatics and Telematics Institute, Centre for Research and Technology Hellas, 42, Arkadias Street,Athens,GR-15234,Greece & Department of Technology Education and Digital Systems, University of Piraeus 150, Androutsou Street, Piraeus, GR Greece