1. University of Piraeus Department of Technology Education and Digital Systems Centre for Research and Technology - Hellas(C.E.R.T.H.) Informatics and Telematics Institute (I.T.I.) N. Manouselis & D.G. SampsonEDMEDIA04, Lugano, Switzerland, 24th June 2004 1 A Multi-criteria Model to Support Automatic Recommendation of e- Learning Quality Approaches Nikos Manouselis, Demetrios G. Sampson e-mail:{nikosm,sampson}@iti.gr

2. University of Piraeus Department of Technology Education and Digital Systems Centre for Research and Technology - Hellas(C.E.R.T.H.) Informatics and Telematics Institute (I.T.I.) N. Manouselis & D.G. SampsonEDMEDIA04, Lugano, Switzerland, 24th June 2004 2 short overview Quality in e-learning Variety of approaches Need for selection Recommender systems Recommendation tasks Recommendation techniques European Quality Observatory (EQO) Model for describing e-learning quality approaches Online repository with quality approaches Recommendation mechanism Example scenario conclusions - future directions

3. University of Piraeus Department of Technology Education and Digital Systems Centre for Research and Technology - Hellas(C.E.R.T.H.) Informatics and Telematics Institute (I.T.I.) N. Manouselis & D.G. SampsonEDMEDIA04, Lugano, Switzerland, 24th June 2004 3 Quality in e-learning Quality: an issue of increasing importance in e-learning Several approaches developed worldwide They address quality control, assurance, and assessment dimensions of quality in e- learning A variety of available methods, criteria, processes, instruments etc.

4. University of Piraeus Department of Technology Education and Digital Systems Centre for Research and Technology - Hellas(C.E.R.T.H.) Informatics and Telematics Institute (I.T.I.) N. Manouselis & D.G. SampsonEDMEDIA04, Lugano, Switzerland, 24th June 2004 4 Variety of approaches The various approaches have different priorities, focusing on different e-learning quality aspects How can e-learning researchers and practitioners identify and select interesting or appropriate approaches for their application scenarios? Issues: Collection / Categorization of quality approaches Recommendation of quality approaches

5. University of Piraeus Department of Technology Education and Digital Systems Centre for Research and Technology - Hellas(C.E.R.T.H.) Informatics and Telematics Institute (I.T.I.) N. Manouselis & D.G. SampsonEDMEDIA04, Lugano, Switzerland, 24th June 2004 5 Variety of approaches The various approaches have different priorities, focusing on different e-learning quality aspects How can e-learning researchers and practitioners identify and select interesting or appropriate approaches for their application scenarios? Issues: Collection / Categorization of quality approaches Recommendation of quality approaches

6. University of Piraeus Department of Technology Education and Digital Systems Centre for Research and Technology - Hellas(C.E.R.T.H.) Informatics and Telematics Institute (I.T.I.) N. Manouselis & D.G. SampsonEDMEDIA04, Lugano, Switzerland, 24th June 2004 6 Recommender systems Any system that produces individualized recommendations as output or has the effect of guiding the user in a personalized way to interesting or useful items in a large space of possible options [Burke, 2002] They help individuals in a community of users to identify items of interest from a potentially overwhelming set of choices [Resnick & Varian, 1997]

7. University of Piraeus Department of Technology Education and Digital Systems Centre for Research and Technology - Hellas(C.E.R.T.H.) Informatics and Telematics Institute (I.T.I.) N. Manouselis & D.G. SampsonEDMEDIA04, Lugano, Switzerland, 24th June 2004 7 Recommendation tasks Recommendation broadly used to support two main user tasks [Herlocker et al., 2004] : Find Good Items: providing users with ranked lists of recommended items based on generated predictions Annotation in Context: generating predictions for an item that the user is currently viewing

8. University of Piraeus Department of Technology Education and Digital Systems Centre for Research and Technology - Hellas(C.E.R.T.H.) Informatics and Telematics Institute (I.T.I.) N. Manouselis & D.G. SampsonEDMEDIA04, Lugano, Switzerland, 24th June 2004 8 Recommendation techniques Collaborative recommendation Recognizing commonalities between users on the basis of commonly rated items Demographic recommendation Based on demographic classes of users Content-based recommendation Matching descriptions of items with user profiles Utility-based recommendation Based on a utility function for each user Knowledge-based recommendation Using knowledge structures for reasoning

9. University of Piraeus Department of Technology Education and Digital Systems Centre for Research and Technology - Hellas(C.E.R.T.H.) Informatics and Telematics Institute (I.T.I.) N. Manouselis & D.G. SampsonEDMEDIA04, Lugano, Switzerland, 24th June 2004 9 European Quality Observatory European Quality Observatory (EQO) initiative Funded by the EC eLearning Programme One of the four initiatives specifically focusing on e- learning quality Aims: To develop a metadata model for describing and categorizing quality approaches To collect quality approaches and store them in an online repository To support e-learning researchers and practitioners in identifying quality approaches suitable for their application needs

10. University of Piraeus Department of Technology Education and Digital Systems Centre for Research and Technology - Hellas(C.E.R.T.H.) Informatics and Telematics Institute (I.T.I.) N. Manouselis & D.G. SampsonEDMEDIA04, Lugano, Switzerland, 24th June 2004 10 EQO Model Metadata Model for the description and categorization of quality approaches for e-learning Extending an initial model of the CEN/ISSS Learning Technologies Workshop (LTW) Quality Assurance Group Expanding description and categorization capabilities of initial CEN/ISSS model Allowing for the description and assessment of a quality approach application

11. University of Piraeus Department of Technology Education and Digital Systems Centre for Research and Technology - Hellas(C.E.R.T.H.) Informatics and Telematics Institute (I.T.I.) N. Manouselis & D.G. SampsonEDMEDIA04, Lugano, Switzerland, 24th June 2004 11 Structure of Model EQO Model v.1.2 Four main categories of elements (available from http://www.eqo.info) GENERAL CONTEXT METHOD EXPERIENCE

12. University of Piraeus Department of Technology Education and Digital Systems Centre for Research and Technology - Hellas(C.E.R.T.H.) Informatics and Telematics Institute (I.T.I.) N. Manouselis & D.G. SampsonEDMEDIA04, Lugano, Switzerland, 24th June 2004 12 EQO web portal Provides access to the online repository Users can search/contribute descriptions of quality approaches Users can search/contribute experiences from quality approaches http://www.eqo.info

13. University of Piraeus Department of Technology Education and Digital Systems Centre for Research and Technology - Hellas(C.E.R.T.H.) Informatics and Telematics Institute (I.T.I.) N. Manouselis & D.G. SampsonEDMEDIA04, Lugano, Switzerland, 24th June 2004 13 EQO Recommendation Module Provides recommendations to the users of the EQO web portal when searching for quality approaches in the repository Utility-based recommender, calculating the suitability of a quality approach (QA) for given application context priorities Multiple criteria engaged Ranking available quality approaches (Find Good Item task)

14. University of Piraeus Department of Technology Education and Digital Systems Centre for Research and Technology - Hellas(C.E.R.T.H.) Informatics and Telematics Institute (I.T.I.) N. Manouselis & D.G. SampsonEDMEDIA04, Lugano, Switzerland, 24th June 2004 14 Recommendation Life-Cycle Quality Approach Contribution Phase Expert contributing a quality approach provides its intended priorities and an initial estimation of its suitability Users contributing an experience express how suitable they found the quality approach for priorities of their application context Quality Approach Recommendation Phase User searching for a suitable quality approach specifies the priorities of the intended application Recommendation module calculates the suitability of a quality approach

15. University of Piraeus Department of Technology Education and Digital Systems Centre for Research and Technology - Hellas(C.E.R.T.H.) Informatics and Telematics Institute (I.T.I.) N. Manouselis & D.G. SampsonEDMEDIA04, Lugano, Switzerland, 24th June 2004 15 Recommendation Life-Cycle QA Contribution Priorities of QA Suitability of QA Expert contributing Quality Approach to EQO repository

16. University of Piraeus Department of Technology Education and Digital Systems Centre for Research and Technology - Hellas(C.E.R.T.H.) Informatics and Telematics Institute (I.T.I.) N. Manouselis & D.G. SampsonEDMEDIA04, Lugano, Switzerland, 24th June 2004 16 Recommendation Life-Cycle QA Contribution Priorities of QA Suitability of QA Expert contributing Quality Approach to EQO repository Experience Contribution Priorities of application Suitability of QA User contributing Experience from the Quality Approach to EQO repository

17. University of Piraeus Department of Technology Education and Digital Systems Centre for Research and Technology - Hellas(C.E.R.T.H.) Informatics and Telematics Institute (I.T.I.) N. Manouselis & D.G. SampsonEDMEDIA04, Lugano, Switzerland, 24th June 2004 17 Recommendation Life-Cycle QA Contribution Priorities of QA Suitability of QA Expert contributing Quality Approach to EQO repository QA Recommendation Priorities of intended application User providing priorities of the application context for which Quality Approach should be suitable Experience Contribution Priorities of application Suitability of QA User contributing Experience from the Quality Approach to EQO repository

18. University of Piraeus Department of Technology Education and Digital Systems Centre for Research and Technology - Hellas(C.E.R.T.H.) Informatics and Telematics Institute (I.T.I.) N. Manouselis & D.G. SampsonEDMEDIA04, Lugano, Switzerland, 24th June 2004 18 Recommendation Life-Cycle QA Contribution Priorities of QA Assessment of QA Expert contributing Quality Approach to EQO repository QA Recommendation Priorities of intended application User providing priorities of the application context for which Quality Approach should be suitable CONTRIBUTION PHASERECOMMENDATION PHASE Experience Contribution Priorities of application Assessment of QA User contributing Experience from the Quality Approach to EQO repository

19. University of Piraeus Department of Technology Education and Digital Systems Centre for Research and Technology - Hellas(C.E.R.T.H.) Informatics and Telematics Institute (I.T.I.) N. Manouselis & D.G. SampsonEDMEDIA04, Lugano, Switzerland, 24th June 2004 19 Recommendation Life-Cycle QA Contribution Priorities of QA Assessment of QA Expert contributing Quality Approach to EQO repository QA Recommendation Priorities of intended application User providing priorities of the application context for which Quality Approach should be suitable CONTRIBUTION PHASERECOMMENDATION PHASE Experience Contribution Priorities of application Assessment of QA User contributing Experience from the Quality Approach to EQO repository

20. University of Piraeus Department of Technology Education and Digital Systems Centre for Research and Technology - Hellas(C.E.R.T.H.) Informatics and Telematics Institute (I.T.I.) N. Manouselis & D.G. SampsonEDMEDIA04, Lugano, Switzerland, 24th June 2004 20 Criteria for calculating suitability Expressing the focus of a quality approach upon several criteria In our case, adopted from [Frydenberg, 2002] the following: 1.Focus on Institutional & executive organization / commitment 2.Focus on Technological infrastructure 3.Focus on Student services 4.Focus on Instructional design and course development 5.Focus on Instruction and instructors 6.Focus on Program delivery 7.Focus on Financial management 8.Focus on Regulatory and legal compliance 9.Focus on Evaluation and assessment procedures

21. University of Piraeus Department of Technology Education and Digital Systems Centre for Research and Technology - Hellas(C.E.R.T.H.) Informatics and Telematics Institute (I.T.I.) N. Manouselis & D.G. SampsonEDMEDIA04, Lugano, Switzerland, 24th June 2004 21 Providing search priorities

22. University of Piraeus Department of Technology Education and Digital Systems Centre for Research and Technology - Hellas(C.E.R.T.H.) Informatics and Telematics Institute (I.T.I.) N. Manouselis & D.G. SampsonEDMEDIA04, Lugano, Switzerland, 24th June 2004 22 Recommendation results

23. University of Piraeus Department of Technology Education and Digital Systems Centre for Research and Technology - Hellas(C.E.R.T.H.) Informatics and Telematics Institute (I.T.I.) N. Manouselis & D.G. SampsonEDMEDIA04, Lugano, Switzerland, 24th June 2004 23 Example scenario (1/3) Two users A and B searching for suitable QAs Each user has different priorities 12 QAs are stored in the repository, they are all candidate for recommendation g User A User B 16525 255 38545 47085 5 35 69555 74560 83575 92570

24. University of Piraeus Department of Technology Education and Digital Systems Centre for Research and Technology - Hellas(C.E.R.T.H.) Informatics and Telematics Institute (I.T.I.) N. Manouselis & D.G. SampsonEDMEDIA04, Lugano, Switzerland, 24th June 2004 24 Example scenario (2/3) U QA f or User A U QA f or User B 1. QA120,1521. QA40,094 2. QA70,0882. QA70,082 3. QA20,0783. QA60,077 4. QA30,0764. QA20,076 5. QA60,0725. QA30,073 6. QA110,0656. QA90,070 7. QA90,0677. QA120,069 8. QA40,0618. QA110,068 9. QA50,0609. QA100,059 10. QA100,05710. QA50,056 11. QA80,04811. QA80,047 12. QA10,00012. QA10,000 The suitability of each QA in the EQO repository is calculated The QAs are ranked according to their suitability

25. University of Piraeus Department of Technology Education and Digital Systems Centre for Research and Technology - Hellas(C.E.R.T.H.) Informatics and Telematics Institute (I.T.I.) N. Manouselis & D.G. SampsonEDMEDIA04, Lugano, Switzerland, 24th June 2004 25 Example scenario (3/3)

26. University of Piraeus Department of Technology Education and Digital Systems Centre for Research and Technology - Hellas(C.E.R.T.H.) Informatics and Telematics Institute (I.T.I.) N. Manouselis & D.G. SampsonEDMEDIA04, Lugano, Switzerland, 24th June 2004 26 Conclusions – Future directions The recommendation module seems to be providing useful hints to the users of the EQO portal evaluation from EQO users is required A utility-based recommender technique seems appropriate for cases similar to the EQO case study where multiple criteria are required to describe items other techniques could be also examined The criteria for calculating suitability could be furthermore elaborated and validated from the users explore the effect of user role in the definition of criteria validation of criteria through user survey

27. University of Piraeus Department of Technology Education and Digital Systems Centre for Research and Technology - Hellas(C.E.R.T.H.) Informatics and Telematics Institute (I.T.I.) N. Manouselis & D.G. SampsonEDMEDIA04, Lugano, Switzerland, 24th June 2004 27 Contact Details Nikos Manouselis and Prof. Demetrios Sampson nikosm@iti.gr, sampson@iti.gr http://www.ask.iti.gr Informatics and Telematics Institute, Centre for Research and Technology Hellas, 42, Arkadias Street,Athens,GR-15234,Greece & Department of Technology Education and Digital Systems, University of Piraeus 150, Androutsou Street, Piraeus, GR-18534 Greece