1. Model and method for evaluation of eQNet “travel well” quality learning resources Assoc. Prof. Dr. Eugenijus Kurilovas, ITC, Lithuania EdReNe final strategic seminar, Barcelona, 26 March 2010

2. LRE particularly seeks to identify LOs that “travel well” (i.e., reusable) across national borders and can be used in a cultural and linguistic context different from the one in which they were created eQNet is a three-year (September 2009-2012) Comenius Multilateral Network funded under the European Commission’s Lifelong Learning programme. The project is coordinated by European Schoolnet and involves 9 Ministries of Education - AT, BE, CZ, IT, LT, NO, PT, SK, SE or agencies nominated to act of their behalf The primary aim is to improve the quality of LOs in LRE. eQNet will do this by establishing a network consisting of policy makers, researchers, and practitioners (teachers) that will develop and apply “travel well” quality criteria to both existing LRE content as well as that to be selected in future from national repositories eQNet project

3. The vision driving the LRE is that a significant percentage of high quality LOs developed in different countries, in different languages and to meet the needs of different curricula can be re-used at European level eQNet will provide a forum for joint reflection and co-operation related to the exchange and re-use of educational content and allow network members to: 1.better share information and expertise particularly related to “travel well” quality criteria (pedagogical, technical and intellectual property rights (IPR) factors) 2.develop new frameworks (including a network of expert teachers) to improve the quality of LOs and metadata in both national repositories and the LRE, including the growing volume of user-generated content and metadata, as well as to improve the multilinguality of LRE content as a result of the translation of metadata, making use, where appropriate, of automatic metadata translation approaches and technologies 3.enable schools to participate in a community of practice related to the use LOs at European level eQNet project

4. Major results will include: the development of “travel well” quality criteria to more easily identify LOs with the potential for cross-border use (this WP is coordinated by the Lithuanian MoE represented by ITC, and EUN) the practical application by teachers of these criteria to >3,500 LOs in the LRE ‘showcases’ of the best of these LOs in a “travel well” section of the LRE portal where necessary, the enrichment of selected LOs with new or better metadata a community of practice for teachers around these LOs eQNet project

5. The aim of the presentation is to suggest scientific model and method for expert evaluation of quality of LOs paying especial attention to LOs reusability level The main problem of all existing approaches in the area is high level of expert evaluation subjectivity Several scientific methods and principles to minimise subjectivity level in expert evaluation of LOs quality were applied, namely: 1.principles of multiple criteria decision analysis for identification of quality criteria 2.technological quality criteria classification principle 3.fuzzy group decision making theory to obtain evaluation measures 4.normalisation requirement for criteria weights 5.scalarisation method for LOs quality optimisation The complex application of these approaches could significantly improve the quality of expert evaluation of LOs and noticeably reduce the expert evaluation subjectivity level An example of practical application of these approaches for LOs quality evaluation is also presented Applied scientific methods and principles

6. To present different approaches to evaluation of Learning Objects (LOs) – previous works To present the principles of identification of quality evaluation criteria To present LOs technological quality evaluation model (criteria tree) To present overall LOs quality evaluation model (incl. technological, pedagogical and IPR criteria) LOs quality evaluation model (criteria tree) – Outline

7. The main notions: LO is referred here as “any digital resource that can be reused to support learning” (Wiley, 2000) Quality evaluation is defined as “the systematic examination of the extent to which an entity (part, product, service or organisation) is capable of meeting specified requirements” (ISO/IEC 14598-1:1999, 1999) Reusability of LOs (or their ability to “travel well” between different contexts and education systems) is considered here as a part of the overall quality of LOs This means that any high quality LO has some reusability level (or potential to “travel well”), but this does not mean that any reusable LO is quality one LOs quality evaluation model (criteria tree)

8. The need for reusability of LOs has at least three elements: Interoperability: LO is interoperable and can be used in different platforms Flexibility in terms of pedagogic situations: LO can fit into a variety of pedagogic situations Modifiability to suit a particular teacher’s or student’s needs: LO can be made more appropriate to a pedagogic situation by modifying it to suit a particular teacher’s or student’s needs (McCormick et al., 2004) LOs quality evaluation model (criteria tree)

9. Learning Object Review Instrument (LORI): Presentation: Aesthetics Presentation: Design for learning Accuracy of content Support for learning goals Motivation Interaction: Usability Interaction: Feedback and adaptation Reusability Metadata and interoperability compliance Accessibility LOs quality evaluation model (criteria tree)

10. Leacock & Nesbit (2007) quality criteria: Content quality: veracity, accuracy, balanced presentation of ideas, and appropriate level of detail) Learning goal alignment: alignment among learning goals, activities, assessments, and learner characteristics Feedback and adaptation: adaptive content or feedback driven by differential learner input or learner modelling Motivation: ability to motivate and interest an identified population of learners Presentation design: design of visual and auditory information for enhanced learning and efficient mental processing Interaction usability: ease of navigation, predictability of the user interface, and the quality of the interface help features Accessibility: design of controls and presentation formats to accommodate disabled and mobile learners Reusability: ability to use in varying learning contexts and with learners from different backgrounds Standards compliance: Adherence to international standards and specifications LOs quality evaluation model (criteria tree)

11. Becta (2007) core pedagogic principles: Inclusion and access Learner engagement Effective learning Assessment to support learning Robust summative assessment Innovative approaches Ease of use Match to the curriculum LOs quality evaluation model (criteria tree)

12. Becta (2007) core design principles: Digital learning resource design Robustness and support Human-computer interaction Quality of assets Accessibility Interoperability Testing and verification Effective communication LOs quality evaluation model (criteria tree)

13. Paulsson and Naeve (2006) have suggested six areas for establishing LO technological quality criteria: A narrow definition A mapping taxonomy More extensive standards Best practice for use of existing standards Architecture models The separation of pedagogy from the supporting technology of LOs LOs quality evaluation model (criteria tree)

14. MELT set of the LOs quality criteria has been divided into five categories: Pedagogical Usability Reusability Accessibility Production LOs quality evaluation model (criteria tree)

15. MELT: the cross-border reuse of content will be more likely if LOs: Have a strong visual element and users can broadly understand what is the intended learning objective or topic. E.g., LOs may have little or no text; and include animations and simulations that are self-explanatory or have just a few text labels or icons / buttons for start, stop, etc. Have been designed to be language customisable and are already offered in more than one language Address curriculum topics that could be considered trans- national Are adaptable from a technical (e.g., LOs are supplied along with an authoring environment or tools) or IPR perspective LOs quality evaluation model (criteria tree)

16. LOs Q4R (Quality for Reuse) criteria: Strategies before LO inclusion in the LOR are based on the following principles: Only build or integrate LOs, which can be certified for quality Interactive LOs are software and as such should answer to software quality criteria Strategies during LO inclusion in the LOR are based on: The principle of reducing form-filling Use of guiding wizards, smart automatic and semi-automatic computer agents to assist in assuring technical interoperability Strategies after LO inclusion in the LOR are based on the following principles: Provide interesting and easily understood user statistics, such as stars, percentages, voting systems Include recommendations for reuse by the user, both to the next user and the designer LOs quality evaluation model (criteria tree)

17. Belgium (Flanders) nationally recognised LOs quality criteria: Scope Criteria: Aims and users (Aims; Target group) Contents (Curriculum area; Relevant source; Source type; Granularity; No advertisements) Geographical coverage (Geographical Limitations (availability – relevance); Language) Accessibility (Technology; Costs; Registration) Source data (Metadata) Quality criteria: Content criteria (Validity; Accuracy; Authority; Actuality; Substantial Coverage; Completeness) Formal criteria (Navigation; User support; Use of ICT and standards) Process criteria (Integrity of information; Site stability; Platform stability) LOs quality evaluation model (criteria tree)

18. Czech LOs quality criteria: Content accuracy Connection with national curriculum Didactics / Pedagogy accuracy Typographic rules Citations, Copyright LOs quality evaluation model (criteria tree)

19. French LOs quality criteria: Curriculum conforming Free rights for pedagogical use in the classroom and out of the classroom Respect of publishers engagements Provide access to expertise the resources Update information must be available Provide the possibility to block advertising (if included) LOs quality evaluation model (criteria tree)

20. Lithuanian LOs quality criteria: Methodical aspects User interface (incl. personalisation) LOs arrangement possibilities Communication & collaboration possibilities and tools Technical features (incl. working stability) Documentation Implementation and maintenance expenditure LOs quality evaluation model (criteria tree)

21. Norwegian LOs quality criteria: Technical interoperability (File formats, standards (IMS QTI & CP, etc.)) Metadata tagging Accessibility (WCAG 1.0) Parallel publishing (two forms of the Norwegian language exist) Pedagogical issues LOs quality evaluation model (criteria tree)

22. Portuguese LOs quality criteria: Technical Scientific Educational Language Values and Attitudes LOs quality evaluation model (criteria tree)

23. Swedish LOs quality criteria: Every web site must have source information and must not conflict with Swedish law The presentation must be clear and easy to navigate The information must be reliable: who is responsible for the information, why the information is published, how often it is updated, etc. Other criteria deal with general interest, language and access requirements LOs quality evaluation model (criteria tree)

24. Swiss LOs quality criteria: National level: Recommendation for description (Application profile) Technical tools Harvesting Local / Partner level: Selection and validation of resources according local criteria and culture Metadata check, enrichment and maintenance Publication (local) LOs quality evaluation model (criteria tree)

25. What is important: Alternatives (LOs to be chosen) Principles of identification of quality evaluation criteria Stakeholders LOs quality evaluation model (criteria tree)

26. The main problem here is how to establish the ‘proper’ set of LOs evaluation criteria which should reflect objective scientific principles of construction a model (criteria tree) for LOs quality evaluation These general principles have been analysed in several multiple criteria decision analysis (MCDA) related works such as (Belton, Stewart, 2002) If the set of decision alternatives is assumed to be predefined, fixed and finite, then the decision problem is to choose the optimal alternative or, maybe, to rank them. But usually the experts (decision makers) have to deal with the problem of optimal decision in the multiple criteria situation where the objectives are often conflicting. In this case, an optimal decision is the one that maximises the decision maker’s utility. An initial candidate set of key factors, which we term criteria should emerge from the problem structuring process In identifying these, the following considerations are relevant to all multiple criteria decision analysis approaches: LOs quality evaluation model (criteria tree)

27. Value relevance: Are the decision makers able to link the concept to their goals, thereby enabling them to specify preferences which relate directly to the concept? Understandability: It is important that decision makers have a shared understanding of concepts to be used in an analysis. Measurability: All MCDA implies some degree of measurement of the performance of alternatives against specified criteria, thus it must be possible to specify this in a consistent manner. It is usual to decompose criteria to a level of detail which allows this. Non-redundancy: Is there more than one criterion measuring the same factor? When eliciting ideas often the same concept may arise under different headings… One can easily check for criteria which appear to be measuring the same thing by calculating a correlation coefficient if appropriate data is available, or carrying out a process of matching as associated with analysis of repertory grids. LOs quality evaluation model (criteria tree)

28. Judgmental independence: Criteria are not judgementally independent if preferences with respect to a single criterion, or trade-offs between two criteria, depend on the level of another. Balancing completeness and conciseness: a number of authors note that desirable characteristics of a value tree are that it is complete, i.e., that all important aspects of the problem are captured, and also that it is concise, keeping the level of detail to the minimum required. Operationality: The model is usable with reasonable effort – that the information required does not place excessive demands on the decision makers. The context in which the model is being used is clearly important in judging the usability of a model. Simplicity versus complexity: The value tree, or criteria set is itself a simple representation, capturing the essence of a problem, which has been extracted from a complex problem description. The modeller should strive for the simplest tree which adequately captures the problem for the decision maker. LOs quality evaluation model (criteria tree)

29. LOs quality evaluation model (technical criteria tree)

30. Software engineering Principle: we can evaluate the software using the two types of evaluation criteria – ‘internal quality’ and ‘quality in use’ criteria: ‘Internal quality’ is a descriptive characteristic that describes the quality of software independently from any particular context of its use (decision makers – scientists, experts in informatics engineering) ‘Quality in use’ is evaluative characteristic of software obtained by making a judgment based on criteria that determine the worthiness of software for a particular project or user (decision makers – users) LOs quality evaluation model (criteria tree)

31. EUN has identified some “travel well” parameters: Language independence / Low language dependence (easily translatable) or Multilinguality Strong visual structure (animations, images and short videos are travelling best) No large semantic density (easy to include in any curriculum) Other parameters more relating to quality in general: Cultural appropriateness: the resource respects the cultural and social norms of teacher and / or pupil Usability: users can find their way through the resource almost intuitively. This may be possible even if the user is unfamiliar with the language of the resource Technical features: easy to access (no login or plug-ins) Openness: open license, free to use, open code (like in flash resources) (Panzavolta, 2009) LOs quality evaluation model (criteria tree)

33. To present the ratings (values) of LOs evaluation criteria (taking into account technological, pedagogical and intellectual property rights (IPR) criteria) To present the method for evaluation of LOs (experts additive utility function) To provide an example of practical evaluation of three LOs alternatives using this method LOs quality evaluation method – Outline

34. Expert evaluation is referred here as the multiple criteria evaluation of the learning software (such as LOs) aimed at selection of the best alternative based on score-ranking results. Learning software multiple criteria evaluation method is referred here as the experts additive utility function including the alternatives’ (i.e., different LOs) evaluation criteria, their ratings (values) and weights. This method is well-known in the theory of optimisation methods and is named “scalarization method”. LOs quality evaluation method

35. where X – LO alternative i = {1, …, m} – evaluation criteria (m=8) – rating (value) of criterion – ‘normalized’ weights of criteria,,

36. The widely used measurement criteria of the decision attributes’ quality are mainly qualitative and subjective. Decisions in this context are often expressed in natural language, and evaluators are unable to assign exact numerical values to the different criteria. Assessment can be often performed by linguistic variables: ‘bad’, ‘poor’, ‘fair’, ‘good’ and ‘excellent’. These values are imprecise and uncertain: they are commonly called fuzzy values. Integrating these different judgments to obtain a final evaluation is not evident. Therefore, (Ounaies et al., 2009) have proposed to use fuzzy group decision making theory to obtain final assessment measures. Linguistic variables conversion into non-fuzzy values: Excellent0.850 Good0.675 Fair0.500 Poor 0.325 Bad0.150 LOs quality evaluation method

37. ValueExplanation Bad (0.150) Failed or feature (criterion) does not exist Poor (0.325) Has poor support and / or it can be done but with significant effort Fair (0.500) Fair support but needs modification to reach the desired level of support Good (0.675) Good support and needs a minimal amount of effort Excellent (0.850) Excellent support and meets the criteria out of the box, minimal effort

38. LOs quality evaluation method Examples of Lithuanian LOs (Physics): LO1: Light diffraction (Šviesos difrakcija) http://mkp.emokykla.lt/imo/lt/mo/330/ LO2: Photo effect (Fotoefektas) http://mkp.emokykla.lt/imo/lt/mo/367/ LO3: Isobar process (Izobarinis procesas) http://mkp.emokykla.lt/imo/lt/mo/395/

39. LOs quality evaluation method (equal weights 0.125) Evaluation criteriaLO1LO1LO2LO3 Technological quality criteria Technological reusability0.850 Design and usability0.850 Working stability0.850 Architecture0.675 Pedagogical quality criteria Multimedia interactivity level0.500 Language independence0.8500.6750.500 Ease of use, intuitiveness0.850 IPR criterion Open licence, cost0.850 Evaluation results (values * 0.125)0.78440.76250.7406

40. LOs quality evaluation method (weights 0.2; 0.1) Evaluation criteriaLO1LO1LO2LO3 Technological quality criteria Technological reusability (weight 0.2)0.1700 Design and usability (weight 0.1)0.0850 Working stability (weight 0.1)0.0850 Architecture (weight 0.1)0.0675 Pedagogical quality criteria Multimedia interactivity level (weight 0.1)0.0500 Language independence (weight 0.2)0.17000.13500.1000 Ease of use, intuitiveness (weight 0.1)0.0850 IPR criterion Open licence, cost (weight 0.1)0.0850 Evaluation results0.79750.76250.7275

41. Table 1 results mean that LO1 meets 78.44% quality in comparison with the ideal, LO2 – 76.25%, and LO3 – 74.06%. According to these experimental evaluation results, LO1 is the best alternative (among the evaluated) from overall quality point of view. Table 2 results mean that LO1 meets 79.75% “travel well” quality in comparison with the ideal, LO2 – 76.25%, and LO3 – 72.75%. According to these experimental evaluation results, LO1 is the best alternative (among the evaluated) from “travel well” quality point of view. LOs quality evaluation method

42. ISO/IEC 14598-1:1999 (1999). Information Technology – Software Product Evaluation – Part 1: General Overview. First edition, 1999-04-15 Becta (2007). Quality principles for digital learning resources. Available online at http://schools.becta.org.uk/index.php?section=re&catcode=ss_res_dig_02& rid=13293 http://schools.becta.org.uk/index.php?section=re&catcode=ss_res_dig_02& rid=13293 Kurilovas, E.; Dagienė, V. (2009). Multiple Criteria Comparative Evaluation of e-Learning Systems and Components. Informatica. 2009, Vol. 20, No. 4, pp. 499-518 [ISI Web of Science] Kurilovas, E.; Dagienė, V. (2009). Quality Evaluation and Optimisation of e- Learning System Components. In: Proceedings of the 8 th European Conference on e-Learning (ECEL’09). Bari, Italy, October 29–30, 2009, pp. 315–324 [ISI Web of Knowledge] [ERIC] – the best paper award Kurilovas, E., Dagiene, V. (2008). Learning Objects and Virtual Learning Environments Technical Evaluation Tools. In: Proceedings of the 7 th European Conference on e-Learning (ECEL’08). Agia Napa, Cyprus, 6–7 November 2008, Vol. 2, pp. 24–33 [ISI Web of Knowledge] [ERIC] – the best paper award References

43. Leacock, T. L. & Nesbit, J. C. (2007). A Framework for Evaluating the Quality of Multimedia Learning Resources. Educational Technology & Society, 10 (2), pp. 44–59 McCormick, R., Scrimshaw, P., Li, N., and Clifford, C. (2004). CELEBRATE Evaluation report. http://celebrate.eun.org/eun.org2/eun/Include_to_content/celebrate/file/Delive rable7_2EvaluationReport02Dec04.pdf http://celebrate.eun.org/eun.org2/eun/Include_to_content/celebrate/file/Delive rable7_2EvaluationReport02Dec04.pdf MELT (2008). EU eContentplus programme’s MELT (Metadata Ecology for Learning and Teaching) project web site. Available: http://melt-project.eun.orghttp://melt-project.eun.org Panzavolta, S. (2009). Travel-well resources. Presentation in the EMINENT conference. Vilnius, 26-27 December, 2009 Paulsson, F., Naeve, A. (2006). Establishing technical quality criteria for Learning Objects. Available: http://www.frepa.org/wp/wp- content/files/Paulsson-Establ-Tech-Qual_finalv1.pdfhttp://www.frepa.org/wp/wp- content/files/Paulsson-Establ-Tech-Qual_finalv1.pdf References

44. Ounaies, H.Z.; Jamoussi, Y.; Ben Ghezala, H.H. (2009). Evaluation framework based on fuzzy measured method in adaptive learning system. Themes in Science and Technology Education. Vol. 1, Nr. 1, 2009, pp. 49–58 Q4R (2008). Quality for Reuse (Q4R) project web site. Available: http://www.q4r.org http://www.q4r.org Van Assche, F., and Vuorikari, R. (2006). A Framework for Quality of Learning Resources. In: European Handbook for Quality, CEDEFOP, 2006 Vargo, J., Nesbit, J. C., Belfer, K., Archambault, A. (2003). Learning object evaluation: Computer mediated collaboration and inter–rater reliability. International Journal of Computers and Applications, Vol. 25 (3), pp 198–205 Wiley. D. A. (2000). Connecting Learning Objects to Instructional design Theory: a definition, a Metaphor, and a Taxonomy. Utah State University. http://www.reusability.org/read/ http://www.reusability.org/read/ References

45. Thank you for your attention !!! Questions ? Dr. Eugenijus Kurilovas Eugenijus.Kurilovas@itc.smm.lt