1. AHELO – feasibility study Informasjonsmøte i KD 1. desember 2009 Spesialrådgiver Jan S. Levy

2. Project background Why undertake the study? Feasibility study goals? After decades of quantitative growth in HE and its growing internationalisation, consensus on the need to ensure quality provision for all… but information gap on learning outcomes. Existing rankings have not focus on LO’s Carry out a feasibility study to provide a proof of concept (Tokyo, 2008) To assess whether reliable cross-national comparisons of HE learning outcomes are scientifically possible and whether their implementation is feasible. A research approach to provide a proof of concept and proof of practicality

3. Project summary: strands of work Basis for discussion of expected learning outcomes delivered through ‘Tuning’ approach. + contextual data Discipline strand in Engineering Basis for discussion of expected learning outcomes delivered through ‘Tuning’ approach. + contextual data Discipline strand in Economics Several perspectives to explore the issue of value- added (conceptually, psychometrics), building on recent OECD work at school level. Research-based “Value- added” or “Learning gain” measurement strand International pilot test of the US Collegiate Learning Assessment (CLA), to assess the extent to which problem- solving or critical thinking can be validly measured across different cultural, linguistic and institutional contexts. + contextual data Generic skills strand

4. Participation in the three strands of surveying Australia, Japan, Sweden Discipline strand in Engineering Belgium (Fl.), Italy, Mexico, Netherlands Discipline strand in Economics Finland, Korea, Mexico, Norway Generic skills strand

5. Project summary (cont.d) Why? The outcomes will be used to assist countries to decide on the next steps. When? The testing window is from August 2010 to April 2011. Who? Data will be collected from a targeted population of students who are near – but before – the end of their first 3-4 year degree. How? Joint steering by governments, agencies and HEIs through IMHE OECD’s role is to establish broad frameworks that guide international expert committees in charge of instrument development. Several options: Use existing instruments, combine existing item pools to cover frameworks, or develop new test material?

6. Feasibility study outputs A final conference and its proceedings to discuss the outcomes of the feasibility study as well as the potential impact of AHELO Three reports on the scientific and practical feasibility of implementing an international assessment focusing on discipline- specific skills in economics and engineering and on generic skills Two reports on the feasibility of reaching international consensus on defining the intended learning outcomes in two contrasted disciplines and in a cross-national fashion One report on the measurement of value-added in higher education, net of students’ incoming abilities and differences in institutional selectivity Less tangible: Enhanced understanding of the project and buy-in from stakeholders

7. Country participation - update  Critical mass attained (10 participants)  Scope for 4-5 more countries  Formal requests from 4 US states, Russian Federation  Discussions with Turkey  Volunteer requests from HEIs (France, Germany, Spain)  Interest from NME non-EEA countries (Kuweit, Colombia)

8. Internasjonal organisering • Ledes av OECDs utdanningspolitiske komite og av IMHE GB • Oppnevnt særskilt gruppe fra deltagende land (GNE) ansvarlig for gjennomføring • Ekspertgrupper på de enkelte temaer • OECD stiller strenge kvalitetskrav til utvikling, oversettelse og tilpasning av undersøkelses- instrumenter • Konsultasjonsorgan med interessegrupper internasjonalt (EI, BIAC, TUAC, ESU, ENQA, EUA..) • Undersøkelsen gjennomføres av internasjonale konsortier i samarbeid med deltagende land

9. Scope of tendered work Project management, survey operations and analyses of results (Module E) Contextual dimension surveys:- student - faculty (Module D)- programme leadership - institutional leadership Methodological/ Instrument development Field implementation and analysis Development of the GenS intrument (Module A) Generic Skills Strand (GenS) Development of the EngS intrument (Module C) Engineering Strand (EngS) Development of the EcoS intrument (Module B) Economics Strand (EcoS) VamS Resarch-based: •No direct assessment involved •Done through commissioned experts (Module F) Value-Added Measurement Strand (VamS) Sole supplier Initial work: Tuning-AHELO Initial work: Contextual Dimension

10. Two proposals received Proposal 1 Consortium led by the National Foundation for Educational Research (NFER) in partnership with Cito B.V. – National Institute for Educational Measurement, and one organisation included as subcontractor: – The Gallup Organisation Europe (Gallup Europe) Proposal 2 Consortium led by the Australian Council for Educational Research (ACER) with the following associates: • Capstan Linguistic Quality Control Agency (Belgium) • Indiana University Center for Postsecondary Research (USA) • University of Twente Centre for Higher Education Policy Studies (Netherlands) • Educational Testing Service (USA) • International Association for the Evaluation of Educational Achievement (Netherlands) • National Institute for Educational Policy Research (Japan) • Statistics Canada (Canada) • University of Florence (Italy) • Westat Inc (USA)

11. The European Classification of Higher Education Institutions. Welcome to U-Map. The European Classification of Higher Education Institutions. U-Map is the third phase of a research project to develop a European Classification of Higher Education Institutions. The classification maps the European higher education landscape and will make the institutional diversity of that landscape more transparent. U-Map will not rank the institutions league-table-style, but will position institutions on a number of dimensions, each representing an aspect of the function and performance of higher education institutions. Click here to go to the finder. About the Project. The concept of diversity has moved rapidly up the political agenda over the past decade. The development of the European Higher Education Area (EHEA) and the European Research Area (ERA) has clearly contributed to this. The Methodology. U-Map has been developed in a dynamic and interactive process in which a number of general guidelines and design principles were used. The consultation of stakeholders was a cornerstone of that process. Watch the demo.

12. Arbeid i Norge • Tre norske institusjoner: NTNU, UMB, HiVe • Generic skills strand • Tilbud fra CAE om å gi seminarer om CLA- undersøkelsen også til andre institusjoner • Planlagt gjennomført i februar/mars 2011 • Rapporter foreløpig planlagt til høsten 2011 • Nasjonal projektleder (NPM) – NIFU STEP i samarbeid med ILS/EKVA • Prosjektorganisert i departementet – prosjektgruppe, styringsgruppe, informasjonsmøter

13. Hvilke typer resultater for Norge? • Bidrag til den internasjonale studien • Internasjonale sammenligninger vil ikke bli laget. Utvalgene vil ikke være representative internasjonalt • Data som kan identifisere respondent blir utilgjengelige • Institusjonene må få rapport med sine resultater • Departementet må få rapport om norske institusjoner som deltok • Offentliggjøring av nasjonale resultater ikke avgjort • Ikke klart om det vil bli gitt noen form for sammenligning med internasjonal benchmark

14. TASK DESCRIPTION Performance Task Each Performance Task requires students to use an integrated set of critical thinking, analytic reasoning, problem solving, and written communication skills to answer several openended questions about a hypothetical but realistic situation. In addition to directions and questions, each Performance Task also has its own document library that includes a range of information sources, such as letters, memos, summaries of research reports, newspaper articles, maps, photographs, diagrams, tables, charts, and interview notes or transcripts. Students are instructed to use these materials in preparing their answers to the Performance Task’s questions within the allotted 90 minutes. http://www.collegiatelearningassessment.org/ http://www.cae.org/content/pdf/CLABrochure2008.pdf

15. Eksempel på performance task