Rannís Erasmus+ Data collection, analysis & impact studies

Data analysis at Rannís  Division for Evaluation and Analysis – until June 2014  5 staff – background; statistics, psychology…  Focus on: I.Data analysis on research, development and innovation in Iceland II.Impact assessment of national research/technology funds III.Secretariat: The Quality Board for Icelandic Higher Education (established in 2010)

Recent Rannís studies „Impact assessment of National Technology Development Fund“

Rannís Performance Contract (1) (Árangursstjórnunarsamningur) Signed 23 April 2013

Rannís Performance Contract (2) Indicators for progress and targets :(Annex to Performance Contract)  Development of comprehensive processes for work flow and internal quality assurance (2014)  Comprehensive data collection for all national/international funds and grant programmes (applications and money spent) – common methodology for analysing data (2014)  Reach targets set for the following programmes: Erasmus+, Nordplus, Artic Studies (2014)  Improving publication of statistical data for national/international funds/grant programmes. Create a statistical database and implement standardized practices in collecting and analysing data. (2014)  Award over 95% of money available for national and international funds (2014)  Customer Service Satisfacton Survey – Rannis as a whole; individual divisions (2015)

Erasmus+ Data collection, analysis & measuring impact The Rannís approach towards Erasmus+

What has IS NA done until E+?  Yearly work programme and yearly reports  Evaluation of Leonardo II ( ) and Socrates – Outsourced  NA customer satisfaction survey (2010)  Results were very positive  Limited utilization of results and no follow-up by IS NA  Quality Awards „best practices “  Started 2003 in Leonardo da Vinci; from 2008 in Comenius; Grundtvig og Erasmus from 2013  Evaluation practices not standardized between fields – LvD external evaluation in beginning/projects in other fields evaluated internally  Geographical analysis of distribution of European funds in Iceland  Collection of all available data from 1995 (comprehensive from 2000) 

Erasmus+ - context for data collection and impact analysing  Requirements in Erasmus+ Legal Basis  First year of Erasmus+ indicates more requirements on setting indicators/targets and measuring progress towards targets and especially measuring IMPACT ( see article 15 in legal basis )  7 year „strategic“ work programmes with key indicators and targets (reviewed annually)  Work programme also includes 7 year strategy for promotion of programme and its results (Communication and Dissemination strategy)  European added value” – Indicators for measuring impact of programme directly linked to EU policy targets(annex I to Legal Basis)  More focus in Erasmus+ than previously on connecting national policy/priorities in education to European education policies/priorities  Europe 2020; Education & Training 2020; Rethinking Skills….

Our emphasis in the NA work programme  Monitor progress towards set objectives and targets  NA operational objectives, indicators and targets (required)  “5 additional” national NA operational objectives  International cooperation  Create baseline for target groups  No analysis/surveys available for certain target groups, e.g. school teachers; adult trainers; HE staff  More frequent evaluations of single target groups – measure trends (quantitative analysis) and impact over time (qualitative)  More effective and focused use of data – more use of available data; gather available data from different national sources  Regular customer satisfaction surveys (first in 2015)

IS NA data collection and analysis (1)  Purpose of data collection and analysis  EU requirements - yearly reports; impact analysis (mid-term/end of Erasmus+)  National requirements; yearly report to National Authorities; use as basis for setting policy objectives  Measuring progress towards operational objectives and targets in the Work Programme (EU objectives & additional NA objectives)  Finding trends  E.g. who is applying?, where do they go, how long do they stay…  Measuring impact – justification for more national funding; policy- making on internationalisation of education (e.g. higher education)  Evaluation and specific cooperation projects – collaboration with other NAs; contribution to studies done by independent research entities.

IS NA Data collection and analysis (2)  Sources of data  EplusLink  BO reports  Mobility Tool  Surveys  Interviews  Other sources – e.g. National Statistical Office

IS NA Data collection and analysis (3) Evidence-based analysis & studies in 2014 WP 1.LLP impact assessment study (expect a mandate from the Ministry of Education) 2.Baseline survey on internationalisation in pre-primary and compulsory schools and upper secondary schools  Aims; a) to analyse available data on involvement of schools to get an overview of the state of play; b) gather more qualitative data on barriers to internationalisation; c) to collect ideas on measures that the NA could put in place to increase the attractiveness of participating in the Erasmus+ programme  Expected results: Quantitative and qualitative state of play on the internationalisation of at these school levels and ideas on how to cater activities to improve the current situation 3.Study on Erasmus Charter for Higher Education and internationalisation policies of higher education institutions in the Nordic countries 4. Baseline study on mobility: a framework for collecting and analysing data on mobility activities 5. Customer Satisfaction Survey – periodically 2015, 2017 and 2019  Aim: a) an online survey under the auspices of the host organisation of the NA, Rannis among all the target audiences it has a mandate to serve; b) focus groups of different target audiences focusing on particular aspects of the NA services, such as distribution of information, counselling, guidance and training, monitoring and follow-up, and quality of products developed.

Example I Example I Erasmus+ Charter for Higher Education (ECHE) - a Nordic study (DK, FI, IS, NO, SE)

Why analyse ECHE? Why a Nordic co-operation project?  More knowledge needed on how higher education institutions (HEIs) organize their internationalisation work  HEIs are interested in a comparative analysis  Previous Nordic studies on mobility (2010 and 2012) indicate that student support is important  Strategies are likely to be important for the development of mobility support structures  Strategies are an essential part of the ECHE-application  Nordic angle provides  An interesting comparative view of different aspects of internationalisation  The European Commission benefits from comparative analysis on the added value of the ECHE

Example II Example II Baseline study on mobility: a framework for collecting and analysing data on mobility activities

Why baseline study?  Bologna (2009) and EU (2011) 20% target on international mobility for higher education graduates and 5% VET (EU target) => => demands comprehensive collection of mobility data as well as analysis of data to measure progress towards target  Limited data available on mobility at school level and in adult education  Limited data available on staff mobility – definition of „staff” missing in HE  Data currently collected/analysed by our own organisations mostly relates to activities funded by EU programmes E+ National Agencies manage => => analysis more of a „formal requirement“  Cross-referencing of own data with other data collected at national level, e.g. on degree mobility, limited => => a comprehensive view of the extent of international mobility activities and the impact of such activities beyond the individual is limited.

Aims of study framework for a comprehensive collection of data  To develop a framework for a comprehensive collection of data on credit and degree mobility (students and staff within all education fields where data is available)  Erasmus+ data will serve as the basis for the framework and the analysis which will follow. a framework for analysing the data  To develop a framework for analysing the data collected, both in order to provide a comprehensive view within a country and to offer a comparative analysis between the participating countries.  The current framework for data collection and analysis used by SIU in Norway will be used as basis for developing the frameworks.

Expected outcome  A framework for collecting data to ensure comparability between years and between countries, and  A framework/model for analysing data => => allows countries to collect data in 2013/ taking stock of mobility in 2013 to provide a baseline study that will be valuable for interim and final evaluation of the impact of Erasmus+

Study Framework  Start of work – September 2014  Development of frameworks and the analysis will take place from September 2014 – end 2015  A collaboration between the Estonian, Icelandic, Norwegian and Swedish NAs