1. Monitoring and Indicators in 2014-2020 WORKSHOP 30 APRIL 2013 Ines Hartwig Impact Assessment & Evaluation Unit DG EMPL 1

2. General observations on status quo in EU 27 Number of indicators/priority varies considerably Some OPs include impact indicators others focus on financial or output indicators Not all indicator sets link output with result Some OP indicators cover large parts of a priority, others only niche type of activities/target groups Monitoring systems often distinguish between Annex XXIII and indicators In general data on performance seem to be of lesser importance than financial data. 2

3. Moving towards result orientation in 2014-202 Common indicators for outputs and results Legal obligation to collect personal data on participants Collect and store individual participants' data (micro-data) AIR only admissible if all data is submitted Suspension possible if there is a serious deficiency in the quality and reliability of the monitoring system or of the indicator data MIS will be subject to EC audits 3

4. Common ESF output indicators PeopleEntities 1 unemployed, incl. LTU* 2 LTU* 3 inactive* 4 inactive, not in education/training* 5 employed, incl. self-employed* 14 number of projects fully or partially implemented by social partners or non-governmental organisations 6 below 25 years* 7 above 54 years* 15 number of projects targeting publ. administrations or publ. services 8 with ISCED 1 or ISCED 2* 9 with ISCED 3 or ISCED 4* 10 with ISCED 5 to 8* 16 number of micro, small and medium-sized enterprises supported 11 migrants, people with a foreign background, minorities (incl. marginalised communities such as Roma)** 12 disabled** 13 other disadvantaged** 4

5. Common ESF result indicators 17 Inactive newly engaged in job searching upon leaving 21 In employment 6 months after leaving 18 In education/training upon leaving 22 In self-employment 6 months after leaving 19 Gaining a qualification upon leaving 23 With an improved labour market situation 6 months after leaving 20 In employment upon leaving 5 Only for a sample of participants. Only reported in 2019 and 2023

6. 6 Indicator Participa nt 123456789 1010 11121314151617181920212223 1 110 0 0 1 0 0 0 10 10 0 0 0 0 1 10 0 0 0 2 0 0 0 0 1 1 0 1 0 0 1 0 0 0 0 0 00 1 0 0 0 1 … Micro data for every participatory record Example: common indicators Labour market age educa tion disadvan tage entities Immediate results Longer- term results Total (AIR) 11 0 0 1 2 1 1 0 1 1 1 0 0 0 0 01 2 0 0 01

7. How to set targets for common result indicators? 7

8. Example: Participants gaining a qualification upon leaving Result target is often expressed in % The % requires a reference value, i.e. the relevant output data. Example: Target 1: 50% gaining a qualification upon leaving 50% of whom? Of all participants in the IP? No, 50% of the low skilled participants (ISCED 1 and 2) gaining a qualification upon leaving Target 2: 40% of the unemployed participants gaining a qualification upon leaving 8

9. Programme-specific indicators Developed in addition to common indicators Set at the level of investment priority or specific objective 9

10. SROP indicators: Priority 2 1.Number of working age people (15-64 years) who participate in training financed in the framework of the priority 2.Number of people who participate in training serving the improvement of key competences supported in the framework of the priority 3.Enterprises affected by the trainings supported in the framework of the priority 4.Number of working age people (15-64 years) who successfully finish a training financed in the framework of the priority (pass final exam, acquire certificate) 5.Number of people who successfully finish a training which serves the improvement of key competences supported in the framework of the priority 10

11. 11 Comparing monitoring data, Priority 2 Annex XXIII indicator Annex XXIII data Indicator data OP indicator Total number of participants 275 434110 805Number of working age people (15-64 years) who participate in training financed in the framework of the priority 169 170Total number of reported people (adding up all indicators) Young people 15-24 165 733

12. SROP indicators: Priority 3 1.Number of teachers accomplished a competence-based training in education methodology supported in the frame of the 3rd priority (head) 2.Number of teachers accomplished a competence-based training in education methodology supported in the frame of the 3rd priority (head) 3.Number of participants in non- formal learning events organized in cultural institutions and organizations supported in the frame of the 3rd priority (head) 4.Number of accumulatively handicapped students provided for an integrated education (head) (30,8 in 2010) 5.Ratio of 8th class students accomplishing not more than level 3 on the reading and text interpretation 6.Number of primary school task assignment locations training at least 80% handicapped students as a result of the activities implemented in the frame of the 3rd priority (pc) 12

13. 13 Comparing monitoring data, Priority 3 Annex XXIII indicator Annex XXIII data Indicator data OP indicator Total number of participants 502 318360 973Total number of reported participants (adding up all indicators)

14. Principles for sound indicators Integrated set of indicators consisting of CI and programme-specific indicators With a limited set of targets. Not all indicators need a target. But the most important ones do. Indicators should cover the main scope of a priority (i.e. target group, type of activity etc.) Targets should also cover large parts of a priority Indicators should be simple, data to be collected in monitoring system Indicators should have a clear definition 14

15. Principles (2) Output and result indicators – no impact indicators Result indicators should be linked to participants/ supported entities, no macro-economic indicators Result indicators need to be logically linked to output indicators. Indicators should be expressed in absolute numbers (targets could be %) 15 NO: Employment ratio (%) of working age YES: unemployed participants in employment upon leaving/6 months/12 months after leaving

16. Steps in designing programme-specific indicators 1.'Deconstruct' the programme with view to the following categories: 1.Target groups (people & entities) 2.Type of planned activities 3.Themes 4.Projects 5.Type of expected results 2.Prioritise within each category 3.Develop indicators e.g. by combining characteristics/ features from the various categories 4.Review intervention logic to ensure that the most important aspects are covered by indicators 16

17. Programme-specific indicators - EXAMPLES - 17

18. 18 By combining different common indicators Advantage: Data have to be collected anyway. Data only needs to be crossed. Less administrative burden linked to data collection

19. Programme Output Indicator: Young low skilled inactive participants 19 With primary or lower secondary education Below 25 years Inactive, not in education or training

20. Result indicator: young low skilled inactive participants gaining a qualification 20 With primary or lower secondary education Below 25 years Inactive, not in education or training Participants gaining a qualification upon leaving

21. Output indicator: low skilled older workers 21 With primary or lower secondary education Above 54 years Employed, including self- employed

22. Result indicator: low skilled older workers gaining a qualification 22 With primary or lower secondary education Above 54 years Employed, including self- employed Participants gaining a qualification upon leaving

23. Longer term result indicator: low skilled older workers gaining a qualification and improving their labour market situation 23 With primary or lower secondary education Above 54 years Employed, including self- employed Participants with an improved labour market situation 6 months after leaving Participants gaining a qualification upon leaving

24. Output indicator: high skilled unemployed 24 With tertiary education Unemployed, including long-term unemployed

25. Result indicator: high skilled unemployed in employment 25 With tertiary education Unemployed, including long-term unemployed Participants in employment upon leaving

26. By combining common indicators with programme specific characteristics/ features Advantages: indicators can take into account specific aspects of the programme. 26

27. Output indicator: NEETS 27 Participants between 16 and 25 years Unemployed, including long-term unemployed Inactive, not in education or training

28. Result indicator: NEETs in education or training 28 Participants between 15 and 24 years Participants in education/ training upon leaving Inactive, not in education or training Unemployed, including long-term unemployed

29. Output indicator: start-ups of high skilled young people supported 29 Start-ups Below 25 years With tertiary education

30. Result indicator: start-ups of high skilled young people sustained 6 months after leaving 30 Start-ups Below 25 years In self- employment six months after leaving With tertiary education

31. Output indicator Disadvantaged pupils 31 Migrant and minorities Pupils below 16 years Disabled Other disadvantage d

32. Result indicator Disadvantaged pupils in education or training upon leaving 32 Migrant and minorities Pupils below 16 years Disabled Other disadvantage d In education or training upon leaving

33. Output indicator : Counselling youngsters (up to 18 years) with migrant background 33 Counselling Below 19 years Migrants and minorities

34. Result indicator : Counselling youngsters with migrant background who finished an internship upon leaving 34 Counselling Below 19 years Migrants and minorities Internship

35. Thank you for your attention Questions? Comments? Contact: Ines Hartwig ines.hartwig@ec.europa.eu 35