1. Rolling Review of Education Statistics
Working Group Meeting “Education and training statistics”
Luxembourg, 23 May 2011
Antonio Consoli, Eurostat Unit B1
23-May-2011

2. Background information
Definition of rolling reviews
Objectives of a rolling review
Identify strengths and weaknesses of the process
Help to improve the quality of the process
Report on achievements of the process
Value added from Rolling Reviews
Assessment of the process from a different angle
Independent expert assessment
External assessment of data quality
Tools for rolling reviews
23-May-2011

3. Rolling Review of Education Statistics
Fit within the Eurostat quality assurance framework
Focused mainly on CVTS and UOE
Included
Self-assessments from quality review of CVTS and UOE
Global user satisfaction survey via the web-site (74 answers)
Partner satisfaction surveys to all MS, Candidates and EFTA (27 answers for CVTS, 23 for UOE)
Final result: improvement actions and recommendations
23-May-2011

4. User satisfaction survey (1)
Respondents
NSA (28), government (22), university or research (17), private (3), business (2), other (2)
Users were asked about:
Users themselves
Purposes they use data for
Relevance of data for their work
Other data sources they use except Eurostat
Quality of data
Quality of the service provided by Eurostat
23-May-2011

5. User satisfaction survey (2)
Users’ opinion is generally positive about data quality
Strengths:
Education statistics are relevant for users, for all data collections UOE, AES and CVTS.
Overall quality is rated high, especially for quality dimensions important for users (accuracy, clarity and accessibility of data).
Users are very satisfied with Eurostat’s support services.
Weaknesses:
User needs more data across European countries and at a more detailed level. All UOE data published by OECD should be made available by Eurostat.
Many users consider that data on IVT have limited comparability over countries and are the least important for them. In addition, the timeliness of CVTS results needs to be improved.
Users are less satisfied with the coherence and the spatial and temporal comparability of data, due to the differences in the national methodologies and the frequent changes in the survey concepts.
23-May-2011

6. Partner satisfaction surveys
Partners were asked about:
Planning and coordination structures in the ESS
Functioning of the ESS
Direction of change
Eurostat’s role in the field of data production
Problems with the data production and suggestions
Assessment of data quality
Resources used to produce the data and for CVTS response burden on enterprises
23-May-2011

7. Partner satisfaction survey (CVTS)
Strengths:
Very positive perception of Eurostat’s role in the preparation of the CVTS 4 and satisfaction with TF meetings. It is anticipated that CVTS 4 will be much better than CVTS 3.
The relevance of all or part of the information collected is high. Even if the Commission Regulation did not require them to carry out this survey, more than half of the reporting countries would continue it.
Data accuracy of the two main CVTS indicators (Percentage of training and non-training enterprises and CVTS strategies and difficulties) and coherence and completeness of CVTS 3 statistics disseminated in Eurostat’s web site are satisfactorily appreciated.
Weaknesses:
High burden imposed to the enterprises and the length of the questionnaire. Some CVTS concepts are difficult to understand by respondents, in particular the IVT concept should be removed from future collections.
High cost, both in terms of human and financial resources. It’s the reason why some countries would stop it without the regulation.
Need to make CVTS micro-data available for analytical purposes and lack of sufficient documentation that accompanies the published CVTS 3 figures.
Timeliness of CVTS 3, as well as the accuracy of some indicators are rated below adequate. Part of the problem can be in the shortage of human resources inside Eurostat.
23-May-2011

8. Partner satisfaction survey (UOE)
Strengths:
Partners are very satisfied with planning and coordination structures in the ESS, the functioning of the ESS in this field and the role of Eurostat in the UOE data production.
Partners do not perceive that the problems they face with the UOE data production are serious or major problems. In particular the joint administration of the UOE data production by UIS/OECD/Eurostat is not perceived as a source for problems.
Partners have been following closely the development in various related topics of the UOE and they are mostly satisfied with recent developments on topics that are most important to them (i.e. ISCED revision and the Commission Regulation).
Partners anticipate improvements, especially in the most ‘problematic’ quality dimensions like timeliness and comparability across countries, with the implementation of the new Commission Regulation.
Weaknesses:
Need for communication of best practises.
Difficulty to find all the requested information and at the detail it is requested. As a consequence, these data need to be estimated to a large extent.
The topic ‘Modernization of the UOE expenditure data’, of high importance to more than half of the partners, should be given more attention by the three organizations.
Timeliness and comparability across countries are the quality dimensions with which partners are least satisfied (heavy burden to validate and prepare data for so many countries and the lack of resources inside Eurostat, methodology not harmonized enough among countries).
23-May-2011

9. Checklist
The Eurostat Statistical Processes Assessment Checklist is a self-assessment tool, addressed to the production unit and covering the quality of the production process and of the statistical output.
It includes information on:
Users
Data Providers
Validation (by the national data providers)
Validation (at Eurostat level)
Statistical confidentiality
Documentation (at Eurostat level)
Data dissemination
Follow-up of the statistical production process
IT conditions
Management, Planning and Legislation
Staff, work situation and competence
Conclusions can be cross checked with opinion of partners and users
23-May-2011

10. Checklist (CVTS) Strengths: Weaknesses:
Users’ interest, relevance of the statistics.
Based on regulations.
Completeness of data and of information on methodology and quality.
Efficiency of data validation at Eurostat.
Current IT application used at Eurostat is reliable enough, integrated and shared.
CVTS production team is well organized.
Weaknesses:
Insufficient available resources at Eurostat for further development and improvement of the production process.
Eurostat does not have the necessary resources to tackle the unmet needs of the users, especially concerning timeliness that is affected by delays at various stages.
Lack of availability of anonymized micro-data.
Heavy burden imposed to the data providers which also results in high unit non-response (not corrected).
More resources would be needed for validation of both primary and secondary variables.
Need to further enhance the completeness of the publications of quality reports.
The IT system used for the CVTS 3 is considered reliable but lacking flexibility and stability.
23-May-2011

11. Checklist (UOE) Strengths: Weaknesses:
Establishment of a regulation coinciding with revision of ISCED
Good contact with users
Limited burden on respondents
Low impact of validation and revisions
Good process documentation
Some balance between the available human resources and the work required
Weaknesses:
Need for more explanations on the data disseminated and more information on quality available to users
Problems with timeliness and missing values in data delivery (addressed in the regulation)
Validation at Eurostat needs to be made automatic
Timeliness, Availability of metadata and Completeness need to be improved
Current IT application at Eurostat not satisfying
23-May-2011

12. Recommendations (CVTS, 1)
Data production:
Reduce the length of the questionnaire with the ultimate aim to reduce response burden (L)
Further work on the harmonization of methods and definitions (also with other surveys) (L)
Improve the flexibility and stability of the IT applications used for data validation (S)
Improve and update regularly the internal documentation about CVTS production (L)
Consider a revision of the CVTS regulation (L)
Improving data and metadata dissemination:
Disseminate the anonymised micro-data based on agreed criteria (S)
Improve timeliness of the data release (M)
Improve the content of ESMS metadata (M)
23-May-2011

13. Recommendations (CVTS, 2)
Improving data quality:
Improve timeliness and coherence (M-L)
Improve comparability between countries (L)
Other matters:
Assess resource needs for CVTS data production inside Eurostat (S)
Assess the benefit from the use of alternative sources for CVT on the frequency of data release and the burden on statistical authorities (L)
23-May-2011

14. Recommendations (UOE, 1)
Data production:
Increase the sharing of knowledge between partners (national and best practices) (S)
Continue with the revision of the production process in line with the ISCED revision (L)
Reduce changes in the design of the data collections from year to year (L)
Assist data providers with the collection of the requested information (M)
Provide more information about UOE estimation practices (including examples) (M)
Boost the interest of both partners and users on the Bologna process indicators (M)
Allocate more effort in the developments on the ‘Modernization of the UOE expenditure data’ topic (but no application on the 2010 ESSnet call for proposal on lifelong learning accounts) (L)
Consider reviewing the UOE tables used for data transmission (L)
Improving data and metadata dissemination:
Improve timeliness of the data release (M)
Improve the content of ESMS metadata (M)
Improve the functionality of Eurostat’s website (look at OECD) (S)
23-May-2011

15. Recommendations (UOE, 2)
Improving data quality:
Improve timeliness and clarity (M-M)
Improve coherence with National Accounts data (L)
Other matters:
Communicate to the Member States the scope of the data collection (M)
Communicate to the Member States the deadlines of the data collection (M)
Assess resource needs for UOE data production inside Eurostat (M)
Consider migration to another IT tool than Oracle/OLAP (M)
23-May-2011