1. The OECD PISA Study in Romania.
THE MINISTRY OF EDUCATION, RESEARCH AND INNOVATION
THE NATIONAL CENTRE FOR CURRICULUM AND ASSESSMENT IN PRE-UNIVERSITY EDUCATION
THE PISA 2009 NATIONAL CENTER
The OECD PISA Study in Romania.
Elements of Impact on Educational Policy
Input for
The Regional Workshop on
EVIDENCE BASED POLICY MAKING
Vienna, 9-10 November 2009
Roxana MIHAIL, PhD.
Head of the Evaluation and Research Department ,
The National Center for Curriculum and Assessment in Pre-university Education
and PISA 2009 ROU National Coordinator

2. ROMANIA
THE MINISTRY OF EDUCATION, RESEARCH AND INNOVATION
THE NATIONAL CENTRE FOR CURRICULUM AND ASSESSMENT IN PRE-UNIVERSITY EDUCATION
THE PISA 2009 NATIONAL CENTER
ISSUES:
1. What is Romania’s “history” related to OECD-PISA Programme?
2. What are (some of) the PISA facts?
3. What lessons have we learned by participating in OECD-PISA?
4. What elements of impact can be highlighted?

3. 1. What is Romania’s “history” related to OECD-PISA Programme?
THE MINISTRY OF EDUCATION, RESEARCH AND INNOVATION
THE NATIONAL CENTRE FOR CURRICULUM AND ASSESSMENT IN PRE-UNIVERSITY EDUCATION
THE PISA 2009 NATIONAL CENTER
1. What is Romania’s “history” related to OECD-PISA Programme?
The acronym: Programme for International Student Assessment
An international comparative study, created by the governments of OECD countries, assessing how well 15-year-old students use their knowledge and skills to meet real-life challenges
A long term programme of assessment, measuring trends over time in students’ performances:
- first cycle – PISA 2000 Reading, Mathematics, Science
PISA 2003 Mathematics, Reading, Science
PISA 2006 Science, Mathematics, Reading
- second cycle - PISA 2009 Reading, Mathematics, Science
PISA 2012 Mathematics, Reading, Science
PISA 2015 Science, Mathematics, Reading

4. ROMANIA
THE MINISTRY OF EDUCATION, RESEARCH AND INNOVATION
THE NATIONAL CENTRE FOR CURRICULUM AND ASSESSMENT IN PRE-UNIVERSITY EDUCATION
THE PISA 2009 NATIONAL CENTER
1. What is Romania’s “history” related to OECD-PISA Programme? (continued 2)
PISA is operating with an innovative concept of literacy: students’ capacity to apply knowledge and skills in key subject areas and to analyse, reason and communicate effectively as they pose, solve and interpret problems in a variety of situations:
- Reading literacy
- Mathematical literacy
- Scientific literacy
It does NOT measure students’ curricular and cross-curricular competencies!
PISA is relevant for lifelong learning: it asks students to report on their own motivation to learn, their beliefs about themselves and their own learning strategies

5. ROMANIA
THE MINISTRY OF EDUCATION, RESEARCH AND INNOVATION
THE NATIONAL CENTRE FOR CURRICULUM AND ASSESSMENT IN PRE-UNIVERSITY EDUCATION
THE PISA 2009 NATIONAL CENTER
1. What is Romania’s “history” related to OECD-PISA Programme? (continued 3)
Is the most comprehensive and rigorous international programme to assess students’ performance and to collect data on the student, family and institutional factors that can help explaining differences in performance
OECD-PISA is applying strict technical standards and quality assurance mechanisms in:
Translation and adaptation of testing materials
School and student sampling
Data collection and validation
Result: high degree of validity and reliability of results
- credibility
- transparency in presenting the outcomes of education in the world’s economically most developed countries, as well as countries at earlier stages of economic development

6. ROMANIA
THE MINISTRY OF EDUCATION, RESEARCH AND INNOVATION
THE NATIONAL CENTRE FOR CURRICULUM AND ASSESSMENT IN PRE-UNIVERSITY EDUCATION
THE PISA 2009 NATIONAL CENTER
1. What is Romania’s “history” related to OECD-PISA Programme? (continued 4)
A global programme:
PISA 2000 – 43 participating countries
PISA 2003 – 41 participating countries
PISA 2006 – 57 participating countries
PISA 2009 – 63 participating countries
WHY participating? E.g.:
- To understand relative strengths and weaknesses of educational systems
To establish benchmarks for educational improvement
To learn from shared experiences about the capacity of educational systems to provide high levels of equity in educational outcomes and opportunities

7.  Did not participate in PISA 2003  Participated in PISA 2006
ROMANIA
THE MINISTRY OF EDUCATION, RESEARCH AND INNOVATION
THE NATIONAL CENTRE FOR CURRICULUM AND ASSESSMENT IN PRE-UNIVERSITY EDUCATION
THE PISA 2009 NATIONAL CENTER
What is Romania’s “history” related to OECD-PISA Programme? (continued 5)
 Participated in PISA 2000 as a PISA+ country (administering instruments in )
Average: 34 out of 43 participating countries
 Did not participate in PISA 2003
Average: ?????
 Participated in PISA 2006
Average: 47 out of 57 participating countries
 Is participating in PISA 2009
- Participation in PISA 2012?????

8. ROMANIA
THE MINISTRY OF EDUCATION, RESEARCH AND INNOVATION
THE NATIONAL CENTRE FOR CURRICULUM AND ASSESSMENT IN PRE-UNIVERSITY EDUCATION
THE PISA 2009 NATIONAL CENTER
3. What are (some of) the PISA facts?
 ROU has less than 0,5% students performing at level 5 at the Reading scale and more than 50% of students at or below level 1!
Challenge: encourage high performance AND minimising poor performance
International report: “Poor performance is relevant to reading literacy”
Reading levels have a significant impact on:
- the welfare of individuals
- the state of society
- the economic standing of countries on international arena
The inequality: for ROU the performance distribution shows the large gap between the 5th and the 95th percentile

9. www.cnceip.ro 3. What are (some of) the PISA facts?(continued 2)
ROMANIA
THE MINISTRY OF EDUCATION, RESEARCH AND INNOVATION
THE NATIONAL CENTRE FOR CURRICULUM AND ASSESSMENT IN PRE-UNIVERSITY EDUCATION
THE PISA 2009 NATIONAL CENTER
3. What are (some of) the PISA facts?(continued 2)
 Between PISA 2000 and PISA 2006 Romanian students performances in Reading literacy decreased significantly:
 In PISA 2000 the average performance was close to 450 score points
 In PISA 2006 the average performance was of 396 score points
The OECD average is 496 for PISA 2006 Reading literacy!
 The same decreasing trend is visible for Romanian Students in IEA-PIRLS and IEA-TIMSS international comparative studies!
 For ROU, on the Reading scale, 49,3% of the total variance in students’ performance is due to the between-schools variance! (FIN=6,3% of the variance)

10. ROMANIA
THE MINISTRY OF EDUCATION, RESEARCH AND INNOVATION
THE NATIONAL CENTRE FOR CURRICULUM AND ASSESSMENT IN PRE-UNIVERSITY EDUCATION
THE PISA 2009 NATIONAL CENTER
3. What are (some of) the PISA facts?(continued 3)
 In Mathematics PISA 2006 ROU students had an average of 415 score points on a 6 level scale, with results comparable with countries like Bulgaria, Chile, Mexico a.s.o.
 The OECD average is 500 score points!
Mathematical literacy = “ … an individual capacity to identify and understand the role that mathematics plays in the world, to make well-founded judgements and to use and engage with mathematics in ways that meet the needs of that individual’s life as constructive and reflective citizen”
 For ROU, on the Mathematics scale, 51,3% of the total variance in students’ performance is due to the between-schools variance! (FIN=7,4% of the variance)
 In ROU the National Curriculum is still not focused on students’ needs for the knowledge society, but on acquiring the Mathematical scientific content, more or less adapted to age level!

11. ROMANIA
THE MINISTRY OF EDUCATION, RESEARCH AND INNOVATION
THE NATIONAL CENTRE FOR CURRICULUM AND ASSESSMENT IN PRE-UNIVERSITY EDUCATION
THE PISA 2009 NATIONAL CENTER
3. What are (some of) the PISA facts?(continued 4)
 In Science PISA ROU students had an average of 418 score points on a 6 level scale, with results comparable with countries like Montenegro, Mexico a.s.o.
Scientific literacy = “… an individual’s scientific knowledge and use of that knowledge to identify questions, to acquire new knowledge, to explain scientific phenomena, to draw evidence-based conclusions about science-related issues and….
Understanding of the characteristic features of science as a form of human knowledge and inquiry;
Awareness of how science and technology shape our material, intellectual and cultural environments
Willingness to engage with science with science-related issues and with ideas of science, as a reflective citizen”
 For ROU, on the Science scale, 49,3% of the total variance in students’ performance is due to the between-schools variance! (FIN=6,3% of the variance)

12. ROMANIA
THE MINISTRY OF EDUCATION, RESEARCH AND INNOVATION
THE NATIONAL CENTRE FOR CURRICULUM AND ASSESSMENT IN PRE-UNIVERSITY EDUCATION
THE PISA 2009 NATIONAL CENTER
3. What are (some of) the PISA facts?(continued 5)
 The impact of demographic and socio-economic factors on variance in student performance in Science is smaller than expected and smaller than the impact of school factors:
13% is accounted for by demographic + SES factors – cf. 15,8% OECD av.
21,5% is accounted for by school factors – cf. 14,1% OECD av.
26% is accounted for by between school variance – cf. 18,8% OECD av.
Causes & factors should be searched for elsewhere:
 Curriculum level ?
 Teacher training level?
 School culture level?

13. www.cnceip.ro 3. What are (some of) the PISA facts? (continued 6)
ROMANIA
THE MINISTRY OF EDUCATION, RESEARCH AND INNOVATION
THE NATIONAL CENTRE FOR CURRICULUM AND ASSESSMENT IN PRE-UNIVERSITY EDUCATION
THE PISA 2009 NATIONAL CENTER
3. What are (some of) the PISA facts? (continued 6)
The effect of the highest level of parents’ education on student performance in Reading since PISA 2000:
in PISA % of the variance is explained, cf. 8% of the OECD average
in PISA % of the variance is explained, cf. 6% OECD average
Causes & factors should be searched for elsewhere:
 Attitude towards reading?
 Attitude towards testing?
 Family environment related to reading?

14. www.cnceip.ro 3. What are (some of) the PISA facts?(continued 7)
ROMANIA
THE MINISTRY OF EDUCATION, RESEARCH AND INNOVATION
THE NATIONAL CENTRE FOR CURRICULUM AND ASSESSMENT IN PRE-UNIVERSITY EDUCATION
THE PISA 2009 NATIONAL CENTER
3. What are (some of) the PISA facts?(continued 7)
The effect of the highest occupational status of parents on student performance in Science since PISA 2000:
in PISA % of the variance is explained, cf. 10% of the OECD average
in PISA % of the variance is explained, cf. 11% OECD average
Significant variation over a six year period!
Causes & factors ?
Structural changes in Romanian society? European mobility?
 The PISA databases are very complex!
 To explore them efficiently we need training and resources!
 Secondary analyses can answer questions about “causes and factors”

15.  There are though, examples of good, efficient practice!
ROMANIA
THE MINISTRY OF EDUCATION, RESEARCH AND INNOVATION
THE NATIONAL CENTRE FOR CURRICULUM AND ASSESSMENT IN PRE-UNIVERSITY EDUCATION
THE PISA 2009 NATIONAL CENTER
3. What lessons have we learned by participating in OECD-PISA?
There are no universal “recipes” for improvement at educational system level!
 Each education system must find its own way of dealing with specific problems!
 There are though, examples of good, efficient practice!

16. ROMANIA
THE MINISTRY OF EDUCATION, RESEARCH AND INNOVATION
THE NATIONAL CENTRE FOR CURRICULUM AND ASSESSMENT IN PRE-UNIVERSITY EDUCATION
THE PISA 2009 NATIONAL CENTER
3. What lessons have we learned by participating in OECD-PISA? (continued 2)
E.G.: In Poland the education reform of schooling system implemented in 1999…
 Now provides a more integrated educational structures for compulsory education
 Over two PISA cycles, Poland raised its average performance through increasing at the lower end of the performance distribution!

17. ROMANIA
THE MINISTRY OF EDUCATION, RESEARCH AND INNOVATION
THE NATIONAL CENTRE FOR CURRICULUM AND ASSESSMENT IN PRE-UNIVERSITY EDUCATION
THE PISA 2009 NATIONAL CENTER
3. What lessons have we learned by participating in OECD-PISA? (continued 3)
PISA is measuring, sample-based, the system level trends, PISA is not reporting at individual level
YET:
our test administrators were reporting (testimonies):
“…. I do not know what is the occupation of my mother because she left Romania four years ago, to work in Spain and I do not know how she is doing…”
We do not know and do not have the means for investigating it further on, the real , complex “testing effects”, on ROU students:
 Psychological, motivational effects
 T he novelty of this testing situation
 T he effects of the test format on ROU students

18. ROMANIA
THE MINISTRY OF EDUCATION, RESEARCH AND INNOVATION
THE NATIONAL CENTRE FOR CURRICULUM AND ASSESSMENT IN PRE-UNIVERSITY EDUCATION
THE PISA 2009 NATIONAL CENTER
3. What lessons have we learned by participating in OECD-PISA?(continued 4)
“Countries with higher national income tend to perform better in Science, 28% of the variation between mean scores can be predicted on the basis of their GDP per capita.”
…. But the GDP is not an educational measure!
The key remains:
 how to make use of the existing resources, no matter how scarce they are?

19. ROMANIA
THE MINISTRY OF EDUCATION, RESEARCH AND INNOVATION
THE NATIONAL CENTRE FOR CURRICULUM AND ASSESSMENT IN PRE-UNIVERSITY EDUCATION
THE PISA 2009 NATIONAL CENTER
3. What lessons have we learned by participating in OECD-PISA? (continued 5)
“In a global context, the future economic and social prospects of both individuals and countries continue to be dependent on the results they actually achieve, not on the performance that might result if they were to operate under average social and economic conditions”
In ROU The PISA results eventually created the pressure needed for changes!!!

20. www.cnceip.ro 4. What elements of impact can be highlighted?
ROMANIA
THE MINISTRY OF EDUCATION, RESEARCH AND INNOVATION
THE NATIONAL CENTRE FOR CURRICULUM AND ASSESSMENT IN PRE-UNIVERSITY EDUCATION
THE PISA 2009 NATIONAL CENTER
4. What elements of impact can be highlighted?
In 2009 the politics converged with the educational policies!
1. The political discourse is informed by some PISA results
2. A top-down decision was taken:
Starting with the school year the National Evaluation at the end of grade VIII will administer tests created following “the model of international standardised tests”

21. ROMANIA
THE MINISTRY OF EDUCATION, RESEARCH AND INNOVATION
THE NATIONAL CENTRE FOR CURRICULUM AND ASSESSMENT IN PRE-UNIVERSITY EDUCATION
THE PISA 2009 NATIONAL CENTER
4. What elements of impact can be highlighted? (continued 2)
3. In 2008, for the first time, the PISA 2006 National report published has been referred to by both politicians, and policy makers
4. PISA released items were analysed by the National Commissions on core subjects, as well as by the working groups for the national examinations
5. Some of the recommendations made for revising the National Curriculum for lower secondary education (gymnasium) are based on the PISA frameworks (2000 and 2006)
6. The item writers for the National Evaluation at grade VIII have served as markers in PISA 2009 exercise

22. 4. What elements of impact can be highlighted? (continued 3)
ROMANIA
THE MINISTRY OF EDUCATION, RESEARCH AND INNOVATION
THE NATIONAL CENTRE FOR CURRICULUM AND ASSESSMENT IN PRE-UNIVERSITY EDUCATION
THE PISA 2009 NATIONAL CENTER
4. What elements of impact can be highlighted? (continued 3)
Aspects of “the top-down effect”:
 The new legislative package to be adopted (the “Law of National Education”) contains references to “ the model of international tests”
 Yet, no medium and long term strategy of participation to the international comparative studies can be secured during economic crisis !

23. ROMANIA
THE MINISTRY OF EDUCATION, RESEARCH AND INNOVATION
THE NATIONAL CENTRE FOR CURRICULUM AND ASSESSMENT IN PRE-UNIVERSITY EDUCATION
THE PISA 2009 NATIONAL CENTER
4. What elements of impact can be highlighted? (continued 4)
Each country / education system is finding its own way of benefiting from the international comparative studies databases….
YET….
 Are there any common ways for building up cooperation at regional level for exploring these national databases?
 What would be the next phase, after using PISA results for “lobbying for educational reforms”, in this region?

24. www.cnceip.ro Information regarding PISA ROMANIA is available on:
THE MINISTRY OF EDUCATION, RESEARCH AND INNOVATION
THE NATIONAL CENTRE FOR CURRICULUM AND ASSESSMENT IN PRE-UNIVERSITY EDUCATION
THE PISA 2009 NATIONAL CENTER
Information regarding PISA ROMANIA is available on:
and
THANK YOU!