1. SSA – Technical Cooperation Fund End of Project Conference The Role of International Achievement Studies (OECD PISA, IEA TIMSS, PIRLS…) Importance of Large-scale International Comparative Student Achievement Studies in Education Dr. Algirdas Zabulionis Anglia Assessment Ltd UK/Lithuania

3. Large-scale international comparative student achievement studies in education can be grouped into two broad categories according to their main purposes: research effective education policy Which of these purposes dominates the other?

4. Components of the Study Framework Research Questions Instruments Study Design Data Analysis Utilisation of Results General Goals Conceptual Framework

5. Main “players” in the field of international comparative studies in education IEA - International Association for the Evaluation of Educational Achievement OECD - Organisation for Economic Co-operation and Development

6. International Association for the Evaluation of Educational Achievement (IEA) Since 1958, IEA has conducted over 30 research studies of cross-national achievement. Currently best known– TIMSS and PIRLS. Nearly 100 countries participate in IEA’s projects. Over 50 years, IEA projects have built a strong global network of researchers, policy analysts, and technical experts in large-scale educational research.

7. From IEA Mission Statements: provides international benchmarks that identify strengths and weaknesses in educational systems; provides high quality data that point to key factors that influence teaching and learning; provides high quality data that direct educational reforms; develops educational systems’ capacity to engage in national strategies for monitoring and improving performance; and, contributes to research in educational evaluation, developing a world-wide network of researchers in this critical area.

8. OECD PISA Launched by OECD in 1997. Aim: – to evaluate education systems worldwide every three years – by assessing competencies of 15-year-olds in the key subjects: reading, mathematics and science. Tests not directly linked to school curriculum – Designed to assess to what extent students at the end of compulsory education, can apply their knowledge to real-life situations and be equipped for full participation in society. Over 70 countries have participated in PISA.

9. IEA TIMSS - Trends in Mathematics and Science Study – TIMSS - Third International Math and Science Study (1995) – SIMSS - Second International Mathematics Study (1980) – FIMSS - First International Mathematics Study (1964) IEA PIRLS – Progress in International Reading Literacy Study OECD PISA – Programme for International Student Assessment

10. Comparing TIMSS/PIRLS and PISA TIMSS Grades 4 - 8 -12 Math & Science 4 years cycle 1995-1999-2003-2007 -2011-2015… PIRLS Grade 4 Reading Literacy 5 years cycle 2001-2005-2011-2016… PISA 15-years old students Reading-Math-Science- Technologies 3 years cycle: 1997-200-2003-2006-2009- 2012-2015…

11. TIMSS/PIRLS Curriculum based Assesses: intended curriculum (what policy requires) implemented curriculum (what is taught in schools) achieved curriculum (what students learn). PISA National curriculum free Assesses: extent to which students near the end of compulsory education have acquired some of the knowledge and skills essential for full participation in society. Comparing TIMSS/PIRLS and PISA

12. TIMSS What maths & science (reading) have you been taught and how much have you learned? PISA What can you do with the maths & science (reading) you have been taught? Comparing TIMSS and PISA

18. PISA-2009 PLUS: Percentage of students at each proficiency level of reading literacy

19. It is the quantity of reading that marks PISA out, not the complexity of the language, which is similarly unfamiliar in both the international studies. The high reading demand of questions in PISA is often accompanied by a relatively lower demand in the mathematics or science required. This reflects the lower level of mathematics or science that students can apply in new contexts as opposed to very familiar ones. Conclusions from ‘Comparison Study for PISA-TIMSS in England'; Ruddock et al., 2006 “ ”

20. TIMSS focuses on the curriculum related tasks, while PISA is literacy-based. Hungarian school system is still highly relies on factual knowledge and traditional teaching strategies, so students are relatively good in tasks which are close to their usual classroom tasks, while they meet relatively few literacy-based tasks and they do not know what to do with these. ‘Comparison of PISA-TIMSS in Hungary’; Balazsi, 2006 “ ”

21. There are More International Studies… IEA CIVIC, CompEd (SITES, ICILS), TEDS… (IEA) PIAAC - Programme for the International Assessment of Adult Competencies, ALL - Adult Literacy and Life skills Survey (OECD) NAEP –National Assessment of Educational Progress (USA) EGRA, EGMA - The Early Grade Reading (Math) Assessment SACMEQ – Southern Africa Consortium for Monitoring Educational Quality (Angola, Botswana, Kenya, Lesotho, Malawi, Namibia, …, Zimbabwe) LAMP - Literacy Assessment and Monitoring Programme (UNESCO) LLECE — Laboratorio Latinoamericano de Evaluación de la Calidad de la Educación (Latin America)

22. Summary There are many interesting international student achievement studies going on around the world. Taking part in a major international survey: –is a challenge for the country concerned; –offers great value to researchers; –presents some risks for education policymakers; –is, in absolute terms, an expensive business. A country needs to be sure the survey it chooses addresses national priorities and the information it will provide will be utilised with value for policymakers, researchers and practitioners.

23. Thank you for your attention!

24. End of project Conference: Focusing on Evidence based Education System SSA – Technical Cooperation Fund www.ssatcfund.org