American Diploma Project 11 September 2009 Andreas Schleicher International Benchmarking International Benchmarking What it means – what it takes Washington, September 11, 2009 Andreas Schleicher Head, Indicators and Analysis Division OECD Directorate for Education

Dimensions for educational benchmarking National educ, social and economic context Structures, resource alloc and policies Social & economic outcomes of education Community and school characteristics Student learning, teacher working conditions Socio-economic background of learners Antecedents contextualise or constrain ed policy The learning environment at school Teaching, learning practices and classroom climate Individ attitudes, engagement and behaviour Output and performance of institutions Quality of instructional delivery Quality and distribution of knowledge & skills Policy Levers shape educational outcomes Outputs and Outcomes impact of learning Individual learner Level A Instructional settings Level B Schools, other institutions Level C Country or system Level D Domain 3Domain 2 Domain 1

National educ, social and economic context Structures, resource alloc and policies Social & economic outcomes of education Community and school characteristics Student learning, teacher working conditions Socio-economic background of learners Antecedents contextualise or constrain ed policy The learning environment at school Teaching, learning practices and classroom climate Individ attitudes, engagement and behaviour Output and performance of institutions Quality of instructional delivery Quality and distribution of knowledge & skills Policy Levers shape educational outcomes Outputs and Outcomes impact of learning Individual learner Level A Instructional settings Level B Schools, other institutions Level C Country or system Level D Domain 3Domain 2 Domain 1 Dimensions for educational benchmarking

Expenditure per student at tertiary level (USD) Tertiary-type A graduation rate A world of change – higher education Graduate supply Cost per student

Expenditure per student at tertiary level (USD) Tertiary-type A graduation rate A world of change – higher education United States Finland Graduate supply Cost per student

Expenditure per student at tertiary level (USD) Tertiary-type A graduation rate A world of change – higher education Australia Finland United Kingdom

Expenditure per student at tertiary level (USD) Tertiary-type A graduation rate A world of change – higher education

Expenditure per student at tertiary level (USD) Tertiary-type A graduation rate A world of change – higher education

Expenditure per student at tertiary level (USD) Tertiary-type A graduation rate A world of change – higher education

Expenditure per student at tertiary level (USD) Tertiary-type A graduation rate A world of change – higher education

Expenditure per student at tertiary level (USD) Tertiary-type A graduation rate A world of change – higher education

Expenditure per student at tertiary level (USD) Tertiary-type A graduation rate A world of change – higher education United States Australia Finland

National educ, social and economic context Structures, resource alloc and policies Social & economic outcomes of education Community and school characteristics Student learning, teacher working conditions Socio-economic background of learners Antecedents contextualise or constrain ed policy The learning environment at school Teaching, learning practices and classroom climate Individ attitudes, engagement and behaviour Output and performance of institutions Quality of instructional delivery Quality and distribution of knowledge & skills Policy Levers shape educational outcomes Outputs and Outcomes impact of learning Individual learner Level A Instructional settings Level B Schools, other institutions Level C Country or system Level D Domain 3Domain 2 Domain 1 Dimensions for educational benchmarking

Education Indicators Programme 2009 edition of Education at a Glance Contribution of various factors to instructional cost per high school student as a percentage of GDP per capita (2006) Percentage points B7.1

American Diploma Project 11 September 2009 Andreas Schleicher International Benchmarking Benchmarking Funding Stakeholder involvement Human capital State and regional context Organisation of schooling Curriculum and assessment High quality learning for every student Leadership and school ethos Effective classroom practice Standards and intervention in inverse proportion to success Teaching quality and professional development Autonomy and accountability Infrastructure Strategies Teaching and learning Teacher quality and professional development opportunities Attention to sectors, subjects, positions and individual learning needs Incentives for teachers to enhance classroom practice Time for teachers to prepare lesson and teaching material, as well as collaboration and exchange of ideas between teachers School leadership Internal processes for audit of quality and effectiveness of teaching across schools, external corroboration Instructional leadership Innovation and knowledge management Staff development, quality assurance and quality improvement Support for other schools or networks, “system leaders” Relationships between teachers, heads, supervisors, inspectors and advisors etc. Devolution of first line responsibility for the quality of educational provision to the point of delivery Local leadership and teacher development Feedback systems which provide information about effectiveness and trends in outcomes Performance indicators, stakeholder surveys, professional evaluation Definition of expected standards Competencies and content, relevance and links to “real world” Alignment of curricula with standards Quality of materials to support work of teachers Assessment approaches around the standards Assessment for learning Assessment of learning and monitoring

National educ, social and economic context Structures, resource alloc and policies Social & economic outcomes of education Community and school characteristics Student learning, teacher working conditions Socio-economic background of learners Antecedents contextualise or constrain ed policy The learning environment at school Teaching, learning practices and classroom climate Individ attitudes, engagement and behaviour Output and performance of institutions Quality and distribution of knowledge & skills Policy Levers shape educational outcomes Outputs and Outcomes impact of learning Individual learner Level A Instructional settings Level B Schools, other institutions Level C Country or system Level D Domain 3Domain 2 Domain 1 Dimensions for educational benchmarking Quality of instructional delivery

International Assessment Seminar 11 September 2009 Andreas Schleicher Lessons from PISA for developing assessments Average performance of 15-year-olds on PISA in science High science performance Low science performance … 18 countries perform below this line

International Assessment Seminar 11 September 2009 Andreas Schleicher Lessons from PISA for developing assessments Durchschnittliche Schülerleistungen im Bereich Mathematik Low average performance Large socio-economic disparities High average performance Large socio-economic disparities Low average performance High social equity High average performance High social equity Strong socio- economic impact on student performance Socially equitable distribution of learning opportunities High science performance Low science performance 15

International Assessment Seminar 11 September 2009 Andreas Schleicher Lessons from PISA for developing assessments 20 Consistency in quality standards Variation in the performance of 15-year-olds in mathematics

International Assessment Seminar 11 September 2009 Andreas Schleicher Lessons from PISA for developing assessments Variation of performance between schools Variation of performance within schools Consistency in quality standards Variation in the performance of 15-year-olds in mathematics OECD (2004), Learning for tomorrow’s world: First results from PISA 2003, Table 4.1a, p.383.

International Assessment Seminar 11 September 2009 Andreas Schleicher Lessons from PISA for developing assessments Strengths and weaknesses of countries in science relative to their overall performance France OECD (2007), PISA 2006 – Science Competencies for Tomorrow’s World, Figure 2.13 Science competencies Science knowledge

International Assessment Seminar 11 September 2009 Andreas Schleicher Lessons from PISA for developing assessments Strengths and weaknesses of countries in science relative to their overall performance Czech Republic OECD (2007), PISA 2006 – Science Competencies for Tomorrow’s World, Figure 2.13 Scientific competencies Scientific knowledge

National educ, social and economic context Structures, resource alloc and policies Social & economic outcomes of education Community and school characteristics Student learning, teacher working conditions Socio-economic background of learners Antecedents contextualise or constrain ed policy The learning environment at school Teaching, learning practices and classroom climate Individ attitudes, engagement and behaviour Output and performance of institutions Quality of instructional delivery Quality and distribution of knowledge & skills Policy Levers shape educational outcomes Outputs and Outcomes impact of learning Individual learner Level A Instructional settings Level B Schools, other institutions Level C Country or system Level D Domain 3Domain 2 Domain 1 Dimensions for educational benchmarking

International Assessment Seminar 11 September 2009 Andreas Schleicher Lessons from PISA for developing assessments Low policy value High policy value Low feasibilityHigh feasibility Money pits Must haves Low-hanging fruits Quick wins Examine individual, institutional and systemic factors associated with high performance Establish the relative standing of states on international standards Extending the range of competencies through which outcomes are assessed Measuring growth in learning A real-time assessment environment that bridges the gap between formative and summative assessment. Monitor educational progress Assuming that every new skill domain is orthogonal to all others

American Diploma Project 11 September 2009 Andreas Schleicher International Benchmarking What it takes…

American Diploma Project 11 September 2009 Andreas Schleicher International Benchmarking Coverage of world economy 77%81% 83% 85%86%87% OECD’s PISA assessment of the knowledge and skills of 15-year-olds Subnational/regional PISA assessments in CountryCoordinated byReported at AustraliaNational authoritiesNational level BelgiumRegionsOECD level BrazilRegionsNational level CanadaNational authoritiesNational level GermanyNational authoritiesNational level ItalyRegionsOECD level MexicoNational authoritiesNational level SpainRegionsOECD level SwitzerlandRegionsNational level United KingdomRegionsOECD level

American Diploma Project 11 September 2009 Andreas Schleicher International Benchmarking What it takes r Implementing PISA volume of the tests, e.g. –3½ hours of main assessment area –1 hour for each of the minor assessment areas each student –2 hours on paper-and-pencil tasks (subset of all questions) –½ hour for questionnaire on background, learning habits, learning environment, engagement and motivation school principals –questionnaire (school demography, learning environment) r Alternatives Suitable if state performance is main interest –Curriculum match / assessment match –Embedding PISA items in state tests –Embedding state items in PISA tests Requires coherence between assessment frameworks.

American Diploma Project 11 September 2009 Andreas Schleicher International Benchmarking Test Items r A unit structure r Authentic stimuli/contexts r High proportion of constructed response items Multiple-choice Short constructed responses Open constructed responses.

American Diploma Project 11 September 2009 Andreas Schleicher International Benchmarking Main products r A set of basic indicators that provide policy makers with a baseline profile of the knowledge, skills and competencies of students in their state relative to those in other countries r A set of contextual indicators that provide insight into how such skills relate to important demographic, social, economic and educational variables r Trend indicators that become available because of the on-going, cyclical nature of the data collections r A knowledge base for further focused policy analysis.

American Diploma Project 11 September 2009 Andreas Schleicher International Benchmarking –All national and international publications –The complete micro-level database …and remember: Without data, you are just another person with an opinion