National Conference on Next Generation Assessments Washington, 8-9 March 2010 Future of Assessments Andreas Schleicher The future of Assessments Lessons learned internationally Washington, 9 March 2010 Andreas Schleicher Head, Indicators and Analysis Division OECD Directorate for Education

National Conference on Next Generation Assessments Washington, 8-9 March 2010 Future of Assessments Andreas Schleicher The future of assessments The Holy Grail? Or the Alchemists’ Stone? r Know w hy you are looking You cannot improve what you cannot measure The yardstick for success is no longer just improvement by national standards but the best performing education systems globally r Know what you are looking for A new assessment culture Responsive to changing skill requirements Capitalising on methodological advances Not sacrificing validity gains for efficiency gains r Know how you will recognise it when you find it Gauging predictive validity Impact on improving learning and teaching r Implications and lessons learned.

National Conference on Next Generation Assessments Washington, 8-9 March 2010 Future of Assessments Andreas Schleicher The old bureaucratic systemThe modern enabling system Hit and miss  Universal high standards Uniformity  Embracing diversity Provision  Outcomes Bureaucratic look-up  Devolved – look outwards Talk equity  Deliver equity Prescription  Informed profession Conformity  Ingenious Curriculum-centred  Learner-centred Interactive  Participative Individualised  Community-centred Delivered wisdom  User-generated wisdom Management  Leadership Public vs private  Public with private Culture as obstacle  Culture as capital

Expenditure per student at tertiary level (USD) Tertiary-type A graduation rate A world of change – higher education Graduate supply Cost per student

Expenditure per student at tertiary level (USD) Tertiary-type A graduation rate A world of change – higher education United States Finland Graduate supply Cost per student

Expenditure per student at tertiary level (USD) Tertiary-type A graduation rate A world of change – higher education Australia Finland United Kingdom

Expenditure per student at tertiary level (USD) Tertiary-type A graduation rate A world of change – higher education

Expenditure per student at tertiary level (USD) Tertiary-type A graduation rate A world of change – higher education

Expenditure per student at tertiary level (USD) Tertiary-type A graduation rate A world of change – higher education

Expenditure per student at tertiary level (USD) Tertiary-type A graduation rate A world of change – higher education

Expenditure per student at tertiary level (USD) Tertiary-type A graduation rate A world of change – higher education

Expenditure per student at tertiary level (USD) Tertiary-type A graduation rate A world of change – higher education United States Australia Finland

Education Indicators Programme 2009 edition of Education at a Glance Latin America then… Hanushek 2009 GDP/pop 1960 Years schooling Asia Sub-Saharan Africa MENA Latin America Europe Orig. OECD

Education Indicators Programme 2009 edition of Education at a Glance GDP/pop 1960 Years schooling Asia Sub-Saharan Africa MENA Latin America Europe Orig. OECD Latin America then and now… GDP/pop 1960 Years schooling Growth GDP/pop 2000 Asia Sub-Saharan Africa MENA Latin America Europe Orig. OECD Hanushek 2009

Education Indicators Programme 2009 edition of Education at a Glance Latin America then and now… Why quality is the key Hanushek 2009 GDP/pop 1960 Years schooling Growth GDP/pop 2000 PISA score Asia Sub-Saharan Africa MENA Latin America Europe Orig. OECD

National Conference on Next Generation Assessments Washington, 8-9 March 2010 Future of Assessments Andreas Schleicher Know what you are looking for The Holy Grail was a well-described object, and there was only one true grail…

National Conference on Next Generation Assessments Washington, 8-9 March 2010 Future of Assessments Andreas Schleicher Schooling in the medieval age: The school of the church

National Conference on Next Generation Assessments Washington, 8-9 March 2010 Future of Assessments Andreas Schleicher Schooling in the industrial age: Uniform learning

National Conference on Next Generation Assessments Washington, 8-9 March 2010 Future of Assessments Andreas Schleicher Schooling in the industrial age: Uniform learning The challenges today: Universal quality Motivated and self-reliant citizens Risk-taking entrepreneurs, converging and continuously emerging professions tied to globalising contexts and technological advance

National Conference on Next Generation Assessments Washington, 8-9 March 2010 Future of Assessments Andreas Schleicher How the demand for skills has changed Economy-wide measures of routine and non-routine task input (US) (Levy and Murnane) Mean task input as percentiles of the 1960 task distribution The dilemma of assessments: The skills that are easiest to teach and test are also the ones that are easiest to digitise, automate and outsource

National Conference on Next Generation Assessments Washington, 8-9 March 2010 Future of Assessments Andreas Schleicher Changing skill demands r The great collaborators and orchestrators The more complex the globalised world becomes, the more individuals and companies need various forms of co-ordination and management r The great synthesisers Conventionally, our approach to problems was breaking them down into manageable bits and pieces, today we create value by synthesising disparate bits together r The great explainers The more content we can search and access, the more important the filters and explainers become

National Conference on Next Generation Assessments Washington, 8-9 March 2010 Future of Assessments Andreas Schleicher Changing skill demands r The great versatilists Specialists generally have deep skills and narrow scope, giving them expertise that is recognised by peers but not valued outside their domain Generalists have broad scope but shallow skills Versatilists apply depth of skill to a progressively widening scope of situations and experiences, gaining new competencies, building relationships, and assuming new roles. They are capable not only of constantly adapting but also of constantly learning and growing r The great personalisers A revival of interpersonal skills, skills that have atrhophied to some degree because of the industrial age and the Internet r The great localisers Localising the global

National Conference on Next Generation Assessments Washington, 8-9 March 2010 Future of Assessments Andreas Schleicher r Education today needs to prepare students… …to deal with more rapid change than ever before… …for jobs that have not yet been created… …using technologies that have not yet been invented… …to solve problems that we don’t yet know will arise r It’s about new… Ways of thinking –involving creativity, critical thinking, problem-solving and decision-making Ways of working –including communication and collaboration Tools for working –including the capacity to recognise and exploit the potential of new technologies The capacity to live in a multi-faceted world as active and responsible citizens.

National Conference on Next Generation Assessments Washington, 8-9 March 2010 Future of Assessments Andreas Schleicher Mathematics in PISA The real world The mathematical World A real situation A model of reality A mathematical model Mathematical results Real results Understanding, structuring and simplifying the situation Making the problem amenable to mathematical treatment Interpreting the mathematical results Using relevant mathematical tools to solve the problem Validating the results

National Conference on Next Generation Assessments Washington, 8-9 March 2010 Future of Assessments Andreas Schleicher National assessment systems differ… Who is assessed System assessment School evaluation Teacher appraisal Student assessment For what? E.g. Accountability Improvement By whom? Evaluators Users of feedback Evaluation agencies How? Methods and procedures, mix of criteria and instruments Mapping of feedback to different units System School Classroom What? Inputs Processes Outcomes With whom? Agents involved

National Conference on Next Generation Assessments Washington, 8-9 March 2010 Future of Assessments Andreas Schleicher Assessment cultures Putting the pressure on top of the education system is the easy part, building capacity is harder Administrative external Participative/internal Summative Formative Interactive, reflective, critical friend Survey Inspectorate Standardised assessment

National Conference on Next Generation Assessments Washington, 8-9 March 2010 Future of Assessments Andreas Schleicher Administrative / external Participative / internal Efficiency gainsValidity gains Large scale and high- stakes summative assessments, typically multiple-choice to contain costs (US, England, Latin America…) Formative classroom- based assessments (e.g. Europe, Asia) Large scale and low-stakes assessments, sample- based administration allows for complex task types (e.g. Northern Europe, Scotland, PISA)

National Conference on Next Generation Assessments Washington, 8-9 March 2010 Future of Assessments Andreas Schleicher …but there are global trends r Multi-layered, coherent assessment systems from classrooms to schools to regional to national to international levels that… Support improvement of learning at all levels of the education system Are largely performance-based Make students’ thinking visible and allow for divergent thinking Are adaptable and responsive to new developments Add value for teaching and learning by providing information that can be acted on by students, teachers, and administrators Are part of a comprehensive and well-aligned continuum, communicate what is expected and hold relevant stakeholders accountable. Integrate, synthesize and creatively apply content knowledge in novel situations Activate students as owners of their own learning and activate students as learning resources for one another Understanding what the assessment reveals about students’ thinking to shape better opportunities for student learning Responding to assessments can enhance student learning if tasks are well crafted to incorporate principles of learning Capitalise on improved data handling tools and technology connectivity to combine formative and summative assessment interpretations for a more complete picture of student learning

National Conference on Next Generation Assessments Washington, 8-9 March 2010 Future of Assessments Andreas Schleicher Know how you will recognise it when you find it The Alchemists’ stone was to be recognised by transforming ordinary metal into gold…

National Conference on Next Generation Assessments Washington, 8-9 March 2010 Future of Assessments Andreas Schleicher Increased likelihood of postsec. particip. at age 19/21 associated with PISA reading proficiency at age 15 (Canada) after accounting for school engagement, gender, mother tongue, place of residence, parental, education and family income (reference group PISA Level 1) Odds ratio College entry School marks at age 15 PISA performance at age 15

National Conference on Next Generation Assessments Washington, 8-9 March 2010 Future of Assessments Andreas Schleicher Relationship between test performance and economic outcomes Annual improved GDP from raising performance by 25 PISA points Percent addition to GDP

National Conference on Next Generation Assessments Washington, 8-9 March 2010 Future of Assessments Andreas Schleicher Increase average performance by 25 PISA points (Total 115 trillion $) bn$

National Conference on Next Generation Assessments Washington, 8-9 March 2010 Future of Assessments Andreas Schleicher Implications and lessons learned The medieval Alchemists’ followed the dictates of a well- established science but that was built on wrong foundations The search for the Holy Grail was overburdened by false clues and cryptic symbols

National Conference on Next Generation Assessments Washington, 8-9 March 2010 Future of Assessments Andreas Schleicher From assessment-inhibited practice towards outcome driven reform Weak focus on processes Strong focus on processes Weak outcome- based management Strong outcome-based management Integrated quality management Good will and trust External control, uninformed prescription Deprivation

National Conference on Next Generation Assessments Washington, 8-9 March 2010 Future of Assessments Andreas Schleicher Some criteria used in the world r Coherence Built on a well-structured conceptual base—an expected learning progression—as the foundation both for large scale and classroom assessments Consistency and complementarity across administrative levels of the system and across grades r Comprehensiveness Using a range of assessment methods to ensure adequate measurement of intended constructs and measures of different grain size to serve different decision-making needs Provide productive feedback, at appropriate levels of detail, to fuel accountability and improvement decisions at multiple levels r Continuity A continuous stream of evidence that tracks the progress of both individual students.

National Conference on Next Generation Assessments Washington, 8-9 March 2010 Future of Assessments Andreas Schleicher Designing assessments r Assessment frameworks A working definition of the domain and its underlying assumptions Organising the domain and identifying key task characteristics that guide task construction Operationalising task characteristics in terms of variables Validating the variables and assessing the contribution they each make to understanding task difficulty Establishing an interpretative scheme.

National Conference on Next Generation Assessments Washington, 8-9 March 2010 Future of Assessments Andreas Schleicher Understanding learning progressions r Learning targets Defining what mastery means for a given skill level r Progress variables Delineate a pathway that characterise the steps that learners typically follow as they become more proficient Evaluation of students reasoning in terms of the correctness of their solutions as well as in terms of their complexity, validity and precision r Levels of achievement Describing the breadth and depth of the learner’s understanding of the domain at a particular level of advancement r Learning performances The operational definitions of what student’s understanding would look like at each of the stages of progress. Wilson, ATC21S

National Conference on Next Generation Assessments Washington, 8-9 March 2010 Future of Assessments Andreas Schleicher Context - Personal - Social/public - Global Competencies Competencies -Identify scientific issues -Explain phenomena scientifically -Use scientific evidence Knowledge -Knowledge of science -Knowledge about science Attitudes -Interest in science -Support for scientific enquiry -Responsibility Identifying Recognising issues that can be investigated scientifically Identifying keywords in a scientific investigation Recognising the key features of a scientific investigation Explaining Applying knowledge of science in a situation Describing or interpreting phenomena scientifically or predicting change Using evidence Interpreting scientific evidence and drawing conclusions Identifying the assumptions, evidence and reasoning behind conclusions Knowledge of science Physical systems (structure of matter, properties of matter, chemical changes of matter, motions and forces, energy and its transformations, energy and matter) Living systems (cells, humans, populations, ecosystems, biosphere) Earth and space (structures of the earth system, energy in the earth system, change in the earth system, earth’s history, space) Technology systems (Concepts and principles, science and technology) Knowledge about science Scientific enquiry (purpose, experiments, data, measurement, characteristics of results) Scientific explanations (types, rules, outcomes) Interest science Indicate curiosity in science and science-related issues and endeavours Demonstrate willingness to acquire additional scientific knowledge and skills, using variety of resources and methods Demonstrate willingness to seek information and have an interest in science, including consideration of science-related careers Support for science Acknowledge the importance of considering different scientific perspectives and arguments Support the use of factual information and rational explanation Logical and careful processes in drawing conclusions

National Conference on Next Generation Assessments Washington, 8-9 March 2010 Future of Assessments Andreas Schleicher Context - Personal - Social/public - Global Competencies Competencies -Identify scientific issues -Explain phenomena scientifically -Use scientific evidence Knowledge -Knowledge of science -Knowledge about science Attitudes -Interest in science -Support for scientific enquiry -Responsibility Identifying Recognising issues that can be investigated scientifically Identifying keywords in a scientific investigation Recognising the key features of a scientific investigation Explaining Applying knowledge of science in a situation Describing or interpreting phenomena scientifically or predicting change Using evidence Interpreting scientific evidence and drawing conclusions Identifying the assumptions, evidence and reasoning behind conclusions OECD Level 6 OECD Level 2 Students can demonstrate ability to understand and articulate the complex modelling inherent in the design of an investigation. Students can determine if scientific measurement can be applied to a given variable in an investigation. Students can appreciate the relationship between a simple model and the phenomenon it is modelling. Students can draw on a range of abstract scientific knowledge and concepts and the relationships between these in developing explanations of processes Students can recall an appropriate, tangible, scientific fact applicable in a simple and straightforward context and can use it to explain or predict an outcome. Students demonstrate ability to compare and differentiate among competing explanations by examining supporting evidence. They can formulate arguments by synthesising evidence from multiple sources. Students can point to an obvious feature in a simple table in support of a given statement. They are able to recognise if a set of given characteristics apply to the function of everyday artifacts.

National Conference on Next Generation Assessments Washington, 8-9 March 2010 Future of Assessments Andreas Schleicher Some methodological challenges r Can we sufficiently distinguish the role of context from that of the underlying cognitive construct ? r Do new types of items that are enabled by computers and networks change the constructs that are being measured ? r Can we drink from the firehose of increasing data streams that arise from new assessment modes ? r Can we utilise new technologies and new ways of thinking of assessments to gain more information from the classroom without overwhelming the classroom with more assessments ? r What is the right mix of crowd wisdom and traditional validity information ? r How can we create assessments that are activators of students’ own learning ? Wilson, ATC21S

National Conference on Next Generation Assessments Washington, 8-9 March 2010 Future of Assessments Andreas Schleicher Low policy value High policy value Low feasibilityHigh feasibility Money pits Must haves Low-hanging fruits Quick wins Examine individual, institutional and systemic factors associated with performance Establish the relative standing of students and schools Extending the range of competencies through which quality is assessed Measuring growth in learning A real-time assessment environment that bridges the gap between formative and summative assessment. Monitor educational progress Assuming that every new skill domain is orthogonal to all others

PISA OECD Programme for International Student Assessment Seeing Japanese schools through the prism of PISA Principled Strategic partnership Negotiated Pragmatic. Top-down Antagonistic. Leading Evidence-driven Achieving high reliability and innovation. Enabling Incentivising. World class performance. Continuous learning and innovation. Good  Great Accommodating Evidence-based Adopting best. practice Regulating. Capacity-building Transparency. Spreading best practice Implementing Accepting evidence Adopting minimum standards Prescribing. Justifying Tackling underperformance Adequate  GoodPoor  Adequate Main focus of assessment Role of government Role of professions Nature of relationship between government and professions Phases of development Main outcomes Improvement in outcomes Reduction of public anxiety. Steady improvement Growing public satisfaction. Consistent quality Public engagement and co-production. Getting the sequencing right

National Conference on Next Generation Assessments Washington, 8-9 March 2010 Future of Assessments Andreas Schleicher –All national and international publications –The complete micro-level database …and remember: Without data, you are just another person with an opinion