Benchmarking with National and International Assessments Larry V. Hedges Northwestern University This paper is intended to promote the exchange of ideas among researchers and policy makers. The views expressed in it are part of ongoing research and analysis and do not necessarily reflect the position of the National Center for Education Statistics, the Institute of Education Sciences, or the U.S. Department of Education.

Advantages of Benchmarking with International Assessments Assess your population versus other nations Permits comparison with an external standard However: Comparisons are informative only if they compare like with like Countries may not be the most relevant external standards (because they differ in too many ways)

International Assessments are Limited International assessments involve compromises across nations with very different curricula and education systems Compromises are essential to assure broader relevance However, Compromises limit the assessment’s relevance for any local purposes.

International Assessments are Limited Compromises involve Content specifications Assessment designs Sampling of ages/grades Background questions International assessments are cross sectional Assessments (cross sectional or longitudinal) are not suited for hypothesis testing

PISA PISA was constructed as a measure of literacy in reading, math, and science It measures life skills rather than academic skills specifically taught in school This has some advantages in constructing a cross national assessment This limits its usefulness for monitoring school policy or the outputs of schools.

PISA Because it is not explicitly tied to school curriculum, the relation of PISA to school policies is not obvious An assessment of 15 year olds is not particularly policy relevant for measuring the output of US schools The background questionnaire material is well suited to international comparisons, but imperfect for US purposes The temptation to use cross sectional comparisons to draw conclusions is likely to be irresistible

IEA Studies (TIMSS and PIRLS) Another benchmarking possibility is the regular cycle of IEA international comparative studies (TIMSS and PIRLS) These assessment instruments are tied more closely to academic skills that are explicitly taught in schools They still offer the possibility of international standards, but are likely to be more relevant to curriculum and instruction and school policies.

IEA Studies (TIMSS and PIRLS) It should be simpler to draw conclusions about their relation to school policies involving instruction Their sampling designs permit international comparison at more relevant ages Background material may be more policy relevant, including extensive material on instruction The temptation to draw conclusion from cross sectional data will still be there

NAEP There already is a program of immediately relevant cross sectional data collection in the US: NAEP This assessment involves no international compromises on content specification It is the de facto standard for measuring academic achievement in the US It has a long time series of achievement scores However, it does have only very limited background information.

NAEP It is already in place and is a trusted source of information about achievement It is based entirely on US standards The sampling is relevant to US policy Background material is limited. Limited background information means there is less temptation to draw conclusions

NCES Longitudinal Studies The NCES longitudinal studies are another potential source of benchmarking opportunities Arguably they are better suited for hypothesis generation than are the cross sectional studies They have far superior background measurement (for research purposes) They could provide comparisons over a wide age range