Key steps in developing assessment standards: Lessons learned in a regional primary learning assessment systems in Sub-Saharan Africa Regional Consultative Meeting on SEAMEO Basic Education Standards and Southeast Asia Primary Learning Metrics 4-5 November 2014 Penang, Malaysia Mioko Saito, UNESCO-IIEP

Presentation structure  IIEP’s involvement in large-scale assessments  Assessment as an integral part of policy cycle  Key steps in developing assessment standards 4

International Institute for Educational Planning (IIEP) and Large-scale Assessments Large-scale Regional Assessment SACMEQ as an independent organization Large-scale National Assessment in Zimbabwe IIEP as an Initiator Large-scale National Assessment in Viet Nam IIEP as a Partner SACMEQ II IIEP as a Partner SACMEQ I IIEP as a Coordinator 2005-present SACMEQ III & IV IIEP as a Supporter WEF 1990 JomtienWEF 2000 DakarWEF 2015 Incheon Large-scale Regional Assessments?

IIEP’s approach to achieve international goals on learning assessments  NOT a short-term crisis management by external agencies  Long-term capacity building to: –Provide advanced technical training required to undertake high-quality, large-scale, scientific studies –Foster an “information culture” where policy decisions are informed decisions.

Policy reform and agenda for action Consultation and debate Policy suggestions Interpretation and reporting Data collection and analysis Specific research questions General policy concerns Programme implementation Policy Cycle Source: Saito (1999)

SACMEQ Working Style  Ownership = Ministries of Education  Not a research project -- but educational planners involved in research  Target population & test framework based on policy and planning context  Highest priority: –"learning by doing" approach –Institutional capacity building –Regional collaboration Self-help approach South-south cooperation Source: Saito & van Cappelle (2010)

Analyze official curricula, school syllabi, textbooks, and examinations Cross-check the descriptions of competency levels with the test blueprint to ensure face validity and construct validity Describe domains (content)Describe skills (behaviours) Construct test blueprint (domains by skills grid) Write trial-test items (twice as many as required) Implement trial testing and complete Rasch and classical item analysis Select test items to achieve (a) balance across test blueprint, and (b) “overlaps” with other studies Implement main data collection Conduct Rasch item calibration and test equating Complete skills audit to identify the specific skills required for success on each item Cluster items with similar difficulties and requiring similar skills into competency levels Develop summary descriptions for each competency level Test Construction Steps Source: Ross et al (2004)

SACMEQ Comparison  Comparisons across sub-groups (sex, location, SES)  Comparisons against Ministry’s benchmark standards (defined by subject experts)  Comparisons across levels of competence  Comparison between time  Comparison between pupils and teachers  Comparison among countries  Comparison with other international studies (IEA- PIRLS, TIMSS, PASEC) Sources: Ross et al (2004); Saito & van Cappelle (2010)

References  Ross, K., Saito, M., Dolata, S., Ikeda, M., & Zuze, L Data archive for the SACMEQ I and SACMEQ II projects. Paris: IIEP- UNESCO.  Saito, M. (1999). A generalizable model for educational policy research in developing countries. Journal of International Cooperation in Education, 2(2), pp  Saito, M. & van Cappelle, F. (2010). “Monitoring the quality of Education: Exploration of concept, methodology, and the link between research and policy”. In: A.W. Wiseman (Ed.) The impact of international achievement studies on national education policymaking. International perspectives on education and society, Vol. 13 (pp. 3-34). Bingley: Emerald Group Publishing. 11

 For more information about SACMEQ:  To apply for the IIEP’s Advanced Training Programme for Educational Planning and Management:  To apply for the IIEP’s Specialized Course on Quantitative Methods for Monitoring and Evaluating the Quality of Education: 12