1. Testing of English- and French- Speaking Students in Canada / Évaluation des élèves anglophones et des élèves francophones au Canada Kadriye Ercikan, UBC, Marielle Simon, uOttawa, Maria Elena Oliveri, UBC & Normand Dufour, MÉLS Canadian Society for the Study of Education Canadian Education Researchers Association May-June 2010, Concordia University, Montreal May-June 2010, Concordia University, Montreal

2. Objective Describe the context, challenges, actual procedures, other solutions and implications for achieving comparability of English and French student test results on large-scale assessments (LSAs) in Canada

3. Context Outside of Quebec, Francophone students form between 1 to 34% of the population Outside of Quebec, Francophone students form between 1 to 34% of the population In Quebec, Anglophone students make up 19% of the population In Quebec, Anglophone students make up 19% of the population Anglophone and Francophone minority groups learn and are tested in their respective language. Anglophone and Francophone minority groups learn and are tested in their respective language. Francophone students are provided education in French in most jurisdictions Francophone students are provided education in French in most jurisdictions Only New Brunswick officially adopted the two national languages Only New Brunswick officially adopted the two national languages

4. Context French Immersion programs are made widely available in all the provinces French Immersion programs are made widely available in all the provinces Comparative comments often made: “As was the case in PISA 200o and PISA 2003, students enrolled in the French-language school systems in Nova Scotia, New-Brunswick, Ontario and Manitoba performed significantly lower in reading than did students in the English-language systems in the same provinces” (Government of Canada, 2008 web site). Comparative comments often made: “As was the case in PISA 200o and PISA 2003, students enrolled in the French-language school systems in Nova Scotia, New-Brunswick, Ontario and Manitoba performed significantly lower in reading than did students in the English-language systems in the same provinces” (Government of Canada, 2008 web site). Test scores are used for obtaining diploma, scholarship decisions, university entrance selection, etc. Test scores are used for obtaining diploma, scholarship decisions, university entrance selection, etc.

5. Context 7 out of 10 jurisdictions surveyed in 2007 said comparability was intended 7 out of 10 jurisdictions surveyed in 2007 said comparability was intended 1 jurisdiction’s tests are developed and administered independently 1 jurisdiction’s tests are developed and administered independently Holistic scoring by teachers in 1 jurisdiction needs validation committees to ensure comparability Holistic scoring by teachers in 1 jurisdiction needs validation committees to ensure comparability The two language versions of a test may be developed based on different target curricula, content specifications or teaching resources The two language versions of a test may be developed based on different target curricula, content specifications or teaching resources

6. Procedures Comparability verification proceduresNumber of jurisdictions Back translation2 Review of language versions by experts 8 Review of language versions by independent groups 4 Pilot-testing with students6 Comparing item difficulties4 Using cultural sensitivity panels4

7. Challenges Ideal conditions for scores to be comparable: Equivalence of tests Equivalence of tests construct definitions construct definitions items items Equivalence of scores Equivalence of scores through parallel tests through parallel tests Comparability of scores through linking procedures Comparability of scores through linking procedures simultaneous calibration simultaneous calibration anchor item linking anchor item linking

8. Challenges Concern with comparability of scores is relatively recent; Concern with comparability of scores is relatively recent; There is great variation in provincial practices; There is great variation in provincial practices; There is preference for development and validation committees over statistical procedures; There is preference for development and validation committees over statistical procedures; The small minority language sample sizes limit: The small minority language sample sizes limit: the types of psychometric procedures to create accurate score scales for the minority students;the types of psychometric procedures to create accurate score scales for the minority students; linking score scales;linking score scales; demonstration of equivalence.demonstration of equivalence.

9. Potential Solutions Adaptation solutions: Adaptation solutions: Use parallel, successive (Rogers et al, 2003), concurrent (Solano-Flores et al, 2002) or simultaneous (Tanzer, 2005) test development strategies Use parallel, successive (Rogers et al, 2003), concurrent (Solano-Flores et al, 2002) or simultaneous (Tanzer, 2005) test development strategies Expert reviews Expert reviews Content (Bolwes et al, 2008) Content (Bolwes et al, 2008) Linguistic (for equivalence of meaning, cognitive requirements, difficulty of vocabulary and expressions and cue given to answer items) Linguistic (for equivalence of meaning, cognitive requirements, difficulty of vocabulary and expressions and cue given to answer items)

10. Potential Solutions Psychometric evidence based on Psychometric evidence based on Classical test theory Classical test theory Unidimensionality analyses (Oliveri et al, in press) Unidimensionality analyses (Oliveri et al, in press) Identification of DIF and its sources Identification of DIF and its sources Student cognitive processes using think aloud processes (TAPs) (Ercikan et al, 2010) Student cognitive processes using think aloud processes (TAPs) (Ercikan et al, 2010)

11. Final Note We reviewed and summarized research, practices and challenges and recommended methodologies for examining comparability of test items and tests given in multiple languages. We reviewed and summarized research, practices and challenges and recommended methodologies for examining comparability of test items and tests given in multiple languages. We want to highlight the importance of optimizing comparability in the development phase of assessments. We want to highlight the importance of optimizing comparability in the development phase of assessments. We also want to encourage further research to investigate the comparability of cognitive processes of students from different language groups as integral components of validity investigations. We also want to encourage further research to investigate the comparability of cognitive processes of students from different language groups as integral components of validity investigations.

12. References Bowles, M. S. & Stanfield, C. W. (2008). A Practical Guide to Standards- Based Assessment in the Native Language. Retrieved on March 28, 2008 from.ncela.gwu.edu/spotlight/LEP/2008/bowles_stansfield.pdf Bowles, M. S. & Stanfield, C. W. (2008). A Practical Guide to Standards- Based Assessment in the Native Language. Retrieved on March 28, 2008 from.ncela.gwu.edu/spotlight/LEP/2008/bowles_stansfield.pdf.ncela.gwu.edu/spotlight/LEP/2008/bowles_stansfield.pdf Ercikan, K., Arim, R.,G., Law, D. M., Lacroix, S., Gagnon, F., & Domene, J. F. (2010). Application of think-aloud protocols in examining sources of differential item functioning. Educational Measurement: Issues and Practice, 29, 24-35. Ercikan, K., Arim, R.,G., Law, D. M., Lacroix, S., Gagnon, F., & Domene, J. F. (2010). Application of think-aloud protocols in examining sources of differential item functioning. Educational Measurement: Issues and Practice, 29, 24-35. Rogers, W. T., Gierl, M.J., Tardif, C., Lin, J., & Rinaldi, C. M. (2003). Differential validy and utility of successive and simultaneous approaches to the development of equivalent achievement tests in French and English. Alberta Journal of Educational Research, 49(3), 290-304. Rogers, W. T., Gierl, M.J., Tardif, C., Lin, J., & Rinaldi, C. M. (2003). Differential validy and utility of successive and simultaneous approaches to the development of equivalent achievement tests in French and English. Alberta Journal of Educational Research, 49(3), 290-304. Solano-Flores, G,, Trumbull, E., & Nelson-Barber, S. (2002). Concurrent Development of Dual Language Assessments: An Alternative to Translating Tests for Linguistic Minorities International Journal of Testing, 2, 107 – 129. Solano-Flores, G,, Trumbull, E., & Nelson-Barber, S. (2002). Concurrent Development of Dual Language Assessments: An Alternative to Translating Tests for Linguistic Minorities International Journal of Testing, 2, 107 – 129.International Journal of TestingInternational Journal of Testing Tanzer, N.K. (2005). Developing tests for use in multiple languages and cultures: A plea for simultaneous development. In: R.K. Hambleton, P.F. Merenda & C.D. Spielberger (Eds.), Adapting educational and psychological tests for cross-cultural assessment. Mahwah, NJ: Erlbaum. Tanzer, N.K. (2005). Developing tests for use in multiple languages and cultures: A plea for simultaneous development. In: R.K. Hambleton, P.F. Merenda & C.D. Spielberger (Eds.), Adapting educational and psychological tests for cross-cultural assessment. Mahwah, NJ: Erlbaum.