1. C R E S S T / U C L A Impact of Linguistic Factors in Content-Based Assessment for ELL Students Jamal Abedi UCLA Graduate School of Education & Information Studies Center for the Study of Evaluation National Center for Research on Evaluation, Standards, and Student Testing Paper presented at the 2003 Annual Meeting of the American Educational Research Association Chicago April 2003

2. C R E S S T / U C L A Validity of Academic Achievement Measures We will focus on construct and content validity approaches: A test’s content validity involves the careful definition of the domain of behaviors to be measured by a test and the logical design of items to cover all the important areas of this domain (Allen & Yen, 1979, p. 96). A test’s construct validity is the degree to which it measures the theoretical construct or trait that it was designed to measure (Allen & Yen, 1979, p. 108). A content-based achievement test has construct validity if it measures the content that it is supposed to measure. A content-based achievement test has content validity if the test content is representative of the content being measured. Examples:

3. C R E S S T / U C L A Two major questions on the psychometrics of academic achievement tests for ELLs: 1. Are there any sources of measurement error that may specifically influence ELL performance? 2. Do achievement tests accurately measure ELLs’ content knowledge?

4. C R E S S T / U C L A l Familiarity/frequency of non-math vocabulary: unfamiliar or infrequent words changed census > video game l Length of nominals: long nominals shortened last year’s class vice president > vice president l Question phrases: complex question phrases changed to simple question words At which of the following times > When Linguistic Modification Concerns

5. C R E S S T / U C L A l Conditional clauses: conditionals either replaced with separate sentences or order of conditional and main clause changed If Lee delivers x newspapers > Lee delivers x newspapers l Relative clauses: relative clauses either removed or re-cast A report that contains 64 sheets of paper > He needs 64 sheets of paper for each report Linguistic Modification (continued) l Voice of verb phrase: passive verb forms changed to active The weights of 3 objects were compared > Sandra compared the weights of 3 rabbits

6. C R E S S T / U C L A CRESST Studies on the Assessment and Accommodation of ELL Students

7. C R E S S T / U C L A Analyses of extant data (Abedi, Lord, & Plummer, 1995) Used existing data from NAEP 1992 assessments in math and science. SAMPLE: Approximately 100,000 ELL and non-ELLs in grades 4, 8, and 12. NAEP test items were grouped into long and short items. Findings l ELL students performed significantly lower on the longer test items. l ELL students had higher proportions of omitted and/or not-reached items. l ELL students had higher scores on the less linguistically complex items.

8. C R E S S T / U C L A Interview study (Abedi, Lord, & Plummer, 1997) 37 students asked to express their preference between the original NAEP items and the linguistically modified version of these same items. Math test items were modified to reduce the level of linguistic complexity. Findings l Over 80% interviewed preferred the linguistically modified items over the original version.

9. C R E S S T / U C L A Impact of linguistic factors on students’ performance (Abedi, Lord, & Plummer, 1997) Two studies: testing performance and speed. SAMPLE: 1,031 grade 8 ELL and non-ELL students. 41 classes from 21 southern California schools. Findings l ELL students who received a linguistically modified version of the math test items performed significantly better than those receiving the original test items.

10. C R E S S T / U C L A The impact of different types of accommodations on students with limited English proficiency (Abedi, Lord, & Hofstetter, 1997) SAMPLE: 1,394 grade 8 students. 56 classes from 27 southern California schools. Findings Spanish translation of NAEP math test. l Spanish-speakers taking the Spanish translation version performed significantly lower than Spanish-speakers taking the English version. l We believe that this is due to the impact of language of instruction on assessment. Linguistic Modification l Contributed to improved performance on 49% of the items. Extra Time l Helped grade 8 ELL students on NAEP math tests. l Also aided non-ELL students. Limited potential as an assessment accommodation.

11. C R E S S T / U C L A Impact of selected background variables on students’ NAEP math performance. (Abedi, Hofstetter, & Lord, 1998) SAMPLE: 946 grade 8 ELL and non-ELL students. 38 classes from 19 southern California schools. Findings Four different accommodations used (linguistically modified, a glossary only, extra time only, and a glossary plus extra time). The glossary plus extra time was the most effective accommodation. Glossary plus extra time accommodation l Non-ELLs showed a greater improvement (16%) than the ELLs (13%). l This is the opposite of what is expected and casts doubt on the validity of this accommodation.

12. C R E S S T / U C L A The effects of accommodations on the assessment of LEP students in NAEP (Abedi, Lord, Kim, & Miyoshi, 2000) SAMPLE: 422 grade 8 ELL and non-ELL students. 17 science classes from 9 southern California schools. Findings Some forms of accommodations may help the recipients with the content of assessment. For example, a dictionary defines all the words in a test, both content and non-content. A Customized Dictionary l easier to use than published dictionary l included only non-content words in the test l ELL students showed significant improvement in performance l no impact on non-ELL performance

13. C R E S S T / U C L A Language accommodation for large-scale assessment in science (Abedi, Courtney, Leon, Mirocha, & Goldberg, 2001) SAMPLE: 612 grades 4 and 8 students. 25 classes from 14 southern California schools. Findings l A published dictionary was both ineffective and administratively difficult to implement as an accommodation.

14. C R E S S T / U C L A Language accommodation for large-scale assessment in science (Abedi, Courtney, & Leon, 2001) SAMPLE: 1,856 grade 4 and 1,512 grade 8 ELL and non-ELL students. 132 classes from 40 school sites in four cities, three states. Findings l Results suggested that linguistic modification of test items improved performance of ELLs in grade 8 l No change on the performance of non-ELLs with modified test l The validity of assessment was not compromised by the provision of an accommodation

15. C R E S S T / U C L A Impact of students’ language background on content-based performance: Analyses of extant data (Abedi & Leon, 1999) Analyses were performed on extant data, such as Stanford 9 and ITBS SAMPLE: Over 900,000 students from four different sites nationwide. Examining ELL and non-ELL student performance differences and their relationship to background factors (Abedi, Leon, & Mirocha, 2001) Data were analyzed for the language impact on assessment and accommodations of ELL students. SAMPLE: Over 700,000 students from four different sites nationwide. Findings l The higher the level of language demand of the test items, the higher the performance gap between ELL and non-ELL students. l Large performance gap between ELL and non-ELL students on reading, science and math problem solving (about 15 NCE score points). l This performance gap was reduced to zero in math computation.

16. C R E S S T / U C L A Normal Curve Equivalent Means and Standard Deviations for Students in Grades 10 and 11, Site 3 School District Reading Science Math MSD M SD M SD Grade 10 SD only16.412.725.513.322.511.7 LEP only24.016.432.915.336.816.0 LEP & SD16.311.224.8 9.323.6 9.8 Non-LEP & SD38.016.042.617.239.616.9 All students36.016.941.317.538.517.0 Grade 11 SD only 14.913.221.512.324.313.2 LEP only22.516.128.414.445.518.2 LEP & SD15.512.726.120.125.113.0 Non-LEP & SD38.418.339.618.845.221.1 All students36.219.038.218.944.021.2 Note. LEP = limited English proficient. SD = students with disabilities.

17. C R E S S T / U C L A Disparity Index (DI) is an index of performance differences between LEP and non-LEP. Site 3 Disparity Index (DI) Non-LEP/Non-SD Students Compared to LEP-Only Students Disparity Index (DI) Math Math Grade Reading Math Total Calculation Analytical 3 53.425.812.932.8 6 81.637.622.246.1 8125.236.925.244.0

18. C R E S S T / U C L A