1. 1/16 CRESST/UCLA Alternative Assessment for English Language Learners Christy Kim Boscardin Barbara Jones Shannon Madsen Claire Nishimura Jae-Eun Park CRESST Conference Los Angeles, CA - January, 2006 If you choose to use this title slide, simply delete the next slide (the two-line title version). This will be slide 1 of your presentation. UCLA Graduate School of Education & Information Studies National Center for Research on Evaluation, Standards, and Student Testing

2. 2/16 CRESST/UCLA Issues of Content Assessment and ELLs NCLB: Inclusion of ELLs in high-stakes tests Are content-based assessments actually measuring students’ content knowledge or are these tests unintentionally assessing students’ language proficiency? Content assessment confounded with language proficiency (Abedi & Leon, 1999, Bailey, 2000) Accommodations for Standardized tests Added challenge for Performance Assessment

3. 3/16 CRESST/UCLA Issues with Current Performance Assessment Evaluation Criteria Language expectations/proficiencies evaluated implicitly (e.g. AP, SAT) Lack of distinction between content and language skills One single score - Insufficient Language skills can compensate for lack of content knowledge

4. 4/16 CRESST/UCLA Research Questions Are we able to differentiate the various cognitive demands (content knowledge and language skills) associated with successful completion of content assessment? Specifically, how much of the content versus language skills contribute to the overall evaluation of student performance? What are the demographic and instructional factors associated with higher achievement in performance assessment?

5. 5/16 CRESST/UCLA Assessment and New Scoring Framework Biology Context Focus on scientific text & Science explanation (National Science Education Standards) Specialized content knowledge & language proficiency CRESST Model-based assessment – Science Explanation Task Explanation Dominant Genre of School-based Science Writing Adding Language Evaluation Component: Functional Linguistic Approach evaluate the effective use of the unique linguistic forms and structures which are specific to communicating scientific knowledge Used in previous studies to evaluate student writing

6. 6/16 CRESST/UCLA Method: Integrated Learning Assessment Text Passage, Reading Comprehension, Explanation Task : Genetics & Physiology Text Passage Explanation Task Holistic: understanding of key biology concepts, effective communication, and overall organization and structure (4-point scale) Content: specifically understanding of the target biology content, use of supportive evidence, and inclusion of prior knowledge – organization, structure and language features not considered (4-point scale) Language: abstraction, informational density, and technicality beyond overall structure of the essay (4-point scale)

7. 7/16 CRESST/UCLA Instructional Practice Indicator: Classroom Assignment Ratings Based on Previous CRESST Research (Achbacher, 1999, Matsumura, 2000)- effective teachers: 1) maintain high standards for student achievement, 2) hold clear goals for student learning, and 3) align their classroom tasks with instructional goals and assessment criteria Incorporation of classroom artifacts as data for instructional practice Adapted to Create Measures for Opportunity to Learn Language Skills – 12 dimensions of OTL related to classroom assignments Clarity of Content Goals: how clearly a teacher articulates the specific scientific skills and biology concepts students are to utilize and gain from completing the biology assignment Clarity of Literacy Goals: looked for literacy goals that were clear, detailed, and specific as to what literacy skills and processes students were to be engaged in while completing the assignment. Level of Literacy Challenge: the degree to which this assignment task provided students with the opportunity to engage meaningfully with biology text Classroom observations – subset 16 teachers

8. 8/16 CRESST/UCLA Method: Participants 1,167 students (grades 8 – 11): 26% ELL 14 high schools from 6 school districts 26 teachers Measures: Outcome - Integrated Learning Assessment: 8.7% of the total variability (Holistic), 4.7% of the total variability (Content), and 5.8% of the total variability (Language) Instructional Practice Indicator: Classroom Assignment Ratings CST-Science (Biology), CST-English Language Arts Teacher and Student Demographic Information (e.g. Years of Experience, ELL, Grade level, gender)

9. 9/16 CRESST/UCLA Research Question 1: Differentiation of Various Cognitive Demands Correlation: 1)Content knowledge and Language skills both Highly Correlated w/ Holistic Score 2)Language and Content slightly lower correlation 3)Slightly higher correlation with CST-Science than CST-ELA for all 3 ILA scores

10. 10/16 CRESST/UCLA Research Question 1: Differentiation Various Cognitive Demands Regression Analysis: Predicting Holistic Score using Content and Language Scores – 84% of the variance explained by the model Content and Language Equally attributed to the Holistic Score Language skills are important component

11. 11/16 CRESST/UCLA Research Question 2: Factors Associated w/ Student Outcome Logistic Hierarchical Model (HM) Level-1 Model Prob[R = 1|B] = P'(1) = P(1) Prob[R <= 2|B] = P'(2) = P(1) + P(2) Prob[R <= 3|B] = P'(3) = P(1) + P(2) + P(3) Prob[R <= 4|B] = 1.0 log[P'(1)/(1 - P'(1)] = B0 + B1*(FEMALE) + B2*(MINORITY) + B3*(EL2) + B4*(GRADE) + B5*(CSTSC) + B6*(CSTELA) + B7*(RCG) log[P'(2)/(1 - P'(2)] = B0 + B1*(FEMALE) + B2*(MINORITY) + B3*(EL2) + B4*(GRADE) + B5*(CSTSC) + B6*(CSTELA) + B7*(RCG) + d(2) log[P'(3)/(1 - P'(3)] = B0 + B1*(FEMALE) + B2*(MINORITY) + B3*(EL2) + B4*(GRADE) + B5*(CSTSC) + B6*(CSTELA) + B7*(RCG) + d(3) Level-2 Model B0 = G00 + G01*(EXP) + G02*(CG_GOALS)

12. 12/16 CRESST/UCLA Research Question 2: Factors Associated w/ Student Outcome

13. 13/16 CRESST/UCLA Summary and Conclusions 1)Explicit Evaluation of Language Skills 2)Language skills significant factor in Performance assessment – Equally contributing to the overall score 3)Develop explicit rubric for language rather than implicit – instructional and assessment implications 4)Explicit Literacy instruction (integrated approach) positively associated with student outcome

14. 14/16 CRESST/UCLA

15. 15/16 CRESST/UCLA

16. 16/16 CRESST/UCLA Christy Kim Boscardin: christyk@ucla.edu