1. KATIE BUCKLEY, HARVARD UNIVERSITY SCOTT MARION, CENTER FOR ASSESSMENT NATIONAL CONFERENCE ON STUDENT ASSESSMENT (NCSA) NATIONAL HARBOR, MD JUNE 22, 2013 Evaluating the Validity of Student Learning Objectives

2. Overview of Presentation 2 Theory of Action for SLOs Validity Argument for SLOs tied to ToA Research Agenda based on Validity Argument Buckley & Marion. CCSSO 2013

3. What is an SLO? Content- and grade/course-specific measurable learning objectives that can be used to document student learning by a teacher over a defined period of time Designed to involve teachers throughout the process and incentivize good teaching practices Can be used as both an accountability tool and as an instructional tool 3 Buckley & Marion. CCSSO 2013

4. Why do we care? Buckley & Marion. CCSSO 2013 4 SLOs currently used in teacher evaluation systems by for teachers in non-tested grades and subjects (NTSG) Upwards of 20 states/districts are using SLOs Nearly 70% of teachers teach in NTSG However Very little research currently exists Much evidence is needed to substantiate the implicit and explicit claims to support the use of SLOs in educator evaluations

5. Theory of Action as a Starting Point A theory of action (ToA) is a useful way to structure a validity evaluation in complex contexts… Researchers have suggested (and we believe) that a theory of action can*: Frame the validity evaluation, Serve as a useful starting point in creating a validity argument, Address the program evaluation and consequential aspects of the system not easily addressed through a validity argument *(Bennett, 2010; Marion & Pellegrino, 2006; Marion, 2010; Ryan, 2002). 5 Buckley & Marion. CCSSO 2013

6. The Challenges with Creating a ToA for SLOs SLOs are more than an assessment or program Student growth is embedded within the SLO process rather than measured externally It is important to convey how teachers should be involved in the SLO process SLOs are flexible and may be adapted to a variety of systems 6 Buckley & Marion. CCSSO 2013

7. Components of an SLO Buckley & Marion. CCSSO 2013 7 SLO is a framework that includes: Meaningful learning expectations (goals) High quality assessments Targets for student performance Targets for aggregate educator performance Instruction from teachers Therefore, an evaluation of SLOs must address all five components

8. Theory of Action for SLOs in a Teacher Evaluation System 8 Buckley & Marion. CCSSO 2013

9. Expanding the ToA with a Validity Argument We expand the ToA to create a validity argument based on Kanes 2006 work We explicate for each condition/input: Propositions Claims Example Evidence Propositions are organized according to Scoring Generalization Extrapolation Decision Consequence* 9 Buckley & Marion. CCSSO 2013

10. Input 1: Student learning goal is rigorous yet attainable Buckley & Marion. CCSSO 2013 10 PropositionClaimExample of Evidence Scoring Goals are of high quality Documentation of quality check on the rigor of the rubric Evidence that goals were evaluated consistently, reliably and accurately Generalization Learning goals reflect the relevant content standards and the associated curriculum of the course Review of the importance and alignment of established learning goals by content experts Extrapolation The learning goals are representative of the broader construct of student learning within the classroom Clear documentation on why the learning goal is an important one for students to learn, ways in which the learning goal requires students to demonstrate deep understanding of the big idea being measured, and ways in which the learning goal ties to overarching goals for the course. Decision Attainment of the learning goals can be used to classify teacher facilitation of student learning Evidence that higher scores on SLOs reflects greater student learning Evidence that higher scores on SLOs reflects more focused teaching of important content frameworks ConsequenceThe SLO does not lead to a narrowing of the curriculum There is no evidence to suggest that that instruction has narrowed to only the domains covered by the SLO

11. Input 2: Assessments accurately and reliably measure student learning of goals Buckley & Marion. CCSSO 2013 11 PropositionClaimExample of Evidence Scoring Student performance is accurately and consistently scored Rater accuracy and consistency studies are clearly documented Generalization Assessment items are representative of broader domain Reliability/generalizability analyses Content validity studies Extrapolation Assessments are/is tailored to match the specific learning goal Documentation of content validity studies between assessment content and learning goal Decision Performance on the assessment(s) can be used to classify student attainment of learning goal Evaluation of conditional standard errors of measurement around key cutscores Misclassification analyses ConsequenceThe SLO does not lead to undesirable behavior such as student cheating on the assessment SLO scores in the state correlate strongly with other measures of student achievement particularly from low stakes tests measuring similar content

12. Input 3: Targets set for students are appropriate and valid Buckley & Marion. CCSSO 2013 12 PropositionClaimExample of Evidence Scoring Determination of whether student meets target is accurately classified Business rules for classifying attainment of student targets are clearly defined. Evidence that student targets are fair and consistent across grades/subjects. Generalization Student attainment of target provides evidence of students ability on learning goal Evidence that student targets are set in a thoughtful manner based on baseline data and prior student performance Extrapolation Student attainment of target provides evidence of overall student achievement Student performance on SLO is strongly correlated with other measures of student performance Decision Student attainment of target can be used to determine the teacher score on the SLO Business rules for aggregating student targets to the teacher level are clearly defined. ConsequenceStudent attainment of the target is the result of student learning and not of undesirable behavior such as teaching to the test SLO scores in the state correlate strongly with other measures of student achievement particularly from low stakes tests measuring similar content

13. Input 4: Targets set for teachers are appropriate and valid Buckley & Marion. CCSSO 2013 13 PropositionClaimExample of Evidence Scoring Determination of whether teacher meets target is accurately classified Evidence that teacher targets are fair and consistent across grades/subjects. Final teacher SLO ratings are strongly correlated with the percent of students who meet their SLOs. Generalization Attainment of target provides evidence of teachers contribution to student learning in the classroom Evidence that teacher targets were chosen based on expert review of degree of student learning considered to be sufficient/exemplary for the teacher within specified time frame. Extrapolation Attainment of target provides evidence of overall teaching effectiveness Evidence that final teacher SLO rating correlates strongly with other measures of teacher effectiveness Decision The teacher or teachers being evaluated on the SLO is/are the one(s) primarily responsible for the students achievement Evidence that teacher developing and evaluated on the SLO is the primary teacher of record in the classroom. ConsequenceStudent attainment of SLO is not due outside factors beyond the teachers control and influence Evidence that student attainment of SLO is no more than weakly correlated with constructs that should be unrelated to SLO attainment.

14. Input5: SLO is adequately tied to teacher instruction Buckley & Marion. CCSSO 2013 14 PropositionClaimExample of Evidence Scoring Teachers appropriately connect standards to learning goals when setting SLOs Teachers document the curricula standards that are tied to the learning goals and which methods of teaching will be used to teach those standards Generalization Teachers have knowledge and skills to reflect on student data as a way to determine students mastery of the curriculum Evidence that educators are able to use assessment results to make data driven decisions regarding their instruction Extrapolation Teachers modify/alter instruction in direct response to data indicating students are off target from meeting their SLO Teachers indicate that the SLO process impacts their instructional plans throughout the school year. Decision Higher scores on SLOs reflects more focused teaching of important content frameworks Student attainment of SLO is related to expert ratings of teacher instructional practice ConsequenceTeacher mobility/attrition among highly effective teachers does not increase as a result of SLO implementation There is no evidence of teacher mobility/attrition among highly effective teachers

15. Prioritizing Studies Buckley & Marion. CCSSO 2013 15 It would be impossible to conduct studies for each claim listed, so how to prioritize? States chosen priorities Analyses that provide monitoring results for system-wide improvement Analyses that address multiple claims Need to also consider short-term vs. long-term analyses What can be done/should be done now based on available data vs. later when the system has produced more data?

16. Research Agenda Buckley & Marion. CCSSO 2013 16 Three Studies 2 based on quantitative data currently available 1 based on qualitative data not yet available Will help to determine whether: SLOs are being implemented as intended Results that are produced are as hypothesized Address multiple claims Analyses that can be used by GA to respond to USEDs requests

17. Research Agenda Buckley & Marion. CCSSO 2013 17 Study 1: Are there differences in the quality and attainment of SLOs: By district, grade, course, course-type? Across school and classroom demographics/characteristics? Over time? Claims addressed Goals are of high quality (Learning Goal input, Scoring proposition) Student attainment of SLO is not due outside factors beyond the teachers control and influence (Teacher Target input, Decision proposition )

18. Research Agenda Buckley & Marion. CCSSO 2013 18 Study 2: What is the relationship between attainment of SLOs and other measures of teacher effectiveness, including: Student growth? Teacher practice? Claims Addressed: Attainment of the learning goals can be used to classify teacher facilitation of student performance (Learning Goal Input, Decision proposition) Student attainment of target provides evidence of overall student achievement (Student Target input, Extrapolation proposition) Attainment of target provides evidence of overall teaching effectiveness (Teacher Target input, Extrapolation proposition) Higher scores on SLOs reflects more focused teaching of important content frameworks (Teacher Instruction input, Decision proposition)

19. Research Agenda Buckley & Marion. CCSSO 2013 19 Study 3: How do teachers and school leaders understand the SLO system and use the system for instructional purposes? Claims addressed: Teachers modify/alter instruction in direct response to data indicating students are off target from meeting their SLO (Teacher Instruction input, Generalization proposition) Higher scores on SLOs reflects more focused teaching of important content frameworks (Teacher Instruction input, Extrapolation proposition)

20. Thank you Buckley & Marion. CCSSO 2013 20 Questions/comments: Katie Buckley (buckley.kate@gmail.com)buckley.kate@gmail.com Scott Marion (smarion@nciea.org)smarion@nciea.org