1. Evaluation Process/Requirements for CAPP Algebra Project

2. Presentation Outline Introduction Evaluation  Goals  Process Outcome Measures Potential Research Questions Data Collection Questions & Discussion

3. Evaluation Goals  Examine the process of decision-making for how to allocate available resources associated with the grant (including personnel)  Who is involved, what type of constraints are there?  Describe how formative evaluation can be used in the schools  Connections between formative assessment results and changes in pedagogy or curriculum Process  Algebra Program participants will be active in helping to define the process of evaluation, key questions  Local knowledge and priorities will be critical to the evaluation design  Unclear how a single policy or approach can be supported statewide given the broad differences in skills, priorities, and school contexts

4. Evaluation Outcome Measures Process  Changes in teacher’s understanding and use of formative assessment in the classroom  Changes in teacher’s self-efficacy and engagement with the assessment process  Student perceptions of the usefulness of formative assessments in their own learning Outcomes  Measure student outcomes in mathematical knowledge  State examinations, district benchmarks, teacher perceptions  Student college-readiness in mathematics  Possibly ACT Explore/Plan These examinations are standardized measures of college readiness and are scaled with each other and the ACT; they could be used to look for “growth”

5. Potential Research Questions Generate findings that will inform the broader research community about the process of integrating formative assessments into schools in meaningful ways.  RQ 1: In what ways will teachers use the capabilities of formative assessments to inform changes in pedagogy or curriculum?  RQ 2: What conditions are critical to successfully support integration of formative assessment into daily/weekly practice?  RQ 3: Can successful integration of mathematics formative assessment into practice improve student outcomes in mathematics?

6. Research Question 1 In what ways will teachers use the capabilities of formative assessments to inform changes in pedagogy or curriculum?  We will create logic models for the functioning of each of the school programs with respect to the intended outcomes  Logic models are explicit statements of how specific aspects of program design will lead to specific program outcomes – for example, how do grade-level meetings on formative assessment outcomes lead to better outcomes for students in mathematics?  The clear descriptions provided by the program logic models will allow the evaluation to examine not only if outcomes are achieved, but how they are achieved (or not) Will allow for good formative data to assist in program improvement  Additional logic models will be created for school cluster (1 HS and 2 middle schools) plans/objectives.

7. Research Question 2 What conditions are critical to successfully support integration of formative assessment into daily/weekly practice?  By understanding the conditions that allow for success across various contexts this evaluation can support the likely implementation of future state or district policies that have formative assessment as a critical component

8. Research Question 3 Can successful integration of mathematics formative assessment into practice improve student outcomes in mathematics?  As much as possible we would like to tie the particular use of formative assessments to improvement in student performance on a wide array of student assessments  Classroom, district, state, and national assessments  Student surveys about engagement/self-efficacy in mathematics  Where Algebra Project participants are willing to introduce design features around implementation (which will allow for a connection between the program and student achievement), we will work with schools to get good estimates of the impact of program implementation  Staggered implementation of intervention, construction of strong comparison groups, roll-out across other schools in districts over time

9. Data Collection Sources/Modes District Data Responsibilities  Student Administrative Data  Administrative data include data that are typically collected by districts at the student level (achievement, grades, attendance, parental education, etc.)  Must have access to these data for comparison School Data Responsibilities  Participants in Project must be willing to participate in surveys, interviews, and classroom observations by researchers  Interviews (1 or 2 per year – 45 minutes per interview)  Survey (1 or 2 short surveys – 15 minutes per survey)  Observations (1-4 classroom observations per year)

10. Data Collection Sources/Modes (cont.)  Portfolio Assembly  The nature of the portfolio will be discussed with school teams during the planning year. Options include student-level portfolios, or Project-level portfolios or both. Student-level – track changes in student performance on clearly identified areas of importance Project-level – track changes and components of the project as curriculum or pedagogical changes are introduced into the classrooms  Student Responses  Student surveys  Student focus groups

11. Potential Spillovers Training a group of teachers to use formative assessments may raise the possibility of formative assessments being used school-wide Integration of standards and concentration on particular methods or difficulties that students have in mathematics across grades may ease transition into the new curriculum based on the common core standards

12. Evaluation Footprint Our goal is to limit the intrusion of the evaluation in the school  Logic models will be created by us through discussions with Project groups in the school and then shared with the school for confirmation and modification It is our goal to try to incorporate and support Project participant’s ideas for evaluating particular aspects of their program

13. Thank You for Your Time Questions?