Designing Student Growth Measures for CTE January 22, 2013 Designing Student Growth Measures for CTE

OTES Provides for Multiple Evaluation Factors

Who should be evaluated? …Any person who is employed under a teacher license issued under this chapter, or under a professional or permanent teacher’s certificate issued under former section 3319.222 of the Revised Code, and who spends at least fifty per cent of the time employed providing student instruction. Would a guidance counselor with a teaching license be evaluated under OTES? No. Would a speech therapist who holds a teaching license be required to be evaluated? No, if only performing speech duties. Would an intervention specialist who works as an inclusion teacher be evaluated? Yes. ORC 3319.111

HB 555 Changes Category A Under the Student Growth Measures, teachers are divided into three categories: Category A: Teachers for whom teacher-level value-added data is available Category B: Teachers for whom data from an assessment that is on the ODE Vendor Approved List is available Category C: Teachers for whom no teacher-level value-added or vendor approved assessment data is available LEAs can choose to implement LEA measures for Category A and Category B teachers. If LEAs choose to implement multiple measures of student growth, they must allocate a specific percentage weight to the measures as shown in the graphic below.

Category A is now divided into A1 and A2

Timeline for Implementation 2013-2014 school year with the following exceptions: TIF/SIG/RttT schools according to their grant timelines or scope of work If bargaining agreement was entered into prior to 9/24/12, immediately upon expiration of agreement

Suggested Implementation Timeline

Definition of Student Growth For the purpose of use in Ohio’s evaluation systems, student growth is defined as the change in student achievement for an individual student between two or more points in time. Measuring growth rather than achievement is more fair to teachers and principals whose students start the school year already behind and below grade level. Excerpted from Measuring Student Growth for Teachers in Non-Tested Grades and Subjects: A Primer

3 Potential Measures for Student Growth Depending on LEA decisions and the availability of data: Value-added Approved vendor-created assessments LEA determined SGMs using SLOs The challenge for measuring student growth is that there is not a single assessment that can be used for all teachers.

3 Measures of Student Growth Value-Added Approved Vendor Assessments LEA Measures EVAAS Reports 4-8 MRM/URM Extended Reporting w/Terra Nova and ACT EOC exams Approved ODE List Vendor had to show a metric for student growth List is fluid Student Learning Objectives (SLOs) Shared Attribution (Other) Vendor Assessments Diagnostic Reliable Valid Fair Rigorous Attributable Comprehensive 3 Measures of Student Growth

1. Teacher Value-Added MUST use if available 10-50% if applicable for 2012-2013 Category A1 >25% effective July 2013 Category A1: full 50% effective July 2014 Category A2-proportionate EVAAS Value-Added metric, aggregated across subject areas 1-year report; or 2- or 3-year rolling average, based on availability Phased-in implementation of reading & mathematics, Grades 4-8 Extended Reporting (other grades & subjects) being piloted In Ohio, Value‐Added refers to the EVAAS Value‐Added methodology, provided by SAS, Inc. This is distinct from the more generic use of the term “value‐ added,” which can represent a variety of statistical modeling techniques. The Ohio EVAAS Value‐Added measure of student progress at the district and school level has been a component of the Ohio Accountability System for several years. Value‐Added calculations currently utilize data from the Ohio Achievement Assessments (OAA). As the new Partnership for Assessment of Readiness for College and Careers (PARCC) assessments become operational and replace the current assessment system, they will be integrated into the Value‐Added calculations. Additionally, the EVAAS data reporting system has added several features to help educators use this important data. Battelle for Kids (BFK) is providing professional development and other related services across the state.

2. Approved Vendor Assessments From ODE-Approved List Vendors demonstrate how assessment can measure growth MUST use for Category B Teachers 10-50% if applicable and no Value-Added data available Category A1-LEA decision in SY2013-2014, not applicable for 2014-2015 Category A2-LEA decision HB 153 requires ODE to develop a list of student assessments that measure mastery of the course content for the appropriate grade level, which may include nationally normed standardized assessments, industry certification examinations, or end‐of course examinations for grade levels and subjects for which the Value‐Added measure does not apply (the non‐tested grades). ODE released a Request for Qualifications (RFQ) so interested vendors could demonstrate that their assessments qualified for use in Ohio schools. The list of approved assessments will be maintained and updated by ODE.

3. LEA-Determined Measures MAY use: LEA decision for Category A and B 0-40% if used in combination with Category A or B measures 24% max for Category A1 in SY2013-2014 0% for Category A1 in 2014-2015 and thereafter Proportionate to Category A2 teacher’s schedule MUST use if Category C Teacher 50% if no Category A or B data available For subjects in which traditional assessments are not an option – such as art or music – LEAs should establish a process to create student learning objectives (SLOs) to measure student progress in those courses.

3. LEA-Determined Measures Three types of LEA-Determined Measures Student Learning Objectives (SLOs) Shared Attribution Approved Vendor assessments Category A2 teachers can use local measures after VA is applied proportionately. Category A1 teachers can use local measures in 2013-2014 only (24% max). Student Learning Objectives process for using measures that are specific to relevant subject matter. Measures must be district-approved and may include: Locally developed assessments; Pre-Post assessments; Performance-based assessments Portfolios. Shared Attribution Measures are student growth measures that can be attributed to a group. These measures encourage collaborative goals and may be used as data in the student growth component. Shared attribution may include: Building or District Value-Added is recommended if available; Building teams (such as content area) may utilize a composite VA score; Building- or District-based SLOs Vendor Assessment as a Local Measure: Teacher Category A (with Value‐Added) also may use Vendor assessments as an LEA‐determined measure if using both.

What is an SLO? A measurable, long‐term academic goal informed by available data that a teacher or teacher team sets at the beginning of the year for all students or for subgroups of students. A measure that demonstrates a teacher’s impact on student learning within a given interval of instruction.

On Educator Evaluation page, click SGMs section. SGM Resources On Educator Evaluation page, click SGMs section. Scroll down on the ODE home page and click “Educator Evaluation”. The page will display next. ODE’s Educator Evaluation webpage with link to SGMs.

SLO Resources Introduction to SLOs Student Learning Objective Guidebook Student Growth Measures for Teachers SLO Writing Template SLO Checklist for Writing & Approving Sample SLOs Snapshot of ODE’s SGM webpage with resources.

Reinforce Best Teaching Practice Through Setting goals for students Using data to assess student progress Adjusting instruction based upon progress

Why is Ohio Using SLOs? Reinforce promising teaching practices Can be used in all subjects/content areas Are adaptable Provide teachers some ownership on how they are evaluated These qualities encompass good teaching practices. Educators using best practices already follow an informal SLO process! SLOs help formalize good teaching by requiring each of these steps and using the growth targets to inform evaluation results.

Potential For Collaboration SLOs can be used to promote collaboration and reflection on practice among teachers.

Measures for SLOs SLOs can be created drawing on different data sources: teacher created assessments performance assessments rubric‐based assessments business & industry certification state or national assessments SLOs are not dependent upon the availability of standardized assessment scores. Instead, SLOs can draw upon different data sources such as end of course exams, performance-based assessment scored by a rubric, or district-created or team-created assessments. SLOs can be highly adaptable, quickly reflecting changes in curriculum and available assessments.

Do all Teachers Need to Write SLOs? ODE recommends that all teachers create at least one SLO this year to gain experience with the SLO process. In subsequent years, teachers should create two to four SLOs per year. Category A1 will not have the option to use local measures beginning July 2014. LEAs can determine which measures teachers will use for the student growth measure. 2-4 SLOs per teacher representative of the teacher’s schedule. We felt like we needed a couple of high school teachers’ schedule as an example here. The group got sidetracked asking how many SLOs a teacher needs. A couple examples would be helpful. Explain each situation and district is unique and move on.

The SLO Development Process LEAs have some flexibility to shape the process to fit local contexts, but ODE recommends the following steps: 1. Review baseline data; 2. Create SLOs; 3. Obtain SLO approval, per local process; ODE recommends teams review and approve; 4. Monitor progress toward attainment of SLO growth targets; 5. Review evidence and evaluate progress towards and attainment of SLO growth targets. ODE also recommends that building-level committees review and approve all of the SLOs so that they develop an understanding of the type and rigor of the SLOs across a school. Please note, however, that LEAs are not required to use the committees and can choose their own evaluator(s) for the SLO process. By centralizing the approval process, the committee will be able to support those teachers who set targets that are too high or too low, ensuring consistency within the building. Also, the committee will ensure that all SLOs are aligned to the academic standards as well as school and district priorities.

High-Quality SLOs Include: Baseline & Trend Data Student Population Interval of Instruction Standards & Content Assessment(s) Growth Target(s) Rationale for Growth Target(s) These are reflected on the SLO writing template.

Student Learning Objective Template

SLO Template Checklist Ensure you keep this template beside you while writing SLOs. If you can check off each box (a couple could be N/A), then your SLO will likely be approved and you will not to revise it. In order to approve an SLO, all checkboxes, save a couple that could be marked N/A, should be checked. If not, send back for revisions and allow approximately 10 days for resubmission.

SLO Approval SLOs are approved at the local level. ODE recommends an existing committee. Provide feedback: both cool & warm. Recommend allowing 10 days for revisions.

SLO Approval Process Committees should go through a calibration process. Read over the entire SLO first. Using the checklist, review the SLO. Discuss whether it meets each criterion and provide feedback to the teacher. Develop a plan for tracking SLOs returned for revisions.

High-quality SLOs include the following criteria: Baseline and Trend Data Student Population Interval of Instruction Standards and Content Assessment(s) Growth Target(s) Rationale for Growth Target(s) 30

Baseline and Trend Data: Identifies source(s) and summarizes student information (test score from previous years, results of pre-assessments) in numerical and narrative form. Reviews trend data to inform the objective and establish the amount of growth that should take place. Identifies student strengths and weaknesses. There are 7 components of an SLO. We will now review each component. Helps the teacher better understand how prepared their students are for the standards addressed during the year/course. Could include: End-of-year data from the previous year Baseline data from a district assessment Pretest Student Work Samples Conduct a detailed analysis of the student data.

Student Population: Includes all students in the class. Describes the student population (number, course, grade level, etc.). Considers any contextual factors that may impact growth. Does not exclude subgroups of students that may have difficulty meeting targets. Provide details. Should cover as many students as possible. If any students are excluded, teacher should acknowledge this and explain why. Students covered under an SLO must be proportional and representative of the teacher’s schedule. For example, if a science teacher teaches four sections of biology and two sections of earth science, two SLOs might cover the biology classes and one might apply to the earth science classes. Note any other important characteristics of the student population. For example, any special needs, ELL, TAG? How representative is this group of students to all of the students the teacher typically instructs?

Interval of Instruction: The duration of the course that the SLO will cover including the start and end dates. Include how frequently the course meets and length of a class period. Include any other relevant information that could impact student growth. Does the class meet daily? Is it an academic year or a semester course?

Standards and Content: Specify which standards the SLO covers. Broad enough to represent the most important learning or overarching skills, but narrow enough to be measured. What content do the students need in order to be successful next year or in the next course? Explain why. Is this a targeted SLO? The content or skill area should represent the essential learning and big ideas/domains of the course content such as key skills or overarching content, and should be selected based upon the identified areas from the data analysis.

Assessment(s): Identify the assessment. Who created/ reviewed it? Describe its structure. Describe how the assessment provides “stretch” for low and high achieving. Identify supplemental assessments. Provide specific details on how multiple tests will be combined into a summative score. Follow assessment guidelines. The content or skill area should represent the essential learning and big ideas/domains of the course content such as key skills or overarching content, and should be selected based upon the identified areas from the data analysis.

Identifying Appropriate Assessments

Selecting Assessments for SLOs Selecting and approving assessments can be one of the most challenging and important steps of the SLO process. Assessments enable teachers to determine growth toward and attainment of the SLO. ODE highly recommends that educators consult the Guidance on Selecting Assessments. This can be challenging, but it is one of the more important steps of the SLO process.

Criteria for Selecting Assessments Was the assessment(s) reviewed by content/industry experts? Is the assessment aligned to both the SLO and the standards? Does the assessment have enough stretch? Is the assessment valid & reliable? Ensure assessments have sufficient “stretch” for both low- and high-achieving students. Assessment contains questions that are of varying difficulty and cover some prerequisite and advanced knowledge or skills. Teachers may not be able to make an informed judgment about the needed stretch of the assessment until they have analyzed the baseline performance. When identifying assessments, educators should keep in mind that assessments must: Be aligned to national or state standards and to the SLO growth target (it measures the skills or content addressed by the SLO). Be reliable, meaning that the assessment produces accurate and consistent results. Be a valid measure, meaning that the assessment measures what it is designed to measure. Be realistic in terms of the time required for administration. The assessment should be, at a minimum, reviewed at the district level by grade-level and/or content-level experts.

Selecting Assessments for SLOs ODE strongly recommends LEAs not allow assessments created by one teacher for use in his/her classroom. If a teacher must create an assessment that is unique to his or her classroom, ODE strongly recommends that the teacher develop the assessment with: a school or district administrator with expertise in assessment, a special educator, an English language learner (ELL) specialist, and/or a content team member. At a minimum, the assessment should be reviewed at the district level by content experts such as grade-level or subject-level content experts.

Growth Target(s) Growth targets should be developmentally appropriate, rigorous, and attainable. Ensure all students have a growth target (not necessarily a passing score). Refer to baseline/pretest data and course requirements. Provide tiered targets to encompass all learners. *Rank order pretest scores and look for breaks in data. Because SLOs are flexible, they allow teachers to create growth targets that are appropriate to the teacher’s individual class, grade, or subject. In the beginning years of SLO implementation, knowing how to set rigorous yet realistic targets may be a challenge for teachers. Growth targets should include specific indicators of growth; such as percentages or questions answered correctly that demonstrate an increase in learning between two points in time. Recommended: The target can be tiered for specific students in the classroom to allow all students to demonstrate growth or the target can be equally applicable to all students in a class, grade, or subject. Using tiered targets for students allows teachers to identify where each student begins the course and to determine the appropriate amount of growth for the student based on their baseline data. Should be rigorous, yet attainable, as determined by the baseline or pretest data. Acceptable growth targets allow the teacher to demonstrate growth for all students while the unacceptable SLOs focus solely on student mastery.

Acceptable & Unacceptable Growth Targets First Unacceptable Target: (This is unacceptable because it does not show growth.) Achievement based for 80% of the students. Second Unacceptable Target: (This is unacceptable for two reasons: (1) students that are scoring at 50 need to make greater gains than only 10 points to reach an acceptable level of growth; (2) teachers must aim to grow all students, so those students scoring high on the pre-assessment need to be challenged with a higher goal and likely an additional assessment to illustrate their growth.)

Rationale for Growth Target(s) Rationale ties it ALL together. Explain how you used the data to establish the targets. Refer to students strengths/weaknesses based upon data. Explain why this content is the most important. Rationales should be aligned to broader school and district goals. *Connect pre-test data + tiered targets = expectations (show connections & accountability) High-quality SLOs include strong justifications for why the growth target is appropriate and achievable for this group of students. The rationale should be a precise and concise statement that describes the student needs and refers to the evidence (data) that informed the creation of targets. When applicable, rationales should also reference school and district goals or priorities.

QUESTIONS?

Regional Student Growth Measure Specialists: Support Carolyn Everidge-Frey, Assistant Director Office of Educator Equity and Talent 614-644-7446 Carolyn.Everidge-Frey@education.ohio.gov SGM@education.ohio.gov Regional Student Growth Measure Specialists: Chad Rice SE Region chad.rice@mvesc.org Mark Robinson NE Region m.robinson@mahoningesc.org Donna Huber Central huber.donna@moesc.net Apryl Ealy NW Region aealy@auglaizeesc.org Katrina Wagoner SW Region katrina.wagoner@hcesc.org

SLO SCORING TEMPLATE

Percentage of students that met or exceeded growth target SLO Scoring matrix Percentage of students that met or exceeded growth target Descriptive rating Numerical rating 90-100 Most Effective 5 80-89 Above Average 4 70-79 Average 3 60-69 Approaching Average 2 59 or less Least Effective 1

Final SLO Rating

Student Growth Measures Evaluation Matrix Teacher Performance 4 3 2 1 Student Growth Measures Above Accomplished Proficient Developing Expected Below Ineffective This is the lookup table.

Research and Best Practices Updated as research and best practices emerge Ohio Education Research Center (OERC)