Colorado Department of Education, Educator Effectiveness Implementing the Specialized Service Professional State Model Evaluation System Colorado Department of Education, Educator Effectiveness

How Do You Feel About SB 10-191? 12/2/2018 How Do You Feel About SB 10-191? Choose a picture from below that best reflects your feelings regarding SB 10-191. Why did you select the picture you did?

What’s on Your Mind? 3

Purposes of S.B. 10-191 A system to evaluate the effectiveness of licensed personnel and continually improve the quality of education and student outcomes. Provide meaningful feedback for professional growth and continuous improvement. Provide a basis for making decisions in the areas of hiring, compensation, promotion, assignment, professional development, earning and retaining non-probationary status, dismissal, and nonrenewal of contract.

Critical Effects of S.B. 10-191 Requires statewide minimum standards for what it means to be an _________ Specialized Service Professional Requires ______ evaluation of all Specialized Service Professionals Requires that all Specialized Service Professionals be evaluated at least ___% on the _________ of the students they support “effective” annual 50 outcomes

Priorities of Implementation Human judgment Data should inform decisions, but human judgment will always be a part of the process Processes and techniques are recommended to improve individual judgment and minimize errors and bias Embodiment of continuous improvement by monitoring Data from pilot and rollout intended to capture what works and what doesn’t Changes in practices and tools Emerging research and best practices

Priorities of Implementation Providing credible and meaningful feedback with: Actionable information Opportunities for improvement Idea that this is a process and not an event Involves all stakeholders in a collaborative process Families, teachers, specialized service professionals, administration, school board, etc. Specialized service professionals involved throughout development process

Priorities of Implementation Takes place within a larger, aligned and supportive system All components of the system must focus on increasing the number of specialized service professionals and students who are successful Turn and Talk Which priority resonates with you and why?

SSP Timeline of Implementation 12/2/2018 SSP Timeline of Implementation Year 1: 2012-13 Create 9 work groups comprised of experts from the field Work groups provide recommendations for definition, standards and elements (aligned with teacher) and measures of student outcomes to State Council for Educator Effectiveness SCEE presents recommendations to State Board of Education CDE facilitates work group creation of draft professional practice rubrics Release application for pilot districts/BOCES; Select pilot sites 19 pilot sites including: 4 BOCES Large suburban/urban districts Small rural districts Online school

SSP Timeline of Implementation 12/2/2018 SSP Timeline of Implementation Year 2: 2013-14 All SSPs evaluated using current system employed in district/BOCES Pilot draft professional practice rubrics for Specialized Service Professionals Collect feedback on professional practice rubrics and measures of student outcomes Refine rubrics and measures of student outcome based on feedback collected Provide training to districts and BOCES wanting to use the State Model System for SSPs

SSP Timeline of Implementation 12/2/2018 SSP Timeline of Implementation Year 3: 2014-15 Statewide implementation of all SSP Standards and Elements, including measures of student outcomes Conduct a validation study for professional practice rubrics using pilot sites If able to earn non-probationary status, this year considered “hold harmless” Year 4: 2015-16 Continued statewide implementation of SSP Standards and Elements, including measures of student outcomes First year an ineffective or partially effective rating would count toward two consecutive years of ineffectiveness for those able to earn non-probationary status

Prior to the beginning of Spring Semester 1. Training 2. Annual Orientation 9. Goal-Setting and Performance Planning 3. Self-Assessment Evaluation Process for Specialized Service Professionals End of May Mid-June May 15 End of September. Prior to the beginning of Spring Semester Train: Prior to the beginning of School. Orient: Within the first week of School. End of June End of September. 8. Final Ratings 4. Review of Annual Goals and Performance Plan 7. End-of-Year Review 5. Mid-Year Review 6. Evaluator Assessment

Building Your Evaluation System 12/2/2018 Building Your Evaluation System Now that you are familiar with the evaluation process, think about how this will work in your setting. Who will serve as evaluators? Will you have more than one person contributing to the evaluation results? How many observations will you conduct? What will those observations look like? How will you handle observing itinerant staff? How can an evaluator provide a meaningful evaluation to an SSP who’s area of expertise they may not share? How can an SSP support an evaluator in gaining knowledge of their expertise in order to provide a meaningful evaluation?

Definition of Specialized Service Professional Effectiveness Effective specialized service professionals in the state of Colorado are vital members of the education team and have the knowledge and skills necessary to ensure that diverse student populations have equitable access to academic instruction and participation in school-related activities. Effective specialized service professionals develop and/or implement evidence-based services or specially designed instruction to meet the unique needs of their students. They support growth and development to close achievement gaps and prepare students for postsecondary and workforce success. They have a deep understanding of the interconnectedness of the home, school and community and collaborate with all members of the education team to strengthen those connections. Through reflection, advocacy, and leadership, they enhance the outcomes and development of their students.

Framework for System to Evaluate Specialized Service Professionals Colorado Department of Education Framework for System to Evaluate Specialized Service Professionals Definition of Specialized Service Professional Effectiveness Quality Standards I. Professional Expertise II. Learning Environment III. High Quality Delivery IV. Reflect on Practice V. Leadership VI. Student Outcomes 50% Professional Practice Standards 50% Measures of Student Outcomes Observations of Professional Practice Expert Input Other Measures Aligned with CDE Guidance Weighting: How Much Does Each Standard Count Towards Overall Performance? Match of Measures of Student Learning to Assigned Duties Weighting: State Scoring Framework: How Do Measures of Quality Standards Result in a Determination of Individual Performance? Effectiveness Ratings Ineffective Partially Effective Effective Highly Effective Appeals Process* Applies when professionals are not at-will employees

State Model Rubric Basics Cumulative in content Each level of the rubric represents an increase in the quality, intensity, consistency, breadth, depth, and complexity of practice Effectiveness marked by the addition of practices that improve the overall performance of the specialized service professional and drives to student/other stakeholder outcomes Standards based Outlines the practices that you must meet to be at standard

Rubric Structure and Rating Level Focus The focus of the Basic rating is on the foundational elements of service provision. The SSP rated as Basic is typically performing at a foundational level and does not meet state quality standards. Every SSP is expected to perform Basic professional practices in their day-to-day work. The focus of Partially Proficient and Proficient levels is what SSP do on a day-to-day basis to achieve state performance standards and assure that students are achieving at expected levels. The focus of Accomplished and Exemplary ratings shifts to the outcomes of the SSP’s practices, including expectations for staff, students, parents and community members, as a result of practices exhibited under Basic, Partially Proficient and Proficient rating levels.

Professional Practices SSP Quality Standards Performance Rating Levels Elements of the Standard Elements of the Standard Professional Practices

Scoring the Rubric Determining the professional practices rating is a three-step process that involves rating the individual elements and standards and using those to determine the overall rating on professional practices. Rating the Elements Rating the Standards Determining the Overall Professional Practices Rating

Understanding the Scoring “Business” Rule Starting in the Basic rating level and moving across, look for the first unchecked professional practice. Move one column back to identify the rating for the element.

Determining the Element Rating Starting in the Basic rating level and moving across, look for the first unchecked professional practice. Move one column back to identify the rating for the element.

Determining the Element Rating Starting in the Basic rating level and moving across, look for the first unchecked professional practice. Move one column back to identify the rating for the element.

Rubric Rating Level Points Standard Basic Partially Proficient Proficient Accomplished Exemplary Element Professional Practices SSP’s performance on professional practices is significantly below the state quality standard. 1 SSP’s performance on professional practices is below the state quality standard. 2 SSP’s performance on professional practices meets state quality standard. 3 SSP’s performance on professional practices exceeds state quality standard. 4 SSP’s performance on professional practices significantly exceeds state quality standard.

12/2/2018 3 2 1 2 2 10 Proficient

Determining Overall Professional Practice Rating 12/2/2018 Determining Overall Professional Practice Rating First, determine each Standard’s contribution to the overall professional practice rating by using the following formula: (Total Pts. Earned for Std.) X (Std. Weight X No. of Stds.) (Number of Elements Associated with Standard) Example: 10 X (.20 X 5) 5 = 2 Once you’ve completed this step for every Standard, you are then able to determine the overall professional practice rating.

Determining Overall Professional Practice Rating 12/2/2018

12/2/2018 Determining Measures of Student Outcomes for Specialized Service Professionals

Framework for System to Evaluate Specialized Service Professionals Colorado Department of Education Framework for System to Evaluate Specialized Service Professionals Definition of Specialized Service Professional Effectiveness Quality Standards I. Professional Expertise II. Learning Environment III. High Quality Delivery IV. Reflect on Practice V. Leadership VI. Student Outcomes 50% Professional Practice Standards 50% Measures of Student Outcomes Observations of Professional Practice Expert Input Other Measures Aligned with CDE Guidance Weighting: How Much Does Each Standard Count Towards Overall Performance? Match of Measures of Student Learning to Assigned Duties Weighting: State Scoring Framework: How Do Measures of Quality Standards Result in a Determination of Individual Performance? Effectiveness Ratings Ineffective Partially Effective Effective Highly Effective Appeals Process* Applies when professionals are not at-will employees

Guardrails for Evaluation System Building 12/2/2018 Guardrails for Evaluation System Building Requirement language from State Board Rule: At least fifty percent of the evaluation shall be based on at least two measures of student outcomes, which measures shall be aligned with the role and duties and the individual SSP being evaluated

Guardrails for Evaluation System Building 12/2/2018 Guardrails for Evaluation System Building Requirement language from State Board Rule: Data used in evaluating SSPs shall be collected from the sites, or a representative sample of the sites, at which the SSP provides services.

12/2/2018 How Do We Get There? Individual SSP Decisions Individual SSP and evaluator will make decisions unique to their professional responsibilities. Weights and measures will vary for every SSP in the district/BOCES. What process will work best for your school, district or BOCES? There is no right or wrong answer to this question! What process will work best given your context, what is feasible, meaningful and valued? SSP Category/Group Decisions Each SSP category will make general decisions about weights of measures, however actual outcome measures may differ. Each SSP category may have different percentages. All SSPs Decisions All SSPs in the district/BOCES will have the same weights or measures included in their evaluation. The weights will remain consistent, however the measures will vary based on the SSP.

Steps for Identifying and Determining Student Outcome Measures 12/2/2018 Steps for Identifying and Determining Student Outcome Measures Step 1: Determine SSP role and responsibilities Step 2: With input from SSP, select and preliminarily weight appropriate outcome measures Step 3: Set criteria for each rating category Step 4: Score results from multiple student outcomes at the end of the year Step 5: Combine weighted scores into a “student outcomes” rating Decision Framework

Student Learning Objective Process: Tools and Resources 12/2/2018 Student Learning Objective Process: Tools and Resources You can use the Excel Tool to record the targets and scales you have established.

Student Learning Objective Process 12/2/2018 Student Learning Objective Process What is the Student Learning Objective Process? Enables specialized service professionals to establish outcomes for individual or groups of students, monitor students’ progress toward those outcomes and evaluate the degree to which students achieve those outcomes using relevant, meaningful measures In evaluation systems, this process serves as a method of measuring student outcomes

Student Learning Objective Process: Strategies for Target Setting 12/2/2018 Student Learning Objective Process: Strategies for Target Setting Possible strategies to consider when setting targets for students: Proficiency data Targets – students meeting grade level expectations Examples: a % of students will pass an exam; a % of students will achieve a specific score or better on a measure Growth data Targets – students growing over the course of instruction or service delivery Examples: a % of students will progress one level; a % of students will make projected growth or better Averaging data Targets – students’ average score on a measure Examples: Students will answer, on average, 80% of the questions correctly; students will achieve a 75% attendance rate

Student Learning Objective Process: Strategies for Target Setting 12/2/2018 Student Learning Objective Process: Strategies for Target Setting Possible strategies to consider when determining how students will be captured in target setting: Individual – SSP sets individual goals for students and then determines how many students met their individual goals Whole Class –SSP sets a goal that would be applied to all students in a class Subgroup of Students –SSP sets a goal for a group(s) of students with similar baseline data levels Case Load – SSP sets a goal for the specific students he or she supports Some of these strategies may work better with some SSP assignments. Can you think of which SSP assignments might work best with which strategy based on your local context?

Student Learning Objective Process: Strategies for Scale Setting 12/2/2018 Student Learning Objective Process: Strategies for Scale Setting SSPs set ranges (how many students are expected to meet the set target) across 4 rating levels: more than expected, expected, less than expected, and much less than expected SSPs collaboratively set ranges with their evaluator Ranges will vary from year to year and possibly class to class or case load to case load as baseline data will vary due to student beginning points Collaboratively setting ranges will allow SSPs and evaluators to calibrate about expected student outcomes

Contact Us EE Leadership Amy Skinner: Communications Skinner_A@cde.state.co.us Katie Lams: Communications Lams_K@cde.state.co.us Britt Wilkenfeld: Data Fellow Wilkenfeld_B@cde.state.co.us EE Leadership Katy Anthes: Executive Director Anthes_K@cde.state.co.us Toby King: Director Evaluation and Support King_T@cde.state.co.us Jean Williams: Evaluation Design Specialist Williams_J@cde.state.co.us Tricia Majors: Project Mgr. Majors_T@cde.state.co.us Disclaimer: The contents of this PowerPoint were developed under a grant from the Department of Education. However, those contents do not necessarily represent the policy of the Department of Education, and you should not assume endorsement by the Federal Government.

Contact Us Implementation Support and Development Dee Blecha 12/2/2018 Contact Us Implementation Support and Development Dee Blecha Blecha_D@cde.state.co.us Mary Bivens Bivens_M@cde.state.co.us Courtney Cabrera Cabrera_C@cde.state.co.us Paul Jebe Jebe_P@cde.state.co.us Sed Keller Keller_S@cde.state.co.us Dawn Pare Pare_D@cde.state.co.us Joslyn Robich Robich_Joslyn2@cde.state.co.us Chris Vance Vance_C@cde.state.co.us