NC Observation Calibration Process 2015 Update & Implementation Plan North Carolina State Board of Education Kimberly Simmons, North Carolina Educator Evaluation System Consultant Cynthia Martin, Ed.D. Service Support Lead

Observation Calibration Training Observation Calibration Training l Observation Engine Process for Educators l Content Online Resources for Educators Resource Sharing Strengthening PLCs Pilot Results 2015 Implementation Plan l 2016 Timeline

What is Observation Calibration Training? Observation Calibration Training (OCT) is an online evaluator training platform combining BloomBoard’s PD resources with Empirical Education’s observer training and calibration tool, Observation Engine. Goals of the OCT: Increase observation skills resulting in stronger scoring calibration Increase rater agreement Provide a common experience for LEAs to host collaborative conversations to improve instructional leadership skills

What is Observation Engine? Online training tool that uses classroom videos to ensure accurate and reliable classroom observations Videos from the MET (Measures of Effective Teaching) library have been master-scored on the NCEES teacher evaluation rubric to create learning opportunities for teacher evaluators in North Carolina. Master scoring including Educator Effectiveness staff working with Empirical Education staff.

Observation Engine NCEES Content 2 Observer Calibration Events – Scoring Studies Rating video practice to increase rater agreement to target scores Scoring Study 1 (video) is administered as an initial assessment of calibration Scoring Study 2 (video) is administered at the end of the year as a post- assessment 19 Targeted Learning Exercises - Lessons Targeted, self-paced online exercises with immediate on-screen feedback. The OCT contains two types of lessons: 2 Full classroom observations 17 Element-specific classroom observations

BloomBoard offers easy access to a rich Marketplace of thousands of learning resources to help educators improve their practice. Find courses, eBooks, articles, and videos from the most trusted content providers, all accessible from a single destination.

Portal to Observation Engine BloomBoard Marketplace is accessible through NCEdCloud using IAM through the Educator Evaluation Professional Development System. 

Portal to Observation Engine Easily access Observation Engine activities via the Marketplace, and use professional learning resources to support and enhance observer calibration

Resources, Personalized for Each Educator Support teachers’ different learning needs, just like students, by using Marketplace content that is searchable by topic, grade level, and more. Support teacher growth with personalized learning pathways aligned to both individual goals and strategic initiatives defined by your school or LEA.

BloomBoard Makes Sharing Resources Easy Share your favorite resources with a colleague, teachers at your site, a PLC, or through your social networks.

High-Quality Content from Trusted Partners BloomBoard works with the most trusted names in education to ensure the Marketplace has valuable, up-to-date resources. Each resource is vetted by a team of experts, and is further endorsed through ratings and reviews provided by educators in the BloomBoard community.

2014-15 Observation Calibration Training (OCT) Pilot 27 LEAs and 5 charter schools registered to participate in the OCT 138 administrators from 20 LEAs participated in the OCT pilot Participants’ scoring improved over the course of the pilot Utilizing OCT practice resources improved scoring, regardless of initial performance level Collaboration is key to NCEES calibration OCT Flyer Link to all 2014/15 pilot reports May want to remove the first bullet, because the information may not be important to SBE

Scoring Accuracy Improved T tests revealed that in Scoring Sstudy 2, percent target agreement was higher (p<.05) and percent discrepant was lower (p<.001) than in Scoring Study 1 Key metrics: Percent Target Agreement: the percent of scores that agree exactly with the target scores Percent Target Discrepant: the percent of scores that disagree with the target scores by 2 or more performance levels (e.g. when the target score is 2 and the score provided is a 4)

Completing OCT Lessons Improved Scoring Regression analysis revealed that the number of lessons completed significantly increased target agreement and decreased target discrepancy This relationship was particularly strong for observers that performed lower on Scoring Study 1

2015-16 Roll-Out Expansion to include 1008 statewide principals 58 LEAs and 11 charter schools 20 LEAs and 5 charter schools have OCT Facilitators Recruitment occurred at Principal READY meetings, Principal Council meetings, Service Support Team outreach Two ways to participate: Individual observer Whole LEA – locally-facilitated, collaborative PD course

Observation Calibration Training Participation Madi41795! Madi41795! Madi41795!   Observation Calibration Training Wikispace

Observation Calibration Training Timeline Dec 10th: Kickoff Feb 8th: SS1 Review/Facilitator Support Mar 1st: Mid-Year Check-In May 19th: Wrap-Up & Reports Dec Jan Feb Mar Apr May Scoring Study 1 Window Dec 10th – Jan 31st Lessons Window Feb 1st – April 1st Scoring Study 2 Window Apr 1st – Apr 29th *Note: exact dates subject to change