1. NC Observation Calibration Process Update & Implementation Plan North Carolina State Board of Education Kimberly Simmons, North Carolina Educator Evaluation System Consultant Cynthia Martin, Ed.D. Service Support Lead

2. Observation Calibration Training
Observation Calibration Training l Observation Engine
Process for Educators l Content
Online Resources for Educators
Resource Sharing Strengthening PLCs
Pilot Results 2015
Implementation Plan l 2016 Timeline

3. What is Observation Calibration Training?
Observation Calibration Training (OCT) is an online evaluator training platform combining BloomBoard’s PD resources with Empirical Education’s observer training and calibration tool, Observation Engine.
Goals of the OCT:
Increase observation skills resulting in stronger scoring calibration
Increase rater agreement
Provide a common experience for LEAs to host collaborative conversations to improve instructional leadership skills

4. What is Observation Engine?
Online training tool that uses classroom videos to ensure accurate and reliable classroom observations
Videos from the MET (Measures of Effective Teaching) library have been master-scored on the NCEES teacher evaluation rubric to create learning opportunities for teacher evaluators in North Carolina. Master scoring including Educator Effectiveness staff working with Empirical Education staff.

5. Observation Engine NCEES Content
2 Observer Calibration Events – Scoring Studies
Rating video practice to increase rater agreement to target scores
Scoring Study 1 (video) is administered as an initial assessment of calibration
Scoring Study 2 (video) is administered at the end of the year as a post- assessment
19 Targeted Learning Exercises - Lessons
Targeted, self-paced online exercises with immediate on-screen feedback.
The OCT contains two types of lessons:
2 Full classroom observations
17 Element-specific classroom observations

6. BloomBoard offers easy access to a rich Marketplace of thousands of learning resources to help educators improve their practice. Find courses, eBooks, articles, and videos from the most trusted content providers, all accessible from a single destination.

7. Portal to Observation Engine
BloomBoard Marketplace is accessible through NCEdCloud using IAM through the Educator Evaluation Professional Development System. 

8. Portal to Observation Engine
Easily access Observation Engine activities via the Marketplace, and use professional learning resources to support and enhance observer calibration

9. Resources, Personalized for Each Educator
Support teachers’ different learning needs, just like students, by using Marketplace content that is searchable by topic, grade level, and more.
Support teacher growth with personalized learning pathways aligned to both individual goals and strategic initiatives defined by your school or LEA.

10. BloomBoard Makes Sharing Resources Easy
Share your favorite resources with a colleague, teachers at your site, a PLC, or through your social networks.

11. High-Quality Content from Trusted Partners
BloomBoard works with the most trusted names in education to ensure the Marketplace has valuable, up-to-date resources. Each resource is vetted by a team of experts, and is further endorsed through ratings and reviews provided by educators in the BloomBoard community.

12. 2014-15 Observation Calibration Training (OCT) Pilot
27 LEAs and 5 charter schools registered to participate in the OCT
138 administrators from 20 LEAs participated in the OCT pilot
Participants’ scoring improved over the course of the pilot
Utilizing OCT practice resources improved scoring, regardless of initial performance level
Collaboration is key to NCEES calibration
OCT Flyer
Link to all 2014/15 pilot reports
May want to remove the first bullet, because the information may not be important to SBE

13. Scoring Accuracy Improved
T tests revealed that in Scoring Sstudy 2, percent target agreement was higher (p<.05) and percent discrepant was lower (p<.001) than in Scoring Study 1
Key metrics:
Percent Target Agreement: the percent of scores that agree exactly with the target scores
Percent Target Discrepant: the percent of scores that disagree with the target scores by 2 or more performance levels (e.g. when the target score is 2 and the score provided is a 4)

14. Completing OCT Lessons Improved Scoring
Regression analysis revealed that the number of lessons completed significantly increased target agreement and decreased target discrepancy
This relationship was particularly strong for observers that performed lower on Scoring Study 1

15. 2015-16 Roll-Out Expansion to include 1008 statewide principals
58 LEAs and 11 charter schools
20 LEAs and 5 charter schools have OCT Facilitators
Recruitment occurred at Principal READY meetings, Principal Council meetings, Service Support Team outreach
Two ways to participate:
Individual observer
Whole LEA – locally-facilitated, collaborative PD course

16. Observation Calibration Training Participation
Madi41795!
Madi41795!
Madi41795!
Observation Calibration Training Wikispace

17. Observation Calibration Training Timeline
Dec 10th: Kickoff
Feb 8th: SS1 Review/Facilitator Support
Mar 1st: Mid-Year Check-In
May 19th: Wrap-Up & Reports
Dec
Jan
Feb
Mar
Apr
May
Scoring Study 1 Window
Dec 10th – Jan 31st
Lessons Window
Feb 1st – April 1st
Scoring Study 2 Window
Apr 1st – Apr 29th
*Note: exact dates subject to change