INTER-RATER AGREEMENT IN KANSAS Summer Principals Academy July 22-24, 2014 Abilene, KS

What is Inter-Rater Agreement? Consistent understanding Consistent agreement Consistent process Consistent credibility The degree 2 raters using the same scale give the same rating in identical situations. 2

Where is Inter-rater Agreement Useful?? Inter-Rater Agreement is useful in all areas of administrative practice including: Kansas Educator Evaluations Kansas Accreditation Personnel Decisions Family Engagement Programs Budget Management 3

What are the Goals of Inter-rater Agreement? The Inter-Rater Agreement Statistical Goal is 75% - agreement with no rating more than one level apart. The Goal is to have an administrative Common Frame of Reference for your district such as: Agreement with no rating more than one level apart. Consistent understanding Consistent agreement Consistent process Consistent credibility 4

How is Inter-rater Agreement Helpful?? What is a Common Understanding? How does Inter-rater Agreement help administrators?? Facilitates Quality Provide Consistent Messaging Impacts Increased Student Growth Enhances Trust Facilitates Quality Feedback Legally Defensible Assists Accurate Data Collection Allows for Increase Cooperation over Time Passes Public Scrutiny 5

District-wide Agreement about External Criteria Agreed Upon External Criteria for: Observations Artifacts Units of Study Processes Program Development Time Limits Walk-Throughs Protocols 6

Pragmatics/Considerations Potential Landmines Discuss with an Elbow Partner Possible Issues that may cause Rater Error Leniency: Benefit of the Doubt Central Tendency: Safety in the Middle Comparing One with Another: Stick to the Rubric Relying on Gut too Much: Stick to the Rubric Evaluating Absent Evidence: Stick to What is Observed 7

How to Train Administrators in using Inter-rater Agreement Inter-Rater Agreement Training – Things to Consider: Synchronous and Asynchronous Experience of the Participants How to Ensure Fidelity Use of Video Examples Use of Case Studies Bias Exists Single versus Multiple Observations Professional Judgment Degree of Familiarity 8

Getting Started in YOUR District Getting Your Bearings Developing a Guidebook Developing a Training Protocol 9

I. Getting Your Bearings Pilot with Administrative Team Pilot Approach: Start with Safe Example: Use a video from YouTube Assess Specific Area of Teaching Student Engagement Teacher Activity Classroom Environment 2. Review current issue/goal/vision/needs 3. Review current practices/instrumentation Rubrics Protocol Training

II. Developing a Guidebook Paper and Electronic: Synchronous and Asynchronous EXCELLENT Administrative Council Endeavor – Ultimately to be used during Professional Learning Times with the Staffs Pieces and Parts Terms/Definitions: Frame of Reference/Common Understanding “Demonstrates” “Frequently” “Usually” 11

II. Developing a Guidebook Pieces and Parts (continued) Examples and Rubrics Areas to Observe Establishes/maintains a culture of safety, respect and rapport Knows how to make content accessible to all learners Plans standards-based instruction and formative assessments Uses a variety of instructional strategies to meet needs Deepens understanding through critical thinking, problem-solving and student reflection Analyzes student performance and provides feedback Collaborates to support student learning 12

II. Developing a Guidebook Pieces and Parts (continued) Exemplars: Video and Artifacts Master-Coded Videos: Leads to high level of agreement Example Artifacts Classroom Rules Lesson Plans Parent Note Home Lesson Plan for Adults: Andragogy 13

III. Developing a Training Protocol Step-by-Step Procedure 1. Initial Training: Goals/Terms Practice Interpreting Rubrics Differentiate between Different Performance Levels with Examples 2. Calibration of the Players Video and Artifacts 3. Review of the System 4. Adjustments (If needed) 5. Implementation 6. Evaluate/Reflect/Adjust 7. Re-Calibrate… 14

Start Slowly Start with “Engagement” with a Small Group Get Agreement Gain Confidence Over Time Write and Refine in Teams 15 Take Aways – To Do List

16 Start with the End in Mind Develop Common Understandings Use Groups to Ensure Buy-In Build Measurable Steps to Reach Each Goal Final Thoughts

Contact Information Teacher Licensure and Accreditation Team Director Scott Myers