A brief introduction to Inter-Rater Agreement & Inter-Rater Reliability

Best Practices in Observing Basis for observation– Knowledge of the criteria Plan for gathering data – Fidelity to process and procedures End result – Quality of the product Observer must align evidence to appropriate component Observer must level evidence accurately Observer must have sufficient evidence to support rating Observer must have skill in engaging educator in conversation around level and direction for future Review this slide – best practices in observing Developed by Teaching Learning Solutions, Inc. FFT Rubrics - ASCD

Developed by Teaching Learning Solutions, Inc. FFT Rubrics - ASCD Bias Definition: Attaching positive or negative meaning to elements in our environment based on personal or societal influences that shape our thinking. A biased judgment is based on outside influences and is not necessarily related to a teacher’s effectiveness. Example: “Mrs. T does so much for the school, she is an excellent teacher.” The actual classroom evidence may not support the rating of the teacher as “excellent.” Review the slide, have participants read Point to make: When making judgments from evidence based on observation, we draw conclusions from what we see and hear. And, while it is human nature to attach positive or negative “meaning” to elements in our environment, it is inappropriate in a professional observation to allow our personal associations – our biases – to affect our judgment. Developed by Teaching Learning Solutions, Inc. FFT Rubrics - ASCD

Understanding Your Own Bias Yield sign to match yield in handouts Developed by Teaching Learning Solutions, Inc. FFT Rubrics - ASCD

Developed by Teaching Learning Solutions, Inc. FFT Rubrics - ASCD Evidence Evidence is a factual reporting of events. It may include teacher and student actions and/or behaviors. It may also include artifacts prepared by the teacher, students, or others. It is not clouded with personal opinion or biases. It is selected using professional judgment by the observer and / or the teacher. Inform participants that collecting strong evidence is critical to making accurate assessments of a teacher’s practice. We’ll now explore the qualities of good evidence. Facilitator notes: Inform participants that we will now prepare to do an observation. Prior to the observation we’ll review the qualities of evidence, and the role of bias. Have participants read this slide. Brief review, proceed to the next slide Developed by Teaching Learning Solutions, Inc. FFT Rubrics - ASCD

Types of Observation Evidence Verbatim scripting of teacher or student comments: “Bring your white boards, markers and erasers to the carpet and sit on your square.” Non-evaluative statements of observed teacher or student behavior: Teacher presented the content from the front of room. Numeric information about time, student participation, resource use, etc. [9:14 – 9:29] Warm-up. 8 of 22 Ss finished at 9:20, sat still until 9:29 An observed aspect of the environment: Desks were arranged in groups of four with room to walk between each group. Review briefly, the contents of this slide Developed by Teaching Learning Solutions, Inc. FFT Rubrics - ASCD

Developed by Teaching Learning Solutions, Inc. FFT Rubrics - ASCD Evidence vs. Opinion… Read each statement. Decide – is it evidence or opinion? Discuss your answer with your elbow partner. Be prepared to discuss some of the statements, or statements about which you have questions. [PARTICIPANT DOCUMENT: Evidence-Opinion-Bias p. 1] Direct participants to the Evidence vs. Opinion statements in their material. Review the directions on the slide. Allow time for participants to work on the statements Debrief Facilitate a debrief and discuss how some statements can be revised to become accurate evidence statements Developed by Teaching Learning Solutions, Inc. FFT Rubrics - ASCD

Other Threats to Observer Accuracy Assessor bias Leniency Central Tendency “Halo” or “Horns” Effect Review each “threat” to accuracy (briefly) Assessor bias: e.g. an Observer likes to see rooms neatly organized with desks in rows, Observers who like “quiet” schools Leniency: e.g. Susan is a great teacher, she does so much with the kids; yearbook, dances, after-school clubs… (even if the evidence doesn’t support a high rating, Susan is rated higher because of other factors Central Tendency: seen in rubrics with odd numbers, proficient is the “middle”. It’s easier to go to the middle than to make a decision higher or lower. Halo / Horns: People perceived by the Observer as being able to do “no wrong” (Halo) or unable to do anything right (horns) Developed by Teaching Learning Solutions, Inc. FFT Rubrics - ASCD

Self-Evaluation Using two different colored highlighters, review your observation write-up. With one color, code statements of evidence. With the other color, code statements of opinion/bias. Write one of your highlighted comments on an index card. The comment can be a statement that you are unsure about being either evidence or opinion. We will discuss some of the comments as a group.

What is Evidence? - Review Actions, by teacher or students Statements or questions, by teacher or students Observable features of the classroom Review the evidence collected previously – is it evidence? Or opinion? After reviewing the contents of this slide, return to the scripted evidence collected during the first observation of Ms. Mahar’s lesson. Assess the evidence with the group. Read the evidence statements – are they evidence, opinion, judgment? Developed by Teaching Learning Solutions, Inc. FFT Rubrics - ASCD

TLS Evidence Rubric Element 2 © Teaching &Learning Solutions, may not be used without permission

1st Observation Practice (10th Grade Mathematics Instruction) Review: What type of evidence must you collect to assess the priorities of the rubrics? FIRST OBSERVATION PRACTICE Materials: Participants should type their notes on a laptop, if at all possible. Otherwise, they should take notes by hand. Facilitator notes: Review the slide contents. Remind participants that they should be aware of the priorities when observing; 1) cognitive engagement, 2) constructivist learning, and 3) 21st Century skills Review question (very brief – ask of the whole group…) Answers: To accurately assess engagement the observer MUST collect evidence of students being engaged, evidence that shows students are thinking. Inform participants that they will be watching a 9th Grade Algebra classroom. The observation is approximately 12 minutes. Provide directions for use of the e-version of the data collection instrument. Developed by Teaching Learning Solutions, Inc. FFT Rubrics - ASCD

Developed by Teaching Learning Solutions, Inc. FFT Rubrics - ASCD Observing Practice Observe the video of the mathematics lesson Collect evidence of Dimensions 1, 5, and 7: With a partner, sort your evidence so that it aligns with the appropriate criteria in your Framework. Be prepared to share your evidence Facilitator notes: Lesson to be observed: Grade 9 Algebra lesson. Ms. Battye is the teacher. Remind participants that they should prepare to work with a partner. Allow them time to prepare their laptops and forms on which they will collect evidence. Evidence should be collected on a Word document. Participants should gather evidence for Domain 3. Inform them that they will follow the evidence gathering protocol. Step 1 – observe and gather evidence (individually) Step 2 – share evidence w/ your partner and sort in to the appropriate components or elements for Domain/Standard 3 (instruction). After gathering and sorting evidence. Discuss the evidence and lesson with your partner. What questions do you have that must be answered? Be prepared to share the evidence and the questions. Debrief: Have different partners share evidence for Domain 3 – one or two pieces. Assess evidence as it’s being shared, if incorrectly placed discuss the evidence. Remind participants that we’re not scoring / rating practice. We’re identifying questions that must be answered before scoring can occur. Developed by Teaching Learning Solutions, Inc. FFT Rubrics - ASCD

Checking Objectivity of Evidence Use the self-check questions to review your evidence collection Have I recorded only facts? Is my evidence relevant to the criteria being examined? Whenever possible, have I quantified words such as few, some, and most? Have I used quotation marks when quoting a teacher or student? Does my selection or documentation of evidence indicate any personal or professional preferences?  Have I included any opinion (in the guise of fact)? Review Self-Check Questions Developed by Teaching Learning Solutions, Inc. FFT Rubrics - ASCD

Developed by Teaching Learning Solutions, Inc. FFT Rubrics - ASCD The Evidence Cycle COLLECT DATA (Evidence) SORT TO ALIGN WITH YOUR FRAMEWORK Interpret: Clarify Conclusions NO! Review this slide Gathering evidence follows a cycle during an evaluation. First, the observer collects data / evidence throughout the observation. Evidence is gathered in conferences, observations, and reviews of artifacts. Second, the observer sorts the evidence to align it properly with the respective domain, standard, and indicator. This sorting process will occur simultaneously with the gathering of evidence as observers internalize the framework. Third, the observer interprets the evidence Finally, after gathering, sorting, and interpreting, the observer will form conclusions about the educator’s practice. Click the arrow on the slide… It’s normal for us to see evidence and move immediately to conclusions / judgments. However, when doing so, we might be making assumptions about practice. This evidence protocol is designed to assure that conclusions are accurate, fair, and not based on any assumptions. Points to make. The gathering, sorting, and interpreting of evidence are important steps in the assessment of teaching practice. Impact on learning… Support needed… Developed by Teaching Learning Solutions, Inc. FFT Rubrics - ASCD

Closure On your 3-2-1-+1 Sheet for Inter-Rater Reliability and Agreement for the Teacher Evaluation Process, list: 3 things you already do well 2 things you can integrate into your practice 1 immediate step you will take Answer the question: “What is the difference between inter-rater agreement and inter-rater reliability? Thank you!