1. A brief introduction to
Inter-Rater Agreement
&
Inter-Rater Reliability

2. Best Practices in Observing
Basis for observation– Knowledge of the criteria
Plan for gathering data – Fidelity to process and procedures
End result –
Quality of the product
Observer must align evidence to appropriate component
Observer must level evidence accurately
Observer must have sufficient evidence to support rating
Observer must have skill in engaging educator in conversation around level and direction for future
Review this slide – best practices in observing
Developed by Teaching Learning Solutions, Inc FFT Rubrics - ASCD

3. Developed by Teaching Learning Solutions, Inc. FFT Rubrics - ASCD
Bias
Definition:
Attaching positive or negative meaning to elements in our environment based on personal or societal influences that shape our thinking.
A biased judgment is based on outside influences and is not necessarily related to a teacher’s effectiveness.
Example: “Mrs. T does so much for the school, she is an excellent teacher.”
The actual classroom evidence may not support the rating of the teacher as “excellent.”
Review the slide, have participants read
Point to make:
When making judgments from evidence based on observation, we draw conclusions from what we see and hear. And, while it is human nature to attach positive or negative “meaning” to elements in our environment, it is inappropriate in a professional observation to allow our personal associations – our biases – to affect our judgment.
Developed by Teaching Learning Solutions, Inc FFT Rubrics - ASCD

4. Understanding Your Own Bias
Yield sign to match yield in handouts
Developed by Teaching Learning Solutions, Inc FFT Rubrics - ASCD

5. Developed by Teaching Learning Solutions, Inc. FFT Rubrics - ASCD
Evidence
Evidence is a factual reporting of events.
It may include teacher and student actions and/or behaviors.
It may also include artifacts prepared by the teacher, students, or others.
It is not clouded with personal opinion or biases.
It is selected using professional judgment by the observer and / or the teacher.
Inform participants that collecting strong evidence is critical to making accurate assessments of a teacher’s practice. We’ll now explore the qualities of good evidence.
Facilitator notes:
Inform participants that we will now prepare to do an observation. Prior to the observation we’ll review the qualities of evidence, and the role of bias.
Have participants read this slide. Brief review, proceed to the next slide
Developed by Teaching Learning Solutions, Inc FFT Rubrics - ASCD

6. Types of Observation Evidence
Verbatim scripting of teacher or student comments:
“Bring your white boards, markers and erasers to the carpet and sit on your square.”
Non-evaluative statements of observed teacher or student behavior:
Teacher presented the content from the front of room.
Numeric information about time, student participation, resource use, etc.
[9:14 – 9:29] Warm-up. 8 of 22 Ss finished at 9:20, sat still until 9:29
An observed aspect of the environment:
Desks were arranged in groups of four with room to walk between each group.
Review briefly, the contents of this slide
Developed by Teaching Learning Solutions, Inc FFT Rubrics - ASCD

7. Developed by Teaching Learning Solutions, Inc. FFT Rubrics - ASCD
Evidence vs. Opinion…
Read each statement. Decide – is it evidence or opinion?
Discuss your answer with your elbow partner.
Be prepared to discuss some of the statements, or statements about which you have questions.
[PARTICIPANT DOCUMENT: Evidence-Opinion-Bias p. 1]
Direct participants to the Evidence vs. Opinion statements in their material.
Review the directions on the slide.
Allow time for participants to work on the statements
Debrief
Facilitate a debrief and discuss how some statements can be revised to become accurate evidence statements
Developed by Teaching Learning Solutions, Inc FFT Rubrics - ASCD

8. Other Threats to Observer Accuracy
Assessor bias
Leniency
Central Tendency
“Halo” or “Horns” Effect
Review each “threat” to accuracy (briefly)
Assessor bias: e.g. an Observer likes to see rooms neatly organized with desks in rows, Observers who like “quiet” schools
Leniency: e.g. Susan is a great teacher, she does so much with the kids; yearbook, dances, after-school clubs… (even if the evidence doesn’t support a high rating, Susan is rated higher because of other factors
Central Tendency: seen in rubrics with odd numbers, proficient is the “middle”. It’s easier to go to the middle than to make a decision higher or lower.
Halo / Horns: People perceived by the Observer as being able to do “no wrong” (Halo) or unable to do anything right (horns)
Developed by Teaching Learning Solutions, Inc FFT Rubrics - ASCD

9. Self-Evaluation
Using two different colored highlighters, review your observation write-up.
With one color, code statements of evidence.
With the other color, code statements of opinion/bias.
Write one of your highlighted comments on an index card. The comment can be a statement that you are unsure about being either evidence or opinion.
We will discuss some of the comments as a group.

10. What is Evidence? - Review
Actions, by teacher or students
Statements or questions, by teacher or students
Observable features of the classroom
Review the evidence collected previously – is it evidence? Or opinion?
After reviewing the contents of this slide, return to the scripted evidence collected during the first observation of Ms. Mahar’s lesson.
Assess the evidence with the group.
Read the evidence statements – are they evidence, opinion, judgment?
Developed by Teaching Learning Solutions, Inc FFT Rubrics - ASCD

11. TLS Evidence Rubric Element 2
© Teaching &Learning Solutions, may not be used without permission

12. 1st Observation Practice (10th Grade Mathematics Instruction)
Review:
What type of evidence must you collect to assess the priorities of the rubrics?
FIRST OBSERVATION PRACTICE
Materials: Participants should type their notes on a laptop, if at all possible. Otherwise, they should take notes by hand.
Facilitator notes:
Review the slide contents.
Remind participants that they should be aware of the priorities when observing;
1) cognitive engagement, 2) constructivist learning, and 3) 21st Century skills
Review question (very brief – ask of the whole group…)
Answers:
To accurately assess engagement the observer MUST collect evidence of students being engaged, evidence that shows students are thinking.
Inform participants that they will be watching a 9th Grade Algebra classroom. The observation is approximately 12 minutes.
Provide directions for use of the e-version of the data collection instrument.
Developed by Teaching Learning Solutions, Inc FFT Rubrics - ASCD

13. Developed by Teaching Learning Solutions, Inc. FFT Rubrics - ASCD
Observing Practice
Observe the video of the mathematics lesson
Collect evidence of Dimensions 1, 5, and 7:
With a partner, sort your evidence so that it aligns with the appropriate criteria in your Framework.
Be prepared to share your evidence
Facilitator notes:
Lesson to be observed: Grade 9 Algebra lesson. Ms. Battye is the teacher.
Remind participants that they should prepare to work with a partner.
Allow them time to prepare their laptops and forms on which they will collect evidence.
Evidence should be collected on a Word document.
Participants should gather evidence for Domain 3.
Inform them that they will follow the evidence gathering protocol.
Step 1 – observe and gather evidence (individually)
Step 2 – share evidence w/ your partner and sort in to the appropriate components or elements for Domain/Standard 3 (instruction).
After gathering and sorting evidence.
Discuss the evidence and lesson with your partner. What questions do you have that must be answered?
Be prepared to share the evidence and the questions.
Debrief:
Have different partners share evidence for Domain 3 – one or two pieces.
Assess evidence as it’s being shared, if incorrectly placed discuss the evidence.
Remind participants that we’re not scoring / rating practice. We’re identifying questions that must be answered before scoring can occur.
Developed by Teaching Learning Solutions, Inc FFT Rubrics - ASCD

14. Checking Objectivity of Evidence
Use the self-check questions to review your evidence collection
Have I recorded only facts?
Is my evidence relevant to the criteria being examined?
Whenever possible, have I quantified words such as few, some, and most?
Have I used quotation marks when quoting a teacher or student?
Does my selection or documentation of evidence indicate any personal or professional preferences? 
Have I included any opinion (in the guise of fact)?
Review Self-Check Questions
Developed by Teaching Learning Solutions, Inc FFT Rubrics - ASCD

15. Developed by Teaching Learning Solutions, Inc. FFT Rubrics - ASCD
The Evidence Cycle
COLLECT DATA
(Evidence)
SORT TO
ALIGN
WITH YOUR
FRAMEWORK
Interpret:
Clarify
Conclusions
NO!
Review this slide
Gathering evidence follows a cycle during an evaluation.
First, the observer collects data / evidence throughout the observation. Evidence is gathered in conferences, observations, and reviews of artifacts.
Second, the observer sorts the evidence to align it properly with the respective domain, standard, and indicator. This sorting process will occur simultaneously with the gathering of evidence as observers internalize the framework.
Third, the observer interprets the evidence
Finally, after gathering, sorting, and interpreting, the observer will form conclusions about the educator’s practice.
Click the arrow on the slide…
It’s normal for us to see evidence and move immediately to conclusions / judgments. However, when doing so, we might be making assumptions about practice. This evidence protocol is designed to assure that conclusions are accurate, fair, and not based on any assumptions.
Points to make.
The gathering, sorting, and interpreting of evidence are important steps in the assessment of teaching practice.
Impact on learning…
Support needed…
Developed by Teaching Learning Solutions, Inc FFT Rubrics - ASCD

16. Closure
On your Sheet for Inter-Rater Reliability and Agreement for the Teacher Evaluation Process, list:
3 things you already do well
2 things you can integrate into your practice
1 immediate step you will take
Answer the question: “What is the difference between inter-rater agreement and inter-rater reliability?
Thank you!