1. The Basic Components of Inter-Rater Reliability
ODE SUMMIT = SIT IN RUBRIC-ALIKE GROUPS
HANDOUT:
Rubric Language
Evidence Examples
Collecting Observation Evidence
IRR-Calibration Plan
Supplies:
Chart paper
Markers
Post-its
Highlighters
FOR ODE SUMMIT: Have extra copies of rubrics
Danielson
Marshall
LEGENDS

2. Objectives
Develop strong understanding of rubric language and performance levels
Strengthen observation skills and explore importance of frequent observation
Calibrating the collection of evidence through multiple measures
Rate the level of performance and provide consistent feedback that builds self-directed learners
2 min
Minimum IRR training should encompass 30-hours (this is akin to learning anything new)

3. Vision of Effectiveness
Think of when you observed an excellent teacher in the classroom or a professional context.
What did you see or hear that made you think that you were observing an effective practitioner?
In a perfect world, what is your vision of educator effectiveness?
Slides 3-6 = 15 min

4. Traits of Effectiveness
Write two pieces of evidence of an effective practitioner on two post-its (one piece of evidence per post-it)
As a table group sort your post-its into like categories
Agree on a label for each category
Slides 3-6 = 15 min
Slides 3-4 = 5 min

5. Traits of Effectiveness
Write two pieces of evidence of an effective practitioner on two post-its (one piece of evidence per post-it)
As a table group sort your post-its into like categories
Agree on a label for each category
Label:___ Label:___ Label:___ Label:___
Slides 3-6 = 15 min
This slide = 5 min

6. Traits of Effectiveness
What happens if we add the four Domains of the InTASC Standards?
Do you need to re-sort your post-its?
The Learner Content Instructional Professional & Learning Practices Responsibilities
Slides 3-6 = 15 min
This slide = 5 min
When you do this at home use the domains/categories specific to your rubric

7. Understanding the Rubric Set up
Danielson
Domains 
Components
Elements
Evidence
LEGENDS
Standards
Performance Indicators
Marshall
Criteria
3 min
Localize! Teach the structure of your rubric – how is it set-up?
How is it a resource for observers of practice?
How is it a resource for teachers in their professional development?

8. Level of Performance
Level 1- Does not meet standards; performs below the expectations for good performance under this standard; requires direct intervention and support to improve practice
Level 2- Making sufficient progress toward meeting this standard; meets expectations for good performance most of the time and shows the continuous improvement; expected improvement through focused professional learning and growth plan
5 min

9. Level of Performance
Level 3- Consistently meets expectations for good performance under the standard; demonstrates effective practices and impact on student learning; continues to improve professional practice through ongoing professional learning
Level 4- Consistently exceeds expectations for good performance under this standard; demonstrates highly effective practices and impact on student learning; continued expansion of expertise through professional learning and leadership opportunities
5 min

10. Where do you live?
It is expected that we strive and are supported to live at a Level 3 and visit Levels 2 & 4
You’re going to move between levels within your year and career!
3 min
This may be a cultural shift for teachers and principals.
Level “3” is where you really consistent control the practice in that standard – “4” may represent areas of passion or areas where we really shine, and “2”
In our rubric, we all have a standard that maybe everyone in Oregon is learning and may be a “2” currently (like ones addressing teachers knowing content standards and CCSS is brand new to most of us!).

11. For Example: Designing Student Assessments
5-min
Draw attention to 1F and talk about this one in particular and the creation of “ok to be at level 2”
Show them where the language supports their “newness” and growth and “allow” them to mark themselves basic knowing the plan to move towards proficient. Create the culture of “We got you (support you)” vs. “Gotcha!”

12. Differentiated Language Levels of Performance
Using your rubric, pick a component, standard or criteria
Highlight key words that show the difference between levels of practice
Discuss and record responses on the organizer provided
Share examples of evidence from your practice that matches the language you identified at each level
HANDOUT: Need blank table of levels of performance for common language
Slide 12 = 5 min
The difference between a “1” and a “4” is vastly apparent; it is the difference between the language in “2” and “3” that is really important to come to a firm and calibrated understanding of prior to engaging in gathering evidence toward the rubric. This is the difference between a proficient practitioner and one that needs more job-embedded supports to grow, or a place that a practitioner will need to focus on for their professional growth goals, etc.
Do this with as many components as you need to identify the patterns of differentiated language

13. Common Language Level 1 Level 2 Level 3 Level 4 No Never Not Clear
Unaware
Does Not
Poor
Little
Low
Inadequate
Uses Few
Some
Attempts
Limited
Moderate
Uneven
Inconsistent
Rarely
Developing
Partially
Consistent
High Quality
Timely
Accurate
Appropriate
Clear
Effective
Aligned
Almost Always
All Students
Highly Effective
Adapted For
Always
Fully Aligned
Extensive
Highly Developed
3 min
You may have also noted shifts in practice:
Example – from Danielson, as you go from left to right, the practitioner starts to have a more student-led style of teaching and learning

14. Identifying Evidence Read the evidence statements provided
Based on your rubric, which component, standard or criteria is it evidence of?
Which statements are evidence of observable practice? Non-observable? If non-observable, where would this evidence be found?
What are examples of evidence from your practice that support components, standards and criteria that aren’t listed? List a few examples of your own in the blank spaces.
Note: Levels of performance are not determined until multiple sources of evidence are collected.
15 min
HANDOUT: Examples of Evidence
Work with an elbow partner to fill-out the Examples of Evidence hand-out by following the prompts on the slides
When your back at your district, cross score with folks from a different working table or other partner – did they sort the evidence into the same categories? Did they think it was observable or non-observable? Where did they think they could find the non-observable evidence?

15. Fact vs. Opinion When Collecting Evidence
Table Talk:
Think about a time that you received an evaluation based on both factual evidence and opinion. What were the differences in…. How you felt? The impact on your professional growth? The relationship with your evaluator?
Why is it important to collect multiple pieces of factual evidence before trying to assess educator practice?
How can factual evidence support educator development?
3 min
Have volunteers share out

16. Which is Stronger? Why? Time Educator Actions Student Actions Time
1:01
1:03
Educator asked kids to stand and led them in “The Days of the Week” song.
Educator asked “What day comes after Saturday?”
16 of the 27 kids stood up for the song.
Steven shouted out, “Monday!” Most students laughed-2 boys physically rolled around and knocked over 2 girls. Steven walked away from the group, and sat in the opposite corner of the classroom
3 min
The top one is stronger it gives direct examples with details.
The teacher is going to be able to reflect more on this type of evidence.
Have participants talk about why. Chart the differences/reasons.
Time
Educator Actions
Student Actions
1:01
1:03
Educator reviewed the days of the week
Educator asked the question about the days of the week
Many students were not listening.
Steven called out over and over again

17. Effective Observation Practices
Eliminate effects of bias. Enter the classroom without judgment and work from evidence.
Collect factual evidence. Write down only what teachers and students say and do. Look for evidence of learning.
Remain, review, reflect. Pause to organize your evidence before aligning. Only rate after multiple sources of evidence are collected.
Frequent, focused, varied with timely feedback.
3 min
Eliminating bias is a training in and of itself, but it is of the utmost importance for a valid and reliable observations and evaluations.

18. Collecting Evidence
What do you see and hear the teacher and students doing?
What evidence can you gather?
Time
Educator Actions
Student Actions
HANDOUT: Collecting Evidence
1 min

19. Selecting Observation Focus
When taking notes, come up with a short-hand system. Don’t worry if you miss something – practice makes perfect.
Sample of how you could scaffold support for focusing your first 3 observations
Cluster1: Managing classroom procedures, Managing student behavior, and Organizing physical space
Cluster 2: Creating environment of respect & rapport, Establishing a culture for learning, Communicating with students
Cluster 3: Using questioning and discussion techniques, Engaging students in learning, Using assessment in instruction
Count off at your table by “3’s” – be ready to gather evidence on the video based on your cluster
3 min

20. Video 1
Ms. Warburton’s 8th grade math lesson: Sorting and classifying equations
CCSS Math 8.EE.C.7a
Collect evidence statements on your cluster
classifying-equations-overview
This slide = 12-mim
Be sure that you are sitting with a like rubric group

21. Video 1 (follow up)
Compare your evidence statements with a cluster partner
With partner, use your rubric and practice aligning/coding evidence statements with components/standards/criteria language.
Make sure each item shared is factual evidence
Each cluster shares out at their table
One piece of evidence from each cluster
In whole group, each cluster will share out one piece of evidence.
Add to your notes any factual evidence your table partners shared that you did not have written down
This slide = 15 min
AT ODE SUMMIT: For sake of training only do 2 at each table plus statement we provide for calibration

22. Calibrating Let’s all look at evidence that aligns with:
At tables: Based on the evidence we have compiled, talk about the level of performance should be assigned for each component.
On the following slides use your clicker to report the ranking for each of the three cluster areas we collected evidence for
Cluster
Danielson
Marshall
LEGENDS
Cluster 1 2c Bh
3.3
This slide = 30 min
First bullet point: Let’s all look at evidence that aligns with ____________. You may need to give participants more factual evidence statement (hopefully one that they have already) (Or rove and find a common one being talked about). Give participants 1-2 minutes to do this, then share out and chart
Second bullet point: Have tables agree on and assign level of performance for given evidence statement, then have tables share out. Did we get the same level? Talk about the fact that you would never assign a level of performance after only one observation – you need multiple observations and evidence to do that(the importance of only using one piece during IRR training). 80% agreement prior to moving on (and the disagreement should not be more than one-level off).

23. What is the performance level for Cluster 1: 2c, Bh, and 3.3?4
Expert Score:

24. Calibrating Let’s all look at evidence that aligns with:
At tables: Based on the evidence we have compiled, talk about the level of performance should be assigned for each component.
On the following slides use your clicker to report the ranking for each of the three cluster areas we collected evidence for
Cluster
Danielson
Marshall
LEGENDS
Cluster 2 2a Bd
3.2
This slide = 30 min
First bullet point: Let’s all look at evidence that aligns with ____________. You may need to give participants more factual evidence statement (hopefully one that they have already) (Or rove and find a common one being talked about). Give participants 1-2 minutes to do this, then share out and chart
Second bullet point: Have tables agree on and assign level of performance for given evidence statement, then have tables share out. Did we get the same level? Talk about the fact that you would never assign a level of performance after only one observation – you need multiple observations and evidence to do that(the importance of only using one piece during IRR training). 80% agreement prior to moving on (and the disagreement should not be more than one-level off).

25. What is the performance level for Cluster 2: 2a, Bd, and 3.2?1 2 3 4
Expert Score:

26. Calibrating Let’s all look at evidence that aligns with:
At tables: Based on the evidence we have compiled, talk about the level of performance should be assigned for each component.
On the following slides use your clicker to report the ranking for each of the three cluster areas we collected evidence for
Cluster
Danielson
Marshall
LEGENDS
Cluster 3 3c Cg
5.2
This slide = 30 min
First bullet point: Let’s all look at evidence that aligns with ____________. You may need to give participants more factual evidence statement (hopefully one that they have already) (Or rove and find a common one being talked about). Give participants 1-2 minutes to do this, then share out and chart
Second bullet point: Have tables agree on and assign level of performance for given evidence statement, then have tables share out. Did we get the same level? Talk about the fact that you would never assign a level of performance after only one observation – you need multiple observations and evidence to do that(the importance of only using one piece during IRR training). 80% agreement prior to moving on (and the disagreement should not be more than one-level off).

27. What is the performance level for Cluster 3: 3c, Cg, and 5.2?1 2 3 4
Expert Score:

28. Effective Feedback Preparing Sharing
Align evidence to the rubric. Code evidence to the rubric and determine the level of performance for each component observed.
2. Prioritize. Select one or two parts of the teacher’s practice on which to focus.
Sharing
Invite dialogue. Welcome and value teacher input.
2. Refer to evidence. Discuss factual evidence from the observation and invite dialogue on the evidence or its alignment to the rubric.
3. Identify development areas. Define specific areas for development and invite the teachers to comment on those areas or to suggest additional areas.
4. Plan concrete action. Together, develop clear and measurable next steps, including resources and a plan for support.
5. Set a timeline. Agree on a plan for implementation and follow-up.
5 min
Back at home, write feedback for practice or know that this is the next step in your

29. Planning for IRR Implementation
Plan - Best practice = 30-hours minimum
Rubric:
Understanding Rubric Format
Levels of Performance
Language of Instruction
Evidence:
Multiple Measures
Observable & Non-Observable
Aligning/Coding to Rubric
Fact vs. Opinion
Awareness of Bias
__ min
ODE SUMMIT: Plan for Implementation

30. Planning for IRR Implementation
Practice Observations:
Observe Practice (Frequent, Focused, Varied)
Organize Evidence (Remain, Review, Reflect)
Aligning/Coding Evidence to Rubric
Rate Performance
80% rule
Effective Feedback
Preparing / Sharing model
What is the plan to come together multiple times to re-calibrate throughout the year(s)?
How will you assess the effectiveness of this IRR/Calibration plan?
__ min
ODE SUMMIT: Plan for Implementation

31. Video 2 Ms. Bannon’s 3rd grade reading lesson: Understanding main idea
CCSS ELA RL.3.2 & ELA.SL.3.2
Collect evidence statements on your cluster
ela-lesson
Slides = 35 min
Slide 25 = 12-min
WILL NOT DO AT ODE SUMMIT

32. Video 2 (follow up)
Compare your evidence statements with a cluster partner
With partner, use your rubric and practice coding evidence statements with components/standards/criteria language.
Make sure each item shared is factual evidence
Each cluster shares out at their table
One piece of evidence from each cluster
In whole group, each cluster will share out one piece of evidence.
Add to your notes any factual evidence your table partners shared that you did not have written down
Slides = 35 min
Slide 26 = 15-min
WILL NOT DO AT ODE SUMMIT

33. Calibrating Let’s all look at evidence that aligns with:
At tables: Based on the evidence we have compiled, talk about the level of performance should be assigned for each component.
On the following slides use your clicker to report the ranking for each of the three cluster areas we collected evidence for
Cluster
Danielson
Marshall
LEGENDS
Cluster 1 3e Aj
3.1
This slide = 30 min
First bullet point: Let’s all look at evidence that aligns with ____________. You may need to give participants more factual evidence statement (hopefully one that they have already) (Or rove and find a common one being talked about). Give participants 1-2 minutes to do this, then share out and chart
Second bullet point: Have tables agree on and assign level of performance for given evidence statement, then have tables share out. Did we get the same level? Talk about the fact that you would never assign a level of performance after only one observation – you need multiple observations and evidence to do that(the importance of only using one piece during IRR training). 80% agreement prior to moving on (and the disagreement should not be more than one-level off).

34. What is the performance level for Cluster 1: 3e, Aj, and 3.1?2 3 4

35. Calibrating Let’s all look at evidence that aligns with:
At tables: Based on the evidence we have compiled, talk about the level of performance should be assigned for each component.
On the following slides use your clicker to report the ranking for each of the three cluster areas we collected evidence for
Cluster
Danielson
Marshall
LEGENDS
Cluster 2 3a Cc
3.2
This slide = 30 min
First bullet point: Let’s all look at evidence that aligns with ____________. You may need to give participants more factual evidence statement (hopefully one that they have already) (Or rove and find a common one being talked about). Give participants 1-2 minutes to do this, then share out and chart
Second bullet point: Have tables agree on and assign level of performance for given evidence statement, then have tables share out. Did we get the same level? Talk about the fact that you would never assign a level of performance after only one observation – you need multiple observations and evidence to do that(the importance of only using one piece during IRR training). 80% agreement prior to moving on (and the disagreement should not be more than one-level off).

36. What is the performance level for Cluster 2: 3a, Cc, and 3.2?1 2 3 4

37. Calibrating Let’s all look at evidence that aligns with:
At tables: Based on the evidence we have compiled, talk about the level of performance should be assigned for each component.
On the following slides use your clicker to report the ranking for each of the three cluster areas we collected evidence for
Cluster
Danielson
Marshall
LEGENDS
Cluster 3 3b Cg
8.4
This slide = 30 min
First bullet point: Let’s all look at evidence that aligns with ____________. You may need to give participants more factual evidence statement (hopefully one that they have already) (Or rove and find a common one being talked about). Give participants 1-2 minutes to do this, then share out and chart
Second bullet point: Have tables agree on and assign level of performance for given evidence statement, then have tables share out. Did we get the same level? Talk about the fact that you would never assign a level of performance after only one observation – you need multiple observations and evidence to do that(the importance of only using one piece during IRR training). 80% agreement prior to moving on (and the disagreement should not be more than one-level off).

38. What is the performance level for Cluster 3: 3b,__, and__?1 2 3 4

39. Wrapping Up
First of many professional learning opportunities
Bend Summit – October 22nd
CCSS Regional Series
Pendleton – October 29 & 29
Wilsonville – November 4 & 5
Redmond – November 7 & 8
Resources
SLG Goal Guidance and samples
Updated Framework, FAQs, “Who is evaluated?”
Toolkit Implementation Web Page
Today’s summit is just the beginning of the professional learning opportunities that will be available. COSA’s regional series “Common Core in Action” will have fall and spring dates. Where the connections between CCSS and Educator Effectiveness will be explored more fully.
Implementation Resources page of the toolkit is where to find all the resources we have talked about today, including the template and sample SLG goals

40. Thank you for attending today!
Providing Feedback
Summit Evaluation
General Questions/Feedback
Thank you for attending today!
Please take a moment to complete the summit evaluation. We will use this information to inform the Bend session.
We’ve tried to answer throughout the session today, but if any additional questions surface, or if you want to give feedback on the SLG Goals guidance document, please This address will be monitored and questions answered either through direct feedback or through updated FAQs.
Thank you for attending!