1. Aligning Academic Review and Performance Evaluation (AARPE)
Virginia Department of Education
Office of School Improvement
Important Facilitator Notes:
Information in quotations indicates scripted material
Information in parentheses indicates facilitator directions/information
___________________________________________
(DECISIONS TO BE MADE PRIOR TO SESSION 3: Who and how will you address Next Steps?; Where did you end up with Standard 3 – this will determine how you modify this slide deck to pick up during Session 3; Will you incorporate work on the Principal standards or not – this will determine how you modify this slide deck; How will you address participants who do not bring the requested evaluation feedback documents; How will your LTPs be incorporated if at all?)
“Does everyone have a name tag and a handout packet? Take a quick minute to make sure that you are seated at the correct table. See labels on each table.”
(Introduce facilitators and quick introductions around the room).
“Today we continue our work on Aligning Academic Review and Performance Evaluation (AARPE) technical assistance. We will spend the better part of our morning using your observations from your first inter-rater reliability observation rounds to assess teacher performance, assess our consistency among observers, and to create specific, effective feedback for teachers. We will also continue our work on establishing Look Fors for performance standards with a focus today on revisiting our work for Standard 3 and then beginning our work on Standard 5.”
(Conduct a brief verbal review of the agenda for the day.)
Aligning Academic Review and Performance Evaluation (AARPE)
Session 3

2. Technical Assistance Initially Offered 2014-2015 1 of 4
Overview and Purpose
To improve instruction and instructional leadership practices by strengthening the alignment between the Performance Standards for Teachers and Principals and the Lesson Planning, Lesson Observation, Professional Development, and Leadership Academic Review Tools.
To develop sample evidence for the sample performance indicators in Teacher Performance Standards for professional knowledge, instructional delivery (planning), assessment of and for learning, and learning environment. The sample evidence for each performance indicator will become a tool that can enhance the division’s observation tools. (District and School Leadership)
“The purpose, roles and responsibilities, outcomes, essential questions, and next steps will be a standing part of each session in order to help all of us keep the main thing the main thing especially given the high stakes for all of us in this room.”
(Review the two bullets.)
“As we discussed earlier, we will be working with our own forms for walkthroughs, observations, and other feedback, etc. This work is not about changing a form.
Regarding Bullet 2: It is our intent that the sample Look For evidence we create during these sessions will support our forms and processes. We will develop the tool so that we can have it in hand for use. The goal is that the same tool(s) and language at the classroom, school, and district levels are defined.
We have looked at the connections between academic review tools and performance indicators. We have developed sample evidence for Standards 1 and 3. It is the beginning. As we continue our work in professional development with staff and in classroom observations, we will revisit the sample evidence to refine it and to add to the list. We will talk more about this later.
We will continue to practice the use of our tools in both this setting and in the school setting.”

3. Technical Assistance Initially Offered 2014-2015 2 of 4
Overview and Purpose (continued)
To develop sample evidence for the sample performance indicators in the Principal Performance Standard for instructional leadership. The sample evidence for each performance indicator will become a tool that can enhance the division’s tools. (District Leadership)
To enhance feedback skills of building and district leaders.
(Review the two bullets)
“Additionally, we will work together to determine when and how you provide feedback? How do you follow-up? What additional support or training would assist you with providing feedback?”

4. Technical Assistance Initially Offered 2014-2015 3 of 4
Outcomes
Each participant will
Set clear and measurable goals for his/her work.
Implement action steps with fidelity.
Monitor and provide feedback.
Adjust based upon monitoring and data.
Repeat the cycle.
“We continue to be committed to modeling what we teach. One important component is to set and abide by specific outcomes for our time together. These continue to be our outcomes for today.”
(Review each bullet)
“Are there any others we should consider?”
(If any additional are agreed upon, make note of it directly on the slide.)

5. Technical Assistance Initially Offered 2014-2015 4 of 4
Roles/responsibilities
District staff will lead this professional development for principals of all second year accredited with warning schools. Update from division leaders
Principals/district staff will use their own work as a starting point and will bring a predetermined number of completed walkthrough feedback forms and formal observation documents to each session throughout the year.
Principals, appropriate district staff, and state OSI-approved personnel will conduct monthly walkthroughs and formal observations per the established timeline.
District staff will support and monitor principals’ delivery of professional development on the sets of sample evidence developed to appropriate school staff.
Lead Turnaround Partner (LTP) will support AARPE training in their work with building leaders and other staff.
“Let’s once again take a few minutes to review our Roles/Responsibilities regarding this technical assistance.”
(Review each bullet adding the following statements. Modify based on your audience.)
“VDOE training materials for each session will be provided to each principal from the district office.
We have already come to consensus on this number at a previous session.
Per our agreement, one day was identified for the principal and designated district leader to conduct walkthroughs and observations for inter-rater reliability. Work completed on this day counts toward the number identified in bullet 2. A calendar has been developed for each school.
Through scheduling and coaching, district staff will support the principal as they conduct professional learning with school staff. A calendar will be developed for this and district staff will attend.
The key is that we are all speaking the same language and using the same tools where appropriate. All of our work by school and district should be tied to the Academic Review tools and performance standards.
Any questions before we proceed?”

6. Technical Assistance Initially Offered 2014-2015
4. Professional Development Norms
Arrive on time and stay the entire session.
Be an active participant. (Complete in-session and between-session work.)
Reserve school/district/personal business for established breaks.
Maintain confidentiality.
Use large sticky notes to indicate additional professional development needed/desired.
(Adjust/modify this slide to meet district expectations. Some suggested remarks follow.)
“We know how challenging it is for you to be away from your school and how easy it can be – especially with technology – to get caught up in what’s going on there instead of being laser focused on the learning going on here. As is the case with any learning experience, setting norms for the day will help ensure a learning environment that is effective for each of you. To that end, let’s review the list.”
(Proceed to review all five bullets, expanding on points as necessary.)
“One additional note--you’ll see that we’ve established a Parking Lot (indicate the chart paper on the wall labeled as such). Throughout our time together, this is the place to note any lingering questions, especially those you’d like answered now but don’t want to ask aloud to the group. It’s also a place to note any additional professional learning needs you discover as we proceed. Use your post it notes from the table, and please post in real time. Sometimes when we wait, we forget. If you are absolutely reluctant to post here, feel free to your facilitator and he/she will post the question anonymously for the group.”

7. Essential Questions
How do the components and criteria of the selected academic review tools align with the Teacher and Principal Performance Evaluation Standards lesson observation tool?
How do we use these tools for school improvement?
How do we know we are making progress?
What are the next steps?
“Virginia’s academic review focused on the alignment of the written, taught, and tested curriculum with tools and supporting resources developed by Strong and Associates. As you know, for the AARPE Academic Review/ Priority technical assistance, we are focusing on the Lesson Planning, Lesson Observations, Leadership, and Professional Development tools as well as selected teacher and principal performance standards. Our work continues around these four essential questions.”
(Read Question 1) “We will spend time looking at this.”
(Read Questions 2 – 4.)

8. Next Steps (Completed by Session 3)
School division report of completed next steps
(Facilitators: Please enter the division’s next steps which should have included: Training dates from AARPE Session 2; Inter-rater Reliability Visits with OSI-approved personnel, division staff, and principals.)

9. Main Areas for Evidence Collection during Observations Where are we?
Standard 1: Professional Knowledge Standard 2: Instructional Planning Standard 3: Instructional Delivery Standard 4: Assessment of/for Learning (formative assessment) Standard 5: Learning Environment Standard 6: Professionalism Standard 7: Student Academic Progress
“The standards in green are those that you have completed and that have been finalized to-date. The standards in blue are the ones that we will continue to refine through this technical assistance. The new teacher performance standard for today’s session is Standard 5: Learning Environment.”

10. Documenting Evidence Tips – Did You…
Avoid terms that express judgment (“neat classroom,” “fun activity,” “caring attitude”)
Avoid words that imply, but do not specify, quantity (“most,” “few,” “several”)
Stick to the five senses
Remember Who, What, When, Where, How
“One purpose of these sessions is to document evidence and, in turn, provide feedback on that evidence. This is all done around a set of defined, agreed upon metrics: the Teacher Performance Standards and Look For.”

11. What is inter-rater reliability?
In AARPE this looks like….
Team of People:
Principal
District representative
OSI-approved personnel
Focusing on in Round 1:
Standard 1 Look Fors
Standard 3 Look Fors
Inter-rater reliability is the consistency between the evidence collection of two or more observers.
“Now we are going to make our transition to our inter-rater reliability work. A quick reminder of our joint definition of inter-rater reliability. It is when observations are made by different people and they are looking for the same strategies or standards being taught - it is important so observations will be similar and you can see an overall picture of what the teacher is capable of doing. It is important to emphasize that we are focused on understanding our observation consistencies and inconsistencies as well as how to address disparities going forward.”

12. Establishing Inter-rater Reliability
Calibrate evidence collection during initial training
Conduct tandem observations and performance reviews with multiple evaluators
“We’ve discussed that one way to establish inter-evaluator reliability from the beginning is through reliability trainings that calibrate evaluators’ evidence collection during observations. After the initial training, the inter-evaluator reliability should be periodically re-established through tandem observations in which multiple evaluators observe the same lesson, compare the evidence they collected, and discuss how they interpreted it. Now that we know the basics about collecting evidence, let’s turn our attention to how we can increase the reliability of our observations and decrease potential biases.
This leads to our next section: decreasing bias during observations. Let’s first look at observer personal bias. This is when observers collect idiosyncratic or irrelevant evidence during lesson observations. We will use a specific protocol to achieve this goal.”
(RESOURCE: OSI Website– Academic Review – Additional Resources for OSI- approved personnel, Principals, and Divisions – Dec. 9 – Providing Effective Feedback.)
The Met Project, 2013

13. Inter-rater Reliability Protocol
Protocol Step 1: Individual: Using your Round 1 evidence documents that you brought today, complete Part I of the Inter-rater reliability Protocol document (pages 1-8) to include noting the specific Look Fors you observed for both Standards 1 and 3 for each of the four teachers.
Protocol Step 2: Partner with your observation team (defined as the staff members who participated in the school-based observation session). Each member shares the Look Fors observed (per teacher/per standard). Discuss any differences (per standard/per teacher). Each member completes the protocol tool pages 9-10.
Protocol Step 3: On your protocol handout, pages 11-14, list the specific questions that the principal will ask the teacher. Share your list with observation team.
Protocol Step 4: Whole group debrief. Each group shares the following for both Standards 1 and 3: (Record responses on the chart paper provided and post when completed.)
a) Differences found in Look Fors for Standards 1 & 3
b) Solutions to avoid these differences in Round 2
c) 2-3 of the feedback questions you’ve agreed upon and why they are important to the teacher feedback cycle.
“We are ready to begin our protocol for debriefing your Round 1 observations and assessing our Inter-rater reliability. Go ahead and clear your spot except for the evidence you brought with you today as well as the Handout you were given when you checked in. As you can see from our protocol, you will start with a self-reflection and we will evolve into a group assessment and we’ll wrap up with a transition to pinpointing specific questions to use for your post-observation conversations with teachers – thereby supporting your growth in providing feedback. Let’s get started with Step 1 above.”
(Review the bullet)
“Remember, Step 1 is independent work. This is the time that you get a sense of your own observation outcomes. While you work, I’ll post slide 10 as a reminder of how our evidence should/should not be documented. Your intention here is to identify which of the specific Look Fors from your collaborative list you observed. You’ll indicate Look Fors directly on your observation forms.”
(Once they begin, circulate and OBSERVE only. Give 20 minutes to start and adjust as needed).
“OK, it appears we are ready to move on to Step 2. The purpose of this step is for your group to begin the conversation about the similarities and differences in your observations. You’ll use pages 9-10 of your handout for this step. This conversation is critical. Pay close attention to the Discuss portion of the handout. Have an in depth discussion that answers each of the questions indicated for each teacher and for both Standards 1 & 3. Each participant should complete their individual form. Go ahead and get started. Remember, you have 4 teachers to discuss so be sure to use your time efficiently and work toward compromise when necessary.”
(Circulate and OBSERVE only. Provide 60 minutes to start and lengthen as necessary. Note that this portion of the conversation is likely to take the longest.)
“It’s time to move on to Step 3. Now that you’ve agreed on your observation similarities and differences, as well as the teachers’ strengths and areas of growth, discuss the feedback that would best suit the teacher and the situation. Individually, document the questions you would pose to each teacher on the handout pages Next, as a group, agree on 2-3 questions per standard per teacher that the principal will ask as a part of the post-conference discussion. Note your questions on the chart paper provided and post it on the wall when you are ready. This gives all of our groups a chance to see a variety of questions. Again, you’ll use pages of your handout for this step.”
(DO NOT distribute chart paper until participants have completed their individual handout pages. Circulate and OBSERVE only. Provide 30 minutes to start. Monitor time and adjust as necessary).
“All right, we are ready for our final step – our whole group debrief. I’m going to give you about 10 minutes to prep your Recorder to respond to the prompts on the slide in front of you. Remember, you are ALL responsible for prepping your Reporter and he/she can phone a friend at any point so be sure to be well versed in your group’s responses.”
(After 10 minutes, begin the debrief and continue until each group has shared. Provide guidance and feedback as is necessary/appropriate.)

14. Enhancing Teacher Quality: Questioning
Standard 5: Learning Environment The teacher uses resources, routines, and procedures to provide a respectful, positive, safe, student-centered environment that is conducive to learning.
Sample Performance Indicators:
5.1 Arranges the classroom to maximize learning while providing a safe environment.
5.2 Establishes clear expectations, with student input, for classroom rules and procedures early in the school year, and enforces them consistently and fairly.
5.3 Maximizes instructional time and minimizes disruptions.
5.4 Establishes a climate of trust and teamwork by being fair, caring, respectful, and enthusiastic.
5.5 Promotes cultural sensitivity.
5.6 Respects students’ diversity, including language, culture, race, gender, and special needs.
5.7 Actively listens and pays attention to students’ needs and responses.
5.8 Maximizes instructional learning time by working with students individually as well as in small groups or whole groups.
“OK, now we will transition back to our work involving identifying Look Fors for our Standards. Today we will begin the work on Teacher Standard 5. These are the Sample Indicators for Standard 5. For this standard, we are looking for evidence: of safe and learning-centered classroom arrangements; clear expectations, rules, and procedures; maximized instructional time; a positive climate; cultural sensitivity and respect for diversity; the teacher actively listening; and a variety of grouping structures to maximize instructional time.”
TQR Teacher Quality Resources, LLC (c) 2005

15. Standard 5 - Look Fors Teachers
Step One: Individual Work - Look through your observations, walkthroughs, and/or written feedback and highlight and label (5.1, 5.2, etc.) any examples of “Look Fors” for Performance Standard 5 Sample Performance Indicators. Step Two: Table Groups– Brainstorm additional examples of “Look Fors” for each Performance Standard 5 Sample Performance Indicator. Create one unduplicated list to be sent to the facilitator’s by your Recorder. (ENTER FACILITATOR ) Step Three: Facilitators share combined lists and assign indicator to each table. Table group discusses examples and determines a final list of sample “Look Fors” for each Performance Standard 5 Sample Performance Indicator. Step Four: Table groups report out. (Group feedback)
“Now we will begin our work on Look For for Standard 5. We will follow a process similar to that for Standards 1 and 3. Take a moment to review the four steps of our process.  Let's begin with Step One.  We will continue to use your ”real work" in this environment - making this relevant to you is critical. To that end, you will start with feedback you provided to teachers just this past school year.  Individually, peruse your selection and highlight any examples of Look Fors you find that can be used specifically for indicators for Performance Standard 5.  Do this work individually. If you do not have your own documents, begin brainstorming Look Fors for the sample performance indicators. Each person should have multiple Look Fors for each indicator by the end of step 1. Begin your work to complete this step. Remember this is INDIVIDUAL work.”
(While participants are working, display slide 10 as a reminder. Circulate and OBSERVE only. Watch time, progress check at minutes and extend time if necessary.)
“Now it's time to begin Step 2. Take the list of Look Fors you created and combine them with the other participants from your table group.  Be sure to vet the list and remove any duplicates and/or any the group deems to be not Look Fors – again note the criteria displayed in the front.  Take the time to review each Look For and make sure you agree that it meets the criteria that we referenced earlier when we talked about evidence. Also be cognizant that your statements are concise and specific- this is not the place for generalities.  In addition to the criteria we displayed earlier, ask yourselves these questions:
is each statement specific;
is each statement concise and observable;
have we avoided generalities;
do we have Look Fors that address both teacher and student behavior;
do we have Look Fors that address what the indicator does look like as well as what it does not look like?
Again, we realize this is a lot to cover; know that your careful review now will go a long way toward making the final list as strong as possible. Once you feel your list is complete – remember each table group should have multiple well written Look Fors for each indicator, please have a member of your group it to (enter presenter name here) - be sure to include the names of each of your table group members in the . You may begin this step. We’ll adjust the time as needed. This is the most important step of the process so please be thorough in your review.”  
(Circulate and OBSERVE only. Watch time, progress check at minutes and adjust to shorten or lengthen as necessary.)
“We've received all lists.  We will work to combine your lists and we will resume this activity at our next Session.”
(Pre-determine whether or not Step Three will occur today or at the start of Session Four.)

16. Questions to Consider:
How Do We Know We Are Making Progress?
Questions to Consider:
Are we focusing on teaching and learning in terms of measurable student progress?
Have we developed a process for progress monitoring at the classroom level, the school level, and the division level?
Do we make adjustments based upon progress monitoring?
“You are the leaders in your school and district . At your level, how do you know you are making progress? Is there a process? Whenever you consider next steps, consider these key questions.”
(One facilitator will take notes about key in the moment remarks/questions throughout the remaining slides.)

17. Classroom, School, Division
How Do We Know We Are Making Progress?
Classroom, School, Division
Set clear and measurable goals
Implement action steps with fidelity
Monitor and provide feedback
Adjust based upon monitoring and data
Repeat the cycle
“Do you have this process in place in such a way that all relevant stakeholders can talk about it at each of these levels? If not, how will you get there? How will our work here help you get there?”
(Provide time for participant responses.)

18. Framework for Inter-Rater Reliability Work
Who: Principal, AARPE Key Division Leader, OSI-approved personnel
When:
One day between Session 2 date and Session 3 date
One day between Session 3 date and Session 4 date
One day between Session 4 date and Session 5 date
These 3 days are identified by the principal, AARPE Division Leader, and OSI-approved personnel and considered as non-negotiable.
What: Recommendations:
4 – 30 minute AND
4 – 15 minute walkthroughs
The school division’s forms will be used by all division staff. (Note: OSI-approved personnel will use VDOE sample observation form and the division’s walkthrough form.)
4. Debriefing: Formal debriefing will be conducted at the next scheduled AARPE session.
(Facilitator will enter the agreed upon date here)
“As we wrap up for the day, this is a reminder about our agreed upon framework. Note that ________ has been determined as the date for our inter-rater observation date.”

19. Next Steps: Division Feedback Skills
Purpose: To enhance feedback skills of building and division leaders.
Questions to consider:
Who provides feedback to principals (formal and informal)?
What tools are used to provide feedback? How are the Principal Performance Standards incorporated in the feedback?
How often is feedback provided in writing?
How is feedback monitored?
What written expectations exist?
What professional development is in place to enhance division leaders’ feedback skills?
Next Session: Division leaders need to bring samples of written feedback provided to principals.
(In addition to supporting the feedback provided by principals to teachers, division leaders will want to align feedback to the Principal Performance Standards. Return to beginning slides to review goals related to division leadership. Discuss academic review findings.)
“Let’s begin our work with Principal Performance feedback by establishing a baseline of our current practices. In your groups, discuss these six questions and come to consensus on your responses. Be prepared to share out in 15 minutes.”
(Circulate and OBSERVE only. This is a chance to see if there are any established practices/expectations on Principal performance feedback. In preparation for the group share, consider having one facilitator take notes about responses, especially should any agreed upon practices emerge.)
“OK, let’s review the questions together. Our goal is to a) identify already established procedures/practices and b) agree on best practices with respect to principal feedback. Let’s start with question 1. Who are the district staff who have the formal responsibility of providing feedback to principals?”
(Allow groups to share their responses. Is there a consensus on a clearly identified person(s) responsible for this? If so, move on. If not, provide the following prompts to support the group in determining who this should be:
Who is the supervisor of principals?
Who is the person(s) most likely to be in a position to observe/monitor the principal?
Who is the person trained on the implementation of the VDOE Principal Performance Evaluation process?
Once a consensus is reached on question 1, move on to question 2.)
“Let’s move on to question 2. What tools are used to provide feedback to principals? How are the Principal Performance Standards incorporated into this process?”
(Allow groups to share their responses. Is there consensus on the tools and the Principal Performance Standards? If not, use these questions to process: What forms/tools have been used to date? What are there benefits/weaknesses? If no standard tool, what do principals need in order to receive effective feedback from district leaders? Have district staff responsible for principal evaluation been trained on the VDOE Principal Performance Standards? If not, what other training/support has been provided to ensure that evaluators of principals are properly prepared to provide feedback? What needs do those district representatives have in order to be properly prepared?)
“All right, question 3. How frequently do we provide feedback and from what input is our feedback formed?”
(Allow groups to share responses. Is there consensus on the frequency and types of feedback? If not, use these questions to process: Is there an established number of formal and informal observations?
Do we observe the principal in a variety of settings? Do we gather data from stakeholders?)
“And finally question 4 – what is the purpose of our feedback to principals? Do our current practices ensure that we accomplish our purpose?”
(Allow group to share responses. Is there consensus on purpose? If not, use these questions to process:
If our purpose is that principals improve their skills as a result of feedback, how do we structure our feedback so that we can measure growth? What voice does the principal have in the feedback process to ensure that our purpose is met? What training/practice do we need as district staff to ensure that we are accomplishing our established purpose?)
If it is not discussed during questions 1-4, prompt the group to discuss questions 5 and 6 in regard to current written expectations in the division (How are the expectations shared? With whom are they shared?) and professional development (Are there on-going practice or professional development sessions where samples of feedback are discussed among leaders?).

20. Next Steps (Completed by Session 4)1.
“As is our practice each session, let’s establish Next Steps to be completed by Session 4.”
(Reminder that this part of the discussion should be led by the key person in the room who has the authority to ensure next steps can be addressed successfully.)

21. Reflection
Principals: What changes will you make as a result of today’s work?
 2. Division Staff: What changes will you make as a result of today’s work?
“As a reflection from today’s work, as well as from the previous sessions, consider how this work will impact your own work in your schools (principals) and divisions (division leaders).
Principals: Share out some ideas of how this work will change your practice.
Division staff: Share out some ideas of how this work will change your practice.”
(Wrap up by offering reminders of key dates as well as what materials participants should be prepared to bring to Session 4.)