1. Educator Evaluation Workshop: Gathering Evidence, Conducting Observations & Providing Feedback MSSAA Summer Institute
July 26, 2012
Massachusetts Department of Elementary and Secondary Education 1

2. Agenda The Role of Evidence in the 5-Step Cycle Artifacts of Practice
Three types of evidence
Roles & responsibilities
Artifacts of Practice
Observations & Feedback
Tips & Strategies
Resources
Massachusetts Department of Elementary and Secondary Education

3. Intended Outcomes
At the end of this session, participants will be able to:
Define “evidence of practice” and understand the role of artifacts, observations, and feedback in the 5-Step Cycle
Understand the value of frequent, unannounced observations with targeted feedback
Identify tools and processes for gathering and organizing evidence that will make evidence collection and feedback more doable in their schools.
Massachusetts Department of Elementary and Secondary Education

4. Every educator is an active participant in the evaluation process
Continuous
Learning
Every educator & evaluator collects evidence and assesses progress
Collaboration and Continuous Learning are the focus
Massachusetts Department of Elementary and Secondary Education
Massachusetts Department of Elementary and Secondary Education 4

5. “I was evaluated today…” sound familiar?
Evaluation  Observations

6. Multiple sources of evidence inform the Performance Rating
Three categories of evidence must be collected for each educator:
Multiple measures of student learning, growth and achievement
Judgments based on observations and artifacts of professional practice
Additional evidence relevant to standards
student/staff feedback ( )
Massachusetts Department of Elementary and Secondary Education

7. Multiple sources of evidence inform the performance rating
This graphic explains the summative performance rating in new educator evaluation framework.
The left column includes the three categories of evidence used in evaluation: Products of Practice (including observations, unit plans, schedules and the like), Multiple Measures of Student Learning (ranging from classroom assessments to MCAS Growth Percentile Scores and MEPA when available), and Other Evidence including, eventually, student feedback.
(A note here: MEPA – the Massachusetts English Proficiency Assessment for English Language Learners – is being replaced by a better assessment being developed by a national consortium of states called WIDA (pronounced “WEE-DUH.”))
The middle column represents the translation of the evidence into an assessment of performance on each of four standards in addition to an assessment of progress on goals.
The right column is the single summative performance rating.
Starting from the left, using a rubric, an evaluator lines up evidence from the three categories on the left to determine a rating on each of the four Standards (for teachers, the four standards are Curriculum, Planning and Assessment, Teaching all Students, Family Outreach & Engagement, and Professional Culture) and an assessment of progress on both student learning and professional practice goals.
Massachusetts Department of Elementary and Secondary Education
Massachusetts Department of Elementary and Secondary Education

8. What does this look like?
Products of Practice related to Standards
Multiple Measures of Student Learning
Other Evidence related to Standards
Artifacts
Teacher-developed unit assessments
Grade level meeting notes
Parent/teacher communication log
PLC meeting notes
Observations
Notes/feedback from short, frequent observations (inside/outside classrooms)
Notes and feedback from announced observations
Student work (quizzes, homework, presentations, etc.)
Portfolios
Performance assessments (including arts, vocational, health & wellness)
Interim assessments
State or district assessments
Student and staff feedback (2013/2014 school year)

9. Implementation Responsibility
Educator Responsibilities:
Documenting action steps completed.
Collecting, organizing and submitting evidence to demonstrate progress toward professional practice and student learning goals.
Evaluator Responsibilities:
Observing practice on a regular basis and providing targeted feedback on performance
Making resources and supports available.
Identification of common artifacts/evidence.

10. Products of Practice: Artifacts

11. It starts with the Educator Plan…
Student Learning Goal: In order to ensure mathematical literacy in each of the three content areas for 8th grade geometry (8.G), I will incorporate at least one essay question into each unit assessment that requires elaboration of mathematical reasoning so that 80% or more of my ELL students demonstrate proficiency on essay questions on the end of the year 8th grade geometry assessment.
Student Learning Goal(s) Planned Activity
Action
Supports/Resources from School/District
Timeline/Benchmark or Frequency
By October 1, I will assess ELL student comprehension and knowledge with formative assessments
By October 15th, I will share this data with my department team and instructional coach and solicit feedback on instructional strategies related to teaching mathematical literacy to ELL students.
By October 30th, I will develop writing objectives for each unit and integrate them into unit assessments.
From November through May, after each unit, I will disaggregate assessment data for ELL students, focusing on mathematical literacy. I will track their progress and adjust instruction as necessary.
Formative geometry assessment
Monthly department team meetings
Monthly one-on-one data analysis with instructional coach and ELL specialist
Unit assessments
Oct. 1: review formative assessment results for my ELL students
Oct. 15: share formative assessment results with department team and instructional coach and identify at least three instructional strategies related to building mathematical literacy with ELL students.
Evidence: meeting notes, 3 strategies
2. October 30th: developing writing objectives for each unit
Evidence: written objectives, essay questions
3. November—June: Administer unit assessments in three content areas and analyze student performance on essay questions
Evidence: student data from essay questions in at least three unit assessments

12. Importance of Strategically Collecting Artifacts
Artifacts should be a sample that demonstrates educator performance and impact
Aligned with educator goals, the Model System Teacher Rubric or school goals
Number of artifacts to collect varies by educator
Artifacts can provide evidence of more than one Standard-Indicator
An annotated summary of Grade 5 unit assessment results can include evidence of practice related to I.C. (Analysis), IV.A.1 (Reflective Practice), and IV.C (Collaboration).

13. Lessons from Early Adopters: Collecting Evidence
Quality not quantity
Guidelines and exemplars will help
Prioritize based on focus areas
Massachusetts Department of Elementary and Secondary Education

14. Products of Practice: Observations & Feedback

15. Observations
The regulations require a minimum of one unannounced observation.
The Model System recommends short, frequent unannounced observations for all educators, as well as at least one announced observation for non-PTS educators and struggling educators.
Observations are one more source of evidence, to support and live alongside artifacts of practice.
They serve a formative AND summative purpose, as do artifacts. They’re the vehicle through which evaluators can provide targeted, ongoing feedback to educators and maintain a dialogue around teaching and learning.
Massachusetts Department of Elementary and Secondary Education

16. Why short, frequent observations?
More opportunities to see patterns of practice
Flexibility in scheduling
Promotes ongoing conversation around teaching and learning
Facilitates observations beyond the classroom
Is 5-15 minutes enough?

17. Observation and Feedback
School-level Administrator Rubric (I-D-2): Typically makes at least two unannounced visits to classrooms each day and provides targeted, constructive feedback to all educators. Acknowledges effective practice and provides redirection and support for those whose practice is less than proficient. Superintendent Rubric (I-D-2): Typically makes at least three unannounced visits to each school to observe principal practice every year and provides targeted, constructive feedback to all administrators. Acknowledges effective practice and provides redirection and support for those whose practice is less than proficient.
We need to go into more detail on Observation here since it is an important source of evidence about educator performance. Here are two more excerpts from the model rubrics for principal (school-level administrator) and superintendent.
Going forward, observations will look quite different from the observations that are done in many schools today. In the model system, principals will be making at least two unannounced observations to classrooms every day, and superintendents are expected to make at least three unannounced visits to each school over the course of the year. The school-level classroom observations described here, in particular, may sound undoable, especially if you’re thinking about it in terms of the “classic” observation that requires scheduling an announced, 45-minute visit to a classroom and setting up a pre- and post-conference.
HANDOUT: Strategies & Suggestions for Observations
Take a moment to read the section on frequent, unannounced observations. Think about how this type of observation differs from what you currently do, and how it will reshape the way we think about observations with regard to educator evaluation.
Will this be a big culture shift for you?
Massachusetts Department of Elementary and Secondary Education 17

18. Principles of high quality observations
Frequent
Focused
Inside/Outside the Classroom
Useful & Timely Feedback
The rubric is not an evaluation tool, but a guide to help identify trends and patterns of practice over time.

19. Feedback
“[O]bservers must learn how to capture classroom events in literal notes, and to talk productively with the teacher about it afterward in a way that is evidence-based and productively points toward actionable improvement.” –John Saphier

20. Principles of Good Feedback
Verbal as well as written
Focused on a few key areas
Based on evidence
Tied to Standards of effective practice
Offers reinforcement for areas of effective practice
Facilitates self-reflection on areas of practice that need refinement and guides the teacher in thinking beyond the lesson observed
“The Proficient performance descriptors represent the expected Standard—the bar we expect all experienced teachers to demonstrate over time.”
Instruct school teams that they will now have the opportunity to begin looking at the performance descriptors for a particular Indicator and corresponding elements found in Standards I and II, starting with Proficient.
Ask participants to count off from 1–7 until every individual has been assigned a number.
Place the numbered tabletop cards at each table, and instruct participants to temporarily “re-sort” into these new “teams,” with the 1’s gathering together, the 2’s gathering together, and so on.
Massachusetts Department of Elementary and Secondary Education
Massachusetts Department of Elementary and Secondary Education

21. Tips & Strategies

22. PLAN
COMMUNICATE EXPECTATIONS
ORGANIZE

23. 1. PLAN
The more concrete the Educator Plan, the easier it is to identify and collect artifacts
Identify common artifacts all or most educators will be expected to collect (unit assessments, parent-teacher logs, etc.)
Share examples of high-quality, valuable evidence during faculty or team meetings
Demonstrate example artifacts that provide evidence of more than one Standard-Indicator

24. 2. Communicate ExpectationsOR

25. 2. Communicate Expectations
Artifacts should be a sample that demonstrates educator performance and impact
Submitted evidence should be tied to educator goals, Standards or Indicators, or school goals
Provide everyone with a clear idea of how and when to share products of practice
? Paper? Online cloud?

26. 3. ORGANIZE Calendar observations
Adopt a process for organizing artifacts and observation notes by Standard/Indicator and/or goals
Paper-based, -driven, or online repository
Sample tools for evidence collection and organization

27. Sample Tools for Evidence Collection and Organization
Included in your packet
Sample Evidence Tool
Completed by . . .
Artifact Cover Page
Educator or Evaluator
(the person who identifies the artifact)
Observation Evidence Collection Tool
Observer/evaluator

28. Massachusetts Department of Elementary and Secondary Education

29. Next Steps – Suggestions for Principals
Read “Strategies and Suggestions for Observations” (p. 39 of the School-Level Planning & Implementation Guide)
Identify options for collecting and organizing evidence at your school and establish a protocol for all educators
Work with your administrative team to set a calendar for observations and evaluations based on the distribution of educators by plan type at your school
Massachusetts Department of Elementary and Secondary Education

30. Massachusetts Model System for Educator Evaluation
Resources
Massachusetts Model System for Educator Evaluation
Massachusetts Department of Elementary and Secondary Education
Massachusetts Department of Elementary and Secondary Education

31. School-Level Planning & Implementation Guide
Content Overview
The Massachusetts Model System for Educator Evaluation
Step 1: Self-Assessment
Step 2: Goal Setting and Plan Development
Step 3: Implementation of the Plan
Step 4: Formative Assessment and Evaluation
Step 5: Summative Evaluation
Appendices: Forms for Educator Evaluation, Setting SMART Goals
This is a quick snapshot of what you will see in the School Guide. The guide itself is structured around the five steps of the evaluation cycle: Self-Assessment, Goal Setting & Plan Development, Implementation, Formative Assessment/Evaluation, and Summative Evaluation. The related forms appear in the Appendix along with other resources including guidance about setting “SMART” goals.
HANDOUT: The 5-Step Educator Evaluation Cycle: Train-the-Trainer Modules
One additional resource will be available to every district and school this spring. ESE will be publishing seven “train-the-trainer” modules with facilitator guides designed to help school-level leadership teams learn about the 5-step cycle. ESE will subsidize approved vendors to offer these modules regionally. The seven sequential modules are as follows:
1 – Overview
2 – Unpacking Rubrics
3 – Self-Assessment and Goal Proposal
4 – S.M.A.R.T. Goal and Educator Plan Development
5 – Gathering Evidence through Artifacts
6 – Gathering Evidence through Observation
7 – Rating Educator Performance
An eighth module will follow this on Rating Educator Impact on Student Learning based on District-Determined Measures, after ESE publishes guidance on Phase 2 of the Educator Evaluation Framework.
Massachusetts Department of Elementary and Secondary Education
Massachusetts Department of Elementary and Secondary Education 31

32. ESE Evaluation Resources
What’s coming?
Summer 2012
Guidance on District-Determined Measures
Training Modules with facilitator guides, PowerPoint presentations, and participant handouts
List of approved vendors
Updated website with new Resources section
Newsletter
Massachusetts Department of Elementary and Secondary Education

33. ESE Evaluation Resources
What’s coming?
Fall/Winter 2012
Solicit and review feedback on Model System; update
Research & develop student and staff feedback instruments
Collect and disseminate best practices
Collect and vet assessments to build a repository of district measures
Internal collaboration to support cross-initiative alignment
EX: Support for use of rubric for teachers of ELLs aligned to RETELL initiative
Massachusetts Department of Elementary and Secondary Education

34. For More Information and Resources:
Visit the ESE educator evaluation website: Contact ESE with questions and suggestions: Presenter: Claire Abbott – 34
Massachusetts Department of Elementary and Secondary Education