Gathering Data for Faculty Evaluation Shelley A. Chapman, PhD Senior Educational Consultant, The IDEA Center

Plan for this Workshop I.Foundations to Gathering Data for Faculty Evaluation II.Logistics of Gathering Data for Faculty Evaluation III.Practicing Classroom Visitations and Observations

I. Foundations Foundations to Gathering Data for Faculty Evaluation Philosophical Underpinnings Administrative Foundations Classroom Observations as Ethnographic Practice Classroom Visitations as the Community of Practice

Philosophical Underpinnings

Assumptions, Beliefs, Values Knowledge Skills Emotions Learning Tree © Shelley A. Chapman 2007

Learning About Faculty Evaluation Assumptions, Beliefs, Values Emotions Knowledge Skills What might be some assumptions, beliefs, and/or values about what a good faculty evaluation system looks like? How do faculty and administrators feel about the System? Are the results being used in a fair and appropriate way? Do faculty and administrator what the system is, which data are collected, how data are used? Can faculty and administrator engage in a holistic system of data collection and Interpretation?

A. Multiple forms of Assessment Student Ratings External Perspective Artifacts Balanced Plan for Summative Evaluation

Evidence of Good Teaching Administer Appropriately Collect 6-8 Reports (more if class size is <10) 30-50% of Overall Evaluation Student Ratings

Evidence of Good Teaching Student Comments-formative Be mindful of standard error of measurement (±.3) Use 3-5 Performance Categories Student Ratings

Evidence of Good Teaching Artifacts Syllabi Graphic Organizers Assignments and project descriptions Rubrics Written Teaching Philosophy/Reflections Samples of Student Work CATs and results

Evidence of Good Teaching Focus Groups of Graduating Students Alumni Surveys Invited Presentations Classroom Observation Classroom Visitation External Perspective

B. Qualitative Research Analyzing at a phenomenon or situation Questions: Why? How? In what way? Goal  Understanding of human or social phenomenon Tends to be inductive

Overarching Goal Feedback Analysis Try Something New Critical Reflection

Administrative Foundations Systematic and Thoughtful Formative and Summative Multi-dimensional and use multiple sources Clear Guidelines should be established* * Adapted from Van Note Chism, H. (2007). Peer review of teaching: A sourcebook. Bolton, MA: Anker.

The Evaluation Program: Link with faculty rewards and awards Present in a straighforward, accurate, and complete way Focus on teaching improvement Use valid and reliable instruments Requires professional judgment to make subjective decisions (even with objective data)

Three-Phase Process for Faculty Evaluation Set Expectations Collect Data Use Data

Create Categories of Performance Below Acceptable Standards Marginal, Needs Improvement Meets Expectations Exceeds Expectations Outstanding Does Not Meet Expectations Meets Expectations Exceeds Expectations

Be Clear Purpose? Summative and formative? Who or what is being reviewed? Outcomes? What if it is strongly negative? Is the person, unit, or program undergoing the review aware of the possible outcomes?

Be Clear (Continued) With whom will the results be shared? What is the nature of my conclusion? Am I authorized to make a decision or am I making a recommendation to a chair or dean? Should I retain copies of the records?* *Adapted from Buller, J.L. (2012). Best practices in faculty evaluation: A practical guide for academic leaders. San Francisco: Jossey-Bass.

Classroom Observation Deductive Approach See Saint Francis University’s “Class Visitation Form” Classroom Observation Form Open Ended (Form A) Framework of Faculty Teaching Performance Evaluation

Classroom Observation Inductive Approach Classroom Observation as Ethnographic Practice

The Process is Observation in the “field” to understand the context… Characterized by “thick” description using the lens of culture

Ethnographic Practice The Product is A report that includes rich descriptions… themes/categories... trends… and some interpretation

Classroom Observation Why? To understand the context of a particular classroom. Who? Trained observer and any class When Mid-way through the course or so What “Observation” in the ethnographic sense How As an ethnographer would take field notes

Classroom Visitation What is the value of peer learning? Gestalt: The whole is greater than the sum of its parts. Faculty learn from their colleagues as they form a “Community of Practice.”

Research on Peer Learning “…peer learning among small groups of teachers was the most powerful predictor of improved student achievement over time.” Quoted by Darling-Hammond, L. (2013). Getting teacher evaluation right: What really matters for effectiveness and improvement. Teachers College Press: NY.

What is a Community of Practice? Group of people who share a concern or a passion for something they do and learn how to do it better as they interact regularly.

Resource: Wenger, E., McDermott, R., & Snyder, W. (2002). Cultivating Communities of Practice: A guide to managing knowledge. Cambridge, MA: Harvard Business Press.

Communities of Practice New Faculty/Adjunct Faculty Tenure Track Faculty Specific type of format Flipped Hybrid Labs or clinicals Large lecture

Communities of Practice Specific type of technique Problem-based Learning Case study method Jigsaw Chalk Talk Think-share-pair

Communities of Practice Specific type technology Clickers Smartboards Video clips Other technology

Communities of Practice Specific type of issues Discipline problems Motivation problems Difficult concepts First-year students/first generation college students Laptops and phones in the classroom

Examples Amanda in Biology for Non-majors Goal: Improve student understanding of biological concepts Help: Tom uses clicker feedback technology for his large political science course. Tom’s Political Science Class 1.How similar is Tom’s class to mine (in demographics)? 2. What type of subject matter is Tom addressing (factual, quantitative, conceptual, theoretical, procedural, etc.) 3. What was his first question for which he used clickers? 4. How did student responses help Tom to inform or adjust his teaching? 5. How often did Tom use clicker feedback throughout the class? 6. What problems occurred with the technology or process? 7. What can I learn from this class that might I adapt and use in my class?

Examples Jeff is teaching a capstone class in the Business/Management major Goal: Help student integrate theory and practice in real life situations Help: Susan uses the case study method in her Marketing class Susan’s Marketing Class 1.How similar is Susan’s class to mine (in demographics)? 2. What types of learning does Susan want her students to apply? Principles? Skills? 3. When did the students receive the case (before or in class)? How long is it? 4. What was Susan’s role with the case? 5. What did students do with the case? Did they work in groups? What was the desired outcome? Presentation? Report? 6. What problems occurred with the process? 7. What can I learn from this class that might I adapt and use in my class?

Examples Jan is a new faculty member, teaching Math in her first semester Goal: Help students to be engaged and on task Juan has been teaching Math successfully at this institution for 8 years Juan’s Class 1.How similar is Juan’s class to mine (in demographics)? 2. What types of learning does Juan want his students to apply? Principles? Skills? Both? 3. How does Juan handle homework? Do students hand it in? Do they use it in class? 4. What was Juan’s role with the homework? Is it graded? 5. Are students distracted by mobile phones and laptops? If so, how does Juan handle that? 6. Were students engaged? If so, why? If not, why not? 7. What can I learn from this class that might I adapt and use in my class?

II. Logistics: Visitation (Peer) Establish Purpose and protocol Identify CoPs Select Tools/ Reflective Questions Invite Participants Prepare Participants Steps to setting up a Classroom Visitation Program

7 Principles for Cultivating a CoP* 1.Design for Evolution 2.Open a Dialogue Between Inside and Outside Perspectives 3.Invite Different Levels of Participation 4.Develop both Public and Private Community Spaces 5.Focus on Value 6.Combine Familiarity and Excitement 7.Create Rhythm for the Community *Wenger, E., McDermott, R., Snyder, W.M. (2002). Cultivating communities of practice: A guide to managing knowledge. Cambridge, MA: Harvard Business Press.

Individualizing Visitations Identify need Match up peer Meet with peer to discuss need Design or select Guiding Questions and Review course materials Visit Class (host faculty introduce visitor) Write short reflection / Debrief Steps to making Classroom Visitation part of a Community of Practice

Visitation Logistics Use for formative purposes (usually for the observer, but can be reciprocal) Visit someone’s class before having someone visit yours Make sure it is a good match—it will meet your specific needs Keep it low stakes and relevant

Before the Visit Ask if the host faculty has a teaching philosophy to share Review the syllabus, where the class is in sequence Review any other course materials pertinent to the visit Ask for any insights about the class that the instructor may want to share

During the Visit Sit in the back of the room Allow yourself to be introduced Use a set of focused questions Take class materials (text, syllabus, etc) Observe and take notes, answering your questions

After the Visit Write a brief reflective statement Meet with the instructor to debrief Plan next steps for your class Invite faculty member to visit as appropriate

II. Logistics: Observations Observer should be trained and should practice Results can be Formative and Summative Meet with the faculty being observed before and after the observation Prepare materials to record observations

Before the Observation Ask if the host faculty has a teaching philosophy to share Ask for any insights about the class that the instructor may want to share Review the syllabus, where the class is in sequence Review any other course materials pertinent to the visit

Review of Materials Formative Purposes : Include discussions of materials in department meetings Encourage a “sharing of materials” session over refreshments Establish protocols for standard syllabus template Provide examples of well-designed assignments Effective and Engaging Provide examples of rubrics

Review of Materials Summative Purposes: Create templates and use them to judge the quality of faculty developed products Create rubrics to use to judge the quality of assignments or assessments

Classroom Observation Log TimeWhat HappenedWhat Was Said

Classroom Observation Log TimeWhat HappenedWhat Was Said 8:05 8:10 8:15 Instructor shut door Students are shuffling papers, opening books. Student comes in late Several students raise hands Female in first row is called on Instructor (I): OK, Class. Let’s begin. Make sure you turned in your homework as you came in. Today we will begin our discussion on the brain. Turn in your textbooks to chapter 5. Is your brain more like a computer or a jungle? Who would like to respond first? Student (S) My brain is a jungle! I am so unorganized! (class laughs)…

Flow of Communication Map Instructor M F F F MMM M M M M M F F F F F F F F F

M F F F F M M M M MM M M M M M M M M MM M M

Examples Business Class with very few women Business Class with students from diverse backgrounds Accounting Class with adjunct instructor who was accused of racial discrimination English instructor who was notorious for being unprepared

During the Observation You will be busy! You won’t get everything! You are capturing all you can. Try to keep up for about minutes

After the Observation Review notes for general impressions Type up report Share report with instructor for feedback Allow instructor to submit written feedback if desired Follow protocol as to who will receive copies of the report Remember that all data are confidential

See Sample Reports Professor Jones Professor Smith

Thank you. Questions?