1. Comments: [NONE] The Art of Meaningful Feedback
Matthew Holley
Scott Renshaw, MD
Jennifer Custer
Jennifer Burba
Indiana University Department of Family Medicine
Introduction: Make some comments about IUSM context, i.e., IUSM is the 2nd largest medical school in the U.S. and is the only one in the state of Indiana. We have 9 campuses across the state. The family medicine clerkship has been offered in a decentralized community based design since the early 1990s. We have over 375 volunteer family physicians currently appointed as teachers of the clerkship. Our family medicine clerkship is a 4-week rotation with feedback provided to learner at mid-rotation and at the end of the rotation. However, we noticing that the comments shared with learners at the completion of the rotations were minimal if any.
Thus, we developed an interactive workshop to highlight the importance of meaningful feedback during a faculty development session for the department faculty. Today will highlight three different activities we used that we believe others can take with them and use at their own institution.
Presented at the Society of Teachers of Family Medicine Annual Conference
May 2013

2. Session objectives The goals of today’s session are:
To articulate the faculty member’s role in giving meaningful feedback.
To discern the importance of giving meaningful feedback to students.
To describe the characteristics of meaning feedback and to practice in creating meaningful feedback.
As a result, the purpose of today’s session is to:
Articulate the faculty member’s role in the feedback process.
Discern the importance of meaningful feedback from both the student’s and faculty’s perspective.
Practice in developing meaningful feedback comments.

3. Types of feedback Formative Summative
Formative feedback is diagnostic information given before the work is completed. It is intended to help the student revise and improve the work.
Summative feedback is a final analysis of the work, on which final grades are based. It can also point toward improving future work of the same type.
What is feedback?
Think of feedback as any response you make to students' efforts. Sometimes the feedback is written as and when you mark students' work; sometimes it is face to face as in a tutoring session. Marks, reports and annotations or comments on written assignments are the most obvious types of feedback that spring to mind.
In fact, consciously or unconsciously, we are giving students feedback all the time: facial expressions, tone of voice, and especially words all say a lot about our expectations and the quality of the responses to those expectations. In some cases, even the failure to provide feedback is a form of feedback; it could be saying, 'You are not important enough for me to spend my time on.'
It is also useful to think about how students' reactions to our feedback can, in turn, serve as feedback to us, telling us about the quality of our feedback to them, and
Formative feedback is diagnostic information given before the work is completed. Like feedforward, it is intended to help the student revise and improve the work.
Summative feedback is a final analysis of the work, on which final grades are based. It can also point toward improving future work of the same type.

4. Why give Feedback? Learners want feedback Learners need feedback
Learners improve from feedback

5. Learners want feedback
Faculty behaviors said to be important by residents & medical students (Schultz, 2001):
96.7% - Be open to questions
95.6% - Give constructive feedback
95.1% - Show enthusiasm for teaching
90.7% - Give timely feedback
60.8% - Observe clinical interaction directly
(Schultz, 2001)
Why is feedback so important?
In a technical psychological sense, there is little learning without feedback. However there is more learning if the following conditions surround it:
Feedback contributes to learning when it is noticed
Feedback contributes even more to learning when the learner reflects on the lessons for next time
Feedback containing advice originating from the student's own recent work is more likely to be given attention, understood and acted upon
Feedback can fuel constructive reflection by the learner.
Feedback on performance is so important that Gibbs and Simpson (2004) said that feedback to the students on their assignments was the single most powerful influence on student achievement. In fact, an item about the importance and quality of feedback shows up on almost all student evaluations of teaching. So in this chapter we will look at some strategies for giving students feedback in ways that will be the most helpful in improving their learning.
The bottom line in the above set of examples is that in giving feedback of any type, you should remember that there is another party to the communication. While you're marking student work, try to imagine you are having a conversation with another person; this might help your feedback come across to them as words from a real person who is trying to help them do better. Your comments should help that other person understand not just what he or she did wrong, but what could be done the next time to do better. And it needs to be provided in a timeframe that allows that feedforward concept to work.

6. However, they don’t get enough…
Learners report feedback is not common
In one study, only 8% of residents were “very satisfied” with feedback process
“Corrective” feedback much less common than “reinforcing feedback
80% of residents reported never or infrequently receiving corrective feedback
17% of residents reported never or infrequently receiving reinforcing feedback
(Isaacson, Posk, et al. ,1995)

7. Activity #1: Recognizing meaningful feedback
Facilitator: To help you understand the student experience of receiving feedback, we like for each of you to work with those around your seat.
We have place throughout the audience envelopes with comments given to students after completing their Family Medicine rotation.
As a group determine a process for categorizing the comments into two categories – meaningful and not-meaningful. As a group, create your own definition of “meaningful” which will help shape your process for sorting the student feedback.
In addition to creating two separate groups, please write your group’s definition or criteria for meaningful feedback.
[Allow 10 minutes for groups to create definition and sort groups]
Debrief:
Have one group provide it’s definition of “meaningful” feedback.
Ask other groups how their definitions were similar or different.
How easy or difficult was it to separate the comments into separate piles?

8. Characteristics of good feedback
It develops self-assessment
It encourages dialogue
It clarifies what good performance is
It is timely
It guides learning
It encourages self-belief
It develops self-assessment
Good feedback helps learners develop self-assessment skills by encouraging them to recognize strengths and weaknesses in their own performance. Try providing students with a marking checklist to hand in with their work, or asking them to grade themselves. Talk to students whose estimates are out of line with yours to find the cause (under-confidence, misunderstanding of task...).
It encourages dialogue
Good feedback encourages dialogue about learning. Instructors often assume student understanding of feedback without checking. If you can get feedback on the feedback, students may be able to help you help them do better by telling you what they do and don't understand in your feedback.
It clarifies what good performance is
Good feedback clarifies what a good performance is. This is the principle of don't just say it's wrong, say what good performance is in this case and what their work needs to make it better.
It is timely
Good feedback comes in time to allow a second chance at learning. Be sure to return your comments while the work is still fresh in the student's mind, and in time to guide any resubmissions. Peer-to-peer assessment and feedback (formulated as soon as the work is handed in) can also be a good way of achieving this goal.
It guides learning
Good feedback provides information to students about their learning. The idea here is helping students understand underlying principles of learning the skill so that they don't focus only on details but rather learn about the less obvious and difficult to define qualities to work toward.
It encourages self-belief
Good feedback encourages positive motivational beliefs. The characteristic that has the highest correlation with success is the belief that you can be successful. Therefore feedback that points the way to future success (feedforward) is more motivating than the grade itself.

9. Activity #2: MAKING feedback MEANINGFUL
Facilitator: Using the process we just completed, we have determined what the characteristics of meaningful feedback are. Now, we’re going to test your knowledge once again.
The next slide is going to provide you with a comment given on a learner’s evaluation. You will need to decide if the feedback is meaningful or not. You can present these statements in a variety of ways. For example, if you were also trying to get your faculty comfortable with using an audience response system (such as clickers) you could incorporate that into this activity. For our purposes today, we going to simply do a thumbs up/thumbs down approach.

10. Good or bad?
Thumbs up
Thumbs down
Learner response is generally poor quality so you give the feedback:
"This is not your best work, but it could be improved by…"
This is a strong example because the focus is on improvement rather than errors, and the phrasing also implies that you believe the student is capable of doing better (in comparison with, say, "This is not good work; here is what’s wrong with it…")
1. Thumbs up
2. Thumbs down

11. Good or bad?
Your learner is making a consistent process error, so you tell them: "You are repeatedly making this error. Be more careful."
The feedback needs to imply that you are working with the student and provide a constructive suggestion, for example: "Note that we’ve seen this problem before. I would suggest doing a final read looking for this particular problem on the next assignment."
1. Thumbs up
2. Thumbs down

12. Good or bad?
The learner’s work is thoughtful and well-prepared. Your feedback is "Good job."
While the praise is nice, it is not very informative. It is best to point out some of the highlights so the student knows what is being praised.
1. Thumbs up
2. Thumbs down

13. WHY DON’T WE GIVE FEEDBACK?
Need safe, reciprocal learning climate – in other words, we are fearful of giving negative feedback
Faculty lack the knowledge and skills to give feedback – Remind them to practice, practice, practice
Not enough time
In some cases, lack of observation time

14. Giving negative feedback
Top Ten Reasons Faculty Avoid Talking to Learners About a Problem “I don’t want to be the bad guy…” “I don’t want to upset him/her…” “I don’t want to make a big deal out of this…” “I don’t want to ruin someone’s career..” “I don’t want to end up in court…” “I’m not sure how he will react…” “I (or others) may have contributed to the problem..” “I know he/she realizes it and will not do it again…” “I think it’s too late in the training or the year…” “I don’t like confrontation…”
Giving “negative” feedback is our second greatest fear. Negative feedback in a constructive effective manner can result in our greatest reward – a successful student.

15. Giving negative feedback
Technique for giving negative feedback:
Focus on observed behaviors
Plan what you will say/write, how you will say/write it
Be timely
Permit the student to self-evaluate first
Reinforce the positive behaviors
Be descriptive
Be specific
Develop an action plan
Summarize the main points and potential action plan for improvement
Keep in mind that:
The student will probably not be surprised…
The student values your feedback
You are probably not the first to identify a problem…
Nip it in the bud…
The school is there to support you…

16. Developing feedback skills
The “Feedback” Sandwich
What was done right
What was done wrong
What to do next time to improve
In some cases you might want to include some back feedback delivery techniques such as the feedback sandwich.

17. Developing feedback skills
The Interactive Feedback Sandwich
Ask – the learner how he/she thought he/she did
Tell – the learner what you observed
Ask – the learner how you can help her improve
(Konopasek, L. ACGME Outcome Project.
New York, NY: Presbyterian Hospital)
Now, here we have the fancier interactive feedback sandwich.
The keys to using this approach are to plan to be concise (limit it 2 or 3 points), be unequivocally clear, and be aware of both verbal and non-verbal cues.
ASK: Used as self-assessment
“How do you think things are going…?”
“How do you think we are doing…?”
“How are your…..coming along?”
Tell: You present/give your feedback
React to the learner’s observation
Include both positive and corrective elements - “I saw that you appeared to have difficulty with….” “I noticed that….” “….commented to me about…”
Give reasons in the context of expectations and objectives
Ask: Discuss next steps to guide learner
“Do you believe my interpretation of the situations is valid?”
“Do you understand my point?”
“What do you think you (we) might do to move you to the next level…?”
“Is there anything you can think of that might help with….”

18. Creating time for feedback
Making It Easier
Performance rubric
Checklist
End of clerkship evaluation form
Audio-recording

19. Activity #3: Putting it all together
Over the course of this session, we have discussed our role in creating meaningful feedback, the importance of providing feedback to students, and even what meaningful feedback looks like. Our final activity provides us with an opportunity to put it all together….
We are passing out a copy of the current student feedback form to each of you.
In a second, we will be showing you a short clip highlight one of the interactions of a recent medical student under your supervision.
After seeing the clip, use the current evaluation form to construct an evaluation using the meaningful feedback techniques we’ve discussed.
[Play YouTube Video]
After video – give individuals time to work on evaluation form.
After finishing as individuals – encourage participants to share their written comments with others at the table.
Return focus to large group and ask if anyone is willing to share their written comments.

20. Negative feedback example:
What feedback would you provide? What action plan would you develop?
Video alternatives:
Leadership Style – Grey’s Anatomy -
Communication – House -
Team Dynamics – Patch Adams -

21. Questions/Comments

22. Resources
Ciccarelli, M. (2012, April). Feedback. Workshop presented at the meeting of the Leadership in Academic Medicine Program (LAMP), Indianapolis, Indiana. Gibbs, G and Simpson, C. (2004). Conditions under which assessment supports students‘ learning. Learning and Teaching in Higher Education, 1 (pp.3-31). Retrieved from Epigeum, Ltd. (2011). Marking and giving feedback. Online module. Isaacson, J.H., Posk, L.K., Lifster, D.G., & Halperin, A.K. (1995). Resident’s perceptions of the evaluation process. Journal of General Internal Medicine, 10(supplement), p. S89. Schultz, K.W., Kirby, J., Delva, D., Goodwin, M., Verma, S., Birtwhistle, R., Knapper, C., & Seguin. R. (2004). Medical students’ and residents’ preferred site characteristics and preceptor behaviours for learning in the ambulatory setting: A cross-sectional survey. BMC Medical Education, 4(12). Retrieved from