Presentation on theme: "Peer evaluation in TBL: A student-developed approach"— Presentation transcript:
1Peer evaluation in TBL: A student-developed approach Derek R. Lane, Ph.D.Associate DeanCollege of Communications and Information StudiesH. Lester Reynolds Engineering Endowed ProfessorJune 1, : : Room C150
2TBL Conference 2007 REFLECTIONS People have problems working togetherTBL is distinctive and importantDee Fink“Students don’t like doing peer evaluations.”Interesting examples of TBL Self/Peer AssessmentCooperative Learning SkillsSelf-directed LearningInterpersonal SkillsRudolph Navari
3TBL Conference 2007 REFLECTIONS TBL and the Business Strategy Game Robert A. Herring, III Winston-Salem State UniversitySample Survey of Student AttitudesLearning Assurance ReportLeadership Skills SCALECollaboration and Teamwork SCALEMaking Student Thinking Visible Jim Sibley University of British ColumbiaStrategies to Create Engaging and Reportable TBL application-oriented activities“Residual Dissent”
4TBL Conference 2007 REFLECTIONS iPeer 2.0 can help facilitate TBL peer evaluationsSophie Spiridonoff / Jim Sibley (UBC)Medical students have mixed opinions about satisfaction with peer evaluationDean Parmlee, Dan DeStephen, Nicole Borges (Wright State U.)
6What Makes a Team Work?“Individual commitment to a group effort: this is what makes a team work, a company work, a society work, and a civilization work.”--Vince Lombardi,Former Green Bay Packers Coach
7Driving Questions RQ1What is the purpose of peer evaluation? RQ2 How do we improve our methods of helping students assess professionalism?Students support that which they help to create.
8Session Overview Purpose of Peer Evaluation: A TBL Retrospective Characteristics of High Performing TeamsImportance of Helpful FeedbackStrategies for Establishing Meaningful Formative (PROCESS) and Summative (OUTCOME) Assessment
9I. Purpose of Peer Evaluation in TBL A TBL Retrospective
10TBL RetrospectiveMichaelsen, L.K., & Black, R. H. (1994). Building Learning Teams: The Key To Harnessing the Power of Small Groups in Higher Education. Growth Partners.
11Performance Evaluation Three Essential Components of Grading SystemIndividual PerformanceIndividual Accountability for Reading/HomeworkAssess Higher-Level Thinking SkillsGroup PerformanceProvides Incentives to Support CohesivenessJustification for Putting Effort into Group WorkPeer Evaluation
12Peer Evaluation Solves Important Motivational Problems Provides Incentives for Equitable Group ParticipationRemoves students’ fear that they will have to choose between receiving a low grade on the group assignments and having to “carry” group work (when other group members fail to do their fair share)
13Performance Evaluation Three Essential Components of Grading SystemIndividual PerformanceGroup PerformancePeer EvaluationGive sufficient weight to each of the componentsMust be clear to students that the instructor thinks each is important (minimum 10%)Must be responsive to student concerns for fairness and equity (reduce social loafing)
14Necessary Components of TBL Tasks Heterogeneous composition of diverse interdependent work teamsClear, specific, and widely shared team goals which encourage group cohesionSufficiently difficult and meaningful team activities that do not allow one person to accomplish the task alone.Regular, descriptive, specific, relevant, timely and usable internal peer feedbackExternal comparisons which are emphasized through immediate and ongoing feedback about organizational performances relative to other teams.
16Team Performance and Effectiveness In a well-run team, the overall performance is superior to that of individual efforts“The whole is greater than a sum of its parts”In a poorly run team, the overall performance is worse than what would be obtained by linearly combining the contributions of many individuals
18The Team Performance Curve ImpactThe Team Performance CurveHigh-PerformanceTeamRealTeamWorkingGroupPotentialTeamFrom: Katzenbach & Smith, The Wisdom of Teams:Creating the High-Performance Organization.Used by permission.Pseudo-TeamTeam Effectiveness
19Building Team Performance Establish urgency, demand performance standards and directionMembers should be selected for skill and skill potential--not personalityPay particular attention to first meetings and actionsSet clear rules of behaviorSet immediate performance-oriented tasks and goalsChallenge the group with fresh facts and informationSpend lots of time togetherExploit the power of positive feedback, recognition, and rewardKatzenbach & Smith, 1993
20Eight Characteristics of High Performance Teams A Clear, Challenging GoalA Results-Driven Structure (clear roles, effective communication)Competent Team MembersUnified CommitmentCollaborative ClimateStandards of ExcellenceExternal Support and RecognitionEffective Principled LeadershipLaFasto, F., & Larson, C. (2001). When teams work best: 6,000 team members and leaderstell what it takes to succeed. Thousand Oaks, CA: Sage.
21III. Importance of Helpful Feedback The importance of process feedback can beillustrated with a simple example. If a personBoards a plane in San Francisco and wants to fly toWashington DC they can expect to see a mountainRange in the first minutes. If all they see isOcean then they are obviously going in the wrongDirection. For students enrolled in TBL classes, thisIs what process feedback in the form of peer evaluationOught to be!
22Formative vs. Summative Peer Evaluation Formative (process) peer evaluation procedures can potentially provide a results-driven structure and establish the criteria or standards of excellence to ensure mutual accountability.Summative (outcome) peer evaluations provide an overall standardized assessment of the extent to which individuals contributed to team success. (Helping Behavior)
27Preliminary Individual Peer Evaluation Criteria Reflect on past group experiencesConsider + - experiencesWhat are the four or five most important issues that contributed to the group success or failure?Develop a list of preliminary criteria you believe are important for group success / peer evaluation
28Preliminary List of Evaluative Criteria Instructions: Develop a list of evaluative criteria that you believe is important for students to use when working in TBL instructional teams in order to be successful.
29Final TBL Evaluative Criteria Instructions:1) Make introductions and share contact information.2) After introductions have been made and contact information shared, decide on a name for your team. The final decision must be unanimous and the name should reflect positively on the qualifications of the individuals in your group. Provide the decision on the line below.Group # ___ Team Name: __________________________________
30Final TBL Evaluative Criteria Share individual preliminary criteria with all group members. Consolidate the preliminary list of criteria and develop a final group list of the criteria that the group is comfortable using for peer evaluations.Use the directions on the next page to develop procedures that incorporate the group criteria that will allow efficient feedback.Submit the final peer evaluation procedures and criteria as well as the Group # and Firm Name along with all of the individual criteria sheets at the beginning of the next class.
31Team Peer Evaluation Procedures & Criteria Instructions: Before proceeding, read the four-page article written by Michaelsen and Schultheiss’ (1988) Making Feedback Helpful.Develop a system for providing performance feedback to the members of your team. The performance feedback system should include:1. Provide a statement of the team goals and objectives that you intend to achieve. These goals should reflect an integration of individual team members’ goals for the course.2. A description of how your team intends to collect the data on which the feedback will be based. Please include a copy of a specific peer grading form that clearly indicates how data will be collected.3. A description of the feedback process you intend to use. Please specify:a. When the feedback will be given? Be specific.b. Who will give it? Be specific.4. Assess the difficulties that you are likely to encounter in implementing your performance feedback system. How will the team overcome these difficulties?5. A statement of how the system provides input into the summative helping behavior grade at the end of the class.
32Characteristics of Helpful Feedback descriptive, not evaluative, and is owned by the sender (not anonymous)specific, not general (focus on specific issues and behaviors)honest and sincereexpressed in terms relevant to the self-perceived needs of the receivertimely and in contextdesired by the receiverusable; concerned with behavior over which the receiver has controlMichaelsen, L.K., & Schultheiss, E. E. (1988). Making feedback helpful. Organizational Behavior Teaching Review 13(3),
33Team Peer Evaluation Procedures & Criteria Your performance feedback system will be evaluated using the following criteria:1. Is the team collecting data they will need to support the achievement of course goals?2. Will the procedures they intend to use support the achievement of their objective(s)?3. Are the procedures they intend to use consistent with the seven characteristics of helpful feedback?4. Are the procedures practical (i.e., can they be implemented effectively given the specific situation in which they will be used)?5. Have they accurately anticipated the problems they are likely to encounter in implementing their performance feedback system and have they developed strategies for overcoming the difficulties?
34Key Elements Critical to Cooperation Positive Interdependenceindividuals succeed only when group succeedsIndividual Accountabilitylearn together to perform better as an individualPromotive Interactionhelp, assist, support, encourage, praise effortsSocial Skills (Professionalism)communication, leadership, decision-making, trust-building, and conflict management skillsGroup Processing
35Code of CooperationEVERY member is responsible for the team's progress and success.Attend all sessions and be on time.Listen to and show respect for the contributions of other members; be an active listener.Criticize ideas, not persons.Resolve conflicts constructively.Pay attention - avoid disruptive behavior.Avoid disruptive side conversations.Only one person speaks at a time.Everyone participates -- no one dominates.Be succinct, avoid long anecdotes and examples.No rank in the room.Attend to your personal comfort needs at any time but minimize team disruption.HAVE FUN!Adapted from the Boeing Commercial Airplane Group
36IU School of Medicine - South Bend TBL Self/Peer Assessment Criteria Cooperative Learning SkillsAttendanceListeningInteraction / ParticipationInformation SharingDefer to Needs of the GroupSelf-Directed LearningPreparationDepth of KnowledgeIdentifies Limits of KnowledgeProblem SolvingConfidenceInterpersonal SkillsGives instructive feedbackAccepts instructive feedbackShows sensitivity and concerns for othersClarity in rationale/reasoning processThis is the first mention of a Code of Cooperation and it is an important one. Presenting this Code as a way of defining desirable team norms and acceptable interpersonal communication is very helpful.The Code sets a norm for behavior - a code of ethics, if you will.Teams should add to this code as desired and as the need arises to address necessary new norms. The code should be viewed as a “living document.” In fact, it is advisable to request additions at the time you present this.Jackson, Hamlett, & Navari
37TBL and the Business Strategy Game Robert Herring, IIIWinston-Salem State U.This is the first mention of a Code of Cooperation and it is an important one. Presenting this Code as a way of defining desirable team norms and acceptable interpersonal communication is very helpful.The Code sets a norm for behavior - a code of ethics, if you will.Teams should add to this code as desired and as the need arises to address necessary new norms. The code should be viewed as a “living document.” In fact, it is advisable to request additions at the time you present this.
38Levine, R. E. , Kelly, A. , Karakoc, T. , & Haidet, P. (2007) Levine, R. E., Kelly, A., Karakoc, T., & Haidet, P. (2007). Peer evaluation in a clinical clerkship: Student attitudes, experiences, and correlations with traditional assessments. Academic Psychiatry, 31(1),
39Who seeks formative feedback? 360 Year 3 Undergraduate Medical Students at University of Aberdeen, UKLess than half the students (46%) collected their formative feedback.Females (p=.004) and higher achievers (p=.020) were more likely to seek formative feedback.Sinclair & Cleland (June, 2007)Journal of Medical Education
40Civil Engineering Design Team Criteria AttendancePunctualityAttitudeRespectPreparednessCompleted ReadingsCompleted HomeworkReady for MeetingsPreparationGoal OrientedFlexibilityQuality Work Completed the First TimeAccountabilityTrustworthy and ReliableGives Proper Notice if Can’t Attend a MeetingIndividual InitiativeTeam commitmentLeadershipWillingness to accept alternative viewpoints
41Little and Cardenas (2001) Quality of Technical Work Ability to CommunicateAbility to Provide LeadershipCommitment to Team and ProjectDemonstrated Effectiveness
42Helping BehaviorSummative (outcome) peer evaluations provide an overall standardized assessment of the extent to which individuals contributed to team success. (Helping Behavior)Standardized base 10n-1 X 10 (6-1)*10 = 50 pts to assignMean of group scoresPercent multiplier100 point distribution (peer*GRAT)
43Successful Teams employ self examination and assessment change basic operating assumptions when necessarymonitor their own interaction patterns and progressare willing to make changesare more attentive to group processesbegin their discussion by attempting to analyze the problem before trying to search for viable solutions
44Session Summary Purpose of Peer Evaluation: A TBL Retrospective Characteristics of High Performing TeamsImportance of Helpful FeedbackStrategies for Establishing Meaningful Formative (PROCESS) and Summative (OUTCOME) Assessment
45Student-Developed Peer Evaluation Procedures & Criteria Feedback—both formative (process) feedback and summative (helping behavior) feedback should be direct, clear, and ensure that group members are accountable to the rest of the team. The value of the process feedback is that all of the members of the team have the opportunity to discuss the feedback and make performance improvements over the course of the semester.
48Ability to Analyze the Problem Hirokawa & Gouran’s Four Essential Functions that Decision-Making Groups Must Perform WellAbility to Analyze the ProblemAbility to Identify Appropriate Criteria for Making a DecisionAbility to Develop Alternative Choices from Which to ChooseAbility to Evaluate the Positive and Negative Aspects of Alternative Choices Prior to Making a Decision**Order is not important but all functions must be performed well!