1. More Writing But Less Grading: Calibrated Peer Review™ Share The Future IV - Tempe, AZ John Wise The Pennsylvania State University

2. Engineering Instructional Services eis@engr.psu.edu Penn State’s Engineering Instructional Services College of Engineering, University Park PA College of Engineering, University Park PA Associate Dean for Undergraduate Studies, Robert Pangborn Associate Dean for Undergraduate Studies, Robert Pangborn Support and Training for Faculty and Teaching Assistants Support and Training for Faculty and Teaching Assistants

3. Engineering Instructional Services eis@engr.psu.edu Review of Bloom’s Taxonomy Knowledge Knowledge Comprehension Comprehension Application Application Analysis Analysis Synthesis Synthesis Evaluation Evaluation Bloom (1956) Taxonomy of educational objectives: the classification of educational goals

4. Engineering Instructional Services eis@engr.psu.edu Problem Statement Large Classes in Particular: Large Classes in Particular:  Predominantly Lecture  Multiple Choice / “Scantron” Grading  Feedback and Writing Difficult All Classes All Classes  Student “Final” Papers Often First Drafts  Product vs. Process Orientation  Concept Misunderstandings

5. Engineering Instructional Services eis@engr.psu.edu History of Calibrated Peer Review (CPR) Large Molecular Science Sections @ UCLA Large Molecular Science Sections @ UCLA Science-Based Model Science-Based Model  Research, Write, Peer Review Now Can Serve Any Discipline and Educational Level Now Can Serve Any Discipline and Educational Level

6. Engineering Instructional Services eis@engr.psu.edu How It Works Create (or Select) Assignment Create (or Select) Assignment Set Timing and Standards Set Timing and Standards Manage Assignments Manage Assignments Review Results Review Results

7. Engineering Instructional Services eis@engr.psu.edu Create Assignment Choose Source Choose Source Define Writing Assignment Define Writing Assignment Write Exemplar Write Exemplar Write Moderate / Poor Examples Write Moderate / Poor Examples Generate Style and Content Questions Generate Style and Content Questions Assess Own Documents for Calibration Assess Own Documents for Calibration (sample)

8. Engineering Instructional Services eis@engr.psu.edu Set Timing and Standards Control Over Text Entry Times Control Over Text Entry Times Control Over Calibration Standards Control Over Calibration Standards Control Over Scoring Scheme Control Over Scoring Scheme

9. Scoring Template: High Difficulty Calibrations To master each calibration students must: Answer 3.0 of 4 (75%) STYLE questions correctly, and Answer 5.3 of 7 (75%) CONTENT questions correctly, and NOT deviate by more than 2.0 points from the RATING of a calibration text Reviews To master reviews students must: NOT deviate by more than 2.0 points from the AVERAGE RATING of the reviewed text Self-Assessment To master self-assessment students must: NOT deviate by more than 1.0 points from the AVERAGE RATING of their text to receive full credit, or NOT deviate by more than 2.0 points from the AVERAGE RATING of their text to receive half credit

10. Directions: Enter the points you want associated with each assignment stage. All points must sum to 100. Each point total can range from 0 to 100. 1.Text quality:XX points 2. Calibrations: XX points over 3 Calibrations 3. Reviews: XX points over 3 Reviews 4. Self- assessment: XX points TOTAL:must sum to 100 points

11. Engineering Instructional Services eis@engr.psu.edu Management Student Reminders Student Reminders Special Requests Special Requests  Late Submissions  Technical Difficulties

12. Engineering Instructional Services eis@engr.psu.edu Review Results Scan Student Results Scan Student Results Identify “Problem” Scores Identify “Problem” Scores Ability to Randomly Review Some or All Student Submissions Ability to Randomly Review Some or All Student Submissions  Can also participate AS a student!

13. Engineering Instructional Services eis@engr.psu.edu Student Name Overall Grade (Out of 100) Text Score (Out of 10) Reviewer Competency Index (RCI) (1-6)Complete [name]91.817.276finished [name]9486finished [name]9175finished

14. Engineering Instructional Services eis@engr.psu.edu Student Point of View Review Materials Review Materials Write Write Calibrate Calibrate  Review three examples, compared to professor Review 3 Random Peers Review 3 Random Peers Review Self Review Self

15. Engineering Instructional Services eis@engr.psu.edu Calibrated Peer Review ™ (CPR) Available from UCLA Available from UCLA  http://cpr.molsci.ucla.edu http://cpr.molsci.ucla.edu Contact me for sample assignment Contact me for sample assignment  jwise@engr.psu.edu jwise@engr.psu.edu  814-865-4020 Contact UCLA regarding traveling workshops Contact UCLA regarding traveling workshops

16. More Writing But Less Grading: Calibrated Peer Review™ Share The Future IV - Tempe, AZ John Wise The Pennsylvania State University