1. The Metacognitive Benefits of Self- and Peer Review Edward F. Gehringer Department of Computer Science North Carolina State University Our work in peer and self-review is funded by Google and NSF http://tinyurl.com/CHEP-SPR

2. Signup sheet  Name  E-mail address  Institution, if not in your domain name

3. Outline  What is metacognition?  Advantages of peer review?  Advantages of self-review?  Self-review  Peer review  Combining peer and self-review  Reviewing vs. being reviewed  Accuracy of peer grading Gehringer, Benefits of self- and peer review efg@ncsu.edu http://tinyurl.com/CHEP-SPR

4. What is metacognition? Gehringer, Benefits of self- and peer review efg@ncsu.edu http://tinyurl.com/CHEP-SPR

5. What is metacognition?  “Thinking about one’s thinking”  I am engaging in metacognition if … I notice I am having more trouble learning A than B. it strikes me that I should double-check C before accepting it as fact. I realize I don’t understand what the instructor wants me to do if I sense that I should write down D so I don’t forget it –John Flavell, 1976 Gehringer, Benefits of self- and peer review efg@ncsu.edu http://tinyurl.com/CHEP-SPR

6. Advantages of peer review? Gehringer, Benefits of self- and peer review efg@ncsu.edu http://tinyurl.com/CHEP-SPR

7. Some advantages of peer review  Feedback is more extensive quicker scalable  Peers may understand students’ comprehension problems better and communicate in language more easily understood  Can’t blame the reader!  Forces students to think metacognitively Gehringer, Benefits of self- and peer review efg@ncsu.edu http://tinyurl.com/CHEP-SPR

8. Advantages of self-review? Gehringer, Benefits of self- and peer review efg@ncsu.edu http://tinyurl.com/CHEP-SPR

9. Some advantages of self-review  Every student gets immediate feedback  Judging correctness of answers deepens students’ understanding of material  Students become aware of own strengths, progress, gaps Gehringer, Benefits of self- and peer review efg@ncsu.edu http://tinyurl.com/CHEP-SPR

10. Self-review: pitfalls  Confounding effects Early studies asked students to predict how well they would do in course. Effort is often included as a criterion.  What is a good self-assessment? Is it the same kind of assessment an instructor would give? Gehringer, Benefits of self- and peer review efg@ncsu.edu http://tinyurl.com/CHEP-SPR

11. Rubrics  Why use a rubric? Tell students what to look for “Fairness” in assessment  Students can help create the rubric  How detailed? Gehringer, Benefits of self- and peer review efg@ncsu.edu http://tinyurl.com/CHEP-SPR

12. Rubric advice http://tinyurl.com/CHEP-SPR

13. Self-review vs. self-grading  Self-grading is more effective on items requiring lower-order cognitive skills. Freshmen vs. seniors …  Would students cheat? ? ? … Wilkowski et al., 2014  70.3% of students in 1 st MOOC awarded selves full credit  9.9% of these submissions were blank or nonsense  8.5% were evid. plagiarized Gehringer, Benefits of self- and peer review efg@ncsu.edu http://tinyurl.com/CHEP-SPR

14. Combining self-review & peer review  was the first review system to incorporate both peer and self-review.  Step 1: Instructor gives students a writing assignment.  Step 2: Students submit their document to CPR. Gehringer, Benefits of self- and peer review efg@ncsu.edu http://tinyurl.com/CHEP-SPR

15. Calibration  Step 3: CPR gives the student 3 calibration documents for review.  Basic idea: Training course for reviewers How they do  how much credence they get Before students review peers, they get 3 works to review  1 exemplary Their agreement with instructor  Reviewer Competency Index Others have known defects Gehringer, Benefits of self- and peer review efg@ncsu.edu http://tinyurl.com/CHEP-SPR

16. How Does CPR Work?  Step 4: After calibration, CPR gives student 3 peer documents for review.  Step 5: CPR gives the student his or her document for self-review.  Step 6: CPR provides a detailed report of the peer review and the self-review. Gehringer, Benefits of self- and peer review efg@ncsu.edu http://tinyurl.com/CHEP-SPR

17. The Review Process  The review process is the same for calibration, peer review, and self-review.  The student answers content questions for each document.  The student answers style questions for each document.  The student assigns a score to each document on a scale of 1 to 10. Gehringer, Benefits of self- and peer review efg@ncsu.edu http://tinyurl.com/CHEP-SPR

18. How self-review fits in  Students get full credit for self-reviewing if their grades agree  within the range set by the instructor between their self-assessment and  the weighted value of their peer reviews. Instructor is notified to check reviews/self-assessment when reviewers of a student did not “train well.” Gehringer, Benefits of self- and peer review efg@ncsu.edu http://tinyurl.com/CHEP-SPR

19. Better to give than to receive?  Would students learn more by … peer-reviewing, or being peer-reviewed? Gehringer, Benefits of self- and peer review efg@ncsu.edu http://tinyurl.com/CHEP-SPR

20. What did students appreciate? Kulkarni et al., 2013, ACM TCHI Gehringer, Benefits of self- and peer review efg@ncsu.edu http://tinyurl.com/CHEP-SPR

21. Other results Shah-Nelson, 2014, OLC Annual Conference Gehringer, Benefits of self- and peer review efg@ncsu.edu http://tinyurl.com/CHEP-SPR

22. “To give is better than to receive”  “Givers”: reviewed, but received no peer feedback  “Receivers”: got peer feedback, but didn’t review  “Receivers” made more signif. gains in writing, especially those at lower proficiency levels. Gehringer, Benefits of self- and peer review efg@ncsu.edu http://tinyurl.com/CHEP-SPR

23. Peer review vs. peer grading  Formative—text feedback  Summative—Likert scale  Should peer review be used summatively? Gehringer, Benefits of self- and peer review efg@ncsu.edu http://tinyurl.com/CHEP-SPR

24. Peer grading—how reliable?  Two studies on Coursera MOOC [2013] Piech et al.: ≥ 26% of grades ± 5% from “ground truth.” Kulkarni et al.: 40% of grades off by 1 letter grade! But …  simplistic calibration  this was, after all, a MOOC Gehringer, Benefits of self- and peer review efg@ncsu.edu http://tinyurl.com/CHEP-SPR

25. Factors affecting accuracy  Students can only effectively peer-review what they understand.  This suggests overweighting scores assigned by better-performing students. Gehringer, Benefits of self- and peer review efg@ncsu.edu http://tinyurl.com/CHEP-SPR

26. Summary  Metacognition: thinking about thinking  Peer review: rapid, extensive feedback  Self-review: deepens understanding  Self-grading: don’t use by itself!  Can combine peer & self-review  Learn more as reviewer than reviewee  Peer grading: not ready for prime time Gehringer, Benefits of self- and peer review efg@ncsu.edu http://tinyurl.com/CHEP-SPR