2. Aug. 9, 2007Gehringer: Improving Course Materials … Through Peer Review … efg@ncsu.edu Expertiza: Improving Course Materials and Learning Outcomes through Peer Review of Student Work Edward F. Gehringer Dept. of Computer Science North Carolina State University Supported by NSF DUE under a CCLI grant NCSU Center for Teaching and Learning NCSU LITRE (Learning in a Technology-Rich Env.)

3. Aug. 9, 2007Gehringer: Improving Course Materials … Through Peer Review … efg@ncsu.edu Outline  Introduction  Expertiza Rationale  Demo  Experiment 1: Improving a Textbook  Experiment 2: Review Wiki Submissions

4. Aug. 9, 2007Gehringer: Improving Course Materials … Through Peer Review … efg@ncsu.edu Introduction  Electronic peer review is students reviewing other students’ work over the Web.  Building resources through electronic peer review gets students working together to improve others’ learning experiences, helps them learn, by performing tasks that are similar to real-world responsibilities, gives them experience in writing their ideas up for an audience of their peers, allows each cohort to “stand on the shoulders” of students in earlier classes.

5. Aug. 9, 2007Gehringer: Improving Course Materials … Through Peer Review … efg@ncsu.edu Homework, traditionally …  For students to demonstrate mastery of the subject. Every student does the same thing— redundant effort. Work is graded and thrown away, never benefiting anyone but the student who did it.  Now the best work can be published, to help others learn.

6. Aug. 9, 2007Gehringer: Improving Course Materials … Through Peer Review … efg@ncsu.edu Outline  Introduction  Expertiza Rationale  Demo  Experiment 1: Improving a Textbook  Experiment 2: Review Wiki Submissions  Conclusion

7. Aug. 9, 2007Gehringer: Improving Course Materials … Through Peer Review … efg@ncsu.edu The Rationale  Improve student learning  Improve teaching  Better utilize resources

8. Aug. 9, 2007Gehringer: Improving Course Materials … Through Peer Review … efg@ncsu.edu Improving student learning  The Expertiza platform improves student learning in these ways: Integrates active and cooperative learning  Active learning allows students to take responsibility for their own learning. Extends active learning to out-of-classroom activities and distance education  DE has been a roadblock to the use of active learning … students viewing lectures remotely can work only by themselves. Discourages plagiarism  Multiple deadlines and milestones make it impossible to submit a finished product obtained from an external source.

9. Aug. 9, 2007Gehringer: Improving Course Materials … Through Peer Review … efg@ncsu.edu Improving teaching  The Expertiza platform improves teaching in these ways: Increases the supply of examples/homework problems/test questions  Students are assigned to make up such examples/questions, and these are peer reviewed. Focuses students on explaining/understanding the concepts that are hardest to master  “Write an example to clarify the hardest concept in lecture k.”

10. Aug. 9, 2007Gehringer: Improving Course Materials … Through Peer Review … efg@ncsu.edu Improving resource utilization  Some work is peer-graded, so teaching assistants can spend more time working with students and less time grading. Having inadequate TA support no longer limits the amount and kinds (e.g., design problems) of homework that can be assigned. Students rely more on their peers for help, less on the course staff.  Makes teaching large classes an advantage! Large classes can produce more and better resources.

11. Aug. 9, 2007Gehringer: Improving Course Materials … Through Peer Review … efg@ncsu.edu Outline  Introduction  Expertiza Rationale  Demo  Experiment 1: Improving a Textbook  Experiment 2: Review Wiki Submissions  Conclusion

12. Aug. 9, 2007Gehringer: Improving Course Materials … Through Peer Review … efg@ncsu.edu

13. Aug. 9, 2007Gehringer: Improving Course Materials … Through Peer Review … efg@ncsu.edu Outline  Introduction  Expertiza Rationale  Demo  Experiment 1: Improving a Textbook  Experiment 2: Review Wiki Submissions  Conclusion

14. Aug. 9, 2007Gehringer: Improving Course Materials … Through Peer Review … efg@ncsu.edu Improving an OOD Text  Our class Master’s level course CS & ECE Substantial DE enrollment  In Fall 2005, we used a new object- oriented design text for the first time, Dale Skrien’s, An Intrduction to Object- Oriented Design and Design Patterns Using Java

15. Aug. 9, 2007Gehringer: Improving Course Materials … Through Peer Review … efg@ncsu.edu Three Homework Assignments  Improve an explanation of a topic covered in the text.  Create a new example of a concept covered in the text.  Write a new exercise for a chapter in the text. All students did not do these exercises in the same order.

16. Aug. 9, 2007Gehringer: Improving Course Materials … Through Peer Review … efg@ncsu.edu The Review Process: How It Begins  Classroom discussion of the most difficult topics  Students select a topic from a list. Several students are allowed to select the same topic, But the number of slots is limited.

17. Aug. 9, 2007Gehringer: Improving Course Materials … Through Peer Review … efg@ncsu.edu Signing up for a First-Round Topic

18. Aug. 9, 2007Gehringer: Improving Course Materials … Through Peer Review … efg@ncsu.edu Student Choices

19. Aug. 9, 2007Gehringer: Improving Course Materials … Through Peer Review … efg@ncsu.edu The Review Rubric  Students review a submission based on the following questions (for examples). Does the example fully illustrate the concept being explained? Is the example easy to understand, i.e., as clear as it could be and still illustrate the concept? Does the example model the real world, i.e., could it be implemented in practice? Is the example code elegant? Does the example use up-to-date Java code or UML? Does the text that accompanies the example explain it well? Is the example more useful than the examples in this section of the book? Does the example seem to be original? Other: How would you rate the submission on factors not reflected in the scores on other questions?

20. Aug. 9, 2007Gehringer: Improving Course Materials … Through Peer Review … efg@ncsu.edu After the Initial Review  Resubmission phase. 2–7 days to revise work in response to reviewer comments.  Grading phase. 3–7 days to make final comments and assign scores.  Review of review phase. Students review each other’s reviews.

21. Aug. 9, 2007Gehringer: Improving Course Materials … Through Peer Review … efg@ncsu.edu Results  17 student submissions selected for text!

22. Aug. 9, 2007Gehringer: Improving Course Materials … Through Peer Review … efg@ncsu.edu Outline  Introduction  Expertiza Rationale  Demo  Experiment 1: Improving a Textbook  Experiment 2: Review Wiki Submissions  Conclusion

23. Aug. 9, 2007Gehringer: Improving Course Materials … Through Peer Review … efg@ncsu.edu Teaching with Wikis  A wiki is, essentially, a Web site that can be edited by any user.  Homework done on wikis promotes collaboration between students.  Problem: How to assess so much writing.  Solution: Peer review.

24. Aug. 9, 2007Gehringer: Improving Course Materials … Through Peer Review … efg@ncsu.edu Our Approach  Students select from a set of topics for a weekly wiki assignment.  The students are given class time to work on the topic in groups.  Then they take a couple of days to finish up their submission.  Their submission is peer-reviewed by other students.  They have a chance to revise their submission.  Students rate the contributions of their partners.  Instructor considers the reviews, revisions, and partner evals in assigning grades.

25. Aug. 9, 2007Gehringer: Improving Course Materials … Through Peer Review … efg@ncsu.edu Students Seem to Benefit  Wiki seems to be preferred as a submission mechanism. “The wiki was really fantastic for creating pages because it guarantees a uniform style for everything, putting the major focus on the content created, rather than formatting issues. It also helped format content so that it was more understandable.”  Contributions are more extensive than we have seen when files are submitted and resubmitted.  Peer review is the only scalable solution. Expertiza gives a framework for review.

26. Aug. 9, 2007Gehringer: Improving Course Materials … Through Peer Review … efg@ncsu.edu Other Uses  Lecture annotations  Code reviews: Contributions to OSS projects  Class “proceedings” in a research course Survey papers or research papers  FAQ in a service-learning course

27. Aug. 9, 2007Gehringer: Improving Course Materials … Through Peer Review … efg@ncsu.edu Future Work  “Peer-assisted” review  Total Quality Management  Automated reviews of reviews

28. Aug. 9, 2007Gehringer: Improving Course Materials … Through Peer Review … efg@ncsu.edu Conclusion  Students can do good work— work that will help their classmates learn the material work that will help the instructor improve the class work that allows each class to “stand on the shoulders” of earlier classes.  The Expertiza approach enables faculty to introduce these exercises into their courses.

29. Aug. 9, 2007Gehringer: Improving Course Materials … Through Peer Review … efg@ncsu.edu For more info …  http://research.csc.ncsu.edu/efg/expertiza  Ed Gehringer, efg@ncsu.edu