Aug. 9, 2007Gehringer: Improving Course Materials … Through Peer Review … Expertiza: Improving Course Materials and Learning Outcomes through Peer Review of Student Work Edward F. Gehringer Dept. of Computer Science North Carolina State University Supported by NSF DUE under a CCLI grant NCSU Center for Teaching and Learning NCSU LITRE (Learning in a Technology-Rich Env.)

Aug. 9, 2007Gehringer: Improving Course Materials … Through Peer Review … Outline  Introduction  Expertiza Rationale  Demo  Experiment 1: Improving a Textbook  Experiment 2: Review Wiki Submissions

Aug. 9, 2007Gehringer: Improving Course Materials … Through Peer Review … Introduction  Electronic peer review is students reviewing other students’ work over the Web.  Building resources through electronic peer review gets students working together to improve others’ learning experiences, helps them learn, by performing tasks that are similar to real-world responsibilities, gives them experience in writing their ideas up for an audience of their peers, allows each cohort to “stand on the shoulders” of students in earlier classes.

Aug. 9, 2007Gehringer: Improving Course Materials … Through Peer Review … Homework, traditionally …  For students to demonstrate mastery of the subject. Every student does the same thing— redundant effort. Work is graded and thrown away, never benefiting anyone but the student who did it.  Now the best work can be published, to help others learn.

Aug. 9, 2007Gehringer: Improving Course Materials … Through Peer Review … Outline  Introduction  Expertiza Rationale  Demo  Experiment 1: Improving a Textbook  Experiment 2: Review Wiki Submissions  Conclusion

Aug. 9, 2007Gehringer: Improving Course Materials … Through Peer Review … The Rationale  Improve student learning  Improve teaching  Better utilize resources

Aug. 9, 2007Gehringer: Improving Course Materials … Through Peer Review … Improving student learning  The Expertiza platform improves student learning in these ways: Integrates active and cooperative learning  Active learning allows students to take responsibility for their own learning. Extends active learning to out-of-classroom activities and distance education  DE has been a roadblock to the use of active learning … students viewing lectures remotely can work only by themselves. Discourages plagiarism  Multiple deadlines and milestones make it impossible to submit a finished product obtained from an external source.

Aug. 9, 2007Gehringer: Improving Course Materials … Through Peer Review … Improving teaching  The Expertiza platform improves teaching in these ways: Increases the supply of examples/homework problems/test questions  Students are assigned to make up such examples/questions, and these are peer reviewed. Focuses students on explaining/understanding the concepts that are hardest to master  “Write an example to clarify the hardest concept in lecture k.”

Aug. 9, 2007Gehringer: Improving Course Materials … Through Peer Review … Improving resource utilization  Some work is peer-graded, so teaching assistants can spend more time working with students and less time grading. Having inadequate TA support no longer limits the amount and kinds (e.g., design problems) of homework that can be assigned. Students rely more on their peers for help, less on the course staff.  Makes teaching large classes an advantage! Large classes can produce more and better resources.

Aug. 9, 2007Gehringer: Improving Course Materials … Through Peer Review … Outline  Introduction  Expertiza Rationale  Demo  Experiment 1: Improving a Textbook  Experiment 2: Review Wiki Submissions  Conclusion

Aug. 9, 2007Gehringer: Improving Course Materials … Through Peer Review …

Aug. 9, 2007Gehringer: Improving Course Materials … Through Peer Review … Outline  Introduction  Expertiza Rationale  Demo  Experiment 1: Improving a Textbook  Experiment 2: Review Wiki Submissions  Conclusion

Aug. 9, 2007Gehringer: Improving Course Materials … Through Peer Review … Improving an OOD Text  Our class Master’s level course CS & ECE Substantial DE enrollment  In Fall 2005, we used a new object- oriented design text for the first time, Dale Skrien’s, An Intrduction to Object- Oriented Design and Design Patterns Using Java

Aug. 9, 2007Gehringer: Improving Course Materials … Through Peer Review … Three Homework Assignments  Improve an explanation of a topic covered in the text.  Create a new example of a concept covered in the text.  Write a new exercise for a chapter in the text. All students did not do these exercises in the same order.

Aug. 9, 2007Gehringer: Improving Course Materials … Through Peer Review … The Review Process: How It Begins  Classroom discussion of the most difficult topics  Students select a topic from a list. Several students are allowed to select the same topic, But the number of slots is limited.

Aug. 9, 2007Gehringer: Improving Course Materials … Through Peer Review … Signing up for a First-Round Topic

Aug. 9, 2007Gehringer: Improving Course Materials … Through Peer Review … Student Choices

Aug. 9, 2007Gehringer: Improving Course Materials … Through Peer Review … The Review Rubric  Students review a submission based on the following questions (for examples). Does the example fully illustrate the concept being explained? Is the example easy to understand, i.e., as clear as it could be and still illustrate the concept? Does the example model the real world, i.e., could it be implemented in practice? Is the example code elegant? Does the example use up-to-date Java code or UML? Does the text that accompanies the example explain it well? Is the example more useful than the examples in this section of the book? Does the example seem to be original? Other: How would you rate the submission on factors not reflected in the scores on other questions?

Aug. 9, 2007Gehringer: Improving Course Materials … Through Peer Review … After the Initial Review  Resubmission phase. 2–7 days to revise work in response to reviewer comments.  Grading phase. 3–7 days to make final comments and assign scores.  Review of review phase. Students review each other’s reviews.

Aug. 9, 2007Gehringer: Improving Course Materials … Through Peer Review … Results  17 student submissions selected for text!

Aug. 9, 2007Gehringer: Improving Course Materials … Through Peer Review … Outline  Introduction  Expertiza Rationale  Demo  Experiment 1: Improving a Textbook  Experiment 2: Review Wiki Submissions  Conclusion

Aug. 9, 2007Gehringer: Improving Course Materials … Through Peer Review … Teaching with Wikis  A wiki is, essentially, a Web site that can be edited by any user.  Homework done on wikis promotes collaboration between students.  Problem: How to assess so much writing.  Solution: Peer review.

Aug. 9, 2007Gehringer: Improving Course Materials … Through Peer Review … Our Approach  Students select from a set of topics for a weekly wiki assignment.  The students are given class time to work on the topic in groups.  Then they take a couple of days to finish up their submission.  Their submission is peer-reviewed by other students.  They have a chance to revise their submission.  Students rate the contributions of their partners.  Instructor considers the reviews, revisions, and partner evals in assigning grades.

Aug. 9, 2007Gehringer: Improving Course Materials … Through Peer Review … Students Seem to Benefit  Wiki seems to be preferred as a submission mechanism. “The wiki was really fantastic for creating pages because it guarantees a uniform style for everything, putting the major focus on the content created, rather than formatting issues. It also helped format content so that it was more understandable.”  Contributions are more extensive than we have seen when files are submitted and resubmitted.  Peer review is the only scalable solution. Expertiza gives a framework for review.

Aug. 9, 2007Gehringer: Improving Course Materials … Through Peer Review … Other Uses  Lecture annotations  Code reviews: Contributions to OSS projects  Class “proceedings” in a research course Survey papers or research papers  FAQ in a service-learning course

Aug. 9, 2007Gehringer: Improving Course Materials … Through Peer Review … Future Work  “Peer-assisted” review  Total Quality Management  Automated reviews of reviews

Aug. 9, 2007Gehringer: Improving Course Materials … Through Peer Review … Conclusion  Students can do good work— work that will help their classmates learn the material work that will help the instructor improve the class work that allows each class to “stand on the shoulders” of earlier classes.  The Expertiza approach enables faculty to introduce these exercises into their courses.

Aug. 9, 2007Gehringer: Improving Course Materials … Through Peer Review … For more info …   Ed Gehringer,