1. Expertiza: Rubric-Based Peer Assessment of Team Projects Edward F. Gehringer Dept. of Computer Science North Carolina State University Supported by NSF DUE under a CCLI grant NCSU Faculty Center for Teaching and Learning NCSU STEM Initiative Center for Advanced Computing and Communication

2. 10/15/2015Gehringer, Rubric-based peer assessment of team projects, efg@ncsu.edu Outline  Expertiza Rationale  Team Assignments  Expertiza “Virtual” Demo  Forthcoming Features  Workshop

3. 10/15/2015Gehringer, Rubric-based peer assessment of team projects, efg@ncsu.edu Introduction  Expertiza began as a peer-review system for co-production of course resources.  Building resources through electronic peer review gets students working together to improve others’ learning experiences, helps them learn, by performing tasks that are similar to real-world responsibilities, gives them experience in writing their ideas up for an audience of their peers, allows each cohort to “stand on the shoulders” of students in earlier classes.

4. 10/15/2015Gehringer, Rubric-based peer assessment of team projects, efg@ncsu.edu Homework, traditionally …  For students to demonstrate mastery of the subject. Every student does the same thing— redundant effort. Work is graded and thrown away, never benefiting anyone but the student who did it.  In Expertiza, students (or teams) sign up for different work  Now the best work can be reused, to help others learn.

5. 10/15/2015Gehringer, Rubric-based peer assessment of team projects, efg@ncsu.edu Major advantages  Expertiza discourages plagiarism Students are not doing the same work. Their work must go through multiple review cycles  Expertiza better utilizes scarce resources Students do most of the reviewing. Instructor/TA spend less time grading.  With Expertiza, large classes are an advantage!

6. 10/15/2015Gehringer, Rubric-based peer assessment of team projects, efg@ncsu.edu Why team assignments?  Teamwork mimics the work environment  Teaming skills are important to success  Teams can tackle larger projects, which diminishes the effort of integration

7. 10/15/2015Gehringer, Rubric-based peer assessment of team projects, efg@ncsu.edu How does Expertiza support teams?  All team members have access to a common submission area. Any member can revise the submission.  Individuals review teams. This simplifies reviewing, and provides for more reviews.  Teammates assess each other’s contribution to the team. This score can be factored into the student’s grade.

8. 10/15/2015Gehringer, Rubric-based peer assessment of team projects, efg@ncsu.edu “Virtual” demo  Team formation By instructor, from the user interface By instructor, by importing a file By students, by issuing invitations  Reviewing  Feedback from author to reviewer  Teammate review

9. 10/15/2015Gehringer, Rubric-based peer assessment of team projects, efg@ncsu.edu The login screen Let’s log in as an instructor …

10. 10/15/2015Gehringer, Rubric-based peer assessment of team projects, efg@ncsu.edu The instructor’s homepage

11. 10/15/2015Gehringer, Rubric-based peer assessment of team projects, efg@ncsu.edu Setting up a wiki assignment

12. 10/15/2015Gehringer, Rubric-based peer assessment of team projects, efg@ncsu.edu Let’s create some teams …

13. 10/15/2015Gehringer, Rubric-based peer assessment of team projects, efg@ncsu.edu In the beginning, there are no teams

14. 10/15/2015Gehringer, Rubric-based peer assessment of team projects, efg@ncsu.edu But let’s create one …

15. 10/15/2015Gehringer, Rubric-based peer assessment of team projects, efg@ncsu.edu Team created; let’s add members

16. 10/15/2015Gehringer, Rubric-based peer assessment of team projects, efg@ncsu.edu 1 st member is added; now for the 2 nd …

17. 10/15/2015Gehringer, Rubric-based peer assessment of team projects, efg@ncsu.edu Team now has two members

18. 10/15/2015Gehringer, Rubric-based peer assessment of team projects, efg@ncsu.edu “Virtual” demo  Team formation By instructor, from the user interface By instructor, by importing a file By students, by issuing invitations  Reviewing  Feedback from author to reviewer  Teammate review

19. 10/15/2015Gehringer, Rubric-based peer assessment of team projects, efg@ncsu.edu Importing teams

20. 10/15/2015Gehringer, Rubric-based peer assessment of team projects, efg@ncsu.edu Here’s the file that’ll be imported

21. 10/15/2015Gehringer, Rubric-based peer assessment of team projects, efg@ncsu.edu Here’s how we import it

22. 10/15/2015Gehringer, Rubric-based peer assessment of team projects, efg@ncsu.edu Now there are five teams …

23. 10/15/2015Gehringer, Rubric-based peer assessment of team projects, efg@ncsu.edu “Virtual” demo  Team formation By instructor, from the user interface By instructor, by importing a file By students, by issuing invitations  Reviewing  Feedback from author to reviewer  Teammate review

24. 10/15/2015Gehringer, Rubric-based peer assessment of team projects, efg@ncsu.edu Students can create teams too …

25. 10/15/2015Gehringer, Rubric-based peer assessment of team projects, efg@ncsu.edu Click on this assignment, Team test 2

26. 10/15/2015Gehringer, Rubric-based peer assessment of team projects, efg@ncsu.edu Now, click on “Your team”

27. 10/15/2015Gehringer, Rubric-based peer assessment of team projects, efg@ncsu.edu This user needs to set up a team

28. 10/15/2015Gehringer, Rubric-based peer assessment of team projects, efg@ncsu.edu Let’s create a team

29. 10/15/2015Gehringer, Rubric-based peer assessment of team projects, efg@ncsu.edu Invite another member to join …

30. 10/15/2015Gehringer, Rubric-based peer assessment of team projects, efg@ncsu.edu Invitation has been sent

31. 10/15/2015Gehringer, Rubric-based peer assessment of team projects, efg@ncsu.edu Now the invited user logs in

32. 10/15/2015Gehringer, Rubric-based peer assessment of team projects, efg@ncsu.edu … and goes to Team test 2

33. 10/15/2015Gehringer, Rubric-based peer assessment of team projects, efg@ncsu.edu Then accepts the invitation

34. 10/15/2015Gehringer, Rubric-based peer assessment of team projects, efg@ncsu.edu … and is placed on the team

35. 10/15/2015Gehringer, Rubric-based peer assessment of team projects, efg@ncsu.edu Advantages of various approaches  Instructor-assigned teams using class rank using demographics  Student-selected teams  Forthcoming: Schedule-based team formation

36. 10/15/2015Gehringer, Rubric-based peer assessment of team projects, efg@ncsu.edu “Virtual” demo  Team formation By instructor, from the user interface By instructor, by importing a file By students, by issuing invitations  Reviewing  Feedback from author to reviewer  Teammate review

37. 10/15/2015Gehringer, Rubric-based peer assessment of team projects, efg@ncsu.edu Let’s review a wiki page

38. 10/15/2015Gehringer, Rubric-based peer assessment of team projects, efg@ncsu.edu Click on the assignment …

39. 10/15/2015Gehringer, Rubric-based peer assessment of team projects, efg@ncsu.edu … then on “Others’ work”

40. 10/15/2015Gehringer, Rubric-based peer assessment of team projects, efg@ncsu.edu Choose a review

41. 10/15/2015Gehringer, Rubric-based peer assessment of team projects, efg@ncsu.edu Read the submitted wiki page

42. 10/15/2015Gehringer, Rubric-based peer assessment of team projects, efg@ncsu.edu Then fill out the review template

43. 10/15/2015Gehringer, Rubric-based peer assessment of team projects, efg@ncsu.edu Save and view the review

44. 10/15/2015Gehringer, Rubric-based peer assessment of team projects, efg@ncsu.edu10/29/08 Gehringer: Expertiza: A Peer-Review System for Collaborative Learning, efg@ncsu.edu After the Initial Review  Resubmission phase. 2–7 days to revise work in response to reviewer comments.  Final review. 3–7 days to make final comments and assign scores.  Metareview phase. Students review each other’s reviews.

45. 10/15/2015Gehringer, Rubric-based peer assessment of team projects, efg@ncsu.edu “Virtual” demo  Team formation By instructor, from the user interface By instructor, by importing a file By students, by issuing invitations  Reviewing  Feedback from author to reviewer  Teammate review

46. 10/15/2015Gehringer, Rubric-based peer assessment of team projects, efg@ncsu.edu Now the reviewee logs in …

47. 10/15/2015Gehringer, Rubric-based peer assessment of team projects, efg@ncsu.edu … and views the scores

48. 10/15/2015Gehringer, Rubric-based peer assessment of team projects, efg@ncsu.edu Click on “View” to expand review

49. 10/15/2015Gehringer, Rubric-based peer assessment of team projects, efg@ncsu.edu Notice the link down at the bottom

50. 10/15/2015Gehringer, Rubric-based peer assessment of team projects, efg@ncsu.edu Use it to give feedback to the reviewer

51. 10/15/2015Gehringer, Rubric-based peer assessment of team projects, efg@ncsu.edu Then the reviewer can view the feedback

52. 10/15/2015Gehringer, Rubric-based peer assessment of team projects, efg@ncsu.edu “Virtual” demo  Team formation By instructor, from the user interface By instructor, by importing a file By students, by issuing invitations  Reviewing  Feedback from author to reviewer  Teammate review

53. 10/15/2015Gehringer, Rubric-based peer assessment of team projects, efg@ncsu.edu Review of teammates

54. 10/15/2015Gehringer, Rubric-based peer assessment of team projects, efg@ncsu.edu Answer these questions …

55. 10/15/2015Gehringer, Rubric-based peer assessment of team projects, efg@ncsu.edu Summary of feedback in Expertiza  Reviews  Feedback to author  Teammate reviews  Metareviews (x reviews y’s review of z)  Surveys  All but surveys can influence the final grade

56. 10/15/2015Gehringer, Rubric-based peer assessment of team projects, efg@ncsu.edu Instructor report

57. 10/15/2015Gehringer, Rubric-based peer assessment of team projects, efg@ncsu.edu New Features Coming to Expertiza  Game mechanics … Achievements system Reputation system Leaderboards Levels  Message boards Students can ask for help; other students can answer  Microtasks/micropayments

58. 10/15/2015Gehringer, Rubric-based peer assessment of team projects, efg@ncsu.edu Forthcoming: Achievement System  Like in computer games.  Categories in which achievement is tracked Quality of submitted work Quality of reviews Contribution to team Helpfulness to other students  Prompt reviews/answers are more helpful, so reward for these …

59. 10/15/2015Gehringer, Rubric-based peer assessment of team projects, efg@ncsu.edu Forthcoming: Message Boards  Student a posts a question  Student b responds  Student a rates student b’s response  Others can rate response too. TAs can also respond  Reliability …

60. 10/15/2015Gehringer, Rubric-based peer assessment of team projects, efg@ncsu.edu Forthcoming: Microtasks/micropayments  Desired enhancements posted Along with amount of “payment”  Students can voluntarily perform the task.  Students can also post tasks.  How students should “pay” for these tasks

61. 10/15/2015Gehringer, Rubric-based peer assessment of team projects, efg@ncsu.edu For more info …  http://research.csc.ncsu.edu/efg/expertiza  Ed Gehringer, efg@ncsu.eduefg@ncsu.edu  Now, for the hands-on demo …

62. 10/15/2015Gehringer, Rubric-based peer assessment of team projects, efg@ncsu.edu Edit a Wiki Page 1. Go to [redacted] 2. You will be redirected to this page: 3. Log in with “unctlt#” and password “unctlt”. 4. Return to UNC_TLT, click on your user # and start editing. 5. Save your page.

63. 10/15/2015Gehringer, Rubric-based peer assessment of team projects, efg@ncsu.edu Review a Wiki Page 1. After you save the page you were editing, log in at http://expertiza.ncsu.edu with the same username and password. http://expertiza.ncsu.edu 2. Click on “Others’ work”, then on “Begin” for one of the reviews. 3. Assign scores and give text comments.