1. MERLOT: The Peer Review of Digital Scholarship Professor Cathy Owens Swift Georgia Southern University Professor Susan M. Moncada Indiana State University Professor Theresa B. Flaherty James Madison University AACSB (April 26, 2003)

2. Presentation What is MERLOT? – Swift The Peer Review Process – Moncada Benefits of Submitting Online Learning Materials to MERLOT & Peer Reviewing – Flaherty Benefits to You (Dean) - Swift Questions or Comments

3. What is MERLOT? Cathy Owens Swift Georgia Southern University

4. Common Issues - Quality Control - Dissemination - Sustainability Institutions invest in developing instructional software without provisions for: Lack of quality, interactive, web-based learning materials Faculty often work in isolation while developing online courses

5. Evaluation of Digital Scholarship Chronicle of Higher Education (6/2/2000) “Counting Digital Scholarship & Teaching in Faculty Evaluations.” –Need peer review of teaching innovations Syllabus, (1/2001) “Faculty Rewards in Digital Instructional Environments” Carnegie Teaching Academy: –Scholarship of teaching definition includes “peer review” Physics Model of Scholarship of Teaching: –Discussions on “review and evaluation by acknowledged national experts”

6. Faculty should remain in control of the teaching/learning process. Peer-reviews will contribute to expanded use and effectiveness of digital learning materials. Assumptions Sharing of materials will maximize everyone’s investments. Faculty need mechanisms to document teaching and learning contributions.

7. Vision & Mission MERLOT’S VISION is to be the place where faculty from around the world will share teaching- learning materials and pedagogy. MERLOT’S MISSION is to improve the effectiveness of teaching & learning by expanding the quantity and quality of peer-reviewed online learning materials that can be easily incorporated into faculty designed courses.

8. Aspects of MERLOT MERLOT is a COOPERATIVE Institutional Partners Organizational Affiliates Individual Members MERLOT is a SET OF PROCESSES Building, organizing, reviewing, and developing applications of online teaching-learning materials Building and sustaining online academic communities. MERLOT is SOFTWARE A user-centered, searchable database of online learning materials, pedagogical support, and people.

9. MERLOT follows the model of peer review of scholarship Individual Review #1 Individual Review #2 Composite Review

10. MERLOT Alliance Partners National Science Foundation (NSF) Grant $400,000 Online peer review training module NLII of Educause AAHE – American Association of Higher Education Education.au Limited IMS Global Learning Consortium, Inc Health Education Assets Library NEEDS – Digital Library for Engineering SMETE.org

11. Discipline Communities Business Biology Chemistry Engineering Health Sciences History Info Tech Music Math Physics Psych Teacher Ed Teaching w/Tech World Languages

12. Parties Involved in MERLOT Administration Team (12) System Partners (22) Campus Partners (2) Discipline Editorial Boards (14) –Editors/Co-Editors (25) –Editorial Review Board Members (150) Peer Reviewers (100+) Others – Authors, Submitters, Users

13. Institutional Partner Commitments Faculty Reviewers – 6-8 faculty to serve as experts Faculty Development & Academic Technology Personnel Travel –Training, presentations, meetings Project Director (10+%) – Planning, coordinating, budgeting, etc. Participation Fee: $25K

14. MERLOT: The Peer Review Process Susan M. Moncada Indiana State University

15. Editorial Boards Support development of discipline communities (14) Editorial Review Board Members (150) –Editors/Co-Editors (25) –Associate and Assistant Editors –Peer Reviewers (100+)

16. Board Qualifications Expertise in scholarship of their field Excellence in teaching Experience in using technology in teaching and learning Connections to professional organizations Experience in conducting peer reviews of online learning resources

17. Board Responsibilities Expand and manage the collection Implement the peer review process Post peer reviews Recruit and train peer reviewers Education and outreach to the community of educators

18. MERLOT’s Business Discipline Accounting Business Law Economics E-commerce Finance General Information Systems International Business Management Marketing

19. TYPES OF MODULES Simulations Tutorials Animations Drills & Practice Quiz/Tests Lecture/Presentations Collections Reference materials

20. Profile of the MERLOT Learning Module Title, author and affiliation Peer Review Link User Comments Link Type of learning material Location (URL for the module) Subject classification Description Submitter Audience

21. Editor’s Evaluation Process Stage 1: –Cursory review to identify worthy modules –Post triage comments and triage value online Stage 2: –Editor assigns “worthy” materials to reviewers –Reviewers apply MERLOT standards to write reviews.

22. MERLOT follows the model of peer review of scholarship Individual Review #1 Individual Review #2 Composite Review

23. Editor’s Evaluation Process Stage 3: Editor sends review to author for feedback and permission to post –Authors can elect to modify materials and request review be modified –Authors can request module be pulled from the repository –Authors can request 2 letters from MERLOT summarizing peer review process and report to 2 people of their choice. Stage 4: Peer review is posted

24. Standard Evaluation Criteria (Strengths & Concerns) 1. Quality of Content 2. Potential Effectiveness 3. Ease of Use

25. 1. Quality of Content Current and relevant Accurate information Clear and concise Informed by scholarship Completely demonstrates concepts Flexibility Integrates/summarizes concept well

26. 2. Potential Effectiveness Specifies learning objectives Identifies prerequisite knowledge Is very efficient Reinforces concepts progressively Builds on prior concepts Demonstrates relationships between concepts

27. 3. Ease of Use Is easy to use Has clear instructions Is engaging Has visual appeal Is Interactive Uses effective navigation techniques All elements work as intended

28. Star Rating System Excellent all around Very good w/few minor concerns (4.0-4.9) Meets/exceeds standards with some significant concerns (3.0-3.9) Standards not met, some limited value (2.0-2.9) Not worth using at all (1.0-1.9)

29. Standards for Scholarly Work* Endeavors require high level of discipline expertise Breaks new ground and is innovative Is of significance Can be replicated or elaborated upon Can be documented Has the potential to be peer reviewed * (Merton, 1973)

30. Benefits of Submitting Online Learning Materials to MERLOT & Peer Reviewing Theresa B. Flaherty James Madison University

31. Who contributes materials to MERLOT? MERLOT materials are added by people who have joined MERLOT (MERLOT members). –Anyone may join MERLOT, and there is no cost or other obligation. Materials may be added by: –the people who created them (author), or –any member who finds a great resource to share with others (submitter).

32. How to Contribute Modules to MERLOT as an Author Step 1: Develop your Module

33. Step 2: Fill out a short form to contribute your material. Can send a note to Editor to request peer review.

35. The Peer Review of this Module

36. Avoids “reinventing the wheel”

37. Documents Teaching Efforts

38. Provides Networking Opportunities

39. Search for members by discipline

40. Peer Reviews are developed through online workspace

41. The peer review can be completed in different segments

42. After completion of the review, the Peer Reviewer is recognized for his/her contribution by a letter from the Editor. This letter can be sent to two other people as well.

43. To become a Peer Reviewer, an individual must: be an instructor at an institution of higher learning demonstrate expertise in the discipline be recognized for excellence in teaching have experience using technology in teaching, and have participated in the activities of the discipline.

44. Benefits of MERLOT (Dean’s Perspective) Qualitative & Quantitative Dimensions to Teaching Efforts –Peer Review is Available for General Public Assist in Evaluation of Faculty Efforts (P&T) Encourage Faculty to Develop Digital Materials that benefit multiple disciplines Encourage Faculty to Provide Service to their Disciplines as Peer Reviewer –Training Provided by MERLOT Encourage Faculty to Use Digital Materials

45. Your Participation in MERLOT System Partner Campus Partner Inform Faculty of Opportunities –Author –Submitter –User –Peer Reviewer Conduct Faculty Training Recognize MERLOT Contributions of Faculty Join MERLOT

46. Questions or Comments?