1. Peer Reviewer Training(f2f) Quality Matters : Inter-Institutional Quality Assurance in Online Learning Updated January 2007 Proposed for use by all QM trainers.

2. After this Peer Reviewer Training… You should be able to: Describe the critical materials, processes, and administrative elements of the Quality Matters online course quality assurance program. Apply the Peer Course Review rubric to review online courses. Write useful recommendations for course improvement. Explain the QM scoring system. Describe the Peer Course Review process and your role in it.

3. QM Materials Overview Today’s Agenda Binder Overview QM Website QM Contact Information QM Resources

4. Introductions Pair up Share name, institution, job, and best distance learning practice. Introduce your partner to the rest of us. Briefly (in one sentence) describe your partner’s best practice.

5. About Quality Matters

6. Peer Course Review Feedback Course Course Meets Quality Expectations Course Revision Instructional Designers Institutions Faculty Course Developers National Standards & Research Literature Rubric Faculty Reviewers Training Quality Matters: Peer Course Review Process

7. For Our Purposes, Quality Is… More than average; more than “good enough” An attempt to capture what’s expected in an effective online course at about an 85% level Based on research and widely accepted standards 85 %

8. What this process is NOT Not about an individual instructor (it’s about the course) Not about faculty evaluation (it’s about course quality) Not about “win/lose” or “pass/fail” (it’s about continuous improvement in a supportive environment)

9. QM Collegial Review vs. Faculty Evaluation A QM Review is Ongoing Focus: course design Outcome: course improvement Non-threatening Team approach that includes faculty Full disclosure to faculty A Faculty Evaluation is Single point in time Focus: delivery Outcome: decision on performance for promotion/tenure Win/lose situation Confidential/secretive

10. Design vs. Delivery The faculty member is integral to both design and delivery. Course Design … is the forethought and planning that a faculty member puts into the course. Course Delivery … is the actual teaching of the course, the implementation of the design. QM is about DESIGN - not delivery or faculty performance

11. Distinguish Between Design vs. Delivery Example: Discussion Board Design: A discussion board has been planned into the course; students have been told how they should participate and how they can expect the faculty to participate. Delivery: How often the faculty member actually participates in the discussion; what the faculty member actually says to students.

12. The Peer Review Team 3 faculty peer reviewers: –must be experienced online instructors –must attend QM training –one MUST be external to the course’s originating institution –there must be a subject matter expert (SME) on the team. NOTE: The SME could also be the external reviewer. AND Faculty course developer: –access to rubric prior to review –involved in pre-review discussions –consulted during review

13. Your Point of View as a QM Peer Course Reviewer Take the students’ point of view Advocate for the student If you can’t find evidence that the standard is met, don’t assume it is or isn’t there….. ask the faculty member.

14. Factors Affecting Course Quality Course design QM REVIEWS THIS Course delivery (i.e. teaching, faculty performance) Course content Course management system Institutional infrastructure Faculty training and readiness Student engagement and readiness

15. Underlying Principles of QM The QM toolset and process are: –based in national standards of best practice, the research literature and instructional design principles –designed to promote student learning –integral to a continuous quality improvement process –part of a faculty-driven, peer review process Course does not have to be “perfect” but better than just “good enough.” (Standards met at about 85% level or better.)

16. Underlying Principles of QM Process designed to ensure all reviewed courses will eventually meet expectations Collegial review process, not an evaluation process Review team must include an external peer reviewer Course faculty or instructor considered part of the review team

17. What’s In It For Institutions … External validation Strengthen accreditation package Raise QA as a priority activity Access to a sustainable, replicable, scalable QA process Inform online course training & practices Provide professional development activities

18. QM Process Provides Institutional toolset and process to meet quality expectations: –Online course design –Student learning –Improved instruction –Assessment and feedback loops –Professional development

19. What’s In It For Faculty … Improve online courses External quality assurance Expand professional community Review other courses and gain new ideas for own course Participation useful for professional development plan and portfolio Receive $150 for each completed peer course review

20. QM in Transition 2003 – August 2006 –QM project funded by FIPSE grant money –materials and some services freely available August 2006 and beyond –QM project funded by MarylandOnline –Some limited materials will be freely available –Other materials available to individuals and institutions at nominal fees –Institutional membership affords full access to materials and services

21. The Instructor Worksheet Worksheet

22. Instructor Worksheet Important part of review Includes info such as: –Institutionally mandated objectives, materials, practices, policies –Materials outside course pages –Types of interaction used & instructor’s statement on the appropriateness of interaction in the course –Additional items that may require review

23. Instructor Worksheet Read it first Refer to it during the review Use in team discussions

24. The Rubric

25. Eight standards: –Course Overview and Introduction –Learning Objectives –Assessment and Measurement –Resources and Materials –Learner Interaction –Course Technology –Learner Support –Accessibility Key components must align.

26. What is Alignment? Critical course elements work together to ensure that students achieve the desired learning outcomes.

27. Course Learning Objectives (II) Resources, Materials (IV) & Technology (VI) Assessment and Measurement (III) Learner Interactions & Activities (V) Key Sections that Must Align

28. Key Standards that Must Align Objectives –Standard II.1: Measurable outcomes –Standard II.2: The module/unit learning objectives describe outcomes that are measurable and consistent with the course- level objectives. Assessment and Measurement –Standard III.1: Measures objectives; consistent with learning activities Learner Interactions and Activities –Standard V.1: Help students achieve the objectives Course Materials –Standard IV.1: Deep and comprehensive enough for students to achieve the objective Course Technology –Standard VI.1: Tools and media support the objectives

29. Rubric Scoring StandardsPointsRelative Value 143Essential 122Very Important 141Important TOTALS 4080 Team of three (3) reviewers One score per standard based on team majority Assigned point value; not sliding scale

30. Assigned Point Values If the standard is met … If the standard is not met … Essential Standards 3 points0 points Very Important Standards 2 points0 points Important Standards 1 point0 points Points are NOT assigned on a sliding scale:

31. Awarding Points Points are awarded for each standard based on 1.the team majority, and 2.the pre-assigned weighting of each standard If 2-3 Reviewers believe that a standard is: met, then the full pre-assigned points are awarded not met, then zero points are awarded

32. How to decide…. For EACH standard: 1.Read the specific review standard and the annotation; review the examples, if needed. 2.Look for evidence that the standard is met in this course. 3.Ask yourself: Does this course meet the standard at an 85% or better level? 4.Decide Yes or No and enter your answer in the web-based rubric form. 5.Include comments/suggestions as documentation.

33. To Meet Expectations… A course must achieve: “Yes” on all 14 of the 3-point “essential” standards. A minimum of 68 out of 80 points 68/80 = 85%

34. Online & Hybrid Courses Rubric designed for application to fully online and hybrid/blended courses Same set of standards apply to both How we achieve the standards may differ For hybrids, focus on pedagogical integration of online and F2F components Refer to Course Format Definitions document

35. Recommendations

36. Writing a Useful Recommendation Constructive Try to offer solutions, not just identify problems. Specific Include a specific example of what is being recommended. Measurable How will the ID/instructor know when the recommendation has been implemented? Sensitive Avoid negative language. Keep recommendations and comments on a positive note. Balanced Point out strengths as well as weaknesses.

37. When MUST you write a recommendation? You MUST write a recommendation if you decide that the course does not meet the specific standard.

38. Improve these Recommendations Assignment instructions weren’t clear. –I wasn’t sure if assignment #1 required a written paper. You didn't tell the students how to find the additional resources. –I had trouble locating the resources; you may want to put a link on the home page for easy access. The text on the page was too hard to read. –The green text on a blue background was difficult to read. Suggest black on white.

39. Improve these Recommendations The learning objectives aren’t measurable. –Rephrase the learning objectives to include an active verb (explain, distinguish, compare, etc.) The assessments are weak. –The assessments are unrelated to the learning objectives. Review your learning objectives and make sure that they are reflected in your exams.

40. Hands-OnPractice

41. Hands-On Practice Work in Pairs Review the Anthropology Course –Go to http://pgcconline.blackboard.comhttp://pgcconline.blackboard.com –Username is fipse –Password is fipse Focus on 14 essential (3 point) standards Write 1 recommendation per standard

42. Steps 1.Get your training pair assignment 2.Find your computer in lab (2 per pair) 3.One person logs into the course 4.Other person logs into QM rubric 5.Read Instructor Worksheet for course Then… follow facilitator’s instructions for first standard

43. General Standard I: Course Overview and Introduction 1.1: Navigational instructions make the organization of the course easy to understand. 1.2:A statement introduces the student to the course and to the structure of the student learning, and, in the case of a hybrid course, clarifies the relationship between the face-to-face and online components.

44. General Standard II: Learning Objectives (Competencies) II.I: The course learning objectives describe outcomes that are measurable. II.2:The module/unit learning objectives describe outcomes that are measurable and consistent with the course-level objectives.

45. General Standard III: Assessment and Measurement III.1:The types of assessments selected measure the stated learning objectives and are consistent with course activities and resources. III.2:The course grading policy is stated clearly. III.3:Specific and descriptive criteria are provided for the evaluation of students’ work and participation.

46. General Standard IV: Resources and Materials IV.I:The instructional materials support the stated learning objectives. IV.2:The instructional materials have sufficient breadth, depth, and currency for the student to learn the subject.

47. General Standard V: Learner Interaction V.1:The learning activities promote the achievement of stated learning objectives. V.2:Learning activities foster instructor- student, content-student, and if appropriate to this course, student-student interaction. V.3:Clear standards are set for instructor response and availability (turn-around time for email, grade posting, etc.)

48. General Standard VI: Course Technology VI.I:The tools and media support the learning objectives and are appropriately chosen to deliver the content of the course.

49. General Standard VII: Learner Support No 3-point elements This standard has no “essential” 3-point elements because it’s primarily concerned with academic support services, student support services and technical support services….usually thought to be the primary responsibility of the institution and not the individual instructor.

50. General Standard VIII: Accessibility VIII.1:The course acknowledges the importance of ADA requirements. To meet this standard, the course must include BOTH of these elements: The course must be offered using software that is accepted as “ADA compliant.”. AND The course should include a brief statement that clearly tells students how to access ADA services at the institution.

51. Showcase Your Courses

52. Course Showcase Think about your own course: –What do you do that meets QM expectations? –What improvements might you make? Share your thoughts with your partner Summarize for the group

53. Scenarios

54. Evaluating Scenarios Divide into groups. Discuss the scenarios that focus on the 14 essential standards. Take the “quiz” with your group. Use the material in your books and each other as references. Jot down your reasoning and be prepared to discuss your decision.

55. Peer Course Review Process

56. About the Course QM is designed to review “mature” courses (taught at least two semesters) QM logo indicates year course met expectations Triggers for subsequent reviews: Faculty request More than 3 years since original review New textbook or instructor Professional or accreditation review pending

57. About the Review On average, a course review takes 7-10 hours Factors affecting review time include Reviewer familiarity with the discipline Reviewer familiarity with the CMS Reviewer familiarity with the QM review process Organization of the course Suggested review methodology: Read Instructor worksheet Familiarize yourself with CMS and course Pre-review team discussion Proceed through standards, but save Standard III (Assessment and Measurement) for last Post-review team discussion Submit your review

58. Your Point of View as a QM Peer Course Reviewer Take the students’ point of view Advocate for the student If you can’t find evidence that the standard is met, don’t assume it is or isn’t there….. ask the faculty member.

59. Timeline Active review period approximately 3 weeks Compiled reports due in approximately 6 weeks Teams: –Have pre-review discussion. –Set Team calendar. –Commit to 2-3 week review period. –Have post-review discussion.

60. Post-Review Reviewers –complete exit interview –receive stipends Faculty –receives Final Review Report –completes Faculty Response Form

61. Review Outcome If meets expectations: –Recognized by Quality Matters –Notifications distributed –ID support provided if requested If does not yet meet expectations: –ID support provided if requested –Team Chair and ID approve revisions –Course meets expectations

62. Roles and Responsibilities

63. Faculty Developer 1.Part of the review team. 2.Provides access to the course. 3.Completes Instructor Worksheet 4.Part of the initial team discussion 5.Receives compiled report 6.Returns Faculty Response Form

64. Peer Reviewers 1.Establish Team Calendar 2.Review the course individually 3.Complete the online web review form 4.Discuss review with Team as needed 5.Complete an Exit Interview Receive…  Recognition as Certified Peer Reviewer.  Compensation

65. Team Chair Reviewer Roles & Responsibilities plus…. 1.Organizes Team calendar 2.Confirm Instructor Worksheet is used 3.Creates draft report from compiled reviews 4.Convenes Team discussions 5.Reviews, edits and submits Team Report Receives…  Recognition as Certified Peer Reviewer and Chair.  Compensation

66. QM To Date

67. QM to Date Overall Participation: –Individuals & programs from 130 institutions across 28 states Course Reviews: –103+ courses reviewed –18 MD schools; 10 non-MD schools Peer Reviewer Rubric Training: –600+ trained

68. Multiple Uses of QM Reported Uses of QM System: Guidelines for initial online course development Quality assurance of existing courses Ongoing faculty professional development Institutional reaccredidation packages Formation of distance learning policies & steering committees

69. Awards - 2005 WCET Outstanding Work (WOW) Award, November 2005. USDLA 21 st Century Best Practice Award, October 2005. Maryland Distance Learning Association (MDLA) Best Program Award, March 2005.

70. QM in Transition 2003 – August 2006 –QM project funded by FIPSE grant money –materials and some services freely available August 2006 and beyond –QM project funded by MarylandOnline –Some limited materials will be freely available –Other materials available to individuals and institutions at nominal fees –Institutional membership affords full access to materials and services

71. Research Findings

72. Overall Course Review Results Upon initial review: 53% meet expectations 22% do not meet expectations - missing at least one essential 3-point element(s) 25% do not meet expectations - missing at least one essential 3 point element(s) and a minimum of 68 points

73. Summary of Course Reviews variety of course management systems –50% Bb, 35% WebCT, 16% WebTycho, 2% other –65% of respondent Reviewers felt unfamiliarity with CMS did not impede review; 34% somewhat variety of disciplines –general studies, information technology, teacher education, engineering, allied health –75% of respondent Reviewers felt unfamiliarity with discipline did not impede review; 25% somewhat

74. Analysis of Infrastructure Whether a course meets/does not meet QM expectations is NOT correlated with: Course’s institution of origin Course management system –53% of Bb courses –54% of WebCT courses –50% of WebTycho courses

75. Analysis of Academic Area Met ExpectationsAcademic Area# Courses 78%information technology9 60%math & statistics10 55%allied health, health, nutrition, nursing11 50%natural sciences11 50%education10 50%history, government, politics6 50%behavioral & social sciences12 50%english, writing, communications10 33%arts & humanities9

76. Post-Course Review Met expectations –Most faculty made suggested improvements even though their course met expectations! Did not meet expectations –Improvements made or in progress –Most improvements made by faculty –Some ask for ID support

77. Common Themes Course reviews revealed 11 common areas for course improvement Elements that are missing in 20% or more of the courses reviewed These are potential targets for –faculty training –special attention in the initial course development phase:

78. Common Areas for Improvement Instructor self-introduction (I.4) 22% Activities that foster interaction(V.2) 22% Technology/skills/pre-req knowledge stated (I.6) 24% Links to academic support, student services, tutorials/resources (VII.2-VII.4) 24-27% Learning objectives at module/unit level (II.5) 27% Netiquette expectations(I.3) 32% Self-check/practice with quick feedback (III.5) 38% B/W alternatives to color content (VIII.4) 54% Alternatives to auditory/visual content(VIII.2) 59%

79. Serving as a QM Peer Reviewer

80. Quality Matters Peer Course Reviewer Certification Process Training Course Review Experience Quality Matters Certification Attend QM Training Trained Assigned to Peer Review Team Submit Report Certified + = Name on QM website Use of QM Logo Eligible for Peer Reviewer Pool Demonstrate Competencies Kane 1/15/05

81. Quality Matters Peer Course Review Process 1. Course Selected 2. Trained Review Team Assigned 3. MOUs and Instructor Worksheet 4. Pre-Review Discussion 5. Individuals complete reviews 7. Final Review Report 6. Team Discussion(s)

82. Next Steps Next Steps

83. Next Steps Within the next week –Return to “Foundations” Blackboard site –Complete Scenarios Quiz –Grade of 85% or greater required to pass training and be a Peer Reviewer

84. Thanks to YOU… Thanks to YOU… Quality Matters!