1. Applying the QM Rubric August 2008
Inter-Institutional Quality Assurance in Online Learning
Applying the QM Rubric
Marker: Display this on the screen as participants enter the meeting room. Add the institution's name, your name and title if you wish.
August 2008

2. After this training you will be able to…
Describe underlying principles of Quality Matters (QM).
Use QM rubric & explain scoring system.
Apply the QM rubric to online course and scenarios.
There are three training session objectives to review with the participants. Be sure to point out that the purpose of this training is to prepare the participants to apply the QM rubric to their own or another’s online course (NOT to teach them how to teach online or to review their own online courses). NOTE: The day’s evaluation is based on how well the participants believe that these objectives were met.
This training is specifically designed for two audiences:
1.  Participants who want to learn about and apply the QM rubric and process
WHO:  Faculty members, instructional designers, administrators who want to know more about QM and apply the process but who are not eligible to become peer reviewers OR who do not plan to serve on a official QM peer review team.
What recognition will you receive?:  You will receive a certificate of completion documenting that you have successfully completing the Apply the QM Rubric workshop.
2.  Participants who want to serve as certified QM Peer Reviewers 
WHO:  Online instructors who meet the following criteria will be eligible to serve as QM Peer Reviewers upon successful completion of this workshop:
Online teaching experience.  You should have taught online for more than two semesters.
Current online teaching experience.  You should have taught within the last 18 months.
Commitment.  You are encouraged to volunteer; this is a self-nomination process.
What should you do?:  As directed by the facilitator, proceed to the QM web site and follow links to Peer Reviewer Workshop.
What recognition will you receive?:  You will receive a certificate documenting at the completion of the online training that you have successfully completed the Peer Reviewer Workshop and are now eligible to serve as a QM peer reviewer.

3. QM Materials Overview Today’s Agenda Folder Overview
QM Website
QM Contact Information
Briefly highlight the following for the participants:
Today’s Agenda: Participants do not usually have a copy of the agenda; this allows the QM trainer to modify the agenda if time problems occur. Point out when lunch and breaks occur; tell participants how to find restrooms and drinking fountain.
Folder Overview: Briefly go over the white sheets on the right side of the folder.
QM Website: If you have an Internet connection, locate the web site at and point out the highlights. NOTE: The QM rubric and other QM documents have been removed from the web site because of copyright and subscriber considerations. Many other documents referred to in the training can be found by clicking the link called Peer Reviewer Documents (located at the bottom of the center column, Training and Events ).
QM Contact Information: Best guarantee of an answer about QM is through Deb Adair, QM Director of Administration. Deb’s address is Be sure to include “QM” in the subject line.
QM Resources: Mention QM workbook (handout). Suggest they save this workbook to make notes about their own courses at a later time (rather than write up the demo course notes in it).

4. Introductions
Share name, institution, job, and best distance learning practice.
Briefly (in one sentence) describe your best distance learning practice.
Alternate #1 - Introductions Exercise:
This exercise is particularly useful if your audience is composed of participants who don’t know each other.
Ask participants to pair up for introductions. Suggest that they interview someone they don’t know (this may require some moving). Give them a few minutes to talk. Go around the room and ask them to introduce either themselves or their partners.
Your challenge here is to convince them to be BRIEF. Encourage a one-sentence response to the best practice.

5. Trying it out with Standard 1.2
A statement introduces the student to
the purpose of the course and to its
components; in the case of a hybrid
course, the statement clarifies the
relationship between the face-to-face
and online components.
Alternate #2 – Apply Standard 1.2
Although the participants haven’t yet been formally introduced to the QM rubric, this exercise provides a hands-on introduction to the day’s activities.
Review Standard 1.2 with the participants, directing them to also look at the point value and annotation in the QM Workbook. Then move to the next slide.

6. Review Scenario
This course reflects a shift in the importance that the world outside the schools increasingly places on thinking and problem solving. Procedural skills alone do not prepare students for that world. Therefore, students deserve a curriculum that develops their mathematical power and an assessment system that enables them to show it. Assessments that match the current vision of school mathematics involve activities that are based on significant and correct mathematics. These activities provide all students with opportunities to formulate problems, reason mathematically, and make connections among mathematical ideas. Students engage in solving realistic problems using information and the technological tools available in real life. Moreover, skills, procedural knowledge, and factual knowledge are assessed as part of the doing of real life mathematics.
1. 5-Tests (Must be taken on campus) 400
Quizzes
3. Project (Optional)
Total number of points possible (no project)
525 (if you do the project)
Grading Scale:
90% to 100% = A 80% to 89% = B 70% to 79% = C 60% to 69% = D 0% to 59% = F
A student who earns a final average of 70% or more on all assessments of objectives and intended learning outcomes for the College Algebra course has successfully fulfilled the general education and other essential core skill goals. Any extra credit problems are given to the entire class. There are no extra credit problems for any one individual during any part of the semester.
You are reviewing a College Algebra course and find the following statement:
Alternate #2 – Introductions Exercise:
Standard (I.2): A statement introduces the student to the course and to the structure of the student learning and, in the case of a hybrid course, clarifies the relationship between the face-to-face and online components.
Refer participants to the above scenario (there should be a copy of it in the folder or you may have to read parts if you have a large group in a big room). Give participants a few minutes to discuss the scenario either in small groups or as a whole and make a decision about whether or not the standard is met.
Participants typically note the following:
the tone of this “welcome” is not friendly, inviting (and may even be threatening considering it’s a Math course)
this presents the instructor’s philosophy but doesn’t clearly give much specific, practical information about the course
the grading policy presented in fuzzy, unclear, nonspecific – and probably raises more questions than it answers
Most participants decide that this scenario does not meet the standard: While the student is introduced to the instructor’s philosophy of the course and the grading scheme, there is no indication of how the learning is structured, how communication will occur and what the student must actually do to complete the course.

7. About Quality Matters
Marker: The following slides deal with the “key” QM concepts.

8. Course Meets Quality Expectations
Quality Matters:
Peer Course Review Process
Faculty Course Developers
Institutions
National Standards & Research Literature
Course
Rubric
Faculty Reviewers
Training
Peer Course Review
Course Meets Quality Expectations
Course Revision
QM Circle Process: This visual highlights the QM review process. Note that the circle elements transition in automatically.
Key points: QM is designed for continuous improvement; the goal is that ALL courses will eventually meet QM expectations.
Method:
Ask participants to look at the diagram and make inferences about it. Typically, they will respond with “it’s a circular process;” “it’s continuous:” etc).
Begin with the COURSE and then continue through PEER COURSE REVIEW, FEEDBACK, COURSE REVISION and Course Meets Quality Expectations. Speak briefly about each item and how it contributes to the process.
Benefits of using QM that you may wish to mention:
A reviewed course WILL meet expectations (although maybe not on the initial review).
The faculty member will receive substantial feedback support.
QM was designed to be a peer-review, collegial process AND
The accreditation team will find plenty of evidence that the institution is striving to achieve quality in its offerings and ensuring that its faculty are skilled and informed online professionals.
Reassure participants that they will learn more about each major item in detail throughout your presentation.
Feedback
Instructional Designers

9. QM = Rubric and Process Rubric Process OFFICIAL
Outcome: earn QM recognition
Tool to assess online courses during formal QM review
Outcome: improve courses,
meet institutional goals
Guide to develop new online courses AND review and update online courses
Must follow official QM guidelines and procedures
Outcome: improve courses, meet institutional goals, demonstrate commitment to quality
INFORMAL
Institutions determine use and procedures
Outcome: earn QM recognition
Tool to assess online courses during formal QM review
Outcome: improve courses, meet institutional goals
Guide to develop new online courses AND review and update online courses
Outcome: earn QM recognition
Must follow official QM guidelines and procedures
Outcome: improve courses, meet institutional goals, demonstrate commitment to quality

10. Factors Affecting Course Quality
Course design QM REVIEWS THIS
Course delivery (i.e. teaching, faculty performance)
Course content
Course management system
Institutional infrastructure
Faculty training and readiness
Student engagement and readiness
There are many factors that affect the quality of an online course. Among these factors are:
the course design (the forethought and planning that goes into an online course,
the course delivery (i.e. the way the course is taught, also known as faculty performance),
the course content,
the course management system and its functionality; technical support
the institutional infrastructure (help desk, online library access, online tutoring access, etc),
a faculty member’s training and readiness for online teaching,
and the students’ role with respect to engagement and readiness for an online course.
QM reviews just one aspect of online course quality – Course Design.

11. The faculty member is integral to both design and delivery.
Design vs. Delivery
The faculty member is integral to both design and delivery.
Course Design … is the forethought & planning that a faculty member puts into the course.
Course Delivery … is the actual teaching of the course, the implementation of the design.
This slide reinforces the concept that QM is about Design, not Delivery. QM recognizes that this is a fine line, but course design is the primary emphasis during a review.
Design is usually about what the faculty members plans and prepares for BEFORE the students arrive in the course; delivery is what happens after the students login.
QM is about DESIGN - not delivery or faculty performance

12. Distinguish between design & delivery…
Example: Discussion Board
Design:
Discussion board planned in course; students told how they should participate & how they can expect the faculty to participate.
Delivery:
How often the faculty member actually participates in the discussion; what the faculty member actually says to students.
Here’s a concrete example of how you might distinguish Course Design from Course Delivery.
Feel free to use an example from your own course if you have one.
Ask the participants if they can think of their own examples and share them with the group.

13. For Our Purposes, Quality Is…
More than average; more than “good enough”
Attempt to capture what’s expected in an effective online course at about an 85% level
Based on research & widely accepted standards
85 %
A QM KEY Concept: Emphasize that QM is not looking for “just good enough.” Looking for above average (approximately 85% or B+). Point out that although this is somewhat subjective, the basis for the decision is based in the research literature and widely accepted standards about effective distance learning.
Peer Reviewers are all experienced online faculty who have attended QM training and learned to apply the rubric. QM relies on the experience, expertise and common sense of its faculty reviewers to conduct reviews fairly and consistently and to judge whether the course meets expectations at the “85%” standard.
The emphasize on above average is also the reason that QM primarily reviews “mature” courses (taught at least two semesters). If the course has been taught over several semesters, the faculty developer has had time to “fine tune” the course and to make management and content improvements.
NOTE FOR FACILITATORS: Many Peer Reviewers find the concept of 85% confusing.  There are actually TWO 85 % thresholds: 1.  The first is that a course must earn at least 72 out of 85 points (or about 85%).  The points from all YES or NO decisions on specific standards are added and must be at the 85% or greater percentage - AND meet all 17 essential standards - to be QM recognized. 2.  The second is that when peer reviewers should use the 85% rule to determine whether or not a specific standard is met. The standard does not have to be 100% to be a "yes". The "85% rule" is a guide for reviewers to gauge whether they will decide "yes" or "no" for that particular standard. The assumption is that experienced online faculty members will have a "gut level"  recognition of what that means.

14. What QM is NOT About… Not about an individual instructor
(it’s about the course)
Not about faculty evaluation
(it’s about course quality)
Not about judgment (it’s about diagnosis and improvement)
Not about “win/lose” or “pass/fail”
(it’s about continuous improvement in a supportive environment)
The purpose of this slide is to focus on what QM is and is not:
about course design, not about the instructor
about course quality, not about faculty evaluation
score that comes out of a QM course review is Diagnostic rather than Judgmental.  It tells us how much revision a course may need.  
about continuous improvement, no about pass or fail
You can acknowledge that QM is walking a fine line because it is difficult to separate the course design and the instructor. However, the intention of a QM review is to focus on the course design.

15. QM Collegial Review vs. Faculty Evaluation
QM Review
Faculty Evaluation
Single point in time
Focus: delivery
Outcome: decision on performance for promotion/tenure
Characteristics
Win/lose situation
Confidential/secretive
Ongoing
Focus: course design
Outcome: course improvement
Characteristics
Non-threatening
Team approach that includes faculty
Full disclosure to faculty
This slide reinforces the distinction between a QM review and a faculty evaluation.

16. The Rubric
Marker: The following slides highlight the origins and organization of the QM rubric.

17. The Rubric Key components must align.
Eight General Standards:
Course Overview and Introduction
Learning Objectives (Competencies)
Assessment and Measurement
Resources and Materials
Learner Engagement
Course Technology
Learner Support
Accessibility
Key components must align.
There are 8 main sections of the rubric (these are the 8 General Review Standards and are noted in red letters in the QM Workbook.
The rubric consists of 40 Specific Review Standards which are distributed over these 8 general categories.
Take the time to point out the organization of the QM Workbook and how the rubric is organized:
Eight General Review Standards
Forty Specific Review Standards
In Table Format: 1st column is Specific Review Standards, 2nd column is point value, next columns are for your YES or NO decision, and the last column is the annotation.
Talk about QM’s intention to be “holistic” and that a QM review is intended to ensure that all parts of the course work together. You can use the metaphor of a cake recipe. Baking a cake is fairly simple if you follow the recipe and correctly measure and add the right ingredients (and the result is also fairly simple and straightforward: if you do it correctly, you end up with a cake). Reviewing an online course is much more complex: not only must you include all the “ingredients” but they must all work together to support the learning objectives. For example, you could have strong, measurable learning objectives but if they don’t align with the assessments, you still don’t have a quality online course.
NOTE: Your discussion of alignment should be brief and introductory; you will spend more time explaining “Alignment” in another section of the presentation.
Alignment: Critical course elements work together to ensure that students achieve the desired learning outcomes.

18. Rubric Scoring In an official QM Review: Standards Points
Relative Value 17 3
Essential 11 2
Very Important 12 1
Important
TOTALS 40 85
If you haven’t already done so, point out the breakdown of the 40 Specific Review Standards by points:
17 Standards are ESSENTIAL. Without all standards being met at the 85% level, this course cannot be a quality online course.
11 Standards are Very Important to an online course
12 Standards are Important.
This results in a total of 40 Specific Review Standards and a total of 85 points.
Emphasize again that each reviewer individually decides Yes or No on a particular standard and enters this decision on the web form. The web form then automatically totals the points (majority rules) and determines whether/not the course meets the standard.
Tell participants that they don’t have to reach consensus; since the instructor will see all decisions and all feedback, it will be very helpful to know that there was a “split decision” on a particular standard and to read the specific recommendations for how to improve.
In an official QM Review:
Team of 3 reviewers initially score individually
One score per standard based on team majority
Pre-assigned point value
Yes/No decision; All/None points
Consensus is NOT required

19. Points are NOT on assigned on a sliding scale…
Standards
If the standard
is met …
is not met …
Essential
3 points
0 points
Very Important
2 points
Important
1 point
This information is important because new Peer Reviewers frequently think they are assigning points (and sometimes think they can assign 2 points for a 3-point standard if the standard is “sort of” met). In practice they decide whether a Specific Review standard is met at the 85% standard, indicate YES/NO on the web-form, and then the web form actually assigns the points.

20. To Meet Expectations…Two Thresholds
A course must achieve:
“Yes” on all 17 of the 3-point “essential” standards.
A minimum of 72 out of 85 points
There are TWO thresholds for meeting expectations and a course must meet both thresholds to meet expectations. Mathematically, this works out to about an 85% standard.

21. The Instructor Worksheet
Marker: The following slides are designed to highlight the importance of the Instructor Worksheet (and the Instructor) to the process.

22. Instructor Worksheet Read it first Refer to it during the review
Use in team discussions
The basic point of this slide is to convince Peer Reviewers to print a copy of the Instructor Worksheet, keep it close by, and refer to it at all stages of the review process.

23. Instructor Worksheet = The Voice of the Instructor
Key piece in the review
Includes info about:
Institutionally mandated objectives, materials, practices, policies
Materials outside course pages
Types of interaction used & instructor’s statement on the appropriateness of interaction in the course
Additional items that may require review
****The Instructor Worksheet for the demo course is on colored paper in the Folder.****
Specifically refer to and discuss the following numbers from the Instructor Worksheet, since these directly impact the standard decisions:
Item 9: Directly related to Standard II.1 Learning Objectives. QM does not hold individual faculty members responsible for measurable learning objectives if the faculty member is REQUIRED to use those objectives. NOTE: The instructor must attach a list of course level objectives and a sample of the module level objectives.
Item 13: Directly related to Standard VI.Course Technology. If the course uses multi-media, the Instructor should provide information about how to access it.
Item 21: Directly related to Standard V.2 Learner Interaction. The instructor decides whether student-to-student interaction is important to this course.
Item 22: The instructor must provide (or link to) his/her institution’s accessibility policy.
Items 24 and 25: The Instructor has an opportunity to list areas of concern and seek responses from the review team.

24. Hands On Practice
Marker: The following slides move the participants through a review of the 17 essential standards.

25. Your Point of View as a QM Course Reviewer…
Take the students’ point of view
Advocate for the student
Can’t find evidence standard is met…don’t assume it is or isn’t there….ask the course developer (faculty member)
Emphasize that Peer Reviewers should assume the STUDENT’s point of view. This is an opportunity to see the course from the student’s point of view (not the faculty member’s). This point of view often reveals aspects of the course that could be improved to make navigation easier and improve the learning environment for the students...aspects that are not obvious to the faculty developer because he/she is just too close to the course.
This might be a good place to ask if any of today’s participants have been students in an online course. Ask them if being a student changed the way they think about online teaching.

26. How to decide…. For EACH standard
Read the specific review standard and the annotation; review the examples, if needed.
Look for evidence that the standard is met in this course.
Ask yourself: Does this course meet the standard at an 85% or better level?
Decide Yes or No and enter your answer in the web-based rubric form.
Include comments/suggestions as documentation.
This slide presents a strategy for reviewing the Specific Standards.
New QM reviewers are always uncertain about how to proceed (they want to do the right thing, but may be confused about the process at this point in the training). This strategy reinforces the key aspects of the QM review AND emphasizes that they are making a YES/NO decision.

27. Hands-On Practice Review the Anthropology Course
Go to
Username is fipse
Password is fipse
Focus on 17 essential (3 point) standards
Write 1 recommendation per standard
You may wish to point out that the Anthropology course dates from 2004 and that it was an actual, unremediated course. It’s not a perfect course…but is typical of many online courses that they may encounter at their own institutions or during a formal QM review.
Peer reviewers will likely find many things that need improvement, but there will also be opportunities to recognize good work. Peer reviewers will experience the “gray” areas that typically come with a “real” course; the challenge is to make decisions about whether or not the standards are met at about an 85% level.

28. Hands-On Practice Find a Partner Locate login handout in folder
Login to ANT 103
Read Instructor Worksheet for course
As the QM facilitator, you are now taking the participants on a forced march through the 17 essential standards. The participants are doing this exercise in small groups (2 or 3 to a group).
Method:
Introduce the General Standard and look at its essential elements. Make sure that participants are using the Instructor Worksheet and the QM Workbook.
Review the items that are significant for each of the elements within the Standard.
Direct the participants to look for these elements in the demo course and determine whether the demo course meets the standard (make a Yes/No decision). If the participants say No, they must be prepared to explain why.
Give them 5-10 minutes to explore the course.
For each essential element, ask the participants to raise their hands if their team thought the course met the standard? If the course did not meet the standard?
Discuss their reasoning; correct any misinformation or confusion.

29. General Standard 1: Course Overview and Introduction
Specific Review Standard
What does this mean?
1.1: Instructions make clear how to get started and where to find various course components.
1.1: Do the students know what to do first?
1.2: A statement introduces the student to the purpose of the course and to its components; in the case of a hybrid course, the statement clarifies the relationship between the face-to-face and online components.
1.2: Do the students know what the learning objectives of the course are, how the course is organized, what learning activities are required, how to communicate, and other general course information?

30. What Are the Consequences of Your Decision?
In a formal QM review:
What does a YES decision mean?
What does a NO decision mean?
For Standards 1.1 and 1.2, draw a table similar to this:
Ask participants to raise their hands and record their choices in the table.
Ask those who said YES what their reasoning was;
Ask those who said NO for their reasoning.
Typically, participants who say YES and participants
Who say NO will have virtually the same feedback.
Ask the group: “What are the consequences of your decision?”
Elicit the following discussion:
The outcome of the QM review is based on the collective wisdom of the peer review team who is working for the online student.  As QM freely admits, it's a somewhat subjective process...with no absolute "right" answer.  Remember, this is your chance to help improve this course…it may be the only time anyone reviews the course in this depth and with "high expectations."  
It's important to consider the consequences of your decision:  If you decide NO on the Standard, then improvements are mandatory.  If you decide YES that the Standard is met, then the recommendations are optional.  So...the team must consider how critical it is to the students who take this course that the recommendations/suggestions are implemented

31. Recommendations
Marker: The following slides deal with writing a positive, useful recommendation as part of a QM review.

32. No Choice! A recommendation MUST be written
if you decide that the course
does not meet the specific standard.
No choice here! If the reviewer decides that the standard is not met, the reviewer MUST explain why the decision was reached and offer specific suggestions for how to remediate it.

33. How to Write a Useful Recommendation
Constructive
Try to offer solutions, not just identify problems.
Specific
Include a specific example of what is being recommended.
Measurable
How will you or the ID/instructor know when the recommendation has been implemented?
Sensitive
Avoid negative language. Keep recommendations & comments on a positive note.
Balanced
Point out strengths as well as weaknesses.
The review team’s recommendations are key to success of a review. A vote of NO on a standard is not helpful without suggestions for improvement. If a course is rated "does not meet expectations," the faculty/designers will want to know how to get the "meets expectation" rating.
Comment on the strengths of the course as well as the weaknesses. Faculty members typically work very hard on their online courses and are quite proud of them. You want to recognize areas that have been well done (and that you plan to consider for your own course) as well as the areas that need improvement. Courses can "meet expectations" but still have suggestions for improvement.

34. Improve these recommendations…
Assignment instructions weren’t clear.
I wasn’t sure if assignment #1 required a written paper.
You didn't tell the students how to find the additional resources.
I had trouble locating the resources; you may want to put a link on the home page for easy access.
The text on the page was too hard to read.
The green text on a blue background was difficult to read. You might want to try basic black on white.
**** Each step comes in on a key stroke to help you walk the group through this exercise.****
Ask participants as a group to suggest improvements for each example. The steps include:
Method: You will first see the recommendation for improvement;
Press a key to reveal the recommendation that needs improvement; it will be in black text.
Ask the group how this recommendation could be improved (there will likely be several ways).
Hit the enter key again, and you will then see a possible improvement in blue text.

35. The learning objectives aren’t measurable.
I had trouble figuring out how the objectives would be measured. I think it would help if you rephrased the learning objectives using active verbs (explain, distinguish, compare, etc.)
The assessments are weak.
I had difficulty connecting the assessments to the learning objectives; they seemed unrelated to the learning objectives. I think you’ll want to review the learning objectives and make sure that they are reflected in the exams.
Method: You will first see the recommendation for improvement;
Hit a key to reveal the recommendation that needs improvement; it will be in black text.
Ask the group how this recommendation could be improved (there will likely be several ways).
Hit the enter key again, and you will then see a possible improvement in blue text.

36. Apply the QM Standards
To the ANT Course (continued)

37. General Standard 2: Learning Objectives (Competencies)
Specific Review Standard
What does this mean?
2.I: The course learning objectives describe outcomes that are measurable.
2.1: Do course-level learning objectives describe an action that the students perform that can be observed or measured?
2.2: The module/unit learning objectives describe outcomes that are measurable and consistent with the course-level objectives.
2.2: Can the module/unit level objectives be observed and measured? Do these module/unit level objectives flow from the course-level objectives?

38. Key sections that must align…
Course Learning Objectives (2)
Resources, Materials (4) & Technology (6)
Assessment and Measurement (3)
Learner Interactions & Activities (5)
Again, highlights the General Review Standards that must work together. Note that although there are only four boxes here, there are actually FIVE standards that align.

39. Remember Bloom?
University of Victoria Counseling Services - an excellent overview of Bloom's Taxonomy written from the student's perspective that provides a list of demonstrated skills and question cues for each level. The site suggests that students who know more about the types of questions that are likely to be asked can better prepare for exams.
Ask participants to take a look at Bloom’s taxonomy and decide whether or not the ANT course level objectives are written at a fairly low level or a fairly high level. This will be useful information as we progress through the essential standards.

40. Think About “Alignment”
Assignment: Read Chapter 4 in text; review study guide, objective 2, page 4.
Assignment: View Chapter 4 PPT; listen to Chapter 4 podcast
Graded Assignment: Prepare table that lists the 7 major biological areas; ask students to define/describe each area and list 3 specific ways this area affects human behavior.
Graded Discussion Forum Question: Psychology is the study of behavior and mental processes. Why, then, are psychologists interested in biological matters such as the nervous system, the endocrine system and heredity? Read and respond to classmates’ postings.
Mid-Term Exam: 50-item multiple choice exam taken in campus Testing Center.
From an Introduction to Psychology Course
Objective: Summarize the relationships that exist between biology and human behavior
This slide lists a learning objective from an Intro to Psych course and five possible examples of activities that might support this objective. Discuss whether/not and HOW each of the examples flows from or supports the objective.
Questions to consider:
If information is delivered in the first assignment, why is the second assignment important? (it serves to reinforce the first assignment and provides visual and auditory alternative delivery)
How does the graded assignment support the learning objective? (it serves as “scaffolding to ensure that students understand the basic content)
How does the graded discussion forum support the learning objective? (this is the major evidence that students have the content and understanding to meet the learning objective)
Is the mid-term necessary to verify that this learning objective is met? (No, the mid-term is an objective test that doesn’t align with the learning objective since there’s no way to measure “summarize” on such an exam. It could be eliminated since feedback is provided in the graded assignment and the graded discussion.)
What would happen if we eliminated the graded assignment and the graded discussion forum? (the learning objective is no longer measured since the mid-term does not provide an opportunity for students to demonstrate that they can “summarize” the relationships.

41. General Standard 2: Learning Objectives (Competencies)
Specific Review Standard
What does this mean?
2.3: All learning objectives are stated clearly and written from the students’ perspective.
2.3: Are the course and module/unit learning objectives written in clear, direct language that the students can easily understand?
2.4: Instructions to students on how to meet the learning objectives are adequate and stated clearly.
2.4: Does this course use narratives, bulleted lists, charts, etc. to tell the students what they must do to meet the course and module/unit level objectives?

42. General Standard 3: Assessment & Measurement (we’ll come back to this one)
3.1: The types of assessments selected
measure the stated learning objectives
& are consistent with course activities
and resources.
3.2: The course grading policy is stated clearly.
3.3: Specific and descriptive criteria are
provided for the evaluation of students’
work and participation.
3.1: The types of assessments selected measure the stated learning objectives and are consistent with course activities and resources.
Typically, we think of “assessments” occurring at the end of a unit or course and you are just beginning your review of this course. Assessments are listed as the third standard to emphasize their importance and their relationship to the learning objectives. The course assessments are also part of “alignment” and should clearly be tied to the learning objectives, materials, and technology.
Below is a possible review strategy for assessments:
Read Standard 3.I and its annotations; review the examples.
Print a copy of the learning objectives which you can refer to as you examine the course.
Review the other 5 standards (4, 5, 6, 7, and 8) of the course, identifying assessments as you find them and considering them in relationship to the objectives and other important course elements.
After you are more familiar with the course, come back to Standard 3 and make your decision.
3.2: The grading policy is stated clearly.
Again, you are taking the student’s point of view. If the grading policy is not clear (or doesn’t correctly add up), write a specific recommendation that suggests how the instructor can make it easier for students to understand it.
3.3: Specific and descriptive criteria are provided for the evaluation of students’ work and participation.
Online students need to understand exactly what’s expected of them and at what level to be successful in the course. Instructors should provide thorough explanations of how course work will be graded, how grades will be derived, how much student participation will contribute to the final grade and other aspects of the course.

43. General Standard 4: Resources & Materials
Specific Review Standard
What does this mean?
4.1: The instructional materials contribute to the achievement of the stated course & module/unit learning objectives.
4.1: Is there evidence of sufficient content for the students to learn the discipline?
Does this content support the objectives, assessments and other course elements?
4.2: The relationship between the instructional materials & the learning activities is clearly explained to the student.
4.2: Can the students easily determine the purpose of all course components & how each will help to achieve the learning objectives?

44. General Standard 5: Learner Engagement
Specific Review Standard
What does this mean?
5.1: The learning activities promote the achievement of stated learning objectives.
5.1: Do the learning activities flow from the learning objectives?
Do the learning activities encourage students to “practice” their learning?
5.2: Learning activities foster instructor-student, content-student, & if appropriate to this course, student-student interaction.
5.2: Does the course encourage student-instructor and student-content interaction?
What is the instructor’s philosophy on student-student interaction?

45. General Standard 6: Course Technology
Specific Review Standard
What does this mean?
6.1: The tools & media support the learning objectives, & are appropriately chosen to deliver the content of the course.
6.1: Do the learning activities flow from the stated course and module learning objectives? Do the activities encourage the students to “practice” their learning?
6.2: The tools & media support student engagement and guide the student to become an active learner.
6.2: Do the tools & media selected encourage the students to be actively engaged in the learning process?
6.3: Navigation throughout the online components of the course is logical, consistent and efficient.
6.3: Is the navigation of the course easy for the students to use and does it effectively guide their movement through the course?

46. General Standard 3: Assessment & Measurement
Specific Review Standard
What does this mean?
3.1 The types of assessments selected measure the stated learning objectives & are consistent with course activities & resources.
3.1: Do the assessments flow from the course and module/unit objectives? Are they consistent with these objectives?
3.2 The course grading policy is stated clearly.
3.2: Do students know how points will be awarded and/or grades assigned? Is the policy on late work explicit?
3.3 Specific & descriptive criteria are provided for the evaluation of students’ work & participation.
3.3: Do students know what work is expected? Is there a description of the expectations for this work and the criteria by which it will be graded? Do the students have access to rubrics, examples, etc?

47. General Standard 7: Learner Support
No “essential” 3-point elements
Primarily concerned with instructional support – an institutional responsibility
This standard has no “essential” 3-point elements because it’s primarily concerned with academic support services, student support services and technical support services….usually thought to be the primary responsibility of the institution and not the individual instructor.

48. General Standard 8: Accessibility
Specific Review Standard
What does this mean?
8.1: The course incorporates ADA standards & reflects conformance with institutional policy regarding accessibility in online and hybrid courses.
8.1: Does the course include these three elements? Offered using software that is accepted as “ADA compliant.”
AND
Include a link to the institution’s ADA policy and/or guidelines.
Include a brief statement that clearly tells students how to access ADA services at the institution.

49. Scenarios
Marker: The slides following this marker step participants through the post-review Evaluation.

50. Evaluating Scenarios Divide into groups.
Discuss the scenarios that focus on the 17 essential standards.
Take the “quiz” with your group. Use the material in your books and each other as references.
Jot down your reasoning and be prepared to discuss your decision.
****Pass out the Quality Matters Peer Reviewer Post-Training Assessment (the scenarios document) that will be a SEPARATE HANDOUT on colored paper in your Folder.****
Participants should work in groups of 3 (this is the number in an actual course review), if possible.
It typically takes about 30 minutes for a group of 20 + participants to complete this exercise.

51. No. Doesn’t tell students what to do first, especially a first-time college student.
No. This is a hybrid course & this example doesn’t explain relationship between online & onsite elements.
No. Objectives are not measurable.
Yes, but what suggestions can you offer for improvement?

52. No, objectives are unnecessarily complex & confusing to a new student in this class.
No, direct link to learning objectives must be provided.
Yes, but students should have requisite web skills.
No. This policy is unnecessarily confusing and complex.
No, descriptive criteria is a key element of this standard & must be provided
Consult with subject matter expert on your team, use common sense & your own experience.

53. No, it’s not clear how these materials are related to the course.
Yes, flashcards are a reasonable learning activity for the stated objective.
Yes. Be sure to consult the Instructor Worksheet for the instructor’s philosophy about student-student interaction in this course.
No. These materials are not integrated into the course.

54. Yes, all links are described and related to the course.
No. The instructions require toggling between multiple areas of the course.
No. The course is delivered through a blog & ; the instructor must provide evidence that it is ADA compliant.

55. Showcase Your Courses
Marker: The slide following this marker guides participants through the Showcase Your Courses activity.

56. Course Showcase Think about your own course:
What do you do that meets the QM standards?
What improvements might you make?
Share your thoughts with your partner
Summarize for the group
The purpose of this activity is to encourage participants to apply the QM rubric to their own online courses and begin to think about revisions/improvements they might make. If time is short, assign each row a particular standard and have them consider only that standard rather than all eight. You might also want to eliminate standards 7 and 8 in order to focus on standards that might be more difficult to implement.

57. Serving as a QM Peer Reviewer
Marker: Direct participants to the QM web page for a list of the steps necessary to be a QM Peer Reviewer.

58. Peer Reviewer Eligibility
You must be ELIGIBLE:
Must complete this f2f Applying the QM workshop
Must complete the assessments in the online Peer Reviewer Certification Module
Approximately one week online
Includes review material
Objective and written assessments
Must have recent (within last 18 months) online teaching experience
How Do I Become a QM Peer Reviewer?
As a trainer, please encourage your participants to “self-select” into the PR training. If they don’t meet the eligibility requirements AND/OR they don’t plan to actually serve on a QM review, there’s no real value in completing the assessments now. If eligibility or desire changes, they can always enroll in the online certification for a small fee.
The procedures for registering and accessing the online Peer Reviewer Certification module are not yet complete. Trainers will be sent this information as soon as it’s available.

59. Evaluation Please fill out the session evaluation in your folder.
Marker: This slide prompts participants to complete the Evaluation Form on colored paper in the folder. This form is very important and should be completed before participants leave the session.

60. Thanks to YOU…Quality Matters!
More information at:
Marker: This is the ending slide of the presentation and can be left up while participants are exiting the room.

61. Quick Review
Marker: Immediately after the lunch break – whenever that occurs in your session, use the following slides as a quick review of the QM underlying principles.
After you complete the review, distribute the Quick Quiz and allow participants time to complete it. Go over the correct answers with the participants so they will have immediate feedback.

62. Underlying Principles of QM…
Based in national standards of best practice, the research literature & instructional design principles
The course under review does not have to be “perfect” but better than just “good enough” (Standards met at about 85% level or better.)
Integral to a continuous quality improvement process
Valued as part of a faculty-driven, peer review process
Designed to promote student learning
In early iterations of the Peer Reviewer process, many participants focused on HOW to do the review, but missed the WHY. This slide emphasizes the key concepts behind QM and how they contribute to a quality course.
It’s not necessary to read these aloud in order… Briefly talk about each item that you may not have raised in earlier slides.

63. Underlying Principles of QM (cont)….
Designed to ensure all reviewed courses will eventually meet expectations
QM is a collegial review process, not an evaluation process
A review team must include an external peer reviewer
Set up so that the course faculty or instructor considered part of the review team
Emphasize the first bullet: If the QM process is followed, ALL courses will meet expectations. This is an important concept that new Peer Reviewers often miss.

64. Who’s On the Team? Formal Peer Review Team 3 faculty peer reviewers
must attend QM training
must be experienced online instructors
one MUST be external to the course’s originating institution
there must be a subject matter expert (SME) on the team NOTE: The SME could also be the external reviewer
chaired by additionally trained QM reviewer
AND
Faculty course developer (Instructor)
access to rubric prior to review
involved in pre-review discussions
consulted during review
Formal Peer Review Team
Be sure to point out that there a 3 faculty peer reviewers who actually complete the web-based review form. One of these reviewers must be a Subject Matter Expert and at least one must be external to the course’s home institution.
Emphasize that the Faculty Course Developer (the Instructor) is a part of the review team and should be consulted by the team.
You might want to ask participants if any of them are Faculty Course Developers…since the Faculty Course Developer is always invited to attend training prior to the review.

65. To Meet Expectations…Two Thresholds
A course must achieve:
“Yes” on all 17 of the 3-point “essential” standards.
A minimum of 72 out of 85 points
There are TWO thresholds for meeting expectations and a course must meet both thresholds to meet expectations. Mathematically, this works out to about an 85% standard.

66. Online & Hybrid Courses
Rubric designed for both
Same set of standards apply to both
How standards achieved may differ
For hybrids, focus on pedagogical integration of online & onsite components
The version of the rubric includes the necessary elements so that it can be used to review a hybrid course.
A course can be reviewed using the hybrid rubric if it meets the following criteria: The course is organized around a Course Management System and the major course elements (including what will be covered at the f2f meetings) are available through the CMS.

67. Quick Quiz
Marker: Immediately after the Quick Review, use the Quick Quiz.
Method:
Distribute the 10-question objective quiz about the rubric, underlying principles and scoring.
It will take participants only about 10 minutes to complete.
Go over the answers with the participants so they have immediate feedback.