Applying the QM Rubric Rubric

Applying the QM Rubric 2011-13 Rubric
Welcome Screen: Show this slide on the screen prior to the session to help your participants identify that they’re in the right place and in the right session. Add your institution and date in a text box to make the information more specific. Last revision date: 05/30/2014 ©2014 MarylandOnline

QM Website www.qmprogram.org
Workshop Overview Today’s Agenda Folder Overview QM Website QM Contact Information About The Content: Briefly highlight the following for the participants: Today’s Agenda: Participants do not usually have a copy of the agenda; this allows the QM trainer to modify the agenda if time problems occur. Point out when lunch and breaks occur; tell participants how to find restrooms and drinking fountain. Folder Overview: Briefly go over the materials in the participant’s folder. NOTE: QM only provides folders and workbooks for QM-sponsored training. Independent sessions: the facilitator is responsible for preparing the packets and making sure that everyone purchases a workbook. Have the participants take out their workbook to use throughout the session but suggest they take notes on paper and not in the workbook so they can use this in the future. QM Website: If you have an Internet connection, locate the web site at and point out the highlights. QM Contact Information: Best guarantee of an answer about QM is through Deb Adair, QM Director of Administration. Deb’s address is Be sure to include “QM” in the subject line. Nancy Ragias, QM’s higher education registrar, can also answer questions about professional development opportunities ©2014 MarylandOnline

Login to MyQM Go to MyQM http://www.qmprogram.org/myQM/
Click on Need an Account? link (lower part of page) or login if you already have an account Review the MyQM site: My Account, My Activity, My Tools, and Workshops Certificates ****HANDOUT: Login to MyQM**** About The Content: MyQM is a personalized, secure area of the Quality Matters Program website. It is a real-time data repository for your institution's complete QM Program history and involvement. It tracks individuals at your institution who are active with the Quality Matters Program - listing trainings, peer reviewer eligibility, courses submitted for review, courses reviewed by those at your institution, attendance at conferences and events, and more. Upon entering MyQM, you have the ability to: Update your profile including contact information Access all QM Program Activity associated with your profile (i.e., training activity, course reviews) Access all QM Program Activity associated with your institution (IR accounts only) View courses submitted for review See real time training registrations Access a current list of eligible peer reviewers for your institution (IR accounts only) To view content in this area, you must login. Remember that your username is the address associated with your profile. ©2014 MarylandOnline

After this workshop, you will be able to:
Identify the underlying principles of QM. (Recognize key QM underlying principles and concepts.) Identify the critical elements of the QM quality assurance program, including the QM Rubric, materials, processes, and administrative components. Apply the QM Rubric to review online courses. Decide if the SPCH 1113 course meets selected QM Rubric standards. Apply the concept of alignment. Write helpful recommendations for course improvement by citing annotations from the QM Rubric and evidence from the course. Content: There are six learning objectives to review with the participants; each is “assessed” to some extent in this workshop. Be sure to point out that the purpose of the workshop is to teach participants about Quality Matters, the QM Rubric, and prepare the participants to apply the QM rubric to their own or another’s online course (not to teach them how to teach online). Note: The day’s evaluation is based on how well the participants believe that these objectives were met. This workshop is specifically designed for two audiences: 1. Participants who want to learn about and apply the QM rubric and process Who: Faculty members, instructional designers, administrators who want to know more about QM and apply the process but who are not eligible to become peer reviewers OR who do not plan to serve on an official QM peer review team. 2. Participants who want to serve as certified QM Peer Reviewers Who: Online instructors who meet the following criteria will be eligible to serve as QM Peer Reviewers upon successful completion of the online Peer Reviewer Course: You must complete the Applying the QM Rubric workshop. You must have taught a for-credit online or blended course within the last 18 months. You must complete the two-week online Peer Reviewer Course and complete the application and MOU to be registered in the QM Database. ©2014 MarylandOnline

And in addition, we expect that you will:
Collaborate and network with colleagues through substantive and timely interactions. Reflect on key QM concepts and whether you wish to serve on a peer review team. Explore the challenges of online teaching by familiarizing yourself with the QM rubric and the SPCH 1113 course. Relate the QM rubric (standards and annotations) to your own course and consider changes that might benefit your online students. Evaluate the QM APPQMR workshop. Content: These are additional “goals” of the workshop. NOTE: You can use these as an example of how to indicate your goals for a course or workshop that are important, but that you cannot or do not plan to assess. ©2014 MarylandOnline

Activity 1: Introductions
Share name, institution, job, and best distance learning practice. Briefly (in one sentence) describe your best distance learning practice. About The Content: Introductions Exercise Ask participants to pair up for introductions. Suggest that they interview someone they don’t know (this may require some moving). Give them a few minutes to talk. Go around the room and ask them to introduce either themselves or their partners. Your challenge here is to convince them to be BRIEF. Encourage a one-sentence response to the best practice. Note: If time is an issue (perhaps you started a few minutes late or you have a large group), remove the second bullet and ask them only to “Share name, institution, role and distance learning experiences.” ©2014 MarylandOnline

Activity 2: Trying it out with Standard 1.2
Standard 1.2: Students are introduced to the purpose and structure of the course. About The Content: Apply Standard 1.2 This exercise is useful as a way of “introducing” key QM concepts. It allows the facilitator to quickly touch on the following concepts: QM is student-oriented; QM is about continuous improvement; QM is practical; QM is balanced (identify strengths and areas for improvement); QM is about collaboration and working with a team; and QM sets a higher standard. Although the participants haven’t yet been formally introduced to the QM rubric, this exercise provides an engaging, hands-on introduction to the day’s activities. Review Standard 1.2 with the participants, directing them to look at the point value and annotation in the QM Workbook. Then move to the next slide. ©2014 MarylandOnline

Review Scenario You are reviewing a College Algebra course and find the following statement: This course reflects a shift in the importance that the world outside the schools increasingly places on thinking and problem solving. Procedural skills alone do not prepare students for that world. Therefore, students deserve a curriculum that develops their mathematical power and an assessment system that enables them to show it. Assessments that match the current vision of school mathematics involve activities that are based on significant and correct mathematics. These activities provide all students with opportunities to formulate problems, reason mathematically, and make connections among mathematical ideas. Students engage in solving realistic problems using information and the technological tools available in real life. Moreover, skills, procedural knowledge, and factual knowledge are assessed as part of the doing of real life mathematics Tests (Must be taken on campus) Quizzes Project (Optional) 50 Total number of points possible 475 (no project) 525 (if you do the project) Grading Scale: 90% to 100% = A 80% to 89% = B 70% to 79% = C 60% to 69% = D 0% to 59% = F A student who earns a final average of 70% or more on all assessments of objectives and intended learning outcomes for the College Algebra course has successfully fulfilled the general education and other essential core skill goals. Any extra credit problems are given to the entire class. There are no extra credit problems for any one individual during any part of the semester. **** Handout: The Review Scenario STD 1.2**** About The Content: Review Scenario: Standard 1.2: Students are introduced to the purpose and structure of the course. Refer participants to the scenario (a copy of the scenario should be in the participant’s folder). Give participants a few minutes to discuss the scenario either in small groups or as a whole and make a decision about whether or not the standard is met. Participants typically note the following: Strengths: The instructor made an effort to share his philosophy with his students; it’s clear he appreciates Math and hopes they will as well. The grading policy is stated but could be clearer. The underlined words are an attempt to highlight important concepts and give them emphasis (although some participants will see these as links). Areas for improvement: the tone of this “welcome” is not friendly, inviting (and may even be threatening considering it’s a Math course) this presents the instructor’s philosophy but doesn’t clearly give much specific, practical information about the course the grading policy presented in fuzzy, unclear, nonspecific – and probably raises more questions than it answers Most participants decide that this scenario does not meet the standard: While the student is introduced to the instructor’s philosophy of the course and the grading scheme, there is no indication of how the learning is structured, how communication will occur and what the student must actually do to complete the course. ©2014 MarylandOnline

Underlying Principles of QM
QM is a faculty-driven, peer review process that is… Collaborative Collegial Continuous Centered - in academic foundation - around student learning About The Content: The underlying principles of QM are a primary reason for this wide-spread adoption. Quality Matters provides a faculty-driven, peer review process that is… Collaborative: QM was designed by and for faculty to share expertise and experience relative to the design of a course. Collegial: The course review process is a collegial discussion between faculty peers committed to Continuous quality improvement. It is not an evaluation. Centered in national standards of best practice, the research literature and instructional design principles designed to promote student learning. ©2014 MarylandOnline

Peer Course Review Process
About The Content: QM Circle Process: This visual highlights the QM review process (point out that it is also on the cover of the QM Workbook). Key points: QM is designed for continuous improvement; the goal is that ALL courses will eventually meet QM expectations. How To Present It: Ask participants to look at the diagram and make inferences about it. Typically, they will respond with “it’s a circular process;” “it’s continuous:” etc). Begin with the COURSE and then continue through PEER COURSE REVIEW, FEEDBACK, COURSE REVISION and MEETS QUALITY EXPECTATIONS. Speak briefly about how each item contributes to the process. Participants will learn more about each item throughout the workshop, so your objective at this point in the presentation is to focus on: QM is about continuous improvement. All courses will eventually meet expectations. QM is both a rubric and a process. NOTE: The information below is for background only; you will go into greater depth about this slide later in the workshop. This diagram illustrates the focus on continuous improvement and summarizes the Quality Matters course review process: COURSE: Beginning at the top of the cycle, institutions decide to examine an online or hybrid course as part of a peer review. The institution may submit its course to QM for a formal review or, if the institution is a QM subscriber, it may conduct the formal course review in-house (both a formal QM review and a formal in-house process lead to QM recognition). The institution may also decide to use the rubric informally and do a review using its own process (this informal process does not lead to QM recognition). PEER COURSE REVIEW: The course is then reviewed by a team of three peer reviewers using the QM Rubric. The QM rubric is based in national standards of best practice, the research literature, and instructional design principles. Peer reviewers must have online teaching experience and complete the Peer Reviewer Training to be eligible to serve on a formal QM review. The peer review team consists of at least one member from an institution other than the course’s home institution. The team also consists of one member from a discipline that matches that of the course. This combination of reviewers ensures a diverse set of perspectives. FEEDBACK: Following the course review, the review team’s feedback is provided to the faculty member or team that developed the course. The feedback consists of two components: Scoring - indicates which QM standards were and were not met by the course. Feedback - a rich set of comments from the reviewers indicating the strengths of the course, areas for improvement and specific recommendations and suggestions for improving the course. COURSE REVISION: Upon initial review, the course may or may not have met Quality Expectations. In either case, the QM review provides support for course revisions and improvement. If a course did not initially meet Quality Expectations, the team chair will re-review the course after revisions. MEETS QUALITY EXPECTATIONS: QM expects that all courses will work toward and achieve quality expectations. The QM review process is not meant to be a test in which a course passes or fails. The overall goal is to provide a system for the improvement of course quality, rather than the simple assignment of a grade or quality level to the initial course. ©2014 MarylandOnline

For Our Purposes, Quality Is…
More than average; more than “good enough” Attempt to capture what’s expected in an effective online course at about an 85% level Based on research and widely accepted standards About The Content: This slide presents a QM KEY concept: Emphasize that QM is not looking for “just good enough.” QM reviewers are looking for above average (approximately 85% or B+). Point out that although this is somewhat subjective, the basis for the decision is based in the standards and the annotations which are grounded in the research literature and widely accepted standards about effective distance learning. Peer Reviewers are all experienced online faculty who have attended QM professional development and learned to apply the rubric. QM relies on the experience, expertise and common sense of its faculty reviewers to conduct reviews fairly and consistently and to judge whether the course meets expectations at the “85%” level. The emphasis on above average is also the reason that QM primarily reviews “mature” courses (taught at least two semesters). If the course has been taught over several semesters, the faculty developer has had time to “fine tune” the course and to make management and content improvements. Note for Facilitators: Many new Peer Reviewers find the concept of 85% confusing. There are actually TWO 85 % thresholds: 1. The first is that a course must earn at least 81 out of 95 points (or about 85%). The points from all MET or NOT MET decisions on specific standards are added and must be at the 85% or greater percentage - AND meet all 21 essential standards - to be QM recognized. 2. The second is that peer reviewers should use the 85% rule to determine whether or not a specific standard is met. The standard does not have to be 100% to be a “met". The "85% rule" is a guide for reviewers to gauge whether they will decide “met" or "not met" for that particular standard. The assumption is that experienced online faculty members will have a "gut level" recognition of what that means. ©2014 MarylandOnline

Factors Affecting Course Quality
About the Content: There are many factors that affect the quality of an online course. Among these factors are: the course design (the forethought and planning that goes into an online course,) the course delivery (the way the course is taught, also known as faculty performance), the course content, the learning management system and its functionality; technical support, the institutional infrastructure (help desk, online library access, online tutoring access, etc.), a faculty member’s training and readiness for online teaching, and the students’ role with respect to engagement and readiness for an online course. QM reviews just one aspect of online course quality – Course Design. NOTE: you don’t have to read this list. You should emphasize design and delivery, of course, but you might also highlight the LMS (particularly if there have been recent issues with it that disrupted classes; or the institutional infrastructure (the availability of online student services or a 24/7 help desk); or student readiness (particularly if it’s a rural population with limited Internet access at home). QM Reviews Course Design ONLY ©2014 MarylandOnline

The faculty member is integral to both design and delivery.
Design vs. Delivery The faculty member is integral to both design and delivery. QM is about DESIGN - not delivery or faculty performance Course Design … is the forethought and planning that a faculty member puts into the course. Course Delivery … is the actual teaching of the course, the implementation of the design. About The Content: This slide reinforces the concept that QM is about Design, not Delivery. QM recognizes that this is a fine line, but course design is the primary emphasis during a review. Design is usually about what the faculty member plans and prepares for BEFORE the students arrive in the course; delivery is what happens after the students login and begin to engage with the content, the instructor and other students. ©2014 MarylandOnline

Distinguish between design and delivery…
Example: Discussion Board Design: Discussion board planned in course; students told how they should participate and how they can expect the faculty to participate. Delivery: How often the faculty member actually participates in the discussion; what the faculty member actually says to students. About The Content: This slide presents a concrete example of how you might distinguish Course Design from Course Delivery. Feel free to use an example from your own course if you have one. ©2014 MarylandOnline

Activity 3: Design vs. Delivery
Find a partner (or two) Share stories about your own experiences with design vs. delivery (professional or personal) Decide on an experience that you’d like to share with the group. This activity is an opportunity to engage participants and usually promotes discussion and laughter. It’s very cathartic to share what worked…and what didn’t (sometimes quite spectacularly). How To Present It: Ask the participants if they can think of their own examples (from “real” life or from their teaching experience) and share them with the group. ©2014 MarylandOnline

What QM is NOT About… Not about an individual instructor (it’s about the course) Not about faculty evaluation (it’s about course quality) Not about judgment (it’s about diagnosis and improvement) Not about “win/lose” or “pass/fail” (it’s about continuous improvement in a supportive environment) About The Content: The purpose of this slide is to focus on what QM is and is not: about course design, not about the instructor about course quality, not about faculty evaluation score that comes out of a QM course review is Diagnostic rather than Judgmental. It tells us how much revision a course may need. about continuous improvement, not about pass or fail You can acknowledge that QM is walking a fine line because it is difficult to separate the course design and the instructor. However, the intention of a QM review is to focus on the course design. Rather than reading the negatives, you might want to focus primarily on what QM is about. Be sure to emphasize that QM users make an effort to avoid the terms “pass” and “fail” since these can carry negative connotations associated with grading (particularly for academics). ©2014 MarylandOnline

About The QM Rubric Eight General Standards: Course Overview and Introduction Learning Objectives (Competencies) Assessment and Measurement Resources and Materials Learner Engagement Course Technology Learner Support Accessibility Key components must align. **** Handout: Participants must have a QM Rubric Workbook to use during the training.**** About The Content: There are 8 main sections of the rubric (these are the 8 General Review Standards and are noted in red letters in the QM Workbook). The rubric consists of 41 Specific Review Standards which are distributed over these 8 general categories. Take the time to point out the organization of the QM Workbook and how the rubric is organized: Eight General Review Standards Forty-one Specific Review Standards In Table Format: 1st column is Specific Review Standards; 2nd column is point value; the 3rd and 4th columns provide space to make a Yes or NO decision; and the last column is the annotation. Talk about QM’s intention to be “holistic” and that a QM review is intended to ensure that all parts of the course work together. You can use the metaphor of a cake recipe. Baking a cake is fairly simple if you follow the recipe and correctly measure and add the right ingredients (and the result is also fairly simple and straightforward: if you do it correctly, you end up with a cake). Reviewing an online course is much more complex: not only must you include all the “ingredients” but they must all work together to support the learning objectives. For example, you could have strong, measurable learning objectives but if they don’t align with the assessments, you still don’t have a quality online course. Note: Your discussion of alignment should be brief and introductory; you will spend more time explaining “Alignment” in another section of the presentation. Alignment: Critical course elements work together to ensure that students achieve the desired learning outcomes. ©2014 MarylandOnline

Key Sections that Must Align
About The Content: The QM Rubric takes a holistic view of the course and that's why it's so important that the learning objectives ALIGN with the assessments, resources and materials, student interaction, and technology standards. Alignment refers to the direct link between the learning objectives (Standard 2) and the assessments and measurements (Standard 3), resources and materials (Standard 4), learner engagement (Standard 5), and course technology (Standard 6). Under the principle of alignment, these later four aspects of the course are driven by and support the learning objectives. As you begin to apply the QM Rubric, pay special attention to the six specific review standards (2.1, 2.2, 3.1, 4.1, 5.1 and 6.1) in the rubric that include alignment matters (and the red word “Alignment” included in each standard box). The diagram on this slide illustrates the dynamic relationship that supports alignment between these standards. Submitted by James Fowlkes & Brenda Boyd ©2014 MarylandOnline

QM Rubric – Standard Point Values
8 General Standards 21 Standards are “Essential” 12 Standards are “Very Important” 8 Standards are “Important” # Standards Points Relative Value 21 3 Essential 12 2 Very Important 8 1 Important About The Content: If you haven’t already done so, point out the breakdown of the 41 Specific Review Standards by points: 21 Standards are ESSENTIAL to an online course and worth 3 points each. Without all essential standards being met at the 85% level, a course cannot be considered a quality online course. For example, note the differences between 1.1 and If standard 1.1 is not met, there will be no (or unclear) instructions on how to begin and student will not know what to do first. If students are confused when they first enter the course, it’s likely they will not persist and may withdraw from the course or just stop participating. Research shows that standard 1.7 is important: students want to know about their instructors (particularly that the instructor is prepared, enthusiastic about the discipline and caring about his/her online students). However, students could learn about their instructors in many other ways (daily announcements or s; discussion forums; and/or through helpful feedback). 12 Standards are Very Important to an online course and are worth 2 points each. These 12 standards are Very Important to an online course and each contributes to the likelihood of student success. In most cases, there could be another way to meet these standards, so a particular specific review standard could not be met and there are other ways to access the information, content or instructions. For example, Standard 1.4 is worth 2 points because it’s important, particularly to online students, that course and institutional policies are clearly stated. However, a student could discover these policies through an institution website or by simply asking the instructor to expand on the syllabus. 8 Standards are Important to an online course and are worth 1 point each. These standards are Important in terms of best practices in online learning and/or in the research literature. Typically, they are other ways for students to access this information, content and/or instructions and the course might stand as a quality course even if the standard is not yet met. For example, take a look at standard 1.5; students could learn about the prerequisites through the course schedule or on a college website (or perhaps even on their registration confirmation). This results in a total of 41 Specific Review Standards and a total of 95 points. ©2014 MarylandOnline

Activity 4: QM Rubric Puzzle
Divide into groups of 3 You will have 10 minutes to: Decide which general standard each specific review standard supports Decide whether your standard is essential, very important or important. Note: your group does not have come to agreement on where to place the specific standard or its relative value; majority rules Facilitator Instructions: Pre-Workshop: Print a Title page for each of the eight general standards (slides are available as a separate ppt file). Print the spreadsheet file that contains each of the 41 specific review standards (you’ll have 8 standards on each page; you’ll cut these into strips for the activity). Put them in an envelope (mix them up) so you don’t lose them. Gather as many rolls of clear tape as you can (one for each 3 participants, if possible). Participants will use the tape to place the Specific Review Standard below its General Standard. Note: the General Standard slides need to be printed on colored paper. Yellow paper for the Title pages and Light Blue for the specific standards. Objectives for this activity: Consider the 8 general standards and how the rubric is organized. Make decisions using the “Power of 2” principle (working with a 3-member team, 2 participants must agree on both the General Standard and the point value) Discuss the value of organization in the rubric and of working with a team. Instructions: Before the session: Tape each general standard around the room, spacing them so participants can tape the specific review standards below each. Activity: Assign the participants to a team (should be 3 to a team but occasionally you may have a team with an additional member) Distribute randomly and as equally as possible the 41 QM specific review standards in the envelope to the teams. You can have each team select them OR you can distribute them by laying them face-down on the desk or table. Participants work in teams of three to assign each specific review standard to a General Standards AND assign the relative point value for each. Participants then tape each specific review standard strip under the appropriate General Standard. Before a standard can be placed, the 2 out of the 3 members of the team must agree on its placement. Debrief: Debrief first by just looking at the number of strips under each General Standard. Make corrections if necessary. Discuss the value of having these important specific review standards organized into a meaningful Rubric…ask them to imagine how difficult it would be to apply just a collection of random standards! ©2014 MarylandOnline

The Faculty Developer (Instructor) Worksheet
Marker: The following slides highlight the importance of and specific information contained in the Faculty Developer (Instructor) Worksheet. ©2014 MarylandOnline

Faculty Developer (Instructor) Worksheet
The Voice of the Instructor Key Piece in the review Includes information about Institutionally mandated objectives, materials, practices, and policies Materials outside the course site Types of interaction used and instructor’s statement on the appropriateness of interaction in the course Read it before you begin the review Refer to it during the review and in team discussions **** Handout: The Faculty Developer (Instructor) Worksheet for the SPCH 1113**** About The Content: Specifically refer to and discuss the following numbers from the Faculty Developer (Instructor) Worksheet, since these directly impact the standard decisions: Item 12: The instructor must provide (or link to) his/her institution’s accessibility policy. Item 17: Directly related to Standard 2.1 Learning Objectives. QM does not hold individual faculty members responsible for measurable learning objectives if the faculty member is REQUIRED to use those objectives. NOTE: The instructor must attach a list of course level objectives and a sample of the module level objectives. Item 20: Directly related to Standard 6. Course Technology. If the course uses multi-media, the Instructor should provide information about how to access it. Item 24: Directly related to Standard 5.2 Learner Interaction & Engagement. The instructor decides whether student-to-student interaction is important to this course. Items 27 and 28: The Instructor has an opportunity to list areas of concern and seek responses from the review team. ©2014 MarylandOnline

Activity 5: Faculty Developer Worksheet
Locate the Faculty Developer (Instructor) Worksheet in your packet Read the Worksheet, be sure to pay close attention to the instructor’s answers to: Course- and Module-Level Objectives Student Engagement Technology Accessibility Assessment The Faculty Developer Worksheet is extremely important in a QM review since it provides the instructor’s perspective. Participants will not take the time to actually read the Faculty Developer Worksheet unless you specifically ask them to do so. Activity: Divide into groups and Ask each group to look at all of these topics: objectives, assessment, student engagement, technology, accessibility. OR Assign each group to look at only one of these topics: objectives, assessment, student engagement, technology, accessibility. Report back on findings: What was helpful? What was unclear? What suggestions for the next iteration of the Faculty Developer Worksheet? ©2014 MarylandOnline

Goals for Hands-on Practice
Introduce the General Standards Use the Faculty Developer Worksheet Take an in-depth look at Alignment Make decisions on specific review standards by looking at an online course Practice writing useful recommendations See how the recommendations affect a course About The Content: This slide reminds you and your participants about why the hands-on practice is important to the workshop. ©2014 MarylandOnline

Your Point of View … As a QM Peer Reviewer, you should:
Take the students’ point of view Advocate for the student Support your decisions with Citations from the standards and annotations Evidence from the course About The Content: Emphasize that reviewers should assume the students’ point of view. This is an opportunity to see the course from the students’ point of view (not the faculty member’s or the institution’s). This point of view often reveals aspects of the course that could be improved to make navigation easier and improve the learning environment for the students...aspects that may not be obvious to the faculty developer because he/she is just too close to the course. Encourage participants to support their decisions with citations from the rubric AND evidence from the course. If they can’t find the evidence they need, they should not assume it is or isn’t there. If the review team needs help, they should consult with the instructor to determine if the required evidence is somewhere in the course. This might be a good place to ask if any of today’s participants have been students in an online course. Ask them if being a student changed the way they think about online teaching. ©2014 MarylandOnline

Accessing the SPCH 1113 Course
Find a Partner Read Faculty Developer (Instructor) Worksheet for course Login to Principles of Speech (SPCH 1113) URL: Username: qmstudent Password: qmstudent **** Handout: How to Login to SPCH 1113 Course**** About The Content: As the QM facilitator, you are now taking the participants on an adventure through the rubric, looking at alignment. The participants are doing this exercise with a partner. Participants are logging into the Principles of Speech” (SPCH 1113) course on the Southern Arkansas Blackboard site. Explain the participants they need to open a new browser, enter the URL, and login with the username and password. The participants will have a handout that contains this information. Options if the SPCH 1113 course isn’t available (the Blackboard site has been extremely reliable, but it’s always good to have options just in case): Ask participants to use their own online courses to do the Hands-on Practice activities (you won’t have an faculty developer (instructor) worksheet). Use a demo course that you use for training on your own campus (you won’t have an faculty developer (instructor) worksheet). ©2014 MarylandOnline

Strategies for Applying the QM Rubric
General Standard 1: Course Overview and Introduction Marker: The following slides present strategies for applying the QM Rubric and introduce General Standard 1. ©2014 MarylandOnline

General Standard 1 About the General Standard Brief Description
from the Rubric To meet the General Standard 1, Course Overview and Introduction: The overall design of the course is made clear to the student at the beginning of the course. The course introduction sets the tone for the course, tells students what to expect, and provides guidance to ensure they get off to a good start. You must share your deep knowledge of course Organization Navigation Expectations (Policies) About The Content: Direct participants to look at the Rubric Workbook as you guide them General Standard 1. 8 Specific Review Standards support the general standard: 2 Essential Standards 2 Very Important Standards 4 Important Standards Introducing General Standard 1: Let's put standard 1 into perspective: You've just finished designing your online or hybrid course, creating rich content, connected activities and aligned assessments. The course is rigorous and meets your pedagogical standards. You're done, right? What you might not recognize is that you're at a pedagogical challenging point of online course design. As the designer you know how the course is organized and where/how students are to begin. But, you haven't shared this with your students! What becomes the easiest thing for a designer to forget at the end of the process - that is, clear instructions to the students about how to begin - becomes the first thing that students are faced with when they click into an online course. Imagine yourself in your f2f classroom. What would happen if you didn’t show up on the first day of the course, stand in the front of the room, welcome your students, provide information and give directions? Right…nothing! The students would at first be confused, then angry…and then leave (perhaps to go visit your department chair’s office).The same is true for your online classroom…you can’t expect your online students to know what to do to get started. As the course designer, you definitely have the inside track on this one. After all, you put the course content, activities, and assessments in the most logical place - you can locate everything in a heartbeat! You know exactly where to click once you login to the course. You might catch yourself thinking, "anyone would intuitively know to click on the correct place to get to where they need to be." But, of course, it doesn't work that way. You have to tell them how to get started in and be successful in YOUR course. The acronym ONE can be useful when you’re getting ready to make sure you’re addressing the important issue of making sure your design includes ways to welcome students to your online course. Organizing to give your students a good start by presenting the whats, whens, whys, and hows of the course design. Navigating and leading students to locate key course components and make connections that will promote a positive feeling for students. Establishing and explaining relationships to help your students recognize learning relationships in your course. How things “fit” together in your course. ©2014 MarylandOnline

How To Decide… For EACH Standard
Read the specific review standard and the annotation; review the examples. Identify key components: ideas, directions, examples, etc. Ask relevant questions Look for evidence that the standard is met in this course. Ask yourself: Does this course meet the standard at an 85% or better level? Decide Met or Not Met. Write Helpful Recommendations. ****Handout: How To Decide**** About The Content: This slide presents a strategy for reviewing the 41 specific review standards. The participants have a copy of this slide as a handout in their packet. Have them take it out and use it throughout the hands-on portion as they learn and apply the specific standards to the SPCH 1113 course. New QM reviewers are always uncertain about how to proceed (they want to do the right thing, but may be confused about the process at this point in the training). This strategy reinforces the key aspects of the QM review AND emphasizes that they are making a MET/NOT MET decision. ©2014 MarylandOnline

Look at a Helpful Recommendation
****HANDOUT: Helpful Recommendations Reference**** About The Content: The success of a QM review depends on the rich, robust feedback that is compiled for the instructor. This graphic emphasizes the components of a helpful recommendation. Direct participants to review the standard, the annotation and course evidence (their decisions should not be based on personal preference). NOTE: This slide visually depicts the components that must be includes when writing a helpful recommendation. Your discussion should be introductory but fairly brief. You will discuss writing helpful recommendations at length when you get to General Standards 7 and 8. ©2014 MarylandOnline

Review Standard 1.1 1.1 Instructions make clear how to get started and where to find various course components. As a Reviewer: What do you do first? Where do you start? (Refer to the “How to Decide” handout) This slide introduces specific review standard 1.1 and asks Participants how they plan to start (you can remind them to refer back to the Handout: How To Decide. NOTE: Each specific review standard covered in this workshop will be introduced in a similar way. The discussion of standard 1.1 will take more time since you are teaching your participants how to use the Rubric, make decisions and how to evaluate the consequences of those decisions. Once you have established this foundation, the other standards will typically take less explanation. ©2014 MarylandOnline

Read the Annotation Instructions provide a general course overview, present the schedule of activities, guide the new student to explore the course website, and indicate what to do first, in addition to listing detailed navigational instructions for the whole course. Instructors may choose to incorporate some of this information in the course syllabus. In this case, students should be directed to the syllabus at the beginning of the course. A useful feature is a “Read Me First” or “Start Here” button or icon on the course home page, linking students to start-up information. Examples: A course “tour” Clear statements about how to get started in the course A “scavenger hunt” assignment that leads students through an exploration of the different areas of the course A graphical table or diagram that depicts the relationship between the online and face-to-face portions of a blended course Blended Courses: Instructions in the online classroom make it clear to students that the course is a blended course, with both online and face-to-face components and activities. Instructions specify the requirements for participation in both the online and face-to-face portions of the course. The introductory information clearly states when and where students should participate each week, and a structured set of topics and a schedule are provided for each face-to-face meeting. ****Handout: Analyze the Specific Review Standard**** This is the annotation for Standard Ask participants to open their QM Workbooks to this standard to view the standard and its annotation. Take time to point out the organization of Standard 1.1’s annotation: Key ideas (information) Where to look Examples Special situations: Blended Courses Ask participants to look at Handout: Analyze the Specific Review Standard for prompts on how to analyze the standard. ©2014 MarylandOnline

Standard 1.1 – How to use the Annotations
Annotations specify: 1. What to look for: Instructions: a general course overview, schedule, website Indication of what to do first Detailed navigation instructions 2. Where to look: Course syllabus “Read Me First” or “Start Here” button or icon 3. Examples Questions to ask: 1. Does the instructor provide instructions about key course components? 2. Do the students know what to do first and how to start the course? This is a “self-check” to see how well participants were able to analyze Standard Ask participants to compare their answers to the slide…and ask if there are any questions. ©2014 MarylandOnline

Apply Standard 1.1 Apply Standard 1.1 to the SPCH 1113 course
Imagine that you are a new student (new to the course, the discipline, and/or the Learning Management System) Consider the following questions: Do you know what to do first? Do you know how to start the course? Can you easily find clear instructions? Note your initial reaction to the course: Do you feel comfortable with the navigation? What are your first impressions about how this course works? About The Content: Standard 1.1 of the Quality Matters Rubric states: Instructions make clear how to get started and where to find various course components. This is seen as an essential standard in the QM Rubric. A word about Standard 1.1: As you’ll note in the annotation, there are many ways to meet this standard. Instructors often prefer using a Start Here button since it’s so clear. As you review a course (or apply standard 1.1 to your own course), keep in mind that just having a Start Here button is not enough. You also need to evaluate it: does it provide clear directions for how students are to get started in this course? Some Start Here areas provide information only and don’t specifically tell the students how to begin the course. How To Present It: Apply standard 1.1 to the SPCH 1113 course: 1. Participants should already be logged in to the SPCH 1113 course. If not, refer them to the handout in their packet with the login information. 2. Imagine that you are a new student (new to the course, the discipline, and/or the course management system) 3. Answer the questions posed on the slide. ©2014 MarylandOnline

Activity 6: Your Decision
MET NOT MET Standard 1.1 About The Content: This slide presents a grid that you can use to record your participants’ decisions. About How To Present It: We recommend polling your participants before you discuss the rationale for the decision. This will give a more accurate “picture” of what your group is thinking (and perhaps indicate areas that need more guidance). After you record participants’ decisions, discuss WHY they decided MET or NOT MET. Ask them to cite the annotation AND provide evidence from the course to support their decision. Be sure to solicit both strengths of the course and areas that need improvement (a QM review should be balanced). After the discussion, be sure to ask participants if anyone has changed their mind (point out that it’s perfectly acceptable, even encouraged, to change your mind when you serve on a QM review team)! ©2014 MarylandOnline

Consequences of Your Decision?
In a formal QM review: What does a MET decision mean? What does a NOT MET decision mean? About The Content: This slide continues the discussion about Standard 1.1: Ask the group: “What are the consequences of your decision?” Elicit the following discussion: The outcome of the QM review is based on the collective wisdom of the peer review team who is working for the online student. As QM freely admits, it's a somewhat subjective process...with no absolute "right" answer. Remember, this is your chance to help improve this course…it may be the only time anyone reviews the course in this depth and with "high expectations." It's important to consider the consequences of your decision: If you decide NOT MET on the Standard, then improvements are mandatory. If you decide MET on the Standard, then the recommendations are optional. So...the team must consider how critical it is to the students who take this course that the recommendations/suggestions are implemented. ©2014 MarylandOnline

Importance of Learning Objectives
General Standard 2: Learning Objectives (Competencies) Marker: The following slides present the importance of General Standard 2, Learning Objectives (Competencies) to the QM Rubric. ©2014 MarylandOnline

from the Rubric To meet the General Standard 2, Learning Objectives: Learning objectives are measurable and are stated clearly. The learning objectives establish a foundation upon which the rest of the course is based. You must provide measurable, precise learning objectives at the course- and module-level objectives so your students will know what is expected of them. About The Content: Direct participants to look at General Standard 2 in their workbook. 5 Specific Review Standards support General Standard 2 and all 5 are Essential Standard. Introducing General Standard 2 General Standard 2 is the foundation of the QM rubric and the key standard for alignment; it is the only general standard in which all specific review standards are essential. It acknowledges the use and importance of course learning objectives and learning objectives at the module/unit level of a course. The learning objectives for the course set the standards and conditions for student learning and are critical to a successful outcome. The learning objectives are also the first element of "alignment," an instructional design concept that ensures that the core elements of a course reinforce each other and work together to promote student learning. ©2014 MarylandOnline

Standard 2 as a Sentence Quality is measurable objectives (2.1) and consistent module-level objectives (2.2), that are written from the student perspective (2.3), with instructions on how to meet them (2.4), that are appropriate for the level of the course (2.5). About the Content: In a succinct sentence, Sasha Thackaberry from Cuyahoga Community College, explains the Specific Review Standards that support General Standard 2. Submitted by Sasha Thackaberry, Cuyahoga Community College ©2014 MarylandOnline

Review Standard 2.1 2.1 The course learning objectives describe outcomes that are measurable. As a Reviewer: What do you do first? Where do you start? (Refer to the “How to Decide” handout) Content Notes: Use the “How to Decide” handout and go over each step with the participants. Be sure they read the specific standard and annotations in their workbooks. ©2014 MarylandOnline

Review Standard 2.2 2.2 The module/unit learning objectives describe outcomes that are measurable and consistent with the course-level objectives (alignment). As a Reviewer: What do you do first? Where do you start? (Refer to the “How to Decide” handout) Content Notes: Use the “How to Decide” handout and go over each step with the participants. Be sure they read the specific standard and annotations in their workbooks. ©2014 MarylandOnline

Standard 2.1 and 2.2 How to apply the Annotations
Questions to ask: Are there course level objectives? Are there module level objectives? Are they measurable? Are they precise? Do they provide criteria for assessment? Do they describe a behavior that can be observed or evaluated? Annotations specify: 1. What to look for: Measurable course and module learning objectives precisely describe what students are to gain from instruction and provide the criteria instructors need to accurately assess student accomplishment. Objectives describe student performance in specific, observable terms. 2. Examples 3. Special situations 4. Alignment This is a “self-check” to see how well participants were able to analyze Standard 2.1 and Ask participants to compare their answers to the slide…and ask if there are any questions. ©2014 MarylandOnline

Standards 2.1 & 2.2 – Your Job Write measurable course and module learning objectives that precisely describe what students will gain from instruction guide instructors to accurately assess student accomplishment. are consistent About The Content: This language (“precisely describe,” “guide instructors,” and “consistent”) is taken directly from the standards and their annotations. Be sure to point this out to participants since it’s critical that they look to the Rubric as the “authority” for making decisions. ©2014 MarylandOnline

Measurable Learning Objective
Component Example Completes this sentence: Upon completion of this course/module, students will be able to (DO SOMETHING). At the end of this course, you will be able to: Begins with an action verb: See: Clemson’s Bloom’s Taxonomy Action Verbs deliver Precisely describes behavior that can be observed or evaluated a carefully constructed persuasive speech to an audience **** Handout: Clemson’s Bloom’s Taxonomy of Action Verbs**** About The Content: This slide presents the structure of a measurable learning objective as well as an example to illustrate it. The Bloom’s Taxonomy Link is to a list of action verbs from Clemson University. The participants have the list of verbs from Clemson in their packets. ©2014 MarylandOnline

Activity 7: Measurable? Precise?
Realize the significance of ethical behavior in the business environment. Document the critical events leading to the beginning of World War I. Identify the seven most serious risks associated with cigarette smoking. Understand the continuing impact of World War II on the European Union. Demonstrate the effects of improper blood handling in a critical care environment. Describe how to create a financial statement using MS Excel. How To Present It: Go through each learning objective and ask participants if it is measurable and/or precise? And if they have suggestions for how to improve the learning objective? Answers: Realize the significance of ethical behavior in the business environment. (Not measurable; an instructor cannot observe “realize.”) Document the critical events leading to the beginning of World War I. (Measurable; precise) Identify the seven most serious risks associated with cigarette smoking. (Measurable; precise) Understand the continuing impact of World War II on the European Union. (Not measurable; an instructor cannot observe “understanding.”) Demonstrate the effects of improper blood handling in a critical care environment. (Measurable; not precise since it’s unlikely the instructor really wants the risk associated with asking students to “demonstrate” improper blood handling) Describe how to create a financial statement using MS Excel. (Measurable. Precise if the instructor wants a written or oral description. Not precise if the instructor wants the students to actually “create” and submit an Excel table) ©2014 MarylandOnline

Bloom’s Taxonomy Lorin Anderson, a former student of Bloom's, led an initiative in the 1990s to update the taxonomy to add relevance for 21st century students and teachers. Bloom's six major categories were changed from noun to verb forms, and some of the categories were renamed. For example, knowledge--the lowest level of the original taxonomy—was changed to remembering. Refer participants to the resource. Bloom’s Taxonomy (Fordhand, 2008) provides an excellent summary of the theory and applications. The site also provides an overview of the revisions to Bloom’s original work. Review the site and related links. Check your understanding of the concepts by taking the “New Bloom’s Taxonomy” interactive quiz. ( When writing course objectives, educators need to be aware of the level at which students are asked to demonstrate or perform mastery of course concepts. Objectives for an introductory course may be appropriately concentrated in the lower levels while objectives for a capstone course will usually be concentrated in the upper levels. Those who use the QM Rubric need to be aware of Bloom’s (or another taxonomy) since it’s a relatively straight-forward concept to apply. Knowing the “Bloom’s level” for each learning objective makes it easier to determine if there is alignment with the assessments. For example, if your learning objective uses the action verb, “define,” then you do not want to have an assessment that is above that level of learning, such as comparing and contrasting a topic. Conversely, if you use an creating level verb, like “develop,” then you do not want to assess the students with just a multiple choice test, which would be at the knowledge level. Cassinelli, C. (2008). Bloom's Taxonomy: Original and revised. In edTEch Vision, Updated Bloom’s Taxonomy. Retrieved September 21, 2012, from ©2014 MarylandOnline

differentiate, organize, attribute retrieve, recognize, recall
Action Verbs generate, plan, produce judge, check, critique differentiate, organize, attribute execute, implement interpret, exemplify, classify, summarize, infer, compare, and explain. retrieve, recognize, recall Cassinelli, C. (2008). Bloom's Taxonomy: Original and revised. In edTEch Vision, Updated Bloom’s Taxonomy. Retrieved September 21, 2012, from ©2014 MarylandOnline

Mnemonic Device: Bloom’s Taxonomy
R U At An Excellent College? Content: This is a simple mnemonic device for remembering the cognitive levels in Bloom’s New Taxonomy. Remind participants that the specific level is not as critical as a general sense of whether the objective is at a low, middle or high level on the taxonomy. NOTE: If you have time, you could ask participants to create their own mnemonic to help remember this important concept. ©2014 MarylandOnline

Bloom’s For Students University of Victoria Counseling Services About The Content: University of Victoria Counseling Services - an excellent overview of Bloom's Taxonomy written from the students’ perspective that provides a list of demonstrated skills and question cues for each level. The site suggests that students who know more about the types of questions that are likely to be asked can better prepare for exams. How To Present It: Ask participants to take a look at Bloom’s taxonomy and decide whether or not the SPCH 1113 course-level objectives are written at a fairly low level or a fairly high level. This will be useful information as we progress through the essential standards. ©2014 MarylandOnline

In search of … a “precise” verb
Verbs from Bloom’s Taxonomy Old and New Version A list of verbs that focus on performance (what the students will do) This slide presents a list of verbs that can be used with both the old and new version of Blooms taxonomy. It is from the University of Texas – Dallas. Spend a few minutes talking about the verbs with the participants. The participants have two handouts in their packet that contain a list of the verbs for old Bloom’s (Clemson’s Blooms Taxonomy of Action Verbs) and the revised Bloom’s taxonomy (New Bloom’s Cognitive Domain Verbs). ©2014 MarylandOnline

Activity 8: Learning Objectives Worksheet
For this activity: Find a partner Locate the Learning Objectives Worksheet in your packet Review the SPCH 1113 course site Complete the worksheet by applying Standards 2.1 through 2.5 to the SPCH 1113 course site ***** Handout: Learning Objectives Worksheet***** How To Present It: Participants should work with a partner to complete the worksheet. Take the time to relate the Standard 2 specific review standards to the SPCH 1113 course and complete all 5 columns which encompasses standards 2.1 – 2.5. At the end of this exercise, participants should be familiar with the content of these specific review standards. ©2014 MarylandOnline

Objectives, Assessments, Materials, Engagement, and Tools/Media
Practice Alignment General Standards 2 – 6 Objectives, Assessments, Materials, Engagement, and Tools/Media Marker: The following slides present the alignment concept and practice applying it to Standards 3.1, 4.1, 5.1 and 6.1. ©2014 MarylandOnline

Key Sections That Must Align
About The Content: Again, highlights the General Review Standards that must work together. About The Content: The QM Rubric takes a holistic view of the course and that's why it's so important that the learning objectives ALIGN with the assessments, resources and materials, student interaction, and technology standards. Alignment refers to the direct link between the learning objectives (Standard 2) and the assessments and measurements (Standard 3), resources and materials (Standard 4), learner engagement (Standard 5), and course technology (Standard 6). Under the principle of alignment, these later four aspects of the course are driven by and support the learning objectives. As you begin to apply the QM Rubric, pay special attention to the six specific review standards (2.1, 2.2, 3.1, 4.1, 5.1 and 6.1) in the rubric that include alignment matters (and the red word “Alignment” included in each standard box). The diagram on this slide illustrates the dynamic relationship that supports alignment between these standards. ©2014 MarylandOnline

Think About “Alignment”
Objectives From Intro Psychology: Course : Summarize the relationships that exist between biology and human behavior Module Identify and define 7 major biological areas. Recognize examples of how each area affect behavior. Assignment: Read Chapter 4 (assume it deals with relationships between biology and human behavior) in text; review study guide, objective 2, page 4. Assignment: View Chapter 4 PPT; listen to Chapter 4 podcast Graded Assignment: Prepare table that lists the 7 major biological areas; ask students to define/describe each area and list 3 specific ways this area affects human behavior. Graded Discussion Forum Question: Psychology is the study of behavior and mental processes. Why, then, are psychologists interested in biological matters such as the nervous system, the endocrine system and heredity? Read and respond to classmates’ postings. Exam: 50-item multiple choice exam taken in campus Testing Center. About The Content: This slide lists a course-level and two module-level learning objectives from an Intro to Psych course and five possible examples of activities that might support this objective. Discuss whether/not and HOW each of the examples flows from or supports the course- and module-level objectives. Questions to consider: If information is delivered in the first assignment, why is the second assignment important? (it serves to reinforce the first assignment and provides visual and auditory alternative delivery) How does the graded assignment support the course- or module-level learning objectives? (it serves as “scaffolding to ensure that students understand the basic content) How does the graded discussion forum support the course- or module-level learning objectives? (this is the major evidence that students have the content and understanding to meet the learning objective) Is the mid-term necessary to verify that the course-level learning objective is met? (No, the mid-term is an objective test that doesn’t align with the learning objective since there’s no way to measure “summarize” on such an exam. It could be eliminated since feedback is provided in the graded assignment and the graded discussion.) What would happen if we eliminated the graded assignment and the graded discussion forum? (the course-level learning objective is no longer measured since the mid-term does not provide an opportunity for students to demonstrate that they can “summarize” the relationships. What else might change if the instructor decided that perhaps the course-level learning objective should be “identify” instead of “summarize” since this is, after all an introductory course? (You could eliminate the graded discussion forum.) ©2014 MarylandOnline

from the Rubric To meet the General Standard 3, Assessment and Measurement: Assessment strategies are designed to evaluate student progress by reference to stated learning objectives; to measure the effectiveness of student learning; and to be integral to the learning process. Assessment is implemented in a manner that not only allows the instructor a broad perspective on the students’ mastery of the content, but also allows students to measure their own learning throughout the course. Your assessments must align with your stated learning objectives and guide students to measure their own learning progress. About The Content: Direct participants to look at General Standard 3 in their workbook. 5 Specific Review Standards support the general standard, 3 Essential Standards and 2 Very Important Standards Introducing General Standard 3 Quality Matters stresses the importance of "measurable" objectives so that these learning objectives can guide instructors to develop assessments that accurately assess student accomplishment. For example, a learning objective that states students will "understand modern art" cannot be measured as there is no way to adequately define and then assess a student's understanding. However, a learning objective that states students will "describe the five characteristics of paintings classed as modern art" is something that can be measured through a variety of assessment techniques. This is a simplistic example, but it illustrates what QM is looking for. The terms "assessment" and "measurement" used in General Standard 3 mean more than just quizzes and exams. "Assessment" refers to any student activity for which the instructor provides feedback that guides the student to improved learning. These assessment activities might range from a brief, automatically-graded self-check to a fully annotated paper to the final essay exam…but each is important because it helps the instructor form an accurate, current picture of student learning and provides crucial, regular feedback to help students assess their own progress. ©2014 MarylandOnline

Review Standard 3.1 As a Reviewer:
3.1 The types of assessments selected measure the stated learning objectives and are consistent with course activities and resources. As a Reviewer: What do you do first? Where do you start? (Refer to the “How to Decide” handout) Content Notes: Use the “How to Decide” handout and go over each step with the participants. Be sure they read the specific standard and annotations in their workbooks. ©2014 MarylandOnline

Standard 3.1: How to use the Annotations
Questions to ask: Do the assessments flow from the course and module/unit objectives? Are they consistent with these objectives? Do you (and will the students) recognize alignment? Annotations specify: Key Ideas (What to look for): Consistent with course and module level objectives. Alignment: Easy to see relationship between learning objectives and assessments. Clear that learning objectives guide students to mastery Examples Alignment Lack of Alignment 3. Special situations: Course objectives are mandated and do not align with assessments This is a “self-check” to see how well participants were able to analyze Standard Ask participants to compare their answers to the slide…and ask if there are any questions. ©2014 MarylandOnline

Activity 9: Apply Standard 3.1
Evidence from the Course Met or Not Met Aligned? The types of assessments selected measure the stated learning objectives and are consistent with course activities and resources. Handout ***Alignment Worksheet **** How To Present It: Login to the SPCH 1113 course and apply Standard 3.1 to the assessments (be sure to read the annotation carefully). Locate the course level objectives and make a list of the types of assessments used in the course. Does the SPCH 1113 course meet Standard 3.1? What evidence from the course supports the participant’s decision. Do the assessments support the course- and module-level objectives, in alignment? The participant should record his or her answer on the worksheet. The participants will continue to work on the worksheet after each of the alignment standards are discussed. After the worksheet is completed the participants will share with the group. ©2014 MarylandOnline

from the Rubric To meet the General Standard 4, Instructional Materials: Instructional materials are sufficiently comprehensive to achieve stated course objectives and learning outcomes. The instructional materials form the core of the course, and these standards respect the instructor’s prerogative in selecting them. The focus of this standard is on supporting the course objectives and competencies, rather than on qualitative judgments about the materials. You must provide rich, robust and appropriate resources and materials that support your stated learning objectives. About The Content: Direct participants to look at General Standard 4 in their workbook. 6 Specific Review Standards support the general standard, 2 Essential Standards, 2 Very Important Standards, and 2 Important Standards Introducing General Standard 4 General Standard 4 prompts you to take a close look at the course’s instructional materials. In the early days of online teaching (way back in the '90s), many faculty thought that building an online course simply involved selecting a good textbook and/or writing out their lectures and posting them within the online class. As our understanding of online learning has evolved, it's become clear that this approach in itself is not sufficient to engage our online students and allow them to meet the learning objectives. Instructors must choose appropriate materials from a wide variety of options ranging from adopting a standard textbook to providing a list of resources from the campus’ online library to selecting learning objects from MERLOT (see the QM Resources List) to having students create their own materials (most likely in a graduate course. Faculty may be confused or over whelmed by the many choices available to them…hence, the importance of using the stated learning objectives as the basis for selecting appropriate instructional resources. General Standard 4 also considers the depth and breadth of your materials, prompts you to appropriately cite them and ensure that students can distinguish required materials from those that are optional. ©2014 MarylandOnline

Review Standard 4.1 As a Reviewer:
4.1 The instructional materials contribute to the achievement of the stated course and module/unit learning objectives. As a Reviewer: What do you do first? Where do you start? (Refer to the “How to Decide” handout) Content Notes: Use the “How to Decide” handout and go over each step with the participants. Be sure they read the specific standard and annotations in their workbooks. ©2014 MarylandOnline

Annotations specify: Key Ideas (What to look for): The course materials and resources enable students to achieve the stated learning objectives. Consult with the team SME (subject matter expert) and use common sense. Focus only on the alignment of the instructional materials with the learning objectives rather than attempt to evaluate the content. Special situations: Use module level objectives if course-level objectives are not measurable 3. Alignment: Part of alignment concept Questions to ask: 1. What instructional materials are used in this course? 2. Do these materials enable students to meet learning objectives? 3. Do I need to consult with the SME? This is a “self-check” to see how well participants were able to analyze Standard Ask participants to compare their answers to the slide…and ask if there are any questions. ©2014 MarylandOnline

Activity 10: Apply Standard 4.1
Evidence from the Course Met or Not Met Aligned? The instructional materials contribute to the achievement of the stated course and module/unit learning objectives. ****HANDOUT: Alignment Worksheet**** How To Present It: Login to the SPCH 1113 course and apply Standard 4.1 to the course materials (be sure to read the annotation carefully). Locate the course level objectives and make a list of the types of materials and/or content used in the course. Does the SPCH 1113 course meet Standard 4.1? What evidence from the course supports the participant’s decision. Do the course materials support the course- and module-level objectives, in alignment? The participant should record his or her answer on the worksheet. The participants will continue to work on the worksheet after each of the alignment standards are discussed. After the worksheet is completed the participants will share with the group. ©2014 MarylandOnline

from the Rubric To meet the General Standard 5, Learner Interaction & Engagement: Forms of interaction incorporated in the course motivate students and promote learning. Engaging students to become active learners contributes to the learning process and to student persistence. You must provide meaningful and productive activities that prompt your students to actively practice their learning. About The Content: Direct participants to look at General Standard 5 in their Workbook. 4 Specific Review Standards support the general standard, 3 Essential Standards and 1 Very Important Standard Introducing General Standard 5 In more traditional face-to-face classrooms, instructors often planned courses around four main elements: lectures, readings, exams and a written paper or project. Students interacted with the instructor and each other by asking questions or participating in discussions within the time allotted in the classroom. Students with more initiative could visit their professors during office hours or perhaps arrange to meet fellow students in a study group. While this approach might work in a f2f classroom on a college campus, it cannot succeed in the online environment. Students need consistent guidance and support from their instructors and meaningful opportunities to engage with the content and their fellow students. Engage the learner with the course content by providing your students with the opportunity to "practice" their learning as preparation for meeting the stated learning objectives. This concept of "practice learning" is particularly important in an online course. You cannot expect your students to simply read the text and then be successful on your assessments. It's your responsibility as the instructor to design opportunities for the students to practice their learning at the appropriate levels to ensure that they can meet your learning objectives.. Interaction between the instructor and students, among students, and between students and course materials is employed to motivate students and foster intellectual commitment and personal development. And it’s critical that instructors also be “present” and engaged in the online classroom…and that the instructor clearly tells his students how and when he will communicate with them, participate in discussions, and provide feedback. Assessments include all activities, assignments, exams, discussions, etc., for which feedback is provided and/or grades assigned. ©2014 MarylandOnline

Review Standard 5.1 5.1 The learning activities promote the achievement of the stated learning objectives. As a Reviewer: What do you do first? Where do you start? (Refer to the “How to Decide” handout) Content Notes: Use the “How to Decide” handout and go over each step with the participants. Be sure they read the specific standard and annotations in their workbooks. ©2014 MarylandOnline

Standard 5.1: Analyze the Standard
Annotations specify: Key Ideas (What to look for): Engage students in activities that directly contribute to the achievement of learning objectives. Examples: readings, student presentations, science labs, class discussions, case studies, role playing, simulations, practice quizzes, tests, etc. 3. Special situations: blended courses 4. Alignment: Part of Alignment Concept Questions to ask: 1. What learning activities are used in this course? 2. Do these activities enable students to meet learning objectives? 3. Are the learning activities engaging and varied? This is a “self-check” to see how well participants were able to analyze Standard Ask participants to compare their answers to the slide…and ask if there are any questions. ©2014 MarylandOnline

Evidence from the Course
Activity 11: Standard 5.1 Standard 5.1 Evidence from the Course Met or Not Met Aligned? The learning activities promote the achievement of the stated learning objectives. ****HANDOUT: Alignment Worksheet**** How To Present It: Login to the SPCH 1113 course and apply Standard 5.1 to the course activities (be sure to read the annotation carefully). Locate the course level objectives and make a list of the kinds of activities used in the course. Does the SPCH 1113 course meet Standard 5.1? What evidence from the course supports the participant’s decision. Do the course activities support the course- and module-level objectives, in alignment? The participant should record his or her answer on the worksheet. The participants will continue to work on the worksheet after each of the alignment standards are discussed. After the worksheet is completed the participants will share with the group. ©2014 MarylandOnline

from the Rubric To meet the General Standard 6, Course Tools and Technology: Course navigation and technology support student engagement and ensure access to course components. The technology enabling the various course components facilitates the student’s learning experience and is easy to use, rather than impeding the student’s progress. You must select the tools and media that best support your learning objectives and provide opportunities for your students to actively practice their learning. About The Content: Direct participants to look at General Standard 6 in their workbook. 5 Specific Review Standards support the general standard, 3 Essential Standards, 1 Very Important Standard, and 1 Important Standard Introducing General Standard 6: General Standard 6: Course Technology deals with the use of instructional media and the endless number of technology tools that are well-suited to online delivery and may be used in online and blended courses. It seems like every week there's a new technology on the market, particularly in the social media area, and it is a challenge to stay abreast of these technological advancements and how they can enhance student learning. The specific function of technology in an online course is to support your learning objectives. In addition, General Standard 6: Course Technology focuses on navigation and the ease of finding elements within the course. The use of tools and media is not required for online or blended courses to meet Quality Matters standards. Many exemplary courses consist of text-only study guides or web pages. Instructors and course designers may not have the equipment, skills, or institution support to develop advanced course components. Online students may still use dial-up internet service; these students will not have the connectivity to receive media requiring significant bandwidth. General Standard 6: Course Technology seeks to assure that those tools that are used have a clear purpose, support the learning objectives and are accessible to students who enroll. ©2014 MarylandOnline

Review Standard 6.1 6.1 The tools and media support the course learning objectives. As a Reviewer: What do you do first? Where do you start? (Refer to the “How to Decide” handout) Content Notes: Use the “How to Decide” handout and go over each step with the participants. Be sure they read the specific standard and annotations in their workbooks. ©2014 MarylandOnline

Annotations specify: Key Ideas (What to look for): The tools and media align with objectives of the course by supporting assessments, instructional materials, activities. Examples: tools and media; not required. Special situations: use module objectives if course-level not measurable Alignment: Part of Alignment Concept Questions to ask: 1. What tools and media are used in this course (if any)? 2. Do the tools and media selected enable students to meet objectives? 3. Is the purpose of the tools and media clear? This is a “self-check” to see how well participants were able to analyze Standard Ask participants to compare their answers to the slide…and ask if there are any questions. ©2014 MarylandOnline

Activity 12: Standard 6.1 Course Tools Media Technology Used
Evidence From the Course Met or Not Met Aligned? Course Tools Media ****HANDOUT: Alignment Worksheet**** How To Present It: Login to the SPCH 1113 course and apply Standard 6.1 to the course tools and media (be sure to read the annotation carefully). Locate the course level objectives and make a list of the types of tools and media used in the course. Does the SPCH 1113 course meet Standard 6.1? What evidence from the course supports the participant’s decision. Do the course tools and media support the course- and module-level objectives, in alignment? The participant should record his or her answer on the worksheet. The participants will continue to work on the worksheet after each of the alignment standards are discussed. After the worksheet is completed the participants will share with the group. ©2014 MarylandOnline

Activity 13: Alignment Worksheet
Complete the Alignment Worksheet if you haven’t already done so. Share and compare your alignment worksheets and any “epiphanies” you had with the group. By this point, participants should have completed the Alignment Worksheet. Ask them to make sure their Alignment Worksheet is complete. Once everyone is finished have them share their worksheet with the group. ©2014 MarylandOnline

Write Helpful Recommendations
General Standards 7 – 8 Learner Support and Accessibility Marker: The following slides give specific directions for writing helpful recommendations and introduce General Standards 7 and 8. ©2014 MarylandOnline

No Choice! You MUST write a Helpful Recommendation if you decide that the course does not meet the specific standard. About The Content: No choice here! If the reviewer decides that the standard is not met, the reviewer MUST explain why the decision was reached and offer specific suggestions for how to remediate it. Encourage reviewers to write a helpful recommendation even if the standard is met. This may be the only time anyone but the instructor and her students looks at this course in-depth…feedback and suggestions for improvement are very important to the instructor. ©2014 MarylandOnline

Look at a Helpful Recommendation
About The Content: The success of a QM review depends on the rich, robust feedback that is compiled for the instructor. This graphic emphasizes the components of a helpful recommendation. Direct participants to review the standard, the annotation and course evidence (their decisions should not be based on personal preference). ©2014 MarylandOnline

Write a Useful Recommendation
Constructive Try to offer solutions, not just identify problems. Specific Include a specific example of what is being recommended. Measurable How will you or the ID/instructor know when the recommendation has been implemented? Sensitive Avoid negative language. Keep recommendations and comments on a positive note. Balanced Point out strengths as well as weaknesses. About The Content: The review team’s recommendations are key to the success of a review. A decision of NOT MET on a standard is not helpful without suggestions for improvement. If a course is rated "does not yet meet expectations," the instructor will want to know how to improve the course so they will receive the "meets expectation" rating. Comment on the strengths of the course as well as the weaknesses. Faculty members typically work very hard on their online courses and are quite proud of them. You want to recognize areas that have been well done (and that you plan to consider for your own course) as well as the areas that need improvement. Courses can "meet expectations" but still have suggestions for improvement. ©2014 MarylandOnline

Can Someone Make Some Brownies?
Mnemonic Device Can Someone Make Some Brownies? This mnemonic device helps participants remember the characteristics of a helpful recommendation: Constructive, Specific, Measurable, Sensitive and Balanced. ©2014 MarylandOnline

Analyze A Recommendation
Constructive Specific Measurable Sensitive Balanced The Start Here button was a great idea, but when I read the information it contained, I still couldn’t tell exactly how to begin the course. It might be very helpful to include a prominent link and directions about what students should do next to actually begin the first lesson of the course. About The Content: Ask participants to look at the sample recommendation on the slide. Identify which parts of the recommendation exemplify each of the five characteristics: Constructive: It might be very helpful to include a prominent link and directions about what students should do next to actually begin the first lesson of the course. Specific: it seemed to contain mostly course information and I still couldn’t tell exactly how to begin the course once I finished reading it. Measurable: … to include a prominent link and directions about what students should do next to actually begin the first lesson of the course Sensitive: It might be very helpful … Balanced: The Start Here button was a great idea… ©2014 MarylandOnline

Activity 14: Improve Recommendations
For this activity you will work in groups. Each group will be assigned one of the recommendations listed below and will rewrite it using the “components” of an effective recommendation. Each group will share their recommendation. Assignment instructions weren’t clear. You didn't tell the students how to find the additional resources. The text on the page was too hard to read. The learning objectives aren’t measurable. Your assessments are weak. The facilitator should break the participants in to 5 groups and assign each group one of the recommendations. Give them 5 minutes to rewrite the recommendation and then ask each group to share the new recommendation. Look for improved recommendations that incorporate the components of an effective recommendation (standard/annotations, evidence from the course, and the 5 characteristics: constructive, specific, measurable, sensitive and balanced). ©2014 MarylandOnline

from the Rubric To meet the General Standard 7, Learner Support: The course facilitates student access to institutional support services essential to student success. In the learner support standard, four different kinds of support services are addressed: technical support, accessibility support, academic services support, and student services support. You must direct your online students to the support services they need to be successful and would be available to them if they attended campus-based courses. About The Content: Direct participants to look at General Standard 7 in their workbook 4 Specific Review Standards support the general standard, 2 Essential Standards, 1 Very Important Standard, and 1 Important Standard Introducing General Standard 7 General Standard 7: Learner Support deals with the importance of providing our online learners with comparable services to those found on campus. In the very early days of online courses, institutions invested in a course management system and technical support (and almost always online registration) but often had difficulty re-tooling other support services. It took much longer for most institutions to recognize that their online students also needed academic support (testing, tutoring, library) and student support services (advising and counseling, in particular). And, of course, for those online students who need special services, it’s critical to offer disabilities support services. Since providing these services is an institutional responsibility, rather than an individual faculty member responsibility, General Standard 7 is designed to ensure that instructors are not penalized if their institutions don’t offer a full array of online services. Note that each of the four specific review standards require links to institutional services, making the assumption that offering these services is not the responsibility of a single instructor within a specific course. Clearly, however, it is the instructor’s responsibility to direct his students to those services which are available and critical to a particular course. For example, in an online History class, you would expect to find significant use made of online library services and resources and, perhaps, links to the Online Writing Lab for help with preparing papers. ©2014 MarylandOnline

Review Standard 7.2 7.2 Course instructions articulate or link to the institution’s accessibility policies and services. As a Reviewer: What do you do first? Where do you start? Refer to the “How to Decide” handout Content Notes: Use the “How to Decide” handout and go over each step with the participants. Be sure they read the specific standard and annotations in their workbooks. ©2014 MarylandOnline

Annotations specify: Key Ideas (What to look for): accessibility policies, accommodations; types of and how to access services Examples: link to institution’s policy; access statement Special situations: no institutional policy Questions to ask: Does the institution have a stated accessibility or accommodation policy? Where are they? Can a student find them? What if there is no stated policy? This is a “self-check” to see how well participants were able to analyze Standard Ask participants to compare their answers to the slide…and ask if there are any questions. ©2014 MarylandOnline

Activity 15: Standard 7.2 7.2 Standard
Link to the Institution’s formal policy included (evidence from the course)? Statement to the students included? Your Decision Recommendations 7.2 About The Content: Activity: Review the SPCH 1113 course site and made a decision on specific standard 7.2. Where did you locate the information (evidence from the course)? Was a statement provided about the service and how to access the service? What is your decision, Met or Not Met? Write a recommendation to the faculty developer. NOTE: If your institution does not have an Accessibility Policy you can request a template developed by Quality Matters to guide your development. To request the template use the following link: Fill in the requested information and QM will send you the template. ©2014 MarylandOnline

Activity 16: Write a Recommendation
Use the evidence you discovered in the SPCH 1113 course to write a recommendation for improving Standard 7.2 Each team will share their recommendation and point out each component of an effective recommendation. ****HANDOUT: Write A Helpful Recommendation**** Participants will use this handout to write a recommendation for Standard 7.2. When participants have finished working ask them to share the recommendation they wrote. Ask them to point out the components of a helpful recommendation (standard/annotation, evidence from the course, characteristics of a helpful recommendation) as they read the recommendation they wrote. ©2014 MarylandOnline

from the Rubric To meet the General Standard 8, Accessibility: The course demonstrates a commitment to accessibility for all students. The accessibility standard incorporates the principles of Universal Design for Learning (UDL) and is consistent with Web Content Accessibility Guidelines (WCAG). You must design your course so students with special needs can access course materials and be successful in your course. About The Content: Direct participants to look at General Standard 8 in the workbook. 4 Specific Review Standards support the general standard, 1 Essential Standard and 3 Very Important Standards Introducing General Standard 8 General Standard 8: Accessibility deals with the importance of designing online courses that are accessible for all students, including those with special needs such as the visual or aural disabilities. Institutional support for accessibility varies widely among institutions. Some institutions are committed to accessibility issues and provide both training and design support for their faculty. Other institutions are more minimal in their approach to accessibility and offer little guidance or support for faculty. The Quality Matters rubric acknowledges this disparity by including only one essential standard (8.1) that requires faculty to assess accessibility in their own courses and provide directions for how their students should access it. ©2014 MarylandOnline

Sites to Explore Universal Design for Learning (UDL)
Site provides “a framework for: designing educational environments that enable all learners to gain knowledge, skills, and enthusiasm for learning reducing barriers to the curriculum providing rich supports for learning.” Web Content Accessibility Guidelines (WCAG) Site provides documents that “explain how to make Web content more accessible to people with disabilities.” About The Content: Standard 8 references accessibility issues and cites these two websites as references for accessibility guidelines (“The accessibility standard incorporates the principles of Universal Design for Learning (UDL) and is consistent with Web Content Accessibility Guidelines (WCAG)”. About How To Present It: Break participants into small groups and ask them to explore both sites. You can use the following prompts to spur discussion: How is the website organized? Is it easy to find information? Is the website written in technical terms or for the non-technical user? What was the most helpful idea or suggestion your group found? What other questions do you have about using this website or applying Standard 8? ©2014 MarylandOnline

Review Standard 8.1 8.1 The course employs accessible technologies and provides guidance on how to obtain accommodation. As a Reviewer: What do you do first? Where do you start? (Refer to the “How to Decide” handout) Content Notes: Use the “How to Decide” handout and go over each step with the participants. Be sure they read the specific standard and annotations in their workbooks. ©2014 MarylandOnline

Annotations specify: Key Ideas (What to look for): LMS accessibility statement, documentation on any content, tools, and software, information on how to obtain accommodation Examples: link to the LMS accessibility statement, link to other tools, content, or software accessibility, statement from instructor Questions to ask: 1. Does the course contain a link to the LMS statement? 2. Are the other tools, content, or software accessible? 3. Do the students know how to obtain accommodation if needed? This is a “self-check” to see how well participants were able to analyze Standard Ask participants to compare their answers to the slide…and ask if there are any questions. ©2014 MarylandOnline

Standard 8.1 - What to Look For
Link to the Learning Management System Accessibility Statement Students will be using Microsoft PowerPoint to create a presentation: Microsoft PowerPoint® will be used to create your e-portfolio. The Microsoft Corporation has developed this software product with accessibility in mind. Product accessibility information is available on their website: Students are asked to watch videos created by their classmates that are not closed captioned. If you are unable to view the videos created by your classmates please contact the Disability Services Center. The Disability Services personnel will work with you to ensure you have access to this content. The center can be reached by calling ( ) or In addition, you can review their website at: Content Notes: This slide provides information and examples of what the participants should be looking for in a course when making a decision on whether or not Standard 8.1 is met. ©2014 MarylandOnline

Evidence from the Course
Activity 17: Standard 8.1 Standard Evidence from the Course Decision Recommendations 8.1 About The Content: Standard 8.1 deals with an important element in ensuring accessibility for your online students. Review the SPCH 1113 course and made a decision for specific standard 8.1. Record the “evidence” from the course to support your decision and write a recommendation to the faculty developer. ©2014 MarylandOnline

Activity 18: Write a Recommendation
Use the evidence you discovered in the SPCH 1113 course to write a recommendation for improving Standard 8.1 Each team will share their recommendation and point out each component of an effective recommendation. ****HANDOUT: Write A Helpful Recommendation**** Participants will use this handout to write a recommendation for Standard 8.1. When participants have finished working ask them to share the recommendation they wrote. Ask them to point out the components of a helpful recommendation (standard/annotation, evidence from the course, characteristics of a helpful recommendation) as they read the recommendation they wrote. ©2014 MarylandOnline

Effects of Helpful Recommendations
SPCH 1113 course Standards 7.2 and 8.1 Marker: The following slide directs participants to consider how helpful recommendations were used to improve the SPCH 1113 course. ©2014 MarylandOnline

Activity 19: Course Improvements
Read the decision and the recommendations from the review team for standards 7.2 and 8.1 (located in your packet) Did you make the same decision? What additional information did you include in your recommendations? Log in to the revised version of the SPCH 1113 course (login information located in your packet) Did the recommendations from the review team help the instructor improve the course? What impact will the revisions have on the student’s experience? Do you feel the revised course meets these standards? About the Content: Participants viewed these standards in the initial course for the “write a recommendation” activities. During this activity they will have an opportunity to review the feedback from the review team and review the course after the faculty developer made revisions. Use the questions on the slide to prompt a discussion of the impact of helpful recommendations. ©2014 MarylandOnline

QM = Process and Rubric Process Rubric OFFICIAL INFORMAL
Outcome: Earn QM recognition Must follow official QM guidelines and procedures Outcome: Improve courses, meet institutional goals, demonstrate commitment to quality Tool to assess online courses during formal QM review Outcome: Improve courses, meet institutional goals INFORMAL Institutions determine use and procedures Guide to develop new online courses AND review and update online courses About The Content: This slide emphasizes that QM is both a Process and a Rubric (tool) for course reviews. Participants (and institutions) should understand that there are significant benefits of using QM informally, but that official QM recognition comes only if the “official” process is followed. ©2014 MarylandOnline

Peer Course Review Process
About The Content: QM Circle Process: This visual highlights the QM review process. Key points: QM is designed for continuous improvement; the goal is that ALL courses will eventually meet QM expectations. How To Present It: Ask participants to look at the diagram and make inferences about it. Typically, they will respond with “it’s a circular process;” “it’s continuous:” etc.). Begin with the COURSE and then continue through PEER COURSE REVIEW, FEEDBACK, COURSE REVISION and MEETS QUALITY EXPECTATIONS. Speak briefly about how each item contributes to the process. Participants will learn more about each item throughout the workshop. This diagram illustrates the focus on continuous improvement and summarizes the Quality Matters course review process: COURSE: Beginning at the top of the cycle, institutions decide to examine an online or hybrid course as part of a peer review. The institution may submit its course to QM for a formal review or, if the institution is a QM subscriber, it may conduct the formal course review in-house (both a formal QM review and a formal in-house process lead to QM recognition). The institution may also decide to use the rubric informally and do a review using its own process (this informal process does not lead to QM recognition). PEER COURSE REVIEW: The course is then reviewed by a team of three peer reviewers using the QM Rubric. The QM rubric is based in national standards of best practice, the research literature, and instructional design principles. Peer reviewers must have online teaching experience and complete the Peer Reviewer Training to be eligible to serve on a formal QM review. The peer review team consists of at least one member from an institution other than the course’s home institution. The team also consists of one member from a discipline that matches that of the course. This combination of reviewers ensures a diverse set of perspectives. FEEDBACK: Following the course review, the review team’s feedback is provided to the faculty member or team that developed the course. The feedback consists of two components: Scoring - indicates which QM standards were and were not met by the course. Feedback - a rich set of comments from the reviewers indicating the strengths of the course, areas for improvement and specific recommendations and suggestions for improving the course. COURSE REVISION: Upon initial review, the course may or may not have met Quality Expectations. In either case, the QM review provides support for course revisions and improvement. If a course did not initially meet Quality Expectations, the team chair will re-review the course after revisions. MEETS QUALITY EXPECTATIONS: QM expects that all courses will work toward and achieve quality expectations. The QM review process is not meant to be a test in which a course passes or fails. The overall goal is to provide a system for the improvement of course quality, rather than the simple assignment of a grade or quality level to the initial course. ©2014 MarylandOnline

About the Course QM is designed to review “mature” courses (taught at least two semesters) QM logo indicates year course met expectations Triggers for subsequent reviews: Faculty request More than 5 years since original review New textbook or instructor Professional or accreditation review pending About The Content: The QM rubric and tools can, of course, be used to review new online courses (or even courses in the development state). However, institutions make a significant investment in time, resources, and funds when they choose to review a course. The QM team suggests reviewing “mature” courses to maximize this investment. Other triggers could also indicate a re-review; it’s really up to the institution to decide how often each course will be reviewed or re-reviewed. ©2014 MarylandOnline

The Peer Review Team AND 3 faculty peer reviewers: Faculty developer:
must be experienced online instructors must attend QM training one MUST be external to the course’s originating institution there must be a subject matter expert (SME) on the team. Note: The SME could also be the external reviewer. AND Faculty developer: access to rubric prior to review involved in pre-review discussions consulted during review About The Content: Be sure to point out that there a 3 faculty peer reviewers who actually complete the Course Review Management System (the web-based review form). One of these reviewers must be a Subject Matter Expert (SME) and at least one must be external to the course’s home institution (although a single person could play multiple roles, could be both the SME and the master reviewer). Emphasize that the Faculty Course Developer (the Instructor) should be consulted by the team as questions or issues arise. ©2014 MarylandOnline

In an Official QM Review
Team of 3 reviewers initially score individually One score per standard based on team majority Pre-assigned point value Met/Not Met decision; All/None points Consensus is NOT required To Meet Expectations…Two Thresholds “Met” on all 21 of the 3-point “essential” standards. A minimum of 81 out of 95 points (81/95 = 85%) About The Content: Emphasize again that each reviewer individually decides Met or Not Met on a particular standard and enters this decision in the course review management system (web-based Rubric tool). The web form then automatically totals the points (majority rules) and determines whether/not the course meets the standard. Tell participants that the peer reviewers don’t have to reach consensus. Since the instructor will see all decisions and all feedback, it will be very helpful to know that there was a “split decision” on a particular standard and to read the specific recommendations for how to improve. ©2014 MarylandOnline

Points are NOT assigned on a sliding scale…
Standards If the standard is met … is not met … Essential 3 points 0 points Very Important 2 points Important 1 point About The Content: This information is important because new Peer Reviewers frequently think they are assigning points (and sometimes think they can assign 2 points for a 3-point standard if the standard is “sort of” met). In practice they decide whether a Specific Review standard is met at the 85% standard, indicate MET or NOT MET in the course review management system (the web-form), and then the web form actually assigns the points. ©2014 MarylandOnline

The 85% Rule Two 85% Uses: A minimum of 81 out of 95 points (81/95 = 85%) Reviewers use their own gauge of 85% as to if a standard is met or not. Standards do not have to be 100% to be marked “Met.” About The Content: There are TWO thresholds for meeting expectations and a course must meet both thresholds to meet expectations. These thresholds are highlighted on slide 31: “Yes” on all 21 of the 3-point “essential” standards and a minimum of 81 out of 95 points. Mathematically, this works out to about an 85% standard (81/95 = 85%). New QM reviewers often have many questions about the 85% standard and often find it confusing. There are actually two 85% issues to consider during a QM review: The first is that a course must earn at least 81 out of 95 points (or about 85%). Each of the 41 specific review standards receives a "yes" or "no" vote regardless of point value. The points are added up and must be at the 85% or greater percentage (AND meet all 21 essential standards) to be QM recognized. The second is that when you are conducting your independent review of the course and making your determination if the specific standard is met, use the 85% rule for yourself in making that judgment. The standard does not have to be 100% to be a "yes". The "85% rule" is a guide for you as a reviewer to gauge whether you will choose "yes" or "no" for that particular standard. Here's an example: A Chemistry course might have 15 course-level learning objectives. Of these, two are not measurable (the other 13 are measurable). Most reviewers would decide YES, this meets Standard 2.1. They would then write strong recommendations for how to improve the two weak objectives. QM is admittedly a subjective process and making decisions about whether a standard is or is not met can be difficult. QM provides extensive annotations and training for its reviewers, but there are likely to be many instances of differing opinions. We know that everyone wants a "right/wrong" answer, but frequently, that's just not possible. Online courses are very complex and likely to be more shades of gray than absolute. And that's the reason QM "hires" three experienced online faculty members to serve on the peer review team. QM relies on the collective experience, knowledge and common sense of the team to arrive at the best possible feedback for the course instructor. ©2014 MarylandOnline

About the Review On average, a course review takes 7-10 hours
Factors affecting review time include Reviewer familiarity with the discipline Reviewer familiarity with the LMS Reviewer familiarity with the QM review process Organization of the course About The Content: Actual review time varies widely. In general the first review takes longer than subsequent reviews (in other words, reviewers become more efficient as they gain experience during reviews). Master Reviewers are advised to include new reviewers on their teams. ©2014 MarylandOnline

Timeline for a QM Review
Pre-Review: (1-2 weeks) Instructor Worksheet Conference Call Set Team Calendar Faculty Developer Active Review: (3 weeks) Reviewers actively review course Post-Review: (1 week) Have post-review discussion, if applicable Revisions: (14 weeks) Course improvements made to meet standards About The Content: QM offers this timeline to help review teams manage their reviews. Some teams complete their work in less time. QM is, of course, flexible if there are extenuating circumstances (illness, change in work schedules, holidays, etc.). ©2014 MarylandOnline

Formal Review Outcome If meets expectations:
Recognized by Quality Matters Notifications distributed Recognized on QM website If does not yet meet expectations: Instructor (and/or ID) make changes Team Chair/Master Reviewer approve revisions Course meets expectations and is recognized About The Content: Emphasize here that ALL courses will meet expectations, either at the time of the initial review or after a period of revision. A typical question at this point is: After the course has been revised, does the entire team need to see it again? The answer is that for most courses, only the Team Chair/Master Reviewer looks at the revised course and makes decisions about whether/not it meets expectations. The Faculty Developer completes a Course Amendment form that explains the changes that were made. The MR checks the course for any changes made and can determine if the course now meets the standard or may write more recommendations and note that the standard is not yet met. ©2014 MarylandOnline

Self-Review Tool Used to do a self-review of online or blended courses
Available through MyQM >> HE tab >> My Tools >> Course Review System Course Review System Start a self-review Lists each specific standard with links to the annotations Allows you to make a decision and add a recommendation Can save and return Once all the standards are completed and saved you can “View” or “ ” the results Can do as many self-reviews as you want (one for every course you teach) No one else has access to the self-review Facilitator: Take a minute to log in and show the participants how to access this tool and how it works. ©2014 MarylandOnline

Serving as a QM Peer Reviewer
Marker: The following slides give specific directions on requirements for becoming a QM Peer Reviewer. ©2014 MarylandOnline

Peer Reviewer Eligibility
How Do I Become a QM Peer Reviewer? You must be ELIGIBLE: Must complete the Applying the QM Rubric workshop. Must complete all assessments in the online Peer Reviewer Course Two weeks online Objective and written assessments Practice Review Must have recent (within last 18 months) online teaching experience in a for-credit course. About The Content: The current procedures for registering and accessing the online Peer Reviewer Course are available at the QM web site ( ©2014 MarylandOnline

Reflections What are the three most significant (or surprising) things you learned by participating in our APP course? What do you intend to do next in regards to beginning to actively use QM and the QM Rubric with your course and/or at your institution? Do you have plans to become a QM peer reviewer? Explain, please. Do you have any general questions about the course review process, the rubric, or Quality Matters? Remind participants that during today’s workshop they have accomplished the following: Identify the underlying principles of QM. (Recognize key QM underlying principles and concepts.) Identify the critical elements of the QM quality assurance program, including the QM Rubric, materials, processes, and administrative components. Apply the QM Rubric to review online courses. Make decisions on whether the SPCH 1113 course meets selected QM Rubric standards. Apply the concept of alignment. Draft helpful recommendations for course improvement by citing annotations from the QM Rubric and evidence from the course. Use the questions on the slide to initiate a discussion of today’s workshop. ©2014 MarylandOnline

Link to the online evaluation:
Wrap Up and Evaluation Link to the online evaluation: ****HANDOUT: Participant Instructions**** Ask participants if they have any final questions. Remind participants that there are instructions in their packet for logging into MyQM and accessing the online evaluation. ©2014 MarylandOnline

Thanks to you… Quality Matters!
For more information visit the QM website at: Marker: You may want to add your own contact information so participants can contact you if they have questions. ©2014 MarylandOnline

Applying the QM Rubric Rubric

Similar presentations

Presentation on theme: "Applying the QM Rubric Rubric"— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

Applying the QM Rubric Rubric

Similar presentations

Presentation on theme: "Applying the QM Rubric Rubric"— Presentation transcript:

Similar presentations

About project

Feedback