Presentation is loading. Please wait.

Presentation is loading. Please wait.

Applying the QM Rubric August 2008 Inter-Institutional Quality Assurance in Online Learning.

Similar presentations


Presentation on theme: "Applying the QM Rubric August 2008 Inter-Institutional Quality Assurance in Online Learning."— Presentation transcript:

1 Applying the QM Rubric August 2008 Inter-Institutional Quality Assurance in Online Learning

2 Describe underlying principles of Quality Matters (QM). Use QM rubric & explain scoring system. Apply the QM rubric to online course and scenarios. After this training you will be able to…

3  Today’s Agenda  Folder Overview  QM Website  QM Contact Information QM Materials Overview

4  Share name, institution, job, and best distance learning practice.  Briefly (in one sentence) describe your best distance learning practice. Introductions

5 Trying it out with Standard 1.2 A statement introduces the student to the purpose of the course and to its components; in the case of a hybrid course, the statement clarifies the relationship between the face-to-face and online components.

6 Review Scenario You are reviewing a College Algebra course and find the following statement: This course reflects a shift in the importance that the world outside the schools increasingly places on thinking and problem solving. Procedural skills alone do not prepare students for that world. Therefore, students deserve a curriculum that develops their mathematical power and an assessment system that enables them to show it. Assessments that match the current vision of school mathematics involve activities that are based on significant and correct mathematics. These activities provide all students with opportunities to formulate problems, reason mathematically, and make connections among mathematical ideas. Students engage in solving realistic problems using information and the technological tools available in real life. Moreover, skills, procedural knowledge, and factual knowledge are assessed as part of the doing of real life mathematics Tests (Must be taken on campus) Quizzes Project (Optional) 50 Total number of points possible 475 (no project) 525 (if you do the project) Grading Scale: 90% to 100% = A 80% to 89% = B 70% to 79% = C 60% to 69% = D 0% to 59% = F A student who earns a final average of 70% or more on all assessments of objectives and intended learning outcomes for the College Algebra course has successfully fulfilled the general education and other essential core skill goals. Any extra credit problems are given to the entire class. There are no extra credit problems for any one individual during any part of the semester.

7 About Quality Matters

8 Peer Course Review Feedback Course Instructional Designers Institutions Faculty Course Developers National Standards & Research Literature Rubric Course Meets Quality Expectations Course Revision Quality Matters: Peer Course Review Process Training Faculty Reviewers

9 QM = Rubric and Process RubricProcess Outcome: earn QM recognition Tool to assess online courses during formal QM review Outcome: improve courses, meet institutional goals Guide to develop new online courses AND review and update online courses Outcome: earn QM recognition Must follow official QM guidelines and procedures Outcome: improve courses, meet institutional goals, demonstrate commitment to quality OFFICIALOutcome: earn QM recognition Tool to assess online courses during formal QM review Outcome: improve courses, meet institutional goals Guide to develop new online courses AND review and update online courses Outcome: earn QM recognition Must follow official QM guidelines and procedures Outcome: improve courses, meet institutional goals, demonstrate commitment to quality INFORMAL Outcome: improve courses, meet institutional goals, demonstrate commitment to quality Guide to develop new online courses AND review and update online courses Outcome: improve courses, meet institutional goals, demonstrate commitment to quality Institutions determine use and procedures

10 Factors Affecting Course Quality  Course design QM REVIEWS THIS  Course delivery (i.e. teaching, faculty performance)  Course content  Course management system  Institutional infrastructure  Faculty training and readiness  Student engagement and readiness

11 Design vs. Delivery The faculty member is integral to both design and delivery. Course Design … is the forethought & planning that a faculty member puts into the course. Course Delivery … is the actual teaching of the course, the implementation of the design. QM is about DESIGN - not delivery or faculty performance

12 Distinguish between design & delivery… Example: Discussion Board Design: Discussion board planned in course; students told how they should participate & how they can expect the faculty to participate. Delivery: How often the faculty member actually participates in the discussion; what the faculty member actually says to students.

13 For Our Purposes, Quality Is… More than average; more than “good enough” Attempt to capture what’s expected in an effective online course at about an 85% level Based on research & widely accepted standards 85 %

14 What QM is NOT About…  Not about an individual instructor (it’s about the course)  Not about faculty evaluation (it’s about course quality)  Not about judgment (it’s about diagnosis and improvement)  Not about “win/lose” or “pass/fail” (it’s about continuous improvement in a supportive environment)

15 QM Collegial Review vs. Faculty Evaluation QM ReviewFaculty Evaluation  Ongoing  Focus: course design  Outcome: course improvement  Characteristics o Non-threatening o Team approach that includes faculty o Full disclosure to faculty  Single point in time  Focus: delivery  Outcome: decision on performance for promotion/tenure  Characteristics o Win/lose situation o Confidential/secretive

16 The Rubric

17 Eight General Standards: 1. Course Overview and Introduction 2. Learning Objectives (Competencies) 3. Assessment and Measurement 4. Resources and Materials 5. Learner Engagement 6. Course Technology 7. Learner Support 8. Accessibility Key components must align. Alignment: Critical course elements work together to ensure that students achieve the desired learning outcomes.

18 Rubric Scoring StandardsPointsRelative Value 173Essential 112Very Important 121Important TOTALS 4085 In an official QM Review: Team of 3 reviewers initially score individually One score per standard based on team majority Pre-assigned point value Yes/No decision; All/None points Consensus is NOT required

19 Points are NOT on assigned on a sliding scale… Standards If the standard is met … If the standard is not met … Essential3 points0 points Very Important 2 points0 points Important1 point0 points

20 To Meet Expectations…Two Thresholds A course must achieve: 1. “Yes” on all 17 of the 3-point “essential” standards. 2. A minimum of 72 out of 85 points

21 The Instructor Worksheet

22 Instructor Worksheet  Read it first  Refer to it during the review  Use in team discussions

23 Instructor Worksheet = The Voice of the Instructor  Key piece in the review  Includes info about: Institutionally mandated objectives, materials, practices, policies Materials outside course pages Types of interaction used & instructor’s statement on the appropriateness of interaction in the course Additional items that may require review

24 Hands On Practice

25 Your Point of View as a QM Course Reviewer… Take the students’ point of view Advocate for the student Can’t find evidence standard is met…don’t assume it is or isn’t there….ask the course developer (faculty member)

26 How to decide…. For EACH standard 1. Read the specific review standard and the annotation; review the examples, if needed. 2. Look for evidence that the standard is met in this course. 3. Ask yourself: Does this course meet the standard at an 85% or better level? 4. Decide Yes or No and enter your answer in the web-based rubric form. 5. Include comments/suggestions as documentation.

27 Hands-On Practice Review the Anthropology Course Go to Username is fipse Password is fipse Focus on 17 essential (3 point) standards Write 1 recommendation per standard

28 Hands-On Practice 1. Find a Partner 2. Locate login handout in folder 3. Login to ANT Read Instructor Worksheet for course

29 General Standard 1: Course Overview and Introduction Specific Review Standard What does this mean? 1.1: Instructions make clear how to get started and where to find various course components. 1.1: Do the students know what to do first? 1.2: A statement introduces the student to the purpose of the course and to its components; in the case of a hybrid course, the statement clarifies the relationship between the face-to-face and online components. 1.2: Do the students know what the learning objectives of the course are, how the course is organized, what learning activities are required, how to communicate, and other general course information?

30 What Are the Consequences of Your Decision? In a formal QM review:  What does a YES decision mean?  What does a NO decision mean?

31 Recommendations

32 A recommendation MUST be written if you decide that the course does not meet the specific standard. No Choice!

33 ConstructiveTry to offer solutions, not just identify problems. SpecificInclude a specific example of what is being recommended. MeasurableHow will you or the ID/instructor know when the recommendation has been implemented? SensitiveAvoid negative language. Keep recommendations & comments on a positive note. BalancedPoint out strengths as well as weaknesses. How to Write a Useful Recommendation

34 Improve these recommendations… Assignment instructions weren’t clear.  I wasn’t sure if assignment #1 required a written paper. You didn't tell the students how to find the additional resources.  I had trouble locating the resources; you may want to put a link on the home page for easy access. The text on the page was too hard to read.  The green text on a blue background was difficult to read. You might want to try basic black on white.

35 The learning objectives aren’t measurable.  I had trouble figuring out how the objectives would be measured. I think it would help if you rephrased the learning objectives using active verbs (explain, distinguish, compare, etc.) The assessments are weak.  I had difficulty connecting the assessments to the learning objectives; they seemed unrelated to the learning objectives. I think you’ll want to review the learning objectives and make sure that they are reflected in the exams.

36 To the ANT Course (continued) Apply the QM Standards

37 General Standard 2: Learning Objectives (Competencies) Specific Review StandardWhat does this mean? 2.I: The course learning objectives describe outcomes that are measurable. 2.1: Do course-level learning objectives describe an action that the students perform that can be observed or measured? 2.2: The module/unit learning objectives describe outcomes that are measurable and consistent with the course-level objectives. 2.2: Can the module/unit level objectives be observed and measured? Do these module/unit level objectives flow from the course-level objectives?

38 Key sections that must align… Course Learning Objectives (2) Resources, Materials (4) & Technology (6) Assessment and Measurement (3) Learner Interactions & Activities (5)

39 Remember Bloom?Bloom

40 From an Introduction to Psychology Course Objective: Summarize the relationships that exist between biology and human behavior Think About “Alignment” Assignment: Read Chapter 4 in text; review study guide, objective 2, page 4. Assignment: View Chapter 4 PPT; listen to Chapter 4 podcast Graded Assignment: Prepare table that lists the 7 major biological areas; ask students to define/describe each area and list 3 specific ways this area affects human behavior. Graded Discussion Forum Question: Psychology is the study of behavior and mental processes. Why, then, are psychologists interested in biological matters such as the nervous system, the endocrine system and heredity? Read and respond to classmates’ postings. Mid-Term Exam: 50-item multiple choice exam taken in campus Testing Center.

41 General Standard 2: Learning Objectives (Competencies) Specific Review StandardWhat does this mean? 2.3: All learning objectives are stated clearly and written from the students’ perspective. 2.3: Are the course and module/unit learning objectives written in clear, direct language that the students can easily understand? 2.4: Instructions to students on how to meet the learning objectives are adequate and stated clearly. 2.4: Does this course use narratives, bulleted lists, charts, etc. to tell the students what they must do to meet the course and module/unit level objectives?

42 General Standard 3: Assessment & Measurement (we’ll come back to this one) 3.1: The types of assessments selected measure the stated learning objectives & are consistent with course activities and resources. 3.2: The course grading policy is stated clearly. 3.3: Specific and descriptive criteria are provided for the evaluation of students’ work and participation.

43 General Standard 4: Resources & Materials Specific Review StandardWhat does this mean? 4.1: The instructional materials contribute to the achievement of the stated course & module/unit learning objectives. 4.1: Is there evidence of sufficient content for the students to learn the discipline? Does this content support the objectives, assessments and other course elements? 4.2: The relationship between the instructional materials & the learning activities is clearly explained to the student. 4.2: Can the students easily determine the purpose of all course components & how each will help to achieve the learning objectives?

44 General Standard 5: Learner Engagement Specific Review Standard What does this mean? 5.1: The learning activities promote the achievement of stated learning objectives. 5.1: Do the learning activities flow from the learning objectives? Do the learning activities encourage students to “practice” their learning? 5.2: Learning activities foster instructor-student, content- student, & if appropriate to this course, student-student interaction. 5.2: Does the course encourage student-instructor and student- content interaction? What is the instructor’s philosophy on student-student interaction?

45 General Standard 6: Course Technology Specific Review StandardWhat does this mean? 6.1: The tools & media support the learning objectives, & are appropriately chosen to deliver the content of the course. 6.1: Do the learning activities flow from the stated course and module learning objectives? Do the activities encourage the students to “practice” their learning? 6.2: The tools & media support student engagement and guide the student to become an active learner. 6.2: Do the tools & media selected encourage the students to be actively engaged in the learning process? 6.3: Navigation throughout the online components of the course is logical, consistent and efficient. 6.3: Is the navigation of the course easy for the students to use and does it effectively guide their movement through the course?

46 General Standard 3: Assessment & Measurement Specific Review StandardWhat does this mean? 3.1 The types of assessments selected measure the stated learning objectives & are consistent with course activities & resources. 3.1: Do the assessments flow from the course and module/unit objectives? Are they consistent with these objectives? 3.2 The course grading policy is stated clearly. 3.2: Do students know how points will be awarded and/or grades assigned? Is the policy on late work explicit? 3.3 Specific & descriptive criteria are provided for the evaluation of students’ work & participation. 3.3: Do students know what work is expected? Is there a description of the expectations for this work and the criteria by which it will be graded? Do the students have access to rubrics, examples, etc?

47 General Standard 7: Learner Support  No “essential” 3-point elements  Primarily concerned with instructional support – an institutional responsibility

48 General Standard 8: Accessibility Specific Review Standard What does this mean? 8.1: The course incorporates ADA standards & reflects conformance with institutional policy regarding accessibility in online and hybrid courses. 8.1: Does the course include these three elements? Offered using software that is accepted as “ADA compliant.” AND Include a link to the institution’s ADA policy and/or guidelines. AND Include a brief statement that clearly tells students how to access ADA services at the institution.

49 Scenarios

50 Evaluating Scenarios  Divide into groups.  Discuss the scenarios that focus on the 17 essential standards.  Take the “quiz” with your group. Use the material in your books and each other as references.  Jot down your reasoning and be prepared to discuss your decision.

51 1. No. Doesn’t tell students what to do first, especially a first-time college student. 2. No. This is a hybrid course & this example doesn’t explain relationship between online & onsite elements. 3. No. Objectives are not measurable. 4. Yes, but what suggestions can you offer for improvement?

52 5. No, objectives are unnecessarily complex & confusing to a new student in this class. 6. No, direct link to learning objectives must be provided. 7. Yes, but students should have requisite web skills. 8. No. This policy is unnecessarily confusing and complex. 9. No, descriptive criteria is a key element of this standard & must be provided 10. Consult with subject matter expert on your team, use common sense & your own experience.

53 11. No, it’s not clear how these materials are related to the course. 12. Yes, flashcards are a reasonable learning activity for the stated objective. 13. Yes. Be sure to consult the Instructor Worksheet for the instructor’s philosophy about student-student interaction in this course. 14. No. These materials are not integrated into the course.

54 15. Yes, all links are described and related to the course. 16. No. The instructions require toggling between multiple areas of the course. 17. No. The course is delivered through a blog & ; the instructor must provide evidence that it is ADA compliant.

55 Showcase Your Courses

56 Course Showcase  Think about your own course:  What do you do that meets the QM standards?  What improvements might you make?  Share your thoughts with your partner  Summarize for the group

57 Serving as a QM Peer Reviewer

58 How Do I Become a QM Peer Reviewer?  You must be ELIGIBLE: Must complete this f2f Applying the QM workshop Must complete this f2f Applying the QM workshop Must complete the assessments in the online Peer Reviewer Certification Module Must complete the assessments in the online Peer Reviewer Certification Module Approximately one week online Approximately one week online Includes review material Includes review material Objective and written assessments Objective and written assessments Must have recent (within last 18 months) online teaching experience Must have recent (within last 18 months) online teaching experience Peer Reviewer Eligibility

59 Please fill out the session evaluation in your folder. Evaluation

60 More information at: Thanks to YOU…Quality Matters!

61 Quick Review

62 Underlying Principles of QM… 1. Based in national standards of best practice, the research literature & instructional design principles 2. The course under review does not have to be “perfect” but better than just “good enough” (Standards met at about 85% level or better.) 3. Integral to a continuous quality improvement process 4. Valued as part of a faculty-driven, peer review process 5. Designed to promote student learning

63 Underlying Principles of QM (cont)…. 6. Designed to ensure all reviewed courses will eventually meet expectations 7. QM is a collegial review process, not an evaluation process 8. A review team must include an external peer reviewer 9. Set up so that the course faculty or instructor considered part of the review team

64 Formal Peer Review Team 3 faculty peer reviewers must attend QM training must be experienced online instructors one MUST be external to the course’s originating institution there must be a subject matter expert (SME) on the team NOTE: The SME could also be the external reviewer chaired by additionally trained QM reviewer AND Faculty course developer (Instructor) access to rubric prior to review involved in pre-review discussions consulted during review Who’s On the Team?

65 To Meet Expectations…Two Thresholds A course must achieve: 1. “Yes” on all 17 of the 3-point “essential” standards. 2. A minimum of 72 out of 85 points

66 Online & Hybrid Courses  Rubric designed for both  Same set of standards apply to both  How standards achieved may differ  For hybrids, focus on pedagogical integration of online & onsite components

67 Quick Quiz


Download ppt "Applying the QM Rubric August 2008 Inter-Institutional Quality Assurance in Online Learning."

Similar presentations


Ads by Google