Presentation on theme: "CMTO Registered Massage Therapist Program Transition to the Federal Inter-jurisdictional Competency Standards A Conversation: November 5, 2014."— Presentation transcript:
CMTO Registered Massage Therapist Program Transition to the Federal Inter-jurisdictional Competency Standards A Conversation: November 5, 2014
A Brief History CMTO Assessments (MCQ and OSCE) have traditionally been built upon job analysis research performed in Ontario The results were a legally-defensible link between what competent RMTs do on-the-job and successful performance on the OSCE and MCQ registration examinations The research results provide the content for the examinations
Registration Examination Cycle Job Analysis Research Defining the knowledge elements that define competent practice Content Outline (drives exam content) Question(item) and test development Examination Assembly and psychometric analysis of results Establishment of Cut Scores (Passing Standards) Award of Registration Recertification (RMTs demonstrate continued competency)
Inter-jurisdictional Competency Standards (IJC Standards) Created in response to federal direction that the RMT credential be more transportable Developed by a consortium of representatives/peers from regulated provinces Canada-wide Job Analysis Study performed Results used to establish IJC Standards
IJC Standards Represent the common body of knowledge representing MT practice in Canada Questions for the CMTO: 1.Do the IJC Standards represent MT practice in Ontario? 2.Can a link between the IJC Standards and the CMTO assessment instruments (OSCE and MCQ) be established?
2014 CMTO IJC Standards Transition Research Study Performed by CMTO RMTs (Subject Matter Experts), psychometric experts, representatives from the other regulated, and soon to be regulated province of Manitoba. Study performed in Spring/Summer of 2014 What procedures were followed? What were the results? How do the results relate to the CMTO Exams?
IJC Standards Transition Procedures – Review of IJC Standards Review of the IJC document: Practice CompetencyDomain & Level Performance IndicatorAssessment Vehicle Cognitive Psychomotor Affective Indicator # Reg Exam MC Reg Exam OP Ed Academic Ed Simulated Ed Clinical 1. Professional Practice Communication Utilize effective written communication Document complete and accurate information. xxx 2 Ensure legibility of information. xxx 3 Employ clear, concise and profession-specific language. xxx 4 Write in a manner that corresponds to the needs of the reader. xxx 5 Employ appropriate formatting, grammar and spelling. xxx
Tasks Identified for Inclusion in Study Based on Assessment Vehicle Only tasks identified as assessable via Reg Exam MC (multiple choice) and/or Reg Exam OP (performance/OSCE) were considered during the study. Tasks meeting study criteria would then be compared with current MCQ and OSCE content.
Tasks/Knowledge Elements: The 12 Considerations 1.Is the task important to test? 2.How difficult is it to test the task? 3.Are there references supporting the task? 4.Where does the task fall on the continuum of priority? 5.What level of professional experience is required to perform the task? 6.Is the task on the current MCQ/OSCE?
Task/Knowledge Elements: The 12 Considerations (continued) 7.Are there any tasks left off? 8.Is the task appropriate for testing in Ontario? 9.Does the task belong on MCQ/OSCE, both? 10.Are there non-assessable/duplicated tasks? 11.Based on final MCQ and OSCE outlines, do the examination specifications need to change (e.g., length, timing, #of stations, etc.,)? 12.Content area weightings for OSCE/MCQ?
Results based on SME opinions: Assessment Question posed to SMEs Assessment Decision Is the task/concept important to test?Yes/No How difficult is the task/concept to test?Very Difficult/Difficult/Moderate/Easy What is the priority to test this task/concept?Not Important/Somewhat Important/Important/Extremely Important Is there a reference available to support assessment of this task/concept? Yes/No What level of professional experience is required to demonstrate competence in this task/concept? Entry-level, Experienced, Advanced
Decision Criterion Applied: Assessment/Question posed to SMEsDecision Required for Task/Concept Inclusion 1. Is the task/concept important to test?Yes 2. How difficult is the task/concept to test?Difficult/Moderate/Easy 3. What is the priority to test this task/ concept? Somewhat Important/Important/Extremely Important 4. Is there a reference available to support assessment of this task/ concept? Yes 5. What level of professional experience is required to demonstrate competence in this task/concept? Entry-level
Questions Tasks on the IJC Standards not on OSCE/MCQ. 2 Tasks on OSCE/MCQ not on IJC 7.IJC deemed comprehensive – no tasks left off 8.All tasks appropriate for Ontario (universal) 9.For tasks that passed review–adopted IJC recommendations for placement on exams 10. Tasks assessable – duplicates eliminated, some editing for clarity
Questions 6-12 (continued) 11.Test Specifications: MCQ stays 150, 125 scored, 4-option multiple-choice, three hours. OSCE stations remained the same, some editing of graded assessments/ competencies to insure fairness to candidates. 12.Content area weightings established.
2015 MCQ Content Outline I.Professional Practice: 26 Tasks, 17% of exam, 22 questions (of 125) 1.1 Communication 1.2 Professional Conduct 1.3 Therapeutic Relationship II. Assessment and Treatment Planning: 63 Tasks, 37% of exam, 46 questions 2.1 Assessment 2.2 Treatment Planning 2.3 Adjunctive Therapeutic Modalities
Becoming an RMT: Certification Examinations/Introduction
MCQ Questions/OSCE Stations Updated MCQ items linked to new IJC Standards-based content outline OSCE stations/tasks scenarios linked to IJC Standards-based content outline Some questions/tasks eliminated from MCQ bank - new questions written New competencies/assessments created for the OSCE
Who develops the OSCE and MCQ? In accordance with International Standards all MCQ question writers and OSCE developers, trainers and raters are RMTs in good standing. SMEs Undergo rigorous training Development is supported by research Supervised by psychometric experts Question/OSCE stations undergo psychometric analysis prior to live scoring
Information Dissemination Notice provided to stakeholders New exams launched in Spring/Summer 2015 All psychometric standards will be followed for: item/examination/station/competency analysis, cut score establishment, score scaling and reporting, ensuring that all subsequent forms are equivalent in content, difficulty and defensibility Cycle shown in slide 3 remains the same
Transition Study results confirm that the 2015 IJC Standards-based CMTO MCQ and OSCE Examinations are linked to competent practice as an RMT in Ontario (and Canada)
CMTO continues to ensure that the RMT Examinations meet standards related to : Validity (questions are supportable with approved references) Reliability (psychometric analyses) Fairness (all candidates treated equitably) Standardization (all candidates receive comparable examination experiences)