Presentation on theme: "A Conversation: November 5, 2014"— Presentation transcript:
1 A Conversation: November 5, 2014 CMTO Registered Massage Therapist Program Transition to the Federal Inter-jurisdictional Competency StandardsA Conversation:November 5, 2014
2 A Brief HistoryCMTO Assessments (MCQ and OSCE) have traditionally been built upon job analysis research performed in OntarioThe results were a legally-defensible link between what competent RMTs do on-the-job and successful performance on the OSCE and MCQ registration examinationsThe research results provide the content for the examinations
3 Registration Examination Cycle Job Analysis ResearchDefining the knowledge elements that define competent practiceContent Outline(drives exam content)Question(item) and test developmentExamination Assembly and psychometric analysis of resultsEstablishment of Cut Scores(Passing Standards)Award of RegistrationRecertification(RMTs demonstrate continued competency)
4 Inter-jurisdictional Competency Standards (IJC Standards) Created in response to federal direction that the RMT credential be more transportableDeveloped by a consortium of representatives/peers from regulated provincesCanada-wide Job Analysis Study performedResults used to establish IJC Standards
5 IJC StandardsRepresent the common body of knowledge representing MT practice in CanadaQuestions for the CMTO:Do the IJC Standards represent MT practice in Ontario?Can a link between the IJC Standards and the CMTO assessment instruments (OSCE and MCQ) be established?
6 2014 CMTO IJC Standards Transition Research Study Performed by CMTO RMTs (Subject Matter Experts), psychometric experts, representatives from the other regulated, and soon to be regulated province of Manitoba.Study performed in Spring/Summer of 2014What procedures were followed?What were the results?How do the results relate to the CMTO Exams?
7 IJC Standards Transition Procedures – Review of IJC Standards Review of the IJC document:Practice CompetencyDomain & LevelPerformance IndicatorAssessment VehicleCognitivePsychomotorAffectiveIndicator #Reg Exam MCReg Exam OPEd AcademicEd SimulatedEd Clinical1. Professional PracticeCommunicationUtilize effective written communication.21Document complete and accurate information.xEnsure legibility of information.3Employ clear, concise and profession-specific language.4Write in a manner that corresponds to the needs of the reader.5Employ appropriate formatting, grammar and spelling.
8 Tasks Identified for Inclusion in Study Based on Assessment Vehicle Only tasks identified as assessable via Reg Exam MC (multiple choice) and/or Reg Exam OP (performance/OSCE) were considered during the study.Tasks meeting study criteria would then be compared with current MCQ and OSCE content.
9 Tasks/Knowledge Elements: The 12 Considerations Is the task important to test?How difficult is it to test the task?Are there references supporting the task?Where does the task fall on the continuum of priority?What level of professional experience is required to perform the task?Is the task on the current MCQ/OSCE?
10 Task/Knowledge Elements: The 12 Considerations (continued) Are there any tasks left off?Is the task appropriate for testing in Ontario?Does the task belong on MCQ/OSCE, both?Are there non-assessable/duplicated tasks?Based on final MCQ and OSCE outlines, do the examination specifications need to change (e.g., length, timing, #of stations, etc.,)?Content area weightings for OSCE/MCQ?
11 Results based on SME opinions: Assessment Question posed to SMEsAssessment DecisionIs the task/concept important to test?Yes/NoHow difficult is the task/concept to test?Very Difficult/Difficult/Moderate/EasyWhat is the priority to test this task/concept?Not Important/Somewhat Important/Important/Extremely ImportantIs there a reference available to support assessment of this task/concept?What level of professional experience is required to demonstrate competence in this task/concept?Entry-level, Experienced, Advanced
12 Decision Criterion Applied: Assessment/Question posed to SMEsDecision Required for Task/Concept Inclusion1. Is the task/concept important to test?Yes2. How difficult is the task/concept to test?Difficult/Moderate/Easy3. What is the priority to test this task/ concept?Somewhat Important/Important/Extremely Important4. Is there a reference available to support assessment of this task/ concept?5. What level of professional experience is required to demonstrate competence in this task/concept?Entry-level
13 Questions 6-1215 Tasks on the IJC Standards not on OSCE/MCQ. 2 Tasks on OSCE/MCQ not on IJCIJC deemed comprehensive – no tasks left offAll tasks appropriate for Ontario (universal)For tasks that passed review–adopted IJC recommendations for placement on examsTasks assessable – duplicates eliminated, some editing for clarity
14 Questions 6-12 (continued) Test Specifications: MCQ stays 150, 125 scored, 4-option multiple-choice, three hours. OSCE stations remained the same, some editing of graded assessments/ competencies to insure fairness to candidates.Content area weightings established.
15 2015 MCQ Content Outline Professional Practice: 26 Tasks, 17% of exam, 22 questions (of 125)1.1 Communication1.2 Professional Conduct1.3 Therapeutic RelationshipII. Assessment and Treatment Planning:63 Tasks, 37% of exam, 46 questions2.1 Assessment2.2 Treatment Planning2.3 Adjunctive Therapeutic Modalities
16 2015 MCQ Outline (continued) Treatment and Patient Self-Care79 tasks, 46% of Exam, 57 Questions3.1 Treatment Principles3.2 Modalities and Techniques3.2.a Massage Skills3.2.b Myofascial Techniques3.2.c Therapeutic Exercise3.2.d. Joint Mobilization3.2.e Thermal Therapy
17 2015 OSCE Content Outline Professional Practice: 17 Tasks, 10% of exam1.1 Communication1.2 Professional Conduct1.3 Therapeutic RelationshipII. Assessment and Treatment Planning:68 Tasks, 35% of exam2.1 Assessment2.2 Treatment Planning
18 2015 OSCE Content Outline (continued) Treatment and Patient Self-Care:101 tasks, 55% of Exam3.1 Treatment Principles3.2 Modalities and Techniques3.2.a Massage Skills3.2.b Myofascial Techniques3.2.c Therapeutic Exercise3.2.d. Joint Mobilization3.2.e Thermal Therapy
19 Becoming an RMT: Certification Examinations/Introduction
20 MCQ Questions/OSCE Stations Updated MCQ items linked to new IJC Standards-based content outlineOSCE stations/tasks scenarios linked to IJC Standards-based content outlineSome questions/tasks eliminated from MCQ bank - new questions writtenNew competencies/assessments created for the OSCE
21 Who develops the OSCE and MCQ? In accordance with International Standards all MCQ question writers and OSCE developers, trainers and raters are RMTs in good standing.SMEs Undergo rigorous trainingDevelopment is supported by researchSupervised by psychometric expertsQuestion/OSCE stations undergo psychometric analysis prior to live scoring
22 Information Dissemination Notice provided to stakeholdersNew exams launched in Spring/Summer 2015All psychometric standards will be followed for: item/examination/station/competency analysis, cut score establishment, score scaling and reporting, ensuring that all subsequent forms are equivalent in content, difficulty and defensibilityCycle shown in slide 3 remains the same
23 Transition Study results confirm that the 2015 IJC Standards-based CMTO MCQ and OSCE Examinations are linked to competent practice as an RMT in Ontario (and Canada)
24 CMTO continues to ensure that the RMT Examinations meet standards related to : Validity (questions are supportable with approved references)Reliability (psychometric analyses)Fairness (all candidates treated equitably)Standardization (all candidates receive comparable examination experiences)