Presentation on theme: "Medication Aide Certification Examination (MACE) and Standard Setting Process Ada Woo, PhD Senior Psychometrician, NCSBN."— Presentation transcript:
Medication Aide Certification Examination (MACE) and Standard Setting Process Ada Woo, PhD Senior Psychometrician, NCSBN
Outline Overview of test development process MACE item development 2010 national MACE standard setting 2010 medication aide job analysis 2011 medication aide knowledge survey
MACE Test Development Process 1.Job Analysis 2.Content Outline 3.Item Writing and Item Review 4.Subject Matter Expert (SME) Panel Review 5.Editorial Review 6.Building Test Forms 7.Setting the Passing Standard 8.Pretesting of Items 9.Administering Examinations
NCSBN and MACE: Historical Background At the 2005 NCSBN Delegate Assembly, following a motion from the floor, the delegates passed the resolution to investigate the possibility of a competency examination for medication assistive personnel. In 2007, the delegates adopted the Medication Assistant-Certified (MA-C) Model Curriculum.
MA-C Model Curriculum Provide the boards with a curriculum for MA-Cs that will assist the boards in their mission of public protection Allow for uniformity in MA-C education across boards and other regulating agencies Consist of 60 hours of didactic training and 40 hours of supervised clinical practicum Content modules comprising the didactic portion are provided in five modules, including recommended times for each
MACE Test Development In August 2008, NCSBN acquired exclusive ownership of the MACE examination. NCSBN is responsible for the development of MACE content. Two MACE item writing and review workshops were conducted during May 12-14 and May 19-21, 2009 in NCSBN offices. Fifteen subject matter experts (SMEs) representing six states and a wide range of expertise participated in the workshops.
Participants from May 12 – 14, 2009 meeting NameNCSBN Area StateJob Title Julie Bucher, RNIORProgram Director Bernadette Murphy, MBA, BSN, RNIORNurse Educator Nancy Stephen, RNIORNurse Manager Anne Citarella, RNIIIAAssistant Professor Victor Palmer, RNIIIAAssistant Program Manager for Allied Health and Long-Term Care Mary Jameson, RNIIMOStaff RN/CNA-CMT-CPR-Insulin Instructor Jill Layne, BSNIIMODirector of Nursing Mary Stassi, RNCIIMOHealth Occupations Coordinator Anne Citarella, RNIIIAAssistant Professor Penny Worthley, RN,C, LNHAIIMOStaff Education Coordinator Avril Marshall, BA, RNIVDCInstructor Dorothy Kritsch, BSN, RN,IVNJProfessional Service Rep. Penny Worthley, RN,C, LNHAIIMOStaff Education Coordinator Avril Marshall, BA, RNIVDCInstructor
Participants from May 19 – 21, 2009 meeting NameNCSBN Area StateJob Title Lisa Bare, RNIIINCAssistant Director of Nursing Bethany Davis, BSN, RNIIINCNursing Instructor Lynn Thorpe, MSN, RNIIINCCompetency Evaluator/examiner for nurse aide candidates
Procedures of Item Writing and Review Meetings/Workshops SMEs discussed a proposed content outline constructed based on the requirement listed in MA-C Model Curriculum. After adopting the content outline, the SMEs established weights/percentages for each content category. SMEs reviewed items administered on recent MACE test forms based on item statistics. In all, 180 items were identified for review. SMEs reviewed items for accuracy and appropriateness of entry-level medication aide range of authorized duties.
Workshop Outcome MACE Item Writing and Review--Summary of Activity Item disposition Total items written/reviewedApproved for pretestFuture rewrites Activity at Meeting Newly written (total number)91 Newly written approved for pretest 52 Client specific items with stats180 Client specific items approved for pretest 57 Items identified as problematic 14 Total Items Approved for Pretest 109 Total Items Reviewed271
MACE Content Outline I. Authorized Duties A. Building Relationships B. Role of Delegating/Supervising Nurse C. Role of MA-C 1. Permitted Duties 2. Restrictions/Limitations D. The Process of Delegation 1. Responsibility of MA-C When Accepting Delegation Tasks E. Specific Legal and Ethical Issues
(cont) MACE Content Outline II. Medication Administration A. Administering and Charting Medications 1. Medication Orders 2. Documentation of Medication Administration 3. Storage 4. Disposal B. Safety and Rights of Medication Administration C. Preparation and Actual Medication Administration D. Prevention of Medication Errors E. Routes of Administration F. Factors Affecting How the Body Uses Medication G. Classifications/Categories of Medications Related to Body Systems and Actions (e.g., antimicrobials, cardiovascular, dermatological, endocrine…) H. Rights of Individuals
(cont) MACE Content Outline III. Medication Concepts and Measurements A. Medication Basics 1. Terminology and abbreviations 2. Dosage Range 3. Actions and Implications 4. Therapeutic and other side effects (e.g. idiosyncratic, paradoxical, antagonist) 5. Precautions 6. Allergic and Adverse Reactions 7. Interactions B. Forms of Medication 1. Liquid 2. Solid and semi-solids C. Mathematics, Weights and Measures
(cont) MACE Content Outline IV. Observation, Care and Reporting A. Causes and Reporting of Medication Errors B. Reporting of Symptoms and Side Effects C. Reporting Any Change From Clients Normal Condition D. Location of Resources and References (e.g., nurse, pharmacist, physician, package/drug insert, drug reference manuals)
National MACE Standard Setting The purpose is to determine a passing score for the National MACE examination in compliance with the Standards for Educational and Psychological Testing Used criterion-referenced standard setting method Used modified-Angoff procedures Used two test forms (total of 78 unique items)
Subject Matter Experts SMEAreaStateWork Setting Bernadette Murphy, MBA, BSN, RN IORState Hospital Nancy Stephen, RNIORState Hospital Jill Layne, BSN, RNIIMOContinuing Care Retirement Community (CCRC) TyAnn Bell, CNA, CMTIIMOCCRC Victor Palmer, RNIIIACommunity College Lynn Thorpe, MSN, RNIIINCCompetency Evaluation Martha Holloway, MS, RNIIIALPublic Schools Jeanne Delicata, MSN, RN, ACNS, BC IVMEVeterans Home Lynn Johnston, MS, BSN, RN IVMEVeterans Home Maryann Ingram, LPNIVMDNursing Home
Standard Setting Procedures 1.Received overview of standard setting process 2.Discussed qualifications for taking the MACE examination and define minimally competent candidates 3.Reviewed content outline 4.Received standard setting training 5.Rated practice items 6.Reviewed and discussed practice item ratings 7.Rated first batch of actual items 8.Rated remaining items 9.Discussed results from round 1 ratings 10.Conducted round 2 ratings
Results Form and Round% Cut-scoreRaw Cut-score (50 item test) Standard Deviation Intraclass Rating Reliability Form 1 Round 177.94%38.97 (39)2.710.74 Form 2 Round 177.31%38.66 (39)2.450.70 Form 1 Round 279.45%39.77 (40)3.120.77 Form 2 Round 278.94%39.47 (40)3.170.75 Ratings became more consistent among judges from round 1 to round 2. Judges unanimously agreed that the national MACE passing standard should be set at 40 (out of 50) for 2010 national MACE test forms.
Post-Workshop Questionnaire Question 1: I feel the training was adequate.Average rating: 3.73 Question 2: I understood the definition of a minimally competent examinee. Average rating : 3.73 Question 3: I understood the rating process.Average rating : 3.82 Question 4: I feel my opinion was considered.Average rating : 3.91 Question 5: The practice exercises helped me understand the process. Average rating : 3.82 Rating scale: 1 = strongly disagree 2 = disagree 3 = agree 4 = strongly agree
Some Feedback from SMEs Staff was wonderful about explaining the process of standard setting. It was a great experience. Great experience. The exercise validated the process of the cut score. This was a wonderful opportunity to be able to participate in this workshop. I look forward to participating again soon in other projects. From a state evaluators and nurse educators perspective, the workshop was relevant in understanding the concepts and dynamics involved in establishing a minimum for safe medication administration of entry level certified medication aides.
(cont) SME Feedback Excellent process of standard setting! Everyones opinion was valued and respected. Thorough explanations. It was a great process! Invite me back! Thank you for allowing me to participate in this process. I see and better understand how the NCSBN works to keep the public safe. I enjoyed the experience and feel the process was very fair and accurate. Very informative and enjoyable! Staff was most helpful. Thanks for the opportunity.
2010 Medication Aide Job Analysis Proposed Timeline and Next Steps October 2010 – August 2011 1. Develop methodology plan 2. Host SME workshop 3. Develop surveys and correspondence 4. Mail pre-survey notice and surveys 5. Collect data 6. Analyze data 7. Conduct nonresponder study 8. Finalize report 9. Publish research brief 10. Host SME Test Specification workshop
2010 Medication Aide Knowledge Survey Proposed Timeline and Next Steps April 2011 – November 2011 1. Develop methodology plan 2. Host SME workshop 3. Develop surveys and correspondence 4. Mail pre-survey notice and surveys 5. Collect data 6. Analyze data 7. Conduct nonresponder study 8. Finalize report 9. Publish research brief
How you can assist in the 2010 Medication Aide Surveys No National Registry Challenges in obtaining mailing addresses for a nationally representative sample Examples from other nursing survey studies: 1.Nurse Aide 2010 JA and KSA Surveys – Randomly selected 6,500 nursing directors from hospitals, nursing homes and long term care facilities based on a facility mailing list 2.NCLEX-RN 2008 practice analysis – Randomly sampled 6,000 NCLEX-RN candidates who passed the examination from January 1 to April 13, 2008.
MACE Publications 1.MACE Item Writing and Review Workshop Report - https://www.ncsbn.org/MACE_Item_Writing_Worksho p_Report.pdf https://www.ncsbn.org/MACE_Item_Writing_Worksho p_Report.pdf 2.MACE Standard Setting Workshop Report - https://www.ncsbn.org/Standard_Setting_Report_for_ MACE.pdf https://www.ncsbn.org/Standard_Setting_Report_for_ MACE.pdf 3.Medication Assistant-Certified (MA-C) Model Curriculum - https://www.ncsbn.org/07_Final_MAC.pdf https://www.ncsbn.org/07_Final_MAC.pdf 4.Quick Reference to MA-C Model Curriculum - https://www.ncsbn.org/07_new_MACChecklist.pdf https://www.ncsbn.org/07_new_MACChecklist.pdf