1 Ada Woo, PhD Senior Psychometrician, NCSBN Medication Aide Certification Examination (MACE™) and Standard Setting ProcessAda Woo, PhDSenior Psychometrician, NCSBN
2 Outline Overview of test development process MACE item development 2010 national MACE standard setting2010 medication aide job analysis2011 medication aide knowledge survey
3 MACE Test Development Process Job AnalysisContent OutlineItem Writing and Item ReviewSubject Matter Expert (SME) Panel ReviewEditorial ReviewBuilding Test FormsSetting the Passing StandardPretesting of ItemsAdministering Examinations
4 NCSBN and MACE: Historical Background At the 2005 NCSBN Delegate Assembly, following a motion from the floor, the delegates passed the resolution to investigate the possibility of a competency examination for medication assistive personnel.In 2007, the delegates adopted the Medication Assistant-Certified (MA-C) Model Curriculum.
5 MA-C Model Curriculum Provide the boards with a curriculum for MA-Cs that will assist the boards in their mission of public protectionAllow for uniformity in MA-C education across boards and other regulating agenciesConsist of 60 hours of didactic training and 40 hours of supervised clinical practicumContent modules comprising the didactic portion are provided in five modules, including recommended times for each
6 MACE Test DevelopmentIn August 2008, NCSBN acquired exclusive ownership of the MACE examination.NCSBN is responsible for the development of MACE content.Two MACE item writing and review workshops were conducted during May and May 19-21, 2009 in NCSBN offices.Fifteen subject matter experts (SMEs) representing six states and a wide range of expertise participated in the workshops.
7 Participants from May 12 – 14, 2009 meeting NameNCSBN AreaStateJob TitleJulie Bucher, RNIORProgram DirectorBernadette Murphy, MBA, BSN, RNNurse EducatorNancy Stephen, RNNurse ManagerAnne Citarella, RNIIIAAssistant ProfessorVictor Palmer, RNAssistant Program Manager for Allied Health and Long-Term CareMary Jameson, RNMOStaff RN/CNA-CMT-CPR-Insulin InstructorJill Layne, BSNDirector of NursingMary Stassi, RNCHealth Occupations CoordinatorPenny Worthley, RN,C, LNHAStaff Education CoordinatorAvril Marshall, BA, RNIVDCInstructorDorothy Kritsch, BSN, RN,NJProfessional Service Rep.
8 Participants from May 19 – 21, 2009 meeting NameNCSBN AreaStateJob TitleLisa Bare, RNIIINCAssistant Director of NursingBethany Davis, BSN, RNNursing InstructorLynn Thorpe, MSN, RNCompetency Evaluator/examiner for nurse aide candidates
9 Procedures of Item Writing and Review Meetings/Workshops SMEs discussed a proposed content outline constructed based on the requirement listed in MA-C Model Curriculum.After adopting the content outline, the SMEs established weights/percentages for each content category.SMEs reviewed items administered on recent MACE test forms based on item statistics. In all, 180 items were identified for review. SMEs reviewed items for accuracy and appropriateness of entry-level medication aide range of authorized duties.
10 MACE™ Item Writing and Review--Summary of Activity Workshop OutcomeMACE™ Item Writing and Review--Summary of ActivityItem dispositionTotal items written/reviewedApproved for pretestFuture rewritesActivity at MeetingNewly written (total number)91Newly written approved for pretest52Client specific items with stats180Client specific items approved for pretest57Items identified as problematic14Total Items Approved for Pretest109Total Items Reviewed271
11 MACE Content Outline I. Authorized Duties A. Building Relationships B. Role of Delegating/Supervising NurseC. Role of MA-C1. Permitted Duties2. Restrictions/LimitationsD. The Process of Delegation1. Responsibility of MA-C When Accepting Delegation TasksE. Specific Legal and Ethical Issues
12 (con’t) MACE Content Outline II. Medication AdministrationA. Administering and Charting Medications1. Medication Orders2. Documentation of Medication Administration3. Storage4. DisposalB. Safety and Rights of Medication AdministrationC. Preparation and Actual Medication AdministrationD. Prevention of Medication ErrorsE. Routes of AdministrationF. Factors Affecting How the Body Uses MedicationG. Classifications/Categories of Medications Related to Body Systems and Actions (e.g., antimicrobials, cardiovascular, dermatological, endocrine…)H. Rights of Individuals
13 (con’t) MACE Content Outline III. Medication Concepts and MeasurementsA. Medication Basics1. Terminology and abbreviations2. Dosage Range3. Actions and Implications4. Therapeutic and other side effects (e.g. idiosyncratic, paradoxical, antagonist)5. Precautions6. Allergic and Adverse Reactions7. InteractionsB. Forms of Medication1. Liquid2. Solid and semi-solidsC. Mathematics, Weights and Measures
14 (con’t) MACE Content Outline IV. Observation, Care and ReportingA. Causes and Reporting of Medication ErrorsB. Reporting of Symptoms and Side EffectsC. Reporting Any Change From Client’s Normal ConditionD. Location of Resources and References (e.g., nurse, pharmacist, physician, package/drug insert, drug reference manuals)
15 National MACE Standard Setting The purpose is to determine a passing score for the National MACE examination in compliance with the Standards for Educational and Psychological TestingUsed criterion-referenced standard setting methodUsed modified-Angoff proceduresUsed two test forms (total of 78 unique items)
17 Standard Setting Procedures Received overview of standard setting processDiscussed qualifications for taking the MACE examination and define minimally competent candidatesReviewed content outlineReceived standard setting trainingRated practice itemsReviewed and discussed practice item ratingsRated first batch of actual itemsRated remaining itemsDiscussed results from round 1 ratingsConducted round 2 ratings
18 Raw Cut-score (50 item test) Intraclass Rating Reliability ResultsForm and Round% Cut-scoreRaw Cut-score (50 item test)Standard DeviationIntraclass Rating ReliabilityForm 1 Round 177.94%38.97 (39)2.710.74Form 2 Round 177.31%38.66 (39)2.450.70Form 1 Round 279.45%39.77 (40)3.120.77Form 2 Round 278.94%39.47 (40)3.170.75Ratings became more consistent among judges from round 1 to round 2.Judges unanimously agreed that the national MACE passing standard should be set at 40 (out of 50) for 2010 national MACE test forms.
19 Post-Workshop Questionnaire Question 1: I feel the training was adequate.Average rating: 3.73Question 2: I understood the definition of a “minimally competent examinee”.Average rating : 3.73Question 3: I understood the rating process.Average rating : 3.82Question 4: I feel my opinion was considered.Average rating : 3.91Question 5: The practice exercises helped me understand the process.Rating scale:1 = strongly disagree2 = disagree3 = agree4 = strongly agree
20 Some Feedback from SMEs Staff was wonderful about explaining the process of standard setting. It was a great experience.Great experience. The exercise validated the process of the cut score.This was a wonderful opportunity to be able to participate in this workshop. I look forward to participating again soon in other projects.From a state evaluator’s and nurse educators perspective, the workshop was relevant in understanding the concepts and dynamics involved in establishing a minimum for safe medication administration of entry level certified medication aides.
21 (con’t) SME FeedbackExcellent process of standard setting! Everyone’s opinion was valued and respected. Thorough explanations.It was a great process! Invite me back!Thank you for allowing me to participate in this process. I see and better understand how the NCSBN works to keep the public safe.I enjoyed the experience and feel the process was very fair and accurate.Very informative and enjoyable! Staff was most helpful. Thanks for the opportunity.
22 2010 Medication Aide Job Analysis Proposed Timeline and Next Steps October 2010 – August 20111. Develop methodology plan2. Host SME workshop3. Develop surveys and correspondence4. Mail pre-survey notice and surveys5. Collect data6. Analyze data7. Conduct nonresponder study8. Finalize report9. Publish research brief10. Host SME Test Specification workshop
23 2010 Medication Aide Knowledge Survey Proposed Timeline and Next Steps April 2011 – November 20111. Develop methodology plan2. Host SME workshop3. Develop surveys and correspondence4. Mail pre-survey notice and surveys5. Collect data6. Analyze data7. Conduct nonresponder study8. Finalize report9. Publish research brief
24 How you can assist in the 2010 Medication Aide Surveys No National RegistryChallenges in obtaining mailing addresses for a nationally representative sampleExamples from other nursing survey studies:Nurse Aide 2010 JA and KSA Surveys – Randomly selected 6,500 nursing directors from hospitals, nursing homes and long term care facilities based on a facility mailing listNCLEX-RN 2008 practice analysis – Randomly sampled 6,000 NCLEX-RN candidates who passed the examination from January 1 to April 13, 2008.
25 MACE PublicationsMACE Item Writing and Review Workshop Report - https://www.ncsbn.org/MACE_Item_Writing_Workshop_Report.pdfMACE Standard Setting Workshop Report - https://www.ncsbn.org/Standard_Setting_Report_for_MACE.pdfMedication Assistant-Certified (MA-C) Model Curriculum - https://www.ncsbn.org/07_Final_MAC.pdfQuick Reference to MA-C Model Curriculum - https://www.ncsbn.org/07_new_MACChecklist.pdf