1 Ada Woo, PhD Senior Psychometrician, NCSBN NNAAP™ (National Nurse Aide Assessment Program): Job Analysis, Standard Setting and Next StepsAda Woo, PhDSenior Psychometrician, NCSBN
2 Outline Overview of test development process 2009 nurse aides job analysis2010 NNAAP standard setting
3 NNAAP Test Development Process Job AnalysisContent OutlineItem Writing and Item ReviewSubject Matter Expert (SME) Panel ReviewEditorial ReviewBuilding Test FormsSetting the Passing StandardPretesting of ItemsAdministering Examinations
4 Job Analysis Process“Evidence of validity based on test content requires a thorough and explicit definition of the content domain of interest… characterization of the domain should be based on job analysis.”(AERA, APA, NCME, Standards for Educational and Psychological Testing, 1999, p.160)
5 Background on Job Analysis Assist NCSBN in describing range of authorized duties and evaluating test plans.Provide validity evidence to support the examination.Conducted at least every five years.Large scale studies using multiple methods.
6 Methodology Document review Review of entry-level CNA daily logs Review of documents specific to CNA work settingsLarge scale surveys to NAEs and entry-level CNAsNon-responder studyMethodology review by external psychometricians
7 Survey Instruments NA survey NAE survey Work environment 115 NA activity statementsLast day of workEducation and trainingDemographicCommentsNAE surveyWork environment115 NA activity statementsComments
8 Sample SelectionRandomly selected 6,500 nursing directors from hospitals, nursing homes and long term care facilitiesEach director received three surveys to be distributed to one NAE and two entry-level CNAsReturn rates:485 facilities opted out of the study21 return surveys due to bad addressesCNA adjusted return rate – 10.7% (n=1,016)NAE adjusted return rate – 11.1% (n=666)
16 Most Performed Activities Activity1 Year or less NAActivity #N% Not Perform Activity% Perform Activity107Ask for help when needed5510.00%100.00%98Respect client's need for privacy/confidentiality5500.20%99.80%19Follow Standard/Universal precautions (e.g., handwashing, personal protective equipment [PPE], isolation guidelines)55756Allow client to do things at his/her own pace5530.40%99.60%93Follow client's plan of care5520.50%99.50%ActivityNurse Aide EvaluatorActivity #N% Not Perform Activity% Perform Activity98Respect client's need for privacy/confidentiality6430.00%100.00%19Follow Standard/Universal precautions (e.g., handwashing, personal protective equipment [PPE], isolation guidelines)660102Follow code of ethics for nurse aides642107Ask for help when needed0.20%99.80%29Use proper body mechanics656
17 Least Performed Activities Activity1 Year or less NAActivity #N% Not Perform Activity% Perform Activity26Perform cardiopulmonary resuscitation (CPR)55148.30%51.70%7Assist client to fill out meal menu55446.90%53.10%23Clear foreign-body airway obstruction (FBAO)54542.60%57.40%39Prepare client for diagnostic test, procedure, or surgery (e.g., NPO)41.50%58.50%46Collect and label stool, urine or sputum specimens55237.30%62.70%ActivityNurse Aide EvaluatorActivity #N% Not Perform Activity% Perform Activity31Apply and monitor restraints65747.50%52.50%7Assist client to fill out meal menu65240.60%59.40%51Perform and record pulse oximetry65640.50%59.50%26Perform cardiopulmonary resuscitation (CPR)65434.10%65.90%23Clear foreign-body airway obstruction (FBAO)64830.10%69.90%
19 Most Important Activities Activity1 Year or less NAActivity #NAvg. ImportStd. err.19Follow Standard/Universal precautions (e.g., handwashing, personal protective equipment [PPE], isolation guidelines)5564.950.0122Apply and respond to client safety alarms5374.91104Report any suspected neglect, mistreatment or abuse4994.90.02103Report and record unusual incidents (e.g., errors, injuries, falls)52094Follow Health Insurance Portability and Accountability Act (HIPAA)5194.87ActivityNurse Aide EvaluatorActivity #NAvg. ImportStd. err.19Follow Standard/Universal precautions (e.g., handwashing, personal protective equipment [PPE], isolation guidelines)6604.930.01104Report any suspected neglect, mistreatment or abuse6354.91103Report and record unusual incidents (e.g., errors, injuries, falls)6324.850.0222Apply and respond to client safety alarms6284.8494Follow Health Insurance Portability and Accountability Act (HIPAA)6294.83
20 Least Important Activities Activity1 Year or less NAActivity #NAvg. ImportStd. err.5Provide nail care4283.810.051Provide foot care4183.837Assist client to fill out meal menu2943.840.0661Assist or encourage family to help with client's care418Provide physical comfort measures (e.g., back rubs)5134.050.04ActivityNurse Aide EvaluatorActivity #NAvg. ImportStd. err.34Assist nurse with dressing change5073.450.057Assist client to fill out meal menu3873.580.0661Assist or encourage family to help with client's care5593.785Provide nail care6023.790.0458Assist client with recreational activities5993.9
22 Conclusions and Next Steps Respondents found activities listed in the surveys to be representative of CNA work.Importance ratings given by CNAs and NAEs were similar.Results obtained from job analysis surveys were used to determine test plan and weighting of test plan areas.
24 Reports and Technical Briefs NNAAP 2009 Job Analysis Report (available on NCSBN website this summer)2011 NNAAP Test Plan Report is available for download at https://www.ncsbn.org/1721.htm
25 NNAAP Test Development Process Job AnalysisContent OutlineItem Writing and Item ReviewSubject Matter Expert (SME) Panel ReviewEditorial ReviewBuilding Test FormsSetting the Passing StandardPretesting of ItemsAdministering Examinations
26 Standard SettingTo determine a point on the ability continuum by which a candidate must exceed to be deemed competentSpecific to the NNAAP examination, the goal of standard setting is to determine the level of nursing ability necessary for certified entry-level nurse aides to practice safely and effectively.Passing standard = passing score = cutscore
27 (con’t) Standard Setting “Where the results of the standard-setting process have highly significant consequences those responsible for establishing the cut scores should be concerned that the process by which cut scores are determined be clearly documented and defensible.”(AERA/APA/NCME, Standards for Educational and Psychological Testing, 1999, p. 54)
28 Assumptions for Standard Setting Certification and licensure is based on an assessment (usually an exam) that is content-valid, scalable and reliable.We can create a defensible process that can quantify "how much" knowledge or skill a person must have to be considered minimally competent.
29 Scores on NNAAP Exam Type of score is very important concept Candidates either "pass" or "fail” their exams.The NNAAP uses a criterion-referenced standard setting process through the definition of minimally competent candidate.We are not asking which candidates were in the top XX% on the exam. We are only concerned with the pass or fail decision.
30 Importance of Standard Setting Certification and licensure examinations are designed to:Protect the public from unsafe practitionersEnsure "best practices" of the professionRequire entry-level professionals to achieve a minimal level of knowledge or skill in their field to perform their job safely and effectively.
31 (con’t) Importance of Standard Setting If standard is set too low, we will certify some candidates who are unqualified.If standard is set too high, we will deny a certification to some candidates who are qualified or competent.Set appropriate standard while referencing the criterion.Briefly mention criterion minimally competent candidates…
32 Important Caveats No “true” value exists Policy decision, related to mission of organization (e.g., public protection)Standards should be revisited when test specifications, scope of practice/range of authorized duties, education programs, and/or other factors change
33 NNAAP Standard Setting Process Overview of standard setting processSMEs discussed qualifications for entry-level NAsSMEs reviewed NNAAP content outlineTraining on standard setting methodologySMEs rated practice itemsSMEs provided round 1 ratings of actual itemsSMEs discussed round 1 ratingsSMEs provided round 2 ratingsGather validity evidenceDetermine cutscore
34 Subject Matter Expert (SME) Panel Representative of practice settings and specialties, regionsKnowledgeable regarding practice analysis, test specificationsAll panelists are RNs with minimum of two years of nursing experienceAll panelists work with CNAsTen SMEs served on the panel of judges
35 SME Panel Characteristics #%NCSBN Geographic AreaI330%II220%IIIIVYear of Post Licensure Experience1 to 15 years16 to 30 years770%31 to 45 years110%Work SettingNursing homeHospitalOther long term careCertifications and Licenses Held (other than RN)*C NA880%LPN/VN440%* Several panelists hold multiple licenses and certifications.
36 The Minimally Competent Candidate To establish a passing standard, we focus on the examinees that possess “just enough” knowledge, skills and ability to practice entry-level nursing safely and effectivelyWe don't focus on high or low ability examineesWe focus on those who are "on the fence"
37 The Minimally Competent Candidate What distinguishes candidates with "just enough" knowledge, skills and ability from the other two groups?Knowledge/skill of the fundamentalsLittle advanced knowledgeHave learned enough to move on (no remedial learning required)“Should” vs. “would”
38 Modified-Angoff Method Rate how minimally competent entry-level CNAs would perform on each NNAAP item.Criterion-referencedCommon in medical and health-related credentialingVast research base, defensibleUsed in NCLEX-RN, NCLEX-PN, MACE and English proficiency standard setting workshops
39 Minimal Requirements for Entering the NA Profession Training program is accepted by state boards of nursing and departments of health servicesNurse aide has minimum 75 hours of class room trainingNurse aide has minimum 16 hours of clinical trainingNurse aide is at least 16 years old and in good mental healthNurse aide works in the following settings: long term care, acute care, assisted living or community health centers
40 Results from Standard Setting Workshop Form and Round% Cut ScoreRaw Cut Score (60 item test)Standard DeviationRating ReliabilityForm A Round 180.34%48.20 (49)0.0040.81Form B Round 181.34%48.80 (49)0.0030.77Form A Round 281.12%48.67 (49)0.79Form B Round 281.87%49.12 (50)0.0020.78Note: Ratings from outlier were removed from the above results.
41 Panel Recommendations SMEs reviewed results of standard setting workshop and their likely impact on NNAAP pass rate.Upon reviewing the information including SMEs’ own knowledge about the industry, all 10 SMEs agreed to a cut score of 47 out of 60 items.This new cut score will be implemented with the launch of 2011 NNAAP test forms.
42 Post-workshop Survey Rating Scale: QuestionMean RatingStandard DeviationQuestion 1: I understood the major goals of the standard setting workshop.3.80.42Question 2: I feel the training was adequate.Question 3: I feel that the discussion of the minimally competent nurse aide candidate was comprehensive.3.90.32Question 4: I understood the group consensus on a minimally competent nurse aide candidate.Question 5: I understood my task throughout the standard setting process.Question 6: I understood the Modified Angoff rating process.3.40.52Question 7: I understood the Beuk rating process. *2.90.57Question 8: I had sufficient time to reflect on my first rating.Question 9: I had sufficient time to reflect on my final rating.Question 10: I feel the group discussions were productive.4.00.00Question 11: I feel my opinion was considered.Question 12: I feel the passing standard recommended by the group was appropriate.Rating Scale:1=Strongly disagree, 2=Disagree, 3=Agree, 4=Strongly agree
43 SME Feedback“I enjoyed the opportunity to meet other NA Educators from different states and be able to understand the variety of challenges each of us face. I feel honored to be a part of this process.”“Supportive, well informed group leaders, appreciative of our input, made the process understandable. I would gladly come back anytime.”“I learned a great deal about the process. I was very impressed with the professionalism and knowledge of the presenters. This is a very comprehensive process and I am honored to be a part of it.”
44 Next Steps 2011 NNAAP forms will be launched in January 1, 2011. These test forms were constructed according to test specifications generated from the 2009 job analysis of nurse aides.All test forms were equated to be comparable in difficulty. Passing score for all 2011 test forms is 47 out of 60.A larger number of items will be pretested in 2011 to build NNAAP item pool.Next NNAAP item writing/review workshop will be hosted in March 2011
45 Reports and Technical Briefs Test Plan Report –https://www.ncsbn.org/2011_NNAAP_Test_Plan-Test_Specifications_Report_links.pdfStandard Setting Report – complete report will be mailed to clients; proprietary information.