Test Blueprinting for Standard and Concept-Based Curricula ATI Education Summit 2017 Lois Churchill, MN, RN Dianna Johnston, DNP, RN, NE-BC

Objectives Explain the benefits of using test blueprinting. List the steps in test blueprinting development. Explain how to set the specifications for the elements at the course and program levels. Explain how to plan for remediation activities after conducting an item analysis.

Purpose of Test Blueprints Tool used to determine parameters of a test prior to its development. Program blueprints outline by semester the percent of each element on tests delivered during the semester. Course blueprints outline the percent of each element on course tests.

Steps in Test Blueprint Development Identify learning elements to be measured Determine percent of items for each difficulty level (Bloom/Anderson & Krathwohl) NCLEX® Content Area Nursing Process QSEN BSN Essentials Massachusetts Nurse of the Future Determine the weight to be assigned to selected elements Program Course Conduct item analysis 55

Percent of Items for Each Cognitive Level Bloom’s Taxonomy - Knowledge, Comprehension, Application, Analysis, Synthesis, Evaluation (Bloom, 1956) Anderson & Krathwohl’s Taxonomy - Remember, Understand, Apply, Analyze, Evaluate, Create (Anderson & Krathwohl, 2001) Foundational Level Remember Understanding Critical Thinking / Clinical Judgment Apply Analyze

RN NCLEX® Test Plan 2016 Updated every 3 years Based on practice analysis (First 12 months of practice) Client Needs Percentage of Items from Each Category/Subcategory Safe and Effective Care Environment Management of Care 17-23% Safety and Infection Control 9-15% Health Promotion and Maintenance 6-12% Psychosocial Integrity Physiological Integrity Basic Care and Comfort Pharmacological Therapies 12-18% Reduction of Risk Potential Physiological Adaptation 11-17%

PN NCLEX® Test Plan 2017 Updated every 3 years Based on practice analysis (First 12 months of practice) Client Needs Percentage of Items from Each Category/Subcategory Safe and Effective Care Environment Coordinated Care 18-24% Safety and Infection Control 10-16% Health Promotion and Maintenance 6-12% Psychosocial Integrity 9-15% Physiological Integrity Basic Care and Comfort 7-13% Pharmacological Therapies Reduction of Risk Potential Physiological Adaptation

Obtaining Information about NCLEX ® Test Plans The detailed test plan for item writers/item reviewers and nurse educators can be accessed at the web sites below: https://www.ncsbn.org/RN_Test_Plan_2016_Final.pdf https://www.ncsbn.org/10136.htm- PN 2017 Detailed Test Plan

Level of Difficulty Bloom’s Taxonomy Knowledge Comprehension Application Analysis Anderson & Krathwohl’s Taxonomy for Learning Remember Understand Apply Analyze Evaluate Create Foundational Thinking vs. Critical Thinking

Nursing Process

QSEN Competencies Quality and Safety Education for Nurses Competencies Program with goal of preparing future nurses KSA necessary to improve quality of care- 6 competencies Massachusetts Nurse of the Future integrates the QSEN competencies and IOM recommendations Quality and Safety Education for Nurses was an initiative to prepare nurses with the competencies needed to provide quality care. Robert Wood Johnson Foundation supported this nationwide project. This project incorporates the institute of medicine competencies. The AACN includes the IOM competencies of the BSN Essentials. Billings and Halstead p 28. IOM competencies: patient centered care, collaboration with interdisciplinary teams, awareness of quality standards, knowledge and use of evidence-based practice and informatics and technology. Billings & Halstead p 336 Nurses play key role in protecting client safety and proving quality care. AACN essential core competencies: critical thinking, health care systems and policies, communication, illness and disease management, ethics, information and health care technologies. Billings and Halstead p 99

AACN Essentials of Baccalaureate Education for Professional Nursing Practice American Association of Colleges of Nursing An autonomous arm of AACN, the Commission on Collegiate Nursing Education (CCNE) ensures the quality and integrity of baccalaureate graduates Quality standards

AACN BSN Essentials (continued)

Massachusetts Nurse of the Future Core Competencies© Competency Definition Patient-Centered Care The Nurse of the Future will provide individualized care that recognizes patient’s preferences, values and needs and respects the patient or designees as a full partner in providing compassionate, coordinated, appropriate, safe care. Professionalism The Nurse of the Future will demonstrate accountability for practicing nursing within established moral, legal, ethical, regulatory and humanistic principles. Leadership The Nurse of the Future will influence the behavior of individuals within their environment in a way that will facilitate the establishment and acquisition/achievement of shared goals. Systems-Based Practice The Nurse of the Future will demonstrate an awareness of and responsiveness to the larger context and system of health care and the ability to effectively call on system resources to provide care that is of optimal quality and value. Informatics & Technology The Nurse of the Future will use information and technology to communicate, manage knowledge, mitigate error and support decision-making. Communication The Nurse of the Future will interact effectively with patients, families, and colleagues, fostering mutual respect and shared decision-making, to enhance patient satisfaction and health outcomes. Teamwork & Collaboration The Nurse of the Future will function effectively within nursing and interdisciplinary teams, fostering open communication, mutual respect, shared decision-making, team learning and development. Safety The Nurse of the Future will minimize risk of harm to patients and providers through both individual performance and system effectiveness. Quality Improvement The Nurse of the Future will use data to monitor outcomes and care processes, and use improvement methods to design and test changes to continuously improve the quality and safety of healthcare. Evidence-Based Practice The Nurse of the Future will identify, evaluate and use the best current evidence coupled with clinical expertise and consideration of patient’ preferences, experience and values to make practice decisions. the minimum expectations for all nurses as they complete their pre-licensure education The ten essential competencies, which guide nursing curricula and practice all the competencies are of equal importance.

Foundational Thinking Ability to remember and understand basic information Clinical manifestations Expected reference range of laboratory results Causes/risk factors for an alteration in health Expected/side/adverse effects of medication or treatment Medication purpose/mechanism of action

Critical Thinking/Clinical Judgment Ability to apply knowledge and use higher level critical thinking skills (assess, analyze, and evaluate) when making decisions about clinical problems a client scenario poses. Define the problem accurately Make inferences based on data Identify and examine alternative solutions Recognize assumptions and faulty reasoning Develop a plan of action Prioritize nursing interventions Perform dosage calculation Analyze new information Draw conclusions about a plan’s effectiveness Revise a plan based on evaluation

Sample Program Test Blueprint   Sample Program Test Blueprint Cognitive Level % NCLEX © Category % Nursing Process % Year Foundational Critical Thinking / Clinical Judgment Management of care Safety and Infection Control Health Promotion & Maintenance Psychosocial Integrity Basic Care & Comfort Pharmacological & Parental Therapies Reduction of Risk Potential Physiological Adaptation Assessment Diagnosis Planning Implementation Evaluation Remember Understand Apply Analyze Evaluate Create Sem1 10 50 40 5 7 3 15 20 25 Sem2 60 12 11 Semester 1 (15 – 20%) (45 – 50%) (40 – 45%) (17-23%) (9-15%) (6-12%) (12-18%) (11-17%) Semester 2 (0 – 5%) (15-25%) (65 – 70%) Avg. 6 13 18

Leveling Elements Establish a range for the percent of items in each element Avoid using exact numbers Assign the range of percent based upon Placement of course in the curriculum Course objectives Degree of target difficulty Place increased weight on higher level processes as students progress through the curriculum (Oermann and Gaberson, 2014)

Test Blueprinting- Traditional vs. Concept-based Curriculum Traditional Curriculum Leveled by course Content is body system focused Concept-based Curriculum Leveled through the curriculum across the lifespan Content is concept-focused Leveling Elements are the same for either curriculum type

Example: Weight of Level of Difficulty/Program Level Recommended range (weight) for the percent of items in each difficulty level by program level Example: (The above percents are for example only and are not recommendations.)

Linking Test Items to the Syllabus Use one of the following to link items on test to syllabus Course objectives Unit objectives Course content

Number of Alternate Items NCSBN has been using alternate item format questions on NCLEX®-RN/PN since 2003. Alternate items give students the opportunity to demonstrate competence in alternative ways such as calculations, identifying graphics. Choice of item format should be based on content being tested. Number of alternate items should depend on the length of the test. Some experts recommend limiting type of alternate format items to no more that 3 per test. (Oermann and Gaberson, 2014)

Key Components of Item Analysis Item Difficulty: Is the item too hard or too easy? Item Discrimination: Who is answering the item correctly?

Using Data from Item Analysis to Plan for Content Review- The p value p value ( also known as difficulty level) is the per cent of students who chose the correct answer. The goal is a p value of 30 to 80% for majority of the items on the test. For purposes of content review, any item with a p value of 50% or less should be reviewed. (A p value of 50% means that half of the students who took the exam chose the incorrect answer)

Using Data from Item Analysis to Plan for Content Review- The Point Biserial Index The point biserial index is an indication of the quality of an item. Can range from -1.00 to +1.00. It tells us whether the high-ability students or the low ability students choose the key. We want a positive point biserial for the key. That means high ability students chose the key. Goal is a 0.30 for the key. For purposes of content review, any item with negative point biserial for the key should be reviewed.

Hint Write items as you prepare for class. Always write rationales for both the key and distractors before you administer the test.

Steps in Item Analysis for Content Review Ask a peer for input about the quality of the item. Two sets of eyes are better than one. Peer Review Can you link it to a course or unit objective? Is it entry-level practice? Can you link it to the NCLEX© test plan? Can you identify the step of the nursing process? For concept-based curricula, is the item at the proper level of difficulty inline with your curriculum? If you have used the item with a prior group of cohorts, did the prior stats meet the acceptable ranges? Relevance Is the stem free of extraneous content? Is the question in the stem clear? Is there only one correct answer? Item Quality

Content Remediation Policies on content remediation should be incorporated into the program’s testing policies. Explore with students either individually or in a group their understanding of the content/concept. Assign activities for review. Retest the content/concept if possible.

Summary and Final Thoughts Questions and comments Plans for moving forward Aha moments

References American Association of Colleges of Nursing (AACN), 2008 The essentials of baccalaureate education for professional nursing practice. Retrieved from http://www.aacn.nche.edu/educational-resources08. Anderson, L. W., & Krathwohl, D. R. (2001). A taxonomy for learning, teaching, and assessing: A revision of Bloom’s taxonomy of educational objectives. Boston, MA: Pearson Education Group. Billings, D., & Halstead, J. (2016). Teaching in nursing: A guide for faculty (5th ed.). St. Louis, MO: Saunders. Bloom, B.S., & Krathwohl, D. R. (1956). Taxonomy of educational objectives: The classification of educational goals, by a committee of college and university examiners. Handbook I: Cognitive domain. New York, NY: Longmans, Green.

References (Continued) Massachusetts Nurse of the Future Nursing Core Competencies© Retrieved from www.mass.edu/nursing National Council of State Boards of Nursing. (2016). NCLEX-RN® Examination: Test plan for the National Council licensure examination for registered nurses. Retrieved from https://www.ncsbn.org/RN_Test_Plan_2016_Final.pdf National Council of State Boards of Nursing. (2017). NCLEX-PN® Examination: Test plan for the National Council licensure examination for nurses. Retrieved from https://www.ncsbn.org/PN_Test_Plan_2017_Educator_v2.pdf Oermann, M., & Gaberson, K. (2014). Evaluation and testing in nursing education. (5th ed.). New York, NY: Springer. Quality and Safety Education for Nurses Institute (2013). Competencies. Retrieved from http://qsen.org/compentencies/