1. PAMELA S. DICKERSON, PhD, RN-BC, FAAN STEPHANIE CLUBBS, MSN, RN-BC, CNS Being A Nurse Planner: What Does It Mean? Part 2: Activity Assessment, Planning, and Evaluation 1

2. Disclosures Purpose/Outcome: Participants will effectively function as nurse planners in approved provider units. Criteria for Successful Completion: Participate in the entire webinar. Complete the evaluation and registration information and return to ONA as directed. Conflict of Interest: There is no conflict of interest for any planner or presenter for this activity. 2

3. Overview of the Series Part I:  Roles, responsibilities, accountability  The approved provider unit  The National accreditation system Part II:  Planning learning activities Part III:  Dealing with challenging situations  Learning activities in context with the provider unit 3

4. Session 2 Objectives Examine processes for an educational activity: Assessment, planning, and evaluation. 4

5. Your Experience What is your length of experience as a nurse planner? Less than one year 1-3 years 3-8 years More than 8 years 5

6. Your Comfort Level How comfortable do you feel in your nurse planner role? Not at all Somewhat Very 6

7. Areas of Discomfort If you answered somewhat or not at all for the previous question, what is your major area of discomfort? Leading a planning committee Knowing what to document Knowing what forms to use Contributing to provider unit evaluation 7

8. Your Provider Unit How many nurse planners are there in your provider unit? I’m it! 2-5 6-10 11 or more 8

9. Quality in Your Educational Activities 9

10. THE PROCESS BY WHICH A DISCREPANCY BETWEEN WHAT IS DESIRED AND WHAT EXISTS IS IDENTIFIED Assessment Of Learning Needs 10

11. Needs Assessment specific to your target audience  Why do they need this content? (what evidence do you have?)  Who are the expected learners? (target audience)  When do we need it? (tomorrow afternoon?) Gap Analysis  What’s the difference between where the learners are now, and where they need to be?  Knowledge, skills, practice Determination of desired learner outcome  What will be the specific end result? What will learners walk away with? (should impact the following):  Professional development for nurses  Quality of patient care Quality in Your Educational Activities: Needs Assessment EDP 1, 2 11 Provider Manual - Chapter 3: Educational Design Process

12. Evidence to support the need for your learning activity  How do you know that learners need this information?  List sources of information, briefly describe relevance – this information is part of your activity file What Is Evidence of a Learning Need and How Do You Document It? EDP 1 12

13. National Sources: NIH, CDC, Accrediting Bodies, CMS, NDNQI State Sources: Health Departments, Legislative Initiatives Learner Specific Sources: Performance/quality improvement initiatives, dashboard data, incident reports, organizational mission/goals, annual survey, requests (phone, verbal, e-mail, evaluation summaries) Example 13

14. © 2012 American Nurses Credentialing Center Identification of a Gap Knows: learner has knowledge about the topic/subject Knows how: learner is capable of applying the knowledge Shows how/does: learner is able to apply knowledge and skills in a simulated setting (shows how) or the practice environment (does) ► Consider info about the gap to determine your outcome Miller’s Model of Clinical Competence 14

15. Example – Today’s program Needs Assessment Methods:  Learner requests  Program evaluation comments  Annual provider survey  Evaluation of provider apps submitted for review 15

16. Example – Needs Assessment Evidence from the Needs Assessment:  Representatives of provider units or those wanting to become provider units have expressed desire for more education on various topics related to the ANCC criteria  What criteria mean  How to implement the criteria  Evidence from provider applications using 2013 criteria shows difficulty in narrative documentation  Particularly in the areas of: needs assessment, gap analysis (Part 2), bias/conflict of interest, and quality outcomes (Part 3). 16

17. Example - Gap Gap Identified:  Gap in knowledge related to documentation expectations and components of effective educational design/evaluation  Gap in skill in incorporating all educational design components to achieve quality educational activities  Gap in practice related to documentation with clear evidence to support the needs of learners and desired outcomes of quality activities 17

18. Example - Outcome Outcome for the Learner:  Participants will effectively function as nurse planners in approved provider units.  Outcome measure: 75% of provider applications completed by people who attended the webinar series will include adequate activity file evidence to demonstrate quality and integrity in their work 18

19. Planning the activity 19

20. What is the expected outcome of the activity?  How will you achieve this? How will you develop/determine:  Objectives?  Content?  Teaching Methods? Quality in Your Educational Activities: Planning EDP 7, 8, 11 20

21. Selection of faculty/authors  Who and Why? Qualifications? Selection of objectives / content / teaching methods  Objectives must be measurable and what the learner is expected to achieve  Content and teaching methods need to support your being able to meet the objectives Determining criteria for successful completion  What will you do, and why?  How will this support your desired outcome? Quality in Your Educational Activities: Planning EDP 6, 7, 11 21

22. Evidence that content for a learning activity is based on best available information – How do you know what resources faculty are using to develop an educational activity? – Has your planning committee ever provided (or suggested) references or resources to your faculty? – Has your planning committee ever shared evidence from needs assessment findings with faculty? How does your activity file reflect this evidence? Content - What Is Evidence, and How Do You Document It? EDP 8 22

23. Information from an organization/website (ex Agency for Healthcare Research & Quality, CDC, NIH…)  Should be current (w/in 5-7 yrs)  Can be published or unpublished Peer-reviewed journal/resource (w/in 5-7 yrs) Clinical guidelines (ex www.guidelines.gov)www.guidelines.gov Expert resource (ex individual, organization, educational institution, book, article, website)  Listing speakers name is not sufficient Textbook Some Choices for Content Evidence 23

24. Ensuring Content Integrity Factors to Consider  Actual or potential conflicts of interest are resolved  Content is based on best available evidence  Independence from any organization providing funding for the activity (commercial interest organization or sponsor – cannot be involved in any aspect of the planning/development of the education)  Free from promotional activity  Have you ever used a content reviewer? What is the purpose of a content reviewer? 24 EDP 9

25. Who needs to be on the planning committee? How many? How do you select members? Why? How do you communicate? Planning Committee EDP 3 25

26. An employee of a commercial interest entity may never be a planning committee member – too much opportunity to control the activity An employee of a sponsor organization may never be a planning committee member BUT you may need to think about the role of this person/organization: is it really sponsorship, or is it perhaps co-providing? When Should A Person NOT Sit On A Planning Committee? 26

27. Disclosures Outcome or objectives Criteria for successful completion Presence or absence of COI – all planners, presenters, faculty, authors, content reviewers  IE – anyone that can influence the content Commercial support – if applicable Non-endorsement of products – no longer relevant, not needed Must be provided to learners prior to the start of an activity 27

28. Criteria for Successful Completion  What will you use?  Why? (hint: what was your gap and outcome?) Verifying Participation with Unique Identifier Planning for Evaluation  What will you use?  Why? (again – think about gap and outcome)  How will the information be used? Planning for the Evaluation (During Planning) EDP 6, 12 28

29. Evaluating the activity 29

30. Verifying Participation Common methods: sign-in sheet, computer log-in Need: name of learner and unique identifier Challenging Situations:  Conferences where people attend different sessions; may earn different number of contact hours  People who sign in but don’t meet criteria for successful completion 30

31. Why Evaluate? Collect data to determine whether you met the needs of learners you identified in your assessment process Collect data about other factors relevant to this group of learners Collect data about learning outcomes Decide what you want to do with this activity in the future:  Offer it again  Modify it  Be done! 31

32. Types of Immediate Evaluation Questionnaire to learners Post-Test Return Demonstration Case Study demonstrating application of knowledge Other 32

33. Wording Evaluation Questions What’s wrong with these questions?  Was this objective met?  Was the faculty effective?  Will you be able to apply what you learned? 33

34. Generating A Summative Evaluation Why?  Capture the aggregate perspectives of learners  Get a “bird’s eye” view of the value of the activity How?  Compile numerical data; report on one form  Compile comments – list each comment; summarize by category (# of comments on application in practice, for example) Note: This is required and must be retained in your activity file 34

35. Summative Evaluation Sample ExcellentGoodModeratePoorN/A Item 1152100 Item 2135000 Item 381009 Item 462109 Speaker 1153000 Speaker 2162000 35 Comments: First session was too long Speaker one was fabulous! Wish we had more time for this session I would like to have attended both breakout sessions – can the schedule be changed for next year?

36. Types of Long-Term Evaluation Surveys to learners 3-6 months after the learning activity Observation of learners in action subsequent to the learning activity Data from other sources: nurse managers, quality / risk management data, patient outcomes, NDNQI indicators, etc. 36

37. Why Long-Term Evaluation? Did the learning make a difference? There is no way to know this at the conclusion of the learning activity. Is the provider unit effective in meeting its goals and following through on outcome measures? Is the provider unit contributing to the overall mission and goals of the organization? 37

38. How Are We Doing? Compared to before you started this webinar series, is your comfort level as a nurse planner: Higher About the same Lower 38

39. Planning for Next Session Our third webinar in this series addresses challenging situations in the nurse planner role. What is your most challenging situation? Documenting evidence of needs assessment and gap analysis Addressing conflict of interest and/or bias Dealing with commercial support or sponsorship Co-providing Evaluating outcomes of learning 39

40. References and Resources ONA Provider Manual / MNA Provider Information ONA web site: www.ohnurses.orgwww.ohnurses.org MNA web site: www.mtnurses.orgwww.mtnurses.org American Nurses Association (2010). Nursing Professional Development Scope and Standards of Practice. Silver Spring, MD. Author. 40

41. Thank You Questions? 41

42. Contact Information Ohio Nurses Association:  Zandra Ohri, MA, MS, RN  zohri@ohnurses.org zohri@ohnurses.org  614-448-1027 Montana Nurses Association:  Pam Dickerson,PhD, RN-BC, FAAN  pam@mtnurses.org pam@mtnurses.org  406-465-9126 42