1. The Quest for Quality: Monitoring and Measuring Outcomes in Continuing Nursing Education Session 3: Monitoring and Measuring Outcomes for Provider Unit Effectiveness Pamela S. Dickerson, PhD, RN-BC, FAAN Director of Continuing Education, Montana Nurses Association President, PRN Continuing Education Presented by Ohio Nurses Association and Montana Nurses Association

2. Disclosures
Purpose: Participants will effectively develop learning and provider unit processes that lead to measurable outcomes demonstrating quality patient care and/or enhanced professional development of registered nurses.
Criteria for Successful Completion: Participate in the entire webinar. Complete the evaluation and registration information and return to ONA as directed.
Conflict of Interest: There is no conflict of interest for any planner or presenter for this activity.

3. Objective
Examine approaches to data collection and analysis related to the effectiveness of a provider unit in improving patient care and enhancing nursing professional development.

4. Reflection
If you have participated in sessions 1 & 2, please rate your current ability to do the following:
Strong = 3
Moderate = 2
Weak = 1
Conduct a needs assessment
Analyze a gap
Measure an activity outcome
Write a provider unit outcome
Your Score:

5. Reflection Data
My score is closest to: 4 7 9 12

6. Self-Assessment Your Score = I know how to Strong = 3 Moderate = 2
Weak = 1
Develop goals and outcome measures for my provider unit
Engage stakeholders in provider unit evaluation
Evaluate provider unit data
Measure provider unit effectiveness in relation to improved patient care or enhanced professional development of registered nurses
Your Score =

7. My total score was closest to a:4 7 9 12

8. Educational Design Process
Accreditation Conceptual Framework ANCC 2013 – used with permission
Educational Design
Process
Assessment of Learning Needs
Planning and Faculty
Design Principles
Achievement of Objectives
Structural Capacity
Commitment
Accountability
Leadership
Resources
Nursing Professional Development: Continuing Nursing Education
Quality Outcomes
Evaluation Process
Evaluation Participation
Goals for Improvement
Value/Benefit to Nursing
Professional Development

9. Provider Unit Outcomes Measures
Outcome: “The impact of structure and process on the organization…and the value/benefit to nursing professional development.” (ANCC, 2013).
Outcome Measurement: “The process of observing, describing, and quantifying predefined indicators of outcomes of performance.” (ANA/NNSDO, 2010)

10. Evaluation in Context Outcome Measurement for Provider Unit
Goals for Provider Unit
Outcome Measures
Provider Unit Structure & Function
Learning
Activities
Activity Evaluation
Outcome Measurement for Provider Unit
PSD, 2013

11. Why Does Your Provider Unit Exist?
Single- focused organization:
Why is continuing education important enough to make it your business?
What does “success” for your organization look like?
Multi-focused organization:
Why is continuing education part of what your organization does?
How does continuing education benefit the organization?

12. Let’s Look at the Organization
What is its mission?
What are its goals?
Who are its stakeholders? (Who cares about the organization, its success, and its functions?)

13. Your Provider Unit What are the goals for your provider unit?
How do your goals match those of organization? (If they don’t, it’s time for some serious reassessment.)

14. Think Like A Reporter
Who
What
Why
Where
How
When

15. What Is The Best Time To Evaluate Your Provider Unit?
Quarterly
Annually
When requested by administration
When the provider application is due

17. What is the best way to frame Provider Unit Evaluation?
How effectively do our structure and processes support our efforts in achieving our goals?
Did we accomplish what we planned in our learning activities as a whole?
Do we have administrative support and funding for our next provider application?
When is our next provider application due?

18. Evaluating Your Provider Unit: Who
Structural Capacity
People
Right mix of people?
Right qualifications of people?
Enough people?
People doing the right things?
Do people know how to do what’s needed as the system evolves?

19. Evaluating Your Provider Unit: What
Structural Capacity: Physical and Material Resources
Office space
Office supplies
Classroom space
Equipment
Up-to-date?
Functional?

20. Evaluating Your Provider Unit - What
Structural Capacity – Financial and Administrative Support
Does primary nurse planner have authority to ensure adherence to criteria?
Is there administrative support for the work of the provider unit?
Are there financial resources for support of structure and processes of the provider unit as well as the educational activities?

21. Evaluating Your Provider Unit - When
It depends…
Some factors may need to be evaluated monthly or quarterly
It may take time to collect data for some outcomes measures, ranging from several weeks to a year or more
Organization may require an annual report and annual goals
Provider renewal time is an excellent opportunity for a comprehensive review and analysis

22. Evaluating Your Provider Unit - Where
Within the context of the provider unit itself
Within the context of the organization
Each will require unique perspectives, data, and analyses

23. Evaluating Your Provider Unit – How
Look at components of structural capacity
Look at components of processes – provider unit functions AND educational design processes
Measure against your goals, which are supported by your predetermined outcomes measures
Engage appropriate stakeholders

24. Stakeholders Who are the people who care? Learners Faculty
Provider Unit Nurse Planners & Key Personnel
Organizational Personnel
Administration
Finance
Risk Management
Others
Community representatives
Advisory Committee or Board
Stockholders or owners, Board of Directors

25. Evaluate Your Provider Unit: Why
Prove that you make a difference
Validate your contribution to quality patient care
Demonstrate your commitment to professional development of registered nurses

26. Example: Provider Unit Effectiveness (how stakeholder input helped improve a quality outcome measure)
Outcome Measure: diversity of learning activities.
Provider Unit Goal: Implement at least 5 web based learning opportunities for nurses in our hospital during calendar year 2013.

27. Stakeholder Input
Learners requested opportunity for web-based learning activities they could access via the hospital’s intranet and complete at their convenience.
Managers requested that contact hours be awarded for activities that did not require nurses to be away from the practice setting for the time required to attend classes.

28. Data
In calendar year 2012, we offered 53 continuing education classes for our nurses – all were “real-time” learning opportunities.
In calendar year 2013, we offered 3 web-based independent study activities. Our goal was 5, but the IT person resigned and has not been replaced.

29. Your Thoughts?
Does this evidence address the question of how stakeholder input has helped to improve a quality outcome measure?
Yes No

31. What Could Make It Stronger?
Did learners feel that their needs were met with these 3 activities?
What is the plan to move forward? Need to advocate for IT resources?
What feedback has been received from managers in relation to their original request?

32. Example: Professional Development
Outcome Measure: leadership development
Provider Unit Goal: Enhance effectiveness of charge nurses and managers through a series of leadership classes and organizational initiatives.

33. Background
Reason for this goal: Fits with organizational goal of improving staff satisfaction and retention through better relationships with front-line leadership.
Evidence:
Research demonstrating that new nurses leave 1st positions in significant numbers, primarily due to unsatisfactory relationships with front-line leaders.
In our facility in 2012, 36% of new RNs left their positions within 18 months of hire.
In our facility in 2012, 64% of RN staff rated their satisfaction as moderate to low and indicated relationship with management a key factor

34. Action Plan
Survey staff nurses to get their perspectives on expectations of charge nurses and managers
Survey charge nurses and managers to get their perspectives on their expectations of their roles
Survey nursing administration to get their perspectives on expectations of charge nurses and managers

35. Action Plan
Based on survey results, meet with nursing administration to develop a plan for role clarification, position description revisions, and other organizational initiatives to support more effective functioning of charge nurses and nurse managers. Plan coordinated educational initiatives.
Establish planning committee to develop a series of leadership classes for front-line managers.
Timeline for combined interventions – March through June, 2013

36. Educational Interventions
Six half-day workshops for front-line managers over a three month period of time.
Content from each session built on previous sessions
Content addressed key areas of concern noted in surveys: staff team communication, interprofessional communication, conflict resolution, time management, workflow management, crisis intervention, dealing with bullying/intimidation, feedback & evaluation

37. Outcomes
In the last 6 months of 2013, only 8% of new nurses (defined as within 18 months of hire) left their positions.
In the staff satisfaction survey conducted in October of 2013, 72% of RNs ranked their satisfaction as moderate to high. Anecdotal comments indicated that the concentrated effort to better prepare and support front-line managers was a key factor.

38. Demonstrating the Evidence

39. More Evidence

40. So What?
Has the provider unit contributed to the professional development of the front-line managers?
Has the provider unit contributed to the professional development of ALL nurses in the organization?

41. Leadership: Another Example
A college of nursing provider unit is aware of national data, supported by the IOM report on the future of nursing, that there is increased need for leadership education for registered nurses. A focus group of CNOs from area healthcare facilities supports the need at the local level.

42. Leadership Example
Provider Unit Goal: Increase leadership capability of alumni and other participants in our CNE activities
Outcome Measure:
Professional Development - Leadership

43. Framing Your Plan For Goal Achievement
Use of an evaluation model
Revised Kirkpatrick
Logic
Moore
Have a clear plan in place for what you’re going to do, what resources will be employed, and what outcome measurement data you will collect – begin with the end in mind!

44. Using the Revised Kirkpatrick Model
What outcomes do we want?
What behaviors are required of these nurse to increase their leadership capability?
Where are the gaps? What are current capabilities and what are the desired capabilities?
What KSAs do learners need to fill these gaps?
What is the best learning strategy to address these needs?

45. Using the Logic Model What do we need? (better prepared leaders)
To address this need, we must:
If we do this effectively, the outcome will be:
When leaders implement this outcome, we will see:
When this happens, organizations will be better because:

46. Using The Moore Model
Does the learner know? (is there a knowledge deficit?)
Does the learner know how? (is there a skill deficit?)
Can the learner show that he/she can use the knowledge? (is there a performance deficit?)
Is the learner able to apply the knowledge? (is there a critical thinking/clinical judgment/application deficit in practice?)

47. The Outcome for The Provider Unit
Provider Unit Goal: Increase leadership capability of alumni and other participants in our CNE activities
Provider Unit Outcome Measure: professional development – leadership
Stakeholders: CON leadership, CNO focus group, alumni, other learners in related activities
Where’s the evidence?
Self-report from participants
Data from CNOs regarding leadership evidence in their organizations among nurses who participated in these activities
Number of nurses engaged in leadership roles in practice or professional settings

48. Self-Assessment Your Score: I know how to Strong = 3 Moderate = 2
Weak = 1
Develop goals and outcome measures for my provider unit
Engage stakeholders in provider unit evaluation
Evaluate provider unit data
Measure provider unit effectiveness in relation to improved patient care or enhanced professional development of registered nurses
Your Score:

49. My total score was closest to a:4 7 9 12

51. The Bottom Line
Evaluation of your provider unit helps you identify strengths, contributions, and areas of opportunity.
Providing evidence of accomplishments of your goals and related outcome measures helps you validate your worth to the organization and the nurses you serve.

52. Questions and Discussion

53. For Additional Information
Pam Dickerson, PhD, RN-BC, FAAN
Zandra Ohri, MA, MS, RN

54. References
American Nurses Association & National Nursing Staff Development Organization (2010). Nursing Professional Development: Scope and Standards of Practice. Silver Spring, MD. Nursesbooks.org
American Nurses Credentialing Center. (2013) ANCC Primary Accreditation Application Manual for Providers and Approvers. Revision 3.0. Silver Spring, MD. Author.
Commission on Accreditation, ANCC. (2012). The Value of Accreditation for Continuing Nursing Education: Quality Education Contributing to Quality Outcomes. Retrieved from:
Dickerson, P. (2012). Rocking the boat: challenges ahead for continuing education providers. Journal of Continuing Education in Nursing 43(10),

55. References
Kirkpatrick, D. (1984). Evaluating Training Programs: The Four Levels. San Francisco: Barrett-Koehler.
Kirkpatrick’s Four Level Evaluation Model. (2012). Retrieved from:
Moore, D., Green, J., & Gallis, H. (2009). Achieving desired results and improved outcomes: integrating planning and assessment throughout learning activities. Journal of Continuing Education in the Health Professions 29(1), 1-15.
W.K. Kellogg Foundation. (2004). Logic Model Development Guide. Battle Creek, MI; Author.