3 Program EvaluationThe process of systematically applying research methods to assess program:designimplementationimprovementoutcomesRossi, P. H., & Freeman, H. E. (1993). Evaluation: A systematic approach (5th ed.). Newbury Park, CA: Sage Publications, Inc.
4 Program EvaluationNeeds Assessment: The process of systematically applying research methods to assess needs, gaps, or areas for improvement
5 EXAMPLES OF EVALUATION INTER-PROFESSIONAL GERIATRIC PROGRAMS
7 Example 1: West Virginia University Gerontology Practitioner CertificateNeeds Assessment and Program DevelopmentK. Hash, J. Englehardt, &Amazing Work GroupFunded by: WV Partnership for Elder Living$500o grant from Benedum Foundation/WVLTCP to assess training needs in geriatrics of interdisiclinary professionals
8 Example 1: West Virginia University Objectives:1.Evaluate the aging education needs of social workers and related professionals (with associate degrees and higher) in the state2.Develop a gerontology practitioner certificate and curriculum3.Develop best practices for recruiting and retaining professionals in the field The School began by gathering the interdisciplinary, statewide workgroup in August. A survey would be developed by this workgroup to distribute through various professional mailing lists and electronic listservs, including our own PCE listserv by September. The survey would assess the aging education needs of direct care workers as well as inquire about the best ways to attract and retain quality workers in the field.
9 Example 1: West Virginia University Objectives:1.Evaluate the aging education needs of social workers and related professionals (with associate degrees and higher) in the state2.Develop a gerontology practitioner certificate and curriculum3.Develop best practices for recruiting and retaining professionals in the field The School began by gathering the interdisciplinary, statewide workgroup. A web-based survey developed by this workgroup to distribute through various professional mailing lists and electronic listservs, including our own PCE listserv by September. The survey assessed the aging education needs of direct care workers as well as inquired about the best ways to attract and retain quality workers in the field.
10 Example 1: West Virginia University Online Survey, N= 391Training Needs77% interested in enrolling in a CEU-based certificateTraining needed: cognitive changes/dementia; end of life; legal issues; HC and aging policyGreat interest in online training The majority of respondents who identified a discipline in terms of education were trained as social workers (46%, n=164), followed by nurses and speech pathologists (each 16%, n=57). Professionals with educational backgrounds in education, counseling, sociology and psychology each represented about 3% of the respondents. Most had earned a bachelors (27%, n=97) or masters (56%, n=195) degree their identified discipline and over 85% also held a professional license. In terms of formal geriatric training, 43% (n=153) identified CE offerings as the source of this training. Thirty-three percent (n=118) indicated that they had no formal training in geriatrics. Largely from WVThe major curriculum areas of a previously offered certificate were listed and respondents asked to base their interest and need on this general curriculumCognitive changes and dementia and end of life issues were each selected by almost half of the respondents. Legal issues and health care and social policies and programs were also of great interest to over 30% of respondents. Respondents currently meet their CE needs primarily through employer-based trainings (33%, n=121) and attendance at the NASW-WV annual conference and the Summer Institute on Aging and other Division of Social Work-sponsored workshops (each selected by over 20% of the respondents).
11 Example 1: West Virginia University “A curriculum is very much needed for our workers to understand the population that we are dealing with and the need is urgent.”“It is very evident that WV NEEDS to offer this certificate.”
12 Example 1: West Virginia University Recruitment and RetentionWhat led them to the field?Job opportunity & Personal ExperienceWhat will keep them in the field?Career growth, Increased Salary & Benefits,Leadership OpportunitiesLove the Field and PopulationWhen asked what led them to work in the field, many noted a job opportunity (57%, n=201) as well as personal and family experiences (29%, n=101). Open-ended responses to this question indicate that many respondents have always “enjoyed” working with this population. Increased opportunity for career growth as well as increased salary and benefits were all selected by over 40% of respondents. Increased opportunity for leadership in the field was selected by 28% (n=98) of those who worked in the field.
13 Example 1: West Virginia University “No motivation is needed. I love the population, and I intend to continue working with them, regardless of pay-rate or continuing education”“I love what I do and just want to keep doing it.”Many noted their love for the field and population as reasons to stay in the field
14 Example 1: West Virginia University Outcomes of Needs AssessmentDevelopment of CEU-based certificateNeeds-identified curriculumPartnerships with other organizations offering trainingsDevelopment of online trainingsRecruitment and retention ideas sharedCertificate Launching in February 2012!
16 Geriatric Consultation Clinic Shawn Berkowitz, MD, CMDPaul R. Gould, LCSW, PhDYoujung Lee, PhD – EvaluatorSuk-Young Kang, PhD – EvaluatorSupported with funding from the John A. Hartford Foundation
17 Binghamton University HPPAE Geriatric Consultation Clinic Interprofessional Learning ExperiencePart of the rotational model for HPPAEMSW students collaborate with Medical Students from Upstate Medical UniversityConduct a 2-part Comprehensive Geriatric AssessmentHome VisitClinic VisitTeam Structure:Clinical Social WorkerGeriatricianStudents from both disciplinesFamily
18 Geriatric Consultation Clinic Student Learning Evaluation Evaluation MethodsA mixed-method using a triangulation design-multilevel model with concurrent data collection and analyses was employed for the evaluation of the project.Data analysis for the focus group was conducted in three steps: (1) open coding, (2) axial coding, and (3) selective coding (Strauss, 1987).ToolsPost-Experience Focus Groups with interprofessional team membersIndex of Interprofessional Collaboration (IIC) (Bronstein, 2002)
19 Geriatric Consultation Clinic Patient Outcomes Evaluation Evaluation MethodsQuantitative comparative analysis:Baseline at time of consult8-12 weeks after consult6 months after consultData CollectedMini-Mental State Exam (MMSE)Zarit Caregiver Burden ScaleUnderstanding of Medication Usage & Side EffectsFS-8 Health Survey
20 Binghamton University Community Enhancement Initiative in Aging Preparedness PurposeBinghamton University’s HPPAE in place for 5 yearsProject placed HPPAE Fellows in non-MSW agenciesAssess impact of program upon participating organizations and overall communityMethodsIndividual interviews with agency administrationFocus group with task supervisors
22 Overview: Geriatric Evaluation & Self-Management (GEMS) Project Tested effectiveness of:Interdisciplinary geriatric home-based assessmentSelf-management support servicesTarget population:Community dwelling older adults with 1+ chronic illnessesProviders:Interdisciplinary teams of social workers and physical therapists (students and professionals) collaborating with PCPs
24 Center for Aging and Disability Education and Research CADER provides comprehensive, validated, skill-based online training programs to ensure that workers have the knowledge and skills needed to meet the challenges in a rapidly changing health and social services landscape. Demonstrating that learners have mastered the competencies needed for effective practice is a key objective of our program. Competence requires professional judgment and critical thinking to translate knowledge, skills, and values into effective practice behaviors.
25 Evaluation ToolsPre and Post Competency Self-Assessment of Individual CoursesParticipants complete pre- & post-test using the approximately 20-item instrument that measures the respondent’s perceptions of their skill level on a scale of 0 to 4 (0=not skilled at all; 4=expert skill)Course EvaluationThis evaluation includes questions about the effectiveness of the online course, including whether course objectives were clearly described and met, the quality and applicability of the course, whether the course expanded learner knowledge and understanding of the topic area, and whether the training program will help them to apply practice skills in the topic area.
26 Evaluation Tools Program Included in the evaluation are questions related to the training program from both the perspective of the individual learner and in relationship to their organization:Individual learner: these include questions related to the expansion of knowledge, development of new skills, and the ability to do a better job as a result of the training program.Supervisor expectations, as well as organizational support are also included.In order to assess the degree to which new learning is maintained, evaluation is repeated at 3 months following completion of the program
28 IPE & SAGE ProgramsThe School of Social Work, College of Nursing, and College of Medicine at Florida Atlantic University developed an interprofessional curriculum to bring social work, nursing, and medical students together for collaborative practice-based learning. This interprofessional curriculum included two components: (a) the Interprofessional Education (IPE) program, and (b) the Senior Aging and Geriatrics Educator (SAGE) Mentoring program.
29 IPE & SAGE Programs Pretest/posttest evaluation (IPE Session 1) (b) Teamwork IQ Quiz(c) reflective journaling (after the IPE & SAGE Mentoring programs)(d) follow-up assignments (after the IPE & SAGE Mentoring programs)(e) surveys before and after the IPE and SAGE Mentoring programs to measure the effectiveness of the programs and reinforce learning from these experiences.
30 HPPAE Model is Competency Based Gero Social Work CompetenciesDeveloped by social workers and aging experts nationallyField tested at demonstration sitesAligned with the 2008 CSWE Education and Accreditation Standards (EPAS)Competencies are measurable practice behaviors—what all graduates are able to do—that are comprised of social work knowledge, skills, and values learned in foundation class and field curriculum.
31 Gero Social Work Competencies Five Domains of practiceValues, Ethics & Theoretical Perspectiveswill be used as example to connect to EPASAssessmentInterventionAging Services, Programs & PoliciesLeadership in Practice (new domain recently added)Each domain has ten competencies (50 total)Gero SW Competencies considered Practice Behaviors under new EPAS
32 Benefits of Geriatric Social Work Competencies (Practice Behaviors) Addresses practice at BOTH micro and macro levelsDirect practicePolicyResearchAdvocacyRecognizes social workers demonstrating leadership in aging need to be proficient in both micro and macro levelsGeriatric social workers providing direct service must be proficient in:identifying service gapsassessing community needsdeveloping programs to help older adults and caregivers cope with agingMacro practitioners must be:knowledgeable about the challenges older adults and families experienceable to identify older adult and family strengths and assets
34 Participant Discussion Describe the current or potential Inter-professional Geriatric Programs at your institutionHow are you currently or what plans do you have to evaluate the program(s)?What successes and barriers have you faced in this evaluation or its planning?