1. Leukemia & Lymphoma Society (LLS) Information Resource Center (IRC) Planning for a Service Program Evaluation - Case Study Public Administration 522 Presented by Ervina Castillo-Newton, John Toadlena and Scott Atole October 16, 2012

2. Leukemia & Lymphoma Society (LLS) Information Resource Center (IRC) “World’s largest voluntary health organization dedicated to funding blood cancer research, education, and patient services”  Private non-profit service organization  Funded by individual and corporate contributions  68 chapters: U.S and Canada  Masters level Information specialists responding to 80,000 inquiries on leukemia, lymphoma, myeloma and other blood cancer patients, etc.  Provide information, guidance & support (phone based) Scott Atole UNM-Main PADM 522 Planning for a Service Program Evaluation Background on Service Program Case Study

3.  Evaluation of Information Resource Center(IRC)  Email – initial contact  Program and academic interest & expertise match  Successful response to RFP  Initial discussions including questions identified in RFP response. Additional details expected in the planning process  RFP: Pilot a patient navigation model for IRC follow up services  Navigate health care systems  Overcome barriers  Evaluation of service enhancement-scheduled phone call follow up to select callers. Need identified by program staff  LLS-service oriented and committed to the evaluation process  Evaluate new programs and ongoing evaluations  Internal funding for this evaluation Scott Atole UNM-Main PADM 522 Planning for a Service Program Evaluation Background on Service Program Case Study

4.  Initial Planning:  3 months (email to launch)  Report provided 6 months later  LLS moved to fund a second/larger evaluation on Patient navigation  EPIC model (steps before the pilot study) Scott Atole UNM-Main PADM 522 Planning for a Service Program Evaluation Background on Service Program Case Study

5. Understanding the Political & Organizational Environment  Pre-project-consulted various sources  Internal/external people  Internet/LLS website  LLS Promotional items/collaterals  RFP  LLS Mission Statement  LLS Organizational structure (background materials/site visit)  Commitment to organizational cause.  Political environment associated with advocacy and organizational commitment  Funding-individual/corporate  LLS operations (private sector-accountability)  How will the evaluation findings be used? Future funding from LLS Board of trustees. Internally competitive  Location of IRD within LLS structure. department Scott Atole UNM-Main PADM 522 Planning for a Service Program Evaluation Assess the Context

6. Understanding the Political & Organizational Environment  LLS values program evaluation  Objective data to support the IRC program enhancement  IRC – immediate program context  7 full time information specialists  80,000 phone calls annually  Navigation model – innovative among other national programs  Navigation to identified subgroup (2 specialists)  National organization  68 chapters – IRC link to local chapters  Need for cancer information (consumer friendly)  Potential for collaboration “community of cancer information service professional” Scott Atole UNM-Main PADM 522 Planning for a Service Program Evaluation Assess the Context

7. Define the Relationship Between Evaluator & Sponsor  Internal/External Evaluator and Funding  LLS External Evaluator/Internal funding  Findings were reported to the LLS Board of Trustees  Primary Point of Contact – LLS Vice President or Patient Services & Disease Programs (Includes IRC)  Advantages/Disadvantages-resistance to int/ext pressures, produce positive findings, objectivity, credibility, timelines(learning/building relationship), potential biases  Clarify roles and the relationship early  Evaluator and Sponsor roles  Funding arrangements  LLS –Participatory Relationship in planning and implementation  LLS – 2 Managers  IRC – 1 Supervisor (Ongoing Liaison) and 1 staff member  Ongoing Education – Responsibilities  LLS Staff – Provide feedback  Evaluator-Drafts, Technical assistance (research process), Assess stakeholder knowledge  Evaluator skill  Relevant experience, experience and interest  Qualitative vs. Quantitative Scott Atole UNM-Main PADM 522 Planning for a Service Program Evaluation Assess the Context

8. Determine the Level of the Evaluation  Leukemia & Lymphoma Society (LLS) is local  Information Resource Center (IRC) is national. Seek to create the connection with local chapters.  LLS Target population  Newly Diagnosed  Those seeking information on clinical trials  Patients residing in Wyoming  Reasoning for Target Population selection (Where will this enhancement be most helpful?):  Limited LLS resources to offer the program to all callers  Navigation approach –reduction in health disparities to vulnerable populations  Wyoming – no local chapters  Outcome vs. Process Evaluation  Primary focus on outcomes  IRC program enhancement – include and document the process Scott Atole UNM-Main PADM 522 Planning for a Service Program Evaluation Assess the context

9. Specify Evaluation Use  LLS-findings will be used for program sustainability and continued and amount of future funding.  Evaluations used to make decisions.  Important to understand the implication of negative findings  Will this program be duplicated? Will the literature/results be made available nationally? Validate Perspectives  Communications with nearly every level of LLS/IRC  Welcome evaluation  Supportive of the project  Important to understand that it is a program evaluation Scott Atole UNM-Main PADM 522 Planning for a Service Program Evaluation Gather Reconnaissance

10.  Potential stakeholders for a service program include:  Two main reasons why the evaluator must engage stakeholders in evaluation planning.  Identify and Invite Stakeholders.  Establish rapport.  Participants involved in the evaluation planning.  “Sponsor” role.  Define Stakeholder Roles and Structure for Their Input.  Importance of engaging the few stakeholder groups.  Survey design. John Toadlena UNM-Gallup PADM 522 Planning for a Service Program Evaluation Engage Stakeholders

11.  Establish Group Processes for Ongoing Stakeholder Involvement.  The communication process.  Methods used.  Evaluation plan timeline.  Preliminary reports. John Toadlena UNM-Gallup PADM 522 Planning for a Service Program Evaluation Engage Stakeholders

12.  Establishment of the stakeholder.  Become immersed in the nature of the program to be evaluated.  “Framework for Program Evaluation in Public Health”.  Understanding the IRC.  Conceptualize Program Theory or Rationale.  Usefulness of logic models and conceptual models.  Evaluation Planning Matrix (EPM). (Holden and Zimmerman, pg.78-79)  Lists the evaluation questions considered.  Short-term, Intermediate, or Long-term outcomes and identifies relevant data sources corresponding to each question.  Outcome surveys and questions. John Toadlena UNM-Gallup PADM 522 Planning for a Service Program Evaluation Describe the Program

13.  Focusing the evaluation.  The planning process.  Study design team.  Preliminary evaluation questions.  Theoretical frameworks.  Ensure Feasibility.  The focus on feasibility.  Control or comparison group.  Assess Potential Data Collection Burden.  Data collection burden an issue on two levels.  How much surveying could the participants tolerate.  What kinds of data collection operation could IRC take on.  Sample size.  Draw conclusions from the results. John Toadlena UNM-Gallup PADM 522 Planning for a Service Program Evaluation Focus the Evaluation

14.  Service Programs are unique  Start with the Mission Statement  Size and scope of the activities  Evaluation  Attitudes  Stakeholders’ knowledge  Stakeholders  Expertise & knowledge  Minimizes resistance  Planning Process  Dissemination  Goals  This case – positive and knowledgeable Ervina Castillo-Newton UNM-WS PADM 522 Planning for a Service Program Evaluation Lessons Learned

15. Comparison Group  Based on comparison of callers  Standard Service through IRC  Enroll in intervention  Information Specialists  Qualified  Trained  Experienced  Outcomes Ervina Castillo-Newton UNM-WS PADM 522 Planning for a Service Program Evaluation Conclusion