1. RRFSS Evaluation: Issues and Strategies RRFSS Workshop, June 19th 2002 Catherine Bingle Sarah Feltis

2. RRFSS Evaluation Project 2001-2 Project background and development Evaluation objectives and framework Methodology, tools, respondents Results Key issues and strategies

3. Evaluation Project Contributors RRFSS Evaluation Work Group RRFSS Working Group 2001 Research assistant and project consultant PHRED Funding

4. Background Evaluation Work Group was formed May 2001 Evaluation framework developed Identified need for support PHRED funding supported hiring of Project Consultant and Research Assistant

5. Evaluation Objectives Evaluate the current state of RRFSS Evaluate the effectiveness of system processes Evaluate the utility of system Identify areas for improvements and successes of RRFSS

6. Expectations of the Evaluation Inform changes to RRFSS procedures Provide a baseline/template for future RRFSS evaluations Contribute to continuous quality improvement

7. Evaluation Framework Developed by Evaluation Work Group Looked at CDC and WHO surveillance system frameworks 4 KEY components Process Issues Collaboration Among Sites and Key Players Utility or Usefulness Cost Effectiveness

8. Data Collection Tools 1st drafts completed in October At least 4 rounds of comment and revision completed in November after review from the Working Group In total 5 self- administered questionnaires and one semi-structured interview were developed and used to collect data

9. Data Collection Tools RRFSS Representative Questionnaire Focus –questionnaire development, project administration, awareness and use of results, data analysis and quality, costs, partnerships, benefits and areas for improvement MOH Questionnaire Focus –familiarity and satisfaction, perceptions of value and impact on health units, feedback on ways to sustain development

10. Data Collection Tools Program Questionnaire Focus –program staff awareness, satisfaction with content of the questionnaire, decision making process, access to results, use of results, impact on programs, and areas for improvements Non-participating Health Units Focus –familiarity, health unit interest in future participation, reasons for not participating, and perceived value of RRFSS

11. Data Collection Tools ISR Interview Focus –working relationship between ISR and health units, questionnaire development process, costs issues and expectations, quality of data and methods, future development of RRFSS MOHLTC Questionnaire Focus –familiarity, impact and value, interest in partnership with RRFSS, strategies for increasing visibility and participation

12. All 6 tools were disseminated December 2001 All data collected by end of January 2002 109 of 151 (72%) of targeted respondents participated RRFSS Representative - 11 of 12 HU’s MOH - 10 of 12 HU’s Non-participating HU’s - 17 of 24 HU’s Program - 69 of 101 program managers/staff MOHLTC - 1 of 1 ISR interview Data Collection and Response Rate

13. Data Analysis Quantitative data Entered and analyzed in SPSS Frequencies Qualitative Data Entered into Word files for content analysis

14. Process Issues Collaboration Use and Usefulness Cost Issues The Future Results

15. Questionnaire development processes Satisfaction with RRFSS questionnaire Quality of data / methods Data analysis RRFSS dissemination RRFSS awareness Process Issues

16. RRFSS Reps (RR) - RR’s and Program Staff (PS), mostly managers, involved in processes for Q development and/or revision PS - about 2/5 were involved RR and PS - nearly all very or somewhat satisfied with processes RRs more satisfied with new module development than with revision processes Process Issues Questionnaire Development Processes

17. PS concerns: –Q dev. process complexity and length –Need to involve staff more RR concerns and suggestions: –greater clarity and documentation of policies, responsibilities, procedures, timelines, decision rationale –greater commitment to process ISR suggestions (in addition to some of above themes) : –procedures to assure more equality of HU access to opportunities for new questions / changes Process Issues Questionnaire Development Processes Areas for Improvement

18. RR - generally meets HU needs MOHs - split bet. very and somewhat satisfied PS - more somewhat than very satisfied, esp. with number of questions relevant to their programs Process Issues Satisfaction with Questionnaire

19. PS and RR - opportunities for more questions - both new topics and expansion of topics RR - explore rotating core RR - explore adjusting core : optional balance Process Issues Satisfaction with Questionnaire Areas for Improvement

20. Most RRs see sample size as adequate Those who do not indicated need for larger sample for subpopulation analyses Most viewed response rate as good, some excellent. Likely based on partial info 67% completion rate, June ‘01. Question quality viewed (ISR) as good overall Process Issues Quality of Data / Methods

21. RR and ISR - further strengthen Q quality procedures - Q review, editing, & testing. ISR - longer Q testing - pilot study (n=100) where possible (responsiveness important too) Assessment of existing modules - keeping flawed modules over time problematic Need feedback mechanism for problems from data user back to Q design Need yearly conference to discuss methods Process Issues Quality of Data / Methods Areas for Improvement

22. RR - dissatisfaction with timeliness of data receipt at HU RR - All HU's had begun to analyze RRFSS data RR - HU's have not enough skilled staff to analyze data and make it usable Process Issues Data Analysis

23. ISR, RR - analysis getting short shrift MOH, RR - analytic capacity a key sustainability challenge RR - major staff capacity differences among HU's Key factor for RR's who were somewhat dissatisfied overall with RRFSS Key concern of HU's not to participate Process Issues Data Analysis

24. RR - most said most or all of HU staff aware of RRFSS PS - 2/3 somewhat, 1/3 very familiar with Q content MOH - most very familiar with current status of RRFSS, most somewhat familiar w/content RR - ½ assess awareness raising processes in HU's as very effective, ½ somewhat eff. Process Issues RRFSS Awareness in HU’s

25. RR - most HU's had disseminated results in some manner; ½ to broader community PS - 2/3 had accessed results. Of these: Most had via their epi/analyst staff Most very satisfied w/how results provided 1/3 needed support to use results - all said it was available. 15% weren't sure if needed supports; ½ didn't need. Process Issues RRFSS Dissemination in HU’s

26. PS - dissatisfaction w/access to results most frequently due to results not being available PS - need for repeated dissemination PS - need better explanation of data methods Process Issues RRFSS Awareness & Dissemination in HU’s Areas for Improvement

27. Most RR very satisfied. Those "somewhat" concerned about cost and staff resources needed for participation Most PS were satisfied, 42% very satisfied w/how RRFSS is implemented in their health unit Process Issues Overall Satisfaction w/RRFSS Implementation

28. At non-participating HU's (epi / alt.), ½ very, ½ somewhat familiar w/RRFSS status Most somewhat familiar w/RRFSS content, some very familiar PHB rep - very familiar w/status, somewhat familiar w/content Process Issues RRFSS Awareness Among Non-Participants

29. Satisfaction with partnership Contributions to decisions, activities Perceived work group effectiveness Ways to improve partnership Collaboration

30. MOH - most very, some somewhat satisfied with how partnership works RR - half very, half somewhat satisfied RR - most say partnership is worth investment ISR very satisfied with effectiveness of working relationship among partners ISR - benefits of partnership: –provides ISR w/feedback on data use –local value of study –collegial, knowledgeable, committed working group Collaboration Issues Satisfaction with Partnership

31. RR - most feel all HU's have equal voice in decisions, although those involved longer seem to influence decisions more HU's do not make equal contributions, due to differences in HU staffing, ability to support involvement, choices to be involved Expectation that these differences will recede as more HU's gain experience Collaboration Issues Contributions to Group Decisions and Activities

32. RR- Working G, Advisory G, and Ad Hoc WG: roughly split 1/2 very, 1/2 somewhat effective Analysis G and Evaluation G - more somewhat than very effective ISR perceived as very effective ISR very satisfied with working relationship among partners Collaboration Issues Perceived Work Group Effectiveness

33. RR - some duplication between Adv. G and Work. G; difficulty of managing WG meetings ISR - –Adv. G'd challenge representing larger group –Increasing # of HU's increases coord. difficulty –Range of expertise among HU's –Meeting structures and scheduling –Critical role of Adv. Group - stability must be ensured Collaboration Issues Perceived Work Group Effectiveness Challenges; Areas for Improvement

34. RR - dedicated funding to Adv. G functions Develop MOU outlining policies, procedures, expectations, roles, conflict resolution proc. Funding for central coordination of RRFSS Annual face to face meetings Schedule chairs and recorders for WG meets Better documentation of Adv. G and TOR Collaboration Issues Ways to Improve Partnership

35. Use of Results Benefits to Health Units from RRFSS Impact on Program Planning & Evaluation Impact of Not Having RRFSS Importance for Public Health Surveillance Needs Not Met By RRFSS Use and Usefulness

36. Virtually all HU's had used RRFSS data Use highest among epi's and program staff. Managers, media, and MOH also cited. Use highest among CDP programs. Use also frequently cited for Env. HP & VPD programs; research & communications staff Use Issues Use of Results - RR

37. 38% used results in their programs Of those who had not, ½ said this was due to results not yet being available Remainder said they had not yet had time Some indicated data not of sufficient importance / relevance to use Use Issues Use of Results - PS

38. Barriers to use: –time required for data analysis and administration –awareness among PS Key needs: –funding for central project coordination –common syntax files –better dissemination efforts in HUs Use Issues Use of Results - RR Barriers; Areas for Improvement

39. Current, timely, local data, filling data gaps Data for emerging and locally relevant issues Staff use for planning and evaluation Better quality than typical HU surveys Strengthened partnership among HUs and epidemiologists, and w/community partners Increased understanding of surveillance Use Issues Key Benefits from RRFSS - RR

40. All MOH -- RRFSS results will have impact on HU ability to meet PP&E standards 8 RR agreed, 3 "too early to say" 44% of PS agreed; 26% did not; 30% not sure Non-part. HU's - 14 agreed; 3 "too early" PHB rep - agreed Use Issues Expected Impact on Program Planning and Evaluation

41. MOH, RR, & PS identified same range of negative impacts on PH effectiveness: Reduced access to local, timely, adaptable, continuous data Reduced ability to measure program objectives, reduced effectiveness in PP&E, and overall effectiveness / accountability Need for additional investment in HU- and topic- specific surveys Use Issues Impact of Not Having RRFSS

42. Majority (8/11) of RRFSS Reps were “very satisfied” w/RRFSS as surveillance method Majority (8/10) of the MOH’s from participating health HU’s view RRFSS as a “very important” tool for public health surveillance 11 of 17 non-participating HU’s also view as a “very important” Use Issues Importance for Public Health Surveillance

43. Virtually all HU’s indicated the need for other community health survey work beyond RRFSS Child and youth health assessment Surveying of sub-populations e.g. pregnant women HU specific topics Sensitive topics Information not obtained through telephone survey Use Issues Needs not Met by RRFSS

44. Key Staff Involvement in RRFSS Return on Investment Cost Reduction Strategies Costs and Cost-Effectiveness

45. Time Annual estimated key staff = 0.68 FTE per HU (range 0.16 to 2.0 FTE per HU) WHO Epidemiologist Other research staff Activities New module dev., data analysis, and RRFSS WG meetings Data dissemination and presentation of data to HU staff Cost and Cost Effectiveness Staff Involvement in RRFSS

46. RRFSS Reps - (5/11) rated as “good” and 3 rated as “excellent” (time, money, and resources) MOH - divided between “very satisfied” and “somewhat satisfied” with their HU’s investment in RRFSS Program Savings - most RRFSS Reps indicated “to early to say” whether RRFSS had resulted in any program saving or would create revenues for HU in future Cost and Cost Effectiveness Return on Investment

47. Support for central coordination of RRFSS Encourage partnerships with other organizations Over time - improvements in organization and efficiency Cost and Cost Effectiveness Cost Reduction Strategies

48. Sustainability Challenges Expanded HU and Provincial Participation Strategies for Increasing Participation Suggestions for Raising RRFSS Visibility The Future

49. MOH Perspective Adequate staff/resources Rapid analysis of data Funding/Cost Keeping program staff involved RRFSS Rep’s Perspective Cost/Funding RRFSS administrative work Questionnaire content The Future Sustainability Challenges

50. ISR Perspective Whether or not able to do all 38 HU Need signed commitment from HU earlier, and for longer term Working group needs support for methodology improvements and analysis The Future Sustainability Challenges

51. HU not currently participating in RRFSS, over half indicated an interest in participating in the future 6 reportedly were considering 4 were actively preparing HU not interested or not considering participating most common reason given for this was COST lack of staff time The Future Expanding HU and Provincial Participation

52. RRFSS Rep’s and MOH were asked to agree or disagree with following statements Important for all health units to participate Important for MOHLTC to participate There should be a province wide sample Funding for RRFSS should be on 100% provincial basis The Future Expanding HU and Provincial Participation

53. RRFSS Reps Perspective Agreement strongest for important for MOHLTC to participate and there should be a province wide sample 6 Strongly, 2 agree important for all HU to participate (3 disagree) 2 strongly, 5 agree funding 100% provincial basis (4 disagree) MOH Perspective Agreement strongest important for all HU to participate Most agreed important for MOHLTC participate and funding 100% provincial basis Agreement was least strong there should be province wide sample The Future Expanding HU and Provincial Participation

54. Ministry agrees to cost share Provision of core questionnaire funding Bring attention to non-participating HU Benefits realized, participation should increase Culture of evaluation in HU Use of RRFSS data The Future Strategies for Increasing Participation

55. Publish/present data Share results and information Target dissemination Link to academic centres Provide key results to PHB to support internal government processes HU provide key results to local politicians The Future Strategies for Raising Visibility

56. Conclusions What conclusions should be drawn from the results, and what are the Next Steps? To be determined … in part by our discussions today What strategic issues are raised? How will they be addressed?

57. Issues, Strategies, and Next Steps Adequate staff/resources for analysis and making data useful Increasing use of data in program planning and evaluation Dissatisfaction with getting question accepted on questionnaire (limited # of questions) Increasing sample size Process for revising old and developing new modules RRFSS coordinators overwhelmed by work

58. Issues, Strategies, and Next Steps Need for stability and support for central work of RRFSS partnership Unequal HU access to ISR, decision-making, workload and resources Upgrading RRFSS web site Sustaining funding for existing participants and new funding for interested ones Provincial representation of data Sharing data among organizations Improved timeliness of data ISR ability to handle increase

59. Thank you Catherine Bingle cbingle@simcoehealth.org Sarah Feltis sarah.feltis@region.york.on.ca