Evaluation is a professional and ethical responsibility and is a core part of PHN professional practice Commitment to evaluation helps build the PHN intelligence about  what interventions work  in what context  why and how Evaluation findings helps improve practice to be more effective and efficient Evaluation Process Evaluation

Evaluation Evaluation determines the extent to which an intervention has achieved the desired health outcomes and assesses the contribution of strategies used Key reasons for evaluation PHN interventions is to assess and improve intervention:  Effectiveness – has the intervention worked  Efficiency – relative effectiveness compared to other interventions  Efficacy – effectiveness under ideal circumstances  Economic Impact – cost-effectiveness and use of resources  Intelligence – inform future planning and theory building  Accountability – justify resource allocation and use

Process Evaluation Relationship of goals and objectives to evaluation Problem reflected in Goal measured by Outcome evaluation Determinants reflected in Objective measured by Impact evaluation Strategies reflected in Strategy Activities measured by Process evaluation Action Statements and Evaluation Levels

Process Evaluation Levels of Evaluation There are several different levels of evaluation in PHN practice: 1.Formative Evaluation - data collected prior to intervention implementation which is used to inform intervention design and assess capacity 2.Process Evaluation – assessed the intervention strategies and capacity building strategies 3.Impact Evaluation – measures whether the intervention objectives have been met 4.Outcome Evaluation – measures whether the intervention goal has been met 5.Economic Evaluation – measures cost-effectiveness of the intervention or intervention strategies

Process Evaluation Qualitative V’s Quantitative methods The two main forms of data gathering used in evaluation include: qualitative and quantitative Quantitative methods focus on numeric data that can be statistically analysed and can test the extent to which an intervention causes change in health status, health behaviour, knowledge, attitude etc Qualitative methods attempt to determine the meaning and experience of the intervention for the target group and other participants Good quality evaluation usually has components of both qualitative and quantitative methods

Process Evaluation Process evaluation assesses intervention implementation and is concerned with Intervention exposure – extent target group are engaged or aware of PHN problem Reach – proportion of target group who participate Participant satisfaction – whether participants are happy and like the intervention activities Delivery – whether activities are implemented as intended Fidelity – assessing performance of intervention materials and components Contextual aspects – aspects of the environments that influence the intervention implementation

Process Evaluation Both quantitative and qualitative methods are used in process evaluation:  Quantitative methods measure reach, delivery and exposure aspects of the intervention  Qualitative methods assess participant satisfaction, fidelity and context elements of intervention delivery. Process evaluation provides rapid feedback on the quality and integrity of the intervention – useful management tool Process evaluation is relatively low cost and is a useful quality assurance tool

Process Evaluation Evaluating education materials Several tools exist for evaluating education materials:  Standard protocol for leaflets and audiovisual materials - considers: attraction, comprehension, acceptability, personal involvement, persuasion  SMOG test - Formula for readability by calculating the number of polysyllabic words  Group leader performance - true/false questionnaire completed by participants

Process Evaluation Methodological component General definitionExample – qualitative and quantitative methods DesignTiming of data collection: when and how often data will be collected Observe classroom activities at least twice per semester with at least 2 weeks of observation Conduct focus groups with participants in the last month of the intervention Data sourcesSource of information (for example, who will be surveyed, observed, interviewed) Both qualitative and quantitative – data sources include participants, teachers/staff delivering sessions records, the environment etc Data collection tools/ measures Instruments, tools and guides used for gathering process-evaluation data Both qualitative and quantitative – tools include surveys, checklists, observations forms, interview guides etc Data collection procedures Protocols for how the data collection tool will be administered Detailed description of how to do quantitative/ qualitative classroom observation, face-to-face or phone interview, mailed survey, focus group etc Data managementProcedures for getting data from field and entered – plus quality checks Staff turn in participant sheets weekly, evaluation coordinator collects and checks surveys and gives them to data entry staff Interviews transcribed and tapes submitted at the end of the month Data analysisStatistical and/or qualitative methods used to analyse or summarise data Statistical analysis and software that will be used to analyse the quantitative data Types of qualitative analysis used Key methodological components to consider in process evaluation

Process Evaluation Process indicators Evaluation indicators are the criteria against with data or observations are assessed for judgement of intervention success or failure Evaluation indicators may come from :  Historical comparisons with similar efforts in the past  Comparisons with contemporary activities  Professional consensus – using the above and professional judgement Finding comparison data may be difficult due to a lack of published results hence collective professional judgement should be applied