Introduction to Evaluation Odette Parry & Sally-Ann Baker

Aim and objectives of our presentation  Define evaluation and examine how it differs from research  Briefly introduce different evaluation designs and data collection approaches -Strengths and limitations  Developing evaluation plans  Governance and ethical issues

What is evaluation?  Evaluation is “a set of procedures to judge a services merit by providing a systematic assessment of it’s aims, objectives, activities, outputs, outcomes and costs.” (NHS Executive, 1997)  Evaluation is learning about ‘what works’ and lessons for future development Evaluation determines how a service is doing

So why evaluate?  Provides evidence of: - Is what we are doing working? - What are the benefits and impacts? - What was successful and what not? And why? - Have/are objectives being achieved?

So why evaluate?  Provides evidence for: - Stakeholders - Further programme development - Staff development - Other organisations - Funders

What do you want to find out?  What is happening and how often?  How is it happening and why is it happening as it is?

Approaches  The ‘What and How Much’ question in evaluation addresses measurable ‘OUTCOMES’ associated with Quantitative approaches (e.g. RCTs and Surveys)  The ‘How and Why’ question addresses PROCESS, associated with Qualitative approaches (e.g. semi-structured interviews, observation and focus groups)

Horses for Courses  Evaluation usually requires both Outcome & Process information  Evaluation may Formative &/or Summative  Small scale evaluations, while most often qualitative, do collect some quantifiable data

Plan your evaluation Evaluation is not a ‘bolt on’ Evaluation key to informing project development and delivery Plan early Involve funders, stakeholders in the planning process

Define your purpose  Clearly define the purpose of the evaluation  What are the main aims and objectives This is a key step in the planning process and will guide how the findings will be used

Develop an evaluation plan What is being evaluated Purpose of the evaluation What is known already The questions the evaluation will address Evaluation method to be used Who & Where will the evaluation take place Timescales The resources available

Be realistic about what can be achieved Often evaluation is a compromise between the ideal and the achievable - the wants of different groups - constraints of methodology - evaluator skills - resource limitations - time limitations - ethical and governance issues

Different ways to collect data  Decide which methods to use in order to get the information you need. You may use one or more of the following: - Existing data - Document analysis - Interviews - Self completed questionnaires - Observation - Focus Group

Using existing data  Use data routinely collected to examine process and outcomes  Can save time but need to ensure that data collected, in a form that can be analysed and consistent with evaluation plan  Evaluation of All Wales Dietetics Scheme used Minimum data sets. Developed in conjunction with WAG and project dieticians to ensure consistency with evaluation aims  Collected data on project activity e.g. courses run, start and completion dates, description of activities, participant details, course outcomes, partnerships built etc

Document Analysis  Policy documents, minutes, operational policies - Track project development, aims of project  Whilst can provide rich information  May only provide part of picture  May be open to subjective interpretation

Observation  Observing participants in activities  Participant vs non participant  Observer effects  Not suitable for some settings  Can be difficult to collect & record data

Interviews  Enables in depth exploration of how people think & feel about certain topics, effectiveness of your intervention etc  Rich data, allows in-depth understanding  Can explore more sensitive area  Can tailor to the individual  No group influence  Resource intensive

Focus groups  Investigation of how groups perceive topics, view the effectiveness of your intervention  Clarify issues identified in surveys  Provide solutions to problems  Less resource intensive and 1 to 1 interviews  Group management issues  Group process issues

Questionnaires  Identification of patterns, trends  Investigate needs, expectations, perspectives, preferences, satisfaction, knowledge  Large samples, relatively lower cost  Ease of analysis  Low response rates  Respondent bias  Language and literacy issues

Analysing your data  Avoid bias, involve more than one person in the task  Address your key questions  Combine data types & findings from different sources  Compare views of different groups  Don’t anticipate results, look for unexpected findings

Evidence of impact Focus on whether:  the purpose has been achieved  the needs of those who take part have been met  there are unintended outcomes arising from the intervention  the intervention has resulted in changes in behaviour  there are barriers to and facilitators of successful implementation

Using your findings  Inform strategy, policy development  Inform budgets  Inform funding proposals  Inform improvement plans & and make changes  Identification of training needs  Identification of areas for future research/evaluation

Presenting evaluation findings  Present balanced view, don’t just report the positive  Consider the needs of your audience,  Different versions of reports or types of presentation might be needed

Reflect on the evaluation process  Were aims and objectives met? If not why  Were methods employed appropriate?  Tools, recruitment methods, analysis etc  Did you reach target groups?  Were resources sufficient?  What changes resulted ?  Future impacts?

Governance Issues  Be careful about crossing over from evaluation into the realms of research – carries additional connotations for ethics approval etc  Take advice  Take an ethical/professional approach to the evaluation, consider the participant and yourself

Ethics and the rights of Participants  Requirements of Data Protection Act (1998)  Some evaluations may require ethical approval Issues to consider - Informed Consent - Deception - Debrief - Withdrawal - Confidentiality

Data Protection Act 1998 – Your responsibility  Only hold genuinely required information  Certain types of sensitive information carry added restrictions and may only be used with permission  Information should only be collected for the purposes for which it was initially required  Data must only be processed in accordance with the legislation  People must not be harmed by how their information is used  People have a right to view information being held about them  Information must be accurate, kept up to date, kept secure and deleted when obsolete

In summary  Evaluation an essential component  Need to plan  Variety of approaches  Methods used dependent on questions to be asked  Consideration of Governance issues

Resources  Data Protection Act see the information commissioners pages aspx aspx  National Research Ethics Service if your evaluation takes place on NHS premises or with Staff, patients ethical approval might be needed check with your local R&D office and LREC administrator  UK Evaluation Society has a useful resource page