Draft Outline of Framework for Evaluation of Family Resource Centres Kieran McKeown Version V6 – 7 th September 2011, CDI Conference in Dublin Family Support Agency

Evaluation Questions 1.The first set of evaluation questions are:  To what extent is the FRC programme achieving its objectives, that is, one or more of its seven outcomes?  Would these outcomes, for individuals and/or communities, have been achieved without the programme?  In order to answer these questions, it is necessary to make a before-and-after comparison of two identical groups: those who participate in the programme (the ‘treatment group’) and those who do not (the ‘comparison group’).  A more tentative answer could be provided by comparing how FRC areas have changed relative to national norms for each of outcome 2.The second set of evaluation questions are:  For any given FRC outcome, what are the factors or ‘pathways’ influencing its effectiveness, for whom, in which group of FRCs, using what type of intervention(s), etc?  In order to answer these questions, it is necessary to have a theory about how the FRC intervention works and to test this against the evidence. This requires a ‘logic model’ for each programme outcome that is being evaluated.

Draft Logic Model of FRC Programme Inputs Pathways Outcomes Changes in Behaviour & Attitudes of Parents & Children: - Individual well-being - Relationships - Support networks - Neighbourhood (trust, safety) Changes in FRC & Community Capacity: - Staff skills, attitudes, practices - Engagement & ‘empowerment’ of community Interventions with parents, children & community: - Type - Duration - Intensity Uptake and impact of interventions may be influenced by characteristics of: - Target group - Target area - Outcome area Improved Child & Parent Outcomes: 1. Healthy / healthier 2. Actively learning 3. Safe from harm 4. Economic security 5. Safe environment 6. Positive networks 7. Participating in society Changes in Local & National Service Capacity: - Response of other services - Inter-agency cooperation - National policy

Comments on Draft Logic Model of FRC Programme  This model is the first draft of a theory on how FRCs work. It is only the first step and requires further refinement within the ‘learning community’ of the FRC programme. It is essential that the evaluation is developed in tandem with the roll-out of the framework in those FRCs who participate in the evaluation, focusing on a small number of substantial and sustained interventions. This logic model also provides a simple framework for writing up an illustrative case study of an intervention.  This approach involves the FRC programme preparing itself for evaluation by specifying exactly what outcomes it is trying to achieve, ensuring that they are achievable and measurable, deciding on the pathways by which they are to be achieved, with whom, and over what time period, etc. This is already part of the task of preparing a local family support strategy, but could now be done by FRCs in the knowledge that their local strategy will be evaluated. In this sense, developing the evaluation framework could be seen as an integral part of the FRC strategy and building FRC capacity.

Further Comments on Draft Logic Model of FRC Programme  From an evaluation perspective, each ‘box’ in the model represents data to be collected; from an implementation perspective, each box represents a series of actions to be considered and delivered.  ‘Inputs’ are ‘FRC interventions’, both ‘programmed’ and ‘other developmental’ activities. ‘Inputs’ also include indicators of need and the factors that may influence participation in FRC activities.  ‘Pathways’, based on other evaluations, could include:  Changes in behaviour and attitudes of individuals, families and neighbourhoods (based on a range of indicators)  Capacity of FRC and community (based on a range of indicators)  Local and national service capacity (including inter-agency working)  ‘Outcomes’ are the seven nationally are agreed objectives for children and families as summarised in the next slide  The big challenge in the evaluation is to refine this model and find robust ways of measuring each concept. There are also challenges for implementation.

Outcomes to be Achieved by FRC Programme  The FRC programme is aligned with national policy on children; in the absence of an explicit family policy, this is national family policy de facto.  There are seven national outcomes for children in the Agenda for Children’s Services (2007), building on the National Children’s Strategy (2000). Every child should be: 1. Healthy, both physically and mentally 2. Supported in active learning 3. Safe from accidental and intentional harm 4. Economically secure 5. Secure in the immediate and wider physical environment 6. Part of positive networks of family, friends, neighbours and community 7. Included and participating in society  The FRC framework has adopted these as outcomes for both children and their parents.

Measuring Outcomes Outcome Possible Sources of Comparative Data 1. Healthy GUI ; SLAN 2007; HBSC Actively learning GUI ; DES; NEWB 2004/5; 3. Safe from harm HSE or DOH; GUI ; 4. Economic security SILC 2009; QNHS 5. Safe environment GUI Positive networks GUI ; SLAN 2007; HBSC Participating in society GUI  A major challenge for both the FRC strategy and its evaluation is that there is no agreed method for measuring national outcomes for purposes of monitoring and evaluation.  Some outcomes (eg 3, 4, 5, 7) do not seem to lend themselves to assessing the effectiveness of child and family services such as FRCs.

Next Steps  Further work will involve: Working with an advisory group of 4 FRCs to identify a set of ‘substantial and sustained interventions’ (2-4) that could be the subject of evaluation. Developing the logic model for these interventions. Finding instruments to measure the concepts, including availability of comparative data. Developing a sampling framework and method of data analysis Case studies could have a useful role within this framework as illustrations of pathways to outcomes that have already been established as effective by the evaluation.  These steps will lead to the preparation of a paper for the FSA detailing the framework for the evaluation.

ENDS HERE

GUI Instruments: Parent (9-yr old cohort) 1.Socio-economic and demographic data on parents/caregivers, including social class, education, income, employment status, marital status, housing type, family type, support networks, friendship networks, leisure activities, childcare arrangements 2.DAS Relationship with partner by primary and secondary caregiver 3.Physical and mental health, including CES-Depression by primary and secondary caregiver 4.Neighbourhood problems by primary caregiver 5.Neighbourhood services by primary caregiver Note that GUI is weak on the measurement of individual parent well-being despite the fact that it has substantial influence on child well-being

GUI Instruments: Child (9-yr old cohort) 1.Child’s EAS [Emotionality, Activity, and Shyness] Temperament by caregiver 2.Pianta child-parent relationship by primary and secondary caregiver 3.Child’s SDQ by teacher and caregiver 4.Child’s academic performance by teacher and primary caregiver 5.Drumcondra Reading & Maths competence by child 6.Attachment to school and teachers by child 7.Bullying and being bullied by child 8.Piers-Harris Self-Concept by child 9.Parenting Style Inventory II by child 10.Neighbourhood problems by child