Presentation on theme: "Proposal and Program Evaluation Ivan Yau. Critical Reflections Different Purposes / Targets lead to different Strategies and Demands on rigorousness of."— Presentation transcript:
Critical Reflections Different Purposes / Targets lead to different Strategies and Demands on rigorousness of program evaluation – to public, visual documentation and stories can be used effectively (have a sense that these are representative), to academic and funder, generally, statistical analysis is used (but should have meaningful no.!!)
Critical Reflections Poor in Program Evaluation in general so far!! – these can be reflected in program evaluation published with poor objectives, evaluation methods and its interpretation. So what are your roles as social workers?
Critical Reflections Evaluation on Program Effectiveness are linked to ‘accountability’ and ‘monitoring’ both. If impact / effectiveness can be demonstrated (accountability), our service can only be monitored by output. Program Evaluation is a progressive and goal-driven process, not a repetitive routine measure, e.g. after a rigorous evaluation on effectiveness, only certain kind of QC or a more in-depth evaluation is required.
Some examples 1. Hong Kong Family Welfare Society (1989) A research report on cognitive- behavioral Group Therapy. - further developed into more refined program on clients with more severe M.I. symptoms, parent group and other with more skill deficits 2. Chronic Disease Self Management Program (CDSMP) – more vigorous screening is required to ensure the commitment in the program and reduce drop-out rate
Critical Reflections Evidence – biased preference 1. Numbers – even converting subjective experience into Numbers!! 2. Counting and Rating 3. Statistical Analysis (T-test, sign- test,…logic of significant level and hypothesis testing)
Program Logic One Linear (common, but oversimplified)
Program Logic Two Holistic View: Analysis of Target and Problem, Mechanism of Change (Intervention), Objectives, Evidence and Expected changes
Mechanism of Change Some Illustrations 1. Skill training – Parenting skill training as an example structured exercise, brief lecture, demonstration, role- play and feedback, experimentation in real life 2. Cognitive restructuring – getting familiar with ABC (differentiating DT /DT, aware of undesirable impact of DT), identifying own DT, disputing these through empirical enquiry, self- restructuring process
Mechanism of Change Some Illustrations 3. Self Understanding / acceptance – self- exploration on critical incidents in trustful environment, feedback from others after intensive interaction, connecting with others deeply and comparing the difference 4. 生死教育 – reflection on the edge of life / death experience or stimulated experience
Some Clarifications Intervention is different from Objective / Effectiveness. Process review is useful for improving the practice. Client ’ s satisfaction is Not the program effectiveness.
Objective Usually viewed as ‘Prevention of Undesirable Consequence’ / ‘Alleviation of the (social) Problem’ Important and Desirable – from who ’ s perspective? Client’s? Influence from stake-holder? Theory-driven? Social Control?
Effectiveness and Change Extent: Magnitude in relation to all participants (sometimes Magnitude alone is sufficient!) Confidence: Change is not the result of chance Causal relationship: can hardly be established
Intervention Intervention must link to mechanism of changes (theory-driven and common sense!!) and desirable outcome. Group proposal – brief session outline first, and further detailed session plan as the group evolves gradually.
Effectiveness I Participants ’ change (pre-post, post only) Issues of measurement: validity (face validity), reliability (items-correlation) and sensitivity Changes reach statistically significant level Single-item comparison Scale construction: literature review, local validation, feedback from participants,…
Effectiveness II Outcome Indicator Criteria for success: how come? Baseline makes comparison feasible (indicating the change) Meaningful (and significant) to reflect the accumulated effort, or indirect evidence to achieve the ultimate goal (community level intervention)
Different Levels – G & P Group and Programs 1. Specific target group 2. Focused, limited intervention 3. Operationalization of Objectives – pre-post measures, post-measures 4. Use of statistical analysis to assess significant changes
Different Levels - Project Project 1. Specific target group with rather long-term intervention 2. Focus on several key outcome indicators 3. Pre-post measures are also feasible
Different Levels - Service Service 1. Diversified target groups with long- term intervention and general / vague objectives (IFSC, DEC,…) 2. Aggregated data is always useful 3. Focused on major groups, or use of randomizations if necessary
Political Agenda Making use of client’s views in advocating for changes, e.g. IFSC, CSSA, SW field work supervision, 自力更新服務 Connecting impact of the program with advocacy / policy change, e.g. 類 風濕性關節炎自我管理課程、中學義工服務、小學學校 社工服務、兒童之家服務、 …
Rounding Up Aggregate data (client’s satisfaction, objective reached, target complaints, etc) is useful for service evaluation, key outcome indicator (decrease in A & E of child injury due to child neglect, decrease in suicide) for project evaluation, while accumulated impact (make use of community resources by NA, support network, no. of high risk case receiving service) for both.
Rounding Up Evaluation should be better linked to continuous improvement (don’t repeat the evaluation every year), service development (promotion of a better service model) or social movement (advocacy) (e.g. changing concepts of chronically ill patient – ownership of health, instead of replying on medical treatment).
Rounding Up Evaluation can be and should be Political – particularly help marginalized group to voice out their views / needs. Social workers should: 1. Be able to link up the objective, intervention and outcome. 2. Be quipped with themselves of different ways / strategies for program evaluation.
Reference 陳錦棠（ 2006 ） 社會服務成效評估：程序邏輯模式之應 用. 香港：香港基督教女青年會 羅致光等編（ 1999 ）靑少年成長與活動案例. 香港：香港 基督敎服務處。 羅致光主編（ 1992 ）靑少年中心活動成效評估 : 方法與案 例. 香港：香港基督敎服務處 United Way of America (2006) Measuring Program Outcomes: A Practical Approach