Presentation is loading. Please wait.

Presentation is loading. Please wait.

Exit presentation Program Logic An introduction Next Exit presentation.

Similar presentations


Presentation on theme: "Exit presentation Program Logic An introduction Next Exit presentation."— Presentation transcript:

1 Exit presentation Program Logic An introduction Next Exit presentation

2 Exit presentation This guide to program logic is designed for managers in government and community organisations, and for new evaluators. Program managers can use program logic to plan and clarify how program activities lead to intended outcomes. Evaluators can use it to define key evaluation questions and shape methods for data collection. This guide covers: what is program logic developing outcomes hierarchies how to use program logic in evaluation (c) 2009. This guide was first published in 1994 as a computer-based tutorial. We have refined it and release it again because people have told us that they still find it useful. The original drew on the work of the NSW Premiers Department and in particular Sue Funnell in designing program logic for public sector programs in NSW. We also acknowledge the support of Jerome Winston. Next Exit presentation

3 Exit presentation Menu 1. What is program logic? 2. Outcomes hierarchies 3. Using generic outcomes hierarchies 4. A program logic framework 5. Identifying evaluation questions 6. References Back to introduction Exit presentation

4 Exit presentation 1. What is program logic? We need to start with some terminology. Program refers to a set of coordinated activities and resources designed to achieve a desired result. A program can be large or small, simple or complex, and have a name such as program, sub-program, project, initiative and so on. However they all have the common feature of aiming to reach a goal, objective, result or outcome. Program logic refers to the causal links between the elements of a program and its intended result or outcome. It shows the assumed chain of cause and effect between the activities and the desired result. By drawing out the logic as a diagram, we have a clearer picture of how the program is assumed to work. NextNext Back to menuBack to menu

5 Exit presentation What is program logic? In the evaluation literature, program logic is a form of program theory, which has been described as: a description of the mechanisms by which the program is expected to achieve its effects (Davidson, 2005) an explanation of the causal links that tie program inputs to expected program outputs (Weiss, 1998) mapping out a programs inputs, activities, outputs, initial outcomes, intermediate outcomes, and longer term outcomes (Donaldson, 2007). Sometimes it is called a logic model:... a way of portraying such theories concretely and visually (Frechtling, 2007)...describe[s] a bounded project or initiative: both what is planned (the doing) and what results are expected (the getting). They provide a roadmap to a specific end (Knowlton and Phillips, 2009). See references at the end of this guideSee references at the end of this guide. NextNext Back Back to menuBackBack to menu

6 Exit presentation What is program logic? Program logic is used to represent a program to help in planning or evaluation. It is an analytical tool and not an end in itself. A program logic can be revised and refined to reflect changing focus on aspects of a program, or actual changes to the program over time. For evaluation, drawing a program logic diagram can help indicate the key questions to ask about a program's performance at different times in the life of the program. It can be used to shape evaluative arguments about the success or otherwise of a program. ??????? ???????? NextNext Back Back to menuBackBack to menu

7 Exit presentation What is program logic? Where does program logic fit in evaluation? The evaluation of a program typically has four steps. Identifying the program logic is usually part of the evaluation design (step 2). It can also be used to support the conclusions in the reporting (step 4). 1.Plan the evaluation 2.Design the evaluation 3.Collect data 4.Analyse data and write a report NextNext Back Back to menuBackBack to menu

8 Exit presentation What is program logic? Analysing the program logic can help answer these questions : how is the program meant to work – is it feasible? to what extent is the program able to be evaluated at this time? is the program strategy logical and appropriate? what are key questions for the evaluation? what information on program performance should be collected? 1.Plan the evaluation 2.Design the evaluation 3.Collect data 4.Analyse data and write a report NextNext Back Back to menuBackBack to menu

9 Exit presentation What is program logic? As a starting point for developing a program logic, it can be useful to think about a program as a simple system with the stages illustrated below. Environment InputsProcesses Immediate results Long-term results NextNext Back Back to menuBackBack to menu

10 Exit presentation What is program logic? Throughout this guide we will use an office skills program as a case study. This can be represented by the diagram below. Environment Funds Suitable courses Young people Community groups funded Courses are run Young people attain skills Young people get jobs Unemployment in the region is reduced InputsProcesses Immediate results Long-term results NextNext Back Back to menuBackBack to menu

11 Exit presentation What is program logic? There are many reasons why a program may not be producing the intended results, for example it is being implemented incorrectly or it does not meet the needs of the target group the measurements being used are not able to show the results the program logic is wrong Environment Funds Suitable courses Young people Community groups funded Courses are run Young people attain skills Young people get jobs Unemployment in the region is reduced InputsProcesses Immediate results Long-term results NextNext Back Back to menuBackBack to menu

12 Exit presentation What is program logic? Here we focus on the last reason - the program logic is wrong. In other words, some of the assumed links represented by the arrows are wrong. In this case, attaining skills does not lead to young people getting jobs. Environment Funds Suitable courses Young people Community groups funded Courses are run Young people attain skills Young people get jobs Unemployment in the region is reduced InputsProcesses Immediate results Long-term results X X Even if these young people have the skills, I wouldnt give them a job unless their attitude was right. Employer NextNext Back Back to menuBackBack to menu

13 Exit presentation What is program logic? Program logic can be visually presented in different ways. One useful way is an outcomes hierarchy. It shows the outcomes a program is intended to produce at different levels, and the assumed links between them. The causal chain flows from the bottom outcome to the top. Outcomes hierarchies are explained in the next section. Appropriate information gathered/ developed Information organised into accessible form Clients exposed to information Increased awareness Changed knowledge Changed attitudes Changed behaviour Program specific outcomes NextNext Back Back to menuBackBack to menu

14 Exit presentation What is program logic? Program logic can also be presented in different ways to reflect different focus. Stakeholders may have their own ideas about how a program should work and describe the program in different ways. Minister for Community Services Course coordinatorProgram Manager – Department If we put more money into it we should get better results. If these kids just turned up and did some work, they would have no trouble getting a job. If the courses were run more efficiently, then more kids would be trained and get jobs. NextNext Back Back to menuBackBack to menu

15 Exit presentation What is program logic? In practice, stakeholders have their own interests which can influence the way they think about the programs logic. Working with stakeholders in designing and refining the programs logic can help to build a better understanding of the programs intended goals and the strategies needed to reach them. Minister for Community Services Program Manager Next sectionNext section Back Back to menuBackBack to menu

16 Exit presentation 2. Developing outcomes hierarchies A program can be described in terms of its intended outcomes. These outcomes can form a logical hierarchy, which can be used as a framework for evaluating the program. NextNext Back to menuBack to menu

17 Exit presentation Developing outcomes hierarchies Firstly consider a programs intended results or outcomes, rather than its objectives or processes. For example, for the office skills program (used as a case study in this guide): An objective To teach participants office skills can be written as an outcome Graduates have suitable office skills An outcome is not a process... but a result that can be measured and observed How many graduates have which skills? NextNext Back Back to menuBackBack to menu

18 Exit presentation Developing outcomes hierarchies While the office skills program was originally described by the Departments manager in terms of aims, objectives and strategies, you can see that it aims to achieve a range of outcomes at different levels. Unemployment is reduced Aim: Objectives: Strategy: To reduce long-term unemployment for target group To teach participants appropriate office procedures To assist course graduates to gain office work positions Design suitable course Fund providers Select students Implement course Graduates have suitable knowledge and skills A suitable office skills course is prepared Participants react favourably to course NextNext Back Back to menuBackBack to menu

19 Exit presentation Developing outcomes hierarchies In fact, the office skills program aims to produce a whole series of outcomes. And we can see the outcomes will have logical links with each other based on cause and effect. Graduates have suitable knowledge and skills Participants react favourably to course For example, we have assumed this outcome has to be achieved first......in order to achieve the higher outcome. NextNext Back Back to menuBackBack to menu

20 Exit presentation Developing outcomes hierarchies The outcomes for the office skills program can be drawn up and arranged into a logical chain of cause and effect, like in this diagram. The cause and effects chain is called an outcomes hierarchy. The logic is that each lower outcome has to be achieved in order to achieve the next higher outcome. Long-term unemployment is reduced Graduates have suitable knowledge and skills Participants react favourably to course Appropriate people apply for the course Graduates gain office positions Office skills course for target group is prepared Information about training needs of target group is available NextNext Back Back to menuBackBack to menu

21 Exit presentation Developing outcomes hierarchies The outcomes hierarchy forms the basis of a program logic framework. Vertically, it shows the sequence of outcomes. Horizontally, it can be used to work out the details of outcomes. We will look at defining attributes, factors and performance information later in the guide (see section 4). Outcomes Attributes of success Factors Performance indicators 7. Long-term unemployment is reduced 6. Graduates get office jobs 5. Graduates attain office skills 4. Participants react favourably to the course 3. Appropriate people apply for course 2. Office skills course for target group is prepared 1. Information on training needs of target group is available Next sectionNext section Back Back to menuBackBack to menu

22 Exit presentation 3. Generic outcomes hierarchies Starting from a blank page to draw up an outcomes hierarchy for a program is not an easy task. Instead, a generic outcomes hierarchy can be a useful starting point. A generic outcomes hierarchy can represent a whole range of different programs that have a common underlying logic or strategy. This section describes some common generic outcomes hierarchies and how they can be used to develop an outcomes hierarchy for a specific program. NextNext Back to menuBack to menu

23 Exit presentation Generic outcomes hierarchies The Office Skills Program (the case study in this guide) can be classed as an educative program: it aims to influence peoples behavior to achieve its overall results it uses strategies involving communication and learning to change a target groups knowledge, skills or attitude. Outcomes 7. Long-term unemployment is reduced 6. Graduates get office jobs 5. Graduates attain office skills 4. Participants react favourably to the course 3. Appropriate people apply for course 2. Office skills course for target group is prepared 1. Information on training needs of target group is available NextNext Back Back to menuBackBack to menu

24 Exit presentation Generic outcomes hierarchies The Quit Smoking Program represented by the outcomes hierarchy on the right is a very different educative program. It is a public education campaign conducted through the media. However, like the office skills program, it aims to influence peoples behaviour using educational strategies. Outcomes 6. Decrease in smoking related disease 5. People quit smoking 4. Changed attitude to smoking in target group 3. Target group is exposed to the campaign 2. Media advertising 1. Information on smokers responses available NextNext Back Back to menuBackBack to menu

25 Exit presentation Generic outcomes hierarchies Although the outcomes for each program are very different, because they use the same kind of strategy, they can be represented by the same generic outcomes at each level. Outcomes 6. Decrease in smoking related disease 5. People quit smoking 4. Changed attitude to smoking in target group 3. Target group is exposed to the campaign 2. Media advertising 1. Information on smokers responses available Outcomes 7. Long-term unemployment is reduced 6. Graduates get office jobs 5. Graduates attain office skills 4. Participants react favourably to the course 3. Appropriate people apply for course 2. Office skills course for target group is prepared 1. Information on training needs of target group is available Program outcomes Changed attitudes, skills or knowledge Target group is exposed to the information Information is organised in accessible form Changed behaviour Appropriate information is available NextNext Back Back to menuBackBack to menu

26 Exit presentation Generic outcomes hierarchies These generic outcomes at each level can be used to form a generic outcomes hierarchy, which broadly represents all educative programs. Program outcomes Changed attitudes, skills or knowledge Target group is exposed to the information Information is organised in accessible form Changed behaviour Appropriate information is available Program outcome Application of new skills, attitudes and knowledge New skills, attitudes and knowledge Program is implemented Workable program is designed Worthwhile goals are established NextNext Back Back to menuBackBack to menu

27 Exit presentation Generic outcomes hierarchies Generic outcomes hierarchies have been developed for seven types of programs, based on their overall strategy. These are: Educative Regulatory Case management Advisory Provision of a service/ product Policy development Revenue raising These seven cover most public sector programs. Select from the list above to view a specific generic hierarchy. Click here to skip the generic outcome hierarchies NextNext Back Back to menuBackBack to menu

28 Exit presentation Generic outcomes hierarchies Educative An educative program aims to influence peoples behaviour to achieve its overall objective. It uses strategies which expose the target group to information and educational processes. The exposure is intended to produce changes in the target groups knowledge, skills or attitude, and, as a result, to change their behaviour. Appropriate information gathered/ developed Information organised into accessible form Clients exposed to information Increased awareness Changed behaviour Program specific outcomes Changed knowledge, skills and/ or attitudes NextNext Back to list of generic hierarchies Back to MenuBack to list of generic hierarchiesBack to Menu

29 Exit presentation Generic outcomes hierarchies Regulatory A regulatory program aims to influence peoples behaviour to achieve its overall objective. It uses strategies based on deterrence which require a regulation prohibiting unwanted behaviour. The assumption behind the program is that potential offenders will be deterred if the target group is aware of both the prohibition and the risk of being caught for violating it. Legislation Licensing/ registration Appropriate inspection and investigation Identification of infringements Appropriate behaviour, compliance Program specific outcomes Potential offenders are aware of the program Conciliation Prosecution of people who fail to comply Potential offenders are deterred NextNext Back to list of generic hierarchies Back to MenuBack to list of generic hierarchiesBack to Menu

30 Exit presentation Generic outcomes hierarchies Case management A case management program aims to influence peoples behaviour to achieve its overall objective. It treats each case (person or group) individually, and sets objectives for them on a case- by-case basis. The assumption behind a case management program is that as each individual achieves their own objectives, the circumstance of all the target group will improve and the overall program objective will be achieved. Reduced long-term dependence and cost to government Life circumstances improved Short-term objectives progressively achieved Selected individual programs satisfactorily implemented Realistic objectives set on a case- by-case basis Accurate identification of needs and capacity on a case-by-case basis Target group access program NextNext Back to list of generic hierarchies Back to MenuBack to list of generic hierarchiesBack to Menu

31 Exit presentation Generic outcomes hierarchies Advisory An advisory program aims to influence peoples behaviour to achieve its overall objective. It uses strategies that encourage individuals in the target group to seek advice about an issue or concern. The advice is intended to change their knowledge, skills or attitudes, and as a result, to change their behaviour. Advisory programs differ from educative programs in that the target groups seek out the advice, rather than being exposed to it. Appropriate information available Information organised into accessible form Target group aware of advisory service Target group seek advice Changed behaviour Program specific outcomes Changed knowledge Changed attitudes NextNext Back to list of generic hierarchies Back to MenuBack to list of generic hierarchiesBack to Menu

32 Exit presentation Generic outcomes hierarchies Service delivery A service delivery program aims to achieve its overall objectives through providing clients with a service or product. A service provision program may be a non-competitive monopoly where the government is the sole provider, or the government may be in competition with other providers. The government may also fund NGOs to provide the service. The clients of the program may vary from customers in the marketplace to clients with specific entitlements. The program outcomes could range from raising revenue to improved health for a target group. Product/ service designed to match clients needs Clients know about product/ service Clients satisfied Program specific outcomes Clients use product/ service Product/ service delivered efficiently and effectively NextNext Back to list of generic hierarchies Back to MenuBack to list of generic hierarchiesBack to Menu

33 Exit presentation Generic outcomes hierarchies Policy development A policy development program aims to change government policies to achieve its objectives. It uses consultation and research to produce policy proposals which are realistic and acceptable to government and major stakeholders. Information about issue is available All stakeholders identified Policies are approved by government Desired change occurs Process perceived as credible by government Stakeholders support policy proposals Stakeholders views taken into account Proposed policies are workable Process perceived as credible by stakeholders NextNext Back to list of generic hierarchies Back to MenuBack to list of generic hierarchiesBack to Menu

34 Exit presentation Generic outcomes hierarchies Revenue raising A revenue raising program usually involves a legislative mandate, with strategies to make the target group aware of its responsibilities, and to collect the revenue. Increased revenue Monies collected Licensing/registration Clients aware of rights/ responsibilities Legislation NextNext Back to list of generic hierarchies Back to MenuBack to list of generic hierarchiesBack to Menu

35 Exit presentation Generic outcomes hierarchies Once the initial hierarchy is designed, it can be improved by talking with stakeholders. Outcomes 7. Long-term unemployment is reduced 6. Graduates get office jobs 5. Graduates attain office skills 4. Participants react favourably to the course 3. Appropriate people apply for course 2. Office skills course for target group is prepared 1. Information on training needs of target group is available For example, in the case study office skills program, one stakeholder said: Well these kids might pick up the skills, but half of them dont want to work anyway. And the outcomes hierarchy was refined (see diagram on the right). Outcomes 7. Long-term unemployment is reduced 6. Graduates get office jobs 4. Participants react favourably to the course 3. Appropriate people apply for course 2. Office skills course for target group is prepared 1. Information on training needs of target group is available Graduates attain office skills Graduates have suitable attitudes NextNext Back Back to menuBackBack to menu

36 Exit presentation Outcomes 7. Long-term unemployment is reduced 6. Graduates develop their careers 4. Participants react favourably to the course 3. Appropriate people apply for course 2. Office skills course for target group is prepared 1. Information on training needs of target group is available Graduates attain office skills Graduates have suitable attitudes Outcomes 7. Long-term unemployment is reduced 6. Graduates get office jobs 4. Participants react favourably to the course 3. Appropriate people apply for course 2. Office skills course for target group is prepared 1. Information on training needs of target group is available Graduates attain office skills Graduates have suitable attitudes Some of the participants in the office skills program said: Now Ive done the course, I never want to work in an office. And the outcomes hierarchy was refined (see diagram on the right). Refining the outcomes hierarchy helps build up a more realistic model of the program. Generic outcomes hierarchies NextNext Back Back to menuBackBack to menu

37 Exit presentation Generic outcomes hierarchies A generic hierarchy can be a useful starting point for developing your outcomes hierarchy and for identifying the basic assumptions behind your program. Once you have talked to program staff and reviewed original program documents, you can se a generic hierarchy as a reference to help you begin drawing up your hierarchy. In practice, many programs are more complex than a generic outcomes hierarchy. You might find your program can be described by using two or more of the generic outcomes hierarchies. The next section shows how to use an outcomes hierarchy to build a program logic framework Next sectionNext section Back Back to menuBackBack to menu

38 Exit presentation 4. Program logic framework Outcomes hierarchies can be used to build evaluation frameworks by looking at the details of the outcomes at each level. Outcomes Attributes of success Factors Performance indicators 7. Long-term unemployment is reduced 6. Graduates get office jobs 5. Graduates attain office skills 4. Participants react favourably to the course 3. Appropriate people apply for course 2. Office skills course for target group is prepared 1. Information on training needs of target group is available NextNext Back to menuBack to menu

39 Exit presentation A program logic framework For example, the details of the Level 5 outcome for the Office Skills Program could be: Outcomes Attributes of success Factors Performance indicators Level 5. Graduates have suitable knowledge, skills and attitudes All graduates demonstrate satisfactory competencies All feel confident to apply for office jobs Cost per graduate is less than $700 Participants literacy levels Teaching standards vary Patterns of graduate test scores (gender, background) Cost per graduate NextNext Back Back to menuBackBack to menu

40 Exit presentation A program logic framework Outcomes Attributes of success Level 5. Graduates have suitable knowledge, skills and attitudes Attributes of success We will look at all the elements of the evaluation framework, starting with attributes of success. In the outcomes hierarchy, each outcome has only been described by a brief statement. It is helpful to identify specific characteristics of a successful outcome. These characteristics are called attributes of success for each outcome. They describe what the outcome is expected to look like if the program is operating successfully. NextNext Back Back to menuBackBack to menu

41 Exit presentation A program logic framework Outcomes Attributes of success Level 5. Graduates have suitable knowledge, skills and attitudes Attributes are defined by the needs of different stakeholders. EmployerProgram Manager - Department Our pilots showed that 60% of this target group should learn the skills. Ill only employ people who know how to do the job properly. 60% of graduates achieve satisfactory competencies NextNext Back Back to menuBackBack to menu

42 Exit presentation A program logic framework Outcomes Attributes of success Level 5. Graduates have suitable knowledge, skills and attitudes Key stakeholders will look for different attributes of success. Students attending courses Community administrator Our budget assumes that cost per graduate will be less than $700. We want to feel confident when we go for job interviews. 60% of graduates achieve satisfactory competencies Cost per graduate is less than $700 Graduates build up their confidence in job seeking NextNext Back Back to menuBackBack to menu

43 Exit presentation A program logic framework Factors Factors are the next element in the framework – factors affect how well the outcome is achieved. One way to consider the factors affecting the program is to do a SWOT analysis. StrengthsWeaknesses OpportunitiesThreats NextNext Back Back to menuBackBack to menu

44 Exit presentation A program logic framework OutcomesFactors Level 5. Graduates have suitable knowledge, skills and attitudes Stakeholders can identify factors. For example, in the office skills program, teachers identify literacy as a weakness. You cant expect all the kids to get the same office skills when half of them cant read or write properly. Literacy levels of students Course coordinator NextNext Back Back to menuBackBack to menu

45 Exit presentation A program logic framework OutcomesFactors Level 5. Graduates have suitable knowledge, skills and attitudes Different stakeholders are likely to suggest different factors. For example, the participants of the office skills program identify the teachers as a factor. Students attending courses The teachers in the course were no good. Literacy levels of students Suitability of teachers NextNext Back Back to menuBackBack to menu

46 Exit presentation A program logic framework Factors Level 5. Graduates have suitable knowledge, skills and attitudes It is important to be aware of factors when it comes to evaluating programs. It enables realistic targets to be set and provides a better understanding of the programs results. Also, once important factors are identified, program managers can plan strategies to deal with them. In this way, they can improve the program and change their assumptions about the programs logic. Literacy levels of students Suitability of teachers Outcomes NextNext Back Back to menuBackBack to menu

47 Exit presentation A program logic framework Outcomes Level 5. Graduates have suitable knowledge, skills and attitudes In the case study office skills program, some changes might be made based on factors. We want to know which teachers are producing the best results. In the future we might have to select our teachers more carefully. Community administrator Factors Literacy levels of students Suitability of teachers NextNext Back Back to menuBackBack to menu

48 Exit presentation A program logic framework Outcomes Level 5. Graduates have suitable knowledge, skills and attitudes Another change might relate to the literacy level of students. Factors Literacy levels of students Suitability of teachers To account for individual differences, we record each students progress separately. We are considering introducing a basic literacy course as well. Teacher NextNext Back Back to menuBackBack to menu

49 Exit presentation A program logic framework OutcomesFactors Level 7. Long-term unemployment is reduced For higher level outcomes, there may be many more factors outside the control of the program. Direct links between these outcomes and the program may be difficult to see. Rapid technological changes Economic restructuring Increased migration NextNext Back Back to menuBackBack to menu

50 Exit presentation A program logic framework Performance information The attributes of success and factors identified in the last two steps give a more detailed picture of each outcome. These can be used to work out what information is needed to see whether the outcome is being achieved – the performance information. The attributes of success and factors identified in the case study suggest a need for information on: how well graduates learnt about office work the quality of teachers in the program the cost effectiveness of the course. Outcomes Attributes of success Factors Performance indicators Level 5. Graduates have suitable knowledge, skills and attitudes 60% of graduates achieve satisfactory competencies Graduates build up their confidence in job seeking Cost per graduate is less than $700 Participants literacy levels Teaching standards vary Patterns of graduate test scores (gender, background) Profile of teachers Cost per graduate NextNext Back Back to menuBackBack to menu

51 Exit presentation A program logic framework OutcomesFactors Level 6. Course graduates get jobs The program logic framework indicates what performance information is needed to assess how well the program is achieving a specific outcome. This information can come from program monitoring or an evaluation study. However, data collection can raise issues. For example, collecting certain data could be costly, unethical, or ultimately not very worthwhile because of the impact of other factors. That information would be great to know, but who has time to chase up the kids? Patterns of graduate employment after the course Data: % of graduates who remain in the job for 3 months (by type) Course coordinator NextNext Back Back to menuBackBack to menu

52 Exit presentation 5. Identifying key questions In the previous section we built a program logic framework based on an outcomes hierarchy. Collecting information on all outcomes could be expensive and inefficient, and priorities need to be set. When we design an evaluation or monitoring system, the framework can be used to decide which outcomes to focus on, what questions to ask, and what information is needed. The issues to consider in deciding on the key evaluation questions to ask include: the purpose of the evaluation the balance between efficiency, effectiveness, appropriateness and equity the interests of stakeholders any issues that have arisen from monitoring the program. NextNext Back to menuBack to menu

53 Exit presentation Identifying key questions While evaluations are used for many different purposes, most of them are a balance between: Program improvement Program justification and What changes are needed?Should the program continue? NextNext Back Back to menuBackBack to menu

54 Exit presentation Identifying key questions Evaluations may include questions that focus on: Efficiency Effectiveness Appropriateness Equity Adequacy Outputs achieved related to inputs. The extent that outcomes match what was intended. How well the program aimed to meet the problem it set out to address; the needs of the client group; and how well it fit government priorities. How fair the program was to all people in the target group. Whether the response was sufficient given the problem or need. NextNext Back Back to menuBackBack to menu

55 Exit presentation Identifying key questions Stakeholders might see different purposes for an evaluation. For example, in the office skills program, the program manager wants the evaluation to answer the following questions. Program Manager - Department I want to know how well the program is being run, and how we can do better next year. Are my budget targets being met? Are all the kids getting a fair chance? Program improvement Efficiency Equity NextNext Back Back to menuBackBack to menu

56 Exit presentation Identifying key questions While the course coordinator wants to know about: Effectiveness I wonder if the course structure is right. Would it be improved by more work experience time? Course coordinator Program improvement NextNext Back Back to menuBackBack to menu

57 Exit presentation Identifying key questions The Minister for Communities wants to know about: Effectiveness Most of the cabinet thinks that these programs are a waste of money. Some say that government should not be involved at all. I want to know whether these programs actually have any impact on unemployment. And I want some good success stories to show the media. Minister for Community Services Appropriateness Program justification NextNext Back Back to menuBackBack to menu

58 Exit presentation Identifying key questions The educational consultant wants to know about: Effectiveness Classroom-based programs cant give people real life skills, and using an educational model just wont work with that target group. Educational consultant Appropriateness Program justification NextNext Back Back to menuBackBack to menu

59 Exit presentation Identifying key questions The purpose of the evaluation will direct you to relevant questions. At the same time, different stakeholders will have their own interests. What is our cost for each person who gets a job? Program Manager - Department Outcomes 7. Long-term unemployment is reduced 6. Graduates get office jobs 5. Graduates attain office skills 4. Participants react favourably to the course 3. Appropriate people apply for course 2. Office skills course for target group is prepared 1. Information on training needs of target group is available Are the right people applying? NextNext Back Back to menuBackBack to menu

60 Exit presentation Identifying key questions Outcomes 7. Long-term unemployment is reduced 6. Graduates get office jobs 5. Graduates attain office skills 4. Participants react favourably to the course 3. Appropriate people apply for course 2. Office skills course for target group is prepared 1. Information on training needs of target group is available Does our course cover all of the competencies you need for office work? Course coordinator NextNext Back Back to menuBackBack to menu

61 Exit presentation Identifying key questions Outcomes 7. Long-term unemployment is reduced 6. Graduates get office jobs 5. Graduates attain office skills 4. Participants react favourably to the course 3. Appropriate people apply for course 2. Office skills course for target group is prepared 1. Information on training needs of target group is available Will these kids have the right skills and attitudes? Employer NextNext Back Back to menuBackBack to menu

62 Exit presentation Identifying key questions Outcomes 7. Long-term unemployment is reduced 6. Graduates get office jobs 5. Graduates attain office skills 4. Participants react favourably to the course 3. Appropriate people apply for course 2. Office skills course for target group is prepared 1. Information on training needs of target group is available What are the changes in youth unemployment levels? Minister for Community Services How many got jobs? NextNext Back Back to menuBackBack to menu

63 Exit presentation Identifying key questions Stakeholders interests may vary, and be at different levels of the hierarchy. Outcomes 7. Long-term unemployment is reduced 6. Graduates get office jobs 5. Graduates attain office skills 4. Participants react favourably to the course 3. Appropriate people apply for course 2. Office skills course for target group is prepared 1. Information on training needs of target group is available Politicians and senior management are usually interested in the higher level outcomes where they are seen as accountable. Minister for Community Services Program staff may want to focus on lower level outcomes where they are familiar and have more control over the results. Course coordinator NextNext Back Back to menuBackBack to menu

64 Exit presentation Identifying key questions Outcomes 7. Long-term unemployment is reduced 6. Graduates get office jobs 5. Graduates attain office skills 4. Participants react favourably to the course 3. Appropriate people apply for course 2. Office skills course for target group is prepared 1. Information on training needs of target group is available When you are planning an evaluation, the framework makes it easier to consider a range of stakeholder viewpoints NextNext Back Back to menuBackBack to menu

65 Exit presentation Identifying key questions Outcomes Performance information Level 6. Course graduates get jobs Regular program monitoring can also raise further questions for an evaluation. Our records show that 80% of the graduates who get jobs are men. Yet, equal numbers of men and women enrol. Whats going on? % of graduates who remain in the job for 3 months Program Manager - Department EndEnd Back Back to menuBack Back to menu

66 Exit presentation You have reached the end of this tutorial – you can return to the main menu to review of the sections, continue to a list of references, or exit the presentation. References References Back to menuBack to menu

67 Exit presentation References Frechtling, J.A, 2007, Logic Modeling Methods in Program Evaluation, Wiley, Inc., Jossey-Bass, San Francisco Torvatn, H., 2008, Book review (Frechtling, Logic Modeling Methods in Program Evaluation), Evaluation and Program Planning W.K. Kellogg Foundation, 2001, Logic Model Development Guide, Using Logic Models to Bring Together Planning, Evaluation & Action Wyatt Knowlton, L and Phillips, C 2008, The Logic Model Guidebook, SAGE Rogers, P.J., Hacsi, T.A., Petrosino, A. & Huebner, T.A (editor), 2000. Program Theory in Evaluation Challenges and Opportunities: New Directions for Evaluation, No. 87, Jossey- Bass. Weiss, Carol H. 1998, Evaluation: Methods for Studying Programs & Policies - 2nd edition, Prentice Hall Davidson, D.E.J., 2004. Evaluation Methodology Basics: The Nuts and Bolts of Sound Evaluation 1st ed., Sage Publications, Inc. Back to menu


Download ppt "Exit presentation Program Logic An introduction Next Exit presentation."

Similar presentations


Ads by Google