1. EVAL 6970: Cost Analysis for Evaluation Dr. Chris L. S. Coryn Nick Saxton Fall 2014

2. Agenda Establishing an analytic framework – Identifying the evaluation problem – Establishing the alternatives – Determining the audience for the evaluation – Selecting the appropriate type of cost analysis – Deciding whether a cost analysis is worth doing Activity

3. Note… Beginning next week, we will begin studying some of the more technical aspects of cost-analysis

4. Identifying the evaluation problem

5. A ‘statement of the problem’ or ‘problem identification’ is perhaps the most critical part of any form of inquiry It helps in defining the specific questions to be investigated and often in formulating hypotheses to be tested (if relevant) It also helps in defining appropriate methods to be used in addressing the problem, answering the research questions, or testing formulated hypotheses

6. The ‘statement of the problem’ should include, at a minimum – Background of the problem (e.g., history, relevance, importance) – Statement of the problem situation (e.g., basic difficulty, need) – Purpose (goal-oriented) with an emphasis on practical outcomes or products – Questions to be answered or objectives to be investigated – Delineation of the research problem (explication of relationships among variables or comparisons to be considered) – Statement of hypotheses (if relevant) – Scope and delimitations (narrowing of focus)

7. Establishing the alternatives

8. Determine all relevant, realistic, and warranted alternatives (i.e., solutions to the problem) that could be reasonably and realistically considered The selection of comparisons should be ‘exhaustive’ to the extent possible – Potential alternatives should not be excluded unless absolutely necessary

9. Determining the audience for the evaluation

10. Identify and enumerate all ‘primary’ and ‘secondary’ audiences – In evaluation, these are sometimes referred to as ‘upstream stakeholders’ (i.e., those making decisions) and ‘downstream stakeholders’ (i.e., those impacted by decisions) Primary audiences generally should receive priority as they will likely be responsible for decision making Even so, sensitivity to secondary audiences is crucial (failure to consider this audience can result in negative, undesirable consequences)

11. Selecting the appropriate type of cost analysis

12. Basic options include, but are not limited to – Cost-effectiveness – Cost-benefit – Cost-utility – Cost-feasibility

13. Nature of the analytic task Determine which method(s) of cost- analysis is(are) appropriate or inappropriate given the problem to be investigated Consider the types of questions addressed, requirements, strengths, and weaknesses of each cost- analysis method discussed last week

14. Type of analysis Analytical questions Measure of costs Measure of outcomes StrengthsWeaknesses Cost- effectiveness Which alternativ e yields a given level of effectiven ess for the lowest cost? Monetary value of resources Units of effectivene ss Easy to incorpor ate standard evaluati ons of effective ness Useful for alternati ves with a single or small number of objectiv es Difficult to interpret results when there are multiple measures of effectivene ss Cannot judge overall worth of a single alternative; useful only for comparing two or more alternatives

15. Type of analysis Analytical questions Measure of costs Measure of outcomes StrengthsWeaknesses Cost-benefit Which alternativ e yields a given level of benefits for the lowest cost? Are the benefits of a single alternativ e larger than its costs? Monetary value of resources Monetary value of benefits Can be used to judge absolute worth Can compare results across a wide variety of alternati ves Often difficult to place monetary value on all relevant benefits

16. Type of analysis Analytical questions Measure of costs Measure of outcomes StrengthsWeaknesses Cost- utility Which alternati ve yields a given level of utility at the lowest cost? Monetary value of resources Units of utility Incorporate s individual preferences for units of effectivenes s Can incorporate multiple measures of effectivenes s into single units of utility Promotes stakeholder participation in decision making Sometimes difficult to arrive at consistent and accurate measures of individual preferences Cannot judge overall worth of a single alternative; only useful for comparing two or more alternatives

17. Type of analysis Analytical questions Measure of costs Measure of outcomes StrengthsWeaknesses Cost- feasibility Can a single alternativ e be carried out within an existing budget? Monetary value of resources None Permits alternati ves that are not feasible to be immedia tely ruled out, before evaluati ng outcome s Cannot judge overall worth because it does not incorporate outcome measures

18. Receptivity of the audience Not all audiences will willingly accept certain types of cost-analysis (e.g., expressing benefits in monetary units in a cost-benefit analysis) Even so, always select the method(s) most appropriate for addressing the problem and questions – This is more a matter of professional integrity than ‘doing what the client wants’

19. Time constraints Most, but not all, cost-analyses require several years from inception to completion Certain methods (e.g., cost-utility), however, can often be completed in much shorter spans of time

20. Available expertise Each method of cost-analysis (i.e., cost-effectiveness, cost-benefit, cost- utility, cost-feasibility) often requires specialized technical skills Not all ‘evaluators’ possess these skills and, therefore, it may be necessary to engage ‘experts’ in any given cost-analysis (e.g., an economist)

21. Deciding whether a cost analysis is worth doing

22. Under certain conditions, cost- analysis may not be the ‘best’ method of evaluation Consider Scriven’s (1974) ‘cost-free’ evaluation argument – If alternatives would result in only insignificant, trivial improvements then other types of evaluation may be more relevant and beneficial

23. Activities

24. With your group, discuss your semester project in terms of the content covered in today’s lecture In doing so, address each of the following to the extent possible and in as much detail as possible – The evaluation problem – Establishment of the alternatives – The audience for the evaluation – The appropriate type of cost analysis – The need for expertise and other resources – Whether a cost analysis is worth doing Discuss your responses to the above with the class