XXIX International Congress of Psychology

XXIX International Congress of Psychology
Cost-Benefit and Cost-Effectiveness Change via Psychotherapy, is it worth the money invested? Werner W. Wittmann Universität Mannheim XXIX International Congress of Psychology No.: S Symposium: New research concepts of modeling and measuring change in clinical psychology and psychotherapy (Chair: Wolfgang Lutz) Berlin, July, 22th., 2008 XXIX International Congress of Psychology – 2008, Berlin

XXIX International Congress of Psychology – 2008, Berlin
Outline Fifteen minutes of very brief psychotherapy for psychotherapists! The focus of this intervention is according to the Howard/Orlinsky model ….R E M O R A L I Z A T I O N! Psychotherapist should focus more on change with economical impact to survive, additionally to clinically significant change and reliable change Program evaluation, cost-benefit and cost-effectiveness analysis Visualizing effect sizes with a helping hand from A. de St-EX A very simple idea, the effect size at the break-even point Some tricks with the Brogden/Cronbach/Gleser cost-benefit equation Learning and bolstering investment decisions via meta-analysis and the return on investment (ROI) concept Examples from psychotherapeutic interventions in the health and rehabilitation area Conclusions and recommendations XXIX International Congress of Psychology – 2008, Berlin

The structure of health systems
Home physician as a distributor Psychotherapy XXIX International Congress of Psychology – 2008, Berlin

Facing the challenges of evidence based competitions in the health systems Psychologists overall and clinical psychologists specifically are anxious to survive in a competitive health system and fear loosing ground Will the inclusion of psychotherapy ruin the finances of the health systems? Is psychotherapy basically a black box, where nobody knows how it works? Can psychotherapy compete with other areas in the health system? What is the available evidence for psychotherapy and its competitors in the health systems? Evidence in terms of cost-benefit and cost-effectiveness! XXIX International Congress of Psychology – 2008, Berlin

Ludwig Erhard: Introduced social free market based economy in Germany, father of the economy miracle after World War II Psychologists love him because of his statement: „Ökonomie ist zu 50% Psychologie!“ (economy is 50% psychology). Many economists smile about this statement, but think about the two nobel awards for economy, which were already given to psychologists. XXIX International Congress of Psychology – 2008, Berlin

Two approaches for summative program evaluation
Cost-benefit- and cost-effectiveness- analysis are two important tools for summative program evaluation Program input in € Program input in € e.g. for productivity e.g. life-quality or life-satisfaction Cost-benefit analysis Cost-effectiveness analysis Output in € Output in meaningful test-scores If you look at our stately mansion of program evaluation we used earlier, here you have two approaches,which are related to the third column or pillar, namely decision aids about programs. XXIX International Congress of Psychology – 2008, Berlin

Cost-outcome distinctions
Input = Resources to be invested Cdir = direct costs Cind = indirect costs Cint = intangible costs OUTCOME Output = Outcome = Benefit U = Utility in monetary terms Udir = direct utility Uind = indirect utility Uint = intangible utility E = Effectiveness (non- monetary outcomes from program evaluations or meta-analysis ) U = Outcome composed of different non-monetary utility measures (e.g.. QALYs= Quality of life in years) Program or intervention in I/O, clinical psychology, medicine or education Comparisons of alternatives: - Intervention 1 - Intervention 2, etc. Here we hint to the terms and concepts used by our colleagues in the business schools, how they make distinctions between different facets of costs and benefits. XXIX International Congress of Psychology – 2008, Berlin

In 1975, President Ford Invited a Few of His Most Trusted Advisers to the White House to Confer on the Future of Education in America In 1975, President Ford Invited a Few of His Most Trusted Advisers to the White House to Confer on the Future of Education in America Gene V Glass Psychotherapy research owes Gene a lot! XXIX International Congress of Psychology – 2008, Berlin

Outcome distributions as evidence, using the metaphor of Antoine de Saint-Exupéry‘s „Little prince“- story Here is an example how to visualize the meaning of an effect size. The effect size measure as Cohen‘s d used in the cost-benefit equation is an important parameter, which is estimated by systematic program evaluation and gets synthesized via meta-analysis. In the story a boa has swalloed an elephant. Well our empirical outcome distributions we use to assess and evaluate program effects often look like this way. Comparing two distributions can thus used as an elephant race to grasp what an effect size measure means. XXIX International Congress of Psychology – 2008, Berlin

The elephant race metaphor: 3 sigma research an unrealistic dream !
CONTROL Cohen‘s d= 3.0 INTERVENTION XXIX International Congress of Psychology – 2008, Berlin

STOP for the break-even
A more realistic race The real effect size STOP for the break-even Break-even effect size is d Empirical effect size is d The return on investment is ROI= d/d + Iz - Iz d control d + Iz - Iz treatment XXIX International Congress of Psychology – 2008, Berlin

The Brogden/Cronbach/Gleser cost-benefit equation
Effect size for the intervention or program (d) Time the effect holds on in years (T) The number of subjects treated or trained (N) Total costs per treated or trained subject (C) The standard deviation of productivity (SDPROD) in € The proportional overlap of effect with productivity (a) These informations can be used in the following cost-benefit equation, to estimate the total net benefit in € Here you find the central cost-benefit equation, which reminds us that the language of business is dollars and not correlation coefficients. Be aware that we as program evaluators always have to translate what we find in our research to other audiences not educated in the sophisticated research tools developed and used in psychological and social science research. So often we have to learn and consider the language and technical terms of others and have to develop communication skills and tricks to the educated lay stakeholders. Visualization tools are very helpful. U = N * T * SDPROD * d * a – N * C Net-Utility = Quantity * Quality Cost XXIX International Congress of Psychology – 2008, Berlin

Question: Given what constraints, gross utility just balances the total costs? Answer: Find the break-even-point, where gross utility = total costs: N * T * SDPROD * d * a = N * C Question: What effect-size do we have to produce for it? Answer: d BREAK-EVEN = C/ (T* a* SDPROD) Here we develop and propose a tool, which is very helpful already long before a program is devloped or applied. This tool can be used a priori ! We can find out how large an effect size must be at the break-even point, where cost and benefit balance each other. We might learn that the effect size at the break-even point is so low, that we are confident enough to offer a program, which outperforms that number. Meta-analysis again can give us information what others already have achieved as effect sizes. XXIX International Congress of Psychology – 2008, Berlin

The standard deviation of productivity, the „sleeping beauty“ of the Brogden/Cronbach/Gleser cost-benefit equation Economists failed to estimate SDProd and bring back the sleeping beauty to real life Two knights from psychology, Frank Schmidt und Jack Hunter developed the first useful estimates They estimated SDProd as 40-70% of salary paid L. Erhard, father of the German economic miracle after the world war II woud have been pleased because these estimates are based on principles of a free market system When does Frank Schmidt get the Nobel prize for economics? XXIX International Congress of Psychology – 2008, Berlin

Question: How large is SDPROD? Answer: Many thanks to Frank Schmidt and Jack Hunter! They have shown via systematic research, that SDPROD estimated in many diverse job areas, turned out to lay between 40 % und 70 % of the annual salary. Newer research, as well as my own German ones, resulted in estimates of 70 % and larger! Assuming that the yearly average salary for the average patient you treat is 37000€, then SDPROD is between € (40%) and 25900€ (70%) These two estimates can later be used in a sensitivity analysis to check the robustness of a cost-benefit analysis. This parameter for decades was „ Achill‘s heel“ of cost-benefit decision theory. The Brogden equation simply could not be applied. Frank Schmidt and Jack Hunter have solved that problem with an ingenious trick, which surely deserved the Nobel prize for economics. Assuming that productivity is influenced by many causes, ist distribution should be normal. They then used the mathematical characteristics of that equation, which is fully symmetrical and one sd below average we are at the 16th percental and one sd above at the 84th percentile. They now asked about the costs of hiring people at the average and these percentiles and aggrgeated these estimates over many experts and job category. They then found out that these estimates were in the 40-70% range of the salary paid. Think about how salary and productivity must be related in a free market system? Adam Smith‘s invisible hand metaphor is working. Those who pay too much get bankcrupt and those who pay too little will loose their best incumbents and also go bankrupt. Those who survive in the market will pay a salary which is mirroring productivity. XXIX International Congress of Psychology – 2008, Berlin

Effect size at the break-even-point applied a-priori to reduce anxiety of different stakeholder groups affected by our program evaluation of psychotherapy: Estimating that the average total cost C = € per patient. We assume that the effect holds on for one year (T = 1) and the outcome measures assessing the treatment have an overlap (path coefficient) of a = .50 with productivity. Yearly salary of the average TK costumer is € using the 40%-70% estimate, SD(PROD) varies between € and €. Conservative estimate: d (BREAK-EVEN) = €/(1*.50*14800€) = ; Optimistic estimate: d (BREAK-EVEN) = €/(1*.50*25900€) = ; Assuming that the effect holds on for T = 2 years, these estimates would be between .12 and .06 ! Therefore we know that we need a sample size large enough to detect effect sizes just over these break-even estimates Computing such a break-even point effect size a priori has several advantages and pay offs. In doing program evaluation you always have to anticipate the fears, anxiety and non-complaince of those being evaluated. The information about the effect size at the break-even point as being low or relatively low may make those who already developed and deliever the program confident that they surely will outperform such an effect size. In another scenario we might fear that a program, which is in the planning phase and for which a prototype is considered to be developed is so costly that it needs a very large effect size at the break-even point. Thus we might either cancel that program idea or try to develop it at a much lower cost budget. Comparing the break-even effect sizes to those we already know from meta-analysis can convince stakeholders about the worth of such programs. XXIX International Congress of Psychology – 2008, Berlin

Power Considerations To be able to detect small differences between IG and CG, we aimed at a sample size of 400 therapists With an average of 5 clients per therapist, this means we need a sample size of about 2000 patients One of the largest evaluation projects on psychotherapy in Germany Liu, et al. (2006) XXIX International Congress of Psychology – 2008, Berlin

THE RETURN ON INVESTMENT CONCEPT (ROI)
Problem: Nobody is satisfied just getting back the money invested after a year. The amount of money returned after a year should be larger. Thus this quotient of money invested to money returned, which is labeled as ROI should be larger then one, i.e. : ROI > 1 But how much larger? To get a feeling what ROI-coefficients are considered as impressive we can look at other areas! Interest rates mirror this for example. As psychologists we have to compete against other stakeholders with respect to investment decisions. Should someone invest money in programs we have to offer or would that investment result into a waste of money or opportunity costs? How can we a-priori answer such a question? As you will see we can answer this question. We will build a bridge of mutual understanding between psychology and economy XXIX International Congress of Psychology – 2008, Berlin

Computing an effect size necessary to compete with others claiming to have a certain ROI Other competitor about investments may claim having a ROI=2, which means after a year for each dollar the investor get two dollars back. Impressive indeed. Can we compete, what effect size do we have to demonstrate to get a similar ROI? Well d necessary = ROI* dBreak-Even and using data from the example above: d necessary = 2 * .22 d necessary = .44! This is a medium sized effect only. So shouldn‘t we be confident in competing? Our competitors about scarce funds might claim having impressive ROI‘s of their programs and might claim so called opportunity costs, once we get the money. Opportunity costs is money lost due to opportunities not taken. If investments go the areas and programs with lower ROI‘s instead of higher ones we loose money. Assume that our program actually demonstrates a d=.80, then we know that our ROI is still larger than those of our competitors about the scarce funds and not investing in us will lead to opportunity costs! What a message for political and financial decision making, if we can bolster it by systematic program evaluation research?! XXIX International Congress of Psychology – 2008, Berlin

The results from a mega-meta-analysis Lipsey,M.W. & Wilson,D.B.(1993)The efficacy of psychological, educational, and behavioral treatment: Confirmation from meta-analysis. American Psychologist, 48(12), XXIX International Congress of Psychology – 2008, Berlin

The Brunswik-lens-equation for relating experimental treatment (ETR) to criteria (CR)
Selection effects due to restriction (enhancement) of range 1 Danger to overestimate 1 Danger to underestimate Sampling error 1 Danger to overestimate (positive error) 1 Danger to underestimate (negative error) Psychometric reliability of experimental treatment and criterion 2 Dangers to underestimate Construct reliability of experimental treatment and criterion 2 Dangers to underestimate (lack of symmetry) There 6 dangers to underestimate against 2 dangers to overestimate A true effect size! XXIX International Congress of Psychology – 2008, Berlin

A taxonomy of assessment variants
Theory derived indices ? Single-act assessment = one specific act in one specific situation SAC Repeated act assessment = repeated observation of one specific act over time and/or situations RAC Single multiple act assessment = one observation of multiple acts, belonging to one behavioral class at one time point or situation SMAC Repeated multiple act assessment = repeated observation of multiple acts belonging to one behavioral class over time and/or situations RMAC XXIX International Congress of Psychology – 2008, Berlin

Multiple-act criteria for complex treatment packages: An example from evaluating health treatments in the German rehabilitation system XXIX International Congress of Psychology – 2008, Berlin

A more narrow multiple act criterion mapping the interests, say monetary outcomes only, tailored to a subgroup of stakeholders XXIX International Congress of Psychology – 2008, Berlin

Multiple act outcome criterion encompassing 27 different outcomes used in our research evaluating psychosomatic treatments in the German rehabilitation system 1. How do you feel at follow-up (one year after) compared to intake? For indicators 2. to 27.: change in ... 2. quality of life 3. state of health 4. state of mind 5. general condition 6. competitiveness 7. complaints/troubles/disturbances 8. state of health 9. dealing with problems/daily pressure 10. life-style concerning health 11. drug consumption 12. relationship to most closely related people 13. relationship to partner 14. family life 15. ability to work 16. number of visits to physician 17. time of inability/absence from work 18. days in hospital 19. well-being 20. dealing with problems/coping 21. capability for self-help 22. endurance of disappointment 23. coping with work 24. ability to take stress 25. getting along with people 26. getting along with problems/limitations/ impairments 27. balanced mood XXIX International Congress of Psychology – 2008, Berlin

All 27 components had to be rated according to differences in change comparing the pre year period after treatment in relation to the one year period after treatment. One point was given for improvement or positive reduction, a zero point for no change, reduction or worsening. So the worst case is a MACT_27 score of zero, the best case a MACT_27 score of 27. XXIX International Congress of Psychology – 2008, Berlin

Distribution of MAC-27-Scores 1 year after treatment N = 367 (Magic-Mountain II-Study) XXIX International Congress of Psychology – 2008, Berlin

Path-analytic (causal) modelling of treatment effects in the magic mountain study causally investigating the southwestern path of the five-data-box conceptualization CR-Box MACT_27 Multiple act outcome consisting of 27 aspects (high scores good) one year after discharge PR-Box Amount of demoralization at intake (high scores low amount) .62* .40* .12* .35* NTR-Box (Nonexperimental Treatment Variables) Quality of relationship to therapist (highscores good therapeutic alliances) Quality of self experiences and insight into own problems (high scores good) Quality of conventional treatment, i.e. nonpsychological -.15* .41* Amount of demoralization at discharge (high scores low amount) Quality of family relationship (high scores good) Amount of live events with negative financial impact (high scores many) Amount of positiv- ely evaluated life events (high scores many) .23* -.22* .16* .29* These three variables are assessed with respect to the one year time period since discharge, i.e. history effects or extra treatment variables Legend: * Significant standarized path coefficients (N = 151) p < .05 EQS model indizes: CHI-square for the independence model: df = R 2 MACT27 = .370 (.609) CHI-square for postulated model above: df = 26 p = .037 adj. R = .335 Bentler normed fit index: Bentler non normed fit index: Comparative fit index (CFI): XXIX International Congress of Psychology – 2008, Berlin

Multiple-act criterion before and after health rehabilitation (EQUA-Study) Status at intake before treatment Score distribution on the multiple act Scale EMEK_27_before (%, N = 569, M = 12,0, SD = 5,3) Status one year after treatment Score distribution on Scale EMEK_27 one year after treatment (%, N = 569, M = 17,1, SD = 7,3) XXIX International Congress of Psychology – 2008, Berlin

Effect size d for EMEK_27 EQUA-Studie (Schmidt, Nübling, Steffanowski & Wittmann, 2002) Multiple act criterion EMEK_27 at intake, score 12.0 Multiple act criterion EMEK_27 one year after treatment: d = (17.1 – 12.0) / 5.3 d = 0.96 The effect size is almost one standard deviation. The average patient treated is almost better of as 84 % of the average patient before stationary psychosomatic rehabilitation! The effect size at the break-even point was estimated (not shown here) as .37 ROI = d / d BREAK-EVEN = 0.96/0.37 ROI = ! XXIX International Congress of Psychology – 2008, Berlin

Monetary benefits computed by traditional analysis (Zielke, 1995) (Objectively assessed data from health insurance companies for psychosomatic inpatient treatment from Germany (Baden-Württemberg, Nordrhein-Westfalen, Rheinland-Pfalz) Tab2.doc im gleichen Verzeichnis Computed cost – benefit - relationship per patient (total): : 2,49 ROI = 2,49 per patient with work contract: 1 : 3,46 ROI = 3,46 XXIX International Congress of Psychology – 2008, Berlin

Summary and conclusions
Anxiety about cost-benefit ratios of psychological interventions is a major hurdle in communicating with stakeholders. Break-even point effect sizes can be calculated a-priori and can help in reducing that anxiety. Comparing that break-even point effect size with those we know from meta-analysis helps in estimating the return on investment (ROI) of psychological and other interventions. Try it, and you may be surprised how competitive in terms of ROI‘s we are in many areas. The Campbell Collaboration(C2) and the Cochrane databases can be checked for empirical evidence with respect to effect sizes. The 15 minutes of brief psychotherapy are over, do you already feel a boost in terms of remoralization? Many of my clients do! Thanks for your attention! XXIX International Congress of Psychology – 2008, Berlin

XXIX International Congress of Psychology

Similar presentations

Presentation on theme: "XXIX International Congress of Psychology"— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

XXIX International Congress of Psychology

Similar presentations

Presentation on theme: "XXIX International Congress of Psychology"— Presentation transcript:

Similar presentations

About project

Feedback