Presentation on theme: "Incentives and Program Evaluation – How to Get What You Want and Measure It Properly Jodi N. Beggs Economists Do It With Models Presented to Carlson Marketing."— Presentation transcript:
Incentives and Program Evaluation – How to Get What You Want and Measure It Properly Jodi N. Beggs Economists Do It With Models Presented to Carlson Marketing July 23, 2010 www.economistsdoitwithmodels.com
Two parts: 1.Consumer behavior and incentives 2.Data analysis and program evaluation
Part 1: Incentives (aka Incentives Work…Except When They Dont) When do they work? Routine effort, direct cause and effect
More noise -> weaker incentives Incentives affect both effort and worker selection Safelite Auto Glass: Piece-rate compensation -> Productivity increases by 44% Half of increase due to increased productivity Other half due to employee self-selection Source: Edward Lazear and Paul Oyer, Personnel Economics, Handbook of Organizational Economics, forthcoming.
Part 1: Incentives (aka Incentives Work…Except When They Dont) When do they work? Routine effort, direct cause and effect But, you get exactly what you pay for… Non-linear incentives and timing
Nonlinear incentives (eg. quotas) give workers an incentive to play a timing game If a worker has made her quota, there is an incentive to push sales to the future Alternatively, there is an incentive to just not try for the rest of the year If a worker has no hope of making her quota, there is an incentive to push sales future Shirking is also a concern here If a worker is on the cusp of making the quota, there is an incentive to hurry sales This often corresponds with discounts that may not be in the firms best interest This behavior can even happen with linear incentives if bonuses are paid yearly, quarterly, etc. Source: Paul Oyer, Fiscal Year Ends and Nonlinear Incentive Contracts: The Effect on Business Seasonality, The Quarterly Journal of Economics, February 1998.
Part 1: Incentives (aka Incentives Work…Except When They Dont) When do they work? Routine effort, direct cause and effect But, you get exactly what you pay for… Non-linear incentives and timing Distortion of multi-dimensional tasks Goldbricking and quota restriction
A sociologists summer research on a factory floor: Compensation: greater of 85 cents per day or piece rate for work turned in Gravy jobs and stinker jobs Piece-rate payout of 9 cents to $1.66 per day Bimodal distribution of piece-rate earnings One group at 45 to 54 cents (and getting the day rate of 85 cents) One group at $1.25 to $1.34 What is going on here? Source: Donald Roy, Quota Restriction and Goldbricking in a Machine Shop, The American Journal of Sociology, March 1952.
Part 1: Incentives (aka Incentives Work…Except When They Dont) When do they work? Routine effort, direct cause and effect But, you get exactly what you pay for… Non-linear incentives and timing Distortion of multi-dimensional tasks Goldbricking and quota restriction Gaming the system in general
Source: http://www.filleritem.com I Cant Imagine This Is Profitable For Amazon…
Part 1: Incentives, continued Incentive strategy and framing also matters Framing
To increase transaction compliance, marketers sometimes temporally reframe the cost of a product from an aggregate one-time expense to a series of small ongoing expenses, often in spite of the fact that the physical payments remain aggregated. Pennies-a-Day strategy How does this affect incentives? Source: John T. Gourville, Pennies-a-Day: The Effect of Temporal Reframing on Transaction Evaluation, The Journal of Consumer Research, March 1998.
Observation: People dislike losses more than they like equivalent gains Loss aversion and reference-dependent utility How can managers and markets use this observation? Long-term effects
Part 1: Incentives, continued Incentive strategy and framing also matters Framing Guaranteed versus probabilistic rewards
In general, people are risk averse…but consider the following choices: $1 or a 1/2 chance of $2 $1 or a 1/3 chance of $3 $1 or a 1/5 chance of $5 $1 or a 1/10 chance of $10 $1 or a 1/100 chance of $100 $1 or a 1/1000 chance of $1000 $1 or a 1/1,000,000 chance of $1,000,000 Not so clear, is it? This can sometimes be leveraged when designing incentives The Publishers Clearing House seems to have figured this out
Part 1: Incentives, continued Incentive strategy and framing also matters Framing Guaranteed versus probabilistic rewards Tournament incentives
Tournament incentives are payouts based on rank order (relative performance) Incentives get stronger as spread between payouts for adjacent places increases However, can have underwater option problem for low and average performers
Part 1: Incentives, continued Incentive strategy and framing also matters Framing Guaranteed versus probabilistic rewards Tournament incentives Follow up: In-kind incentives can sometimes be powerful, contrary to what traditional economic theory would predict. This is likely because people tend to artificially restrict their consumption as a means of self-control, so a gift that the person likes but wouldnt buy herself can be more satisfying than the equivalent amount of cash. For more, see Mental Accounting Matters by Richard Thaler.
Part 1: Incentives, continued When dont incentives work? Work involving cognition and/or creativity Tunnel vision and suppression of intrinsic motivation
Output-based incentives can inhibit out-of-the-box thinking Eg. evidence shows that commissioned art is viewed as less creative Output-based incentives can also inhibit intrinsic or moral incentives and have perverse effects Day care center – fine for late parents actually increased tardiness NIS 10 per child for delays over 10 minutes Fine added to monthly bill Increase didnt go away once fine was abolished Pay enough or dont pay at all Source: Daniel Pink, Drive; Uri Gneezy and Aldo Rustichini, A Fine Is a Price, The Journal of Legal Studies, January 2000; Uri Gneezy and Aldo Rustichini, Pay Enough or Dont Pay at All, The Quarterly Journal of Economics, August 2000.
Part 1: Incentives, continued When dont incentives work? Work involving cognition and/or creativity Tunnel vision and suppression of intrinsic motivation Non-actionable output-based incentives
Roland Fryer: In stark contrast to simple economic models, our results suggest that student incentives increase achievement when the rewards are given for inputs to the educational production function, but incentives tied to output are not as effective. Relative to achievement-increasing education reforms of the past few decades, student incentives based on inputs produce similar gains in achievement at lower costs. Qualitative data suggest that incentives for inputs may be more effective because students do not know the educational production function, and thus have little clue how to turn their excitement about rewards into achievement. Angrist and Lavy find that paying for passing has a greater impact on girls performance than on boys performance Source: Roland Fryer, Jr., Financial Incentives and Student Achievement: Evidence from Randomized Trials, working paper, 2010; Joshua Angrist and Victor Lavy, The Effects of High Stakes High School Achievement Awards: Evidence from a Randomized Trial, The American Economic Review, 2009.
Part 1: Incentives, continued What can a manager or marketer do instead? Autonomy, mastery, purpose
Huffington Post: Foursquare: Showing Mastery Via Badges: Badges can be powerful in a community setting when the badges are public, noticeable and respected. It also doesnt hurt if the badges are difficult (but not too difficult) to obtain.
Endowed progress effect People provided with artificial advancement toward a goal exhibit greater persistence toward reaching the goal Endow people with 2 completed steps out of 10 rather than 0 completed out of 8 and they are more likely to complete all steps Reframing from not undertaken to undertaken but incomplete Also decreases completion time Effect depends on perceptions of task completion rather than misunderstanding of sunk costs Attainability also matters Source: Joseph C. Nunes and Xavier Dreze, The Endowed Progress Effect: How Artificial Advancement Increases Effort, The Journal of Consumer Research, March 2006.
Part 1: Incentives, continued What can a manager or marketer do instead? Autonomy, mastery, purpose Status
Status in loyalty programs: Increasing the number of elites in the top tier dilutes perceptions of status Adding a subordinate tier enhances status Tiers below the second tier do not affect those at the top but can make those in the tier immediately above feel more elite Those who do not qualify for status prefer hierarchies with multiple tiers Status-laden labels (eg. gold and silver) on their own signal an increasingly selective hierarchy Source: Xavier Dreze and Joseph C. Nunes, Feeling Superior: The Impact of Loyalty Program Structure on Consumers Perceptions of Status, The Journal of Consumer Research, April 2009.
Part 2: Program Evaluation (aka Showing Causality Is Harder Than it Looks) The fundamental program evaluation problem
Source: http://xkcd.com/552/ The Post Hoc, Ergo Propter Hoc Fallacy in a Nutshell:
Source: http://gregmankiw.blogspot.com/2009/08/least-surprising- correlation-of-all.html Suppose we were to graph average SAT scores by the number of bathrooms a student has in his or her family home. That curve would also likely slope upward. (After all, people with more money buy larger homes with more bathrooms.) But it would be a mistake to conclude that installing an extra toilet raises yours kids' SAT scores. -- N. Gregory Mankiw The Least Surprising Correlation of All Time
Part 2: Program Evaluation (aka Showing Causality Is Harder Than it Looks) The fundamental program evaluation problem Why life should be like a middle school science project or a clinical trial How do researchers get around this problem? Natural experiments!
The effect of babies on lifetime earnings Why not just compare earnings of those who had kids at 24 to those who had kids at 25? Two more valid comparisons: Women who had children at 24 to women who miscarried at 24 and had children at 25 Women who had children at 24 by accident to women who had children at 25 Findings: 1 year delayed childbirth for women in 20s = 10% higher lifetime earnings Result goes away once women reach their 30s Source: Amalia R. Miller, The Effects of Motherhood Timing on Career Path, working paper, 2005.
What is the effect of veteran status on earnings? Researchers can control for what is observable- age, education, gender, race, etc. But, cant just compare identical military and non-military people There is some difference that made some of them choose the military and others not, and this difference could be correlated with earning potential Vietnam draft lottery allowed researchers to circumvent this selection bias Findings: Based on data from Vietnam War veterans, veterans earned 15% less than equivalent non-veterans Finding limited to white males and Vietnam War, so may not be generalizable Source: Joshua D. Angrist, Lifetime Earnings and the Vietnam Era Draft Lottery: Evidence from Social Security Administrative Records, The American Economic Review, June 1990.
Part 2: Program Evaluation, continued Other things to consider in program evaluation Hawthorne Effect and Mere-Measurement Effect Impact of feedback on subsequent behavior and loyalty Salience and attention (pilot program effect) Participant selection The Counterfactual
For more information, go to http://www.economistsdoitwithmodels.com/Carlson