DESIGN ISSUES, CHOICES Between vs. within-subject design? Dealing with wealth effects Strategy method Eliciting preferences Eliciting beliefs

Between- vs. Within-Subject Design Between-subject design: Each subject participates only in one treatment. We compare treatment averages. Within-subject design: Each subject participates in more than one treatment. Can observe the same person in more than one condition.

Between- versus Within- Subject Design Let y be the outcome we are interested in and  the treatment effect. “i” is an index for subject i. t is for “treated” and u is for “untreated”. Then, Within person design: i = yi1 – yi0 Between person design:  = yt* - yu*

Within-Subject-Design: Allows individual comparison Control for individual fixed effects (things that we do not observe, that are constant for an individual, e.g. how much she cares about others’ payoffs, how “rational” she is etc.) More powerful statistical tests than possible with between-subjects design, especially when the sample size is small. Cheaper to run, since you need to use fewer subjects. But, there is an “order effect” problem. – In the second treatment subjects have learned something already—what happens in the first treatment can affect what happens in the 2nd Possible Solution: reverse order to control for order effects—some subjects go through first treatment A then treatment B, some subjects go through first treatment B then treatment A. Question for you: What if we have 5 treatments? 10 treatments? Another disadvantage is that subjects may get a hint of what you are trying to study (since they see all treatments),and that might affect what they do.

In between-subjects design, we rely on randomization to be able to say something about treatment effects. How? Suppose that each individual has different characteristics that we cannot see. Since the same individual does not go through both treatments, we are comparing different people under the different treatments. But, if you assign people to treatments randomly, and if you have a large enough sample, the groups under different treatments will be similar. You can therefore say something about treatment effects.

Wealth effects Subjects’ monetary endowment can change during the experiment if you pay every round, especially important in a within-subject design Also, people might hedge. One solution: Reward one (or more) of the rounds randomly Also economizes on data.

How “valid” are laboratory data? 1. Internal validity: Do the data permit correct causal inferences? If you design your experiment well and control the things you need to control, internal validity can be achieved. 2. External validity: Is it possible to generalize from lab to field? Can what you observe in the lab say something about the “real world”? The experimentalist’s answer is yes. =>When possible, use experiments in conjunction with field experiments and/or naturally occurring data. Studies that show that behavior in “games” correlate with real choices are very popular.

Strategy Method Used in game-theoretic experiments, mainly. Elicit a whole strategy rather than just one action. The idea is that, instead of just playing the game and responding to whatever the other person does, subjects are asked to indicate an action at each possible information set. e.g. Suppose we are interested in a sequential prisoner’s dilemma game. Player 1 moves first, picks C or D, then player 2 decides after observing player 1’s move.

Here, the strategy method would involve asking: -What will you do if player 1 plays Cooperate? -What will you do if player 1 plays Defect? So, we elicit responses for each contingency. Then, once player 1 makes a decision, player 2’s payoffs are automatically determined according to what her strategy said for that contingency. Advantage: Get more information about actions in parts of the game tree that are not likely to be reached. Disadvantage: Especially in games where some actions lead to anger or emotions, the strategy method may not be able to predict what will happen. Moreover, we may be inducing the person to think differently about the game (perhaps we are pushing them toward more rationality by having them consider a whole plan).

Inducing Risk-Neutrality (GRADS) Controlling for risk preferences can be important for proper inference from behavior in experiments. We can either get info on risk preferences by giving subjects lottery choices (like we did in class with the Holt-Laury task) at the end of any experiment, or we can try to make subjects behave in a risk-neutral way (induce risk-neutrality). One method that has been used to induce risk-neutrality (not clear if it works or not in reality!) is denominating the payoffs in the experiment in terms of “lottery tickets”. Specifically, let n be the number of tickets earned, out of a theoretical maximum of N tickets. The second stage of the experiment is a lottery, where there is a high prize (wH) and a low prize (wL), and the probability of getting the high prize is n/N. The expected utility from this lottery is then given by: EU=(n/N) U(wH)+ (1-(n/N))U(wL) (Note: if U(.) is concave, the subject is risk-averse).

Without loss of generality, normalize U(wH)=1 and U(wL)=0 Without loss of generality, normalize U(wH)=1 and U(wL)=0. The above equation reduces to EU=n/N, which is linear in n. Which means, even a risk-averse subject will be risk-neutral in decisions that have payoffs denominated in terms of lottery tickets. Example: Consider the choice between two decisions: Decision 1 gives 50 tickets (out of a possible 100) with certainty, and Decision 2 gives 25 tickets and 75 tickets with equal probability. Both decisions imply a 50% chance of winning the high prize. Therefore, any EU maximizer will be indifferent between the choices, i.e. a risk-averse subject will behave like a risk-neutral subject when the payoffs themselves are in lottery tickets rather than sure money.

Eliciting Beliefs (GRADS) How do we find out what subjects believe about the realization of a random variable, or about what another subject is likely to play? Having information about subjects’ beliefs is important both in individual decision problems and in games. Example from individual decision-making: Suppose we want to find out how subjects’ beliefs about an uncertain event will change with new information. I can give people information, then ask what they think. E.g. There are two possible urns (urn A and urn B) with different numbers of red and black balls inside. Suppose that A has 6 six red and 4 black, and urn B has 6 black and 4 red. One of them will be chosen randomly and we don’t know which one (the balls inside of the chosen urn are not visible). Suppose each urn is equally likely. I draw one ball from the chosen urn, it’s red. I want to know what is the subject’s perceived probability of the chosen urn being urn A after this information. How can I elicit this belief truthfully? Example from game theory: Prisoner’s dilemma Before subjects make their decisions in a PD, both players are asked what is the probability with which they think the other player will cooperate/defect? And then they play the game.

THE GOOD THING ABOUT BELIEF ELICITATION: Beliefs can be very informative to understand the motivations behind subjects’ behavior Beliefs are of particular importance to check the rationality of decisions (e.g. guessing game) GENERAL ISSUES/PROBLEMS WITH BELIEF ELICITATION Experimenter-Demand Effect (you may make people think about stuff they would not have thought about) Directs focus on particular problems, e.g., guessing game! (if you ask them what others are likely to do, you make them think about it) Desire to be consistent: people state beliefs to “match” their actions Desire to justify actions: Someone defects or does something “selfish”, and states that she thought the other person would defect also. Truthful elicitation mechanisms can be complicated to explain (e.g., payment dependent on distance measure between true outcome and expected outcome) Procedures’ incentive compatibility can depend on risk-neutrality. Can pollute incentives in the experiment if people “hedge” decisions and beliefs, to guarantee a sure payoff.

In general, suppose that the state can be either A or B In general, suppose that the state can be either A or B. I want to find out the probability with which the subject thinks that the state is A. I want to elicit this in some way that will lead people to be truthful about what they think. A commonly-used method for belief elicititation is the “quadratic scoring rule” procedure. We ask the subject to report her subjective probability for state A happening, and then pay her according to the actual realization of the state. Let r be the reported probability for state A. Let I be an indicator variable: I=1 if state A realizes. I=0 if state A does not realize. Then, Payoff=1-(r-I)2 Payoff=1-r2 if I=0, 2r-r2 if I=1 Note that the worst payoff (=0) happens when you assign probability 1 to A and the actual state is not A, and the best payoff (=1) happens when you assign probability 1 to A and the actual state is A.

This mechanism is “incentive-compatible” (induces truthful revelation) if the subject is a risk-neutral expected utility maximizer. If her subjective probability is p, and her reported probability is r, the subject’s expected payoff is: EU=p(2r-r2)+(1-p)(1-r2) It is possible to show that the EU will be maximized when r=p (when the subject tells the truth). Verify this! Potential problem: If you are not risk-neutral, the mechanism is no longer incentive compatible.

Even if you find a rule that works for risk-averse people also (e. g Even if you find a rule that works for risk-averse people also (e.g. “guess the correct state, I’ll pay you 5 dollars if right), there may be incentives to hedge if both beliefs and actions are rewarded. For example, suppose that in a Prisoners’ Dilemma game again, I want to know the subject’s belief that the other player will play cooperate. If both the guess (about the other player’s action) and the payoffs from the original game are paid at the same time, the subject might think “I can guarantee myself a sure payoff if I say “he will defect” and if I play “cooperate”. Because, if he really cooperates, I get good payoffs from the game, if he defects, I get payoffs from the belief payment. So, we may not be eliciting the correct beliefs. In order to avoid this type of hedging, you could reward EITHER beliefs OR actions randomly. That is, you ask the subject what he believes, and then he plays the game, but he knows that his payment from the experiment will either depend on the accuracy of the belief or the payoff from the game, and not both (you can determine which one will count at the end of the experiment by flipping a coin). This would get rid of the incentives to hedge.