Presentation is loading. Please wait.

Presentation is loading. Please wait.

Lecture 2 Principles of Economic Experiments and Experimental Design.

Similar presentations

Presentation on theme: "Lecture 2 Principles of Economic Experiments and Experimental Design."— Presentation transcript:


2 Lecture 2 Principles of Economic Experiments and Experimental Design

3 But 1 st …results from last week’s 2 nd experiment  Goal was to find out the role our subconscious plays in decision making  General result:  the 2 nd guess in not necessarily better than the 1 st, but the average of the two is better than the 1 st  What this means is that even though you don’t have any more info on the correct answer, subconsciously you know whether you over- or underestimated your 1 st guess

4 But 1 st …results from last week’s 2 nd experiment  Our result:  Remember, the actual distance from Vancouver to HCMC is roughly 11,500km  Circumference of Earth = 40,000km  Distance from Earth to Moon = 400,000km  Average of 1 st guess = 228,051km  Average of 2 nd guess = 180,498km  Average of both guess = 204,274km

5 But 1 st …results from last week’s 2 nd experiment  Conclusions:  The subconscious is powerful, and we need to understand it better  Why was our 2 nd guess (and average of the two) better than our 1 st when our 1 st guess was supposed to be our best guess?  Many students need to travel to get a better sense of distance  Yet another reason to look into Study Abroad

6 Tutorials next week!  Tutorials begin next week  We will have our 1 st experiment  It is very important to show up on time  If you have issues with the tutorial section you are in, come see me at the break  Issues must be serious (i.e. scheduling conflict with other course/work)

7 Course Schedule DateLectureTutorial May 9IntroductionNO TUTORIALS May 16Experimental DesignNO TUTORIALS May 23NO CLASSESExperiment #1: Markets May 30Data Analysis: QualitativeNO TUTORIALS June 6Data Analysis: QuantitativeExperiment #2: Public Good June 13MarketsHow to use excel, Sample questions June 20Public GoodsASS#1 due June 27MIDTERMNO TUTORIALS July 4Game TheoryExperiment #3: Ultimatum July 11Social PreferencesExperiment #4: Crime July 18Social IDSample questions July 25CrimeASS #2 due August 1NO CLASSESNO TUTORIALS August 8DevelopmentNO TUTORIALS

8 Tutorials SectionDayTimeRoomTA D101Tues.10:30-11:20WMC 1651Bakhit D10210:30-11:20EDB 9651Graeme D10312:30-1:20WMC 2220Ji Hoon D10412:30-1:20BLU 11901Bakhit D10511:30-12:20K8664Graeme D10611:30-12:20RCB 6136Bakhit D1073:30-4:30WMC 2501Ji Hoon D1083:30-4:30RCB 6125Bakhit D109Wed.10:30-11:20WMC 3510Yang D11011:30-12:20WMC 2268Yang D11112:30-1:20WMC 1651Ji Hoon D1121:30-2:20WMC 2501Ji Hoon

9 TA Information NameOfficeOffice hoursemail Yang360512:30-1:30 (Wed.) Bakhit36212:30-3:30 (Wed.) Graeme26963:00-4:00 (Mon.) Ji Hoon16553:30-4:30 (Thurs.)

10 Principles of Economic Experiments

11 Big Questions to be Answered  How do you choose and present the rules governing an experimental economy?  How do you choose and motivate subjects?

12 I. Realism and Models

13 What is the goal of designing an experiment? 1.To make the lab resemble the real-world as much as possible?  Too complex  The more complex the design of an experiment, the more expensive it is to conduct  Reality has an infinite amount of detail  Need to choose only the most important details relevant to the research question  e.g. rules and rewards not fashion style and scent of air

14 What is the goal of designing an experiment? 2. To replicate the assumptions of the formal, theoretical model?  Even if the observed behaviour of subject is consistent with the implications of the formal model, this only serves as weak support for the model  It would be stronger if you had observed the same behaviour by relaxing some of the more stringent assumptions of the model  e.g. # of sellers in a competitive market

15 What is the goal of designing an experiment? 3. Your goal should be to find a design that offers the best opportunity to learn something useful and answer the questions that motivate your research

16 Analogy to Art  An artist wishes to express a human event, say slavery  He is unable to re-enact the event since it took place so long ago  He finds it undesirable to replicate it closely for moral reasons  He chooses a medium, say canvas or stone  The quality of his painting will be judged by how well it simplifies reality to capture and communicate the essence of being a slave  Likewise, an experiment should be judged by its impact on our understanding, not how close it replicates reality

17 Analogy to Art The Captive Slave (1827) by British portraitist John Philip Simpson

18 II. Induced-Value Theory

19 How does the experimenter gain control of the subjects?  Induced-value Theory:  Proper use of a reward allows an experimenter to induce pre-specified characteristics in experimental subjects  With the proper reward, the subjects’ innate characteristics become largely irrelevant  this is extremely important when we want to start analyzing and interpreting the results of an experiment

20 What are the necessary conditions to induce subjects’ characteristics? 1.Monotonicity  Subjects always prefer more reward  Don’t choose a reward that people are bounded by  e.g. pieces of cake, glasses of juice, or anything people can get full of *NOTE: the best and most commonly used reward is cash  Easy to satisfy

21 What are the necessary conditions to induce subjects’ characteristics? 2. Salience  Relation between actions and rewards implements the desired institution 1.Fixed payment (e.g. $5 to show-up)  NOT SALIENT because payment does not depend on subjects’ actions 2.Performance-based payment (e.g. $1 per point of profit earned)  This IS SALIENT  Salience is what differentiates surveys from controlled economic experiments

22 What are the necessary conditions to induce subjects’ characteristics? 3. Dominance  Subjects are only motivated by their reward (i.e. not motivated by what others are getting)  Need for privacy  This is why many experiments are conducted in a lab using computer terminals as the interface

23 What have experimenters learned from induced-value theory? 1.To create a controlled economic environment, need to motivate subjects by paying them in cash 2.Average payment should exceed the average opportunity cost of the subjects 3.Find subjects whose opportunity costs are low and whose learning curves are steep (e.g. undergrads!) 4.Create the simplest possible economic environment in order to promote salience and reduce ambiguities in interpreting the results

24 What have experimenters learned from induced-value theory? 5. Check instructions and verify subjects understand in “dry runs” or quizzes 6.Avoid “loaded” words in instructions  e.g. Prisoner’s Dilemma  actions A & B vs. Loyal & Betray 7. Do not deceive or lie to subjects  Salience and dominance are lost if subjects doubt the announced relation between actions and rewards



27 Experimental Design

28 Introduction  How we design our experiment dictates the questions we can ask and answer  Last week’s “Battle of the Sexes” experiment was very simple in design, and thus could only make very limited comments on coordinating behavior

29 Introduction  How could we have changed last week’s experiment to make comments on: 1.The effects of using “hockey” and “ballet” as the labels for the actions 2.The effects of being punished for not coordinating

30 Introduction 2121 ab A3,31,1 B0,03,3 2121 hockeyballet hockey3,31,1 ballet0,03,3  Consider the following experimental design

31 Introduction 2121 ab A3,31,1 B0,03,3 2121 ab A 1,0 B0,-13,3 NO PUNISHMENTPLAYER 2 IS PUNISHED  Now, consider the following experimental design

32 What is the Goal of an Experimental Design?  SHARPEN the “focus” variables and minimizing the BLURRING of “nuisance” variables

33 What is the Goal of an Experimental Design?  Focus variable:  The few variables whose effects you are interested in  This is, in fact, the point of the experiment!  AKA Treatment variable  e.g. the labeling of actions and the severity of punishment

34 What is the Goal of an Experimental Design?  Nuisance variable:  Other variables that are of no direct interest, but may affect your results  Types: 1.Controllable (e.g. sex, age, education, income, etc.) 2.Uncontrollable (e.g. subject’s interest, alertness, amount of fatigue)

35 I. Direct Experimental Control Constants and Treatments

36 How do we Sharpen Focus Variables?  Focus variables are controlled for at 2 or more different levels  Need to vary all treatment variables independently to obtain the clearest possible effects  Need to ensure all possible explanations for our outcome of interest (i.e. ability to coordinate) are covered

37 How do we Sharpen Focus Variables? No punish w/ punish A & B observations Hockey & Ballet observations No punish w/ punish A & B observationsNONE Hockey & Ballet NONEobservations Confounded TreatmentsIndependent Treatments If we notice a difference in behaviour for the treatments we have observations for, it is impossible to know whether it was the labelling or the punishment that caused it.

38 II. Indirect Experimental Control Randomization

39 How do we Blur Nuisance Variables?  Uncontrollable nuisance variables can cause inferential errors if they are confounded with focus variables  A variable is confounded if it is correlated with both the outcome variable and the treatment variables  Independence among controlled variables prevents some confounding problems  Need to ensure that any subject does not have a biased opportunity to be in a particular experimental session based on some controlled variable (e.g. sex)

40 How do we Blur Nuisance Variables?  Randomization provides indirect control of uncontrolled nuisance variables by ensuring their independence of treatment variables  EXAMPLE: Role Assignment by Order of Attendance  Subjects’ personal idiosyncrasies and habits are uncontrollable  Don’t assign early birds to one role and late comers to the other role  Randomizing roles based on order of attendance ensures differences between players is due to their roles, not due to differences in subjects’ personal characteristics

41 How do we Randomize?  Types of randomization techniques: 1.Completely Randomized  Each treatment is equally likely to be assigned in each period of an experimental trial  Quite effective when you can afford to run many periods  Independence is established after many periods  This can be improved upon (i.e. fewer periods) with the appropriate combination of control and randomization

42 How do we Randomize? 1.Completely Randomized EXAMPLE PeriodTreatment 1A&B, punish 2Hockey & Ballet, no punish 3A&B, punish 4A&B, no punish 5Hockey & Ballet, punish

43 How do we Randomize? 2. Random Blocks  Difference from completely randomized design is that 1 or more nuisances are controlled as treatments rather than randomized i.Between Subjects  Treatments are only varied across subjects  Subjects only receive 1 treatment  Our original Battle of the Sexes class experiment used this design ii.Within Subjects  Treatments are varied for each subject  In other words, every single subject is exposed to every single treatment  Subjects receive each treatment in a random order

44 How do we Randomize? i.Between Subjects  ADVANTAGES 1.Avoids carryover effects common in Within Subject Design 2.Lowers the chances of subjects suffering boredom after a long series of tests 3.Lowers the chances of subjects becoming more accomplished through practice and experience, and thus skewing the results

45 How do we Randomize? i.Between Subjects  DISADVANTAGES 1.Practicality  Requires a large number of subjects to generate useful data since subjects are only exposed to 1 treatment 2.Individual variability & Assignment bias  Since subjects are only part of 1 group, it is difficult to control for all possible individual differences 3.Environmental factors  Usually arise from poor experimental design  Suppose, for time reasons, you test one group in the morning and one in the afternoon  Many studies show that most people are at their mental peak in the morning, so this will certainly have created an environmental bias

46 How do we Randomize? ii.Within Subjects  ADVANTAGES 1.This gives as many data sets as there are conditions for each subject 2.Requires far fewer subjects than Between Subjects Design 3.Provides a way of reducing the amount of error arising from natural variance between individuals

47 How do we Randomize? ii.Within Subjects  DISADVANTAGES 1.Carryover effects where the first treatment adversely influences the others  e.g. Fatigue and Practice »In a long experiment, with multiple conditions, the participants may be tired and thoroughly fed up of researchers prying and asking questions and pressuring them into taking tests. »This could decrease their performance on the last study.

48 How do we Randomize? 3. Crossover Design  Variation of Within Subject Design  Used when you suspect your treatment variables have carryover effects (i.e. effects that last for some time)

49 How do we Randomize? 3. Crossover Design EXAMPLE  Back to the Battle of Sexes: to punish or not  Suppose we are concerned that being in the “punishment” treatment (P) first will affect behaviour in the subsequent “no punishment” treatment (N)  Simple Design: NP and PN  This design confounds time and learning with the treatment variables  Crossover Design: NPN and PNP  Using this design, the difference in the average outcome for P and N indicates the effect of your focus variables

50 How do we Randomize? 4. Factorial Design –Most important general method when you have 2 or more treatment variables –More efficient than completely randomized design

51 How do we Randomize? 4. Factorial Design EXAMPLE  2 treatment variables: A,B  Levels: A={L,M,H}, B={L,H}  3x2 factorial: each of the 6 treatments {LL,LH,ML,MH,HL,HH} occurs k times  If k=4 then 3x2x4=24 trials are required

52 How do we Randomize? 4. Factorial Design  Required number of trials increases quickly as the number of treatments (and levels of treatments) increases

53 How do we Randomize? 4. Factorial Design EXAMPLE  Suppose we have 2 treatment variables, each with 2 levels  2 2x2 = 2 4 = 16 trials  Suppose we have 3 treatment variables, each with 2 levels  2 2x2x2 = 2 8 = 256 trials *NOTE: both of these examples uses k=1

54 How do we Randomize? 5. Fractional Factorial  Alleviates the problem with the increasing number of trials  Basic idea: run a balanced subset of the Factorial Design  Less robust than Factorial Design

55 How do we Randomize? 5. Fractional Factorial EXAMPLE  3 treatments, each with 2 levels {+,-}  8 possible treatments: {+++,++-,+-+,+--,-++,-+-,-- +,---}  If you choose the first 4 or every other one then you will have unbalanced treatments because some variables are held constant or some pairs of variables are correlated

56 How do we Randomize? 5. Fractional Factorial EXAMPLE  To get a balanced subset, use the following rule:  The 3 rd element equals the product of the first two  Balanced subset: {+++,+--,-+-,--+} *NOTE: the balanced subset of a 2x2x2 factorial design requires ½ the number of treatments (i.e. 4 vs 8) and greatly reduces the number of trials (i.e. 16 vs. 256)

57 What are some of the chronic nuisances in experiments? 1.Experience and Learning  Subjects’ behaviour changes over time as they come to better understand the lab environment  “learning” is a nuisance when we want to test a static theory, but a focus when we want to characterize behavioural change over time  When it is a nuisance, control it by: 1.Using only experienced subjects (CONTROL AS A CONSTANT) 2.Using a balanced Crossover Design (CONTROL AS A TREATMENT)

58 What are some of the chronic nuisances in experiments? 2.Fatigue and Boredom  Not all experiments are exciting to participate in  Keeping experimental session under 2 hours is a good rule of thumb  Using the occasional payoff switchover is another way to help keep attention 3.Noninstitutional Interactions  Subjects’ behaviour may be affected by interactions outside the lab  e.g. Battle of Sexes: players getting together at a break and coordinating on a strategy  Careful monitoring during the break or a change in parameters after the break is advisable

59 What are some of the chronic nuisances in experiments? 4. Selection Biases  When subjects or behaviour is unrepresentative because their selection was biased  Biased implies not random  Problems with self-selection  e.g. recruiting subjects from a advanced finance course for a market trading experiment 5.Subject/Group Idiosyncrasies  A group of subjects may somehow reinforce each other in unusual behaviour  Need replications with different subjects

60 Don’t forget, tutorials start next week!

Download ppt "Lecture 2 Principles of Economic Experiments and Experimental Design."

Similar presentations

Ads by Google