Presentation is loading. Please wait.

Presentation is loading. Please wait.

The Basics of Experimentation

Similar presentations


Presentation on theme: "The Basics of Experimentation"— Presentation transcript:

1 The Basics of Experimentation
Lecture Outline I. Independent variables A. Definition B. Manipulated vs. Subject II. Dependent variables III. Operational definition A. Experimental operational definitions B. Measured operational definitions C. Scales of Measurement D. Evaluation of operational definitions 1. Reliability 2. Validity i. face ii. content iii. predictive iv. construct

2 IV. Evaluation of the experiment
Internal Validity Extraneous variables Confounding variables Classic threats to internal validity History Maturation Testing Instrumentation Statistical regression Selection Subject mortality Selection interactions External Validity IN CLASS CONFOUNDS PROJECT

3 What is an experiment? Smart-Pill Study Independent Variable?
Operational definition

4 Independent variable vs. Subject variabls
Subject Variables characteristics about our subjects that we might be interested in but we cannot randomly assign Age Height Weight Gender not a true independent variable not manipulated AKA predictor variables

5 Experimental Operational Definitions Measured Operational Definitions
Both independent variables and dependent variables must be carefully defined Operational Definitions Experimental Operational Definitions Operational Definition of IV Measured Operational Definitions Operational Definition of DV

6 How we operationally define our constructs can affect the outcome of a study.
Schacter (1959) The effect of anxiety on the need to affiliate Manipulated anxiety by telling subjects different stories about how much shock will hurt Gp1 Dr Gregor Zilstein It is necessary for the shocks to be intense” Gp 2 The shock will be more like a tickle Measured whether the subjects waited with others or alone

7 Sarnoff, and Zimbardo (1961)
Does anxiety cause a need to affiliate Male subjects we are studying the physiology of the mouth High anxiety Suck on Pacifier Baby bottle Breast shield Low anxiety Whistles Kazoos

8 We also must operationally define nonconstruct variables
Drug exposure – not an abstract concept like anxiety Amnt. Route of administration IV, IP, SC, oral Injection volume

9 When we operationally define our measurement (DV) we also consider what scale of measurement we are using Scales of measurement Nominal Ordinal Interval ratio

10 Nominal This is the simplest level of measurement
Classification by name only Classifies items into two or more distinct categories You are either one or the other You can be Democratic or Republican, we don’t know how democratic or republican you are. Gender Ethnicity Marital status

11 Jenni (1976) Book carrying techniques type1 - wrap one or both arms around the books with the short edges of the books resting on the hip or in front of the body type 2 - support the books by one arm and hand at the side of the body, with the long edges of the books approximately horizontal to the ground. Observed 2,626 individuals in six colleges in the US, one in Canada, and one in Central America. 82% of females used type 1 96% of males used type 2

12 A higher scale value has more of that attribute.
ordinal A higher scale value has more of that attribute. The intervals between adjacent scale values are indeterminate. Scale assignment is by the property of "greater than," "equal to," or "less than." Movie ratings (0, 1 or 2 thumbs up). U.S.D.A. quality of beef ratings (good, choice, prime). Gold, Silver, Bronze - olympics The rank order of anything. Adds the arithmetic operation of greater than and less than

13 Fahrenheit and Centigrade
Interval Intervals between adjacent scale values are equal the difference between 8 and 9 is the same as the difference between 76 and 77. Standard operations of addition, multiplication, subtraction, and division can be performed Fahrenheit and Centigrade

14 There is a rational zero point for the scale. Ratios are equivalent
e.g., the ratio of 2 to same as the ratio of 8 to 4. Examples Degrees Kelvin. Annual income in dollars. Length or distance in centimeters, inches, miles, etc. Allows Multiplication and division of scale values.

15 Ratio and interval scales tend to be preferred because the statistics that can be used on them tend to be more powerful t-tests and Anova vs. Chi Square

16 Evaluating Operational Definitions
Reliability vs. Validity Precision vs. Accuracy Throwing Darts - or Sighting a gun

17 Forms of Reliability Reliable definition of your I.V.
let’s say you hypothesize that hungry rats will learn a maze more quickly (when food is the reward) than rats that aren’t hungry. What is the IV? What is the DV? Hunger is a hypothetical construct how can we make a reliable operational definition of this construct?

18 Reliable operational definitions of your DV
How can we be certain that our measurements always come out the same? This can be examined in multiple ways 1) Inter-rater reliability Do different observers agree with the measurement have 2 people make the observations and see if their observations are correlated. should be high positive correlation

19 2) Test-Retest reliability
If the same individual is measured twice with the same instrument – you should get similar results GRE IQ 3) Inter-item reliability individual items on a questionnaire designed to measure the same thing should be highly correlated Cronbach’s Alpha you will calculate for the survey project

20 Validity Are we manipulating the variable we intend manipulate?
Anxiety example Schacter vs. Zimbardo Are we measuring the variable that we intend to measure? IQ test – does it really measure intelligence?

21 Some Types of Validity Face validity
Forms of validity Some Types of Validity Face validity The degree to which a manipulation or measurement is self-evident Does it look like it should measure what it intends to? Does it just make sense? Heart rate and anxiety? This is a weak form of validity. May be a problem if face validity is low though subjects might not take it seriously

22 Students often question the content validity of exams
The degree to which the content of a measure reflects the content of what is measured. Students often question the content validity of exams “You only asked me about the things I didn’t know.” “You didn’t ask me about the stuff I did know.”

23 Predictive validity The degree to which a measure, definition, or experiment yields information allowing prediction of actual behavior or performance If people do well on the IQ test and it is really getting at intelligence, we would expect them to do well at other tasks that require intelligence. School Other tests Difficult jobs

24 Construct validity Is the entire measure valid?
The degree to which an operational definition accurately represents the construct it is intended to manipulate or measure AND ONLY that construct

25 Construct validity continued
Does the test discriminate between constructs Lets say I designed a new test to predict how well adjusted people are The Kaiser attitude adjustment survey but it also measures verbal aptitude? In other words people’s verbal aptitude score is highly correlated with their Kaiser attitude adjustment survey score Problem?

26 Evaluating the validity of the experiment itself
The forms of validity that we have discussed so far have been in terms of defining our variables operational definitions We also need to determine whether the experiment itself is valid Internal validity The degree to which a researcher is able to show a causal relationship between antecedent conditions and subsequent behavior External Validity How well the Findings of the Experiment generalize or apply to situations in real life

27 Experiments tend to be high internal validity and low in external validity
contrived (controlled) situations rats in cages exposed to cocaine far from real life human cocaine use. Keep in mind though that an experiment that is low internal validity (confounded) also has low external validity. The experiment can tell us nothing because it is confounded.

28 Experiments are most concerned with achieving high internal validity.
Confounding Experiments are most concerned with achieving high internal validity. External validity is less of a concern. Internal validity Occurs when we are relatively certain that the changes in the DV were do to the levels of the IV if there could be other reasons for the changes, then our experiment is not internally valid. These other reasons (if they exist) are known as confounds

29 Extraneous Variables versus Confounds
The better control we have over an experiment the more likely our internal validity will be good. But – there are often other things that are changing during an experiment. If these other things (variables) are outside the experimenters control we call them extraneous variables Could be anything. changes in noise changes in temperature equipment failure differences in subject preparation (some more alert than others)

30 Extraneous variables should always be avoided if possible
but they are not a threat to the internal validity of the experiment unless they vary systematically with the independent variable. When this occurs it is known as a confound Thus, a confound is a type of extraneous variable Extraneous variables are not necessarily confounds

31 Confounds are the worst form of extraneous variable
If an extraneous variable varies systematically with the independent variable we cannot know if the changes we see in the dependent variable are due to the IV or the confounding variable. we cannot infer causality thus, low internal validity

32 Jack hammering going on in adjacent building
Extraneous variables alone are not necessarily confounds and won’t necessarily be a threat to the internal validity of an experiment Alzheimer’s rat experiment Morris water maze old rats young rats Drug no drug What is the IV, DV, Subject Variable? Jack hammering going on in adjacent building All rats experienced the jack hammering Extraneous or confound?

33 Extraneous variables alone, however, do not cause Type 1 errors
Keep in mind extraneous variables are not good even if they are not confounding. Makes it more likely you will make a Type II error Type II error Not finding an effect that is actually there failing to reject the Null hypothesis when it is actually false Because extraneous variables increase the variability of responding (behavior) it makes it more difficult to find significant effects thus, it lowers our power. Extraneous variables alone, however, do not cause Type 1 errors Finding an effect that is not really there Rejecting the Null hypothesis when it is true Confounds can lead to Type 1 errors

34 It would have been a confound if I had only ran the control rats during the jack hammering, and the drug exposed rats when it was quiet. I didn’t do that, but the jack hammering could have increased the variability in responding to make it more difficult to find a difference If I find a difference despite this extraneous variable I would not be concerned about the internal validity of the study. I found an effect under poor conditions.

35 8 classic threats to Internal Validity
Donald Campbell identified 8 kinds of extraneous variables that can threaten the internal validity of experiments Each of these is a potential confound

36 1) History (External Events)
Refers to the history of the experiment Changes in the subjects environment that might affect the results What if during testing of a new anxiety treatment The 9/11 attacks take place What if while testing treatments for asthma summer smog occurs in the city 1970s when head start began Sesame Street also began proper controls can eliminate this issue

37 2) Maturation refers to any internal (physical or psychological) changes in subjects that could affect the dependent measure Performance could get worse boredom fatigue Performance could get better Within subject designs always have to worry about this Performance at the end of a session could be better or worse because of the above reasons Make sure your IV conditions are evenly distributed across time A classic issue in medical research is spontaneous recovery people sometimes just get better over time.

38 3) Testing This refers to effects on the DV produced because of previous administration of the same test. single group pretest/posttest design Might think about initial test, figure things out they wish they had done differently. GRE/Kaplan/GRE increase in performance could be the result of taking the test previously Practice effect

39 4) Instrumentation When some feature of the measuring instrument itself changes over the course of an experiment Obviously if your equipment breaks that would be a problem What if using human assessment? train someone to observe behavior aggression They could get better at observing aggression through practice If one group is assessed early in the semester and one late in the semester could be a problem (instrumentation confound)

40 5) statistical regression
This can occur whenever people are assigned to conditions based on extreme scores on a test Regression toward the mean Draw the normal curve IQ – look at the smartest people make some manipulation Test them again likely to score more toward average The scores will tend to decrease or regress to the mean regardless of manipulation Tony Delk nine 3-pointers An early study of Sesame Street indicated that it was especially helpful to the most disadvantaged kids Kids with the lowest scores improved the most

41 6) Subject Mortality If more subjects drop out of one condition compared to another Can be a clear problem in animal studies high dose of cocaine – 50% mortality dealing with a “super” subgroup of rats in human studies dropout of experimental condition because aversive or boring those that stay are a special subgroup of people not the same as the control group Children in head start that are having trouble might drop out leaving only the better students thus, higher scores

42 7) selection Whenever the researcher does not randomly assign to the different conditions of the experiment When we randomly assign to groups we spread individual differences among our subjects across the conditions without random assignment individual differences could affect the results of our study. Self selection bias when participants choose which group they will be in perhaps parents choose to place their kids in head start because they are more motivated for their kids to learn

43 8) selection interactions Selection & History
A selection threat can combine with another threat to form a selection interaction Selection & History Head start kids are more motivated (parents chose program) and more likely to watch Sesame street when it came on Selection & Maturation Head start kids are more motivated and thus more likely to develop skills faster than the control condition Selection & Testing Head start kids are more motivated, so they figure out trick questions better leading to improved performance on the second test Selection & Instrumentation Perhaps observers in control group get bored and less likely to pick up on kids improvement Selection & Mortality Perhaps you lose more motivated (good) students from the control group to other preschool programs Selection & Regression If kids that are the furthest behind are assigned more often to the treatment group – they would show greater improvement

44 Work on CONFOUND WORKSHEETS


Download ppt "The Basics of Experimentation"

Similar presentations


Ads by Google