1. Lecture 2 Outline: Thu, Jan 15
Chapter 1.2: Statistical Inference and Study Design
Types of Inference
Observational Studies vs. Randomized Experiments
Confounding Variables
Randomized Experiments
Inference to Populations: random sampling studies vs. non-random sampling studies

2. Drawing Conclusions
An inference is a conclusion from the data about some broader context that the data represent.
e.g., one egg in a container is rotten -- the rest are rotten; when we flick on a light switch, the light turns on -- flicking on the light switch will generally cause the light to turn on.
A statistical inference is an inference justified by a probability model linking the data to a broader context. Statistical inferences include measures of uncertainty about the conclusions (e.g., p-values, confidence intervals)

3. Two “broader contexts” in statistics
Population inference: an inference about population characteristics, like the difference between two population means
Causal inference: an inference that a subject would have received a different numerical outcome had the subject belonged to a different group.

4. Causal Questions
Medicine: How effective is a new drug? What is the effect of smoking on one’s chance of developing cancer?
Psychology: What change in an individual’s normal solitary performance and behavior occurs when people are present? What changes in an individual’s moral behavior occur when the individual is commanded by authority?
Economics: What is the effect of a change in taxes on labor supply and investment behavior? What is the effect of a change in the minimum wage on employment?
Education: What is the effect of smaller class sizes on achievement?

5. Types of Causal Studies
Observational study: Study in which group status is observed, i.e., beyond the control of the researcher.
Controlled experiment: Study in which group status is controlled by the researcher.
Randomized experiment: Study in which group status is assigned by a chance mechanism.

6. Examples of Causal Studies
Motivation and creativity study (case study 1.1.1)
Sex discrimination study (case study 1.1.2)?
How much health damage does atomic bomb cause? -- comparison of chromosomal aberrations of Japanese atomic bomb survivors near blast and those far from blast.
How many deaths does being a solider in a war (prevent or) cause? -- comparison of death rates in Navy and out of Navy during Spanish American war.
How many heart attacks does taking estrogen (prevent or) cause? -- Comparison of heart attack rates of menopausal women taking estrogen and women not taking estrogen

7. Causal Inference
Main point: statistical inferences of causation can be made from randomized experiments, but not from observational studies.
In an observational study, one cannot rule out the possibility that confounding variables are responsible for group differences in the observed outcome.
In an observational study, one cannot rule out the possibility of reverse causality or simultaneous causality. Which came first – the chicken or the egg? Beta-carotene intake and morbidity.

8. Confounding Variables
A confounding (lurking) variable is a variable that is related to both group membership and the outcome. Its presence makes it hard to establish the outcome as being a direct consequence of group membership.
Examples:
Sex discrimination study
Death rates in and out of Navy study
Estrogen study
Although it is possible to control for known confounding variables (via multiple regression), in an observational study we can never be sure that there are not unknown confounding variables that are responsible for group differences in outcome.

9. Association Is Not Causation
There is a close relationship between the salaries of Presbyterian ministers in Massachusetts and the price of rum in Havana. Are the ministers benefiting from the rum trade or supporting it?
A study showed that cigarette smokers have lower college grades than non-smokers. Does the road to good grades lie in giving up smoking?

10. Do Observational Studies Have Value – Yes!
Establishing causation is not always the goal (prediction may be the goal)
Establishing causation may be done in other ways.
Experiments not always practical or ethical
Analysis of observational data may lend evidence toward causal theories and suggest the direction of future research.

11. Criteria for Establishing Causation From Obs. Studies
The association is strong.
The association is consistent.
Higher doses are associated with stronger responses.
The alleged cause precedes the effect in time.
The alleged cause is plausible.
Examples:
Smoking and lung cancer
Radiation from atomic bomb and chromosomal aberrations

12. Randomized Experiments
In a randomized experiment, an impersonal chance mechanism is used to assign the units to groups.
In a randomized experiment, any relationship between important variables and group membership can only arise through chance.
Suppose that there is a treatment and control group and the treatment group has a higher observed response than the control group.
In a randomized experiment, the difference must be due to either to the treatment or to the play of chance in the random assignment of units to the groups.
Statistical inference provides a method for describing how confident we can be that an observed difference between the treatment and control groups did not arise due to chance.

13. Statistical Inference in the Motivation-Creativity Study
The creativity scores tended to be larger in the “intrinsic” than in the “extrinsic” group. Either the intrinsic questionnaire caused a higher score or else the more creative writers happened to be placed in the “intrinsic” group. The probability (p-value) associated with this latter possibility is
Moderate to convincing evidence that taking the intrinsic question in fact caused writers to be more creative.

14. Law of Large Numbers and Replication
The Law of Large Numbers: Draw independent observations at random from any population with finite mean . Decide how accurately you would like to estimate As the number of observations drawn increases, the mean of the values eventually comes as close to the mean as you specified and stays that close.
The law of large numbers guarantees that if enough units are used, any important variables (whether we are aware of them or not) will be divided roughly equally between the two groups.

15. Random Assignment in JMP
To randomly assign units to two groups of size n_1 and n_2 in JMP:
Create a column called “random.” Right click on the top of this column, click on formula, click on the random function and then click on Random Uniform.
Click on Tables, Sort and then sort by random.
Create a column “group.” Label the first n_1 units in the table as Group I and the rest of the units as Group II

16. Inference to Populations
Goal: Make conclusions about aspects of a population (e.g., mean income in U.S.) based on a sample.
Two types of sampling designs.
Random sampling study: Units are selected by the investigator from a well-defined population through a chance mechanism with each unit having a known (>0) chance of being selected.
Non-random sampling study: Units selected in way other than through chance (e.g., units selected by taking volunteers)

17. Simple Random Sample
Simple random sample of size n: Subset of population of size n selected in such a way that every subset of size n has same chance of being selected.
Equivalent to drawing units out of a hat without replacement.
Simple random sample in JMP:
Click on Tables, Subset, then put the number n in the box “Sampling Rate or Sample Size.”

18. Random Samples and Statistical Inference
Statistical inference about the population can be made based on the sample for random sampling studies (Ch ) by using the sampling design.
The sample might turn out to be nonrepresentative (i.e., have markedly different characteristics than the population) but:
We can describe the uncertainty due to the chance mechanism of random sampling accurately because we know how the sample was generated (Chapter 1.4.1).
The law of large numbers guarantees that if we take a large enough sample, the sample mean (and other characteristics) will be very close to the population mean.

19. The Literary Digest Poll
The Literary Digest Poll. In the 1936 presidential election, the Literary Digest predicted an overwhelming victory for Landon over Roosevelt.
Roosevelt won the election by a landslide – 62% to 38%. What went wrong?
The sample was taken by mailing questionnaires to 10 million people whose names and addresses came from sources like telephone books and club membership lists million peopled returned the samples.

20. Biased Samples
Selection Bias: When the procedure for selecting a sample results in samples that are systematically different from the population.
When a selection procedure is biased, taking a large sample does not help. This just repeats the basic mistake on a large scale.
Causes of selection bias:
Voluntary Response Sample
Undercoverage
Nonresponse

21. Statistical Inferences Permitted by Study Designs
Display 1.5
Examples:
Motivation and Creativity study
Sex Discrimination study
Researchers measured the lead content in teeth and IQ scores for all 3,229 children attending first and second grade between 1975 and 1978 in Chelsea and Somerville, Mass. IQ scores for those with low lead concentrations found to be significantly higher than for those with high lead concentrations.
Conceptual Exercises 1-12 in Ch. 1 relate to statistical inferences permitted by study designs.

22. Summary
“Random samples and randomized experiments are representative in the same sense that flipping a coin to see who takes out the garbage is fair.” (Chapter 1.5.5)
Two key advantages of random samples and randomized experiments over nonrandom samples and observational studies are the following:
Uncertainty about representativeness can be incorporated into the statistical analysis. If randomization were abandoned, there would be no way to express uncertainty accurately.
Law of large numbers guarantees that if we have a large sample size, we will get accurate results. Without using randomization, our results could be systematically biased even for a large sample.