URBDP 591 A Lecture 16: Research Validity and Replication Objectives Guidelines for Writing Final Paper Statistical Conclusion Validity Montecarlo Simulation/Randomization Evaluating Empirical Research

Guidelines for Writing Final Paper Structure your paper around these key elements: Statement of the Research Problem:  Research Question: What is the central research question that you are investigating?  Include in your introduction a brief statement describing the importance of the topic.  Plan of the paper: Literature review: Past Research: Strengths and weaknesses of prior research you might discuss one or more of the following:  How existing research has overlooked or given inadequate attention to your topic.  How does your research improve upon the existing literature? Conceptual Model and Research Hypothesis Develop a conceptual model to describe your question and hypothesis. Define important concepts and describe how you plan to operationalize important variables. What do you expect to find? What are your expectations?

Research Methods: The actual design or your proposed research. What is the overall research method proposed: a. Sampling: What is the target population? What sampling procedures will you use? b. Research Design: What general type of research design is most appropriate for your study? c. Measurement: List all relevant variables - how are they related to the concepts used in the conceptual/theoretical framework? Describe and evaluate information on measurement reliability and validity. d. Statistical Analysis: What specific statistical analysis techniques will be used in your research?

Expected Results Explain how your chosen strategy will answer your questions. Why the methodology selected is most appropriate. Limitation Are any relevant shortcomings/limitations ? Discuss limitations of your approach. Reference Complete list of references. Remember to add Page number for quotations. Format for quotations Ben-Akiva, M. and Bowman, J. L. (1998) Integration of an Activity-based Model System and a Residential Location Model. Urban Studies 35,

WRITING TIPS  Use simple and direct writing.  Be concise  Avoid jargon  Use active form  Spell check your paper For additional information on writing see: The Elements of Style, Fourth Edition by William Strunk Jr., and E.B. White. Available also on-line.

Research Question Hypothesis Formation Based on Lit. Research., propose some “new knowledge” Research Design Determine how to obtain the data to test the RH: Data Collection Carrying out the research design and getting the data. Data Analysis Data collation and statistical analysis Hypothesis Testing Based on design properties and statistical results Draw Conclusions Decide how your “new knowledge” changes “what is known” about a certain phenomenon the “Research Loop” Theory Development

Applying the Research Loop The “research loop” is applied over and over, in three ways… Testing a research hypothesis –The first test of a research hypothesis -- using the “best” design you can Replication –being sure your conclusions about a particular hypothesis are correct by repeating exactly the same research design –the main purpose of replication is to acquire confidence in our methods, data and resulting conclusions Convergence –testing variations of the research hypothesis: using variations of the research design (varying population, setting, task, measures and sometimes the data analyses –the main purpose of convergence is to test the limits of the “generalizability” of our results what design/analysis changes lead to different results?

Conclusion Validity Statistical Conclusion Validity The extent to which an effect has occurred above chance levels and how well the investigation can detect a difference (or relationship) that actually exists.

Elements of Conclusion Validity: Sufficient Power. Reasonable evidence to find that that the cause and effect covary. Significant covariation.

Threats to Conclusion Validity Low statistical power Violated assumptions of statistical tests Fishing and the Error Rate problem Low reliability of measures Poor reliability of treatment implementation

Hypothesis testing Step 1: Set up hypothesis –you should determine whether it is 1-tailed or 2-tailed test Step 2: Compute test statistics Step 3: Determine p-value of the test statistic –for a pre-determined alpha, you can find the corresponding critical limit Step 4: Draw conclusion –reject H 0 if p-value < alpha (ie greater than the critical limit) –accept H 0 if p-value > alpha (ie less than the critical limit)

A statistical hypothesis is a statement about the parameters of a probability distribution. Null hypothesis H o :  1 =  2 Alternative hypothesis H a :  1   2 11 22  is the mean of a distribution

The Four Components to a Statistical Conclusion the number of units (e.g., people) accessible to study the effect of the treatment or independent variable relative to the noise the odds the observed result is due to chance the odds you’ll observe a treatment effect when it occurs sample size effect size alpha level power

Statistical significance vs practical importance Significance and importance –A statistically significant result is one which we can be confident is real and reproducible - not just the result of random variation. –Whether the observed discrepancy (the effect size) is large enough to have any practical importance is quite a different matter. If the sample size is too small... –The test may not have sufficient power to establish whether a difference is significant, even though the difference may be large enough to have substantial practical consequences if it is “real”. If the sample size is unnecessarily large... –Differences may be established as statistically significant, and yet be too small to have any practical consequences. The optimum sample size… –is just large enough to detect differences of a size which the researcher believes to be of practical importance. This firstly involves a professional assessment of how large a difference is important, followed by a power analysis to determine the required sample size.

Randomization Testing: Method of testing hypotheses and sometimes also determining confidence intervals for parameters. Randomization tests determine the significance level of a test statistic obtained from a set of data by comparing the statistic with a randomization distribution. What is a randomization distribution? It is a distribution of test statistic values that is obtained by randomly reordering the observed data values (without replacement) until all possible permutations are obtained. Most hypotheses of interest are alternatives to a null hypothesis of randomness. A randomization test tells us how likely it is that a certain pattern in the data arose by chance. Randomization Tests

Original data Randomly permuted data 95%5% Threshold Test statistic under Null Hypothesis Replicate Distribution of test statistic Distribution of test statistic Hypothesis Testing Hypothesis Testing Significance thresholds based on Permutation test

The basic approach in a randomization test is to calculate a test statistic from the observed data, and then randomly reshuffle the data a large number of times, recalculating the test statistic for each iteration. Randomization Test

These statistics are used to generate a distribution of values, and the observed value is compared to the distribution to see whether the observed case is an event that was unlikely to have occurred through chance; that is, if it is a tail value from the distribution. Randomization Test

Randomization techniques allow the determination of the significance of an observed test statistic by comparing it to the distribution of values obtained by randomly reordering the data. We ask: Is the observed value an unusually large or small value compared to what might occur by chance? This is typically done in one of two ways: 1) In some cases, random data values (e.g., spatial positions or other measurements on individual subjects) are generated as random values. 2) In comparisons between groups, only the group memberships are randomized while the same set of measurements are maintained. These tests sometimes are referred to as permutation procedures (the randomization is done by reordering the positions of elements in a array). Randomization Techniques

How to do a randomization test: 1.Identify hypotheses (null and alternate). 2.Choose a test statistic. 3.Compute the test statistic for the original data. 4.Resample. Recompute the statistic for each of the ‘new samples.’ When you stop resampling, you will have a randomization distribution. 5. Accept or reject the null hypothesis by comparing the original test statistic to the randomization distribution. Randomization Test

Evaluating Empirical Research: A Synthesis - Identify Key Variables and Design Variables and their measurement scales Research Hypotheses, Approaches, and Design Measurement Reliability and Validity - Evaluate Research Validity Measurement Reliability and Statistics Internal Validity Equivalence of Groups Control of External Variables Measurement Validity External Validity Population Validity Ecological Validity - Issues of Interpretation