Experimental Research Design Causality & Validity Threats to Validity –Construct (particular to experiments) –Internal –External – already discussed

Threats to Construct Validity Mono Method bias –Remove common method variance –Establish convergent & discriminant validity Hypothesis guessing Inappropriate level of construct –Usually lower than what occurs in natural setting

Threats to Internal Validity Time –History, Maturation Measurement –Testing, Instrumentation Selection –Randomization, Statistical regression, Mortality Contamination –Diffusion, Compensation Interaction among threats

Internal Validity Establish that cause led to the effect –Identify non-experimental (uncontrolled) factors that affect the dependent variable or the relation b/w the independent and dependent variables

10 Threats to Internal Validity Time History –Factors occurring along with the independent variable An event occurring at the time of the intervention that overshadows (or reduces the effect of iv) and causes changes in dv –E.g., change in manufacturing material along with training influences performance quality

FIGURE 7.1 Copyright © 2003 John Wiley & Sons, Inc. Sekaran/RESEARCH 4E

10 Threats to Internal Validity Time Maturation –Biological or psychological processes or factors that occur w/the passage of time & affect the dependent variable E.g., job experience, socialization etc. affect performance

FIGURE 7.2 Copyright © 2003 John Wiley & Sons, Inc. Sekaran/RESEARCH 4E

10 Threats to Internal Validity Measurement Testing –Studies w/pre-post measures or multiple measures Increased or decreased sensitization to post- test measures Changes in participants due to measurement itself Increased fatigue or resentment Self-consistency effects

10 Threats to Internal Validity Instrumentation –Change in the measures from pre to post test Change in observers’ frames of reference Changes in participants’ frames of reference (e.g., beta & gamma changes p. 43 Saks) Change in instrumentation quality (e.g., performance quality measurement)

10 Threats to Internal Validity Instrumentation –Use of different measurement techniques Establish expected convergent & discriminant validity Importance of using different measures to overcome percept-percept problem

Threats to Internal Validity Selection Randomization –Randomly assign to experimental conditions Only in experimental designs Matching on key variable if random assignment not possible

Threats to Internal Validity Selection Randomization –Randomly select from population Calculate response rate Establish comparability of respondents to population on key features Stratified (v. cluster?) sampling

10 Threats to Internal Validity Selection Statistical Regression –Upon repeat measurement, scores usually “regress” or “move toward” the mean Usually when extreme scorers are selected into the study –Can be classified as ‘error’ variation

10 Threats to Internal Validity Selection Mortality –Features of participants who do not complete the study affect the results –Reduced power in testing effects

10 Threats to Internal Validity Contamination Diffusion/imitation of treatments –When people in control groups inadvertently know about or are affected by intervention and that changes their behavior

10 Threats to Internal Validity Contamination Compensatory Effects –Knowledge about intervention and changes in control group members’ behavior is more purposeful

10 Threats to Internal Validity Interaction among threats –When multiple threats operate at the same time obscuring the effect of the IV on the DV

Threats to Internal Validity Time-related –History, maturation Measurement –Testing, Instrumentation Selection –Randomization, Statistical regression, mortality Contamination –Diffusion, Compensation Interaction among threats