Independent vs Dependent Variables PRESUMED CAUSE REFERRED TO AS INDEPENDENT VARIABLE (SMOKING). PRESUMED EFFECT IS DEPENDENT VARIABLE (LUNG CANCER). SEEK TO EXPLAIN DEPENDENT VARIABLE. DIRECTIONALITY IS STATED OR IMPLIED. SMOKING  LUNG CANCER.

Internal Validity IS THERE A CAUSAL RELATIONSHIP BETWEEN INDEPENDENT VARIABLE & DEPENDENT VARIABLE (COOK & CAMPBELL, 1979, P.38). TYPE I ERROR: REJECTING NULL WHEN IT IS TRUE (FALSE POSITIVE). CONCLUDE TREATMENT MORE EFFECTIVE THAN CONTROL WHEN REALLY SAMPLING FLUCTUATIONS ARE RESPONSIBLE.

Internal Validity FINDINGS SHOWN TO RESULT ONLY FROM EFFECT OF INDEPENDENT VARIABLE & CAN’T BE INTERPRETED AS REFLECTING EFFECTS OF EXTRANEOUS VARIABLES. SUBJECTS’ PERFORMANCE ON DEPENDENT VARIABLE CAN ONLY RESULT FROM EFFECT OF INDEPENDENT VARIABLE.

Internal Validity IS INDEPENDENT VARIABLE REALLY RESPONSIBLE FOR DIFFERENCES DETECTED? OR ARE THERE COMPETING HYPOTHESES THAT WERE NOT ADDRESSED OR MEASURED. THREATS TO INTERNAL VALIDITY SHOULD BE ADDRESSED PRIOR TO THE STUDY.

Threats to Internal Validity HISTORY MATURATION TESTING INSTRUMENTATION STATISTICAL REGRESSION SELECTION MORTALITY

Threats to Internal Validity 1. HISTORY: IN ADDITION TO INDEPENDENT VARIABLE, ANOTHER EVENT MAY HAVE EFFECTED DEPENDENT VARIABLE BETWEEN 1ST & 2ND MEASUREMENT. ADVERTISEMENT. MEDIA COVERAGE.

Threats to Internal Validity 2. MATURATION: REFLECTS PASSAGE OF TIME NOT EXPERIMENT. EFFECTS OF AGE ON DEPENDENT VARIABLE. AS YOU GET OLDER YOU BECOME WISER. STRENGTH ON ATHLETICISM. PALLIATIVE CARE IN 1ST YEAR OF PROGRAM.

Threats to Internal Validity 3. TESTING: FACT THAT YOU ALREADY TOOK TEST MAY LEAD TO  SCORE WHEN YOU TAKE TEST AGAIN. TEST MAY INFLUENCE OUTCOME, NOT INDEPENDENT VARIABLE. EXPERIENCE GAINED THROUGH REPEAT TESTING.

Threats to Internal Validity 4. INSTRUMENTATION: CHANGES IN MEASUREMENT OF VARIABLES MAY ACCOUNT FOR DIFFERENCES FOUND. CALIBRATION OF MEASUREMENT (DIFFERENT TYPES OF EQUIPMENT, DIFFERENT WAYS OF DIAGNOSING, DIFFERENCES IN RATERS.)

Threats to Internal Validity 5. STATISTICAL REGRESSION (REGRESSION TO THE MEAN) GROUPS SELECTED FOR EXTREME SCORES WILL, ON REPEAT TESTING WITH SAME OR SIMILAR TEST, HAVE SCORES CLOSER TO MEAN SCORE (BECAUSE THE MORE DEVIANT THE SCORE, THE LARGER MEASUREMENT ERROR).

Threats to Internal Validity 6. SELECTION BIAS: BIAS IN SELECTING SUBJECTS IS RESPONSIBLE FOR RESULTS, NOT INDEPENDENT VARIABLE. SMOKING CESSATION CLINIC. PSYCHOLOGY STUDENTS.

Threats to Internal Validity 7. MORTALITY: LOSS OF SUBJECTS FOR ANY REASON. THOSE WHO ARE LEFT ARE DIFFERENT THAN THOSE WHO REMAIN & THESE DIFFERENCES ARE RESPONSIBLE FOR THE OUTCOME, NOT THE INDEPENDENT VARIABLE.

Threats to Internal Validity MUST BE ADDRESSED IN THE DESIGN PHASE PRIOR TO IMPLEMENTING THE PROJECT.

External Validity GENERALIZABILITY OF FINDINGS TO OTHER SETTINGS OR SAMPLES WILL THIS WORK IN ANOTHER SETTING WITH DIFFERENT SUBJECTS?

Threats to External Validity 1. SAMPLING 2. HAWTHORNE EFFECT 3. NOVELTY EFFECTS 4. INTERACTION BETWEEN HISTORY & TREATMENT 5. EXPERIMENTER EFFECTS

Threats to External Validity 1. SAMPLING LACK OF RANDOM SELECTION PROCEDURES MAKES IT NOT POSSIBLE TO GENERALIZE TO OTHER POPULATIONS. PROBABILITY SAMPLING. CONVENIENCE SAMPLE OF UNIVERSITY STUDENTS OUTSIDE LIBRARY.

Threats to External Validity 2. HAWTHORN EFFECT EFFECT OF BEING OBSERVED CAUSED CHANGES IN BEHAVIOR NOT INTERVENTION. KNOWLEDGE OF BEING IN STUDY CAUSES PEOPLE (BOTH SUBJECTS AND RESEARCHERS) TO CHANGE THEIR BEHAVIOUR.

Threats to External Validity 3. NOVELTY EFFECTS WHEN TREATMENT IS NEW BOTH SUBJECTS & RESEARCHERS MAY ALTER BEHAVIOUR; ENTHUSIASM, HOPE. OVER TIME, THESE EFFECTS DISAPPEAR

Threats to External Validity 4. INTERACTION BETWEEN HISTORY & TREATMENT SOME EXTERNAL EVENTS MAY INTERACT WITH TREATMENT. DIETARY REDUCTION IN CHOLESTEROL AT SAME TIME AS OAT BRAN &  CHOLESTEROL.

Threats to External Validity 5. EXPERIMENTER EFFECTS PERFORMANCE OF SUBJECTS IS INFLUENCED BY CHARACTERISTICS OF RESEARCHER. RESULTS MAY NOT BE SAME WITH OTHER RESEACHERS.

Threats to External Validity MUST BE ADDRESSED IN THE DESIGN PHASE PRIOR TO IMPLEMENTING THE PROJECT.

INTERNAL VERSUS EXTERNAL VALIDITY? TRY TO ADDRESS BOTH. IF THERE IS CONFLICT, OPT FOR STRONEGR INTERNAL VALIDITY AND HANDLE EXTERNAL VALIDITY WITH REPLICATIONS. IF FINDINGS NOT INTERNALLY VALID, CAN’T POSSIBLE BE EXTERNALLY VALID.

RELIABILITY RELIABILITY OF MEASUREMENT INSTRUMENT IS THE DEGREE OF CONSISTENCY WITH WHICH IT MEASURES WHAT IT PURPORTS TO BE MEASURING. WEIGHT ON SCALE.

RELIABILITY ARE {EXAM} SCORES OBTAINED FOR YOUR SAMPLE TRUE? DO THEY REFLECT THE TRUE DIFFERENCES IN SUBJECTS?

RELIABILITY ERROR VARIANCE (MEASUREMENT ERROR) IS THE EXTENT OF VARIABILITY IN TEST SCORES DUE TO ERROR RATHER THAN TRUE MEASUREMENT.

RELIABILITY NON-SAMPLING MEASUREMENT ERROR. TOTAL ERROR SAMPLING ERROR

RELIABILITY OBSERVED SCORE = TRUE SCORE + MEASUREMENT ERROR. MEASUREMENT ERROR CAN BE DUE TO CHANCE OR RANDOM ERROR = RELIABILITY. CAN ALSO BE SYSTEMATIC ERROR = VALIDITY.

RELIABILITY RELIABILITY: EXTENT TO WHICH INSTRUMENTS YIELDS THE SAME RESULTS ON REPEATED MEASURES. CONCERNED WITH CONSISTENCY, ACCURACY, PRECISION, STABILITY, EQUIVALENCE AND HOMOGENEITY

RELIABILITY STABILITY: ABILITY TO PRODUCE THE SAME RESULTS WITH REPEATED TESTING. TEST-RE-TEST RELIABILITY

RELIABILITY HOMOGENEITY; INTERNAL CONSISTENCY ALL ITEMS IN THE TOOL MEASURE THE SAME CONCEPT. IQ. SPLIT-HALF RELIABILITY, KUDER- RICHARDSON, CRONBACH’S ALPHA

RELIABILITY EQUIVALENCE: TOOL PRODUCES THE SAME RESULTS WHEN PARALLEL TESTS ARE USED. DIFFERENT IQ TESTS. DIFFERENT OBSERVERS, RATERS USING INSTRUMENT TO MEASURE SAME PHENOMENA AT SAME TIME. INTER-RATER RELIABILITY

RELIABILITY EQUIVALENCE: PERCENT AGREEMENT RELIABILITY COEFFICIENT KAPPA, WEIGHED KAPPA