1. Reliability and Validity
Introduction to Study Skills & Research Methods (HL10040)
Dr James Betts

2. Lecture Outline: Definition of Terms Types of Validity
Threats to Validity
Types of Reliability
Threats to Reliability
Introduction to Measurement Error.

3. Some definitions…
Validity
Reliability
Objectivity

4. Types of Experimental Validity
Internal
External

5. Validity
Logical
Statistical

6. Logical Validity

7. Statistical Validity

8. Logical/Statistical Validity

9. Interesting Example: Breast Cancer
Incidence: ~1 % (0.8 %)
(i.e. a positive result should be detected for approximately in every 100 women tested)
Sensitivity: ~90 % (87 %)
(the mammogram is sensitive enough that approximately in every 100 breast cancer patients will receive a positive result)
Specificity: ~90 % (93 %)
(the mammogram is specific enough that approximately in every 100 healthy patients will receive a negative result).
Data from Kerlikowske et al. (1996)

10. Quick Test
What is the probability that a patient receiving a positive result actually has breast cancer?

11. Threats to Internal Validity

12. Threats to Internal/External Validity
e.g. O1 T O2
PLACEBO P O3 R O4

13. Threats to Internal/External Validity

14. Threats to External Validity

15. Reliability
Subject ml.kg-1.min ml.kg-1.min ml.kg-1.min-1
Subject ml.kg-1.min ml.kg-1.min ml.kg-1.min-1
Subject ml.kg-1.min ml.kg-1.min ml.kg-1.min-1

16. Reliability
Subject ml.kg-1.min ml.kg-1.min ml.kg-1.min-1
Subject ml.kg-1.min ml.kg-1.min ml.kg-1.min-1
Subject ml.kg-1.min ml.kg-1.min ml.kg-1.min-1

17. Reliability
Subject ml.kg-1.min ml.kg-1.min ml.kg-1.min-1
Subject ml.kg-1.min ml.kg-1.min ml.kg-1.min-1
Subject ml.kg-1.min ml.kg-1.min ml.kg-1.min-1

18. Types of Reliability
Subject ml.kg-1.min ml.kg-1.min ml.kg-1.min-1
Subject ml.kg-1.min ml.kg-1.min ml.kg-1.min-1
Subject ml.kg-1.min ml.kg-1.min ml.kg-1.min-1

19. Rater Reliability (Objectivity)
Intrarater reliability
Interrater reliability

20. Threats to Reliability

21. Measurement Errors
Ultimately, reliability is dependent on the degree of measurement error in a given study
The overall error in any measurement is comprised of both systematic and random error
We will address measurement error further next week…

22. Literature Search Assignment
The handout lists 8 questions which can be answered through retrieving the corresponding source articles
Answer as many as possible and hand them in at next week’s lecture
Does not contribute to final assessment but will be marked
DO NOT contact author or order articles.

23. Selected Reading
Atkinson, G. and A. M. Nevill. Statistical methods for assessing measurement error (Reliability) in variables relevant to sports medicine. Sports Medicine. 26: , 1998.
Holmes, T. H. Ten categories of statistical errors: a guide for research in endocrinology and metabolism. American Journal of Physiology. 286: E
Thomas J. R. & Nelson J. K. (2001) Research Methods in Physical Activity, 4th edition. Champaign, Illinois: Human Kinetics