1. Reliability and Validity
Sa’ed H. Zyoud

2. The error of research
Random error (Chance) (increase sample number) (Precision: free from RE)
Precision is the degree to which repeated measurements under unchanged conditions show the same results
Systematic error (Bias) (matching+ blinding) (Accuracy: free from SE)
Accuracy is the degree of closeness of measurements of a quantity to that quantity's actual (true) value
Sampling error (type 1 error, type 2 error)
Measurement error (wrong stat, wrong in scoring)

4. Reliability Means "repeatability" or "consistency".
A measure is considered reliable if it would give us the same result over and over again (assuming that what we are measuring isn't changing!).
There are four general classes of reliability estimates, each of which estimates reliability in a different way.

5. Reliability (continued)
Inter-Rater or Inter-Observer Reliability
Intra-Rater or Intra-Observer Reliability (Test-Retest Reliability)
Internal Consistency Reliability
Inter-method reliability.

6. Inter-Rater or Inter-Observer Reliability
Used to assess the degree to which different raters/observers give consistent estimates of the same phenomenon
(i.e. is the variation in measurements when taken by different persons but with the same method or instruments.

7. Establish reliability on pilot data or a subsample of data and retest often throughout.
There are a number of statistics which can be used to determine inter-rater reliability.
Cohen's kappa, intra-class correlation.

8. Test-Retest Reliability
Used to assess the consistency of a measure from one time to another.
This approach assumes that there is no substantial change in the construct being measured between the two occasions.
The amount of time allowed between measures is critical.
intra-rater reliability is the degree of agreement among multiple repetitions of a diagnostic test performed by a single rater

9. Internal Consistency Reliability
Used to assess the consistency of results across items within a test.
We are  looking at how consistent the results are for different items for the same construct within the measure.
Example  
"I like to eat bran bread“
“I've enjoyed eating bran bread in the past”
“I hate bran bread” -

11. Kinds of Internal Reliability
There are a number of statistics which can be used to determine Internal Reliability.
Inter-item Correlation
Cronbach's Alpha (a)

12. cronbach's alpha ----------------------Internal consistency
α ≥ Excellent.
9 > α ≥ Good.
8 > α ≥ Acceptable.
7 > α ≥ Questionable and Acceptable.
6 > α ≥ Poor.
5 > α Unacceptable

13. Inter-method reliability- is the variation in measurements of the same target when taken by a different methods or instruments, but with the same person

14. Validity
Validity is the degree of closeness of measurements of a quantity to that quantity's actual (true) value
(i.e. Does it measure what you think it measures.
This is more familiarly called Construct Validity.

15. Types of Construct Validity
Translation validity
Face validity
Content validity
Criterion-related validity (Known groups validity)
Predictive validity
Concurrent validity
Convergent validity
Discriminant validity

16. Translation validity

17. Face Validity
“On its face" does it seems like a good translation of the construct.
i.e. Does it appear to measure what it is supposed to measure?
Weak Version: If you read it does it appear to ask questions directed at the concept.
Strong Version: If experts in that domain assess it, they conclude it measures that domain.

18. Content Validity
How well elements of the test relate to the content domain?
For example, a depression measure should cover the checklist of depression symptoms

19. Criteria-Related Validity (Known groups validity)
involves the correlation between the test and a criterion variable (or variables)

20. Predictive Validity
A high correlation would provide evidence for predictive validity –
i.e. it would show that our measure can correctly predict something that we theoretically thing it should be able to predict.

21. Concurrent Validity
Assess the operationalization's ability to distinguish between groups that it should theoretically be able to distinguish between.
As in any discriminating test, the results are more powerful if you are able to show that you can discriminate between two groups that are very similar.

22. Convergent Validity
Examine the degree to which the operationalization is similar to (converges on) other operationalizations that it theoretically should be similar to.
To show the convergent validity of a test of arithmetic skills, one might correlate the scores on a test with scores on other tests that purport to measure basic math ability, where high correlations would be evidence of convergent validity.

23. Discriminant Validity
Examine the degree to which the operationalization is not similar to (diverges from) other operationalizations that it theoretically should be not be similar to.
To show the discriminant validity of a test of arithmetic skills, we might correlate the scores on a test with scores on tests that of verbal ability, where low correlations would be evidence of discriminant validity.

24. Conclusion validity

26. Differentiate between prevalence and incidence
Prevalence: The total number of cases of a disease in a given population at a specific time
Incidence:  the number of new cases of a specific disease occurring during a certain period in a population at risk.

27. odds ratio
odds ratio: the ratio, used particularly in case-control studies or cohort study (exposed versus non-exposed), estimates the chances of a particular event occurring in one population in relation to its rate of occurrence in another population.
Odds ratio= a/b / c/d
Odds ratio= a*d/c*b a b
a/b c d
c/d

28. a b
a/b c d
c/d
Relative risk is a ratio of the probability of the event occurring in the exposed group versus a non-exposed group
Here, a = 20, b = 80, c = 1, and d = 99. Then the relative risk of cancer associated with smoking would be
Smokers would be twenty times as likely as non-smokers to develop lung cancer.

29. Smokers are ???? times more likely to have lung cancer than non-smokers