Math Candel Maastricht University

1.Internal validity Do your conclusions reflect the “true state of nature” ? 2.External validity or generalizability Do your conclusions hold for other persons, situations, and times ?

1.Bad operationalisation of the constructs you want to measure 2.Incorrect design of the study 3.Errors in the statistical analysis 4.Wrong choice of the sample 5.Sample too small 6.Non-response, dropout and non-compliance

Does active coping lead to an increase in the quality of life as experienced by rheumatic patients ? If we carry out an empirical study, and the answer is yes, how sure can we be that this really is the case ?

How did we measure quality of life ? How good is this measure ? E.g. Sickness Impact Profile (SIP; 1989, 1994): –Emotional stability –Social behavior –Somatic Autonomy –Mobility control –Mobility range –Psychological autonomy

Social behavior (e.g.): I am cutting down the length of visits with friends I am doing fewer community activities I stay away from home only for brief periods of time Emotional stability (e.g.): I often act irritable toward those around me, for example, snap at people, give sharp answers, criticize easily I act disagreeable to family members, for example, I act spiteful, I am stubborn

Suppose we administer this scale on a different day, does it lead to the same scores on the different subscales ? (1) How reliable is the scale ? One could calculate a correlation between scores obtained at two different occasions: test-retest reliability Remedy for unreliability: Repeated measurements Motivation for multi-item measures !

(2) Are we sure that the scale is measuring what we would like to measure ? Does the scale have validity ? If the scale is reliable, this is still not guaranteed. The scale may still be invalid !

Content validity Is the scale representative of the domain to be measured ? Is there something missing ? Nomological or construct validity Does the scale behave as one would expect the scale to behave (e.g. on theoretical grounds) ? E.g. the score on SIP should correlate with the score on another quality of life measure

1.Respondent’s tendency to agree (or disagree) with each statement 2.Respondent’s tendency to give answers that are socially accepted 3.Respondent’s tendency to give extreme answers 4.Respondent’s tendency to bring some variation into his or her answers 5.Desire of the respondent to please the interviewers/researchers ……..

How good was the design of the study ? Are alternative explanations possible for the relations found ? Suppose the design of the study was: BaselineInterventionPostmeasurement What problems may be involved ?

Suppose the weather was good during the period between pre- and post-measurement, so there were less complaints related to the disease  History effects Rheuma is a progressive disease. One may expect a decrease in quality of life over time.  “Maturation”

Remedy: install a control group BaselineInterventionPostmeasurement BaselinePost measurement

Other effects that also may be under control: Testing effects  E.g. subjects are made aware of their functional health status and increased complaints may be due to pre-measurement Selection effects  The subjects are not assigned in a random fashion to the treatment and control group

Other possible threats: Diffusion of treatments  E.g. when patients are retrieved from the same hospital (rheumatology clinic) they may learn about each others therapy Remedy: Randomize at the level of hospitals

Observational studies: confounders ! E.g. the relation between smoking and blood pressure may also be explained by differences in physical exercise and differences in nutrition of smokers and non-smokers Remedies: Stratify on confounders, and perform analysis for each stratum Look for persons in both groups that match each other on confounders Include confounders as covariates in the analysis

Was the selection based on the variable that we would like to find an effect upon ? Watch out for regression towards the mean !

Quality of life before Quality of life after

Quality of life before Quality of life after

Quality of life before Quality of life after

Quality of life before Quality of life after Mean

Quality of life before Quality of life after Mean

Quality of life before Quality of life after Mean

Quality of life before Quality of life after Mean

Quality of life before Quality of life after Mean

Quality of life before Quality of life after Mean

Quality of life before Quality of life after Mean

In the treatment group not all people are strongly motivated to follow the program (and possibly to do some exercises at home) So a therapy may be effective but this may not be found in the study. Should we leave out the non-compliers ? Non-compliance: not all people are willing to follow the procedures as prescribed by a doctor or a program

The more often you do a statistical test, the larger the probability that you may find an effect, even if there is none The chance of a type I error may increase

Increase of type I error due to multiple testing

Select a sample that is large enough to be able to detect an effect (in an experiment) or a relation (in an observational study) If there is an effect or relation, but we are not able to detect it, we make a type II error

Relation between type II error and sample size

How can one maximize the external validity or the generalizability ? –Take a random sample from the population: best alternative, but not always feasible –Take a heterogeneous sample: stratified sampling: in case of interaction no average effects can be reported –Take a sample with subjects similar to modal persons for the strata: even per stratum there is some risk in interpreting relations

The target population is not reached: Are the participants different from the non- participants ? Are the people that dropout from the study different from the others ? Remedies: Minimize nonparticipation and dropout Obtain information on nonparticipation and dropout The target population is reached, but is one able to generalize to other target populations ?

Interaction between selected sample and treatment Interaction between time and treatment Interaction between setting and treatment Strategies to get insight into these interactions: -Replicate study with other groups, at other times and within other settings -Search literature for prior evidence