Lecture 6: Reliability and validity of scales (cont) 1. In relation to scales, define the following terms: - Content validity - Criterion validity (concurrent and predictive) - Construct validity (discriminant and convergent) - Responsiveness 2. Use of proxy respondents

Validity Depends on purpose: –screening: discrimination –outcome of treatment: responsiveness, sensitivity to change –prognosis: predictive validity

Content and face validity Judgment of “experts” and/or members of target population Does measure adequately sample domain being measured? Does it appear to measure what it is intended to measure? (eyeball test)

Criterion validity Criterion (“gold” standard) Concurrent criterion validity –e.g., screening test vs diagnostic test Predictive criterion validity –e.g., cancer staging test vs 5-year survival

Construct validity Is the theoretical construct underlying the measure valid? Development and testing of hypotheses Requires multiple data sources and investigations: –Convergent validity: measure is correlated with other measures of similar constructs –discriminant validity: measure is not correlated with measures of different constructs

Responsiveness of measures Ability to detect clinically important change over time or differences between treatments Requirement of evaluative measures Two approaches: –external responsiveness (validation against change in external criterion) –internal responsiveness: compute effect size of ratio of change score to measure of variability (different formulae)

Validity of Delirium Index (DI) Convergent validity –DI will be correlated with measures of current health/function: Current ADL disability (Barthel Index) Current severity of illness (clinical judgment and physiological severity score from APACHE)

Validity of Delirium Index (DI) Discriminant validity: –DI will be weakly correlated with measures of previous health/function: premorbid level of ADL disability severity of dementia comorbidity (prior)

Validity of DI (cont) DI score will predict probability of one-year survival DI score will be sensitive to: –changes in medication exposures –changes in environmental factors

Spearman correlation coefficients between Delirium Index and 3 baseline measures of current status

Spearman correlation coefficients between Delirium Index and 3 baseline measures of prior status

Delirium severity and survival Proportional hazards regression of delirium severity in delirium cohort Mean of 1st 2 DI scores Results –significant interaction: DI predicted survival in patients with delirium alone, not in those with dementia

Effects of medications and environmental factors Repeated in-hospital measures of DI, medications (medical record), and environmental factors (using checklist) Repeated measurements analysis performed to investigate associations of changes in risk factors with change in DI score.

Environmental risk factors for delirium Source: McCusker et al, JAGS 2001, 49:

Medications and DI Medication changes (increase in # medications, use of anticholinergic medications) were associated with increase in DI score.

Responsiveness of DI Internal responsiveness: ability to capture real change over time –effect size (change in mean DI/SD at baseline) –standardized response mean (change in mean DI/SD of change) External responsiveness: are changes in the DI correlated with other measures of change?

Internal responsiveness of the Delirium Index (DI) at 8 weeks

Validity of DI: Conclusions In patients with delirium +/- dementia: –Correlated with measures of current function –Responsive to change over time –Responsive to changes in environmental factors and medications In patients with delirium alone: –Predicts survival

Example: Inflammatory Bowel Disease Questionnaire (IBDQ) Disease-specific HRQoL measure 30 items covering 4 domains –bowel symptoms –systemic symptoms –emotional function –social function How to validate?

Validation of Inflammatory Bowel Disease Questionnaire Questionnaires administered to 42 patients with inflammatory bowel disease on 2 occasions, 1 month apart: –IBDQ –Global ratings of function –Global ratings of change by physician and a relative –Disease Activity Index – Emotional function scale from generic HRQoL instrument

Validation of IBDQ: predictions and results Correlation of 0.5 or more between patients’s global rating of change and IBDQ subscale on bowel symptoms (result: 0.42) Correlation of 0.3 or more between change in Disease Activity Index and IBDQ subscale on bowel symptoms (result: 0.33) Correlation of 0.3 or more between change in Disease Activity Index and IBDQ subscale on systemic symptoms (result: 0.04) Correlation of 0.5 or more between change in generic measure of emotional function and IBDQ subscale on emotional function (result: 0.76)