1 Identifying and Selecting Measures for Health Disparities Research Anita L. Stewart, Ph.D. University of California, San Francisco Clinical Research with Diverse Communities EPI 222, Spring April 9, 2009

2 Selecting Measures for Your Own Study: The Problem u You are beginning a study u You know the concepts (variables) of interest u Question: Which measure of ________ should I use? »A popular measure »One that a colleague used successfully »Create your own

3 Inappropriate Measures can Result in: u Conceptual inadequacy –Measuring wrong concept u Poor data quality (e.g. missing data) u Poor variability u Poor reliability and validity u Inability to detect true associations –e.g., no measured change in outcome when change occurred

4 Selecting Measures for Your Research u Goal: find a measure of your concept that has been developed using stringent measurement development methods u Your task: find measures and review them for all steps in measurement development process

5 Process of Selecting Good Measures for Your Studies Define concept (variable) Specify study context Review measures’ properties --conceptual and psychometric adequacy Pretest best 1-2 measures Select final measure Identify potential measures

6 Process of Selecting Good Measures for Your Studies Define concept (variable) Specify study context Review measures’ properties --conceptual and psychometric adequacy Pretest best 1-2 measures Select final measure Identify potential measures

7 Concept/Construct u A variable that is relatively abstract as opposed to concrete –e.g. health status, stress, acculturation vs. height, body temp u An abstraction based on observations of certain behaviors or characteristics u Cannot be assessed directly

8 Measures of Concepts u Concepts are defined and operationalized in terms of observed indicators or measures u Measures are “proxies” for the latent variables we cannot directly observe

9 Define Concept For Your Study u Define concept from your perspective, taking into account your… –study questions –target population u For outcomes, describe: –how intervention or independent variables might affect it –specific types of changes you expect

10 Example of Concept: Interpersonal Processes of Care u Interpersonal processes between physicians and patients –Communication, decision making, respectfulness, empathy –Emphasis on processes relevant to vulnerable patients

11 Purpose of Measuring Concept u Describe how your concept fits into your research question(s) u Outcome measure? u Determinant of health? u Identify need for intervention?

12 Example: Interpersonal Processes of Care u Interpersonal processes of care may influence patient outcomes –Proximal outcomes »Knowledge of condition and recommendations »Adherence to recommendations »Patient satisfaction –Ultimate outcomes »Health status

13 Process of Selecting Good Measures for Your Studies Define concept (variable) Specify study context Review measures’ properties --conceptual and psychometric adequacy Pretest best 1-2 measures Select final measure Identify potential measures

14 Specify Context for Measures u Study characteristics affecting choice of measures –Nature of target population –Practical constraints u Nature of population (patients) –Lower educational level? Limited literacy? –Healthy or ill?

15 Practical Constraints u Preferred mode of administration u Acceptable respondent burden u Budget to pay for measures u Translations needed u Method of data entry u Time frame – time to select and pretest measures

16 Process of Selecting Good Measures for Your Studies Define concept (variable) Specify study context Review measures’ properties --conceptual and psychometric adequacy Pretest best 1-2 measures Select final measure Identify potential measures

17 Locate Potential Measures u Identify candidate measures for all concepts u Redundancy OK for now u DO NOT develop your own questions unless it is absolutely necessary

18 Locating Measures u For major dependent or independent variables –Multi-item measures with known psychometric properties u For background variables and covariates –Standardized survey measures

19 Locating Measures: Possible Sources u Compendia u Organizations and research centers u Government agencies u National and state surveys u Large relevant research studies

20 Helpful Handouts to Locate Measures u CADC measurement core website –Locating measures for health disparities research –Measures compilations and reviews u Summary of surveys – National Center for Health Statistics u Instructions for accessing questionnaires from several national and state health surveys

21 Handout: Locating Measures for Health Disparities Research u To link to websites, need to log on to CADC website

22 Locating Measures: Compendia u Specific measures of various concepts are compiled, reviewed, listed, or otherwise provided –Books SEE HANDOUT »Many books review and critique various measures –Web »A few websites

23 Compendia by Web u National Cancer Institute website u Health behavior constructs: theory, measurement, and research –Reviews concepts and measures of constructs such as perceived control, social support, and perceived vulnerability

24 Locating Measures: Organizations and Research Centers u Some organizations and research centers specialize in measurement and provide public access websites

25 RAND Health Program u Measures, scoring manuals, and citations u Specialty: –Quality of care, patient satisfaction –Health-related quality of life »Generic and disease specific –Medical Outcomes Study (MOS) measures –Adults and children (surveys and tools)

26 Ottawa Health Decision Centre u Patient and physician decision aids u Patient measures, e.g., decisional conflict, decisional regret, stage of decision making, decision self-efficacy

27 Commonwealth Fund Surveys u Health insurance u Medicare u Health care quality, patient centered care, underserved populations u Child health development, care of the elderly u State health policy, international health policy

28 Commonwealth Survey of Physicians u 2006 International Health Policy Survey of Primary Care Doctors –Use of information technology –Access to care –Availability of financial incentives –Chronic care management

29 MacArthur Research Network on Socioeconomic Status and Health u Reviews measures in several domains: –Psychosocial –Social and physical environment –Socioeconomic status (SES) »SES across the lifecourse u MacArthur Network on SES and Health

30 Duke University Center for Demographic Studies u National Long Term Care Survey (with NIA) u Prevalence and patterns of functional limitations, medical conditions, recent medical problems, use of health care, and housing and neighborhood characteristics

31 Roadmap K12 Data Resource Center u Links to over 2 dozen national and state surveys u Provides overview for each: domains, time frame, population, scope, sample size, and contacts cDataSetResources/

32 Locating Measures: Government Agencies u Several federal and state government agencies provide measures for use in health and health disparities research –Information about measures (e.g. clearinghouse) –Actual measures

33 Agency for Healthcare Research and Quality (AHRQ) National Quality Measures Clearinghouse

34 Agency for Healthcare Research and Quality (AHRQ) u Medical Expenditure Panel Survey (MEPS) – u Consumer Assessment of Health Plans Survey (CAHPS) –

35 Department of Veterans Affairs u National Survey of Veterans, u Survey of Veteran Enrollees’ Health and Reliance Upon VA, u Both include measures of demographics and socioeconomic status, military background, health, health insurance, and VA and non-VA benefits usage.

36 National Cancer Institute u Special initiatives on measures u Health Information National Trends Survey (HINTS) –Compiled cancer screening questions, identified best ones, conducted extensive pretesting

37 Centers for Disease Control and Prevention u Behavioral Risk Factor Surveillance System –Surveys of health behaviors

38 National and State Surveys u Good sources of descriptors, covariates u Usually don’t have multi-item measures

39 CDC: National School-Based Youth Risk Behavior Survey (YRBS) u Survey conducted every other year –random national sample of youth in grades 9-12 –Most states conduct survey u Measures of substance use, risky sexual behaviors, diet, physical activity, overweight

40 California Health Interview Survey (CHIS) u A behavioral risk surveillance survey modeled after the CDC BRFSS –Numerous languages u –Go to “Methodology – review questionnaires”

41 National Center for Health Statistics (NCHS) u Surveys and data collection systems u Can download –Any survey or portion of survey u Handouts: –Summary of surveys –Instructions for accessing questionnaires

42 NCHS National Health Care Surveys: Surveys of Physicians u Family of provider-based surveys u Provide objective, reliable information about –organizations and providers –services rendered –patients they serve

43 Locating Measures: Large Research Studies u Many large-scale, multi-center and longitudinal studies have developed and used measures on health-related topics u Increasingly, they are posting these on “study” websites

44 Study of Women’s Health Across the Nation (SWAN) u Physical, biological, psychological, and social changes of women during their middle years u Questionnaire can be downloaded

45 Sacramento Area Latino Study on Aging (SALSA) Study u NIA funded longitudinal study of Latinos in the Sacramento region u Started in 1996 –each person followed for up to 5 years

46 Locating Measures: Finding Authors of Measures u Published research using measure you are interested in –Unpublished measures often described in methods –Authors may provide measures

47 Process of Selecting Good Measures for Your Studies Define concept (variable) Specify study context Review measures’ properties --conceptual and psychometric adequacy Pretest best 1-2 measures Select final measure Identify potential measures

48 Review Potential Measures for: u Conceptual adequacy for your study u Psychometric adequacy in target group(s) u Practicality, acceptability in your study u Translations available if needed

49 Conceptual Adequacy for Your Study u Concept being measured “matches” the concept you defined –Sometimes can only be determined by reviewing items u If not a perfect match –How close is it to your concept? –Can it be modified to get at missing components?

50 Conceptual Adequacy u You are interested in reports of perceived discrimination in health care setting u Measures of discrimination pertain to: –Discrimination over the lifecourse –Discrimination in various settings (work, school) u Not adequate for your purpose

51 Conceptual Adequacy: interpersonal processes of care u You are interested in variety of processes relevant to vulnerable patients u Measures available –Designed for mainstream populations –Focus on communication »Missing decision making and interpersonal dimensions

52 Psychometric Adequacy for Your Study u In samples similar to your target group: –good variability –low percent of missing data –good reliability –good validity u As an outcome for planned intervention –responsive, sensitive to change in similar population –able to detect expected magnitude of change

53 Reliability u Extent to which an observed score is free of random error u Population-specific; reliability increases with: –sample size –variability in scores (dispersion) –a person’s level on the scale

54 Internal Consistency Reliability: Cronbach’s Alpha u Requires multiple items measuring same construct u Extent to which items measure same construct (same latent variable) u It is a function of: –Number of items –Average correlation among items –Variability in your sample

55 Minimum Standards for Internal Consistency Reliability u For group comparisons (e.g., regression, correlational analyses) –.70 or above is minimum –.80 is optimal – above.90 is unnecessary u For individual assessment (e.g., treatment decisions) –.90 or above (.95) is preferred JC Nunnally, Psychometric Theory 3 rd ed, McGraw-Hill, 1994

56 Validity u Does a measure (or instrument) measure what it is supposed to measure? u And… Does a measure NOT measure what it is NOT supposed to measure?

57 Validation of Measures is an Iterative, Lengthy Process u Validity is not a property of the measure –Validity is a property of a measure for particular purpose and sample –Validation studies for one purpose and sample may not serve another purpose or sample u Accumulation of evidence: –Different samples –Longitudinal designs

58 Construct Validity Basics A process of answering the following questions: u What is the hypothesis? u What are the results? u Do the results support (confirm) the hypothesis?

59 Practical Considerations: Match to Your Context Study context u Need permission to use? Any cost of using? u Scoring rules available? u Method of administration appropriate? u Short forms if needed? Appropriate for your sample u Reading level u Translations u Acceptability, respondent burden

60 Practical - Obtaining Permission u Need permission to use or to adapt? u Public domain –If items are published or in the public domain, usually don’t need permission u Private or proprietary –Need to write to author or distributor –Allow 4-6 weeks to obtain measure and/or permission

61 Practical - Cost to Use or to Score Measures u Cost of administering and scoring –Fee for each “instrument” purchased –Cost of any needed scoring software? –Cost to have it scored by source? »Cost per “instrument”?

62 Practical - Scoring u Are scoring instructions clearly documented? –Do you have a scoring codebook? u Are computer scoring programs available? u (Cost of scoring)

63 Practical – Short Forms? Are there reliable and valid short-forms available if you need it? Many measures have short forms, but they typically have not been tested as thoroughly Shorter forms can have low variability, reliability, validity, and sensitivity to change

64 Reading Level u Is reading level appropriate for your target population? –Special concern in lower SES, limited English proficiency groups u If reading level not known –Make your own judgment –Pretest with target population

65 Availability of Translations if Needed u If you need measure in another language, are there translations available? –Official (published and tested) –Unofficial (by some other researcher)

66 Translation Availability and Quality Is the measure available in the language of your target populations? Yes No Know method of translation Assess adequacy or quality of translation Perform translation using state-of-the-art methods A resource issue

67 Acceptability u Ease with which measure can be used in your setting and population u Acceptability to target population –respondent burden –culturally sensitive u Acceptability to interviewers –amount of training needed

68 Respondent Burden u Real burden –Length, convenience, time needed to complete u Perceived burden –a function of item difficulty, distress due to content, perceived value of survey, expectations of length u Some population subgroups may have more difficulty, take longer to complete

69 Process of Selecting Good Measures for Your Studies Define concept (variable) Specify study context Review measures’ properties --conceptual and psychometric adequacy Pretest best 1-2 measures Select final measure Identify potential measures

70 Pretest Potential Measures in Your Target Population u Select best measures for all concepts in your conceptual framework –existing instrument in its entirety –subscales of relevant domains (e.g., only those that meet your needs)

71 Pretest u Pretesting essential for priority measures (e.g., outcomes) u Pretest is to identify: –problems with method of administration –unacceptable respondent burden –problems with questions or response choices –words and phrases that do not mean what you intended to target population

72 Pretest Methods u April 21 class by Anna Nápoles u Using qualitative methods in developing and testing concepts, measures, and interventions

73 Summary u Methods for selecting/reviewing measures described here are “ideal” u Apply these methods to your most important measures –e.g., outcomes, key independent variables

74 Homework u See handout for class 2 homework u Complete rows 1-13 in matrix –Use form posted on the website –Remaining rows are for the next measurement lecture u responses to –By Monday April 13