Presentation is loading. Please wait.

Presentation is loading. Please wait.

Issues in Selecting Measurement Tools in Assessing Eligibility, Process and Outcomes in Social and Behavioral Intervention Research Patrick R. Clifford,

Similar presentations


Presentation on theme: "Issues in Selecting Measurement Tools in Assessing Eligibility, Process and Outcomes in Social and Behavioral Intervention Research Patrick R. Clifford,"— Presentation transcript:

1 Issues in Selecting Measurement Tools in Assessing Eligibility, Process and Outcomes in Social and Behavioral Intervention Research Patrick R. Clifford, Ph.D. Professor and Associate Dean for Research University of Medicine and Dentistry of New Jersey School of Public Health NIH Summer Institute on Social and Behavioral Intervention Research July 9 – 13, 2012

2 A Prefatory Note Given the allocated timeframe combined with the breadth of material specific to measurement issues relevant to social and behavioral intervention research (not to mention the diversity of participant academic training and professional experience), the information presented must be viewed as an overview of issues related to measurement tool selection (i.e., this presentation centers on relatively few issues thought to be of particular importance).

3 Intervention Research Whether conducting program evaluation or intervention outcomes research, it is important to be clear about the purpose(s) for which the data will be used.Whether conducting program evaluation or intervention outcomes research, it is important to be clear about the purpose(s) for which the data will be used. Clarity of purpose will markedly affect what data are to be collected, how they will be collected, and how they will be presented.Clarity of purpose will markedly affect what data are to be collected, how they will be collected, and how they will be presented. Most intervention research/evaluation decisions regarding outcomes have their pros and cons.Most intervention research/evaluation decisions regarding outcomes have their pros and cons.

4 Important Initial Questions What is/are the Research Question(s) of interest?What is/are the Research Question(s) of interest? What are the related Research Hypotheses?What are the related Research Hypotheses? What variables (e.g., independent, dependent, covariate) are needed to investigate the research hypotheses and answer the research question?What variables (e.g., independent, dependent, covariate) are needed to investigate the research hypotheses and answer the research question? How will the data be collected (e.g., RI, IVR, online)?How will the data be collected (e.g., RI, IVR, online)? How do said variables need to be operationalized?How do said variables need to be operationalized? Do relevant psychometrically sound (with respect to the population of interest) measures/procedures exist?Do relevant psychometrically sound (with respect to the population of interest) measures/procedures exist?

5 Screening for Eligibility Factors to Consider: –Participant recruitment (e.g., media, internet, clinical prog.) –Participant characteristics (e.g., homeless, underage) –Study inclusion/exclusion criteria –Invasiveness of data collection tool (e.g., blood, tissue) –Subject burden (e.g., travel, assessment time, scheduling) –Sensitivity of questions (e.g., illegal or sexual behavior) –Research Interviewer training (e.g., client engagement) –Laboratory services (e.g., blood analysis services) –Yield (i.e., total screened/eligible & provide consent) –Costs (e.g., instrument development and administration)

6 Potential Consequences of Screening Burdensome, invasive, and/or sensitive screening activities can:Burdensome, invasive, and/or sensitive screening activities can: diminish recruitment effortsdiminish recruitment efforts result in an underpowered studyresult in an underpowered study contribute to sample biascontribute to sample bias Overly restrictive inclusion/exclusion criteria reduces external validity (i.e., generalizability)Overly restrictive inclusion/exclusion criteria reduces external validity (i.e., generalizability) Electronic (e.g., IVR) and internet methods of data collection can be efficient but raise data integrity and respondent concerns (e.g., erroneous and missing data, respondent identity and state of being).Electronic (e.g., IVR) and internet methods of data collection can be efficient but raise data integrity and respondent concerns (e.g., erroneous and missing data, respondent identity and state of being).

7 Screening Criteria, Tools and Methods To summarize, considerations pertaining to screening criteria, the selection of measurement tools, and the method of screening for participant eligibility are critical, and if not performed adequately can undermine the value of the best designed studies. To summarize, considerations pertaining to screening criteria, the selection of measurement tools, and the method of screening for participant eligibility are critical, and if not performed adequately can undermine the value of the best designed studies.

8 Process Evaluation Process evaluation centers on efforts to assess the extent to which an intervention is implemented as planned and treatments delivered as intended (i.e., the extent to which the study protocol is followed)Process evaluation centers on efforts to assess the extent to which an intervention is implemented as planned and treatments delivered as intended (i.e., the extent to which the study protocol is followed) Process evaluation is important for understanding and interpreting study findings (e.g., it would be a mistake to interpret negative findings as the result of an ineffective treatment if the treatment was not delivered appropriately; sometimes referred to as a Type III error)Process evaluation is important for understanding and interpreting study findings (e.g., it would be a mistake to interpret negative findings as the result of an ineffective treatment if the treatment was not delivered appropriately; sometimes referred to as a Type III error)

9 Assessing Program Implementation Factors to consider: –Resources (e.g., office space, equipment) –Interventionist training (e.g., performance criterion) –Interventionist supervision (e.g., protocol adherence) –Treatment group assignment (e.g., randomization) –Participant participation (e.g., attendance, engagement) –Data collection activities (e.g., at relevant time points) –Research Interviewer training (e.g., performance criteria) –Research Interviewer supervision (e.g., protocol adherence)

10 Inadequate Implementation Process Exposure to ineffective treatments/interventionsExposure to ineffective treatments/interventions Exposure to sub-threshold levels of the interventionExposure to sub-threshold levels of the intervention Reduced design sensitivity and powerReduced design sensitivity and power Increased probability of type II errorIncreased probability of type II error Increased likelihood of confoundingIncreased likelihood of confounding Interpretation of study findings severely hamperedInterpretation of study findings severely hampered Costly (i.e., inefficient use of research resources to include money, effort, and time)Costly (i.e., inefficient use of research resources to include money, effort, and time)

11 Assessing Intervention Outcomes To maximize social and behavioral intervention outcomes research, it will be necessary to assess the participants and their relevant behaviors of concern (e.g., smoking, exercise, alcohol use) prior to the intervention (i.e., baseline), during the intervention (e.g., mediator variables), end-of- intervention, and post-intervention completion, which may have unexpected results (e.g., enhanced intervention effects and reduced design sensitivity).To maximize social and behavioral intervention outcomes research, it will be necessary to assess the participants and their relevant behaviors of concern (e.g., smoking, exercise, alcohol use) prior to the intervention (i.e., baseline), during the intervention (e.g., mediator variables), end-of- intervention, and post-intervention completion, which may have unexpected results (e.g., enhanced intervention effects and reduced design sensitivity).

12 Measurement Tool Considerations Complexity of the relevant constructs (e.g., dimensionality of measures, abstract vs. tangible)Complexity of the relevant constructs (e.g., dimensionality of measures, abstract vs. tangible) Sensitivity of the information to be collected (e.g., illegal behavior, consequences of honest reporting)Sensitivity of the information to be collected (e.g., illegal behavior, consequences of honest reporting) Method of data collection (e.g., in-person interview, IVR, internet) and potential benefits and disadvantagesMethod of data collection (e.g., in-person interview, IVR, internet) and potential benefits and disadvantages Invasiveness of the data collection tool (e.g., blood sample vs. questionnaire) as it can affect attritionInvasiveness of the data collection tool (e.g., blood sample vs. questionnaire) as it can affect attrition Instrument psychometric properties (i.e., reliability, validity)Instrument psychometric properties (i.e., reliability, validity) Data analysis plan (e.g., continuous vs. categorical)Data analysis plan (e.g., continuous vs. categorical) Correlations among variables (e.g., repeated measurements, multicollinearity)Correlations among variables (e.g., repeated measurements, multicollinearity)

13 Instrument Selection Considerations The measure should be sufficiently sensitive to identify meaningful changeThe measure should be sufficiently sensitive to identify meaningful change The measure should be standardized (i.e., possess acceptable psychometric properties and provide a measure of the behavior of interest) and minimal (e.g., SIP vs. DrInC, Procidano Support Measures)The measure should be standardized (i.e., possess acceptable psychometric properties and provide a measure of the behavior of interest) and minimal (e.g., SIP vs. DrInC, Procidano Support Measures) The resources allocated to the measurement of a construct should be proportional to its importanceThe resources allocated to the measurement of a construct should be proportional to its importance Selected measures should be the most cost effective available to measure the constructs of interestSelected measures should be the most cost effective available to measure the constructs of interest

14 Additional Considerations Data collection via the internetData collection via the internet Assessment reactivity effectsAssessment reactivity effects –Frequency of assessment –Comprehensiveness of the assessment battery Participant tracking systems to minimize attrition Participant tracking systems to minimize attrition

15 Electronic Data Collection AdvantagesAdvantages –Does not require available research staff to collect data –Participant convenience regarding assessment completion –Eliminates research staff data entry errors and costs –Access to broad populations of interest DisadvantagesDisadvantages –Data integrity concerns (i.e., missing and erroneous data) –Respondent identification –Respondent state of being

16 Electronic Data Collection Bots Originally, referred to computer programs that generate automated responses to Web survey formsOriginally, referred to computer programs that generate automated responses to Web survey forms Human Spam Bots refers to people who repeatedly complete and submit surveys on behalf of someone elseHuman Spam Bots refers to people who repeatedly complete and submit surveys on behalf of someone else

17 Procedures to Enhance the Integrity of Electronic Data Record time spent in activityRecord time spent in activity Check for items with suspect responsesCheck for items with suspect responses Include impossible item response choices (e.g., ask where person heard about the study and include responses that are incorrect)Include impossible item response choices (e.g., ask where person heard about the study and include responses that are incorrect) Check response patterns (e.g., same response choice across close-ended questions or minimal responses to open-ended questionsCheck response patterns (e.g., same response choice across close-ended questions or minimal responses to open-ended questions Check for suspicious email addresses (e.g., abcd@gmail.comCheck for suspicious email addresses (e.g., abcd@gmail.com

18 Research Assessment Effects Both the frequency and comprehensiveness of the assessment battery contribute to improved clinical outcomesBoth the frequency and comprehensiveness of the assessment battery contribute to improved clinical outcomes Reduced research design sensitivityReduced research design sensitivity Increased type II errorIncreased type II error Methodological confoundMethodological confound Transfer of less effective, or possibly even ineffective, treatments to community settingsTransfer of less effective, or possibly even ineffective, treatments to community settings

19 The More Salient Aspects of the Research Protocol Interacting with a Research InterviewerInteracting with a Research Interviewer Frequent assessment (e.g., quarterly)Frequent assessment (e.g., quarterly) Comprehensive assessmentComprehensive assessment Extended post-treatment follow-up periodExtended post-treatment follow-up period Location: (i.e., treatment and research typically occur in the same settingLocation: (i.e., treatment and research typically occur in the same setting Inclusion of secondary data sources (e.g., collateral involvement, laboratory tests)Inclusion of secondary data sources (e.g., collateral involvement, laboratory tests)

20 Minimize and/or Control for Potential Assessment Reactivity Effects Reduce assessment frequency (e.g., semi-annual assessments as opposed to quarterly)Reduce assessment frequency (e.g., semi-annual assessments as opposed to quarterly) Reduce the comprehensiveness of the assessment battery (e.g., limit to domains necessary for answering research hypotheses)Reduce the comprehensiveness of the assessment battery (e.g., limit to domains necessary for answering research hypotheses) Use separate settings for treatment and research related activitiesUse separate settings for treatment and research related activities Include a minimal or no assessment control groupInclude a minimal or no assessment control group Consider using missing data study designs/analyses (i.e., often referred to as efficiency designs)Consider using missing data study designs/analyses (i.e., often referred to as efficiency designs)

21 Data Collection Considerations Reduce study participant-research staff interactions (e.g., use of computerized assessments, telephone vs. in-person interviews)Reduce study participant-research staff interactions (e.g., use of computerized assessments, telephone vs. in-person interviews) Limit data collection to essential measuresLimit data collection to essential measures Consider data instruments that contain fewer itemsConsider data instruments that contain fewer items Avoid using highly correlated measures of the same constructAvoid using highly correlated measures of the same construct To the extent possible, use existing information (e.g., medical record data) as opposed to participant self-report dataTo the extent possible, use existing information (e.g., medical record data) as opposed to participant self-report data Exclude non-essential collateral reports and laboratory testsExclude non-essential collateral reports and laboratory tests

22 Participant Tracking Systems Contribute to enhanced participant retention rates and mitigate attritionContribute to enhanced participant retention rates and mitigate attrition Obtain contact information from at least two persons who always know how to get in-touch with the participant (e.g., parents, relatives, friends)Obtain contact information from at least two persons who always know how to get in-touch with the participant (e.g., parents, relatives, friends) Record participants full name, SS #, date and place of birth, drivers license # and state issued, employment contact information, full names and addresses of parents (if living)Record participants full name, SS #, date and place of birth, drivers license # and state issued, employment contact information, full names and addresses of parents (if living) Most importantly, obtain consent to use all available means to locate the participant for follow-upMost importantly, obtain consent to use all available means to locate the participant for follow-up


Download ppt "Issues in Selecting Measurement Tools in Assessing Eligibility, Process and Outcomes in Social and Behavioral Intervention Research Patrick R. Clifford,"

Similar presentations


Ads by Google