Presentation is loading. Please wait.

Presentation is loading. Please wait.

1 Class 7 Pretesting measures and considerations in modifying or adapting measures November 10, 2005 Anita L. Stewart Institute for Health & Aging University.

Similar presentations


Presentation on theme: "1 Class 7 Pretesting measures and considerations in modifying or adapting measures November 10, 2005 Anita L. Stewart Institute for Health & Aging University."— Presentation transcript:

1 1 Class 7 Pretesting measures and considerations in modifying or adapting measures November 10, 2005 Anita L. Stewart Institute for Health & Aging University of California, San Francisco

2 2 Steps 5-7: Pretesting and Adapting 1. Specify context 2. Define concept for your study 3. Identify and review potential measures for a) conceptual and psychometric adequacy b) practicality and acceptability c) translation available if needed 4. Select best candidates 5. Pretest selected measures in your groups 6. Choose best ones based on pretest results, OR 7. Adapt if necessary to address problems this week

3 3 Purposes of Pretest by Context u In your search for measures, can look for evidence of pretesting –Evaluate whether measure was pretested and adequacy of methods u When creating a battery of measures –You can conduct pretest to help you decide between alternative measures u When using specific measures –You can conduct pretest to identify problems

4 4 Example of Measure Undergoing Cognitive Testing u Consumer Assessment of Health Plans Study (CAHPS TM ) –Harris-Kojetin et al., 1999 (in readings) u Describes extensive pretesting in developing and refining the CAHPS TM u Provides example of good pretesting

5 5 Step 5: Pretest in Target Population u Pretesting essential for measures being applied to any new population group –Especially priority measures (e.g., outcomes) u Pretest is to identify: –problems with method of administration, respondent burden, procedures –problems with questions or response choices

6 6 Types of Pretests u General debriefing pretest (N=10) u In-depth cognitive interviewing (N=5-10 each group) u Large pretest (N=100) –test measurement properties prior to major study

7 7 Types of Pretests u General debriefing pretest (N=10) u In-depth cognitive interviewing (N=5-10 each group) u Large pretest (N=100) –test measurement properties prior to major study

8 8 General Debriefing Pretest u Goal –Identify problems with the procedures –Estimate time needed to complete instrument –Identify serious problems with items u Procedures –Subjects answer entire questionnaire –At end, debrief –Close to true task

9 9 Debriefing Questions After Administration of Survey.. Ask respondents: u Were any questions confusing? u Which words were hard to understand? u Which questions were difficult to answer? caused distress? u Was questionnaire too long? u Confusing instructions?

10 10 Patient Acceptance Scale u 6-item scale about burden of a survey u Negative affect burden –Feeling embarrassed, upset, annoyed, uncomfortable u Questionnaire length u Ease of answering questionnaire Zimmerman M et al., Med Care, 1994;32:603-608

11 11 Example of General Pretest to Select Best Measure of Patient Satisfaction u Compared 4 questionnaires on –Response rates, missing data, completion time, patient ratings of the questionnaire u 10 evaluation items at end of questionnaire –Clarity of questions, ease of finding an answer, questions about unimportant issues, ease of completion, too long, layout confusing, lacked important questions Perneger TV et al., A randomized trial of four patient satisfaction questionnaires Med Care, 2003;41(12):1343-1352

12 12 Example of General Pretest to Select Best Measure of Patient Satisfaction u Compared 4 questionnaires on –Response rates, missing data, completion time, patient ratings of the questionnaire u 10 evaluation items at end of questionnaire –Clarity of questions, ease of finding an answer, questions about unimportant issues, ease of completion, too long, layout confusing, lacked important questions Perneger TV et al., A randomized trial of four patient satisfaction questionnaires Med Care, 2003;41(12):1343-1352

13 13 Problems with General Pretests Respondents… u often don’t understand the task. u don’t want to appear as if they didn’t understand u have a hard time telling you anything was wrong u easier to say everything was fine

14 14 In-Depth Cognitive Interviews u Derived from social and cognitive psychology to explore processes respondents use to answer survey questions

15 15 Cognitive Interviews Examine 4 Steps in Answering Questions u Comprehend the question (as intended) u Retrieve the information –various strategies used to access memory u Judgment formation - formulate an answer –calculate or judge the correct information u Edit response - decide what to report –is answer embarrassing, socially undesirable? Sudman S et al., Thinking About Answers, Jossey-Bass, San Francisco, 1996

16 16 Purpose of Cognitive Interviews To learn.. u.. if respondents understand words and phrases as intended (meaning) u.. about the process of answering the questions u.. whether items are unacceptable u.. about the usefulness of response choices –whether response choices are adequate –how they use the response choices

17 17 Two Types of Cognitive Interviews u Think aloud interviews –Respondent asked to think aloud as they answer question u Probe interviews –Interviewer asks specific questions to elicit how respondent answered question –Scripted and spontaneous probing

18 18 Think Aloud Interviews u Thorough examination of the entire thought process of creating answers u Require that respondents verbalize their thought processes as they are answering items –Hard task for most people u Mostly done at “cognitive interviewing” labs, e.g., at National Institutes of Health

19 19 Steps in Cognitive Interview Pretesting u Identify most problematic items and develop scripted “probes” u Decide on order of administration –Probes after each question or at the end u Recruit sample for cognitive interviews u Administer entire questionnaire –Administer probe questions u Summarize results

20 20 Probe Questions to Explore Meaning u What does the word _______ mean to you? u What does the phrase ________ mean to you?

21 21 Example: Probing the Meaning of a Phrase I asked you about how the office staff treated you personally …. What does the phrase “office staff” mean to you?

22 22 Example: Probing the Meaning of a Phrase What does the phrase “office staff” mean to you? “the receptionist and the nurses” “nurses and appointment people” “the person who takes your blood pressure and the clerk in the front office”

23 23 Example: Probing the Meaning of a Phrase What does the phrase “office staff” mean to you? “the receptionist and the nurses” “nurses and appointment people” “the person who takes your blood pressure and the clerk in the front office” u We changed the question to receptionist and appointment staff

24 24 Types of Probes (cont.) u Retrieval –How did you remember that? u Judgment –Why did you pick that number for your answer? u Response –Do you think that most people answer this question honestly? Collins D. Quality of Life Research 2003. 12:229-38.

25 25 Types of Probes (cont.) u Redundancy –How is the phrase “give you advice about your diet and exercise” different from the phrase “talk to you about your diet and exercise”?

26 26 Types of Cognitive Probes (cont.) u Cultural appropriateness and meaning: –I asked you how often doctors asked you about your health beliefs? What does the term ‘health beliefs’ mean to you? “.. I don’t want medicine” “.. How I feel, if I was exercising…” “.. Like religion? --not believing in going to doctors?”

27 27 Types of Cognitive Probes (cont.) u Cultural appropriateness and meaning: –I asked you how often doctors asked you about your health beliefs? What does the term ‘health beliefs’ mean to you? “.. I don’t want medicine” “.. How I feel, if I was exercising…” “.. Like religion? --not believing in going to doctors?” u We changed the question to “personal beliefs about your health”

28 28 Overcoming “Social Desirability” in Cognitive Interviews u Ask respondents whether they think others… –would have difficulty answering a question –would answer the question honestly

29 29 Example of Probe on Difficulty: CES- D Item “During the past week, how often have you felt that you could not shake off the blues, even with help from family and friends” u Probe: Do you feel this is a question that people would or would not have difficulty understanding? –Latinos more likely than other groups to report people would have difficulty TP Johnson, Health Survey Research Methods, 1996

30 30 Example: Probing the Process of Answering u I asked you _____ and you answered____. –Why did you pick this answer? –What were you thinking of when you picked this answer? –Can you tell me what you were thinking when you answered this way? »Can you give me some examples? u What came to mind when I asked you _____?

31 31 Example: Use of Response Scale u Do diverse groups use the response scale in similar ways? u Re questions about cultural competence of providers –Interviewers reported that Asian respondents who were completely satisfied did not like to use the highest score on the rating scale CPEHN Report, 2001

32 32 Sample Result: Use of Response Scale u In an exercise class of Samoans, instructor asked them to rate the difficulty of the exercise he just did on a 1-10 scale u They did not understand what he meant by a 1-10 scale –“Western” metric?

33 33 Cognitive Interviewing: Example of Probing Questions Acceptability of Questions u I asked you _____. –Did you find this question offensive? –Was it distressing to answer this question?

34 34 Example of Probe for Acceptability u When I asked you how often you felt discriminated against by doctors because of your race or ethnicity, you answered (read answer given). –Were you offended by this question?

35 35 Interviewer’s Role: Spontaneous Probing u Probe on items that appear to be problematic u If a long pause in answering.. –“I noticed you pausing - what came to mind when you I asked you that question?”

36 36 Other Cues to Problems in Face-to- Face Pretests When administering the survey… u Be aware of behavioral cues related to specific items or to questions in general –Discomfort –Yawning –Looking at their watch

37 37 Steps in Cognitive Interview Pretesting u Identify most problematic items and develop scripted “probes” u Decide on order of administration –Probes after each question or at the end u Recruit sample for cognitive interviews u Administer entire questionnaire –Administer probe questions u Summarize results

38 38 Two Approaches to Probing u Concurrent probing –Ask probes immediately after respondent has given answer to survey item –Advantage-information is fresh in respondent’s mind u Retrospective probing –Ask probes after entire interview –Advantage-able to assess standard administration of items

39 39 Steps in Cognitive Interview Pretesting u Identify most problematic items and develop scripted “probes” u Decide on order of administration –Probes after each question or at the end u Recruit sample for cognitive interviews u Administer entire questionnaire –Administer probe questions u Summarize results

40 40 Special Script for Recruiting People for Cognitive Interviewing u Respondents often do not understand their role u Explain to them how their help fits into the larger study, goal of main study, process of creating questions u Explain their role clearly: –“help us learn how to ask better questions” –“help us make questions clearer for others”

41 41 Recruiting Subjects for Cognitive Interviewing u Explain who you are, purpose of pretest, their role, expected duration (1-1.5 hours) –Subjects told it is a pretest and that their task is to help by identifying problems u Pay subjects since it is a special demand on them (if you can) u If survey is long, may have to pretest different sections on different subjects  Requires more total subjects

42 42 Steps in Cognitive Interview Pretesting u Identify most problematic items and develop scripted “probes” u Decide on order of administration –Probes after each question or at the end u Recruit sample for cognitive interviews u Administer entire questionnaire –Administer probe questions u Summarize results

43 43 Administer Questionnaire and Probes u Administer entire questionnaire –Or break into sections u Administer probes u Audiotape entire interview including probes –Transcribe audiotapes RESULT: information on the entire process of administering the questionnaire

44 44 Steps in Cognitive Interview Pretesting u Identify most problematic items and develop scripted “probes” u Decide on order of administration –Probes after each question or at the end u Recruit sample for cognitive interviews u Administer entire questionnaire –Administer probe questions u Summarize results

45 45 Summarize Data on Pretest Interviews u Summarize problems and nature of problems for each item u Determine how important problems are u Data become basis for possible revisions/adaptations

46 46 Behavioral Coding u Review transcripts to identify problems u Two types of problems: –Problems with standard administration of items »respondent and interviewer “problem” behaviors –Problems identified with specific probe questions

47 47 Examples of Behavioral Codes for Standard Administration Interviewer behavior u Hard to read - interviewer experiences difficulty reading question Respondent behavior u Repeat question - respondent asks to have the question repeated u Asks for clarification

48 48 Summarize Behavioral Coding For Each Item u Proportion of interviews with each problematic behavior u For standard administration –# of occurrences of each problem divided by N »e.g., 7/48 respondents requested clarification u For probes –# of occurrences of each problem divided by N for probe »e.g., 2/15 respondents did not understand meaning based on response to probe

49 49 Behavioral Coding Summary Sheet: Standard Administration (N=20) Item # Interviewer: difficulty reading Subject: asks to repeat Q Subject: asks for clarification 12/2001/20 2000 3 3/202/20 401/200

50 50 Examples of Behavioral Codes Based on Probes u Probe on meaning –Open-ended responses indicate lack of understanding u Probe on use of response options –Open-ended responses indicate response options are problematic

51 51 Example: Probing the Meaning of a Phrase I asked you how often doctors asked you about your health beliefs. What does the term “health beliefs” mean to you? “.. I don’t want medicine” “.. How I feel, if I was exercising…” “.. Like religion? --not believing in going to doctors?” Behavioral coding: Count number of times a subject did NOT understand meaning as intended

52 52 Behavioral Coding Summary Sheet: Standard Administration (N=20) + Probes Item #Probes Meaning unclear Interviewer -difficulty reading Subject: asks to repeat Q Subject: asks for clarification 1102/102/2001/20 200000 3154/151/203/202/20 415001/200

53 53 Summarize Behavioral Coding u Determine if problem is common –Items with only a few problems may be fine –Items are questionable when »several types of problems were found »several subjects experienced the same problem

54 54 Behavioral Coding Summary Sheet: Standard Administration (N=20) + Probes Item #Probes Meaning unclear Interviewer -difficulty reading Subject: asks to repeat Q Subject: asks for clarification 1102/102/2001/20 200000 3154/151/203/202/20 415001/200

55 55 Summarize Behavioral Coding (cont.) u Determine if common problems with an item are serious –Gross misunderstanding of the question –Yields completely erroneous answer –Couldn’t answer the question at all u Some less serious “common” problems can be addressed by improved instructions or a slight modification

56 56 Now What! u Issues in adapting measures based on pretest results

57 57 Options for Adapting or Revising u Depend on whether you are using a standard measure or a new measure

58 58 Writing New Items u One approach to finding problems with a standard measure in diverse groups –Write some new items that you think will be better for this group –Write them in the same format as the standard measure u Can test the adequacy of the original measure and a “modified” measure that includes your new items –A way to begin to develop new items

59 59 Adapting Measures u If a measure needs modification in your opinion –Can contact author(s) to see if they would like to work with you –Be sure your opinion is based on extensive pretests in several groups by your group and/or other researchers –Work with a measurement specialist to assure that your proposed modifications are likely to solve the problem

60 60 Internal versus External Validity u Internal validity - your study is able to answer your research questions with a measure u External validity - your results on a measure can be compared to other studies using the same measure

61 61 Tradeoffs of Using Adapted Measures u Modifying measures –Can improve internal validity - able to answer your question with this measure –You lose external validity - can’t compare your scores to other studies u Can test measure in original and adapted form –If “adapted” measure works better, use it u If “adapted” measure does not work better –Can still use original measure (it may not work either)

62 62 Handouts: Examples of Probe Questions u Sample Cognitive Interview Protocol –Ron Hays and Leo Morales u Selected probes from Interpersonal Processes of Care study u Summary of NCHS working paper series on cognitive methods (website)

63 63 Homework for Next Class (Dec 2) u Identify 4-5 questions in measure you have selected that might be a problem for your target population –Phrases, length, response choices u Write one “probe” question for each problem u Conduct 2 pretests on persons similar to those you are interested in u Summarize briefly your results


Download ppt "1 Class 7 Pretesting measures and considerations in modifying or adapting measures November 10, 2005 Anita L. Stewart Institute for Health & Aging University."

Similar presentations


Ads by Google