1. Thoughts on the Role of Surveys and Qualitative Methods in Evaluating Health IT National Resource Center for HIT 2005 AHRQ Annual Conference for Patient Safety and Health Information Technology

2. Starting with Surveys… Design issues Design issues –comparison groups Instrument development Instrument development Sampling methodology Sampling methodology –probability of selection –size, response rate for significance *and* power Survey administration Survey administration Data analysis Data analysis

3. Instrument Development: Key Considerations Internal validity Internal validity –are you getting what you think you are getting? Question ordering Question ordering –easy to difficult –logic to flow and transitions Keep it short Keep it short Pilot test! Pilot test!

4. Sampling Methodology: Key Considerations Define your population, sampling frame Define your population, sampling frame Random selection of participants Random selection of participants –No ‘cherry picking’ (eliminate bias) –Everyone has a non-zero chance to be selected Stratification and over sampling Stratification and over sampling Target response rate for a statistically meaningful result Target response rate for a statistically meaningful result –Significance and power

5. Survey Administration: Key Considerations Selecting your mode of administration Selecting your mode of administration –Self-administered questionnaire (SAQ) Mail, Web Mail, Web –Telephone interviews –In-person Tips on increasing response rate Tips on increasing response rate –Personalize initial outreach –Use the Dillman approach –Questionnaire format –Optimize follow-up: bang for buck

6. “When you can’t express it in numbers … your information is of an unsatisfactory kind” -- Lord Kelvin (1824 – 1907) So, why use surveys *and* qualitative methods?

7. … Because evaluating Health IT is a SOCIAL science

8. What is qualitative methodology? A non-numerical examination and interpretation of observations for the purpose of discovering underlying meanings and patterns. Qualitative methods provide a powerful tool for understanding the context surrounding quantitative data and for explaining phenomena that cannot be measured statistically due to a lack of validated instruments, sample size, or passage of time.

9. Using qualitative methods... Qualitative methods offer highly valid results through an in-depth understanding of processes, attitudes, and effects. Qualitative methods offer highly valid results through an in-depth understanding of processes, attitudes, and effects. Qualitative methods do not produce statistically significant results and are generally less reliable than quantitative results. Qualitative methods do not produce statistically significant results and are generally less reliable than quantitative results.

10. Qualitative Methods: Observation Direct observation by individual(s) of activities, processes, or behaviors Direct observation by individual(s) of activities, processes, or behaviors –Is implementation going as planned? –Validating self-reported survey results –Requires skilled observers ($) Method of observation/data capture varies Method of observation/data capture varies –E.g., Time and motion studies

11. Qualitative Methods: Semi-structured Interviews Qualitative Interviews Qualitative Interviews –Use discussion guide or list of themes Not a static, closed-ended questionnaire Not a static, closed-ended questionnaire –Key stakeholder perspectives –Detailed information, flexibility Methodology varies Methodology varies –Sampling approach can be scientific –Content analysis on notes or transcripts

12. Qualitative Methods: Focus Groups Moderated discussion with 10-12 respondents Moderated discussion with 10-12 respondents –discussion guides w. pre-selected themes Improved validity Improved validity –getting all “on the same page” Can explain contradictory statistical results Can explain contradictory statistical results

13. Qualitative Methods: Case Studies An in-depth study of a small set of sites. An in-depth study of a small set of sites. –Observation –Interviewing –Review and/or content analysis of background information, documents, communications, etc. –Informal conversations and encounters Provides rich descriptions of activities and the influences of external factors on activities. Provides rich descriptions of activities and the influences of external factors on activities. Highly valid, but not generalizable. Highly valid, but not generalizable. Case studies are NOT equivalent to program site visits. Case studies are NOT equivalent to program site visits.

14. Analyzing Qualitative Data Process is variable Process is variable –but for any study assumptions should be clearly described General approach General approach –Coding raw qualitative information Nudist or ATLAS.ti Nudist or ATLAS.ti –Identify themes and trends –Identify and describe outliers –Draw conclusions carefully and caveat Analyses should occur both within and across individual data collections (e.g. within a single interview and across all interviews). Analyses should occur both within and across individual data collections (e.g. within a single interview and across all interviews).

15. Recommended References Coffey, A., and Atkinson, P. (1996). Making Sense of Qualitative Data: Complementary Research Strategies. Thousand Oaks, CA: Sage. Coffey, A., and Atkinson, P. (1996). Making Sense of Qualitative Data: Complementary Research Strategies. Thousand Oaks, CA: Sage. Edmunds, Holly. The Focus Group Research Handbook. McGraw Hill. 2000. Edmunds, Holly. The Focus Group Research Handbook. McGraw Hill. 2000. Hyman, Herbert H., with William J. Cobb, Jacob J. Feldman, Clyde W. Hart, and Charles Herbert Stember. Interviewing in Social Research. University of Chicago Press, 1954. Hyman, Herbert H., with William J. Cobb, Jacob J. Feldman, Clyde W. Hart, and Charles Herbert Stember. Interviewing in Social Research. University of Chicago Press, 1954. Krueger, A. Richard. Focus Groups: A Practical guide for Applied Research. Sage Publishing. 1994. Krueger, A. Richard. Focus Groups: A Practical guide for Applied Research. Sage Publishing. 1994. Miles, M.B, and Huberman, A.M. (1994). Qualitative Data Analysis, 2nd Ed., p. 10-12. Newbury Park, CA: Sage. Miles, M.B, and Huberman, A.M. (1994). Qualitative Data Analysis, 2nd Ed., p. 10-12. Newbury Park, CA: Sage. NSF “User Friendly Handbook for Mixed Method Evaluations.” 1997. NSF “User Friendly Handbook for Mixed Method Evaluations.” 1997. Payne, Stanley L. The Art of Asking Questions. Princeton University Press, 1951. Payne, Stanley L. The Art of Asking Questions. Princeton University Press, 1951. Turner, Charles F., and Elizabeth M. Martin, eds. Surveying Subjective Phenomena. Russell Sage, 1984. Turner, Charles F., and Elizabeth M. Martin, eds. Surveying Subjective Phenomena. Russell Sage, 1984.