1. The Q-Bank Database of Question Evaluation Reports PHIN Vocabulary and Messaging Community of Practice July 21, 2009 Aaron Maitland, NCHS

2. Outline Background on Question Evaluation and Q-Bank Development of Q-Bank Future Goals Demonstration

3. Background Question evaluation has been a vital part of the survey enterprise for a long time. Explosion of question evaluation research due to collaboration between cognitive psychologists and survey methodologists in the 1980s. Widely held belief that these studies have improved individual questionnaires.

4. Why evaluate survey questions? Researchers want to know if their questions are … –valid or provide meaningful information about the intended concept. –reliable or provide consistent information over time and across observations. “Fix” questions with problems.

5. Model of the Survey Response Process Four mental processes that respondents use to answer survey questions. Comprehension Recall Judgment Response selection

6. Question Evaluation Methods Qualitative –Cognitive Interviewing –Focus Groups Quantitative –Behavior coding –Statistical modeling Mixed-method approach

7. Cognitive Interviewing Example Would you say {your} health in general is excellent, very good, good, fair, or poor? –“Many times [respondents] answers seemed to depend on who they were comparing themselves to, which in part seemed to be due to this question’s placement right after the physical limitations section. People sometimes compared themselves to those who could not get around or take care of themselves; others compared themselves to those their own age or those with similar health situations. This also results in incomparable data.”

8. The Problem of Accessibility Historically, cognitive test findings have been relatively inaccessible. Implications –Knowledge is lost –Resources are wasted –Lack of transparency and accountability Need for a medium to make findings available

9. What is Q-Bank? Question Evaluation Reports Individual Questions Test findings Methods Question characteristics Report information

11. Who should use Q-Bank? Question designers Survey data users Survey methodologists

12. Development 2002 NCHS began development 2005 interagency effort –Steering Committee National Center for Health Statistics Census Bureau Bureau of Labor Statistics National Cancer Institute National Science Foundation –Establish common lexicon

13. Establishing a Common Lexicon Question topic –More than 100 question topics Information type –Subjective vs. Objective questions Type of response categories –Numeric vs. textual –Open ended vs. closed ended Response errors –Interviewer difficulty –Problematic terms –Ambiguous concepts –Overly complex –Assumption/double-barrel –Questionnaire effects –Recall/estimation difficulty –Biased/sensitive –Inadequate response options

14. Current Organization Steering Committee Chair: Kristen Miller (NCHS) Other agencies: BLS, Census, NCI, NSF NCHS Q-Bank Administrator Programmer Data Entry Staff

15. Future Goals Include other evaluation methods Increase participation Developing standards for evaluation reports Potential integration with other survey methods tools

16. Q-Bank Website: http://www.cdc.gov/qbank/home.aspx Q-Bank e-mail: qbank@cdc.gov