Brief Historical Overview of the Budapest Initiative and Testing Activities January 2010 Palais des Nations, United Nations Geneva, Switzerland

Measuring health status for international comparisons… What constitutes core health? What measures should be used? What standards for producing the data? Are the data internationally comparable? What methods should be used for comparisons?

Historical Overview: Establishment of the TF and First Meeting May 2004: UNECE sponsored a Joint Meeting on the Measurement of Health Status with WHO and Eurostat Broad consensus on a) the need for multiple indicators to provide a full statistical picture of population health, both for individual country use and for international comparisons; and b) the immediate need for an indicator of ‘health state’ Recommendation to set up a Task Force on the measurement of heath state under the aegis of the UNECE The first meeting was held in Budapest in November 2005, and following the City Group convention, now known as the Budapest Initiative (BI)

Historical Overview: Purpose and Objectives Purpose: To develop a new common instrument to measure health state suitable for inclusion in national interview surveys which will provide basic necessary information on population health. Objectives: To develop a question set to assess overall health state through a number of domains of functioning; To describe trends in health over time within a country, across subgroups of a population, and across countries; and To do so in the framework of official national statistical systems.

Historical Overview: Framework, Definitions, Domains Development of a conceptual framework defining the concepts of health status and health state Paper: Health as a Multi-Dimensional Construct and Cross-Population Comparability (2005) Adopted a set of criteria for selecting functional domains Paper: Criteria For and Selection of Domains for the Measurement of Health Status (2005) Selection of domains Developed a question set to measure selected domains Paper: Conceptual and Logistic Issues in Item Construction and Proposal Questions for Domains (2005)

What is Health State? Health State measures functional ability as opposed to other aspects of health: Determinants and risk factors Disease states Use of health care Environment barriers and facilitators Functioning is measured in terms of ‘capacity’ not ‘performance’ ‘Within the skin’ Without the use of aids in a reasonable environment

Health Status and Health States Health states (capacities) Determinants of health Genetic contribution Lifestyle/behaviour Death Physical environment Physiological risk factors and risk markers Diseases, symptoms and injuries Domain 1 Domain2 N2.... HEALTH STATUS DomainN1 Economic environment Social environment Physiological risk factors and risk markers Quality of life Wellbeing Health State: an individual’s levels of functioning within a set of health domains.

BI Focus and Guiding Principles for the Question Set Measure the health states of individuals; Operationalize health state as functional ability; Focus on capacity rather than performance; Measure across a parsimonious number of health domains, capturing the most variation in health; Meet high standards of validity and international comparability; and Simple and clear, and easily translated into many languages.

Criteria for the Selection of Domains Relevance: face validity, breadth of domains, importance for population health monitoring, draw on selected key ideas of the ICF; Feasibility: reasonable for inclusion in health interview surveys, consistency in meaning across multiple social contexts, heterogeneity, parsimony of domains Measurement: statistical & structural independence, clear series of levels within domains, within/near the skin, suitability for preference measurement. In order to identify the domains of health to be included, a set of criteria were established:

Final 7 Domains for BI Questions Vision Hearing Walking / Mobility Cognition Affect Pain Fatigue* The original set of 10 domains included dexterity and social functioning/ relationships, both of which have been dropped. Anxiety was combined with depression in the current domain of affect.

Criteria and Considerations for Item Construction Number of questions per domain Ensuring uni-dimensionality Duration of the recall period Dealing with assistive devices and medications Item wording and response categories Positive vs. negative wording 4- vs. 5-category response sets Summary/preference health measures

Testing the Question Set

Testing the BI Questions Round 1: November – December 2006 BI-only test (4 countries, 3 languages): Australia, Canada, Italy, U.S. Cognitive testing of all 7 domains Results reviewed at January 2007 TF meeting Separate studies and reports, inconsistent analysis methods, some general findings Outcome: BI-M1 question set, submitted to Eurostat Three rounds of testing have been conducted as part of the development of a BI question set.

Testing the BI Questions Round 2: November 2007 – January 2008 BI-ESS collaborative test (7 countries, 6 languages): Bulgaria, Germany, Portugal, Spain, Switzerland, U.K, U.S. Cognitive testing of 6 domains (absent Vision) Improvements: Evidence-based methodology & systematic comparative analysis of patterns Analysis meeting in U.S. February 2008, all participants Results documented in report Outcome: Further changes for cognitive testing recommended

WG-BI Similarities Substantive overlap/Differing perspectives: Question domains include the same basic and complex activities Disability includes interaction with environment and civil rights perspective Requirements for question sets: Minimize burden on national data collection Parsimony in the number of indicators and measures; domains are succinct, clearly defined Reasonable expectation of high quality responses from samples of the general public, demonstrated validity of measures International comparability, relevant at national and international level Focus on aspects of health that are more likely to produce comparable data Need for cross national cognitive and field testing Questions must be simple, clear, easily translated into many languages Amenable to multi-modes of collection

Testing the BI Questions Round 3 (Cognitive Test): January 2009 – March 2009 UNESCAP-WG-BI collaborative test (10 countries): Cambodia, Canada, Fiji, Kazakhstan, Maldives, Mongolia, South Africa, Sri Lanka, Philippines, U.S. Cognitive testing of 11 domains (additions: Upper Body, Communication, Learning, and Life Activities) Improvements: On-line data management and analysis tool Analysis meeting in U.S. May 2009 Outcome: Further changes for field test recommended

Testing the BI Questions Round 3 (Field Test): June 2009 – August 2009 WG-BI-UNESCAP collaborative test (5 countries): Cambodia, Kazakhstan, Maldives, Sri Lanka, Philippines Cognitive testing of 11 domains (additions: Upper Body, Communication, Learning, and Life Activities) Improvements: Approx 1,000 field interviews at each site

Testing Issues and Methodological Developments

General Issues for Testing Cross-cultural comparability: Do the survey questions work consistently across all countries and subgroups? Translation comparability: Do terms (both in the question and in the response set) have the same meaning across countries? Validity: Do respondents interpret questions consistently regardless of country, language, or demographic? Do respondents use the same thought processes to answer questions? If not, then, why are there differences? What about the countries, languages or demographic subgroups generate different response processes? How can we “fix” or manage these differences through question design?

Objectives for the Test Protocol Purpose: To develop systematic comparable method with joint analysis. Evidence based Joint and coordinated interviewing Similar protocol Similar sample Understanding of differences (at a minimum) Joint and coordinated analysis With interview data Evidence based (as opposed to opinion)

Lessons Learned Semi-structured cognitive interviews offers critical and unique insight into cross-national question performance Transparency is critical Of data from interviews Of the process for drawing conclusions Data collection oversight Better data management

Because of a physical, mental or emotional problem, do you have difficulty concentrating, remembering or making decisions? IDImmediate words after question Response Category Description of any response Difficulty Activity, experience or Situation Discussed took meds ? If so, was this considered? Way in which answer was formed; pattern of Calculation S1"None"No difficultyNoneForgetting an umbrella when the forecast is for rain. noShe was thinking about her ability to remember & concentrate in general. Even though she occasionally forgets things it's not a problem. clause ignored S2"Sometimes I have trouble remembering things" a little difficulty Easy to answer, but this difficulty she sees as normal stuff & isn't concerned about it; i.e., this isn't a disability walking in to the kitchen and forgetting why she's there noa little difficulty means that sometimes she forgets things - it's frequency of forgetting, not significance of things forgotten; 1st clause ignored Example BI chart

Q-Notes On-line data entry and analysis tool Allows for continuous oversight Facilitates quick but thorough analysis Designed around analysis principles

Next Steps Toward BI-M2 Activities in 2010

Next Steps for the BI Questions The BI-M1 did not meet all of the objectives for the question set (some domains absent, some room for improvement). Testing to date has been focused on developing BI-M2. BI-M2 will be based on lessons learned from: 2009 UNESCAP cognitive and field tests (reviewed here) 2010 European cognitive tests (Round 4) 2010 experience with U.S. National Health Interview Survey

2010 Testing - Round 4 Comparative Cognitive Interviewing Project Testing will return to the European region, just as in Round 2. Spain (Spanish) Portugal (Portuguese) Italy (Italian) France (French) Germany (German) Switzerland (French, German, Italian) United States (English, Spanish)

Project Group Coalition Testing will be conducted by: Members of the 1 st comparative group (from Round 2) Methodologists from QEM workshop at NCHS US and Canada

Project Goals Continue methodological work begun in Round 2 testing. Narrative interviewing vs. structured probes Interpretations of vague response categories across cultures and languages Use of software for data collection. Examine issues raised at QEM. Use of mixed-method (PROMIS-WG; cognitive and field tests) Continue examination of WG/BI extended set, specifically what questions capture. Patterns of interpretation Patterns of calculation Both in-scope and out-of-scope interpretations

Domains for Inclusion in the Test Cognition Communication Affect Learning Pain Fatigue PROMIS subset