1. Homework Summary Impressive job and nice opportunity for me to learn about your research Impressive job and nice opportunity for me to learn about your research –Health insurance market reforms –Payment policy to support best practice –Public health particularly around reproductive health, obesity and unhealthy behaviors, workforce training, and resource allocation

2. Public Health Policy Is there a compelling argument to be made that a policy is needed to protect the public’s health Is there a compelling argument to be made that a policy is needed to protect the public’s health Is it clear that the relevant industries won’t attempt to correct the problem Is it clear that the relevant industries won’t attempt to correct the problem Might educating consumers activate market forces to correct a public health problem Might educating consumers activate market forces to correct a public health problem Is it clear that proposed legislation will correct the problem Is it clear that proposed legislation will correct the problem

3. Payment Policy Financial incentives and disincentives to optimize care challenging without evidence base of best practice for specific cases Financial incentives and disincentives to optimize care challenging without evidence base of best practice for specific cases Internal disagreements within profession hard to resolve with legislation Internal disagreements within profession hard to resolve with legislation In general profession doesn’t like to be told what it can and cannot do In general profession doesn’t like to be told what it can and cannot do

4. Dear Dr. Michaels (Asst Secretary of OSHA): On behalf of the Alliance for Academic Internal Medicine (AAIM)—the consortium of five academically focused specialty organizations representing departments of internal medicine at medical schools and teaching hospitals in the United States and Canada—I write to provide the alliance’s perspective regarding the September 2, 2010, letter to your agency, filed by Public Citizen and other organizations, petitioning for federal regulation and enforcement of resident and fellow physician duty hours. AAIM respectfully urges you to deny this petition based on our 4,000 members’ collective personal and professional knowledge of the Accreditation Council for Graduate Medical Education (ACGME) as an effective oversight organization for graduate medical education (GME).

5. Insurance Reforms and Other Imminent Policy Changes Many of you identified specific policies that are or are about to be implemented that are relevant to your research interests Many of you identified specific policies that are or are about to be implemented that are relevant to your research interests Opportunity to study this change as a way to inform the policy process Opportunity to study this change as a way to inform the policy process

6. Evaluating Health Policy through Natural Experiments Andrew B. Bindman, MD Professor Medicine, Health Policy, Epidemiology & Biostatistics UCSF

7. Asking a Policy Research Question Interesting to you Interesting to you Testable hypothesis Testable hypothesis Novelty Novelty Value regardless of direction of result Value regardless of direction of result Ethical Ethical Importance/number impacted Importance/number impacted Timeliness Timeliness

8. Learning About Policies Read the newspaper Read the newspaper –NY Times, Washington Post, Politico Health newswire services Health newswire services –California Healthline (www.chcf.org) (www.chcf.org)www.chcf.org –Kaiser Health News (www.kaiserhealthnews.org) Academic faculty Academic faculty Community-based organizations Community-based organizations Directly from policy decision-makers Directly from policy decision-makers

9. My Research Interest Health consequences of public policies Health consequences of public policies Access to and quality of care for low- income, diverse, and patient populations vulnerable to poor health because of their social circumstances Access to and quality of care for low- income, diverse, and patient populations vulnerable to poor health because of their social circumstances

10. Medi-Cal: California’s Medicaid Program 6.8 million beneficiaries 6.8 million beneficiaries $47 billion last year $47 billion last year 2 nd largest use of general fund (17%) 2 nd largest use of general fund (17%) Pays for 1 in every 2 births in the state Pays for 1 in every 2 births in the state Approximately half of beneficiaries are Latino Approximately half of beneficiaries are Latino Provides 2/3rds of safety net funding Provides 2/3rds of safety net funding

11. Medicaid Population: Clinical Question Many Medicaid beneficiaries have interruptions in coverage (churning) Many Medicaid beneficiaries have interruptions in coverage (churning) Many uninsured gain Medicaid coverage when hospitalized Many uninsured gain Medicaid coverage when hospitalized Does Medicaid coverage provided at the time of a hospitalization adequate or do interruptions in Medicaid enrollment have a negative impact on the health of beneficiaries? Does Medicaid coverage provided at the time of a hospitalization adequate or do interruptions in Medicaid enrollment have a negative impact on the health of beneficiaries?

12. Designing a Research Study Randomized trial Randomized trial –Feasibility?

13. Designing a Research Study Randomized trial Randomized trial –Unethical and impractical Observational study Observational study –Compare the experiences of beneficiaries who have interruptions in coverage with those who have continuous coverage

14. Reverse Causality Interruption in coverage might not predict worse health outcome so much as worse health might predict whether or not have interrupted coverage Interruption in coverage might not predict worse health outcome so much as worse health might predict whether or not have interrupted coverage Bias of higher admissions among those with continuous coverage Bias of higher admissions among those with continuous coverage

15. Designing a Research Study Randomized trial- is it feasible? Randomized trial- is it feasible? Observed variation - is it biased? Observed variation - is it biased? Natural experiment - does a good one exist? Natural experiment - does a good one exist?

16. Natural Experiments A natural or quasi-experiment is a naturally occurring instance of observable phenomena which approximate or duplicate the properties of a controlled experiment. In contrast to laboratory experiments, these events aren't created by scientists, but yield data which nonetheless can be used to make causal inferences. A natural or quasi-experiment is a naturally occurring instance of observable phenomena which approximate or duplicate the properties of a controlled experiment. In contrast to laboratory experiments, these events aren't created by scientists, but yield data which nonetheless can be used to make causal inferences.experimentlaboratoryexperimentsexperimentlaboratoryexperiments

17. What Are the Elements of a Good Natural Experiment in Health Policy Policy implementation not biased by patient characteristics such as health status Policy implementation not biased by patient characteristics such as health status Policy can be effectively tied to a “treatment” exposed group Policy can be effectively tied to a “treatment” exposed group Access to before/after data Access to before/after data

18. Medicaid Population: Policy Question Federal law requires re-determination of eligibility for beneficiaries at a minimum of every 12 months but states have option to do more frequently Federal law requires re-determination of eligibility for beneficiaries at a minimum of every 12 months but states have option to do more frequently Beneficiaries who do not “re-sign up” are dropped from program Beneficiaries who do not “re-sign up” are dropped from program Does frequency of a state’s re-enrollment process increase the number of beneficiaries with interruptions in coverage and if so is this in turn associated with patients’ health? Does frequency of a state’s re-enrollment process increase the number of beneficiaries with interruptions in coverage and if so is this in turn associated with patients’ health?

19. Natural Experiment of Interrupted Medicaid Coverage California extended Medicaid re-enrollment period for all children in California from every 3 to every 12 months on January 1, 2001 California extended Medicaid re-enrollment period for all children in California from every 3 to every 12 months on January 1, 2001 Extension of eligibility re-determination period should be associated with an increase in continuity of Medicaid coverage, but should not except through its influence on continuity of coverage be associated with the health status of children. Extension of eligibility re-determination period should be associated with an increase in continuity of Medicaid coverage, but should not except through its influence on continuity of coverage be associated with the health status of children.

20. Challenging Issues in Studying Natural Experiments Learning about a policy change as it is about to happen or after the fact makes it harder to collect baseline data Learning about a policy change as it is about to happen or after the fact makes it harder to collect baseline data

21. Primary Data Collection Can be challenging to organize in time to assess pre-policy condition Can be challenging to organize in time to assess pre-policy condition Lots of work but lots of control over data collection (eg surveys, physiological measures, etc) Lots of work but lots of control over data collection (eg surveys, physiological measures, etc) –Time consuming –Expensive Difficult to maintain over time Difficult to maintain over time

22. Secondary Databases Pre-existing data that are often collected for an alternative purpose Pre-existing data that are often collected for an alternative purpose Individual level or sometimes aggregate data Individual level or sometimes aggregate data Examples: Examples: –National surveys –Registries –Study cohorts –Administrative data

23. Secondary Data: Advantages/Challenges Efficient - cheap, fast and often very large Efficient - cheap, fast and often very large Little control on what was collected Little control on what was collected Collection is often longitudinal/repeated cross- sectional Collection is often longitudinal/repeated cross- sectional Potential to analyze temporal changes Potential to analyze temporal changes If source is a payer or a provider may have incomplete capture If source is a payer or a provider may have incomplete capture Can be scooped by others with access to same data Can be scooped by others with access to same data

24. Finding Secondary Data www.ctsi.ucsf.edu/research/celdac www.ctsi.ucsf.edu/research/celdac www.ctsi.ucsf.edu/research/celdac www.phpartners.org/health_stats.html www.phpartners.org/health_stats.html www.phpartners.org/health_stats.html

25. Medicaid Data for Studying Interruptions in Coverage Comprehensive and detailed regarding eligibility Comprehensive and detailed regarding eligibility Fee for service claims complete Fee for service claims complete Missing claims information for beneficiaries in managed care Missing claims information for beneficiaries in managed care Won’t reflect experience of beneficiaries when they aren’t covered Won’t reflect experience of beneficiaries when they aren’t covered

26. Statewide Hospital Patient Discharge Abstracts Comprehensive capture of all hospitalizations in state regardless of payer Comprehensive capture of all hospitalizations in state regardless of payer Includes information on hospital admission diagnoses Includes information on hospital admission diagnoses Provides payer source at time of hospitalization Provides payer source at time of hospitalization

27. Ambulatory Care Sensitive Conditions: AHRQ Prevention Quality Indicators 1. Condition with acute course and window for intervention 2. Condition with chronic course amenable to self-management ACS Conditions Acute Conditions: –Dehydration –Ruptured Appendicitis –Cellulitis –Bacterial Pneumonia –Urinary Tract Infection Chronic Conditions: –Asthma –Hypertension –COPD –Diabetes Mellitus –Heart Failure –Angina

28. Statewide Hospital Patient Discharge Abstracts Provides payer source at time of hospitalization but not over time Provides payer source at time of hospitalization but not over time Critical question for hospitalizations for ambulatory care sensitive admissions is what the insurance status was prior to the admission since many uninsured gain coverage in association with the hospitalization Critical question for hospitalizations for ambulatory care sensitive admissions is what the insurance status was prior to the admission since many uninsured gain coverage in association with the hospitalization

29. Linked CA Hospital Discharge and Medicaid Eligibility Files OSHPD: Hospital Discharge Data 19982003 DHS: Medi-Cal Enrollment Database 19982003 Demographics Monthly enrollment history Aid Category (e.g. TANF or SSI) FFS, managed care Other insurance Diagnosis (ICD-9 Code) Month/Year of admission Linkage

30. Pre/Post Study of Re-Enrollment Policy Change for Children Children 1-17 years with a minimum of 1 month of Medicaid coverage in California Children 1-17 years with a minimum of 1 month of Medicaid coverage in California Outcome = time to a hospital admission for an ambulatory care sensitive condition Outcome = time to a hospital admission for an ambulatory care sensitive condition Main predictor = time period Main predictor = time period –Pre policy change = Jan ‘99-December ‘00 –Post policy change = Jan ‘01-December ‘02

31. Children 1-17 Years in California Medicaid Before and After Policy to Change Enrollment 1999-20002001-2002 N 3,288,171 3,288,1713,230,120 Mean Age (yrs) 99 % Female 5051 Ethnicity (%) Hispanic Hispanic5456 Black Black1312 Asian Asian88 Other Other2524 Aid Group (%) TANF TANF4750 SSI SSI33 Other Other5047 Managed Care (%) 4741

32. Children with Continuous Medicaid Enrollment by Time Period 49 62 Years of Enrollment Percentage

33. Probability of a Hospitalization for an ACS Condition Over Time

34. Children: Adjusted Risk of ACS Hospitalization Relative Hazard P-Value P-Value Post policy 0.74 0.74<.0001 Age 0.88 0.88<.0001 Female 0.97 0.970.0175 Ethnicity Hispanic Hispanic 3.26 3.26<.0001 Black Black 4.70 4.70<.0001 Asian Asian 1.10 1.100.0926 Other Other 2.97 2.97<.0001 Aid Group TANF TANF 1.47 1.47<.0001 SSI SSI 24.90 24.90<.0001 Managed Care 0.82 0.82<.0001

35. Hospital Savings Reduction in number of hospital days 15,635 Average daily Medicaid payment for a hospitalization for an ACS condition $1,129 Total hospital savings $17,651,915 Medicaid Coverage Costs Increased number of continuously enrolled months of Medicaid coverage TANF/Other SSI SSI1,395,283 41,524 41,524 Average monthly capitation rate TANF/Other SSI SSI $96 $96$370 Total Medicaid Costs for increased coverage TANF/Other SSI SSI$134,393,685 $15,350,490 $15,350,490 Total Medicaid Coverage Costs $149,744,175 Hospital Savings and Medicaid Coverage Costs Associated with Enrollment Extension in 2001 – 2002

36. Quasi- (natural) Experiments "Estimating the internal validity of a relationship is a deductive process in which the investigator has to systematically think through how each of the internal validity threats may have influenced the data. Then the investigator has to examine the data to test which relevant threats can be ruled out.... When all of the threats can plausibly be eliminated it is possible to make confident conclusions about whether a relationship is probably causal." Cook and Campbell

37. Limitations Could secular changes other than the policy change explain the observed differences? Could secular changes other than the policy change explain the observed differences?

38. Strengthening the Design of a Natural Experiment Pre/post changes ideally with a comparison group not exposed to policy Pre/post changes ideally with a comparison group not exposed to policy “Difference in differences” “Difference in differences” Need to establish conceptual basis for selection of specific comparison group Need to establish conceptual basis for selection of specific comparison group

39. Potential Comparison Group: Adults in Medi-Cal Medicaid eligibility re-determination period did not change during study period for adults Medicaid eligibility re-determination period did not change during study period for adults Therefore, would not expect a decrease over time in hospitalizations for ambulatory care sensitive conditions among adults Therefore, would not expect a decrease over time in hospitalizations for ambulatory care sensitive conditions among adults

40. Comparison Group: Adults in Medicaid Adults with Medicaid coverage Adults with Medicaid coverage –1999-2000 = 62% –2001-2002 = 60% Adjusted relative hazard of a hospitalization for an ACS condition for adults in post vs pre period= 1.11 Adjusted relative hazard of a hospitalization for an ACS condition for adults in post vs pre period= 1.11

41. Second Comparison Group: Children with Continuous Coverage Comparison of children with continuous coverage in each time period revealed no significant difference in hospitalizations for ambulatory care sensitive conditions Comparison of children with continuous coverage in each time period revealed no significant difference in hospitalizations for ambulatory care sensitive conditions Suggests no difference in treatment approach to ambulatory care sensitive conditions over time period of study Suggests no difference in treatment approach to ambulatory care sensitive conditions over time period of study

42. How Do We Know This is About Access to Ambulatory Care? Hospitalization rates among children for non ambulatory care sensitive conditions (appendicitis and gastrointestinal obstruction) did not change over time Hospitalization rates among children for non ambulatory care sensitive conditions (appendicitis and gastrointestinal obstruction) did not change over time

43. Most with Interruption in Medicaid Coverage Do Not Have Alternative for Ambulatory Care At the time of hospitalization At the time of hospitalization –59% regain Medi-Cal with admission –7% remain uninsured –33% had another form of insurance

44. Policy Implications States need to become more aware of the hidden costs in their Medicaid policies States need to become more aware of the hidden costs in their Medicaid policies Continuity of Medicaid coverage can support better health and decrease wasteful spending on hospitalizations that could have been avoided with less costly outpatient care Continuity of Medicaid coverage can support better health and decrease wasteful spending on hospitalizations that could have been avoided with less costly outpatient care

45. Translating Research into Policy Results of study used Results of study used –in testimony to California legislature to prevent more frequent eligibility re- determination as part of budget cut process –in Congress to support Maintenance of Effort requirements as a part of CHIP reauthorization Published in scientific journals for other states to consider in their policy making Published in scientific journals for other states to consider in their policy making

46. Homework Identify a plausible natural experiment to evaluate a policy relevant to your area of research Identify a plausible natural experiment to evaluate a policy relevant to your area of research Describe data you could use to study it and possible comparison group(s) Describe data you could use to study it and possible comparison group(s)

48. Examples of Challenging Natural Experiments Evaluation of voluntary managed care Evaluation of voluntary managed care –voluntary implementation can have health selection bias Evaluation of the benefits of insurance coverage Evaluation of the benefits of insurance coverage –uptake by uninsured and “crowd out” of privately insured can make it hard to isolate who got “treatment” of insurance

49. Are There Opportunities for Randomized Evaluations of Health Policies? Randomized designs are least susceptible to bias Randomized designs are least susceptible to bias Political considerations often make this approach impractical in health policy interventions Political considerations often make this approach impractical in health policy interventions May be opportunities to use a lottery in implementing policies that have more demand than supply (a wait list) May be opportunities to use a lottery in implementing policies that have more demand than supply (a wait list)