Presentation is loading. Please wait.

Presentation is loading. Please wait.

On the Bias in Estimating Program Effects Using Clinic Based Data Ashu Handa University of North Carolina at Chapel Hill Mari-Carmen Huerta London School.

Similar presentations


Presentation on theme: "On the Bias in Estimating Program Effects Using Clinic Based Data Ashu Handa University of North Carolina at Chapel Hill Mari-Carmen Huerta London School."— Presentation transcript:

1 On the Bias in Estimating Program Effects Using Clinic Based Data Ashu Handa University of North Carolina at Chapel Hill Mari-Carmen Huerta London School of Economics

2 Objective Can clinic based data give good estimates of program impact? –Most large scale nutrition interventions do not have accompanying social experiment Nature of program; costs; political constraints –Clinic based data on nutritional status is available virtually everywhere –Useful to know whether these data can be used for program evaluation

3 Approach Compare non-experimental and experimental estimates of program impact Intervention: Progresa –Experimental estimates of impact on child height already exist –Derive non-experimental estimate using data on beneficiaries from health clinics –Compare the two estimates to assess bias

4 Why Experiments? Randomly selected comparison group allows for –Control of observed characteristics that might affect outcome –Control of unobserved characteristics that might affect outcome (selection bias) Less of a problem in a mandatory program such as Progresa Bias caused by using non-experimental comparison group may be negative or positive

5 Summary of Experimental Results Gertler et.al. and Behrman & Hoddinott (BH) –Use same basic data set; two measurements months apart –Sample is kids age months at baseline –Gertler: Estimates growth in height in cms –BH: Estimate growth in height measured by z-score –Both include child, household and community level control variables –Both report positive and significant estimates in the range of 15% of mean growth (1 cm per year) –Gertler: Only impact is on kids months of age

6 Clinic Based Sample Individual data from all Progresa clinics between end 1997 and end 1999 Different dates of incorporation –Use to identify program impact Select kids with two measures of height taken months apart (median=13 months) –Measure 1 in early 1998; measure 2 in early 1999 Estimate growth in height measured by z-score (same as BH) Kids 0-48 months of age; no control variables

7 Identification Strategy Use childs incorporation date to define length of exposure to program (define 9 groups) Selection problem – control group initially healthier –Growth specification will eliminate some bias Full ExposurePartial ExposureNo Exposure Initial HAZ Growth in HAZ Age in months Exposure in months1670 % of sample751411

8 Estimation

9 OLS Estimates of Exposure Time on Growth Sample:FullFully Exposed ExposureBetaT T excluded Obs R-sq

10 OLS Estimates of Exposure Time on Growth: Interactions with Age Sample:FullFully Exposed BetaT T Log(exposure) Log(exposure) x 0-4 months excluded Obs R-sq

11 Non-experimental estimates relative to benchmark –20% for age group –25% for age group –40% for age group using listed treatment Comparison of point estimates BH Experimental SampleNon-Experimental Age group *0-48**12-48***12-36*** Beta T * Listed instead of actual treatment ** Difference in predicted means using regression equation *** Coefficient of dummy variable indicating whether child ever exposed

12 Discussion Clinic based estimates significantly lower –20% to 40% depending on specification Downward bias due to measurement error –Listed treatment not equal to actual treatment –Listed treatment measures average impact of total program different concept Omitted variable bias reduces estimate –Omitted control variables (not used in clinic-based study) positively related to participation but negatively related to growth –Leads to downward bias in non-experimental impact

13 So how reliable is the non-experimental estimate? The glass is half empty –Estimates are positive, but significantly lower than benchmark –Leads to conclusion that program less effective than it actually is The glass is half full –Gives positive and significant estimates –Cost of measuring impact is virtually zero –Understanding program operation allows assessment of nature of bias How close do we really need to be for policy?


Download ppt "On the Bias in Estimating Program Effects Using Clinic Based Data Ashu Handa University of North Carolina at Chapel Hill Mari-Carmen Huerta London School."

Similar presentations


Ads by Google