1. Family Resources Survey Proposals for the treatment of unlinked FRS data* Valerie Christian & Philip Clarke

2. 2 Background to this project FRS 0809 - first survey year for which we have been able to link* consenting respondents data across the entire year. BUT …. the consent rate achieved in 0809 was just over 60 per cent. ** For the 40 per cent of data not linked, this project aims to assess for any extent of non-consent bias and to adjust non-consenters data accordingly.

3. 3 Where we are now and our eventual aim This work is still in the early stages…..* Eventual aim – combination of all adjustments (i.e. for both consenters and non-consenters) into one improved dataset. **

4. 4 What is the case for adjustments to FRS? CASELOAD capture – all FRS data and administrative data compared (extract from Table M6 for FRS 0809)

5. 5 What is the case for adjustments to FRS? MONETARY capture – based on consenters raw (ungrossed) FRS 0809 data

6. 6 What does all this mean? While the unadjusted FRS continues to be fit for purpose* it does not always appear fully capture state benefits in household income. This shortfall – partly reduced by linking consenters* - justifies the work of this project to adjust non-consenters data (in 0809 almost 40 per cent of all FRS respondents).

7. 7 Considerations in assessing non- consenters data So far, any approach to adjusting non-consenters data* is based on what we know of consenters ** For example, from linking consenters data we know the type of adjustments made to selected DWP benefits

8. 8 What adjustments are made to consenters? Overall net increase to incomes following linking … comprising four distinct patterns of adjustment: those which adjust the survey estimate up (correcting for survey underestimation) those which adjust the survey estimate down (correcting for survey overestimation) those which input a value where none was reported to survey (correcting where survey gives false negatives or hidden recipients) those which completely remove a value* reported to survey (correcting survey for false positives)

9. 9 How large are these adjustments? For pensioner benefits combined (AA, DLA, PC, RP* ) (Adjustments are required for ½ of all cases) adjustments which change the survey estimate up or down range from – £315 to + £227 account for 89% of all pensioner adjustment cases adjustments to input a value** increase survey estimates between + £31 to + £249 and account for 4% of all pensioner adjustment cases. adjustments to remove a value reduce survey estimates by – £330 to – £6 and account for the remaining 7% of all pensioner adjustment cases

10. 10 Our approaches so far: What proportion of cases are adjusted after linking? The net impact of linking consenters has led to small upward adjustments in their income. However among recipients of pensioner benefits, adjustments are needed for half* of consenters. We have therefore assumed a similar proportion** of non-consenters would need an adjustment of some sort.

11. 11 Our approaches so far: Stage 1: Determining which non-consenters data to adjust A logistic regression model was fitted to pensioner consenters using a range of descriptive FRS variables* to explain the likelihood of requiring an adjustment amongst these consenters This regression helps to determine the most significant variables in explaining the need to adjust or not to adjust Applying this model to non-consenters gives the likelihood for requiring an adjustment to the FRS amount reported by this group.

12. 12 Our approaches so far: How successful is explanatory power of our regression model? Here we assess how successful our model is in predicting a probability of adjustment among consenters Predictions align when an adjustment has been made and predicted probability > 0.5 or when no adjustment was made and the predicted probability < 0.5 * The pensioner regression gives 59** per cent of consenter cases as correctly predicted. Among these cases, factors increasing*** the likelihood of needing an adjustment are martial status, region, age. Those decreasing **** likelihood include material deprivation, and poor health.

13. 13 Our approaches so far: Stage 2: Determining which non-consenters data to adjust Applying this model to non-consenters can give the likelihood for requiring an adjustment to the FRS amount actually reported by this group However to actually assign the adjustment status to each non- consenter, this probability is combined with the selection of a random number between 0 and 1 Non-consenters would be given a need to adjust status if the random number was less than or equal to the models predicted probability

14. 14 Our approaches so far: Stage 3: Determining a monetary adjustment for need to adjust non-consenters Once the model has identified the non-consent cases to be adjusted an appropriate monetary adjustment will be assigned to those cases. The correctness of adjustment will then be assessed against appropriate* caseload and monetary administrative totals. To accommodate any bias (evident if discrepancies are maintained between adjusted FRS and admin data) the model proposes to re-visit the designation of adjust / dont adjust to achieve a result which better matches the admin total.

15. 15 Assumptions made so far …… Work described so far is based on models currently in use across DWP. These assume non-consenters behaviours mirror those of consenters. Apart from the end stage to re-adjust monetary values to satisfy comparisons with similar admin expenditure totals – our basic model does not account for situations where non-consenters may be different from the known group of consenters.

16. 16 Questioning assumptions made so far …… We recognise that these (i.e. consent / non-consent) groups are self- selected and it may not be appropriate to assume similarity when designing a model. If bias is detected some attempt should be made to identify the different behaviours and to make different assumptions as to their adjustment. One regression approach which tests for and estimates any selection bias is the two stage Heckman selection model*. A further step of this project can consider this model to identify any bias and build this into assumptions in proposing adjustments

17. 17 Current status of project to adjust non-consenters data Work is ongoing – both in terms of agreeing the most suitable approach to modelling, but also with regard to other key issues concerning the level of detail (and complexity) to which we should model Some key questions…… 1) is it necessary to apply all four types of adjustments for those non- consenters needing adjustment? …… 2) is it necessary to model adjustments at the level of individual benefits / income streams, or would grouping by client type or benefit type (children / pensioners / working-age adults) suffice? …. 3) will adjustments adequately retain the functionality* of current dataset ? 4) are proposals repeatable on an annual basis?

18. 18 Concluding thoughts …. Comparisons which show a net FRS undercount are so far best known in relation to DWP benefits* …. with some evidence of this trend for earnings amongst those with lowest incomes. How far an undercount is relevant to other income sources (e.g. private pensions data; council tax & housing benefit data; Child Maintenance data) is so far unknown.

19. 19 Concluding thoughts …. We aim to extend the linking of consenters data to as many income sources that are available*. Adjustments for non-consenters will also be considered Keen to link income strands likely to be received across the income distribution** (i.e. not only benefits – many of which are disproportionately received by lower income households) We plan to report on the ongoing progress of this work to adjust for non-consenters at other similar public presentations

20. 20 Contacts at DWP: Our team address: team.development@dwp.gsi.gov.uk We invite and welcome your comments / experience in this new area of work