Presentation on theme: "Daphne Kounali and Tony Robinson Department of Mathematical Sciences"— Presentation transcript:
1Daphne Kounali and Tony Robinson Department of Mathematical Sciences Merging Local Research Data with PLASC: An application on the Appropriateness of Free School Meals as a Measure of Deprivation in Educational Research in Hampshire.Daphne Kounali and Tony RobinsonDepartment of Mathematical SciencesUniversity of Bath
2The HARPS study The HARPS project is an acronym for ‘Hampshire Research with Primary Schools’Objective: Assess the impact of school composition upon student academic progress. The main aim of the study is to estimate and better understand compositional effects at the primary school level.Definitions: Compositional effects are a subset of contextual effects. Compositional effects attempt to operationalise the influences of fellow pupils characteristics on individual pupil’s performance.Compositional variables included in this study will be; social economic status (SES), ethnicity, gender, prior achievement, special educational needs (SEN) and age.A further remit of the study is to evaluate the probity of FSM as a proxy for the economic element of SES.
3Study DesignThe research design is both quantitative and qualitative. Like a set of Russian dolls the project design is of 3 nested parts:A large scale analysis of over 300 primary schoolsA study of a subsample of 46 schools in the Basingstoke and Dean area.More detailed case studies of 12 schools.
4The Basingstoke sub-sample One purpose of the Basingstoke subsample is to test the sensitivity of the quantitative measures and methodologies, including FSM, used to identify compositional effects.To this end we have collected and are analysing detailed family background information from the year 3 children in the 46 Basingstoke schools. These schools have been selected so as to compare the findings from these schools with those of the overall Hampshire cohort.This will not only test the probity of measures such as FSM for the Hampshire dataset but will also throw light on their use for school effectiveness studies in general.
5The Basingstoke subsample dataset contains family background data on 1653 pupils from a total of 2012 students attending 46 out of all 50 schools in the Basingstoke and Dean area. Relevant to economic status these data include:Occupational group, Working status, Home ownership, Whether in receipt of Working Tax Credit, Whether in receipt of FSM.Other information was collected on ethnic background, family structure, out-of-school childcare arrangements and cultural behaviour. These data were collected at the beginning of 2004 for year 3 pupils (7 year old) from these schools.
6One other measure that is becoming popular in research is the Index of Multiple Deprivation (IMD)(Noble et al., 2004)However this relates not directly to individuals but to the small geographical area in which they live, known as a low level Super Output Area (SOA) containing on average about 1500 people. IMD is a composite index based on indices grouped within seven domains:Income; Employment; Health deprivation and disability;Education, skills and training; Barriers to housing and services; Living environment; Crime
7The Deprivation Geography of Hampshire based on the Multiple Deprivation Index Our Basingstoke Schools on the Deprivation MapThe schools are represented by their DFES codes
8As a competitor to FSM, the IMD or its income component could be used as a distant proxy for economic deprivation but technical problems arise when it is desired to assign a deprivation index to an individual who may or may not share the fortunes of an average neighbour.In general it is not too difficult to assign a deprivation index to a school using IMD information from the SOAs from which the school draws its pupils. This is then an alternative to using the percentage of pupils eligible for FSM as an indicator of economic deprivation at the school level. Note that the education component of the IMD contains achievement data on performance at Key Stages 2, 3 and 4 which may well preclude it as independent information used as input to an analysis of such performance.
9FSM and how it is usedFSM is frequently used as a factor in representing economic disadvantage in educational research(studies of attainment or truancy)LEAs incorporate FSM figures in calculating SEN and AEN provisionOnly statutory available information on economic disadvantageUsed as a proxy indicator of economic disadvantage and more generally low SES
10Current FSM eligibility The current eligibility criteria are that parents do not have to pay for school lunches if they receive any of the following:Income SupportIncome-based Jobseeker's AllowanceSupport under Part VI of the Immigration and Asylum ActChild Tax Credit, provided they are not entitled to Working Tax Credit and have an annual income (as assessed by HM Revenue & Customs) that does not exceed £14,155the Guarantee element of State Pension Credit.Children who receive Income Support or income-based Job Seeker's Allowance in their own right qualify as well.
11What is FSMFSM eligibility is a measure of extreme socio-economic disadvantageAt pupil level indicates the pupil’s family economic disadvantageAt school level its aggregate indicates proportion of families suffering extreme economic disadvantageIt is not a direct indicator of non-economic SES disadvantage
12CriticismsThe popularity of such uses of FSM eligibility in studies in educational research is based on its availability. There is no other measure of socio-economic disadvantage that is universally or even widely available at individual pupil level.Registration is voluntary – affected by shame, social class differentials?Schools encourage registration because it affects school incomeDo families consistently qualify over time?Often used as a broad indicator of SES but for example does not directly indicate cultural capital
13The use of Large-Scale Data-Bases on Educational Data Opportunities for advantages of using large pre-existing data setsConceptual and logistic problems in working with these data setsStatistical Issues germane to the analysis of large, complex data bases
14The prosA means for identifying or enumerating cases: they can provide the sampling frame – can be especially helpful in identifying rare events i.e. our subsample statistics indicate that 7.5% incidence of FSM eligibility in the Basingstoke and Dean area compared with a yearly estimate of 9% betweenProvide a broad, if imprecise overview: Frequent existing large data bases can be utilized to measure and define the magnitude and distribution of a problem to more definitive study requiring primary data collection.This is particularly important in studies in social sciences and education where the phenomena under study are not themselves directly measurable and must be studied indirectly through the measurements of other observable phenomena.This is true for both the outcomes we are measuringi.e. we want student progress and we measure test performanceas well as the predictors usedi.e. we want socioeconomic disadvantage and we measure a number of indicators for this including FSM.
15Could provide directions of the size of measurement error in our predictors as well as the direction of bias induced by non-response. Our Basingstoke subsample was found to have higher intakes (baseline tests) in literacy, lower incidence of SEN and FSM compared with the rest of Hampshire.Provide opportunities to Validate the quality of research data in terms of what these indicators actually measure as well as address missingness. In our sample for example non-responders were found to be twice more likely to be FSM eligible (using the PLASC information)
16Conceptual and structural pitfalls They can be frequently flawed. There are 2 reasons for this and both related to the fact that those who design and compile data bases cannot consider the particular needs of every person who eventually might use them.First, PLASC is primarily used for administrative functions related to LEA funding and related policies not always explicitly stated and could be changing. It is simply not designed or maintained to maximize data quality or consistency. Codes and records layouts (i.e. SEN) may be changed periodically.Data elements (i.e. UPN’s from the earlier years might be unreliable pupil identifiers and the same applies for school identifiers after school mergings or closings) are often incomplete and unreliably coded. Information might be inconsistently recorded across LEAs (i.e. literacy tests at reception)
17Pitfalls continue …Second, even in well-maintained data bases (Hampshire LEA with a long history of research collaborations) the original goals that dictated what data were collected will not coincide with those of subsequent users and coding practices (i.e. SEN assessed externally) might make secondary analysis difficult if not impossible. Inclusion of both persons and services might be linked to program policies that might change over time. Another example is associated with the LEA’s communication with the individual school systems and heterogeneity among schools in the type of the actual tests administered (e.g. QCA). In most cases we do not know:How were the data collected and processedThe time-frame of data collectionCompleteness and coherence among databases of different LEAs
18Another example of complex problems in usage Our original population cohort consists of (51% boys) of all Hampshire pupils who took the baseline test during and their KS1 tests during We have test results for approximately 84% of this cohort. The Hampshire-wide size of the cohort of Reception pupils from PLASC 2001/02 is and the size of the Yr2 pupils from PLASC 2003/04 isThe pattern of longitudinal losses in terms of test-results seems quite typical for Hampshire for this phase, judging for other historical cohort data that we also had access to. A wave of around 2000 pupils are lost as we move in time because they are leaving Hampshire schools and another wave of 2000 new pupils are coming-into to these schools.Because of data-inconsistencies related to correctly identifying and recording school changes (school merging or closing) we end-up looking at pupils for the phase Baseline to KS1 data. These pupils come from 318 (Infant or Junior and Primary Schools) at baseline and 302 schools at KS1.
19Back to FSM in HARPS and the pros Some subsample statisticsOut of the 1653 Basingstoke families 124 were receiving FSM (7.5%)Of these 13 claimed to receive working tax-creditSingle parents and non-working couples account for the majority of FSM recipientsSo FSM is a very coarse indicator, can we do better?What about the economically deprived that do not fall below the FSM economic threshold?No indication of how much the “better off” actually are?
20A scale of economic status We estimate a latent score for the unobserved economic status using relevant dataand Use thisInstead of FSMas input to modellingX1Y1X2LatentIndexY2Y3X3X1: Working statusX2: SES (assessed by occupation records)X3: education level of the parentY1: Tax-CreditY2: FSM eligibilityY3: Home RentingA graphical representation of theMIMIC model (Multiple-Indicator-Multiple-Cause)
21FSM stabilityThere is a widely held view that FSM eligibility status is stable over time as the following quote shows:
26Assessing the effect of deprivation on school performance So, FSM does NOT work
27More Consequences of the choice of deprivation index Ranking school Under-Performance School Ranks for the Lowest threshold of the KS1 testing scale:Comparison of school rankings based on FSM and our index suggest that schools can be judged performing worse when FSM is used instead of the deprivation index.So, when we use FSM we tend to under-estimate school performance.This is because FSM eligibility under-estimates the prevalence of economic deprivation.Moreover, we also under-estimate the uncertainty associated with these rankings.The differences in rankings affect mostlyAverage (typical) performing schoolsThe plots depict the school-level residuals(probit scale) against their rank-order.Large negative values in the probit scale characterize the best performing schools.
28Moreover, the bias also seem to be differential The area where FSM under-estimates school performance is below the diagonal line. Higher Residual Ranks indicate Worse Performance61% - 71% (depending on the residuals considered for each threshold of the KS1 scale)of the schools are judged as performing worse when using FSM rather than using our index.Moreover, the bias also seem to be differentialMore deprived schools (in terms of intake and/or FSM eligibility) being affected more.
29Derivation of the index of disadvantage A MIMIC model (Multiple-Indicator-Multiple-Cause) model was used for the derivation of a scale representing economic disadvantage. Binary data on three indicators of economic disadvantageY1: working tax-credit, Y2: FSM eligibility, Y3: Home renting) are usedin a two-parameter logistic model (Rasch-model) as follows:logit[Pr(yijj)]= diji+ dij ij, for indicator i=1,2,3 and subject jwhere di’ is a three-dimensional vectorfor the i-th element equal to 1 and all other elements equal to zeroand is the latent unobserved measure of economic disadvantage.The parameter estimates for i (location parameters, or intercepts) represent the potency of each indicator.The parameters i (factor loadings) represent the discriminating ability of each indicator.
30We also specified a structural model for the latent economic disadvantage, allowing the mean indicator potencies to differ between groups as follows:j=0+1X1+2X2+3X3+di1X1,where X1: working status, X2: SES assessed from occupation records, X3: level of educationThe term di1X1 allows us to introduce a direct effect of working tax credit on the receipt of working tax credit, apart from its indirect effect through the latent variable.This is needed since only employed people would bein receipt of tax-credit.
31The intercepts’ estimates suggest that the estimated indicator potencies increase from FSM to renting with FSM being the most difficult benefit to get. FSM eligibility does have the highest discriminating ability (slope) as expected followed by home renting.As expected the estimated parameter indicates that the odds of receipt of of working tax-credit are significantly reduced for non-working couples compared with the typical couple where one is working full-time and the other part-time.
32Higher SES reduces economic disadvantage compared to middle classes Low or unknown SES increases economic disadvantage.Higher education level is associated with reduced economic disadvantage.Non-working couple are significantly more likely to be economically disadvantaged, as expected, compared with the typical working status of a couple in this sample (one working full-time and one part time) with the single part-time parents following closely.Couples consisting of partners where only one full-time working time are also significantly more disadvantaged compared to the typical working couple which appears similar to couples where both partners work full-time.It is interesting to note that couples consisting of partners where one is working full-time and the other is self-employed, are significantly less disadvantaged compared with the typical couple
33very disadvantaged children even among low SES families Working tax-credit is a muchMore useful indicator especiallyFor middle class occupationsFSM refers tovery disadvantaged childreneven among low SES families
34Receipt of Working tax credit as an indicator of economic disadvantage seem to INDEXoccupation based SES, MOSTLY but not always.This might be due to bias associated withreporting occupation and/orassumptions on occupation rankings.Receipt of Working tax creditas an indicator of economic disadvantage seem to INDEXlevel of parental education less well.It seems that level of education is consistent with economic disadvantage for the very poor.The economic value of vocational qualifications seem less clear for more typical incomes.This could also be due to the fact that in our sample90% of the responders are women and only 60% of these women contribute to the determination of SES.
35Further developments Improvements in the construction of the index: Allow for differential measurement error through joint modelling instead of plugging-in index estimates.In the structural part of the model include interaction terms for education level and the identity of the major bread-winner.Address problems related to endogeneity i.e. economic disadvantagemight affect both school intakes as well as KS1 test results,and there might be both direct and indirect effectsof economic disadvantage on school performance.
37As far as the use of PLASC data is concerned the conclusion is Documentation and Documentation and …The importance for the research community to get involved with the development and progress of maintaining and improving such invaluable data resources as the PLASC data.