OFFICE OF EDUCATION Consultations – 13 August 2014 Work to date related to the ELP loading

Two national projects commissioned by the SPWG Investigate any improvements that could be made to the DLBOTE measure. Conduct a cost-benefit analysis of a trial of a more accurate, nationally consistent ELP measure.

OFFICE OF EDUCATION Is the DLOBE measure identifying the right students for support? Three issues: Limited coverage Low accuracy Significantly underestimates target cohort size

OFFICE OF EDUCATION Misalignment between Disadvantaged LBOTE measure and low English proficiency These are the students who are captured by the disadvantaged LBOTE measure These are the students who need to be captured at a minimum by a low ELP measure Note: based on 2012 student background data extracted from ERN Students with low ELP are around 4 times more likely to have parents who have achieved Year 12 school education than to have parents of the lowest school education level Parental Education Levels

OFFICE OF EDUCATION Improvements to Disadvantaged LBOTE Group Size of group Size equivalence (group size as % of target cohort size) Intersect (no. in group also in target cohort) Coverage (% target cohort represented) Accuracy (% of group accurately identified) Target cohort: Students with limited English proficiency (44,494) Disadvantaged LBOTE21,62649%5, %25.3% Disadvantaged LBOTE, OR LBOTE and <=1 year in Aust school 42,95797%18, %42.8% Disadvantaged LBOTE, OR refugee students 25,50157%7, %27.9% Disadvantaged LBOTE OR NAP students 25,94858%8, %32.2% NAP: students eligible for New Arrivals Program

OFFICE OF EDUCATION Alternatives to Disadvantaged LBOTE GroupSize of group Size equivalence (group size as % of target cohort size) Intersect (no. in group also in target cohort) Coverage (% target cohort represented) Accuracy (% of group accurately identified) Target cohort: Students with limited English proficiency (44,494) Disadvantaged LBOTE21,62649%5, %25.3% LBOTE and not born in Aust62,086140%17, %27.5% LBOTE and <=2 years in Aust school 47,311106%24, %52.7% LBOTE and <=1 year in Aust school 23,91654%14, %61.3% LBOTE and <=2 years in Aust school and not born in Aust 18,00541%10, %57.5% Refugee students6,40914%2,8606.4%44.6% NAP students5,90413%3,3727.6%57.1% Refugee OR NAP students12,15927%6, %49.5%

OFFICE OF EDUCATION Conclusions The Disadvantaged LBOTE measure should not be used in the funding model None of the improvements or alternatives to the Disadvantaged LBOTE measure investigated reached the required level of efficacy A new, nationally consistent measure is needed that is both more accurate and has greater validity The NSW trial of EAL/D Learning Progression demonstrates the EAL/D measure provides an accurate identification of students with low English language proficiency.

OFFICE OF EDUCATION NSW trial of the EAL/D instrument EAL/D Learning Progression: EAL/D was developed by ACARA in 2011 The purpose was to indicate the level of support needed for students to access learning area content Provides broad descriptions of the characteristics of learner groups at each of four phases of English language learning (Beginning, Emerging, Developing and Consolidating) across four language modes – listening, and speaking, reading and writing The trial focused on: 1. Reliability of teacher judgements (inter-rater agreement and correlations between teacher judgements) 2. Appropriateness of EAL/D assessments for resource allocation purposes (discriminant, concurrent, measurement, and structural validity) 3. Teacher acceptance of EAL/D

OFFICE OF EDUCATION Reliability Reliability statistics included exact and adjacent agreement rates, Kappa rates and score reliability coefficient

OFFICE OF EDUCATION Validity – Discriminant evidence Can teachers discriminate the four language modes appropriately? Multitrait-Multimethod analysis method Teacher 2 judgement All correlations on the same mode exceed correlations between different modes, providing a satisfactory level of discriminant validity evidence

OFFICE OF EDUCATION Construct validity - Concurrent validity evidence (relationship between EAL/D levels and NAPLAN results) A high level of congruence is evident between EAL/D assessments and NAPLAN results for the same students

OFFICE OF EDUCATION Measurement quality of EAL/D assessment data Research questions Are the modes measuring a single latent ability – general English proficiency? Are the scales used for measuring the four modes functioning as intended? Can the assessed levels be summarised to derive a single measure of proficiency? The degree to which the modes are measuring the same construct The Rasch Partial Credit Model was used to provide evidence that the data fits a uni-dimensional model. Diagnostic analyses demonstrated: Global model fit statistics Mean and standard deviation of standardised residuals Proportion of most unexpected responses Mode level fit statistics (infit and outfit) Category probability curve/Item Characteristics Curves Thresholds were ordered as expected

OFFICE OF EDUCATION Summary of EAL/D trial in NSW 1.The trial demonstrated the EAL/D instrument is reliable and easy to use 2.Teachers could readily use the EAL/D to discriminate consistently between different development levels within and across each of the four language modes 3.There is strong evidence from the trial to support the use of the EAL/D to derive a single measure of the ELP for each student for the purposes of resource allocation. Teacher survey results also indicated: 1.Teachers’ support for use of the EAL/D for resource allocation in place of the current ESL phase tool 2.Professional learning provided in the trial was adequate for confident and consistent judgements using the EAL/D 3.Teachers required more time to plan and do assessments to make informed phase judgements

OFFICE OF EDUCATION Overlapping between ELP and SEA Note: 2014 preliminary EAL/D data, NSW government schools SEA Quarters