The national assessment system and the use of individual student data: the case of Hungary Educational Information Systems for Innovation and Improvement OECD / SSRC / Stupski Workshop October 2010 New York City Gábor Halász ELTE University, Budapest

The National Assessment of Basic Competencies (NABC) The key features of the system: a system in permanent evolutionThe key features of the system: a system in permanent evolution A system used in various ways by various stakeholdersA system used in various ways by various stakeholders The use of individual student data The dimension of innovation

For more details see the Hungarian country background report to the OECDReview on Evaluation and Assessment Frameworks for Improving School Outcomes (to be published)

Thank you for your attention!

The system ofNational Assessment of Basic Competencies Every pupil assessed in grades 6th, 8th and 10th National tests in literacy and numeracy Individual student identification number used Background data collected on students and schools Centralised scoring of tests at national level by the national managing agency (Educational Authority) Reporting at four levels –National –National level (for the public and decision-makers) –Maintainer –Maintainer level (to every local self-government that maintain schools and other maintainers) –School –School level (to every school) –Pupil –Pupil level (to every school and parent)

A system in evolution Year Which grades? Centralised national scoring and 9Sample (20 students from each school) and 10Sample (20 students, each program types from grade 10) , 8 and 10 No change No changeMixed (Sample in some grades, full in others) 2007-No change Full Full (Every student from every school)

A system in evolution 2001 – First results published (with value added scores based on estimation related with SES) school and maintainer reports publicly available (for those possessing access codes) introduction of individual pupil identifier codes 2009 – data on every pupil available for parents and teachers value added scores calculated on the basis of individual pupil identifier codes

The uses of the system NABC Different stakeholders want to use it for different purposes Summative assessment of individual pupils (---) School evaluation and accountability (+?) Formative assessment of individual pupils for improving learning (--) School improvement (++) Informing parents for school choice (+) Research (+)

The dominant use: school improvement Schools Reports on schools (with comparative data) Reports on municipalities (with comparative data) Reports on individual pupils Municipalities Capacity building Incentives, sanctions School improvement (e.g. curriculum, CPD)

The use of the system: Number of downloads of various types of reports (reports of 2009 downloaded after March 2010) Maintainer (municipality) reports3 919 School reports22 227

The use of data for school improvement In 2004 three from four headmasters reported on using the school results for internal development (survey result) In 2007 every third municipality put these results to use in its managerial work (survey result) Schools may use school reports for marketing themselves marketing themselves

The school report (form the website of a lower secondary school)

The use of individual student data Individual student identifiers and longitudinal measurement initiated by researchers unsatisfied with data quality for measuring value added Long and difficult discussions with the national parliamentary Data Protection and Freedom of Information Commissioner… … ending with an agreement on a technical solution that assures privacytechnical solution

Key issues/1 Privacy (this has not become an issue) Communication with parents (how to enrich the information parents get and how to make data more readable for them) Motivation and capacity (how make schools, teachers and local communities to be more active users of school and pupil level data) Incentives and sanctions (make schools and municipalities act) High costs (this is a major concern)

Key issues/2 Quality of data (controlling better school level data collection) Resource allocation and priorities (investing into improve the quality of data or into improving the use of existing data for development) Speed of feedback (8-9 months – making formative use difficult) Various interests to use the system differently (competing expectations but school improvement dominates – this makes the system relatively exempt from conflicts)

The technical solution/1 The personal identification number (PIN) is generated at school level and remains in the school Those doing the national data processing do not have access to the PIN of individual pupils Test data can be connected with individual pupils only through the PIN of the individual pupil The PIN of the individual pupil is known only by (1) the principal, (2) teachers authorized by him/her, (3) the parents of the pupil

The technical solution/2 Access to test data of individual pupils through two different channels –Through a publicly available website for parents (they have access only to the data of their own child) –Through a non-public website for teachers (here the principal can give various authorizations for access to individual teachers) On the non-public website teachers can create various reports on individual pupils (e.g. data on specific test items, times series) Access to individual pupil data of downloads from the public website (teachers) entry to the non-public website (parents)

Assessment and innovation A National Strategy for Education Sector Innovation prepared in 2010, in which…National Strategy for Education Sector Innovation …assessment and evaluation are seen as key components… …in two dimensions: –The impact of assessment and evaluation on innovation in the education system (eg. enhancing evidence based practice, supporting school development) –Innovations within the assessment and evaluation system (eg. new ICT solutions, new ways of using data, new forms of access)

Assessment and innovation NABC seems to be an important carrier of innovations in both dimensions –Innovative solutions are emerging permanently within in the system (e.g. the test-generator) –A number of schools do sophisticated school and pupil level analyses (and they also establish plans to improve performance in specific areas or in the case of particular pupils)