Presentation on theme: "Building the evidence of system performance What are we learning about the tertiary education system? ATEM NZ Conference 10 July 2007."— Presentation transcript:
Building the evidence of system performance What are we learning about the tertiary education system? ATEM NZ Conference 10 July 2007
Page 2New Zealand Ministry of Education Lies, damned lies and statistics ▲ Government collects large amounts of data about education – and tertiary education in particular. From it, we can work out many facts. ▲ But facts without context are not much more than trivia. And data is useful only if it helps us to answer ‘important’ questions. ▲ The really important questions: ▲ how well do our students do in completing their studies? ▲ how well do they do after they leave study and in what ways? ▲ how much does our system add value? ▲ It is all about gathering evidence
Page 3New Zealand Ministry of Education This presentation ▲ Looks at the Ministry of Education’s tertiary education analytical programme ▲ Where it came from ▲ Current priorities ▲ Some examples ▲ What do you want? And what is missing?
Page 4New Zealand Ministry of Education Building evidence of performance in tertiary education ▲ Government, institutions and students all have a stake. ▲ Need to improve study outcomes for many. ▲ Need to get good value for money. ▲ Policy shift from emphasis on participation to outcomes. ▲ Funding changes. ▲ Historically, lack of evidence has hampered our understanding.
Page 5New Zealand Ministry of Education Where we came from ▲ Unit record data has been collected since early 90s on parts of the tertiary education system ▲ From 2000, the collection was enhanced – with the SDR ▲ So there was a great and rich resource ▲ But little use was made of it, except internal analyses to underpin some policy projects ▲ With the 2002 reforms, the ministry needed to take a system view of performance – and saw a greater need for public accountability information ▲ So we set about creating an evidence base from the available data ▲ to underpin policy ▲ to provide accountability to public ▲ and to support sector planning
Page 6New Zealand Ministry of Education Where we came from ▲ Where to start …. ▲ First, we analysed information and evidence needs ▲ Conclusions ▲ a focus on performance ▲ and on outcomes ▲ a system focus ▲ But to assess the performance of a system, we need to look at value created ▲ And that means primarily looking outside the system itself ▲ what was a person’s status before ▲ and what happens after leaving ▲ plus something about how things work inside
Page 7New Zealand Ministry of Education Where we came from ▲ While we have great data, one issue was its usability ▲ A key issue was the lack of links across time and across providers ▲ NSNs would eventually resolve that issue, but waiting would take years ▲ So in the meantime, statistical matching enabled us to do completion rates
Page 8New Zealand Ministry of Education Where we came from ▲ But looking at SDR data – even with matching – didn’t give us outcomes data ▲ Especially given the need to assess performance by what happens after study ▲ The integrated dataset on Student Loan Scheme borrowers allows us to establish precise linkages between an individual’s education profile and post-study income – a good proxy for outcomes ▲ Although coverage is limited – it applies only to those who use the loan scheme – this was also a breakthrough ▲ But issues remain – data on industry training and targeted programmes is held separately – and how do we measure pre-entry status ▲ The NSN is now at a stage where we can start to address these issues
Page 9New Zealand Ministry of Education We set ourselves some goals ▲ Our mission ▲ leadership in analysis …. ▲ production and dissemination …. ▲ timely, authoritative and relevant the tertiary education system its interaction with the wider education sector its contribution to New Zealand’s longer term social and economic outcomes. ▲ Our role ▲ we look at a system that is made up of components ▲ the TEC looks at components within a system ▲ but we apply common or complementary analytical approaches to common data
Page 10New Zealand Ministry of Education What is happening now ▲ We do summary reports ▲ like Profile and Trends and TES monitoring ▲ they are comprehensive ▲ and summarise analyses ▲ And more detailed reports on aspects of the system ▲ Mainly our approach is quantitative not qualitative ▲ We focus on four themes ▲ Learners ▲ Post-study outcomes ▲ Supporting the system ▲ Research and knowledge creation ▲ Education Counts has over 60 reports on many aspects of the system ▲ Examples …
Page 11New Zealand Ministry of Education What is happening now
Page 12New Zealand Ministry of Education What is happening now More examples ▲ Who doesn’t pay back their loans? ▲ What determines the research performance of staff? ▲ How long do people spend in tertiary education? ▲ A changing population and the NZ tertiary sector ▲ Participation in tertiary education – a birth cohort approach ▲ He tini manu reo ▲ Maori student performance in degrees
Page 13New Zealand Ministry of Education So what do we find? ▲ Some examples ▲ Post-study outcomes ▲ Passing courses and completing qualifications ▲ Research performance
Page 14New Zealand Ministry of Education Outcomes – estimated earnings by level and gender
Page 15New Zealand Ministry of Education Outcomes – estimated earnings 5 years post-study by level and ethnic group
Page 16New Zealand Ministry of Education So what do we find? Completions ▲ 50%-55% completion rate for bachelors degree. ▲ 80% pass rate for bachelors courses. ▲ Rates are higher at postgraduate level and lower at sub- degree level. ▲ 25% pass all their courses without gaining a qualification. ▲ Full-time students have much higher rates, even allowing for the extra time part-timers need to complete. ▲ NZ compares below average internationally, but reasonably well when adjusted for its high level of part- time study.
Page 17New Zealand Ministry of Education So what do we find? Completions
Page 18New Zealand Ministry of Education Bibliometric analysis of university research
Page 19New Zealand Ministry of Education Bibliometrics vs PBRF scores Biological sciences
Page 20New Zealand Ministry of Education Where are we going? ▲ Breakthroughs in data infrastructure lead to analytical breakthroughs ▲ Links between agencies’ datasets ▲ Broadening of integrated data ▲ Some projects ▲ More on research and on the PBRF How has the PBRF affected attrition in research degrees? Statistical analyses of shifts ▲ Industry training ▲ Transitions – how does school performance impact on tertiary participation and on tertiary achievement ▲ Improving statistics – quality, timeliness and usability
Page 21New Zealand Ministry of Education The challenge for the future ▲ As we break new ground, we shift the frontier ▲ We are changing the way we look at and talk about tertiary education ▲ And the new Tertiary Education Strategy – what demands is that making? ▲ Hard to get at information ▲ For instance …. ▲ Improving research connections and linkages … ▲ … to create economic opportunities ▲ Increasing language, literacy and numeracy levels for the workforce
Page 22New Zealand Ministry of Education And how can you help ▲ What do you need to know? ▲ What is missing?
Page 23New Zealand Ministry of Education ▲ Our task – build the evidence base ▲ Our achievement ▲ lots more material ▲ better statistics ▲ better analysis ▲ But a way to go ▲ Our challenge ▲ data infrastructure ▲ improve the relevance of the analyses ▲ the hard questions