1. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 1 Consider the Evidence Evidence-driven decision making for secondary schools A resource to assist schools to review their use of data and other evidence 1

2. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 2 Evidence-driven decision making Today we aim to think about how we use data and other evidence to improve teaching, learning and student achievement improve our understanding, confidence and capability in using data to improve practice discuss how we make decisions think about our needs and start to plan our own evidence-based projects

3. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 3 Evidence-driven eating You need to buy lunch. Before you decide what to buy you consider a number of factors: how much money do you have? what do you feel like eating? what will you be having for dinner? how far do you need to go to buy food? how much time do you have? where are you going to eat it?

4. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 4 Evidence-driven teaching I had a hunch that Ana wasn’t doing as well as she could in her research assignments, a major part of the history course. What made me think this? Ana’s general work (especially her writing) was fine. She made perceptive comments in class, contributed well in groups and had good results overall last year, especially in English. How did I decide what to do about it? I looked more closely at her other work. I watched her working in the library one day to see if it was her reading, her use of resources, her note taking, her planning, or what. At morning tea I asked one of Ana’s other teachers about Ana’s approach to similar tasks. I asked Ana if she knew why her research results weren’t as good as her other results, and what her plans were for the next assignment. I thought about all of this and planned a course of action. I gave her help with using indexes, searching, note taking and planning and linking the various stages of her research.

5. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 5 Consider the Evidence A resource to assist schools to review their use of data and other evidence What is meant by ‘data and other evidence’?

6. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 6 Evidence Any facts, circumstances or perceptions that can be used as an input for an analysis or decision how classes are compiled, how classes are allocated to teachers, test results, teachers’ observations, attendance data, portfolios of work, student opinions … Data are one form of evidence

7. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 7 Data Known facts or measurements, probably expressed in some systematic or symbolic way (e.g. as numbers) assessment results, gender, attendance, ethnicity … Data are one form of evidence

8. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 8 Which factors are data? Evidence to consider before buying lunch how much money you have what you feel like eating what you’ll be having for dinner how far you need to go to buy food how much time you have where you’re going to eat what your diet allows

9. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 9 Evidence-driven decision making We have more evidence about what students know and can do than ever before – their achievements, behaviours, environmental factors that influence learning We should draw on all our knowledge about the learning environment to improve student achievement explore what lies behind patterns of achievement decide what changes will make a difference

10. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 10 What evidence does a school have? Demographics Student achievement Perceptions School processes Other practice

11. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 11 Demographics What data do we have now to provide a profile of our school? What other data could we create? School Students Staff Parents/caregivers and community

12. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 12 Demographics Data that provides a profile of our school School – decile, roll size, urban/rural, single sex or co- educational, teaching spaces … Students – ethnicity, gender, age, year level, attendance, lateness, suspension and other disciplinary data, previous school, part-time employment … Staff – gender, age, years of experience, qualifications, teaching areas, involvement in national curriculum and assessment, turnover rate … Parents/caregivers and community – socio-economic factors, breadth of school catchment, occupations …

13. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 13 Student achievement What evidence do we have now about student achievement? What other evidence could we collect? National assessment results Standardised assessment results administered internally Other in-school assessments Student work

14. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 14 Student achievement Evidence about student achievement National assessment results - NCEA, NZ Scholarship - details like credits above and below year levels, breadth of subjects entered… Standardised assessment results administered internally - PAT, asTTle … Other in-school assessments - most non-standardised but some, especially within departments, will be consistent across classes - includes data from previous schools, primary/intermediate Student work - work completion rates, internal assessment completion patterns, exercise books, notes, drafts of material - these can provide useful supplementary evidence

15. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 15 Perceptions What evidence do we have now about what students, staff and others think about the school? Are there other potential sources? Self appraisal Formal and informal observations made by teachers Structured interactions Externally generated reports Student voice Other informal sources

16. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 16 Perceptions Evidence about what students, staff, parents and the community think about the school Self appraisal - student perceptions of their own abilities, potential, achievements, attitudes … Formal and informal observations made by teachers - peer interactions, behaviour, attitudes, engagement, student-teacher relationships, learning styles, classroom dynamics … Structured interactions - records from student interviews, parent interviews, staff conferences on students … Externally generated reports - from ERO and NZQA (these contain data but also perceptions) … Student voice - student surveys, student council submissions … Other informal sources – views about the school environment, staff and student morale, board perceptions, conversations among teachers …

17. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 17 School processes What evidence do we have about how our school is organised and operates? Timetable Classes Resources Finance Staffing

18. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 18 School processes Evidence about how our school is organised and operates School processes - evidence and data about how your school is organised and operates, including: Timetable –structure, period length, placement of breaks, subjects offered, student choices, tertiary and workforce factors, etc Classes - how they are compiled, their characteristics, effect of timetable choices, etc Resources - access to libraries, text books, ICT, special equipment, etc Finance - how the school budget is allocated, how funds are used within departments, expenditure on professional development Staffing - policies and procedures for employing staff, allocating responsibility, special roles, workload, subjects and classes

19. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 19 Other practice How can we find out about what has worked (or not) in other schools?

20. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 20 Other practice How we can find out about what has worked in other schools? Documented research – university and other publications, Ministry of Education’s Best Evidence Syntheses, NZCER, NZARE, overseas equivalents … Experiences of other schools – informal contacts, local clusters, advisory services, TKI LeadSpace …

21. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 21 What can we do with evidence? Shane’s story A history HOD wants to see whether history students are performing to their potential. She prints the latest internally assessed NCEA records for history students across all of their subjects. As a group, history students seem to be doing as well in history as they are in other subjects. Then she notices that Shane is doing very well in English and only reasonably well in history. She wonders why, especially as both are language-rich subjects with many similarities. The HOD speaks with the history teacher, who says Shane is attentive, catches on quickly and usually does all work required. He mentions that Shane is regularly late for class, especially on Monday and Thursday. So he often misses important information or takes time to settle in. He has heard there are ‘problems at home’ so has overlooked it, especially as the student is doing reasonably well in history.

22. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 22 Shane’s story (cont...) The HOD looks at the timetable and discovers that history is Period 1 on Monday and Thursday. She speaks to Shane’s form teacher who says that she suspects Shane is actually late to school virtually every day. They look at centralised records. Shane has excellent attendance but frequent lateness to period 1 classes. The HOD speaks to the dean who explains that Shane has to take his younger sister to school each morning. He had raised the issue with Shane but he said this was helping the household get over a difficult period and claimed he could handle it. The staff involved agree that Shane’s regular lateness is having a demonstrable impact on his achievement, probably beyond history but not so obviously. The dean undertakes to speak to the student, history teacher, and possibly the parents to find a remedy for the situation.

23. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 23 23 Thinking about Shane’s story What were the key factors in the scenario about Shane? What types of data and other evidence were used? What questions did the HOD ask? What happened in this case that wouldn’t necessarily happen in some schools?

24. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 24 Shane’s story - keys to success The history HOD looked at achievement data in English and history. She looked for something significant across the two data sets, not just low achievement. Then she asked a simple question: Why is there such a disparity between in these two subjects for that student? She sought information and comments (perceptions evidence and data) from all relevant staff. The school had centralised attendance and punctuality records (demographic data) that form teacher could access easily. The action was based on all available evidence and designed to achieve a clear aim.

25. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 25 Evidence-driven strategic planning If we use evidence-driven decision making to improve student achievement and enhance teaching practice … … it follows that strategic planning across the school should also be evidence-driven.

26. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 26 Evidence-driven strategic planning. INDICATORS FROM DATA asTTle scores show a high proportion of year 9 achieving below curriculum level NCEA results show high non- achievement in transactional writing Poor results in other language NCEA standards etc. STRATEGIC GOAL To raise the levels of writing across the school Strategic action Develop a writing development plan which addresses writing across subjects and levels, including targets, professional development and other resourcing needs etc. ANNUAL PLAN Develop and implement a plan to raise levels of Writing at year 9 Development plan to be based on an analysis of all available data and to include a range of shared strategies etc. YEAR TARGET Raise writing asTTle results year 9 boys from 3B to 3A etc. Appraisal P D Self review School charter EVALUATION DATA asTTle writing results improve by … Perception data from Yr 9 staff indicates … Evaluation of effectiveness of range of shared strategies, barriers and enablers … etc

27. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 27 The evidence-driven decision making cycle Trigger Explore Question Assemble Analyse Interpret Intervene Evaluate Reflect

28. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 28 The evidence-driven decision making cycle Trigger Clues found in data, hunches ExploreIs there really an issue? Question What do you want to know? AssembleGet all useful evidence together Analyse Process data and other evidence Interpret What information do you have? Intervene Design and carry out action Evaluate What was the impact? Reflect What will we change?

29. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 29 The evidence-driven decision making cycle Explore Check data and evidence to explore the issue Question Clarify the issue and ask a question Assemble Decide what data and evidence might be useful Analyse data and evidence Trigger Data indicate a possible issue that could impact on student achievement Evaluate the impact on the intervention Intervene Plan an action aimed at improving student achievement Interpret Insights that answer your question Speculate A teacher has a hunch about a problem or a possible action Act Carry out the intervention Reflect on what has been learned, how practice will change

30. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 30 The evidence-driven decision making cycle EXPLORE QUESTION ASSEMBLE ANALYSE TRIGGER EVALUATE INTERVENEINTERPRET SPECULATE ACT REFLECT

31. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 31 The evidence-driven decision making cycle. Explore data Survey of students shows that this is only partially true Question What are the characteristics of students who are poor at writing ? Assemble more data & other evidence: asTTle reading, homework, extracurric, attendance, etc. Analyse NQF/NCEA results by standard Trigger Significant numbers not achieving well in writing Analyse non NQF/NCEA data and evidence Intervene Create multiple opportunities for writing; include topics that can use sport as context; connect speaking and writing. PD for staff. Interpret information Poor writers likely to play sport, speak well, read less, do little HW A teacher has a hunch - poor writers might spend little time on homework Evaluate Has writing improved? Reflect How will we teach writing in the future?

32. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 32 Evaluate and reflect Summative evaluation – assess how successful the intervention was; decide how our practice will change; report to board Formative evaluation – at every stage in the cycle we reflect and evaluate Are we are on the right track? Do we need to fine-tune? Do we actually need to complete this?

33. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 33 Types of analysis We can compare achievement data by subject or across subjects for an individual student groups of students whole cohorts The type of analysis we use depends on the question we want to answer

34. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 34 Inter-subject analysis Have my students not achieved a particular history standard because they have poor formal writing skills, rather than poor history knowledge?

35. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 35 Intra-subject analysis What are the areas of strength and weakness in my own teaching of this class?

36. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 36 Longitudinal analysis Are we producing better results over time in year 11 biology?

37. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 37 The evidence-driven decision making cycle >Trigger Clues found in data, hunches ExploreIs there really an issue? Question What do you want to know? AssembleGet all useful evidence together Analyse Process data and other evidence Interpret What information do you have? Intervene Design and carry out action Evaluate What was the impact? Reflect What will we change?

38. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 38 Asking questions Evidence-driven decision making starts with asking good questions You can tell whether a man is clever by his answers. You can tell whether he is wise by his questions. Nobel Prize winner, Naguib Mahfouz

39. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 39 Trigger questions How good/poor is …? What aspects of … are good/poor? Is … actually changing? How is … changing? Is … better than last year? How can … be improved? Why is … good/poor? What targets are reasonable for …? What factors influence the situation for …? What would happen if we …? Formative or summative?

40. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 40 Summative questions A target in the school’s annual plan is for all year 10 boys to improve their writing level by at least one level using asTTle (e.g. from 4B to 4A). Have all year 10 boys improved by at least one asTTle level in writing?

41. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 41 Questions about policy We have been running 60-minute periods for 5 years now. What effect has the change had?

42. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 42 Formative questions from data The data suggest our students are achieving well in A, but less well in B. What can we do about that?

43. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 43 Formative questions from data A significant proportion of our school leavers enrol in vocational programmes at polytechnic or on-job. How well do our school programmes prepare those students?

44. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 44 Questions from hunches I suspect this poor performance is being caused by … Is this true? We reckon results will improve if we put more effort into... Is this likely? I think we’d get better results from this module if we added … Is there any evidence to support this idea?

45. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 45 Hunches from raw data

46. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 46 Hunches from raw data Is the class as a whole doing better in internally assessed standards than in externally assessed standards? If so, why? Are the better students (with many Excellence results) not doing as well in external assessments as in internal? If so, why? Is there any relationship between absences and achievement levels? It seems not, but it’s worth analysing the data to be sure.

47. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 47 The evidence-driven decision making cycle Trigger Clues found in data, hunches >ExploreIs there really an issue? Question What do you want to know? AssembleGet all useful evidence together Analyse Process data and other evidence Interpret What information do you have? Intervene Design and carry out action Evaluate What was the impact? Reflect What will we change?

48. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 48 Question – Explore – Question It looks like our students are doing well in A but not in B. What can we do about it? EXPLORE … what else should we be asking? Is this actually the case? Is there anything in the data to suggest what we could do about it?

49. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 49 Question – Explore – Question We have been running 60-minute periods for a year now. Did the change achieve the desired effects? EXPLORE … what else should we be asking? How has the change impacted on student achievement? Has the change has had other effects? Is there more truancy? Is more time being spent in class on assignments, rather than as homework?

50. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 50 The evidence-driven decision making cycle Trigger Clues found in data, hunches ExploreIs there really an issue? >Question What do you want to know? AssembleGet all useful evidence together Analyse Process data and other evidence Interpret What information do you have? Intervene Design and carry out action Evaluate What was the impact? Reflect What will we change?

51. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 51 A very good question Specific and with a clear purpose Able to be investigated through looking at data and other evidence Likely to lead to information on which we can act

52. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 52 Questions with purpose What do we know about reported bullying incidents for year 10 students? MAY BE BETTER AS Who has been bullying whom? Where? What are students telling us? What does pastoral care data tell us? Were some interventions more effective with some groups of students than others?

53. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 53 Write more purposeful questions 1.What are the attendance rates for year 11 students? 2.What has been the effect of the new 6-day x 50-min period structure? 3.How well are boys performing in formal writing in year 9? 4.What has been the effect of shifting the lunch break to after period 4?

54. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 54 More purposeful questions 1.How do year 11 attendance rates compare with other year levels? Do any identifiable groups of year 11 students attend less regularly than average? 2.Is the new 6-day x 50-min period structure having any positive effect on student engagement levels? Is it influencing attendance patterns? What do students say? 3.Should we be concerned about boys’ writing? If so, what action should we be taking to improve the writing of boys in terms of the literacy requirements for NCEA Level 1? 4.The new timing of the lunch break was intended to improve student engagement levels after lunch. Did it achieve this? If so, did improvements in student engagement improve student achievement? Do the benefits outweigh any disadvantages?

55. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 55 The evidence-driven decision making cycle Trigger Clues found in data, hunches ExploreIs there really an issue? Question What do you want to know? >AssembleGet all useful evidence together Analyse Process data and other evidence Interpret What information do you have? Intervene Design and carry out action Evaluate What was the impact? Reflect What will we change?

56. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 56 Assembling the evidence We want to know if our senior students are doing better in one area of NCEA biology than another. So … we need NCEA results for our cohort. It could be that all biology students do better in this area than others. So … we also need data about national differences across the two areas.

57. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 57 Are our data any good? A school found that a set of asTTle scores indicated that almost all students were achieving at lower levels than earlier in the year. Then they discovered that the first test had been conducted in the morning, but the later test was in the afternoon and soon after the students had sat a two-hour exam.

58. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 58 Think critically about data Was the assessment that created this data assessing exactly what we are looking for? Was the assessment set at an appropriate level for this group of students? Was the assessment properly administered? Are we comparing data for matched groups?

59. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 59 Cautionary tale 1 You want to look at changes in a cohort’s asTTle writing levels over 12 months. Was the assessment conducted at the same time both years? Was it administered under the same conditions? Has there been high turnover in the cohort? If so, will it be valid to compare results?

60. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 60 Cautionary tale 2 You have data that show two classes have comparable mathematics ability. But end-of-year assessments show one class achieved far better than the other. What could have caused this? Was the original data flawed? How did teaching methods differ? Was the timetable a factor? Did you survey student views? Are the classes comparable in terms of attendance, etc?

61. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 61 The evidence-driven decision making cycle Trigger Clues found in data, hunches ExploreIs there really an issue? Question What do you want to know? AssembleGet all useful evidence together >Analyse Process data and other evidence Interpret What information do you have? Intervene Design and carry out action Evaluate What was the impact? Reflect What will we change?

62. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 62 Analysing data and other evidence Schools need some staff members who are responsible for leading data analysis Schools have access to electronic tools to process data into graphs and tables All teachers do data analysis Data is not an end in itself – it’s one of the many stages along the way to evidence-driven decision making

63. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 63 Basic analysis

64. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 64 Basic analysis Divide the class into three groups on the basis of overall achievement Identify students who are doing so well at level 2 that they could be working at a higher level Find trends for males and females, those who are absent often, or have many detentions Compare this group’s external assessment success rate with the national cohort.

65. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 65 Reading levels – terms 1 and 4.

66. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 66 Making sense of the results Think about significance and confidence How significant are any apparent trends? How much confidence can we have in the information?

67. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 67 Making sense of the results This table shows that reading levels overall were higher in term 4 than in term 1. Scores improved for most students. 20% of students moved into level 5. But the median score is still 4A. Is this information? Can we act on it?

68. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 68 Information Knowledge gained from analysing data and making meaning from evidence. Information is knowledge (or understanding) that can inform your decisions. How certain you will be about this knowledge depends on a number of factors: where your data came from, how reliable it was, how rigorous your analysis was. So the information you get from analysing data could be a conclusion, a trend, a possibility.

69. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 69 Information Summative information is useful for reporting against targets and as general feedback to teachers. Formative information is information we can act on – it informs decision-making that can improve learning.

70. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 70 Questions to elicit information Did the more able students make significant progress, but not the lower quartile? How have the scores of individual students changed? How many remain on the same level? How much have our teaching approaches contributed to this result? How much of this shift in scores is due to students’ predictable progress? Is there any data that will enable us to compare our students with a national cohort? How does this shift compare with previous Year 9 cohorts?

71. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 71 Reading levels – terms 1 and 4

72. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 72 Words, words, words … Information can … establish, indicate, confirm, reinforce, back up, stress, highlight, state, imply, suggest, hint at, cast doubt on, refute … Does this confirm that …? What does this suggest? What are the implications of …? How confident are we about this conclusion?

73. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 73 The evidence-driven decision making cycle Trigger Clues found in data, hunches ExploreIs there really an issue? Question What do you want to know? AssembleGet all useful evidence together Analyse Process data and other evidence >Interpret What information do we have? Intervene Design and carry out action Evaluate What was the impact? Reflect What will we change?

74. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 74 Making sense of information Data becomes information when it is categorised, analysed, summarised and placed in context. Information therefore is data endowed with relevance and purpose. Information is developed into knowledge when it is used to make comparisons, assess consequences, establish connections and engage in dialogue. Knowledge … can be seen as information that comes laden with experience, judgment, intuition and values. Empson (1999) cited in Mason (2003)

75. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 75 Interrogate the information Is this the sort of result we envisaged? If not, why? How does this information compare with the results of other research or the experiences of other schools? Are there other variables that could account for this result? Should we set this information alongside other data or evidence to give us richer information? What new questions arise from this information?

76. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 76 Interrogate the information Does this relate to student achievement - or does it actually tell us something about our teaching practices? Does this information suggest that the school’s strategic goals and targets are realistic and achievable? If not, how should they change, or should we change? Does the information suggest we need to modify programmes or design different programmes? Does the information suggest changes need to be made to school systems?

77. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 77 Interrogate the information What effect is the new 6-day x 50-min period structure having on student engagement levels?

78. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 78 Interrogate the information What effect is the new 6-day x 50-min period structure having on student engagement levels? Do student views align with staff views? Do positive effects outweigh negative effects? Is there justification for reviewing the policy? Does the information imply changes need to be made to teaching practices or techniques? Does the information offer any hint about what sort of changes might work?

79. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 79 The evidence-driven decision making cycle Trigger Clues found in data, hunches ExploreIs there really an issue? Question What do you want to know? AssembleGet all useful evidence together Analyse Process data and other evidence Interpret What information do you have? >Intervene Design and carry out action Evaluate What was the impact? Reflect What will we change?

80. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 80 Professionals making decisions How do we decide what action to take as result of the information we get from the analysis? We use our professional judgment.

81. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 81 Professional decision making We have evidence-based information that we see as reliable and valid What do we do about it? If the information indicates a need for action, we use our collective experience to make a professional decision

82. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 82 Professionals making decisions Have my students not achieved a particular history standard because they have poor formal writing skills, rather than poor history knowledge? The answer was ‘Yes’... so I need to think about how to improve their writing skills. How will I do that?

83. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 83 Professionals making decisions Do any particular groups of year 11 students attend less regularly than average for the whole cohort? The analysis identified two groups – so I need to think about how to deal with irregular attendance for each group. How will I do that?

84. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 84 Professionals making decisions You asked what factors are related to poor student performance in formal writing. The analysis suggested that poor homework habits have a significant impact on student writing. You make some professional judgements and decide Students who do little homework don’t write enough You could take action to improve homework habits - but you’ve tried that before and the success rate is low You have more control over other factors – like how much time you give students to write in class So you conclude – the real need is to get students to write more often

85. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 85 Deciding on an action Information will often suggest a number of options for action. How do we decide which action to choose? We need to consider what control we have over the action the likely impact of the action the resources needed

86. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 86 Planning for action Is this a major change to policy or processes? What other changes are being proposed How soon can you make this change? How will you achieve wide buy-in? What time and resources will you need? Who will co-ordinate and monitor implementation?

87. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 87 Planning for action Is this an incremental change? Or are you just tweaking how you do things? How will you fit the change into your regular work? When can you start the intervention? Will you need extra resources? How will this change affect other things you do? How will you monitor implementation?

88. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 88 Timing is all How long should we run the intervention before we evaluate it? When is the best time of the year to start (and finish) in terms of measuring changes in student achievement? How much preparation time will we need to get maximum benefit?

89. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 89 Planning for evaluation We are carrying out this action to see what impact it has on student achievement We need to decide exactly how we’ll know how successful the intervention has been To do this we will need good baseline data

90. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 90 Planning for evaluation What evidence do we need to collect before we start? Do we need to collect evidence along the way, or just at the end? How can we be sure that any assessment at the end of the process will be comparable with assessment at the outset? How will we monitor any unintended effects? Don’t forget evidence such as timetables, student opinions, teacher observations …

91. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 91 The evidence-driven decision making cycle Trigger Clues found in data, hunches ExploreIs there really an issue? Question What do you want to know? AssembleGet all useful evidence together Analyse Process data and other evidence Interpret What information do you have? Intervene Design and carry out action >Evaluate What was the impact? Reflect What will we change?

92. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 92 Evaluate the impact of our action Did the intervention improve the situation that triggered the process? If the aim was to improve student achievement, did that happen?

93. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 93 Evaluate the impact of our action Was any change in student achievement significant? What else happened that we didn’t expect? How do our results compare with other similar studies we can find? Does the result give us the confidence to make the change permanent?

94. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 94 Evaluate the impact of our action A school created a new year 13 art programme. In the past students had been offered standard design and painting programmes, internally and externally assessed against the full range of achievement standards. Some students had to produce two folios for assessment and were unsure of where to take their art after leaving school. The new programme blended drawing, design and painting concepts and focused on electronic media. Assessment was against internally assessed standards only.

95. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 95 Evaluate the impact of our action Did students complete more assessments? Were students gain more national assessment credits? How did student perceptions of workload and satisfaction compare with teacher perceptions from the previous year? Did students leave school with clearer intentions about where to go next with their art than the previous cohort? How did teachers and parents feel about the change?

96. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 96 Evaluate the intervention How well did we design and carry out the intervention? Would we do anything differently if we did it again? Were our results affected by anything that happened during the intervention period - within or beyond our control? Did we ask the right question in the first place? How useful was our question? How adequate were our evaluation data?

97. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 97 Think about the process Did we ask the right question in the first place? How useful was our question? Did we select the right data? Could we have used other evidence? Did the intervention work well? Could we have done anything differently? Did we interpret the data-based information correctly? How adequate were our evaluation data? Did the outcome justify the effort we put into it?

98. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 98 The evidence-driven decision making cycle Trigger Clues found in data, hunches ExploreIs there really an issue? Question What do you want to know? AssembleGet all useful evidence together Analyse Process data and other evidence Interpret What information do you have? Intervene Design and carry out action Evaluate What was the impact? >Reflect What will we change?

99. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 99 Future practice What aspects of the intervention will we embed in future practice? What aspects of the intervention will have the greatest impact? What aspects of the intervention can we maintain over time? What changes can we build into the way we do things in our school? Would there be any side-effects?

100. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 100 Future directions What professional learning is needed? Who would most benefit from it? Do we have the expertise we need in-house or do we need external help? What other resources do we need? What disadvantages could there be? When will we evaluate this change again?

101. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 101 Consider the Evidence Terminology

102. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 102 Terminology Terminology used in the evidence-driven decision making cycle Trigger Clues found in data, hunches ExploreIs there really an issue? Question What do you want to know? AssembleGet all useful evidence together Analyse Process data and other evidence Interpret What information do you have? Intervene Design and carry out action Evaluate What was the impact? Reflect What will we change?

103. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 103 Trigger Data, ideas, hunches, etc that set a process in action. The trigger is whatever it is that makes you think there could be an opportunity to improve student achievement. You can routinely scan available data looking for inconsistencies, etc. It can be useful to speculate about possible causes or effects - and then explore data and other evidence to see if there are any grounds for the speculation.

104. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 104 Explore Initial data, ideas or hunches usually need some preliminary exploration to pinpoint the issue and suggest good questions to ask.

105. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 105 Question This is the key point: what question/s do you want answered. Questions can raise an issue and/or propose a possible solution.

106. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 106 Assemble Get together all the data and evidence you might need – some will already exist and some will have to be generated for the occasion.

107. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 107 Analyse Process sets of data and relate them to other evidence. You are looking for trends and results that will answer your questions (but watch out for unexpected results that might suggest a new question).

108. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 108 Interpret Think about the results of the analysis and clarify the knowledge and insights you think you have gained. Interrogate the information. It’s important to look at the information critically. Was the data valid and reliable enough to lead you to firm conclusions? Do the results really mean what they seems to mean? How sure are you about the outcome? What aspects of the information lead to possible action?

109. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 109 Intervene Design and implement a plan of action designed to change the situation you started with. Be sure that your actions are manageable and look at the resourcing needed. Consider how you’ll know what has been achieved.

110. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 110 Evaluate Using measures you decided in advance, assess how successful the intervention has been. Has the situation that triggered the process been improved? What else happened that you maybe didn’t expect?

111. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 111 Reflect Think about what has been learned and discovered – and what practices you will change as a consequence. What did we do that worked? Did this process suggest anything that we need to investigate further? What aspects of the intervention can be maintained? What support will we need?

112. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 112 Terminology Other terms used in Consider the Evidence

113. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 113 Terminology Analysis A detailed examination of data and evidence intended to answer a question or reveal something. This simplistic definition is intended to point out that data analysis is not just about crunching numbers - it’s about looking at data and other evidence in a purposeful way, applying logic, creativity and critical thinking to see if you can find answers to your questions or reveal a need. For example, you can carry out a statistical analysis of national assessment results in the various strands of English across all classes at the same level. You could compare those results with attendance patterns. But you might also think about those results in relation to more subjective evidence - such as how each teacher rates his/her strengths in teaching the various strands.

114. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 114 Terminology Aggregation A number of measures made into one. This is a common and important concept in dealing with data. A single score for a test that contains more than one question is an aggregation - two or more results have been added to get a single result. Aggregation is useful when you have too few data to create a robust measure or you want to gain an overview of a situation. But aggregation can blur distinctions that could be informative. So you will often want to disaggregate some data – to take data apart to see what you can discover from the component parts. For example, a student may do moderately well across a whole subject, but you need to disaggregate the year’s result to see where her weaknesses lie.

115. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 115 Terminology Data Known facts or measurements, probably expressed in some systematic or symbolic way (eg as numbers). Data are codified evidence. (The word is used as a plural noun in this kit.) The concepts of validity and reliability apply to data. It helps to know where particular data came from; how data were collected and maybe processed before you received them. Some data (eg attendance figures) will come from a known source that you have control of and feel you understand and can rely on. Other data (eg standardised test results) come from a source you might not really understand; they may be subject to manipulation and predetermined criteria or processes (like standards or scaling). Some data (eg personality profiles) may be presented as if they are sourced in an objective way but their reliability might be variable.

116. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 116 Terminology Demographics Data relating to characteristics of groups within the school’s population. Data that provides a profile of people at your school. You will have the usual data relating to your students (gender, ethnicity, etc) and your staff (gender, ethnicity, years of experience, etc). Some schools collect other data, such as the residential distribution of students and parental occupations.

117. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 117 Terminology Disaggregation See aggregation When you disaggregate data, you take aggregated data apart to see what you can discover from the component parts. For example, a student may do moderately well across a whole subject, but you need to disaggregate the year’s result to see where her weaknesses lie.

118. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 118 Terminology Evaluation Any process of reviewing or making a judgement about a process or situation. In this resource, evaluation is used in two different but related ways. After you have analysed data and taken action to change a situation, you will carry out an evaluation to see how successful you have been - this is summative evaluation. But you are also encouraged to evaluate at every step of the way - when you select data, when you decide on questions, when you consider the results of data analysis, when you decide what actions to take on the basis of the data - this is called formative evaluation.

119. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 119 Terminology Evidence Any facts, circumstances or perceptions that can be used as an input for an analysis or decision. For example, the way classes are compiled, how a timetable is structured, how classes are allocated to teachers, student portfolios of work, student opinions. These are not data, because they are not coded as numbers, but they can be factors in shaping teaching and learning and should be taken into account whenever you analyse data and when you decide on action that could improve student achievement.

120. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 120 Terminology Information Knowledge gained from analysing data and making meaning from evidence. Information is knowledge (or understanding) that can inform your decisions. How certain you will be about this knowledge depends on a number of factors: where your data came from, how reliable it was, how rigorous your analysis was. So the information you get from analysing data could be a conclusion, a trend, a possibility.

121. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 121 Terminology Inter-subject analysis A detailed examination of data and evidence gathered from more than one learning area. Inter subject analysis can answer questions or reveal trends about students or teaching practices that are common to more than one learning area. For example, analysing the results of students taking mathematics and physics subjects can indicate the extent to which achievements in physics are aided or impeded by the students’ mathematical skills.

122. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 122 Terminology Intervention Any action that you take to change a situation, generally following an analysis of data and evidence. This term is useful as it emphasises that to change students’ achievement, you will have to change something about the situation that lies behind achievement or non-achievement. You will take action to interrupt the status quo.

123. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 123 Terminology Intra-subject analysis A detailed examination of data and other evidence gathered from within a specific learning area. Intra subject analysis can answer questions or reveal trends about student achievement or teaching within a subject or learning area. For example, an analysis of assessment results for all students studying a particular subject in a school can reveal areas of strength and weakness in student achievement and/or in teaching practices, etc. Comparison of a school’s results in a subject with results in that subject in other schools is also intra subject analysis.

124. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 124 Terminology Longitudinal analysis A detailed examination of data and evidence to reveal trends over time. Longitudinal analysis in education is generally used to reveal patterns in student achievement, behaviour, etc over a number of years. Results can reveal the relative impact of different learning environments, for example. In this resource, it is suggested that longitudinal analysis can be applied to teaching practice and school processes. For example, the impact of modified teaching practices in a subject over a number of years can be evaluated by analysing the achievements of successive cohorts of students.