1. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 1 Consider the Evidence A resource to assist schools to review their use of data and other evidence 2 What is meant by ‘data and other evidence’?

2. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 2 Evidence Any facts, circumstances or perceptions that can be used as an input for an analysis or decision how classes are compiled, how classes are allocated to teachers, test results, teachers’ observations, attendance data, portfolios of work, student opinions … Data are one form of evidence

3. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 3 Data Known facts or measurements, probably expressed in some systematic or symbolic way (e.g. as numbers) assessment results, gender, attendance, ethnicity … Data are one form of evidence

4. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 4 What evidence does a school have? Demographics Student achievement Perceptions School processes Other practice

5. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 5 Demographics Data that provides a profile of our school School - decile, roll size, urban/rural, single sex or co- educational, teaching spaces … Students - ethnicity, gender, age, year level, attendance, lateness, suspension and other disciplinary data, previous school, part-time employment … Staff - gender, age, years of experience, qualifications, teaching areas, involvement in national curriculum and assessment, turnover rate … Parents/caregivers and community - socio-economic factors, breadth of school catchment, occupations …

6. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 6 Student achievement Evidence about student achievement National assessment results - NCEA, NZ Scholarship - details like credits above and below year levels, breadth of subjects entered … Standardised assessment results administered internally - PAT, asTTle … Other in-school assessments - most non-standardised but some, especially within departments, will be consistent across classes - includes data from previous schools, primary/intermediate Student work - work completion rates, internal assessment completion patterns, exercise books, notes, drafts of material - these can provide useful supplementary evidence

7. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 7 Perceptions Evidence about what students, staff, parents and the community think about the school Self appraisal - student perceptions of their own abilities, potential, achievements, attitudes … Formal and informal observations made by teachers - peer interactions, behaviour, attitudes, engagement, student-teacher relationships, learning styles, classroom dynamics … Structured interactions - records from student interviews, parent interviews, staff conferences on students … Externally generated reports - from ERO and NZQA (these contain data but also perceptions) … Student voice - student surveys, student council submissions … Other informal sources – views about the school environment, staff and student morale, Board perceptions, conversations among teachers …

8. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 8 School processes Evidence about how our school is organised and operates School processes - evidence and data about how your school is organised and operates, including: Timetable –structure, period length, placement of breaks, subjects offered, student choices, tertiary and workforce factors, etc Classes - how they are compiled, their characteristics, effect of timetable choices, etc Resources - access to libraries, text books, ICT, special equipment, etc Finance - how the school budget is allocated, how funds are used within departments, expenditure on professional development Staffing - policies and procedures for employing staff, allocating responsibility, special roles, workload, subjects and classes

9. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 9 Other practice How we can find out about what has worked in other schools? Documented research – university and other publications, Ministry of Education’s Best Evidence Syntheses, NZCER, NZARE, overseas equivalents … Experiences of other schools – informal contacts, local clusters, advisory services, TKI LeadSpace …

10. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 10 Evidence-driven strategic planning. INDICATORS FROM DATA asTTle scores show a high proportion of Yr 9 achieving below curriculum level NCEA results show high non- achievement in transactional writing Poor results in other language NCEA standards etc STRATEGIC GOAL To raise the levels of writing across the school Strategic action Develop a writing development plan which addresses writing across subjects and levels, including targets, professional development and other resourcing needs etc ANNUAL PLAN Develop and implement a plan to raise levels of Writing at Year 9 Development plan to be based on an analysis of all available data and to include a range of shared strategies etc YEAR TARGET Raise writing asTTle results Yr 9 boys from 3B to 3A etc Appraisal P D Self review School charter EVALUATION DATA asTTle writing results improve by … Perception data from Yr 9 staff indicates … Evaluation of effectiveness of range of shared strategies, barriers and enablers … etc

11. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 11 The evidence-driven decision making cycle TriggerClues found in data, hunches ExploreIs there really an issue? QuestionWhat do you want to know? AssembleGet all useful evidence together AnalyseProcess data and other evidence InterpretWhat information do you have? InterveneDesign and carry out action EvaluateWhat was the impact? ReflectWhat will we change?

12. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 12 The evidence-driven decision making cycle EXPLORE Check data and evidence to explore the issue QUESTION Clarify the issue and ask a question ASSEMBLE Decide what data and evidence might be useful ANALYSE data and evidence TRIGGER Data indicate a possible issue that could impact on student achievement EVALUATE the impact on the intervention INTERVENE Plan action to improve student achievement INTERPRET Insights that answer your question SPECULATE A teacher has a hunch about a problem or a possible action ACT Carry out the intervention REFLECT on what has been learned, how practice will change

13. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 13 The evidence-driven decision making cycle Explore data Survey of students shows that this is only partially true Question What are the characteristics of students who are poor at writing ? Assemble more data & other evidence: asTTle reading, homework, extracurric, Attendance, etc Analyse NQF/NCEA results by standard Trigger Some of our students are poor at writing Analyse non NQF/NCEA data and evidence Intervene Create multiple opportunities for writing; include topics that can use sport as context; connect speaking and writing. PD for staff. Interpret information Poor writers likely to play sport, speak well, read less, do little HW A teacher has a hunch - poor writers might spend little time on homework Evaluate Has writing improved? Reflect How will we teach writing in the future?

14. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 14 Types of analysis We can compare achievement data by subject or across subjects for an individual student groups of students whole cohorts The type of analysis we use depends on the question we want to answer

15. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 15 Inter-subject analysis Have my students not achieved a particular history standard because they have poor formal writing skills, rather than poor history knowledge?

16. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 16 Intra-subject analysis What are the areas of strength and weakness in my own teaching of this class?

17. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 17 Longitudinal analysis Are we producing better results over time in year 11 biology?

18. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 18 The evidence-driven decision making cycle >TriggerClues found in data, hunches ExploreIs there really an issue? QuestionWhat do you want to know? AssembleGet all useful evidence together AnalyseProcess data and other evidence InterpretWhat information do you have? InterveneDesign and carry out action EvaluateWhat was the impact? ReflectWhat will we change?

19. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 19 Asking questions Evidence-driven decision making starts with asking good questions You can tell whether a man is clever by his answers. You can tell whether he is wise by his questions. Nobel Prize winner, Naguib Mahfouz

20. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 20 Trigger questions How good/poor is …? What aspects of … are good/poor? Is … actually changing? How is … changing? Is … better than last year? How can … be improved? Why is … good/poor? What targets are reasonable for …? What factors influence the situation for …? What would happen if we …?

21. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 21 Questions from hunches I suspect this poor performance is being caused by … Is this true? We reckon results will improve if we put more effort into... Is this likely? I think we’d get better results from this module if we added … Is there any evidence to support this idea?

22. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 22 The evidence-driven decision making cycle TriggerClues found in data, hunches >ExploreIs there really an issue? QuestionWhat do you want to know? AssembleGet all useful evidence together AnalyseProcess data and other evidence InterpretWhat information do you have? InterveneDesign and carry out action EvaluateWhat was the impact? ReflectWhat will we change?

23. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 23 Question – Explore – Question It looks like our students are doing well in A but not in B. What can we do about it? EXPLORE … what else should we be asking? Is this actually the case? Is there anything in the data to suggest what we could do about it?

24. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 24 Question – Explore – Question We have been running 60-minute periods for a year now. Did the change achieve the desired effects? EXPLORE … what else should we be asking? How has the change impacted on student achievement? Has the change has had other effects? Is there more truancy? Is more time being spent in class on assignments, rather than as homework?

25. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 25 The evidence-driven decision making cycle TriggerClues found in data, hunches ExploreIs there really an issue? >QuestionWhat do you want to know? AssembleGet all useful evidence together AnalyseProcess data and other evidence InterpretWhat information do you have? InterveneDesign and carry out action EvaluateWhat was the impact? ReflectWhat will we change?

26. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 26 A very good question Specific and with a clear purpose Able to be investigated through looking at data and other evidence Likely to lead to information on which we can act

27. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 27 Questions with purpose What do we know about reported bullying incidents for year 10 students? MAY BE BETTER AS Who has been bullying whom? Where? What are students telling us? What does pastoral care data tell us? Were some interventions more effective with some groups of students than others?

28. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 28 The evidence-driven decision making cycle TriggerClues found in data, hunches ExploreIs there really an issue? QuestionWhat do you want to know? >AssembleGet all useful evidence together AnalyseProcess data and other evidence InterpretWhat information do you have? InterveneDesign and carry out action EvaluateWhat was the impact? ReflectWhat will we change?

29. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 29 Assembling the evidence We want to know if our senior students are doing better in one area of NCEA biology than another. So … we need NCEA results for our cohort. It could be that all biology students do better in this area than others. So … we also need data about national differences across the two areas.

30. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 30 Think critically about data Was the assessment that created this data assessing exactly what we are looking for? Was the assessment set at an appropriate level for this group of students? Was the assessment properly administered? Are we comparing data for matched groups?

31. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 31 The evidence-driven decision making cycle TriggerClues found in data, hunches ExploreIs there really an issue? QuestionWhat do you want to know? AssembleGet all useful evidence together >AnalyseProcess data and other evidence InterpretWhat information do you have? InterveneDesign and carry out action EvaluateWhat was the impact? ReflectWhat will we change?

32. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 32 Analysing data and other evidence Schools need some staff members who are responsible for leading data analysis Schools have access to electronic tools to process data into graphs and tables All teachers do data analysis Data is not an end in itself - it’s one of the many stages along the way to evidence-driven decision making

33. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 33 Making sense of the results Think about significance and confidence How significant are any apparent trends? How much confidence can we have in the information?

34. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 34 The evidence-driven decision making cycle TriggerClues found in data, hunches ExploreIs there really an issue? QuestionWhat do you want to know? AssembleGet all useful evidence together AnalyseProcess data and other evidence >Interpret What information do we have? InterveneDesign and carry out action EvaluateWhat was the impact? ReflectWhat will we change?

35. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 35 Information Knowledge gained from analysing data and making meaning from evidence. Information is knowledge (or understanding) that can inform your decisions. How certain you will be about this knowledge depends on a number of factors: where your data came from, how reliable it was, how rigorous your analysis was. So the information you get from analysing data could be a conclusion, a trend, a possibility.

36. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 36 Words, words, words … Information can … establish, indicate, confirm, reinforce, back up, stress, highlight, state, imply, suggest, hint at, cast doubt on, refute … Does this confirm that …? What does this suggest? What are the implications of …? How confident are we about this conclusion?

37. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 37 Interrogate the information Is this the sort of result we envisaged? If not, why? How does this information compare with the results of other research or the experiences of other schools? Are there other variables that could account for this result? Should we set this information alongside other data or evidence to give us richer information? What new questions arise from this information?

38. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 38 The evidence-driven decision making cycle TriggerClues found in data, hunches ExploreIs there really an issue? QuestionWhat do you want to know? AssembleGet all useful evidence together AnalyseProcess data and other evidence InterpretWhat information do you have? >InterveneDesign and carry out action EvaluateWhat was the impact? Reflect What will we change?

39. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 39 Professional decision making We have evidence-based information that we see as reliable and valid What do we do about it? If the information indicates a need for action, we use our collective experience to make a professional decision

40. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 40 Deciding on an action Information will often suggest a number of options for action. How do we decide which action to choose? We need to consider what control we have over the action the likely impact of the action the resources needed

41. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 41 Planning for evaluation What evidence do we need to collect before we start? Do we need to collect evidence along the way, or just at the end? How can we be sure that any assessment at the end of the process will be comparable with assessment at the outset? How will we monitor any unintended effects? Don’t forget evidence such as timetables, student opinions, teacher observations …

42. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 42 The evidence-driven decision making cycle TriggerClues found in data, hunches ExploreIs there really an issue? Question What do you want to know? AssembleGet all useful evidence together AnalyseProcess data and other evidence Interpret What information do you have? Intervene Design and carry out action >EvaluateWhat was the impact? ReflectWhat will we change?

43. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 43 Evaluate the impact of our action Did the intervention improve the situation that triggered the process? If the aim was to improve student achievement, did that happen?

44. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 44 Evaluate the impact of our action Was any change in student achievement significant? What else happened that we didn’t expect? How do our results compare with other similar studies we can find? Does the result give us the confidence to make the change permanent?

45. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 45 The evidence-driven decision making cycle TriggerClues found in data, hunches ExploreIs there really an issue? QuestionWhat do you want to know? AssembleGet all useful evidence together AnalyseProcess data and other evidence InterpretWhat information do you have? InterveneDesign and carry out action EvaluateWhat was the impact? >ReflectWhat will we change?

46. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 46 Future practice What aspects of the intervention will we embed in future practice? What aspects of the intervention will have the greatest impact? What aspects of the intervention can we maintain over time? What changes can we build into the way we do things in our school? Would there be any side-effects?

47. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 47 Future directions What professional learning is needed? Who would most benefit from it? Do we have the expertise we need in-house or do we need external help? What other resources do we need? What disadvantages could there be? When will we evaluate this change again?

48. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 48 What now? How can we apply this model in our school?