What is impact? Impact is seen as the positive and negative, intended or unintended long-term results of a project. Impact is a measure of change. Impact is not quick to see. Importance of measuring: Learn more from projects by assessing impact. Measure the ‘profit’ of an investment. If impact is credible, measurable and positive, it acts a positive promotion for a project – this can lead to influence of policy development.
Input Money invested in an AZSTT project. Resources required to make a project happen. Output Direct and tangible products from the project, i.e, x people trained, y schools involved. Outcomes/Impact Changes to people resulting from the project, i.e, qualifications, increase in achievement, promotion, improved personal capabilities. Activity
How is the impact of a project measured? We use certain indicators to measure impact. These are normally defined by the projects initial goals and targets. They can be qualitative or quantitative. Achievement, progress, enthusiasm of students. Before and after questionnaires measure changes in understanding of individuals Subjective views from individuals. Focus groups. Positive changes in approaches to teaching.
How can the Trust measure its Impact? 1. Papers published and cited 2. Website statistics – use of CPDUs and resources 3. Case Studies Ways of looking at impact of the Trusts work:
International Journal of Science Education. British Educational Research Journal. Educational Research. Journal of Research in Science Teaching. Curriculum Journal. Teacher Development. Innovations in Education and Teaching International.
TitleAuthorsJournalCitedYear Developing attitude to science scales for use with children of ages from five to eleven years Tony Pell & Tina Jarvis International Journal of Science Education 422001 Effect of the Challenger Experience on Elementary Children’s Attitudes to Science Tony Pell & Tina Jarvis Journal of Research in Science Teaching 152002 Developing attitude to science education scales for use with primary teachers Tony Pell & Tina Jarvis International Journal of Science Education 112003 Primary teachers' changing attitudes and cognition during a two-year science in-service programme and their effect on pupils Tony Pell & Tina Jarvis International Journal of Science Education 122004 Factors influencing elementary school children's attitudes toward science before, during, and after a visit to the UK National Space Centre Tony Pell & Tina Jarvis Journal of Research in Science Teaching 172005
What can web stats tell us about the popularity of the AZSTT website? How can impact be measured?
The AZSTT website contains information on funded projects, published papers and resources produced through the funded projects. Google analytics can be used to see interesting patterns of the use of the website. Feedback on CPDUs can also be accessed via the content management system.
Lowest numbers in August.... Best time to reach the AZSTT audience? Oct/Nov & Jan/Feb. Why? Percentage of new visits is also lowest in these months.
Popularity of resources on the AZSTT website Resources produced by AZSTT projects are now split into two categories on our new website: Curriculum Materials CPDUs
CPDU Feedback Talking Science “I found this CPD session very relevant and interesting. It really reinforces the importance of using dialogue and other principles of AfL in science lessons.” Fostering Curiosity In Early Years Science “It was Fantastic.” Science Clubs “Very useful. Am thinking of using the resources for girl guides. Girls need encouragement to gain confidence in science.” Top rated CPDUs are: Talking Science - 5 Fostering Curiosity In Early Years Science - 5 STAY - 5 Thinking Beyond the Classroom “I enjoyed this and will use some of the activities in my own area, Horticulture.”
Case studies allow us to go into greater detail to find out more about the impact of a project. We can learn from this to make future projects more sustainable.
Improving Science Together Bath Spa, 2000-2002. 20 Primary schools and 4 Secondary schools in Bristol and South Gloucestershire. Focus on Assessment; developing lesson plans and accompanying assessment sheets. Huge focus on subject leadership. Produced two major resources for dissemination.
Resources from the Project 1. A set of online materials on the AZSTT website to support primary teachers in planning for focussed assessment of scientific enquiry. 2. Online CPDU for primary teachers on the AZSTT website on the use of ‘floorbooks’ as a formative assessment strategy in science.
Strategy for looking at the Impact of the project Impact on practice in science enquiry assessment in the project schools. Impact on the subsequent careers and practice of the project teachers involved. Impact on practice of teachers outside the IST project. Data from AZSTT website on wider dissemination.
Objectives One: Tracing and interviewing of teachers, past and current. Two: Use of IST resources with new teachers. Three: AZSTT archive and website data.
Objective One Tracing and interviewing of IST teachers, past and current. Headteachers : 5 – Still Headteacher 1 – Advisor for workforce and governance 1 – Priest 3 – Retired 10 – Unsure SSLs: 3 – Still at same school 4 - @ different schools in the area 1 - @ Graduate School of Education UoB 3 – Unknown whereabouts but contactable 8 – Unsure Key teachers: 4 - @different schools in the area 5 - @ same school 1 – Unknown whereabouts but contactable
There are a number of ways of looking at the impact just within the tracing of past and current teachers: Teachers who have remained at the same school in the same position – did the project have longevity? Teachers who have stayed at the same school and been promoted – how did the project impact on professional development? Teachers who have moved to another school in the area which was also an IST project school – did they find the resources produced from the project were being used differently? Teachers who have moved to another school not associated with the project – could what was learnt be implemented into a new school? Teachers who were not involved in the project, but are at the project schools – was there a legacy of the project left behind?
Cath Foote Avon Primary Original IST Key Teacher still at the same school Abbie Cowland Bankleaze Primary New science coordinator at original project school Sam Nunn Bowsland Green New science coordinator, currently trailing new science assessment strategies. Erica Cheary Callicroft Junior Was key teacher at the time of the project, now SSL at same school. Liz Doorbar Deputy Head at Westbury-on-Trym. Was key teacher at Charborough Road Primary school, now deputy head at new school. Eric Rydon Charborough Road Primary New Science Coordinator at original project school. Nicola Bailey Headteacher at Charborough Road Primary. Still Headteacher at the same school. Emma Gundy St Michaels Primary New science coordinator. Kirsteen Craig Sea Mills Junior Current Headteacher Amy Easterbrook Infant teacher at Highdown Was subject leader at Wheatfield Primary during project. Dr Angela Greenwood Still head at Little Stoke Primary, (but leaving end of year). Heather Allen Little Stoke Primary (still there) Jo Davey Shield Road Primary (still there)
Objective Two Using IST resources with new teachers Teachers external to the project who have been IST resources: Floorbooks: Emma Rigarlsford Primary Science Leading Teacher – AZSTT project Kay Coverdale Member of College and has been using Floorbooks. Linda JamesHas been using Floorbooks Anna Peart Has been using Floorbooks Rachel Brooks Recently introduced to them on Science Subject Leaders course, now using them in schools Focussed assessment sheets: Sarah Doneghan Member of College and has been using the sheets across her cluster. Helen GraingerUses both Floorbooks and Focussed assessment sheets
Sarah on Planning Sheets “I absolutely love them.” “Great way of developing progression and development of skills.” “As County Durham AST, I recommend them to all schools and in a recent Ofsted science subject inspection a school using them got outstanding in all areas.”
Rachel on Floorbooks “They encourage learners to contribute their ideas and we all know science is a continual assessment from initial thoughts to testing and challenging ideas, Floorbooks allow me to do this.” “Swiftly provides evidence. They are easy and efficient.” “Interactive and engaging.” “So easily shared with parents and carers.”
Why have Floorbooks been such a popular resource? Over 12 years the basic idea hasn't changed, or needed to change. Integrity of the idea never lost even if context changes. Not a tip or a trick, very open and straightforward approach which doesn’t need lots of time to set up. Very valuable approach which works from early years to early secondary. Easy to see usefulness in AfL and evidence for APP. Has value and validity. Natural part of Assessment.
Further Impact Primary Science Quality Mark submissions – what can we learn about science in primary schools from the PSQM? Teachers mentioned they were using AZSTT assessment frameworks.
Objective Three AZSTT archive and website data. Again through the use of Google Analytics, the popularity of the resources produced by the Improving Science Together project can be looked at.
Floorbooks CPDU Floorbooks page views per month are smaller than that of the Sc1 Scientific Enquiry section of the website. However, a much longer time is spent on these pages. Floorbooks are not so popular across the world, mainly only in Australia, Netherlands, USA, UAE, Guernsey and New Zealand. But have a huge audience in the UK.
Why such a wide impact? Dissemination strategy worked very well. Routes of dissemination: ASE South West Region Meeting Dissemination Day to Primary Science subject Leaders from other South Gloucestershire and Bristol LEA schools ASE Annual Meeting via the AZSST website Through the work of university tutors and academic publications
Future of Dissemination The role of key people. How can we make sure that dissemination occurs through other routes? The COLLEGE!
Final thoughts.... For project leaders just finishing up... For project leaders just starting... For the College... Collaboration is key, this conference is a prime example of that.