Follow up workshop on the Evaluation of AUT STEM Programme 13 March 2019 Follow up workshop on the Evaluation of AUT STEM Programme Pauline Hoyle Director Cinnabar Consultancy

AUT STEM-TEC seminar Aims of this session: To define the elements of the Theory of Change for the AUT STEM -TEC programme To consider the elements needed for effective evaluation of the AUT STEM-TEC programme Outcomes of this session: To agree the Theory of Change for AUT STEM -TEC programme To define the parameters of the strategy for evaluating the AUT STEM-TEC programme

What makes evaluation effective? To be effective evaluation needs to measure achievement of intended outcomes and additional impacts related to the mission be embedded in the learning process and post-activity actions evaluation and learning processes are merged so that learning instruments simultaneously function as evaluation tools evaluation is integrated into planning and implementation of post-activity intervention triangulate evidence of impact from a variety of sources not to be heavy and not difficult to implement be useful for participants and providers Additional benefits: Evaluation of interventions and its outcomes speeds up integration of successful strategies (embedding) If you have tangible evidence it is easier to convince colleagues, leaders – more dissemination Scaffolding professional learning

Theory of Change for STEM –TEC Evaluation Evaluation is about impact on outcomes of the programme What are the STEM-TEC most important outcomes? Do we want to measure the impact on all outcomes of the programme – long, medium or short term? On what five key outcomes do we want to evidence of impact ? How do we measure impact of the programme now? What outcomes are we currently measuring? Where are the gaps?

AUT STEM-TEC programme Mission statement: Increasing the number and improving the quality of STEM graduates for the New Zealand knowledge economy We aim to achieve this by: Conducting high quality research in STEM education Implementing best pedagogical strategies in the teaching and learning of STEM subjects Ensuring equity and creating opportunity via social and community engagement Are these agreed?

Theory of Change for STEM -TEC Our mission Increasing the number and improving the quality of STEM graduates for the New Zealand knowledge economy Who we work with What we do What we achieve ( outcomes) How we evaluate and measure out impact

Theory of Change for STEM -TEC Who we work with: Young people in schools, universities particularly at AUT Teachers in schools, colleges and universities particularly at AUT STEM academics, professionals and entrepreneurs STEM partners in informal learning organisations, industry and employers Missing?? Parents Public How well does this relate to our mission?

Theory of Change for STEM –TEC What we do Essential promoting and sponsoring local and international student competitions in STEM subjects seminars and hands-on science and technology workshops with secondary school students engaging the public with hands-on science and technology activities at museums CPD for school teachers Important Research projects Collecting innovative supplementary resources for engaging students and enhancing teaching and learning of STEM subjects Useful Community events Exchange visits including sabbaticals AUT students and staff sharing knowledge, making connections with influential entrepreneurs, STEM graduates, and technology leaders ( STEMpreneurs Club) joint conferences and seminars with the effective use of the resources

Theory of Change for STEM –TEC What we do Not agreed as to whether essential, important or useful – needs discussion focus groups with students on global collaboration publications ( including refereed journal articles, papers in refereed conference proceedings, and refereed book chapters, books, e books and reports) presentations at national and international conferences and universities Additional ideas Build educational expertise within STEMTEC centre CPD for university/tertiary lecturers   How well does this relate to our mission?

Theory of Change for STEM –TEC Outputs: something produced from an engagement Measurements: actions to know what happens Outcomes: impact of actions which contribute to the achievement of the mission Evaluation strategy: the process for knowing whether your outcomes are contributing to your mission Evaluation tools: actions to measure whether your meeting your defined outcomes

Theory of Change for STEM –TEC Outcomes Long term outcomes: ??? ?? Medium term outcomes: Short term outcomes:

UCL Evaluation Strategy Mixed methodology: analysis of young peoples’ responses pre and post engagement the impact observed by the teachers of young people impact noted by tutors (staff or students) evaluation tasks embedded into engagement strands (where young people engage in activities of 30+ hours) external evaluator observations. Evidence of impact from: initial reaction to events changes in participants’ knowledge and/or practice over 3-6 months for medium impact activities or 6-9 months for high impact activities, changes in participants’ institution (i.e. teachers practice or school provision).

Research behind self –reported evaluation Guskey ( 2000) model of impact levels Level 1 participant’s reactions (immediately) Level 2 participants learning from engagement (immediately and after 6-8 weeks) Level 3 organisational support and change (after 6-8 weeks and 6-9 months) Level 4: participants’ use of new knowledge and skills (after 6-9 months) Level 5: Student outcomes (after 6-9 months) For CPD/engagement to be effective and impactful: CPD/engagement outcomes are clearly stated and their achievement evaluated post-CPD/engagement intervention and its outcomes are clearly planned and reflected in the Action Plan evaluation process is embedded in the planning and implementation of post-CPD/engagement actions

Theory of Change for STEM –TEC Evaluation Evaluation is about impact on outcomes of the programme What are the STEM-TEC most important outcomes? Do we want to measure the impact on all outcomes of the programme – long, medium or short term? On what five key outcomes do we want to evidence of impact ? How do we measure impact of the programme now? What outcomes are we currently measuring? Where are the gaps?

Theory of Change for STEM –TEC Evaluation Agreed actions:

AUT STEM-TEC seminar Outcomes of this session: To agree the Theory of Change for AUT STEM -TEC programme To define the parameters of the strategy for evaluating the AUT STEM-TEC programme