1. Follow up workshop on the Evaluation of AUT STEM Programme
13 March 2019
Follow up workshop on the Evaluation of AUT STEM Programme
Pauline Hoyle
Director
Cinnabar Consultancy

2. AUT STEM-TEC seminar Aims of this session:
To define the elements of the Theory of Change for the AUT STEM -TEC programme
To consider the elements needed for effective evaluation of the AUT STEM-TEC programme
Outcomes of this session:
To agree the Theory of Change for AUT STEM -TEC programme
To define the parameters of the strategy for evaluating the AUT STEM-TEC programme

3. What makes evaluation effective?
To be effective evaluation needs to
measure achievement of intended outcomes and additional impacts related to the mission
be embedded in the learning process and post-activity actions
evaluation and learning processes are merged so that learning instruments simultaneously function as evaluation tools
evaluation is integrated into planning and implementation of post-activity intervention
triangulate evidence of impact from a variety of sources
not to be heavy and not difficult to implement
be useful for participants and providers
Additional benefits:
Evaluation of interventions and its outcomes speeds up integration of successful strategies (embedding)
If you have tangible evidence it is easier to convince colleagues, leaders – more dissemination
Scaffolding professional learning

4. Theory of Change for STEM –TEC Evaluation
Evaluation is about impact on outcomes of the programme
What are the STEM-TEC most important outcomes?
Do we want to measure the impact on all outcomes of the programme – long, medium or short term?
On what five key outcomes do we want to evidence of impact ?
How do we measure impact of the programme now?
What outcomes are we currently measuring?
Where are the gaps?

5. AUT STEM-TEC programme
Mission statement:
Increasing the number and improving the quality of STEM graduates for the New Zealand knowledge economy
We aim to achieve this by:
Conducting high quality research in STEM education
Implementing best pedagogical strategies in the teaching and learning of STEM subjects
Ensuring equity and creating opportunity via social and community engagement
Are these agreed?

6. Theory of Change for STEM -TEC
Our mission
Increasing the number and improving the quality of STEM graduates for the New Zealand knowledge economy
Who we work with
What we do
What we achieve ( outcomes)
How we evaluate and measure out impact

7. Theory of Change for STEM -TEC
Who we work with:
Young people in schools, universities particularly at AUT
Teachers in schools, colleges and universities particularly at AUT
STEM academics, professionals and entrepreneurs
STEM partners in informal learning organisations, industry and employers
Missing??
Parents
Public
How well does this relate to our mission?

8. Theory of Change for STEM –TEC What we do
Essential
promoting and sponsoring local and international student competitions in STEM subjects
seminars and hands-on science and technology workshops with secondary school students
engaging the public with hands-on science and technology activities at museums
CPD for school teachers
Important
Research projects
Collecting innovative supplementary resources for engaging students and enhancing teaching and learning of STEM subjects
Useful
Community events
Exchange visits including sabbaticals
AUT students and staff sharing knowledge, making connections with influential entrepreneurs, STEM graduates, and technology leaders ( STEMpreneurs Club)
joint conferences and seminars with the effective use of the resources

9. Theory of Change for STEM –TEC What we do
Not agreed as to whether essential, important or useful – needs discussion
focus groups with students on global collaboration
publications ( including refereed journal articles, papers in refereed conference proceedings, and refereed book chapters, books, e books and reports)
presentations at national and international conferences and universities
Additional ideas
Build educational expertise within STEMTEC centre
CPD for university/tertiary lecturers  
How well does this relate to our mission?

10. Theory of Change for STEM –TEC
Outputs: something produced from an engagement
Measurements: actions to know what happens
Outcomes: impact of actions which contribute to the achievement of the mission
Evaluation strategy: the process for knowing whether your outcomes are contributing to your mission
Evaluation tools: actions to measure whether your meeting your defined outcomes

11. Theory of Change for STEM –TEC Outcomes
Long term outcomes:
??? ??
Medium term outcomes:
Short term outcomes:

12. UCL Evaluation Strategy
Mixed methodology:
analysis of young peoples’ responses pre and post engagement
the impact observed by the teachers of young people
impact noted by tutors (staff or students)
evaluation tasks embedded into engagement strands (where young people engage in activities of 30+ hours)
external evaluator observations.
Evidence of impact from:
initial reaction to events
changes in participants’ knowledge and/or practice over 3-6 months for medium impact activities or 6-9 months for high impact activities,
changes in participants’ institution (i.e. teachers practice or school provision).

13. Research behind self –reported evaluation
Guskey ( 2000) model of impact levels
Level 1 participant’s reactions (immediately)
Level 2 participants learning from engagement (immediately and after 6-8 weeks)
Level 3 organisational support and change (after 6-8 weeks and 6-9 months)
Level 4: participants’ use of new knowledge and skills (after 6-9 months)
Level 5: Student outcomes (after 6-9 months)
For CPD/engagement to be effective and impactful:
CPD/engagement outcomes are clearly stated and their achievement evaluated
post-CPD/engagement intervention and its outcomes are clearly planned and reflected in the Action Plan
evaluation process is embedded in the planning and implementation of post-CPD/engagement actions

14. Theory of Change for STEM –TEC Evaluation
Evaluation is about impact on outcomes of the programme
What are the STEM-TEC most important outcomes?
Do we want to measure the impact on all outcomes of the programme – long, medium or short term?
On what five key outcomes do we want to evidence of impact ?
How do we measure impact of the programme now?
What outcomes are we currently measuring?
Where are the gaps?

15. Theory of Change for STEM –TEC Evaluation
Agreed actions:

16. AUT STEM-TEC seminar Outcomes of this session:
To agree the Theory of Change for AUT STEM -TEC programme
To define the parameters of the strategy for evaluating the AUT STEM-TEC programme