1. “What's measured improves” Peter Drucker
Yvonne Pelham, Manager, Educational Outreach, EAD
IEEE Symposium on Pre-University Teacher Training
Hilton Tampa Airport Westshore
19 – 20 May 2012
For the next 20 minutes, I’d like to focus our attention on the Teacher In-Service Program outcomes, measures of success.

2. Topics of Discussion Outcome, measures of success, and impact
Metrics on workshops and in-service presentations
Impact assessment
TISP champion opportunities
I will briefly provide a perspective on measuring outcomes, success and impact
and will also present an analysis of the current data that we have been collecting over the years,
and begin to describe a plan for a qualitative study of the impact of the program.
I will conclude with a couple of points on how you as TISP champions can contribute in this area.
3/27/2017

3. Evaluating Outcomes, Measuring Success and Determining Impact
To evaluate outcome, define the vision/aspiration, and determine if the program activities fulfill that vision
To measure success, define the goals/objectives and measure efforts using a common set of metrics
To determine impact, define the value of the program and assess the effort using key performance indicators
Evaluating Outcomes, Measuring Success and Determining Impact.
When we talk about outcome, you must first define your vision for the program and take a look at the activities and their results to see if they support that vision.
Program success is measured by comparing a standardize set of metrics to a defined set of objective.
And third, if we want to determine impact, we ask the question what difference does it make, what is the value of the efforts in terms of impact to the target audience.
3/27/2017

4. Vision
Through a sustained program lead by IEEE volunteers in every section, IEEE will be recognized by the pre-university community as a leading provider of professional development for science, technology, engineering and mathematics teachers.
Ideally, Educational Activities would like to see all section have a sustained and active Teacher In Service program as part of their core volunteer effort
and if and when attained, IEEE will be sought after as a provider of professional development for science, technology, pre-engineering and mathematics teachers. 4

5. How to Get There: Long-Term Goals
IEEE Teacher In-Service Champions in all sections
Empower IEEE “champions” to develop collaborations with local pre-university education community
Increase the level of understanding of the needs of educators
Identify ways that engineers can assist schools and school systems (i.e. teacher training, curriculum enhancements, etc.)
Technological Literacy
Enhance the level of technological literacy of pre-university educators and students
How do we get there,
The stated long-term goals focuses on two aspects The first is to empower a critical mass of volunteers to be the TISP champions who will develop collaborations with local pre-university education community.
Through the train the trainer program, these volunteers will gain an understanding and appreciation of the needs of the pre-university teacher and through direct contact with the local schools; will determine the best way that IEEE can assist.
The other aspect is the focus on enhancing technological literacy to the degree that teachers and students are aware of and appreciate technology and its benefits to society. 5

6. Measures of Success 2,203 trained attendees (81% volunteers)
Empower IEEE “champions”
Technological literacy of pre-university educators
Technological literacy of pre-university students
Understanding of the needs of educators
School systems assisted by IEEE
2,203 trained attendees (81% volunteers)
93% agreed that program enhanced technological literacy
94% believe that student’s technological literacy would increase (anecdotal)
Sustained programs in several sections
168 presentations reported
To show evidence of the success of TISP, we took a look at each stated goal and provided evidence to show the program is meeting its goals.
Certainly the training workshops that have taking place over the years have helped to create teams of TISP champions. Around 1800 volunteers have completed the train the trainer workshop. From that we are aware of X sections who have followed up with the teacher presentations.
This is reported as 168 presentations. As Doug may have mentioned earlier, we know many more of been conducted, so we encourage all to let EA know.
There is great potential to expand this program to other sections. We are only constrained by budget and bandwidth.
From these presentations, we have tabulated results of the teacher questionnaires that the TISP volunteers have distributed and collected and produced a cumulative summary that allows us to report that 93% of teacher felt that the in-service presentation enhanced their technology literacy and that these teachers felt that the TISP hands-on activities/lesson plans would increase their students level of technological literacy.

7. Current Metrics
Workshop Evaluations to measure effectiveness of the training experience
Did the training inspire the attendees to get involved in outreach activities with teachers and schools?
Did the training increase understanding and provide resources to enable collaboration?
Teacher Questionnaires to measure usefulness of the in-service presentation
Will the teachers use the resources provided in the classroom?
Did the in-service affect level of technological literacy?
So let’s take a closer look at the current metrics that we are capturing. We collect data on the train the trainer workshops and also request feedback from teachers who have participated in an in-service presentation.
We collect data from the training workshop to measure effectiveness of the training experience.
The evaluations allow us to answer the question such as “Did the training inspire the attendees to get involved in outreach activities with teachers and schools” and “Did the training increase understanding and provide resources to enable collaboration?”
We also look at the feedback on each of the segments of the workshop to see where we make improvements in the training workshop.
As you think back to your own experience going through training, you may have recommendation that can help improve the delivery of the training. You will get an opportunity on Sunday to discuss.
We also look at Teacher Questionnaires to measure usefulness of the in-service presentation and ask questions such as “Will the teachers use the resources provided in the classroom?” and “Did the in-service affect level of technological literacy?”
3/27/2017

8. Training Workshops – Empowering Champions
When we look at the distribution of trained volunteers across the regions, you can see that there are opportunities in several areas.
In Region 9, we had several large event involving student members which is reflected in the numbers.
A realistic goal may be to determine the number of trained volunteers proportional to membership in that region.
If we want to relate this data of number of trained volunteers to our stated vision of section involvement, I would break down the data by section within the regions to see if we are creating that critical mass
3/27/2017

9. Training Workshops – Empowering Champions
If you drill into R1-6, you will see that there are also opportunities in regions 1-6.
3/27/2017

10. Effectiveness of Training Workshops
38% prior
68% after
When we look at how well the training workshop inspired the volunteers to get involved.
Across the 18 training workshop where we collected the workshop evaluations,
all results showed a positive gain in the number of attendees who agreed that they would work with teachers, students and schools.
Prior to the workshop, only 38% of the attendees responded positively.
After the experience, 68% responded positively.
The chart just showed the results from a sampling of training workshops. In the Brazil training workshop where 13 countries were represented A little more than half of the participants responded positively. This in effect spread the program broadly from a single event. (Did we get any reported presentations from this event?)

11. More stats on TISP Workshop Feedback
72% said the workshop definitely provided valuable information/resources to participate in pre-university outreach
69% said the workshop contributed to their understanding of the impact engineers can make through pre-university outreach
Additional stats collected from the 18 workshop indicate positive responses to questions about resources and understanding. There is certainly room for improvement here, so once again, your input would be invaluable to shape the training workshop

12. Teacher Feedback – Questionnaire Cumulative Results
93% of attendees who attended a TISP Presentation and completed a feedback form are teachers.
68% of educators teach secondary level and 25% teach primary level.
The subject areas most educators teach are science, technology and math.
93% felt that the program added to their technical knowledge base.
92% stated that they will use the concepts presented in their classroom instruction in the next 6 months.
94% agreed that the “hands-on” portions of the presentation were helpful in understanding the concepts discussed.
When we tabulate the data from the teacher feedback questionnaires that you send to EA , we update a cumulative reports.
Most of the respondents are teachers with a very small percentage being administrators.
Most of the teachers teach secondary level and interesting, 25% teach primary level. 93% felt that the program added to their technical knowledge base.
Over 90% stated that they will use the concepts presented in their classroom instruction in the next 6 months and agreed that the “hands-on” portions of the presentation were helpful in understanding the concepts discussed.

13. Impact on Technological Literacy
Impact on teacher 91% agreed that the presentations increased their level of technological literacy.
Perceived impact on student
94% agreed that the hands-on activities will increase their student’s level technological literacy
Very important indicator of the success of the teacher presentations conducted by the volunteer are the questions related to Technological Literacy.
91% agreed that the presentations increased their level of technological literacy.
94% agreed that the hands-on activities will increase their student’s level technological literacy
3/27/2017

14. Program Evaluation
Impact Assessment to analyze the Teacher In-Service Program
Ability to impact science, technology, engineering and mathematics education
Impact on teacher and student attitudes about engineering and technology
Impact on student achievement levels
Now that TISP is well established in several countries, we can put effort towards research study on the impact of the program.
We would want to understand the programs ability to impact science, technology, engineering and mathematics education,
and the impact on teacher and student attitudes about engineering and technology.
What we would not be measuring is the impact on student achievement levels,
as we recognized that a change in student achievement levels could not be attributed to exposure to a single activity since the IEEE hands-on activity is used as part of a larger unit of instruction.
3/27/2017

15. Key Research Questions
How does TISP affect teacher’s pedagogy for science, technology and mathematics?
How do the in-service sessions affect teachers attitude towards teaching technology subjects in their classroom?
How do IEEE hands-on activities affect what students know about engineering, technology, and related science and mathematic topics?
How do the activities affect student’s attitudes towards science, technology, engineering, mathematics activities, and careers?
How do the activities impact understanding and attitudes of female students and under-represented group in engineering and technology fields?
The Key Research Question we would ask are similar to questions
that the Boston MOS Engineering is Elementary is using in their research study as reported by the American Institute for Research at the NAE/AAES Convocation in April of this year.
These questions include:
How does TISP affect teacher’s pedagogy for science, technology and mathematics?
How do the in-service sessions affect teachers attitude towards teaching technology subjects in their classroom?
How do IEEE hands-on activities affect what students know about engineering, technology, and related science and mathematics topics?
How do the activities affect student’s attitudes towards science, technology, engineering, mathematics activities, and careers?
How do the activities impact understanding and attitudes of female students and under-represented group in engineering and technology fields?
3/27/2017

16. 2009 Pilot Teacher Assessment: Survey Questions
Number of IEEE lesson plans or activities that have been used, in full or in part, in the classroom
Reasons(s) that influenced your inability to use the lesson plans
The lesson plan(s) supports classroom instruction for the subject areas that I teach
The students expressed greater interest in science, technology, engineering or math after completing these hands-on activities (anecdotal)
A small pilot survey was conducted in 2009 to report quantitative and anecdotal data to assess effectiveness of the in-service presentations conducted.
The pilot was conducted with educators in two areas.
75 teachers who attended the Fort Worth TISP presentations. The sample include teachers who participated in either 2007 or 2008 sessions.
Also, with the help of a lead volunteer in South Africa who has championed the program in that region, distributed a localized version of the same six-item questionnaire to teachers. It was during holiday time in South Africa so unfortunately we did not receive a significant number of responses from that group.
3/27/2017

17. 2009 Pilot Teacher Assessment: Results
Of the respondents,
48% taught in the secondary school
57% were science teachers
48% used 2 to 5 lesson plans in the classroom
Of the 26% that did not use the lesson plans, 6 out of 11 stated that they ran out of time or was unable to fit into schedule
74% agreed that the lesson plans supported their classroom instruction
88% agreed that the students expressed more interest after completing the activities
The questionnaire was sent via postal mail and electronically to 75 teachers in Texas. 31% were returned over a 4 week period.
We ask teachers to indicate how many IEEE lesson plans or activities that have been used, in full or in part, in the classroom. And if none were used, indicate why.
We also asked them to respond to the questions whether the lesson plan(s) supports classroom instruction for the subject areas that I teach. We also asked teachers to respond the question of student’s interest.
48% taught in the secondary school, and 57% were science teachers.
Nearly half used 2 to 5 lesson plans in the classroom. Of the 26% that did not use the lesson plans, 6 out of 11 stated that they ran out of time or was unable to fit into schedule.
74% agreed that the lesson plans supported their classroom instruction and 88% agreed that the students expressed more interest after completing the activities.
So the take away is that when working with the schools, that the events are arranged so that scheduling will permit the application of the new techniques in the classroom within the school year.
It also says to me that follow-on relationship are needed to continue the support for the teachers following the event.
3/27/2017

18. Assessment Effort
Engage a research firm to examine TISP impact on teacher attitudes, classroom practice and knowledge
Identify section(s) to participate in the study
To do a full scale research, we would engage a research firm to examine TISP impact on teacher attitudes, classroom practice and knowledge and Identify and invite section(s) to participate in the study.
3/27/2017

19. Opportunities to contribute to the enhancement of TISP
Make recommendations on enhancements to the train-the-trainer workshops
Remind teams to report TISP activities and distribute feedback questionnaires to update global statistics
Make recommendations on follow-on activities with teachers
Participate in assessment research
As I promised in the beginning, I want to close by saying that you as TISP champions have an opportunities to contribute to the enhancement of TISP
by making recommendations on enhancements to the train-the-trainer workshops,
reminding teams to report TISP activities
and distribute feedback questionnaires to update global statistics,
making recommendations on follow-on activities with teachers and
volunteer to participate in assessment research
3/27/2017

20. Let’s have dialogue
3/27/2017