“What's measured improves” Peter Drucker Yvonne Pelham, Manager, Educational Outreach, EAD IEEE Symposium on Pre-University Teacher Training Hilton Tampa Airport Westshore 19 – 20 May 2012 For the next 20 minutes, I’d like to focus our attention on the Teacher In-Service Program outcomes, measures of success.

Topics of Discussion Outcome, measures of success, and impact Metrics on workshops and in-service presentations Impact assessment TISP champion opportunities I will briefly provide a perspective on measuring outcomes, success and impact and will also present an analysis of the current data that we have been collecting over the years, and begin to describe a plan for a qualitative study of the impact of the program. I will conclude with a couple of points on how you as TISP champions can contribute in this area. 3/27/2017

Evaluating Outcomes, Measuring Success and Determining Impact To evaluate outcome, define the vision/aspiration, and determine if the program activities fulfill that vision To measure success, define the goals/objectives and measure efforts using a common set of metrics To determine impact, define the value of the program and assess the effort using key performance indicators Evaluating Outcomes, Measuring Success and Determining Impact. When we talk about outcome, you must first define your vision for the program and take a look at the activities and their results to see if they support that vision. Program success is measured by comparing a standardize set of metrics to a defined set of objective. And third, if we want to determine impact, we ask the question what difference does it make, what is the value of the efforts in terms of impact to the target audience. 3/27/2017

Vision Through a sustained program lead by IEEE volunteers in every section, IEEE will be recognized by the pre-university community as a leading provider of professional development for science, technology, engineering and mathematics teachers. Ideally, Educational Activities would like to see all section have a sustained and active Teacher In Service program as part of their core volunteer effort and if and when attained, IEEE will be sought after as a provider of professional development for science, technology, pre-engineering and mathematics teachers. 4

How to Get There: Long-Term Goals IEEE Teacher In-Service Champions in all sections Empower IEEE “champions” to develop collaborations with local pre-university education community Increase the level of understanding of the needs of educators Identify ways that engineers can assist schools and school systems (i.e. teacher training, curriculum enhancements, etc.) Technological Literacy Enhance the level of technological literacy of pre-university educators and students How do we get there, The stated long-term goals focuses on two aspects The first is to empower a critical mass of volunteers to be the TISP champions who will develop collaborations with local pre-university education community. Through the train the trainer program, these volunteers will gain an understanding and appreciation of the needs of the pre-university teacher and through direct contact with the local schools; will determine the best way that IEEE can assist. The other aspect is the focus on enhancing technological literacy to the degree that teachers and students are aware of and appreciate technology and its benefits to society. 5

Measures of Success 2,203 trained attendees (81% volunteers) Empower IEEE “champions” Technological literacy of pre-university educators Technological literacy of pre-university students Understanding of the needs of educators School systems assisted by IEEE 2,203 trained attendees (81% volunteers) 93% agreed that program enhanced technological literacy 94% believe that student’s technological literacy would increase (anecdotal) Sustained programs in several sections 168 presentations reported To show evidence of the success of TISP, we took a look at each stated goal and provided evidence to show the program is meeting its goals. Certainly the training workshops that have taking place over the years have helped to create teams of TISP champions. Around 1800 volunteers have completed the train the trainer workshop. From that we are aware of X sections who have followed up with the teacher presentations. This is reported as 168 presentations. As Doug may have mentioned earlier, we know many more of been conducted, so we encourage all to let EA know. There is great potential to expand this program to other sections. We are only constrained by budget and bandwidth. From these presentations, we have tabulated results of the teacher questionnaires that the TISP volunteers have distributed and collected and produced a cumulative summary that allows us to report that 93% of teacher felt that the in-service presentation enhanced their technology literacy and that these teachers felt that the TISP hands-on activities/lesson plans would increase their students level of technological literacy.

Current Metrics Workshop Evaluations to measure effectiveness of the training experience Did the training inspire the attendees to get involved in outreach activities with teachers and schools? Did the training increase understanding and provide resources to enable collaboration? Teacher Questionnaires to measure usefulness of the in-service presentation Will the teachers use the resources provided in the classroom? Did the in-service affect level of technological literacy? So let’s take a closer look at the current metrics that we are capturing. We collect data on the train the trainer workshops and also request feedback from teachers who have participated in an in-service presentation. We collect data from the training workshop to measure effectiveness of the training experience. The evaluations allow us to answer the question such as “Did the training inspire the attendees to get involved in outreach activities with teachers and schools” and “Did the training increase understanding and provide resources to enable collaboration?” We also look at the feedback on each of the segments of the workshop to see where we make improvements in the training workshop. As you think back to your own experience going through training, you may have recommendation that can help improve the delivery of the training. You will get an opportunity on Sunday to discuss. We also look at Teacher Questionnaires to measure usefulness of the in-service presentation and ask questions such as “Will the teachers use the resources provided in the classroom?” and “Did the in-service affect level of technological literacy?” 3/27/2017

Training Workshops – Empowering Champions When we look at the distribution of trained volunteers across the regions, you can see that there are opportunities in several areas. In Region 9, we had several large event involving student members which is reflected in the numbers. A realistic goal may be to determine the number of trained volunteers proportional to membership in that region. If we want to relate this data of number of trained volunteers to our stated vision of section involvement, I would break down the data by section within the regions to see if we are creating that critical mass 3/27/2017

Training Workshops – Empowering Champions If you drill into R1-6, you will see that there are also opportunities in regions 1-6. 3/27/2017

Effectiveness of Training Workshops 38% prior 68% after When we look at how well the training workshop inspired the volunteers to get involved. Across the 18 training workshop where we collected the workshop evaluations, all results showed a positive gain in the number of attendees who agreed that they would work with teachers, students and schools. Prior to the workshop, only 38% of the attendees responded positively. After the experience, 68% responded positively. The chart just showed the results from a sampling of training workshops. In the Brazil training workshop where 13 countries were represented A little more than half of the participants responded positively. This in effect spread the program broadly from a single event. (Did we get any reported presentations from this event?)

More stats on TISP Workshop Feedback 72% said the workshop definitely provided valuable information/resources to participate in pre-university outreach 69% said the workshop contributed to their understanding of the impact engineers can make through pre-university outreach Additional stats collected from the 18 workshop indicate positive responses to questions about resources and understanding. There is certainly room for improvement here, so once again, your input would be invaluable to shape the training workshop

Teacher Feedback – Questionnaire Cumulative Results 93% of attendees who attended a TISP Presentation and completed a feedback form are teachers. 68% of educators teach secondary level and 25% teach primary level. The subject areas most educators teach are science, technology and math. 93% felt that the program added to their technical knowledge base. 92% stated that they will use the concepts presented in their classroom instruction in the next 6 months. 94% agreed that the “hands-on” portions of the presentation were helpful in understanding the concepts discussed. When we tabulate the data from the teacher feedback questionnaires that you send to EA , we update a cumulative reports. Most of the respondents are teachers with a very small percentage being administrators. Most of the teachers teach secondary level and interesting, 25% teach primary level. 93% felt that the program added to their technical knowledge base. Over 90% stated that they will use the concepts presented in their classroom instruction in the next 6 months and agreed that the “hands-on” portions of the presentation were helpful in understanding the concepts discussed.

Impact on Technological Literacy Impact on teacher 91% agreed that the presentations increased their level of technological literacy. Perceived impact on student 94% agreed that the hands-on activities will increase their student’s level technological literacy Very important indicator of the success of the teacher presentations conducted by the volunteer are the questions related to Technological Literacy. 91% agreed that the presentations increased their level of technological literacy. 94% agreed that the hands-on activities will increase their student’s level technological literacy 3/27/2017

Program Evaluation Impact Assessment to analyze the Teacher In-Service Program Ability to impact science, technology, engineering and mathematics education Impact on teacher and student attitudes about engineering and technology Impact on student achievement levels Now that TISP is well established in several countries, we can put effort towards research study on the impact of the program. We would want to understand the programs ability to impact science, technology, engineering and mathematics education, and the impact on teacher and student attitudes about engineering and technology. What we would not be measuring is the impact on student achievement levels, as we recognized that a change in student achievement levels could not be attributed to exposure to a single activity since the IEEE hands-on activity is used as part of a larger unit of instruction. 3/27/2017

Key Research Questions How does TISP affect teacher’s pedagogy for science, technology and mathematics? How do the in-service sessions affect teachers attitude towards teaching technology subjects in their classroom? How do IEEE hands-on activities affect what students know about engineering, technology, and related science and mathematic topics? How do the activities affect student’s attitudes towards science, technology, engineering, mathematics activities, and careers? How do the activities impact understanding and attitudes of female students and under-represented group in engineering and technology fields? The Key Research Question we would ask are similar to questions that the Boston MOS Engineering is Elementary is using in their research study as reported by the American Institute for Research at the NAE/AAES Convocation in April of this year. These questions include: How does TISP affect teacher’s pedagogy for science, technology and mathematics? How do the in-service sessions affect teachers attitude towards teaching technology subjects in their classroom? How do IEEE hands-on activities affect what students know about engineering, technology, and related science and mathematics topics? How do the activities affect student’s attitudes towards science, technology, engineering, mathematics activities, and careers? How do the activities impact understanding and attitudes of female students and under-represented group in engineering and technology fields? 3/27/2017

2009 Pilot Teacher Assessment: Survey Questions Number of IEEE lesson plans or activities that have been used, in full or in part, in the classroom Reasons(s) that influenced your inability to use the lesson plans The lesson plan(s) supports classroom instruction for the subject areas that I teach The students expressed greater interest in science, technology, engineering or math after completing these hands-on activities (anecdotal) A small pilot survey was conducted in 2009 to report quantitative and anecdotal data to assess effectiveness of the in-service presentations conducted. The pilot was conducted with educators in two areas. 75 teachers who attended the Fort Worth TISP presentations. The sample include teachers who participated in either 2007 or 2008 sessions. Also, with the help of a lead volunteer in South Africa who has championed the program in that region, distributed a localized version of the same six-item questionnaire to teachers. It was during holiday time in South Africa so unfortunately we did not receive a significant number of responses from that group. 3/27/2017

2009 Pilot Teacher Assessment: Results Of the respondents, 48% taught in the secondary school 57% were science teachers 48% used 2 to 5 lesson plans in the classroom Of the 26% that did not use the lesson plans, 6 out of 11 stated that they ran out of time or was unable to fit into schedule 74% agreed that the lesson plans supported their classroom instruction 88% agreed that the students expressed more interest after completing the activities The questionnaire was sent via postal mail and electronically to 75 teachers in Texas. 31% were returned over a 4 week period. We ask teachers to indicate how many IEEE lesson plans or activities that have been used, in full or in part, in the classroom. And if none were used, indicate why. We also asked them to respond to the questions whether the lesson plan(s) supports classroom instruction for the subject areas that I teach. We also asked teachers to respond the question of student’s interest. 48% taught in the secondary school, and 57% were science teachers. Nearly half used 2 to 5 lesson plans in the classroom. Of the 26% that did not use the lesson plans, 6 out of 11 stated that they ran out of time or was unable to fit into schedule. 74% agreed that the lesson plans supported their classroom instruction and 88% agreed that the students expressed more interest after completing the activities. So the take away is that when working with the schools, that the events are arranged so that scheduling will permit the application of the new techniques in the classroom within the school year. It also says to me that follow-on relationship are needed to continue the support for the teachers following the event. 3/27/2017

Assessment Effort Engage a research firm to examine TISP impact on teacher attitudes, classroom practice and knowledge Identify section(s) to participate in the study To do a full scale research, we would engage a research firm to examine TISP impact on teacher attitudes, classroom practice and knowledge and Identify and invite section(s) to participate in the study. 3/27/2017

Opportunities to contribute to the enhancement of TISP Make recommendations on enhancements to the train-the-trainer workshops Remind teams to report TISP activities and distribute feedback questionnaires to update global statistics Make recommendations on follow-on activities with teachers Participate in assessment research As I promised in the beginning, I want to close by saying that you as TISP champions have an opportunities to contribute to the enhancement of TISP by making recommendations on enhancements to the train-the-trainer workshops, reminding teams to report TISP activities and distribute feedback questionnaires to update global statistics, making recommendations on follow-on activities with teachers and volunteer to participate in assessment research 3/27/2017

Let’s have dialogue 3/27/2017