1. Can You Enhance Knowledge and Stimulate Excellence One STEM Unit at a Time? AEA – October 16, 2014 Panel: Evaluating STEM Professional Development Interventions Mary Piontek, PhD Center for Evaluation & Education Policy at Indiana University

2. Evaluation Overview Focused on the impact of the STEM program’s professional development on teacher and classroom-level outcomes: Changes in teachers’ knowledge and skills in STEM disciplines and Changes in teachers’ instructional practices within their classrooms across other disciplines.

3. Evaluation Design and Methods Evaluation data sources include Web-based self-report surveys from teachers, principals, students, and parents; Teacher online feedback forms (required documentation and reflection on implementation of curricular units); Online Q&A discussion forum (facilitated by PD trainers); and Observations of teaching of the STEM curricular units (by site support staff).

4. Program Overview Focuses on helping students “learn how to learn,” emphasizing logic, inquiry, problem-solving, mathematical thinking, and experimental scientific thinking. Enrichment STEM curriculum (14 curricular units for 4th-8th grades) implemented in 94 sites during the 2013-14 school year; curriculum not designed to be graded.

5. Program Overview Primarily an afterschool program (with recent option for in- school program design). 64 hours of STEM enrichment contact; 20-30 students per program. Co-taught by two instructors (at 6 th -8 th grade level usually a mathematics and a science instructor).

6. Program Overview Teachers receive 5.5 days of required professional development per year: orientation workshop (1-day, summer); curricular unit workshops (2-day sessions, during school year); and culminating webinars (2-3 hours, spring).

7. Changing Context New staffing = director and support staff. Director oversees multiple programs in the organization, expects concentrated focus on teacher PD outcomes. Revived interest in teacher and classroom outcomes by board members and funders; continued “murmurings” about student academic outcome data.

8. Changing Context Uncertain priorities for continued funding (operates as an outreach component of an organization directly funded in state’s annual budget). Pressure on the larger organization to demonstrate impact on STEM educational landscape.

9. Changing Context In school, embedded academic day option for sites (began in 2012-13 school year in a few sites ad hoc, expanded in 2013-14, semi-formalized in 2014-15): “elective class.” Aligned with Common Core State Standards and Next Generation Science Standards, but not intended to replace current classroom curriculum. Original impetus/communicated benefits = reduce school costs [afterschool staffing, facilities, security, and buses]; eliminate conflicts with non- academic programming [sports, clubs]. Yet to be determined benefits = expanded student access to STEM, broader site participation across state.

10. Evaluation Findings 96% of principal survey respondents agree or strongly agree that teachers in their schools have enhanced their regular classroom instruction because of the program. 81% of principal survey respondents agree or strongly agree that teachers in their schools sought out additional professional development opportunities in STEM disciplines because of the program. 70% of principal survey respondents agree or strongly agree that teachers have sought out opportunities to serve as instructional mentors in STEM disciplines to their peers because of the program.

11. Evaluation Findings Classroom teaching duties and instruction are directly influenced by teachers’ experiences as instructors in the program. Having students identify problems/issues and strategies for addressing them, work in pairs/teams to collect and analyze information, and engage in group discussions to reflect on their learning. Using real-world examples in their teaching of content.

12. Evaluation Findings I am already experienced in PBL and applying real world connections to problems for student engagement, understanding and buy-in. I give credit to [the program] for providing me with more of these types of connections and for giving me a science/math connection that I might not have considered. Generally, [the program] has a great sense of what is appropriate and a balance of what to connect for the middle school students' mindset.

13. Evaluation Findings This year I had great success with my own classroom of 30 students when I used [activity name] from the [lesson name]. My students were never so eager to start Science, work with a group of peers, and get the job done. They were literally sad when I had to end each lesson for the day, due to time constraints. They wanted to continue the lessons and gave audible groans when we had to end. Seeing their excitement and enthusiasm provoked me to want to incorporate even more lessons into my regular classroom. In addition, the students felt important and worthy, as they were receiving [name of program’s] lessons. I could not be happier!

14. Evaluation Findings Overall the implementation of the curriculum by instructors during the 2013-2014 program year demonstrated high fidelity and consistent quality. Out of 110 observations conducted by site observers, 25% (28) demonstrated moderate fidelity and 75% (82) demonstrated high fidelity.

15. Evaluation Findings Instructors are demonstrating high quality preparation, organization, and implementation of the curricular units; appropriately using facilities, space, and equipment; fostering student participation and team work; creating purposeful activities; supporting student engagement with STEM and STEM content learning; promoting inquiry and problem solving; and facilitating reflection, relevance, and making connections by students.