Challenges in Evaluating a Middle School STEM Program Emphasizing Engineering and Technology Janet K. Matulis, Ph.D. Nancy Knapke, M.A. Presented at the Annual Meeting of the American Evaluation Association in Orlando, Florida November 11-14, 2009

Background Mercer, Auglaize, K-8 STEM Program of Excellence (MAKE-it) Consortium of 3 school districts, business, community, and higher education partners Middle school initiative Rural west-central Ohio Personalized inquiry- and activity-based STEM curriculum Leverages high school program of excellence, Project Lead The Way

Curriculum Curricular focus on engineering and technology Grades 5-6 Laser Engraver Computer Numerical Control Machine Vinyl Cutter Engineering is Elementary curriculum Gateway to Technology curriculum Gateway to Technology curriculum 5th Grade Activities: STEM Camp Intro to Engineering Day 7th Grade Activities: STEM Fair “Trink-it” Project 6th Grade Activities: Engineering Day Manufacturing Field Trips 8thth Grade Activities: Advanced “Cutting Edge” Project 2

Teacher Professional Development Content-based PD Engineering is Elementary PD in Boston Manufacturing equipment training by IHE faculty Teacher collaboration Intra-district collaboration Inter-district collaboration Teaming based on subject

Challenges Measuring student achievement overall Timing and length of grant Intervention “immersion” Standardized test and/or short-cycle assessments Measuring engineering & technology knowledge, skills, and interests “Pinning down” exactly what to measure! Identifying and developing instruments Teachers Students 4

Challenges Survey management Analyzing survey data Developing online surveys with extensive graphics Administering and managing student surveys across three school districts Who best to manage – client or evaluator? Administering and managing teacher surveys Analyzing survey data Roles for responsibility switched Cleaning data challenging for large-scale student survey How can data be used to guide instruction?

Course of Action Existing instruments used Assessing teachers’ attitudes Teachers’ Beliefs Inventory Assessing students’ knowledge General Engineering Knowledge Test (from Engineering is Elementary) Modified existing instruments Math and science foci replaced with engineering and technology foci COURSE OF ACTION

Course of Action New instruments developed Assessing students’ knowledge of, and interest in, engineering careers Extensive collaboration by project team! Survey administration and management Data analysis and interpretation Focus: using data to guide instruction! COURSE OF ACTION 2) 7

Lessons Learned Instruments with engineering & technology foci for teachers & students limited Modification of instruments with math & science foci is possible Look for opportunities! Consider appropriateness across grades May be able to cover more grades than you think!

Lessons Learned Data pointed to specific gaps in student knowledge Students’ knowledge of “what an engineer does” Guided teachers’ instruction Real-life experience with manufacturing equipment made a difference! Student interest soared Students grasped idea of “what an engineer does” 9