1. Challenges in Evaluating a Middle School STEM Program Emphasizing Engineering and Technology
Janet K. Matulis, Ph.D.
Nancy Knapke, M.A.
Presented at the Annual Meeting of the
American Evaluation Association in Orlando, Florida
November 11-14, 2009

2. Background Mercer, Auglaize, K-8 STEM Program of Excellence (MAKE-it)
Consortium of 3 school districts, business, community, and higher education partners
Middle school initiative
Rural west-central Ohio
Personalized inquiry- and activity-based STEM curriculum
Leverages high school program of excellence, Project Lead The Way

3. Curriculum Curricular focus on engineering and technology Grades 5-6
Laser Engraver
Computer Numerical
Control Machine
Vinyl Cutter
Engineering
is Elementary
curriculum
Gateway to
Technology
curriculum
Gateway to
Technology
curriculum
5th Grade Activities:
STEM Camp
Intro to Engineering Day
7th Grade Activities:
STEM Fair
“Trink-it” Project
6th Grade Activities:
Engineering Day
Manufacturing Field Trips
8thth Grade Activities:
Advanced “Cutting Edge” Project 2

4. Teacher Professional Development
Content-based PD
Engineering is Elementary PD in Boston
Manufacturing equipment training by IHE faculty
Teacher collaboration
Intra-district collaboration
Inter-district collaboration
Teaming based on subject

5. Challenges Measuring student achievement overall
Timing and length of grant
Intervention “immersion”
Standardized test and/or short-cycle assessments
Measuring engineering & technology knowledge, skills, and interests
“Pinning down” exactly what to measure!
Identifying and developing instruments
Teachers
Students 4

6. Challenges Survey management Analyzing survey data
Developing online surveys with extensive graphics
Administering and managing student surveys across three school districts
Who best to manage – client or evaluator?
Administering and managing teacher surveys
Analyzing survey data
Roles for responsibility switched
Cleaning data challenging for large-scale student survey
How can data be used to guide instruction?

7. Course of Action Existing instruments used
Assessing teachers’ attitudes
Teachers’ Beliefs Inventory
Assessing students’ knowledge
General Engineering Knowledge Test (from Engineering is Elementary)
Modified existing instruments
Math and science foci replaced with engineering and technology foci
COURSE OF ACTION

8. Course of Action New instruments developed
Assessing students’ knowledge of, and interest in, engineering careers
Extensive collaboration by project team!
Survey administration and management
Data analysis and interpretation
Focus: using data to guide instruction!
COURSE OF ACTION 2) 7

9. Lessons Learned
Instruments with engineering & technology foci for teachers & students limited
Modification of instruments with math & science foci is possible
Look for opportunities!
Consider appropriateness across grades
May be able to cover more grades than you think!

10. Lessons Learned Data pointed to specific gaps in student knowledge
Students’ knowledge of “what an engineer does”
Guided teachers’ instruction
Real-life experience with manufacturing equipment made a difference!
Student interest soared
Students grasped idea of “what an engineer does” 9