1. Replacing “Ready, Aim, Fire” with “Research, Inform, Action”

2. Today’s Session The Kansas City STEM Alliance and the Kansas City Area Education Research Consortium collaborate with schools and nonprofits to recruit and transition youth into STEM programs across the metropolitan area. Come explore how research with local students, schools, out-of-school programs, and community volunteers informs program decisions and illustrates a return-on-investment of resources leveraged by businesses, K-12 schools, higher education, and foundations. Also participate in discussions about how what Kansas City is learning can be translated to inform efforts in your region. 2

3. Our Vision The vision of the KC STEM Alliance us is to see that a diverse, innovative and sustainable STEM workforce becomes a reality. By developing an environment that leverages the strengths of educators, STEM organizations, and local industry we can create a collaborative network to encourage and sustain STEM careers. 3 Working together will allow this…

4. KC STEM Alliance Partners 4

5. Building on Our Success: 2006-2012 More than 27 area school districts and higher education partners to provide curriculum, professional development for teachers Nationally recognized Project Lead the Way (PLTW) and US FIRST - 15,000 students in the Kansas City Metro area Since 2006 – well over 20,000 students have participated in these two programs. Significant resources – private, public sectors and school districts 5

6. Our strategy Ongoing program evaluation and data collection Increase participation of existing initiatives Seek out new partnerships and opportunities Raise awareness about STEM in Kansas City Secure support from area companies and organizations 6

7. Mission Statement Our shared goal is to improve P-20 education for all students in the Kansas City metropolitan area by providing powerful tools for data-driven educational research, evaluation and implementation. Schools and School Districts Collaborating Universities Leading Community Organizations Local Education Agencies, Foundations, Chambers and Economic Development Entities, Community and Other Colleges, State Education Departments of Kansas & Missouri.

8. Beyond Descriptive KC-AERC works with local education programs to help create measures of program effects on students Traditional quantifiable measures –Ex. Reading or Math gains pre and post tests Non-quantifiable skill measures –Ex. Socio-emotional skills, attitudes towards science

9. Methods Discuss desired program goals and effects with program leaders Research appropriate instruments for analyzing effects on students Devise appropriate statistical model –Mixed methods of quantitative and qualitative Collaborate with program leaders on final design

10. Quantitative Methods Treatment and control groups to identify if effects can be linked to program –“Gold Standard” in attempting to find causality –Many times difficult to find and engage a control group Regression analysis to factor a variety of background characteristics into results Use of Pre and Post tests to identify growth through program (T- tests, ANOVA)

11. Qualitative Methods Focus groups to gather thoughts from those closest to subject area Use quantitative methods to identify strong and weak areas of program, qualitative investigation of why some aspects work Interviews with leaders for in depth details of program implementation

12. KC STEM Alliance Initial Research Questions What are the effects of STEM programs on high school achievement? What are the effects of PLTW, FIRST Robotics and other STEM programs on post-secondary educational outcomes? Why do students participate/not participate in STEM activities? Measure in-class variability in implementation of PLTW as measured by scores on the exit exams? How much of this variability can be explained by differences in students’ academic backgrounds (other math course taken, grades earned etc.)? Design Methodology

13. 13 Lower participation rate among minorities (when compared to overall school population – FIRST) Low female participation in STEM engineering programs – but, high in biomedical programs Higher participation of females in urban districts Urban programs are less robust enrollment, participation, funding and support What did we learn in the first 6 months:

14. What are issues raised by these pilots? Process Data for Program Improvement: Comprehensive Program Implementation with School Districts Definitions of Program Participation Understanding Within-Group Variation in Number of PLTW Courses Complete Understanding FIRST Robotics Participation Systematically PLTW Course Outcome Data Outcome Data for Program Impact: High School Data Formats and Availability Data on College Outcomes

15. Logic Model for KC STEM Alliance Evaluation

16. Highlights of FIRST Robotics Surveys 2011-12 – FIRST Robotics, the majority of participants are White and Male. – Majority have lived in the United States their whole life, come from households where English is spoken, and have highly educated parents (i.e., college degree or beyond) who in turn hold high expectations for their children. – Slightly more than half of students surveyed (54%) were participating in FLL for the first time.

17. Highlights of FIRST Robotics Surveys FIRST Coaches reported: 72% participating in a FIRST competition Of those participating, 44% participated in more than one competition. The number of mentors per team ranged from 0-15, with 24% of coaches reporting their teams did not have even one mentor. Every single coach whose team had a mentor(s) reported their mentors helped in “mechanical component design;” less than half of coaches reported that their mentors helped with marketing, business plans, computer applications, or website development.

18. Highlights of FIRST Robotics Surveys FIRST Volunteers/Mentors (N = 77) reported: Majority of Event Volunteers were White and Male. Almost all volunteers lived in the Kansas City metropolitan area. Majority employed full-time and represented a variety of occupations. When asked about their satisfaction with their work in the FIRST Robotics program, they gave responses that were generally very positive; however, there were relatively less positive responses to questions about the perceived effectiveness of information provided about the volunteer job.

19. KC 2012 FIRST Robotics Mentors

20. Think with a partner: What is the difference between “EVALUATION” and “ASSESSMENT? ”

21. Typically, “EVALUATION” = program level and “ASSESSMENT” = individual level TYPES of DATA: Participant Level (interest, engagement, achievement) Program Level (quality activities, professional development) Systems Level (Policy changes)

22. How can you make this happen for your organization?

23. Research, Inform, Action Two-Way Street Evaluation Planning to Implementation to Utilization –Step 1: Program Purpose –Step 2: Data –Step 3: Plan to Obtain Data –Step 4: Communication/Share Data

24. Step 1: Document the nature and purpose of the program What is the program? What is the need? How is the program meeting the need? What are the goals of the program in terms of measurable outcomes? –Short-term –Long-term Note: Increases in knowledge, skills, and attitudes are laudable goals.

25. Research, Inform, Action Inputs Outputs Measurable Outcomes/Goals Evidence/Possible Sources of Evidence (Data)

26. How can you make this happen for your organization?

27. Step 2: Identify data needed to measure progress against goals Choose just the important data; avoid collecting data you will not use Look for data among the people you directly serve Consider: –Who, when, and how often participants attend –What participants know and can do Note: Evidence of learning is more valuable than what participants say they learned.

28. Step 3: Develop a plan to obtain the data with current resources Brainstorm with team: –What constitutes success? –What evidence can be collected? –What notes/data can frontline staff record? –What data do you need assistance with gathering and understanding? RE – visit goals to see if they capture everything that is important

29. Step 4: Share how the data will be used Provide feedback to stakeholders about how to improve what we do Information to different audiences: –Current and future funders –Schools/School districts –Parents/Students/Families –Participants

30. Research, Inform, Action Two-Way Street Evaluation Planning to Implementation to Utilization –Step 1: Program Purpose –Step 2: Data –Step 3: Plan to Obtain Data –Step 4: Communication/Share Data

31. Replacing “Ready, Aim, Fire” with “Research, Inform, Action” Dr. Leigh Anne Taylor Knight Executive Director, KC-AERC lknight@kcaerc.org 913-396-3214 Laura Loyac0no Director, KC STEM Alliance loyaconol@kcstem.org 816-665-3823