Asking the Right K-12 Questions How to Answer Them to Evaluate K-12 STEM Outreach and Engagement Carlos Rodriguez, Ph.D., Principal Research Scientist American Institutes for Research April 2012 NAE/AAES Convocation 1

 For STEM Education Success:  Give adequate instructional time to science as well as math especially in K-5  Enhance K-12 teachers’ capacity ◦ Deep knowledge of STEM subject matter ◦ Understanding how students’ learning develops in STEM fields, the kinds of misconceptions students may develop, and strategies for addressing students’ evolving needs  PD to instructional leaders to create school conditions that support student achievement in STEM 2

 For ST E M Education Success: ◦ In-School and Out-of-School experiences ◦ Integrated curriculum, i.e., hands-on, direct teach, labs or experiments ◦ Low student to teacher ratios ◦ PBL (problem based learning) in comprehensive open-ended projects (e.g., robotics, “canonical” engineering, i.e., projects that involve multiple engineering tasks-mechanical, electrical, civil, etc.) ◦ High emphasis on self-directed learning tasks 3

 Defined outcomes – drives interventions. Success is measured against the intended results.  Persistence enables effective interventions to take hold - includes proactive leadership, sufficient resources and support at the district and school levels.  Personalization – develops students as individuals. Individual differences, uniqueness and diversity are recognized and honored.  Challenging content anchors knowledge and skills students master. Students understand the link between content rigor and career opportunities.  Engaged adults believe in the potential of all students – they support, stimulate interest and create high expectations. Educators play multiple roles as teachers, coaches, mentors, tutors and counselors. Teachers develop and maintain quality interactions with students and each other. Active family support is sought and established. These design principles of programs that work, in informal and informal seetings, appear to comprise a package rather than an à la carte menu. 4

 Wouldn’t pre-collegiate engineering education have to be embedded in changing schools and systems fundamentally from the ground-up and focus on the need for individual deep change in teaching and learning to support and sustain system-level change?  So the questions we ask must include questions that inquire into individual changes in beliefs and practices among students and teachers. And, also recognize the importance of the actions of larger institutions and the mandates and constraints under which they operate. 5

So, units of analysis could lead to at least four areas of inquiry: 1) inquiry into K-12 engineering education as a catalyst for school renewal, 2) inquiry into factors that support scale; 3) inquiry into teacher change, 4) inquiry into causes of student achievement gains through pre-collegiate engineering education. 6

 How do programs at the local, regional, and national level increase awareness contribute to a comprehensive ecosystem for K-12 students to choose ST E M as a career option?  How should K-12 systems work with industry and government to implement or expand program implementation in underserved communities?  How should communities work together with other organizations (non-profits, industry, and government) to establish the K-12 Engineering ecosystem in underserved communities? 7

 Engineering is Elementary – EiE - Current research questions include:  What do students know about engineering and technology?  How does the EiE curriculum affect what students know about engineering and technology?  How does the EiE curriculum affect students’ understanding of related science and mathematics topics?  How does the EiE curriculum affect students’ attitudes towards STEM activities and careers? 8

 Engineering is Elementary – EiE - Current research questions include (continued):  How does the EiE curriculum impact the understanding and attitudes of female students, students of color, low-income students, and other populations underrepresented in STEM fields?  How does EiE professional development affect teachers’ pedagogy for STEM topics?  How does EiE professional development affect teachers’ attitudes towards teaching engineering and technology in their classrooms? 9

Nature of evaluation questions  Emphasis on causal questions: To what degree are causal questions central, and to whom? ◦ Types of causal knowledge: Is the focus more on the effects of known causes or the causes of known effects?  Emphasis on Probable Questions – understanding the conditions for effects  Emphasis on Conclusive Questions Knowledge production and accumulation: How important is knowledge accumulation? 10

Evaluation constraints Degree of program development: Has the program been implemented effectively, and with clear logic agreed on among stakeholders? Resource constraints: Will there be evaluation capacity, ongoing funding, and adequate time? Political constraints: Is there agreement among stakeholders on the influence or use of findings? 11

 What public policies and practices are needed to increase the number of K-12 students (both mainstream and underserved) going into ST E M careers?  What can you learn from your “interventions” to assist in the development of a national partnership which can improve the K-12 ST E M ecosystem?  Then, based on the above, what ACTIONS MUST BE TAKEN, WHEN AND BY WHOM? 12

 The acquisition of knowledge, skills, and habits of mind;  Opportunities to put these into practice;  Developing sense of competence, confidence, and progress;  Motivation to be in, a sense of belonging to, or self-identification with the field; and  Information about stages, requirements, and opportunities.  Expanding Underrepresented Minority Participation: America's Science and Technology Talent at the Crossroads 13

 Which ones do you want or need to answer? 14