1. Dr. Kathleen Haynie Haynie Research and Evaluation November 12, 2010

2. DR K-12: Seeks to enable significant advances in student and teacher STEM learning Begin with a research question about how to improve preK-12 STEM learning and teaching Develop, implement, and study effects of innovative educational resources, models, or technologies NSF makes 53-75 awards per year Exploratory projects 3 years, <= $450,000 Full R&D projects up to 5 years, <= $3,500,000 Scale up projects up to 5 years, <= $5,000,000

3. Scale Up and Study Effectiveness Synthesize and Theorize Explore Hypothesize, and Clarify Design, Develop, and Test Implement, Study Efficacy, and Improve Cycles of Research and Development

4. Proposals must include plans for formative and summative evaluation of development and research work Evaluation questions Data to be gathered and analysis plans Expertise of evaluators Evaluation should focus on the validation of, fidelity to, and the usefulness of the development and research processes to achieve the targeted outcomes. Objectives include: (1)ensuring that the project is making satisfactory progress toward its goals; (2)recommending reasonable, evidenced-based adjustments to project plans; (3)determining the value of the outcomes of the project; and (4)attesting to the integrity of outcomes reported by the project

5. Formative Evaluation Focus Summative Evaluation Focus Proposal Materials Development Usability Piloting Implementation Piloting Project Foci Broad Strokes Evaluation Plan, Adequate Budget Logic Modeling, Evaluation Plan, Evidence of Process Instrument Design, Data Collection and Analysis, Formative Feedback, Evidence of Outcomes/Impact Evaluation Foci

6. Example of DR K-12 Evaluation Timeline

7. Need for planning and teaching resources in STEM area Expertise: *Inquiry- based curricula *Innovative design *STEM content *Educational research & evaluation Partners: *Primary PI’s organization *HRE, *Teacher- partners InputsActivities Outcomes Impacts Develop materials: *Define user needs *Establish criteria for materials *Define features *Define content *Write materials Build collaborative team (curriculum, tech, research) Successful model of collaboration across areas of expertise STEM planning and teaching resources Teacher Usability *Planning needs *Instruction value Teacher learning Research and Project Evaluation Conduct Research and Evaluation Contribute to available open educational resources Development Model of planning and teaching resources in other content areas Research knowledge and Evaluation results: Leveraging teaching resources in educational settings STEM Logic Model Process for developing teaching supports in other STEM areas Pilot Testing: *Iteratively with mats development *Involve teacher- partners *User-testing *Implementation study Implementation fidelity= OTL (students)

9.  Logic modeling process lays foundation for determining evaluative questions and methods  NSF asks that the evaluation plan “pose significant questions that can be addressed empirically and that are central to the project’s goals and objectives, as well as contributing to understanding that meets current and expected educational demands of the nation on world- class criteria.”  Evaluation plan should be vetted with project team members, and, ideally, with project Advisors

10. General Evaluative Approach Matrix for NSF Projects

11.  Process data ◦ Evidence of team collaboration (e.g., observe team meetings, team notes) ◦ Evidence of decision-making in materials design and development  What are the design specifications?  Who are the decision-makers?  What decisions are made, and associated rationales?  What opportunity costs of decisions?  Outcomes data ◦ What features, tools, content, functions, etc.? ◦ To what extent meet design specifications? ◦ Evidence from stakeholders as to usability, value, affordances, limitations, likely impact

12.  Outcomes data ◦ How valuable are different features or content sections of the materials? ◦ Are resources / materials likely to be engaging to students? ◦ Are materials clear/intuitive to teachers? ◦ How easily are instructional plans /activities performed? ◦ What is the likely added value of these resources? ◦ What refinements to materials result from user testing?  Data sources ◦ Usability studies with potential user group  Observations, interviews, focus groups, tracking of materials use (i.e., mouse-clicks on electronic materials), surveys  Solicit feedback on what is helpful, useful, feasible, user- friendly for their purposes in actual educational settings

13.  Data sources ◦ Implementation study in actual educational settings  (Classroom?) observations, teacher/administrative/ student interviews, focus groups, surveys  Solicit feedback on how well resources function within that setting  Outcomes data ◦ What subset of the materials or resources do educators use in actual settings? How much time spent on each? ◦ What educational practices are tied to use? ◦ What opportunities for student learning are afforded by the materials? How do materials contribute to school, district goals or attainment of educational standards? ◦ How do educational impacts compare with baseline or with comparison group? ◦ What refinements to materials result from piloting?

14.  From day 1, build strong and trusting relationships with all decision-makers  Determine the decision-makers  Stick with the evaluative evidence  Assessment your audience’s listening - readiness to hear and digest findings  Provide balanced feedback: what’s working, what might be improved

15.  Start well in advance of May, when most Annual reports are due  Use the evaluation plan as a guide ◦ Be sure to focus on evaluation questions and methods ◦ Can include emergent findings, but try to connect with original project goals  Make sure evaluation write-up is well-vetted with project team  Unless final evaluation report, be sure to include formative recommendations that are realistic

16.  Thank you, and go to it!!!  Dr. Kathleen Haynie Director of Haynie Research and Evaluation kchaynie@stanfordalumni.org 609-466-2990 haynieresearch.com