1. Noyce Evaluation University of Minnesota April 20, 2006 Jim Appleton Marjorie Bullitt Bequette Frances Lawrenz Ann Ooms Deena Wassenberg Technical assistance: David Ernst

2. Overall goals for our project To contribute to the knowledge base about effective strategies for attracting and retaining high quality STEM teachers To collaboratively develop a plan to evaluate the Noyce Program that will document overall program accomplishments while celebrating the uniqueness of each project To conduct the evaluation and disseminate findings in a utility-oriented fashion

3. Our responsibilities We are: Collecting and categorizing evaluation plans and instruments Conducting a comprehensive review of the STEM recruitment and retention literature Working with ORC MACRO to make effective use of their data We will: Work with all the projects to design a program evaluation through virtual and face-to-face meetings Conduct the evaluation Disseminate the results in a user-friendly fashion

4. We need you to be effective We need your help to: Refine our literature data base Optimize the effectiveness of the evaluation variables and instrument data bases Plan and conduct the program level evaluation

5. Plan for this two session introductory conference Showcase our materials and explain how we think they might be useful Obtain feedback on how to improve Discuss what might be useful in an overall evaluation of the Noyce Program Determine the most effective use of the evaluation time at the PI conference

6. Outline of today’s presentation A tour of the Web site Demo of the literature data base Summary of the literature findings Logic Model Evaluation variables and instruments Setting the stage for tomorrow’s discussion

7. Q & A Questions and Answers about the Introduction

8. A tour of the Web site

19. Q&A Questions and Answers about the Web site overall

27. Q&A Questions and Answers about using the Literature Data Base

28. Ongoing review of the R&R literature Looking for factors that affect recruitment and retention Chose empirical articles from our database that (based on abstracts) had significant results on factors affecting recruitment and retention Starting with larger N, quantitative work; integration of other studies will follow We summarized recent articles and used RAND (2004) summaries of older work Factors were grouped into larger categories Our more detailed summary will be posted

29. Literature examined so far Adams, 1996 Arnold, Choy, & Bobbitt, 1993 Baker, 1988 Ballou, 1996 Ballou & Podgursky, 1997 Brewer, 1996 Bempah, Kaylen, Osburn, & Birkenholz, 1994 Boe, Bobbitt, Cook, Whitener, & Weber, 1997 Bond, 2001 Carroll, Reichardt, Guarino, & Mejia, 2000 Darling-Hammond, Chung, & Frelow, 2002 Eberhard, Reinhardt- Mondragon, & Stottlemyer, 2000 Galchus, 1994 Hall, Pearson, & Carroll, 1992 Hansen Lien, Cavalluzzo, & Wenger, 2004 Hanushek, Kain, & Rivkin, 2001 Hanushek & Pace, 1995 Henke, Geis, Giambattista, & Knepper, 1996 Henke, Zahn, & Carroll, 2001 Hounshell & Griffin, 1989 Ingersoll, 2001 Ingersoll, 2003 Ingersoll & Kralik, 2004 Jacobson, 1988 Kirby, Berends, Naftel, 1999 Kirby & Grissmer, 1993 Loeb (2000) Lankford, Loeb, Wyckoff, 2002 Marso & Pigge, 1997 Miech & Elder, 1996 Mont & Rees, 1996 Murnane, Singer, Willett, Kemple, & Olsen, 1991 Odell & Ferraro, 1992 Pigge, 1985 Plecki, Elfers, Loeb, Zahir, & Knapp, 2005 Rickman & Parker, 1990 Seyfarth & Bost, 1986 Shen, 1997 Shen (Autumn, 1997) Shen, 1998 Shen, 1999 Shin, 1994 Shin, 1995 Shugart and Hounsell, 1995 Stinebrickner, 2001a Stinebrickner, 2001b Stinebickner 2002 Stockard & Lehman, 2004 Theobald, 1990 Tran, Young, Mathison, & Hahn, 2000 Villar & Strong, 2005 Weiss, 1999 Young, Place, Rinehart, Jury, & Baits, 1997

30. What research has shown to affect retention Characteristics of teachers: Race/ethnicity Gender Experience Age Type of training program Area taught Academic ability/achievement Family and fertility choices Reasons for choosing to teach Certainty of intention to teach

31. What research has shown to affect retention Teacher preparation program characteristics Most of the large N quantitative work that we’ve examined focuses on the type of program (alternative, master’s 5 th year, major in education or in a discipline), not program components. Some studies examine the effects of course requirements generally. Both program type AND program components matter, though.

32. What research has shown to affect retention Mentoring and induction programs Again, details on what helps are underexamined. Salary Pay affects retention, interacting with gender, race/ethnicity, other local salaries and conditions, subject taught, potential for advancement (and salaries for those positions), and salary scale/highest salary.

33. What research has shown to affect retention School and district setting “School culture” Race/ethnicity of students; also distribution of race/ethnicity Student ability Student SES; also distribution of SES School size Number of classes taught Classes taught in area of specialization Spending (amount and patterns) Incidence of crime/violence

34. Putting this all together: Tracking teacher characteristics (affective as well as demographic), program and mentoring experience (and the connections between those two), salary, district conditions, and more, can help each project improve and can help all projects learn from each other.

35. Q&A Questions and Answers about what the literature has shown

37. Theoretical Framework: LOGIC MODEL DESCRIPTION Our proposed Noyce Logic Model contains our efforts to delineate several perspectives: The Noyce Program Ideal Depicted by the main path as well as bold headings preceded by addition signs (e.g., “+Plan to teach”) Decision points en route to becoming a STEM teacher Indicated by diverging routes from the main path describing alternative options and the Noyce Ideal in bold headings Dashed boxes denote retention/recruitment by school or program Important STEM major decision factors Influenced by attributes of the candidate, pre-service program, and school/district (depicted as bulleted lists on the main path) Depicted as thought bubbles emerging from the decision point

38. Theoretical Framework: LOGIC MODEL DESCRIPTION The Noyce Program Ideal: Diverse and smart STEM majors will be enticed by scholarships and stipends to enter pre-service programs Programs will provide adequate and relevant training These STEM majors will graduate, begin teaching in their field and at high need schools, and fulfill the obligations of their scholarship/stipend. These new teachers will continue to teach at high-need schools beyond the obligation period?

39. Theoretical Framework: LOGIC MODEL DESCRIPTION Decision Points En Route to Becoming a STEM Teacher: STEM majors may: Plan to teach or plan for a non-teaching STEM career If planning to teach, either enter a certification program or teach without certification If entering a program, upon graduation decide to teach or to not teach If choosing to teach, decide if it will be at a low- or high-need school If at a high-need school, decide whether to remain over time.

40. Theoretical Framework: LOGIC MODEL DESCRIPTION Important STEM Major Decision Factors along These Paths: Interests, career values, career pay and importance of monetary compensation, importance of certification, challenge of financial costs, desire and requirement to teach What value in workplace, social justice beliefs, program/funding requirements, training, fulfillment of job, perception of support and appropriate level of challenge

41. Theoretical Framework: LOGIC MODEL GRAPHIC

43. Q&A Questions and Answers about the logic model

44. Project Evaluation Resources

48. Project evaluation variables, methods and instruments Collected from your evaluation plans – Thank You! Categorize these based on the kind of information collected and how it was collected Categorize and present any specific evaluation instruments you provided (most now available on our Web site)

49. What you are doing: Of those responding (41 of 65, 63%), 92.7% of Noyce programs submitted a detailed evaluation plan to us. We’ve categorized these into: Evaluation of the program itself Evaluation of post-program activity Evaluation methods

50. What you are doing: VARIABLES

54. What you are doing: THE PROGRAM ITSELF Program Recruitment (61.0%) Noyce Student Performance in Program (61.0%) Demographics (58.5%) Program Retention (24.4%)

55. What you are doing: POST- PROGRAM MONITORING Noyce Teacher Effectiveness (63.4%) Monitoring of Noyce Teachers (61.0%) School/District Retention (41.5%) Transition Experiences/Support for Teachers (39.0%) Coordination Between Programs or Institutions (31.7%) Fulfillment of Scholarship Requirements (29.3%) School/District Recruitment (22.0%) School/District Characteristics (14.6%) Teaching Assignment Characteristics (2.4%)

56. What you are doing: WAYS OF GATHERING DATA Interpret carefully: These were only used if the documents, observations, self-report data could not be better classified elsewhere. Formative Program Effectiveness Data (63.4%) Summative Program Effectiveness Data (61.0%) Self-Report Data (61.0%) Specific Analyses or Methodologies (56.1%) Research Questions/Evaluation Goals (34.1%) Observations (19.5%) Document Analysis (4.9%)

57. What you are doing: SUMMARY Program itself: many are gathering data on recruitment, demographics, and student performance; fewer on program retention Post-program monitoring: several to many are gathering data on teacher effectiveness, monitoring/fulfillment of scholarship requirements, school/district retention, teacher transition experiences/support, and inter-program/institution coordination; fewer on school/district recruitment and characteristics (including teaching assignment characteristics) Ways of gathering data: mostly formative and summative program effectiveness, and self-report (caution)

58. What you are doing: INSTRUMENT DESCRIPTIONS

59. What you are doing: INSTRUMENTS

60. Q&A Questions and Answers about the project evaluation resources

61. Involvement oriented evaluation: We need you! Evaluations should be designed to document the context and the full range of effectiveness Involvement-oriented evaluations provide informed objectivity relationship to site goals and context exemplary designs motivation to provide data more use of the evaluation

62. What we know already You are doing a great deal of evaluation This work is varied and addresses many different stages of the progress of an aspiring teacher Through coordinating and sharing this work, evaluations (individual and overall program) can be improved

63. Planning for Friday Discussion: Resources What are the valuable components of the existing resources? How could the existing resources be improved? What additional resources could be provided?

64. Planning for Friday Discussion: Evaluation What might be important questions for the evaluation to address? What should we emphasize in the evaluation? From whom should data be gathered? What sort of data gathering methods would be most appropriate? What data/report dissemination strategies would be most useful?

65. Planning for Friday Discussion: PI conference What overall Program evaluation issues should be addressed at the PI meeting in June? What presentation format should be used? What outcomes should we expect? What “deliverables” should we hope to obtain?

66. Planning for Friday Discussion: Participation Think about the questions (these are now posted on our Web site) Feel free to email before the session begins (lawrenz@umn.edu) if you like, to send a question to be shared or to arrange a time to appear onscreenlawrenz@umn.edu During the discussion you can e-mail through chat to have your comments read by presenter You can also e-mail at any time through chat to be scheduled to join the discussion verbally (and in person if you have a camera) Please feel free to raise any other questions or issues that you feel are important

67. Q & A Questions and Answers about Involvement Oriented Evaluation or Friday’s Presentation

68. See you tomorrow! 3:30-5:00 Eastern 2:30-4:00 Central 1:30-3:00 Mountain 12:30-2:00 Pacific