1. Interaction with and Reflection on the Noyce Program Evaluation Data Frances Lawrenz Pey-Yan Liou Christina Madsen Christopher Desjardins Allison Kirchhoff University of Minnesota

2. Noyce Program Evaluation Group Our evaluation project has four major components: –Preparation of an extensive literature review pertaining to recruitment and retention –Thematic synthesis through content analysis of project information –Statistical analyses to produce quantitative models of the program development –Execution of an overall program evaluation plan through collaboration (participatory approach) with existing projects.

4. Methodology: Data sources Surveys –PI online survey (N=66) –Scholar online survey (N=555) –Disciplinary faculty online survey (N=80) Interviews –Scholars in progress (N=12) –Districts in progress (N= 17) Monitoring data collected for NSF Self-response bias

5. Methodology: Analyses Frequencies, cross tabs, means Open-ended responses categorization Factor and cluster analyses Regression and HLM analyses using 3 outcome variables: –Factor score: Commitment to teaching in high needs school (Influence of Noyce) –Would you have become a teacher if you hadn’t received the Noyce scholarship? –Would you have decided to teach in a high need school if you hadn’t participated in the Noyce scholarship program?

6. PI Survey

7. Components of the PI survey Project overview Noyce money Program characteristics and organization Contacts and partners

8. Project overview 75 active Noyce projects with 88% of the projects responding –Of the 9 non-responding PIs, 7 were in their first or second year and subsequently had no data to report Projects in 29 states responded, with 12 responding from Texas Total of 141 strands reported with an average 1 or 2 teacher education strands per responding PI –8 strands was the maximum –Strand breakdown: 49 Undergraduate program leading to a bachelor’s degree (34.8%) 17 Teaching credential (no degree) (12.1%) 27 Post-bac or graduate program (no master’s awarded) (19.1%) 27 Graduate program (19.1%) 21 Other (14.9%)

9. Noyce money 55% indicated that Noyce scholarship paid for over 75% of scholars’ tuition (n = 66) (PI Sec II Q4) 60% indicated that their teaching preparation program would continue mostly unchanged without the Noyce funding (n = 65) (PI Sec II Q7) –In contrast, 62% reported that the funding made their teacher certification program different (n = 65) (PI Sec II Q8) In general, PIs indicated that Noyce funding: –Greatly increased their ability to recruit a variety of students –Had a lesser effect on perceptions about teaching careers –Had a lesser effect on relationships with community, districts, STEM faculty, and industry. (PI Sec II Q9)

10. Funded activities Miscellaneous Meetings Mentoring PI Sec II Q 1, number of respondents for each item ranged from 65 to 56.

11. Program characteristics Required activities (PI Sec III Q2) –Most: Teaching methods classes specific to candidate’s subject area; student teaching experience –Least: Mentoring during the candidate’s second year of teaching Required field experience (PI Sec III Q3) –Most: Supervised actual classroom teaching in high needs schools (mode # of hrs > 121). –Least: Education field experience working outside of schools with young people like those who attend high needs schools in your area (mode # of hrs 1-40) Collaborations with school districts (PI Sec III Q5) –Most: School districts providing practicum sites for scholars –Least: The school districts agreeing to hire all scholar who successfully complete the program.

12. Screening criteria for selecting Noyce scholars Criterian% responding yes GPA6598.5 Personal statement 6497.0 Letters of recommendation 6090.9 Undergraduate status 6681.8 Structured interviews 6373.0 Bachelor’s degree 6572.3 Previous experience 6468.8 Standardized tests 6336.5 STEM courses beyond required for bachelor’s6336.5 Subject matter test6330.2 Attitude tests or measures6317.7 Observation of a teaching event6414.1 PI Sec III Q 3

13. Scholar Survey

14. Components of the scholar survey Project overview Program characteristics and organization Teaching environment and experience The decision to become a teacher Background and experience Overall experience

15. Project overview Of the 555 scholars responding to the survey: –46% were teaching full-time/part-time –31% were still in their certification program but not yet a full-time teacher –13% were still in their program but also teaching full-time –8% completed a program but never taught –1% left their program without completing certification –1% taught after being certified and were working in education but not as teachers –1% taught after being certified but were no longing working in education

16. Program characteristics Features of teacher certification program (Scholar Sec II Q1) –Most: Classes in teaching methods specific to their subject area. –Least: Guaranteed job at a participating school district. Developing specific strategies for teaching students from diverse racial and ethnic backgrounds was the most extensively explored opportunity across the programs (Scholar Sec II Q4)

17. Teaching environment 83% of the 320 responding scholars are currently teaching in a high needs school (Scholar Sec III Q1) 68% of the 324 responding scholars are in schools where over 50% of students receive free or reduced lunch (Scholar Sec III Q3) 41% of the 322 responding scholars are in schools where there has been 15% teacher attrition over the last 3 years (Scholar Sec III Q3) 78% of the 318 responding scholars have held leadership positions within the last 3 years (Scholar Sec III Q5)

18. Decision to become a teacher 53% of scholars decided to become a STEM teacher during adulthood (23 or older) (n = 540) (Scholar Sec IV Q1) 87% of scholars learned about the Noyce scholarship after they decided to become a teacher (n = 543) (Scholar Sec IV Q4) 61% of scholars learned about the Noyce scholarship from their advisor (n = 586) (Scholar Sec IV Q5)

19. Factors influencing STEM teacher decision Scholar Sec IV Q3, respondents may have multiple selections (respondents vary from 516 to 535)

20. Effect of Noyce scholarship on teaching Would you have become a teacher if you had not received the Noyce Scholarship? Would you have taught in a high needs school if you hadn’t participated in the Noyce program? Yes Possibly No I will not teach in a high needs school Scholar Sec IV Q6 (n = 543) & Q7 (n = 542)

21. Faculty Survey

22. Faculty survey results 91% were familiar with the Noyce program (n = 79) (Fac Q1) 44% interact with STEM education faculty daily and 25% interact weekly (n = 79) (Fac Q2) –Half of those interacting daily were in mathematics Faculty encouraged students to pursue teaching because of the students’ interest or because of the students’ personalities (Fac Q6a) Most suggested response from faculty to students interested in teaching was that they should get actual experience teaching and that teaching is a ‘noble’ profession (Fac Q6b) 57% indicated that availability of scholarships affects advice they give to STEM majors (n = 69) (Fac Q6c)

23. Ability of STEM majors who intend to teach compared to those that do not Fac Q4 (n = 80)

24. Perceived effect of Noyce on STEM teacher certification program Fac Q1 (n = 71)

25. Group Work 1

26. Group work 1: Data analysis Work in small groups or individually for 10 minutes to develop a question that you would like us to explore At the end of the 10 minutes hand in your question and we’ll select a suggestion(s) to analyze We’ll discuss these results and implications after our next group work activity

27. PI Suggestions 1.Career/non-career changers 2.Timing of teaching decision 3.Unique program characteristics

28. Career/non career changers Overview Demographics High needs teaching

29. Overview Career changers –½ of all responding scholars indicated they were ‘career changers’ (n=532) –Of the scholars indicating being career changers: All scholars at seven institutions indicated they were ‘career changers’ 25 scholars at one institution indicated they were ‘career changers’ –Of the scholars indicating not being career changers: All scholars at four institutions indicated they were ‘non- career changers’ 20 scholars at one institution indicated they were ‘non-career changers’

30. Demographics Of the 267 ‘career changing’ scholars: –Program type 67% are/were in graduate programs (masters) 14% are/were in teaching credential programs (no degree) –Teaching status 47% are/were teaching full-time 5% completed a teacher program, but never taught Of the 265 ‘non career changing’ scholars: –Program type 37% are/were in undergraduate programs 9% are/were in teaching credential programs (no degree) –Teaching status 43% are/were teaching full-time 10% completed a teacher program, but never taught

31. High needs teaching Teaching high needs school Teaching another type of school Career changers Non-career changers Scholar Sec III Q1 & Sec V Q4, career changers (n=267), non-career changers (n=265)

32. Timing of teaching decision Scholars were asked, “Did you first learn about the Noyce scholarship before or after you decided to become a teacher?” Program type Recruitment strategies

33. Program type Scholar Sec I Q1 & Sec IV Q 4, Before (n=74), After (n=469)

34. Recruitment strategies Scholar Sec IV Q4 & Q5, Before (n=74), After (n=469)

35. Unique program characteristics Experiences in teacher certification program Required field experiences

36. Experiences Scholars responded to 14 experiences in their teacher certification program (Scholar Sec II Q1) –Of the 14 experiences, 2 were unique: Guaranteed job at a participating school district –27% of scholars indicated they were guaranteed a job (n=544) –Of the scholars indicating yes, 38% were from graduate programs (masters) whereas 9% were from teaching credential programs (n=144) Mentoring provided by certification program during the second year –32% of scholars indicated they received second year mentoring by their program (n=524) –Of the scholars indicating yes, 38% were from graduate programs (masters) whereas 16% were from teaching credential programs (n=165)

37. Required field experiences Scholars responded to 5 required field experiences in their teacher certification program (Scholar Sec II Q 2) –Of the 5 field experiences, 2 were unique: Education field experience working outside of schools (e.g. summer camp) with young people like those who attend high needs schools in the area –11% of scholars indicated they received education field experience outside of schools (n=541) –39% of the scholars were in undergraduate program leading to a bachelors degree whereas 12% were from teaching credential programs (n=57) Research field experience in your subject area –27% of scholars indicated they received research field experience (n=539) –44% of scholars were in graduate programs (masters) whereas 11% were in teaching credential programs (n=143)

38. Group Work 2

39. Group Work 2: Discussion Spend a few minutes thinking about the discussion questions included in your packet. Then in a small group, you’ll discuss either the first or second question. We’ll then reconvene and discuss as a large group ideas that were raised during the small group discussion.

40. Inferential Statistics

41. Factor analysis Factor analysis was utilized to discover underlying factors and as a psychometric procedure for the development and refinement of the questionnaire One main goal is to combine many items into a construct, and use this construct to do more analyses

42. Factor analysis results of Noyce scholar survey Constructed FactorCronbach’s Alpha Number of Items Commitment to teaching in a high needs school (influence of Noyce) 0.8936 Preparation for high needs school 0.73013 Path to teaching 0.7227 District/school high needs environment 0.7165 Personal beliefs towards teaching 0.6118 School teaching environment 0.7754 Mentoring experience 0.7246

43. Commitment to teaching in high needs school (influence of Noyce) Become a teacher (Scholar Sec IV Q8a) Complete the certification program (Scholar Sec IV Q8b) Take a teaching job (Scholar Sec IV Q8c) Teach in a high needs school (Scholar Sec IV Q8d) Remain teaching in a high needs school for the full term of your commitment (Scholar Sec IV Q8e) Remain teaching in a high needs school beyond the full term of your commitment (Scholar Sec IV Q8f)

44. 1. teach in a high needs school (Scholar Sec IV Q8d) 2. remain teaching in a high needs school for the full term of your commitment (Scholar Sec IV Q8e) 3. remain teaching in a high needs school beyond the full term of your commitment (Scholar Sec IV Q8f) 4. become a teacher (Scholar Sec IV Q8a) 5. take a teaching job (Scholar Sec IV Q8c) 6. complete the certification program (Scholar Sec IV Q8b) Hierarchical cluster analysis of commitment to teaching in a high needs school 123456123456

45. Influence of Noyce vs. timing Commitment to teaching in a high needs school (Influence of Noyce scholarship) Did you first learn about the Noyce scholarship before or after you decided to become a teacher? Commitment to teaching in a high needs school (Influence of Noyce scholarship) Spearman’s rho N Did you first learn about the Noyce scholarship before or after you decided to become a teacher? Spearman’s rho N -.250** 530 **. Correlation is significant at the 0.01 level (2-tailed). 1: before 2: after (Scholar Sec IV Q4)

46. AfterBefore Commitment to high needs teaching Low High Spearman’s rho = -0.250 Influence of Noyce vs. timing Did you first learn about the Noyce scholarship before or after you decided to become a teacher? * Not the actual relationship, just a visual representation (Scholar Sec IV Q4)

47. Influence of funding on timing of teaching decision Would you have become a teacher if you had not received the Noyce scholarship? (2, N=542) = 83.296, p<0.001 (Scholar Sec IVQ4) (Scholar Sec IVQ6)

48. Hierarchical Linear Models (HLM) Also known as multi-level analysis, is a more advanced form of simple linear regression and multiple linear regression Why is HLM used in analyzing Noyce data? The program-level variables help to explain between-program variance rather than within- program variance Scholars in the same program tend to be more alike than in other projects

49. Three outcome variables in HLM Factor score: Commitment to teaching in a high needs school (influence of Noyce) Would you have become a teacher if you hadn’t received the Noyce scholarship? (Scholar Sec IV Q6) Would you have decided to teach in a high needs school if you hadn’t participated in the Noyce scholarship program? (Scholar Sec IV Q7)

50. Group Work 3

51. Group Work 3: Searching for HLM PI predictors Brief overview of predictor analyses Spend a few minutes reading through the PI level variables handout Work in small groups or individually for 10 minutes to discuss the PI level variables that you think they would influence the outcome variable “Commitment to teaching in a high needs school (influence of Noyce)”

52. 1: Commitment to teaching in a high needs school What proportion of the total tuition scholars need to pay is provided by the Noyce scholarship funding? (0.10, p=0.013) (PI Sec II Q4) Your ability to recruit participants from ethnicities under-represented in STEM into your certification/licensure program (0.10, p=0.047) (PI Sec II Q9f) Your relationships with businesses or industry (0.13, p=0.049) (PI Sec II Q9l) Interactions with children from different cultures (-0.33, p=0.01) (PI Sec III Q2b) Teaching methods class(es) specific to the candidate’s subject area (0.36, p=0.032) (PI Sec III Q2d) Student teaching experience (-0.29, p=0.015) (PI Sec III Q2g)

53. 1: Commitment to teaching in a high needs school What proportion of the total tuition scholars need to pay is provided by the Noyce scholarship funding? (0.10, p=0.013)  (0.12, p=0.001) (PI Sec II Q4) Your ability to recruit participants from ethnicities under- represented in STEM into your certification/licensure program (0.10, p=0.047)  (0.11, p=0.001) (PI Sec II Q9f) Your relationships with businesses or industry (0.13, p=0.049) (PI Sec II Q9I) Interactions with children from different cultures (-0.33, p=0.01) (PI Sec III Q2b) Teaching methods class(es) specific to the candidate’s subject area (0.36, p=0.032) (PI Sec III Q2d) Student teaching experience (-0.29, p=0.015) (PI Sec III Q2g)

54. We would like to thank NSF and all those involved with the Noyce project!! Thank you for your participation and valuable ideas!