Presentation on theme: "Strategies for Incorporating Evaluation into Teacher Scholarship Programs: A Case Study of Math for America Los Angeles Teacher Fellows Program Pam Mason."— Presentation transcript:
Strategies for Incorporating Evaluation into Teacher Scholarship Programs: A Case Study of Math for America Los Angeles Teacher Fellows Program Pam Mason Executive Director, MƒA LA Rebecca Eddy President, Cobblestone Applied Research & Evaluation, Inc.
A Partnership of Mathematicians, Business People and Educators
The mission of Math for America is to improve mathematics and science education in US public secondary schools by building a corps of outstanding STEM teachers and leaders. Our goal is to support those outstanding STEM teachers already in the classroom and to increase the number of mathematically talented individuals entering teaching.
In 2008, The University of Southern California (USC), Claremont Graduate University (CGU) and Harvey Mudd College (HMC) teamed together with MƒA to create Math for America Los Angeles (MƒA LA).
To improve math education in our public schools by attracting, training, and retaining highly qualified secondary school mathematics teachers.
To provide the support teachers require to answer a critical need and bring about a real, sustained change.
Monthly Professional Development Curriculum and Teaching Support National Board Certification Ongoing classroom observations and instructional support by experienced advisors Social and network building opportunities
Clusters of Fellows at individual schools create effective learning environments for students Hawkins High School
Montclair High School Miguel Contreras Learning Complex
Alain Leroy Locke Senior High School Green Dot Public School James Monroe High School
Helen Bernstein High School Fairfax High School
Los Altos High School East Los Angeles Performing Arts Academy
Ramon C. Cortines High School
Professional Development Outreach Program Designed by Pam Mason, MƒA LA Executive Director Darryl Yong, Associate Chair of Mathematics at Harvey Mudd College
MƒA LA Fellows have attended regional and national conferences for mathematics teachers and have given formal presentations.
Conferences attended during the school year: CMC South Conference in Palm Springs CMC North Conference in Asilomar NCSM Conference in Denver NCTM Conference in Denver Photo by Solares Photography Reprinted with permission from the National Council of Teachers of Mathematics
A tightly networked community Regular meetings Experienced coaches Social events
ACCORDING TO MƒA LA COACHES “… a feeling of camaraderie …”
Purpose of Evaluation Setting goals Identifying indicators to measure program implementation and outcomes Keeping the program on track Drawing conclusions about the merit of the program
Evaluation designed to answer key evaluation questions Uses a combination of qualitative and quantitative methods Quasi-experimental design to compare students of MfA LA Fellows vs. students on non-Fellows in the same schools
Evaluation Question To what extent is the implementation of MfA LA program activities occurring as planned? How well have Fellows’ students performed on mathematics achievement outcomes such as CST scores compared to other students? Do students of Fellows show any additional benefits from being in their classrooms such as course-taking patterns or attitudes about mathematics in comparison to students of non-Fellows? Are there differences between the Fellows’ experiences for those at cluster sites versus individual sites? Is there evidence that the MfA LA program improves teacher recruitment, quality, and retention in the teaching profession? What are the strengths of the MfA LA program? What are areas of improvement for MfA LA?
Purpose Track program implementation and measure outcomes Math for America Los Angeles is committed to a comprehensive evaluation of its programs and operations Design Mixed method: Quantitative and qualitative data sources to answer evaluation questions Matched control group quasi- experimental design for student achievement data Evaluation Activities Focus groups, interviews, classroom observations, document analysis, assessment of student test scores & surveys
What is a logic model? Common elements in a logic model: Inputs Activities Outputs Outcomes “Logic models help us plan, implement, evaluate, and communicate more effectively.” Taylor-Powell, E., & Henert, E. (2008)
University of Wisconsin – Extension: templates, examples W. K. Kellogg Foundation: Development Guide foundation-logic-model-development-guide.aspx
Situation Get PillsTake PillsFeel Better INPUTS OUTPUTSOUTCOMES
Inputs What is invested in MfA LA Inputs What is invested in MfA LA Activities What MfA LA does Activities What MfA LA does Participation Who MfA LA reaches Participation Who MfA LA reaches Short-term Med-term Long-term Situation Students under- achieve in math; U.S. needs STEM workforce to be globally competitive Situation Students under- achieve in math; U.S. needs STEM workforce to be globally competitive Outputs Outcomes Assumptions Highly qualified math teachers produce high achieving students Recruiting, training & supporting highly-qualified individuals will produce highly qualified teachers who stay in the profession Assumptions Highly qualified math teachers produce high achieving students Recruiting, training & supporting highly-qualified individuals will produce highly qualified teachers who stay in the profession External Factors Students in high poverty schools in LA less likely to receive instruction from highly qualified teachers Students in high poverty schools in LA less likely to perform well in mathematics External Factors Students in high poverty schools in LA less likely to receive instruction from highly qualified teachers Students in high poverty schools in LA less likely to perform well in mathematics Fellows receive stipends + tuition support Fellows receive coaching from mentors Fellows receive prof. support through PD Staff Exec. Director Coaches Financial Support (Grants, Donors) Time Cred. Program Staff Exec. Director Coaches Financial Support (Grants, Donors) Time Cred. Program Fellows receive training Students receive math instruction from trained Fellow Schools receive trained teachers/ leaders Fellows receive moral/ emotional/ instructional support from other Fellows Fellows teach in classrooms/ work in math depts Districts receive trained teachers Fellows have improved pedagogical skills Fellows maintain content knowledge in math Fellows feel supported & respected as a teacher Fellows are more confident to lead in classroom & in department Fellows continue to teach in schools Students acquire strong math skills Fellows retained in math profession LA schools/ districts have highly qualified math teachers LA schools/ districts have high achieving math students Students from LA competitive in global economy Students maintain achievement & interest in math/ other STEM fields National Board Certification & continued professional growth Teachers at Fellows’ school collaborate with Fellows MfA LA Teacher Fellows Program Logic Model
NeedInputs Outcomes Outputs Activities Participation Product Long-termShort-term What needs are we addressing? Teachers lack support and professional development experiences that prevents recruitment & retention of high- quality mathematics secondary school teachers
NeedInputs Outcomes Outputs Activities Participation Product Long-termShort-term What resources do we need for the program? Director/ staff expertise, funding
NeedInputs Outcomes Outputs Activities Participation Product Long-termShort-term What activities are planned to address the needs? Who will participate? Fellows attend monthly PD sessions 10 months per year & attend professional conferences; receive coaching support; stipends Monitoring Implementation
NeedInputs Outcomes Outputs Activities Participation Product Long-termShort-term What are the short- term and long-term changes we expect to see in participants? Fellows acquire content knowledge, classroom management & planning skills (short-term) Fellows are retained in the profession & are effective secondary math teachers (long-term) Determining Program Merit
See the following list and determine in which category each applies For example: Bake cookie dough Is this a need, activity, outcome, etc.?
1. Start with a need 2. Identify one activity (Outputs) Specify how much of that activity needs to happen to address the need (Activities) Establish who will be reached by the activity and what is required of their participation (Participation) 3. What resources are required? (Inputs) 4. What is your goal for meeting the needs? (Outcomes)
What are some common needs that you identified? In thinking about designing an evaluation, what are some indicators that we will track? Outputs Outcomes How can we design an evaluation to effectively address the key evaluation questions and include the indicators for implementation and outcomes?
Evaluation QuestionAnswer To what extent is the implementation of MfA LA program activities occurring as planned? Implementation: Main activities were implemented as planned (e.g., PD, coaching); Slight adjustments to coaching and professional development to meet the needs of more experienced Fellows How well have Fellows’ students performed on mathematics achievement outcomes such as CST scores compared to other students? CST: Fellows’ students performed similarly to a matched group of students of non-Fellows across all subject areas CAHSEE: Fellows’ students performed similarly to students of non-Fellows; 11 th grade students of Fellows were more likely to pass than students of non-Fellows Do students of Fellows show any additional benefits from being in their classrooms such as course- taking patterns or attitudes about mathematics in comparison to students of non-Fellows? Course-Taking Patterns: Students of Fellows (57%) were more likely to advance to another math class in the next school year than students of non-Fellows (51%) for both years (2010/11 & 2011/12) Student Attitudes: At pretest, students of Fellows and non-Fellows had similar attitudes towards mathematics; fall survey to be administered and analyzed to determine if there are differences at posttest Are there differences between the Fellows’ experiences for those at cluster sites versus individual sites? Fellows at “cluster sites” reported more support, lesson collaboration, and similar philosophies about teaching than Fellows at “individual sites” Is there evidence that the MfA LA program improves teacher recruitment, quality, and retention in the teaching profession? Recruitment: Some Fellows reported that MfA LA provided additional motivation to enter the teaching profession Retention: 97% of Fellows have been retained; 91% of Fellows plan to stay in teaching field after MfA LA Teacher Quality: Fellows have strong academic backgrounds (i.e., Praxis, college major) and build from this basis as indicated by coaches comments & classroom observations What are the strengths of the MfA LA program? Program Strengths: Professional and Community Support; Quality of Program Activities (e.g., PD, workshops); Financial Support What are areas of improvement for MfA LA? Professional Development: Lesson planning; Small-group discussions Coach-Fellow Mentoring Model: Critical and constructive feedback from coaches; Clarification of coaches’ roles; Training for coaches
2009 Cohort “I am very thankful for all that MƒA LA has been done to make me a better teacher.”
2009 Cohort “I am grateful for such a wonderful chance to inspire young minds...”
2009 Cohort “ I always consider it an honor to say that I am a fellow in such a wonderful organization
Renewing and expanding our commitment to improve student achievement and mathematics education in the greater Los Angeles community with each new cohort of fellows.