1. Education Policy Initiative at Carolina UNC Teacher Quality Research Initiative: Collaborating for Program Accountability and Improvement September 29, 2015 Kevin C. Bastian Director, Teacher Quality Research Initiative Education Policy Initiative at Carolina UNC Chapel Hill

2. Education Policy Initiative at CarolinaThe University of North Carolina at Chapel Hill 2 Outline Background on the UNC Teacher Quality Research Initiative In-depth discussion on key research projects Individual-level data sharing and TPP improvement

3. Education Policy Initiative at CarolinaThe University of North Carolina at Chapel Hill 3 UNC System Priorities UNC System Overall Goal: Preparing more, higher quality teachers and school leaders for North Carolina Public Schools Key Strategies to Address the Goal Recruitment Preparation Beginning Teacher Support Teacher Quality Research – Focused on outcomes/impacts

4. Education Policy Initiative at CarolinaThe University of North Carolina at Chapel Hill 4 What is the Teacher Quality Research Initiative? Originated in 2008 Partnership between the University of North Carolina General Administration, the Education Policy Initiative at Carolina (EPIC), and the 15 UNC System institutions that prepare teachers and school leaders Main goal: To conduct high-quality research and disseminate research evidence to improve the quality of teachers and school leaders prepared in North Carolina Integral component of the UNC system’s mission to prepare more and better teachers and school leaders for the public schools of North Carolina

5. Education Policy Initiative at CarolinaThe University of North Carolina at Chapel Hill 5 Motivations for the Teacher Quality Research Initiative North Carolina as an accountability state Stalled progress in student performance and concerns about educational inequities Widespread attention on evidence-based reform Compelling evidence on the importance of teachers and teaching

6. Education Policy Initiative at CarolinaThe University of North Carolina at Chapel Hill 6 Structure of the Teacher Quality Research Initiative Annual scope of research work between the UNC-GA and EPIC UNC Council of Education Deans Meetings—opportunities for dialogue, feedback and reflection Production of policy briefs, reports, and journal articles Benefits of working within a university system

7. Education Policy Initiative at CarolinaThe University of North Carolina at Chapel Hill 7 Teacher Quality Research Initiative Studies Teacher portals analyses Program effectiveness analyses Recent Graduate Survey Job placement rates North Carolina Teaching Fellows evaluation Returns to teacher experience North Carolina New Teacher Support Program evaluation Graduate degrees in NC public schools edTPA validity and reliability studies NCTQ predictive validity study Personality traits and beginning teacher outcomes UNC MSA and Principal Fellows evaluations

8. Education Policy Initiative at CarolinaThe University of North Carolina at Chapel Hill 8 Teacher Portals Analyses Static characteristic capturing teachers’ preparation prior to entering teaching Track demographic and compositional data on the NC teacher workforce Assess the performance of teachers entering the profession through different routes Value-added models: Statewide and within-school comparisons Teacher evaluation ratings Defined Portals UNC undergraduate, graduate degree, and licensure only NC private undergraduate, graduate degree, and licensure only Out of state undergraduate, graduate degree, and licensure only Teach For America Visiting International Faculty Alternative Entry Unclassifiable

9. Education Policy Initiative at CarolinaThe University of North Carolina at Chapel Hill 9 Teacher Portals Distribution

10. Education Policy Initiative at CarolinaThe University of North Carolina at Chapel Hill 10 Summary of Portals Value-Added Results Teacher Portal Significantly More Effective than UNC UG Significantly Less Effective than UNC UG No Different than UNC UG UNC Graduate Degree 1 (HS Math) 09 NC Private Undergraduate Degree 0 3 (ES Science, MS Math, MS Science) 8 NC Private Graduate Degree 1 (HS Science) 07 Out-of-State Undergraduate Degree 0 4 (ES Math, ES Science, HS Math, and HS Social Studies) 7 Out-of-State Graduate Degree 1 (HS English) 010 UNC Licensure Only 1 (HS Science) 08 Out-of-State Licensure Only 0 1 (ES Reading) 2 Teach For America 9 (ES Math, ES Science, MS Math, MS Reading, MS Science, MS Algebra, HS Math, HS Science, HS Social Studies) 02 Visiting International Faculty 2 (ES Math and ES Reading) 1 (HS Math) 6 Alternative Entry 0 3 (MS Math, HS Math, and HS Social Studies) 8 Unclassifiable 0 2 (MS Science and HS Math) 8

11. Education Policy Initiative at CarolinaThe University of North Carolina at Chapel Hill 11 Summary of Portals Evaluation Rating Results Teacher Portal Leadership Classroom Environment Content Knowledge Facilitating Student Learning Reflecting on Teaching UNC Graduate Degree 0.048 * 0.059 * 0.085 * 0.044 * 0.058 * NC Private Undergraduate Degree 0.0130.011-0.0030.009 NC Private Graduate Degree 0.057 * 0.066 * 0.071 * 0.0340.043 Out-of-State Undergraduate Degree -0.005-0.008-0.004-0.010-0.004 Out-of-State Graduate Degree 0.0100.0160.011 * 0.008-0.007 UNC Licensure Only -0.0170.0180.045 * -0.008-0.014 Out-of-State Licensure Only -0.058-0.0760.038-0.0020.034 Teach For America 0.105 * 0.073 * 0.066 * 0.067 * 0.066 * Visiting International Faculty -0.043 * 0.037 0.014-0.025 Alternative Entry -0.051 * -0.032 * -0.017 * -0.041 * -0.045 * Unclassifiable -0.079 * -0.055 * -0.035 * -0.059 * -0.067 *

12. Education Policy Initiative at CarolinaThe University of North Carolina at Chapel Hill 12 Beginning Teacher Retention by Portal

13. Education Policy Initiative at CarolinaThe University of North Carolina at Chapel Hill 13 UNC Program Effectiveness Analyses Track employment outcomes and school characteristics Assess the performance of initially-prepared teachers from each UNC system institution Value-added models: Statewide, within-school comparisons, student subgroups, selection vs. preparation Teacher evaluation ratings Overall analyses compare graduates of UNC system institutions to a common reference group of all non-UNC system initially-prepared teachers Program-specific analyses compare graduates of a specific institution to teachers entering through other routes of preparation

14. Education Policy Initiative at CarolinaThe University of North Carolina at Chapel Hill 14 Job Placement Rates by UNC System Institution

15. Education Policy Initiative at CarolinaThe University of North Carolina at Chapel Hill 15 Where Do UNCC Initially-Prepared Teachers Work?

16. Education Policy Initiative at CarolinaThe University of North Carolina at Chapel Hill 16 School Characteristics for UNC System Graduates School Characteristic ASUECUECSUFSUNCATNCCUNCSUUNCAUNCCHUNCCUNCGUNCPUNCWWCUWSSU % Free or Reduced-Price Lunch 56.0862.0865.7465.0465.6364.4243.1455.0547.3455.0361.5573.1257.0559.0774.98 Minority Percentage 36.2256.4155.6568.6068.7673.6148.0532.1452.2149.5754.0368.0746.8528.7671.77 Performance Composite 70.3464.6361.4763.1360.2760.7870.9670.4571.3468.5964.7461.1368.7969.5155.52 NBC Teacher Percentage 15.0212.7313.536.4310.9510.7515.3918.1115.7813.9412.398.1913.1417.069.40 Returning Teacher Percentage 82.7980.3380.0678.3378.9277.2581.3483.5181.0780.5181.4779.0681.3983.4377.50 School Type Elementary 53.6161.8659.9557.1142.5958.6019.6246.0154.1066.6860.5564.2761.7954.6661.62 Middle School 18.8215.6117.4818.5518.3018.6726.2515.8315.8715.6117.0811.5716.1717.0816.67 High School 26.8721.9722.0923.9038.1722.7354.0037.6729.7216.8421.8523.9921.5626.0520.20 Other 0.700.570.490.450.950.000.130.490.310.880.520.170.482.211.52 Teacher-School Year Count 89849861824156995111573935815262157976682237649863443594

17. Education Policy Initiative at CarolinaThe University of North Carolina at Chapel Hill 17 Summary of Programs Value-Added Results Institution Number of Significantly Higher Value-Added Comparisons Number of Significantly Lower Value-Added Comparisons Number of Statistically Insignificant Comparisons ASU 0010 ECU 2 (MS Read & MS Algebra I) 08 ECSU 005 FSU 008 NCA&T 0 2 (5 th Grade Science & MS Math) 5 NCCU 1 (5 th Grade Science) 06 NCSU 2 (ES Math & HS Algebra I) 08 UNCA 0 1 (ES Math) 7 UNCCH 1 (MS Math) 1 (MS Algebra I) 8 UNCC 3 (MS Read, HS Biology, & HS English I/II) 07 UNCG 1 (ES Math) 3 (MS Read, HS Biology, & HS English I/II) 6 UNCP 0 2 (8 th Grade Science & HS English I/II) 6 UNCW 3 (ES Math, 5 th Grade Science, & MS Math) 07 WCU 1 (5 th Grade Science) 09 WSSU 005

18. Education Policy Initiative at CarolinaThe University of North Carolina at Chapel Hill 18 UNC Programs Value-Added Results

19. Education Policy Initiative at CarolinaThe University of North Carolina at Chapel Hill 19 UNC Programs Evaluation Rating Results

20. Education Policy Initiative at CarolinaThe University of North Carolina at Chapel Hill 20 edTPA Validity Analyses Partnering with East Carolina University to conduct construct and predictive validity analyses on locally-evaluated edTPA data Locally-evaluated edTPA can provide faculty and staff: Common language and expectations for candidate performance Direct evidence about the extent to which candidates demonstrate specific knowledge and skills edTPA can leverage systems of continuous improvement Locally-evaluated edTPA may not guide TPPs to adapt/adopt more effective practices if the data are not valid and predictive of outcomes for program graduates

21. Education Policy Initiative at CarolinaThe University of North Carolina at Chapel Hill 21 Construct Validity of Locally-Evaluated Portfolios edTPA Construct edTPA RubricFactor 1Factor 2Factor 3 Planning Planning for Content Understanding 0.73 0.22-0.11 Planning to Support Varied Student Learning Needs 0.72 -0.040.18 Using Knowledge of Students to Inform Teaching 0.45 0.010.35 Identifying and Supporting Language Demands 0.77 -0.060.12 Planning Assessment to Monitor and Support Student Learning 0.69 0.16-0.02 Instruction Learning Environment0.20 0.61 -0.03 Engaging Students in Learning-0.01 0.82 0.04 Deepening Student Learning0.03 0.76 0.07 Subject-Specific Pedagogy0.01 0.61 0.19 Analyzing Teacher Effectiveness0.030.18 0.62 Assessment Analysis of Student Learning0.030.11 0.73 Providing Feedback to Guide Further Learning0.040.03 0.71 Student Use of Feedback-0.090.02 0.89 Analyzing Students’ Language Use0.14-0.02 0.72 Using Assessment to Inform Instruction0.130.03 0.73

22. Education Policy Initiative at CarolinaThe University of North Carolina at Chapel Hill 22 Predictive Validity of Locally-Evaluated Portfolios edTPA MeasuresStd. EVAAS Estimate Planning factor0.063 Instruction factor0.213 ** Assessment factor0.178 * Planning for Content Understanding0.121 Planning to Support Varied Student Learning Needs0.050 Using Knowledge of Students to Inform Teaching0.029 Identifying and Supporting Language Demands0.092 Planning Assessment to Monitor and Support Student Learning0.102 Learning Environment0.156 + Engaging Students in Learning0.271 ** Deepening Student Learning0.210 ** Subject-Specific Pedagogy0.229 * Analyzing Teacher Effectiveness0.134 Analysis of Student Learning0.124 Providing Feedback to Guide Further Learning0.194 ** Student Use of Feedback0.195 * Analyzing Students’ Language Use0.215 * Using Assessment to Inform Instruction0.154 + Standardized Total Score0.184 ** Cases209

23. Education Policy Initiative at CarolinaThe University of North Carolina at Chapel Hill 23 Predictive Validity of NCTQ Ratings Established formal collaboration with NCTQ in spring 2014 Assess the relationship between NCTQ’s TPP ratings and measures of teacher performance in North Carolina Study premise: An underlying justification for standards for teacher preparation programs is that meeting the standards should lead to higher quality preparation practices, better teacher performance, and better student outcomes If premise holds, TPP have an incentive to undertake reforms to increase their scores/ratings on standards If premise does NOT hold, TPP efforts could be better spent undertaking reforms that do lead to improve teacher and student outcomes

24. Education Policy Initiative at CarolinaThe University of North Carolina at Chapel Hill 24 Research Background for NCTQ Analyses Full sample includes 4,513 first and second year teachers in 2011-12 and 2012-13 in NC public schools Ran overall program rating, standards, and indicators models Ran models for 1 st year teachers only and for 1 st and 2 nd year teachers combined Took three approaches to address missing data—case-wise deletion, multiple imputation, and dummy variable replacement

25. Education Policy Initiative at CarolinaThe University of North Carolina at Chapel Hill 25 Summary of NCTQ Results NCTQ Ratings Associations with Teacher Performance Measures Teacher Value-AddedTeacher Evaluation Ratings Total Tests Significant Positive Associations Significant Negative Associations Total Tests Significant Positive Associations Significant Negative Associations NCTQ Overall Program Rating 42103080 Totals with NCTQ Standards 1241551403123 Selection Criteria 14301080 Early Reading 4001000 English Language Learners 4001080 Struggling Readers 4001001 Elementary Mathematics 4001000 Elementary Content 4011090 Middle School Content 4011008 High School Content 6001001 Classroom Management 1400100 Lesson Planning 14001003 Assessment and Data 14421000 Student Teaching 14211000 Secondary Methods 1000 60 Outcomes 14601000

26. Education Policy Initiative at CarolinaThe University of North Carolina at Chapel Hill 26 Limitations and Takeaways Limitations: Generalizability, missing data, and conceptual alignment between NCTQ standards and teacher outcome variables Takeaways In NC there is not a strong relationship between NCTQ’s ratings and meeting their standards and the performance of teacher preparation program graduates Results do not suggest that teacher preparation is unimportant or that the preparation components rated by NCTQ are unimportant Results suggest creating outcome-rich environments for preparation programs, setting higher standards for admission into programs, and continuing to identify ways to measure program quality

27. Education Policy Initiative at CarolinaThe University of North Carolina at Chapel Hill 27 From Accountability to Program Improvement Research Current research may encourage programmatic reforms and help teacher preparation programs better consumers of evidence but it CANNOT formatively drive evidence-based improvements With what data can teacher preparation programs make evidence-based improvements? Teacher preparation programs need individual-level data on program graduates Helps teacher preparation programs determine whether preparation practices are aligned with the working environments of their graduates Helps teacher preparation programs assess how variation in graduates’ preparation experiences explain variation in their characteristics and performance

28. Education Policy Initiative at CarolinaThe University of North Carolina at Chapel Hill 28 Data Sharing in the UNC System Initiative designed to stimulate a culture of evidence and program improvement by providing teacher preparation program with individual- level data on their program graduates In the UNC system we are providing teacher preparation programs with separate data files per school year, with each file containing data on all the individuals who were initially prepared to teach by a teacher preparation and employed as teachers in that school year

29. Education Policy Initiative at CarolinaThe University of North Carolina at Chapel Hill 29 What Data are Being Shared? Employment StatusTeacher Characteristics Classroom Characteristics School CharacteristicsTeacher Outcomes 1.District and school 2.Number of pay periods 3.First pay period 4.Last pay period 5.Amount of time worked (full-time equivalency status) 1.Teaching experience 2.Graduate degree status 3.National Board Certification status 4.Licensure areas 5.Licensure basis 6.Exams taken 7.Exam scores 8.Teaching a tested- grade/subject-area 1.Number of classes taught 2.Average class size 3.Grade level(s) taught 4.Subject-area(s) taught 5.Race/ethnicity proportions 6.Free and reduced- price lunch proportions 7.Gifted proportion 8.Disabled proportion 9.Limited English Proficient proportion 10.Average days absent 11.Average prior achievement scores 12.Average prior achievement level 1.Urbanicity 2.School size 3.Percentage free and reduced- price lunch 4.Short-term suspension rate 5.Violent acts rate 6.Race/ethnicity percentages 7.Total per-pupil expenditures 8.Per-pupil expenditures in spending categories (e.g. regular instruction) 9.AYP percentage 10.State accountability status and growth 11.Performance composite 12.Teacher credentials—percentage fully-licensed, novice, holding an advanced degree or NBC 13.Pupil to teacher ratio 14.Teacher stay ratio 1.Returns to the state’s public schools 2.Returns to the same school 3.Teacher value- added estimate (across 10 separate subject- areas) 4.Quintile for value- added estimate

30. Education Policy Initiative at CarolinaThe University of North Carolina at Chapel Hill 30 Connections to Program Improvement

31. Education Policy Initiative at CarolinaThe University of North Carolina at Chapel Hill 31 Thoughts on Program Improvement These data tell you where to look Programs need individual-level data on program graduates Important to conduct drill-down studies to understand WHY Important to have multiple outcomes to assess TPP performance Program improvement aided when outcomes point in the same direction Opportunities to leverage performance assessment data and to examine candidate success on new licensure examinations Program improvement requires leadership, capacity, and culture

32. Education Policy Initiative at CarolinaThe University of North Carolina at Chapel Hill 32 Future Research Directions edTPA analyses Student teaching and clinical placements Personality traits/non-cognitive characteristics School leader programs analyses Assistant principal experiences Incorporation of additional outcome measures Program-specific research and data use

33. Education Policy Initiative at CarolinaThe University of North Carolina at Chapel Hill 33 Thank You! Contact Information kbastian@email.unc.edu http://publicpolicy.unc.edu/ http://tqdashboard.northcarolina.edu/