1. Has NCLB Improved Teacher and Teaching Quality for Disadvantaged Students? Laura M. Desimone Thomas M. Smith Vanderbilt University David Frisvold University of Michigan Presented at the Society for Research on Educational Effectiveness conference, Lansdowne, Virginia, December 10-12, 2006. Laura M. Desimone Thomas M. Smith Vanderbilt University David Frisvold University of Michigan Presented at the Society for Research on Educational Effectiveness conference, Lansdowne, Virginia, December 10-12, 2006.

2. Research Questions 1.What were the gaps in teacher and teaching quality for students in poverty compared to their more advantaged peers in 2000, and to what extent did those gaps narrow by 2003? 2. Are improvements in teacher quality and/or the narrowing of teacher quality gaps associated with state implementation of NCLB?

3. Why is Teacher and Teaching Quality Important? More Experience & Content Knowledge Improved Instruction Certification Increased Student Learning Teacher QualityTeaching Quality

4. State Implementation of NCLB NCLB requires states to take steps to ensure that poor and minority children “are not taught at higher rates than other children by inexperienced, unqualified, or out-of-field teachers.” Time period: 2000-2003 Target: 2005-2006

5. Conceptual Framework: Policy Attributes Theory Policy ConsistencySpecificityAuthorityPowerStability

6. Data State Policy Database 2000 National NAEP 2000 and 2003 State NAEP

7. Measures  Instruction -Conceptual Emphasis -Conceptual Strategies -Procedural Teaching  Full vs. partial certification  Inexperienced Teacher: 2 Years or Fewer  BA or higher in mathematics

8. RQ1: What were the gaps in teacher and teaching quality for students in poverty compared to their more advantaged peers in 2000, and to what extent did those gaps narrow by 2003? Mean comparisons 2000 & 2003 national and by state and free lunch status

11. Summary of State-by-State Mean Comparisons on Teacher Quality Indicators

12. RQ2: Are improvements in teacher quality and/or the narrowing of teacher quality gaps associated with state implementation of NCLB? Do states that are stronger on the policy attributes have higher teacher quality in 2000 than states with weaker policy attributes? Do states that are stronger on the policy attributes have higher teacher quality in 2000 than states with weaker policy attributes? Are changes in state policies between 2000 and 2003 associated with increases in teacher quality? Are changes in state policies between 2000 and 2003 associated with increases in teacher quality? Was the implementation of policies between 2000 and 2003 associated with a reduction in poverty gaps in teacher quality? Was the implementation of policies between 2000 and 2003 associated with a reduction in poverty gaps in teacher quality?

13. State Policy Data Compiled from existing national data sources Compiled from existing national data sources –Education Week’s Quality Counts –American Federation of Teachers’ report on states –Key State Policies by the Council of Chief State School Officers –Report cards on standards by the Thomas B. Fordham Foundation Chose measures available in both 2000 and 2003 Chose measures available in both 2000 and 2003

14. State Policy Measures  Alignment of standards and assessments (consistency) NCLB: “ State assessments shall be aligned with the State ’ s challenging academic content and student academic achievement standards ” [(C)(ii), p.115 STAT. 1450]. Our measure: state used criterion-referenced assessments in middle school mathematics that had undergone an external alignment review in 2000 and in 2003.  Alignment of standards and assessments (consistency) NCLB: “ State assessments shall be aligned with the State ’ s challenging academic content and student academic achievement standards ” [(C)(ii), p.115 STAT. 1450]. Our measure: state used criterion-referenced assessments in middle school mathematics that had undergone an external alignment review in 2000 and in 2003.

15. State Policy Measures  Clear and detailed standards (specificity) NCLB: “ challenging academic standards shall include challenging academic content standards in academic subjects that specify what children are expected to know and be able to do and contain coherent and rigorous content ” [SEC. 1111; p. 115 STAT. 1445] Our Measure: state had “ clear and specific standards ” in middle school mathematics.  Clear and detailed standards (specificity) NCLB: “ challenging academic standards shall include challenging academic content standards in academic subjects that specify what children are expected to know and be able to do and contain coherent and rigorous content ” [SEC. 1111; p. 115 STAT. 1445] Our Measure: state had “ clear and specific standards ” in middle school mathematics.

16. State Policy Measures  Providing professional development resources (authority1) NCLB: To improve the academic achievement of the disadvantage can be accomplished by “ significantly elevating the quality of instruction by providing staff in participating schools with substantial opportunities for professional development ” [SEC. 1001; p.115 STAT. 1440]; Our measure : state provides assistance to low-performing schools

17. State Policy Measures  Providing resources to low-achieving schools (authority2) NCLB: To improve the academic achievement of the disadvantage can be accomplished by “ distributing and targeting resources sufficiently to make a difference to local educational agencies and schools where needs are greatest ” [SEC. 1001; p.115 STAT. 1440]; Our measure : state provides professional development resources

18. State Policy Measures  Ranking schools (power1) and number of possible sanctions (power2) NCLB: “…a local educational agency shall identify for school improvement any elementary school or secondary school served under this part that fails, for 2 consecutive years, to make adequate yearly progress ” [(b)(1)(A), p. 115 STAT. 1479]. Our measures : (1) state assigns ratings to all schools or identifies low-performing schools (2) number of possible sanctions

20. Do States With Stronger Policy Attributes Have Higher Teacher Quality In 2000 Than States With Weaker Policy Attributes? Three-level hierarchical linear model on the NAEP 2000 national sample: Three-level hierarchical linear model on the NAEP 2000 national sample: Where Q ijs = inexperienced teacher (2 or fewer years of experience), certification status, level of preparedness to teach different mathematics topics, whether or not the teacher has a degree in mathematics, as well as teachers’ use of different instructional strategies, including conceptual emphasis, conceptual strategies, and procedural teaching. Where Q ijs = inexperienced teacher (2 or fewer years of experience), certification status, level of preparedness to teach different mathematics topics, whether or not the teacher has a degree in mathematics, as well as teachers’ use of different instructional strategies, including conceptual emphasis, conceptual strategies, and procedural teaching.

22. Cross-sectional Relationship Between Policy and Teacher/Teaching Quality in High-Poverty Schools

23. Changes in state policy between 2000 and 2003 11 states adopted measures to assist low- performing schools (authority1) 11 states adopted measures to assist low- performing schools (authority1) 9 states started or increased the resources they gave to professional development (authority2) 9 states started or increased the resources they gave to professional development (authority2) 23 states began ranking schools according to achievement results (power) 23 states began ranking schools according to achievement results (power) Most states had clear and detailed standards as early as 2000, but a handful of states implemented them between 2000 and 2003 (specificity) Most states had clear and detailed standards as early as 2000, but a handful of states implemented them between 2000 and 2003 (specificity) Most states conducted an alignment review in 2000; four states conducted an alignment of their standards and assessments in 2003 (consistency) Most states conducted an alignment review in 2000; four states conducted an alignment of their standards and assessments in 2003 (consistency)

24. To examine the relationships between change in state policies and change in teacher quality, while controlling for change in state-level poverty, we estimated the following model: Where Q st represents average teacher quality in state s at time t (t = 2000, 2003), POLICY represents each of the five state policies in the analysis (Power, Consistency, Specificity, Authority1, and Authority2) Where Q st represents average teacher quality in state s at time t (t = 2000, 2003), POLICY represents each of the five state policies in the analysis (Power, Consistency, Specificity, Authority1, and Authority2) Also, we ran models separately for high and low poverty schools Also, we ran models separately for high and low poverty schools

26. Was the implementation of policies between 2000 and 2003 associated with a reduction of poverty gaps in teacher quality? where Q ijst represents the characteristics of teacher i in school j in state s at time t where Q ijst represents the characteristics of teacher i in school j in state s at time t

29. Summary: Poverty Gaps and How They Have Changed Small poverty gaps exist and didn’t change much from 2000 to 2003 Small poverty gaps exist and didn’t change much from 2000 to 2003 Disadvantaged students in advantaged schools are worse off than their advantaged peers in disadvantaged schools Disadvantaged students in advantaged schools are worse off than their advantaged peers in disadvantaged schools

30. Summary: Associations of Teacher Quality with NCLB-Related Policy The relationship between policy attributes and teacher quality is not much different for advantaged and disadvantaged students. The relationship between policy attributes and teacher quality is not much different for advantaged and disadvantaged students. States that provided professional development resources (authority) had students whose teachers were more likely to use conceptual teaching strategies; state with more sanctions had students whose teachers used more procedural instruction and had higher content knowledge. States that provided professional development resources (authority) had students whose teachers were more likely to use conceptual teaching strategies; state with more sanctions had students whose teachers used more procedural instruction and had higher content knowledge. Implementation of certain policies was weakly associated with improvements in teacher quality in several cases, but generally did not reduce the relationship between poverty and teacher quality. Implementation of certain policies was weakly associated with improvements in teacher quality in several cases, but generally did not reduce the relationship between poverty and teacher quality. –Power mitigated it, specificity made it worse

31. Challenges to Studying Policy Effects on Teacher Quality Complex nature of interactions of state and local policy (direct and indirect effects) Complex nature of interactions of state and local policy (direct and indirect effects) Implementation of simultaneous multiple policy levers Implementation of simultaneous multiple policy levers Time ordering: policy may first reflect reaction to a problem, and only several years later would positive results be expected; how long does it take for policy to affect trends in teacher quality? Time ordering: policy may first reflect reaction to a problem, and only several years later would positive results be expected; how long does it take for policy to affect trends in teacher quality? Real change vs. random fluctuation in short time span (e.g., 3 years) Real change vs. random fluctuation in short time span (e.g., 3 years) Quality of measures Quality of measures

32. Significance Teacher and teaching quality affect student outcomes Teacher and teaching quality affect student outcomes Currently we are implementing multiple policy levers to improve teaching: (1) merit pay, (2) professional development (mentoring, coaching, induction), (3) school and curriculum reforms, (4) NCLB requirements, (5) teacher preparation reforms (6) recruitment Currently we are implementing multiple policy levers to improve teaching: (1) merit pay, (2) professional development (mentoring, coaching, induction), (3) school and curriculum reforms, (4) NCLB requirements, (5) teacher preparation reforms (6) recruitment

33. What policies work to improve teaching, retain good teachers, and recruit potentially good teachers? What policies work to improve teaching, retain good teachers, and recruit potentially good teachers? Use student achievement as a measure of effectiveness Use student achievement as a measure of effectiveness Ask how teachers are changing in response to policy levers Ask how teachers are changing in response to policy levers Ground studies in theory about policy and teacher behavior Ground studies in theory about policy and teacher behavior Directly compare effects of 2 or 3 key policy levers, e.g., preparation, incentives, mentoring/coaching/professional development Directly compare effects of 2 or 3 key policy levers, e.g., preparation, incentives, mentoring/coaching/professional development Future Questions and How We Might Pursue Them