Inside the ‘Black Box’ Of School Improvement: Measuring Change James P. Spillane Northwestern University Chicago March 11, 2008 Institute of Education Sciences, Spencer Foundation, National Science Foundation, Carnegie Foundation

Organizing for Teaching and Learning Formal Structure Formal positions Organizational routines Relational Structure: A Social Network Approach Norms and Beliefs Network Structure Substance of Interactions

Managing and Leading Instruction Heroics of Leadership Principal Plus Styles Behaviors The missing piece - the practice of leading & managing A Distributed Perspective Attention to both formal and informal leaders Practice is about interactions, not just actions The Subject Matters

Research Instruments Logs Experience Sampling Method (ESM) log Leadership Daily Practice (LDP) Log End of Day Log (EOD) Social Network Survey Mixed Method Studies

Today’s Talk Motivating the work: Why Look Inside Schools? Social Network Survey Design & Validation Organizational ‘measures’ Norms & beliefs Structure Substance

Motivating the Work Why Look Inside Schools?

Motivating the Work ‘Mediator Variables’ “factors that must be changed or modified first by the treatment before main outcomes can be affected” (Petrosino, 2000, p. 50). “generative mechanism through which the focal independent variable is able to influence the dependent variable of interest” (Baron & Kenny, 1986, p. 1173). Build and test program/intervention theory (Weiss,1997). Hypotheses generating/theory development

Anchoring the Work: What to Measure First Dimension: Normative Structure Social Trust Collective Responsibility Innovation Second Dimension: Relational Structure Tie strength Tie span Third Dimension: Substance of Interactions

Social Network Instrument Design and Validation Work

Social Network Survey Screen shot from SSSNQ Version 1 - Math Advice Questions

Social Network Instrument Screen Screen Shot from SSSNQ Version 2 – Math Advice Questions Page 1

Social Network Instrument Screen Screen Shot from SSSNQ Version 2 – Math Advice Questions Page 3

Social Network Instrument Screen Screen Shot from SSSNQ Version 2 – Math Advice Questions Page 2

School staff network survey M/R R/M Randomization math name generator math name interpreter RWLA name generator RLWA name interpreter RWLA name generator RWLA name interpreter math name generator math name interpreter

Three ways in which network data is susceptible to bias How many alters are named –fatigue or satisficing Who is named –censoring or priming Why those names were chosen/alter attributes –question scope redefinition

Average number of alters listed By subject area and treatment group Subject area R/M (n = 126) M/R (n = 138)Difference RWLA * Math Total *

Some Results The Instrument in Use

The First Dimension: Norms and Beliefs Why should we care: –Trust moderates sense of vulnerability and uncertainty that accompanies change. –Trust enables joint problem solving. –Trust among school staff is linked to gains in student performance. The Evidence from the field –Teacher-teacher trust –Teacher-principal trust

strongly agree strongly disagree agree disagree Trust among teachers

strongly agree strongly disagree agree disagree

The Second Dimension: Relational Structure Frequent and influential interactions enable transfer of tacit, complex knowledge and joint problem solving. Relationships that span the formal organization allow for new ideas and minimize group think. Relationships that encompass all school staff members affects the degree to which staff members are engaged in joint work, and the consistency of advice provided.

Measuring Relational Structure Tie Strength –Average number of advisors –Average number of advisors with at least weekly contact –Average influence Tie Span –Average number of advisors in other grade level teams/departments –Average number of advisors external to the school Network Concentration –Percent of ties to formal leaders –Distribution of advisors by number of advisees

Color corresponds to staff role Reading/Writing/ Language Arts Math Science Social Studies Special Education 6th Grade Special Subjects Administration A B C Relational Structure

Color corresponds to staff role Reading/Writing/ Language Arts Math Science Social Studies Special Education 6th Grade Special Subjects Administration Relational Structure M2M2 Math Coordinator M2M2 M2M2

Average in-degree: M 2 associates vs. Non-M 2 associates M2M2 Non-M 2 N=6N=95N=16N=35N=541

In math network, 8 out of 10 schools have two 6 th grade team members connected to math department Spanning Sub-units Color corresponds to staff role Reading/Writing/ Language Arts Math Science Social Studies Special Education 6th Grade Special Subjects Administration

In RWLA network, 7 out of 10 schools have zero or one 6 th grade team member connected to RWLA department Spanning Sub-units Color corresponds to staff role Reading/Writing/ Language Arts Math Science Social Studies Special Education 6th Grade Special Subjects Administration

In about half of the schools, an M 2 associate provides the bridge between the 6 th grade team members and the math department Spanning Sub-units Color corresponds to staff role Reading/Writing/ Language Arts Math Science Social Studies Special Education 6th Grade Special Subjects Administration M2M2 M2M2 M2M2 M2M2 coordinator M 2 Math

Average external out-degree: M 2 associates vs. Non-M 2 associates M2M2 Non-M 2 N=5N=86N=16N=29N=445 M2 participants seek advice from more sources outside of their schools compared to their colleagues, suggesting that the M2 participants may be key brokers between their schools and external sources.

How is the red group different from the blue group?

In some schools, one or two key players provided practically all of the advice (a high network concentration). In other schools, many teachers were involved (a lower network concentration). Generally, math networks were more concentrated than RWLA networks. Structure: The Distribution of Influence more concentrated less concentrated

In some schools, one or two key players provided practically all of the advice (a high network concentration). In other schools, many teachers were involved (a lower network concentration). Generally, math networks were more concentrated than RWLA networks. Concentrated Influence? more concentrated less concentrated M 2 Math teachers Self-contained teacher Math teacher M 2 Math teacher/coordinator

The Third Dimension –The Substance of interactions among staff -- what teachers talk about! –Instruction is a multi-faceted practice including Deepening content knowledge Course content teaching strategies, Materials Strategies for low performing students Assessing students understanding

RWLA Network Average out-degree by school

Math Network Average out-degree by school

Math Network Percent of ties that deal with each content area

The instructional practices and assessments discussed or shown in these presentations are not intended as an endorsement by the U.S. Department of Education.