1. Will That Work for Us? Interpreting Research from The Memphis Striving Readers Project (MSRP) Presented by Ric Potts, MCS; J. Helen Perkins, U of M; Elizabeth Heeren, MCS; Rorie Harris, MCS; and Jill Feldman, RBS 2008 International Reading Association Research Conference Atlanta, GA

2. Session Overview Introduction to the Striving Reader’s grant Overview of Memphis SR research design Year One Impact Analyses Collection of implementation fidelity data –implications for practitioners and researchers Planned (Ongoing) Analyses Q & A /Group Discussion

3. Introduction: Memphis Striving Readers Project (MSRP) Ric Potts, PI – MSRP Memphis City Public Schools

4. Memphis-The City The City of Memphis has a population of 642,251. 63.1% African American 31.3% Caucasian 4.1% Hispanic

5. And one Elvis

6. Approximately 70 percent of adolescents struggle to read. The young people enrolled in middle and high school who lack the broad literacy skills to comprehend and learn advanced academic subjects will suffer serious social, emotional, and economic consequences. »Reading at Risk: The State Response to the Crisis in Adolescent Literacy, Oct. 2005

7. Urban Child Institute The State of Children in Memphis and Shelby County 2006 “Under-educated children have no future.”

8. Urban Child Institute The State of Children in Memphis and Shelby County 2006 by U.S. standards roughly 75 percent of students in Tennessee fail to meet national grade appropriate standards, and Memphis is at the bottom in Tennessee.... Memphis is one of the least-educated cities in America.

9. Motivation behind Memphis Striving Readers Project Memphis is among the poorest and least-educated cities in the US –30.1% of all children live in poverty –24.3% of adults have less than a HS education –36.7% have HS diploma or equivalent –30.5% have Assoc. or some college –8.5% have at least a BA MCS is 21st largest K12 district in US >116,000 students –Over 95% of MCS’ 196 schools are Title I schools –71% of MCS students qualify for free/reduced price lunch –MCS students are 87% AA; 9% White; 4% “other” –In 85% of MCS schools, 33% of students change schools during year –In 2003-04, the system-wide graduation rate was 61 percent –71% of students in grades 6-8 scored below the 50 th percentile on TCAP (Reading/Language Arts)

10. Striving Readers – A Federal Response In 2005, the Department of Education called for proposals for the Striving Readers grant. In March, 2006, Memphis was one of eight sites awarded the grant.

11. Memphis Striving Reader Program Targeted Schools SchoolGrade Span Total EnrollmentTotal # Of Non-Special Education Students Scoring In Bottom Quartile In Reading School 26-81,021414 School 16-81,033384 School 66-8700251 School 56-8765245 School 86-8547178 School 46-8486196 School 36-8976357 School 76-8877274

12. The Whole School Intervention: Memphis Content Literacy Academy (MCLA) Overview presented by J. Helen Perkins, SR Co-PI University of Memphis

13. A Change Model

14. No Knowledge First Exposure Deeper Learning with Limited Capacity Practice with Coaching Refined and Expanded Capacity Expertise & Ability to Coach Others A Capacity-Building Model for Teacher Development (Cooter & Cooter, 2003) Emphasis: “Deep Training” (180 hours over two years) …

15. Memphis Content Literacy Academy Infusing Simultaneously Across Core Subject Areas Scientifically-based Reading Research (SBRR) Strategies in… Vocabulary Reading Comprehension Reading Fluency

16. Benefits to Teacher – “Laureates”… Advanced Training (180 hours) on scientifically-based reading instruction (SBRR) for urban children A Master Teacher “Coach” to Assist (30 hours) with Implementing New Strategies (in their own classrooms!) Twelve (12) Graduate Semester Hours of Credit from University of Memphis (FREE) (applicable to an advanced degree) Can Seek “Highly Qualified” Endorsement in Reading Books and Materials (FREE) Success in Helping Children Achieve “AYP” Principal Support

17. MCLA Year 1: Selected Strategies Fluency Choral Reading Paired reading Guided, repeated, oral reading (pairs)

18. Comprehension Question Generation Three- Level Retelling Oral Graphic Organizor Written Comprehension monitoring Expository Text Patterns Multiple Strategies

19. Vocabulary Development Pre-instruction of vocabulary Repeated, multiple exposures Semantic Maps

20. Classroom Organizational Tools & Strategies: Year 1 CREDE Standards Whole class v. collaborative small group Reading Next Elements Use of leveled materials (e.g., National Geographic)

21. CREDE Formatting of Professional Development Training http://crede.berkeley.edu/standards/standards.html

22. Classroom Action Plans (CAPs) Spring 2008 Science, Social Studies, & ELA Your task is to develop a series of class lessons where you teach academic vocabulary in a unit of your choice. You must have at least one vocabulary learning strategy/activity that occurs: 1. BEFORE students read the assigned text, 2. DURING the reading assignment, and 3. AFTER the reading assignment

23. MCLA Classroom Model Gradual release of responsibility (teacher modeling, guided practice, independent practice, independent use) Integration of 12 literacy strategies (vocabulary, fluency & comprehension) Development of Classroom Action Plans (CAPs) (content area lesson plans for strategy implementation including procedures for student assessment) On-site support provided by coaches Use of Curriculum Resource Center (CRC) materials

24. The Principals’ Fellowship Literacy Leadership Practices Real World Problem Solving Create “Literacy Materials Centers” Early Identification w/ Intense/Focused Remediation Research-Informed Decision Making Involve Families Needs-Based Scheduling Matching the Most Successful Teachers with “Critical Condition” Kids

25. READ 180, Our Targeted Intervention Overview provided by Elizabeth Heeren, SR Grant Coordinator Memphis City Schools

27. Program Components Student workbooks for Independent Practice in small and whole group rotations Support materials for differentiated instruction in small group rotation Tools for student placement and assessment

28. Key Elements of READ 180 Fidelity of Implementation 90 minute classes Certified teachers (LA or Reading) District Instructional Support District Technological Support Scholastic training (site-based and on-line)

29. R180 Correlations to Reading Next Recommendations for Adolescent Literacy Direct, explicit comprehension instruction Motivation and self-directed learning Strategic tutoring Differentiated texts (levels and topics) Technology component Ongoing formative assessment Extended time for literacy Professional development (long-term and on- going)

30. Memphis Implementation We have 8 schools in the Striving Readers Grant, with up to 120 randomly selected R180 students at each school. Students receive R180 instruction for 2 years. Each student placed in R180 falls in the lowest quartile of TCAP (Reading score). Each student in R180 is paired with a similar student from the lowest quartile who does not receive the treatment (for impact comparison).

31. MSRP Research Design Overview presented by Jill Feldman, SR Research Director Research for Better Schools

32. Overall MSRP Goals To determine: 1.The effects of MCLA on core subject teachers’ knowledge and use of SBRR 2.The separate and combined effects of MCLA and Read 180 on students’ reading achievement levels, especially students who are identified as struggling readers 3.The separate and combined effects of MCLA and Read 180 on students’ achievement in core subjects, especially students who are identified as struggling readers

33. Funding, staff, curriculum resource center, facilities, incentives, research materials Principals 45 hours of Principal Fellowship participation 100% of principals incorporate plan for using MCLA strategies in SIP 100% attendance of key MCLA events 80% of principals report actively supporting the program 100% of MCLA schools have allocated space for the CRC Teachers 90 of hours of MCLA training/yr for 2 years (180 hours) Engage in weekly coaching sessions or as needed to meet teachers’ differentiated needs 8 CAP “cycles” completed each year for two years 100% of teachers complete performance measures identifying supplemental resources available/those necessary to support content area instruction Students 50% of students attend 4 classes taught daily by teachers participating in MCLA Students learn to use 7 of 8 MCLA CAP strategies Outputs Principals Awareness of and interest in staff implementation of MCLA concepts and strategies Increased advocacy for school- wide use of MCLA strategies Teachers Increased knowledge about MCLA strategies Improved preparedness to use research-based literacy strategies to teach core academic content Increased use of direct, explicit instruction to teach research- based comprehension, fluency, and vocabulary strategies in content area classes Integrated use of multiple MCLA strategies to support ongoing development of content-related instructional units Students Increased familiarity with and use of MCLA strategies when engaging with text Increased internalization of literacy strategies Increased confidence engaging with content related texts Increased interest in school/learning Short–term Outcomes Principals Improved school climate School-wide plans include focus on content literacy Improved instructional leadership Teachers Increased effectiveness supporting students’ content literacy development Continued collaboration among community of teachers to develop and implement CAPs Students Improved reading achievement and content literacy: 10% increase in students scoring proficient in Reading/LA and other subject areas of TCAP mean increase of five NCEs on ITBS Increased performance on gateway and EOC exams Long-term Outcomes Higher Quality Teaching Principals Attend 45-hour sessions/yr (2 yrs) Participate in motivational, recruitment and celebratory events Discuss MCLA at faculty meetings Conduct walkthrough observations Provide opptys for teacher collab Allocate space for CRC materials Teachers Attend 30 weekly 3-hour MCLA training sessions/yr (2 years) Develop and implement 8 CAPs per year in collab content-area groups Meet with coaches for feedback to improve impl of MCLA strategies Learn to use of leveled texts to support SR content literacy needs Students Learn to use MCLA strategies to read/react to content related text ( MCLA Program Logic Model Higher Student Achievement

34. Study Design and Analytic Approach: MCLA Study Design MCLA: Evaluate teacher and student outcomes –experimental design –randomly assigning schools (to treatment and control conditions) Teacher outcomes include – preparedness – frequency of literacy strategy use Analytic Approach MCLA: Two-level HLM –spring ITBS and TCAP scores as a function of teacher and school variables

35. Analytic Decisions Missing Data –students missing pretest score(s) deleted from impact analysis on relevant measure(s) –teachers missing pretest score deleted from impact analysis on measure Covariates –include all student- and school-level covariates in the model –run the model –eliminate the school covariate with the lowest significance level (highest p-value) not less than 0.2 – repeat steps 2 and 3 until the remaining covariates had p-values less than 0.2 – repeat steps 2-4 for the student covariates

36. MCLA: Random Assignment of Schools

37. Demographic Characteristics of Year 1 MCLA Student Sample

38. Baseline Comparisons of Students in MCLA Treatment and Control Schools

39. Selected Characteristics of the Year 1 Teacher Sample for MCLA Impact Analyses

40. All Variables Included in MCLA Impact Analytical Models for Year 1

41. READ 180 Logic Model

42. R180 Study Design Analytic Approach Study Design: Evaluate student outcomes using RCT based on random assignment of students to conditions across schools Student outcome measures : –reading achievement (ITBS) –core content areas (TCAP) Analytic Approach: Cross-sectional ITT analyses of reading and core content area achievement Two-level models using spring ITBS and TCAP scores as a function of student and school variables

43. READ 180: Enrolled Students

44. Variables Included in READ 180 Impact Analytic Models (Year One): Dependent and Independent

45. Variables Included in READ 180 Impact Analytic Models (Year One): Covariates

46. Year One Impact

47. Comparison of Teachers in MCLA Treatment and Control Schools on Year-End Indices for Preparedness and Frequency of Use

48. MCLA Impacts on Students (Year One)

49. READ 180 Impacts on Students (Year 1)

50. Collection of Data about Implementation Fidelity

51. Implications for Researchers and Practitioners What are our purposes for collecting implementation data? 1.To provide other districts with information about outcomes they might expect when implementing similar interventions with their struggling readers* 1.To set the context for understanding student outcomes *Requires MCS to place the needs of the field above local concerns

52. Reasons to Collect “Double Data” R180 evaluation is intended to test effects of a replicable intervention in the real-world: 1.Without the support of external evaluators 2.In ways that emulate what districts will need to do to: monitor implementation obtain process feedback

53. Reasons to Collect “Double Data” Collecting data about MCLA and R180 fidelity helps researchers explain patterns of impact findings can be useful in identifying predictors of outcomes

54. What Is the Role of the Researcher? RBS collects data about: –Impact (MCLA & R180) –Implementation fidelity To better understand impact or lack thereof (MCLA & R180) To support development of MCLA (only) –Counterfactual To compare effects to what would have happened in SR schools in the absence of MSRP

55. What is the Role of MCS? Implement R180 & MCLA Monitor the implementation process –Ensure implementation is “on model” –Refine service delivery based on formative data

56. Defining Implementation Fidelity: MCLA Innovation Configuration Mapping

57. MCLA Implementation Framework Developing an Innovation Configuration (IC) Map (Hall & Hord, 2006) –Operationally defines levels of implementation fidelity among clusters of “key active ingredients” –Iterative process involving key stakeholders Development team (University of Memphis) Grantee (Memphis City Public Schools) Researchers (Research for Better Schools)

58. MCLA: Roles & Responsibilities MCS Administrators: Participate in Principal’s Fellowship Support recruitment and retention efforts Link MCLA w/School Improvement Plan Observe MCLA teachers (once/marking period) Allocate space for CRC materials Protect/respect role of coach Developer: Design MCLA curricula (for teachers & principals) Facilitate writing team activities Meet weekly with instructors (& coaches) Disseminate research about adolescent SR

59. MCLA Training Provided by the Developer: 3-hour weekly principal meetings (fall;Year 1) 3-hour weekly teacher training sessions per content area (180 hours over 2 years)* PD for coaches in Mentorship; Urban education; Adolescent lit Provided by MCS (coaches): On-site observation of CAPs Model/co-teach strategies Feedback Supplemental resources *has included coaches since spring 2007

60. MCLA Innovation Configuration Map Framework

61. Instrument Development With the IC map guiding development, the following measures were designed to collect data about MCLA implementation: Surveys –Teacher knowledge about & preparedness to use MCLA strategies –Teacher demographic characteristics –Teachers’ MCLA Feedback Interviews –Principals, coaches, development team, and MCS administrators Teacher Focus Group Discussions

62. Operationally defining components: “Job Definition”

63. Aligning the IC Map and Instrument Development: “Job Definition” – Teacher Survey

64. “Job Definition” - Principal Interviews

65. MCLA Innovation Configuration Map Framework

67. Where the rubber hits the “runway”… MCLA Classroom Implementation

68. Operationally defining components: Implementation of Lesson Plans

69. Implementation of lesson plans: Collecting classroom observation data

71. MCLA: Implementation Barriers Barriers: Limited development/planning time Need for coaches with disciplinary content knowledge Challenges in establishing a critical mass of enrolled teachers at each school CRC materials not received until spring 2007 Pressure to focus on TCAP test preparation (spring) Difficulty maintaining principal attendance at weekly meetings

72. MCLA: Planned Implementation Changes Changes: Adoption of CREDE (UC-Berkeley) JPA instructional model Reduction in the number of CAPs required of teachers Shortened class schedule/more intensive work with coaches Inclusion of special education teachers among those eligible to enroll Restructured Principal Fellowship (includes other school leaders; meets monthly)

73. Defining Implementation Fidelity: R180 Rorie Harris Memphis City Public Schools

74. Findings Related to Implementation Scheduling –Scheduling 90 minute blocks in schools using the Middle School concept is difficult. Teams of core content teachers traditionally have 55 minute classes. –Interruptions to the 90 minute block occur. Special Education Students –READ 180 will only suffice as a SPED student’s intervention if the teacher is SPED-certified.

75. Findings Related to Implementation Use of Technology –Technology issues can negatively affect instructional time. Parents & Students –Some parents do not want their children in Reading Intervention classes. They feel like this is a “label.” –Classroom management issues impact instruction. –Student mobility affects the scope and sequence of reading instruction.

76. Findings Related to Implementation School Administration –Without administrator “buy-in” to the importance of smaller classes and protection of the 90 minute block, fidelity is not supported. Read 180 Teachers –It is challenging to encourage ALL teachers to engage in on-line professional development and/or to attend network meetings. –Teacher turn-over brings out the need for repeated initial training and reduces the development of teacher leaders.

77. Indicators of Read 180 Implementation Scholastic identifies several key program aspects –Teacher Training/Professional Development –Computer Hardware/Software Use –Use of Read 180 Materials –Group Rotation –Class Size –Classroom Environment –Student Engagement

78. Sources of Implementation Data Classroom observations during the school year (Fall & Spring) Read 180 program databases (SAM) Enrollment and course-related data from district databases Surveys administered to students (Fall & Spring) and teachers (Spring) Information collected during professional development programs

79. MCS Data Linked to Implementation Indicators MCS Data SourceKey Program Area Completion of Scholastic RED Course Teacher Training Attendance at district-wide Read 180 Network Meetings Teacher Training Fall & Spring Classroom Observations Computer Hardware & Software Use Group Rotations Class Size Classroom Environment Use of Read 180 Materials Enrollment Data Class Size

80. MCS Data Linked to Implementation Indicators MCS Data SourceKey Program Area Student Usage Data from SAM Computer Hardware & Software Usage Student Surveys Classroom Environment Student Engagement Use of Read 180 Materials Teacher Survey Computer Hardware/Software Use Classroom Environment Group Rotations Use of Read 180 Materials

81. Overview of Year One Conclusions Jill Feldman, RBS

82. (Brief) Conclusions & Discussion READ 180: No significant Year One student impact Late startup (Most) students will receive two years of intervention Planned Future Analyses: Three-level analyses planned to examine whether teacher characteristics exert a moderating effect on student outcomes Exploratory analyses of relationships between amount of READ 180 instruction and effects on student outcomes

83. (Brief) Conclusions & Discussion MCLA: Significant (moderate) impact on teachers’ frequency and preparedness to use MCLA strategies No significant impact on students’ achievement in reading or core content areas Discuss: –Subjectivity of measure (“Hawthorne Effect”) –Teacher findings support program logic model –Explore relationship between impact and participation in PD

84. Next Steps…

85. Planned Exploratory Analyses Re-run HLM impact analyses to test effects of teacher variables on outcomes –Preparedness and use of MCLA strategies –Age –Experience as teacher (& years at MCS) –PD in year prior to MCLA

86. Planned/ongoing analyses Individual student’s growth over time Rerun HLM with student-level variables –# MCLA teachers –Student’s school attendance ITS analyses –Using TCAP Spring 2003 & 2004 scores Correlating R180 data with TCAP & ITBS –for possible use as covariates in HLM

87. Now It’s Your Turn Ask the panel Share your experiences –Triumphs –Tribulations

88. Thank you for joining us! For additional information contact: feldman@rbs.org