1. Team Read Improving Literacy in the Seattle School District
PA 590C – Program Evaluation
Fall 2007
Anne Debuyserie

2. (Team Read Mission Statement, 1998)
“...a Seattle School district tutoring program dedicated to increasing the reading skills of elementary students through year-long coaching by high school students ... The program's goals for participating high-school tutors are to develop work experience, a sense of responsibility and accomplishment, and to learn the rewards of community service.”
(Team Read Mission Statement, 1998)

3. What is Team Read?
Private-public partnership to improve literacy in Seattle Public Schools
Cross-age tutoring program using high school student as tutors
Started in March 1998 with a three-phase implementation
June 1999: 10 schools, 335 students, 300 coaches
June 2000: addition of 7 schools
Phase 1: March 1998, reading sessions in 4 schools
Phase 2: October 1998, 2 schools added
Phase 3: February 1999, 4 more schools added
2nd yr expansion: for a total of 17 elementary schools, coaches from 15 HS

4. Team Read- Org Chart

5. Team Read Stakeholders
Craig McCaw (and wife Susan): Businessman
John Stanford: Seattle Schools Superintendent
Joan Dore: Seattle Schools Reading Specialist
Tricia McKay: Team Read Program Mgr
Team Read Advising Board: Seattle School District Staff, Alliance for Education pgm representatives, Mc Caws, community representatives.
Additional investors joined by year 2.
Craig McCaw, his wife Susan and John Stanford are at the origin of Team Read. They approached Reading Spec Joan Dore who came up with the Team Read idea.
Alliance for Education: formed in mid 90s to provide private investment and involvements in the Seattle Public Schools. The McCaw Foundation directs Team Read funding through the alliance and collaborates on reading-related initiatives
Additional investors: Bruce and Jolene McCaw, Keith and Mary Kay McCaw, Dennis and Beth Weibling, and the Laurel Foundation.

6. Who is Team Read? Coaches Readers Strict selection guidelines
High expectations from Team Read leaders
Compensation
Readers
School selection comes first
Student selection based on reading level
Eligibility based on need
First-come, first-serve basis
Low socio-economic background
4 types of staff are involved in Team Read: coaches, readers, site coordinators and volunteers
Coaches selection guidelines:
Mini 14 yrs old
Registered at a Seattle Pub HS
Mini 2.7 GPA and/or higher grades in reading or related subjects
Good attendance record
Fluent in English
Must have good references
Coaches expectations:
Strong interest in working with children
Be responsible
Accountable for behavior and actions
Strong work ethic (contact site coord if late or miss)
Clear procedure in place for disciplanary action (verbal, written warnings leading to dismissal)
Compensation: 3 options
Hourly pay $6.13, paid monthly
College tuition based on accumulated hrs worked, plus a 50% match, directly sent to college where tutor registered. Tuition must be used within 1 yr HS graduation.
School credit for “service learning” or school-to-work course requirements.
Additionally, ALL coaches who have taught for at least 1 yr get the opportunity to receive a 4 yr scholarship of $2,500/yr. One recipient of the John Stanford scholarship is chosen every year. The runners-up get a smaller scholarship.
Coaches approx work 1112 hrs during a school yr.
Readers
Schools are 1st selected based on their willingness to participate in the pgm
Each school identifies 25 2nd, 3rd, 4th and 5th graders reading below grade level
Student Intervention Teams (SIT) select students or site coordinators approach teachers to identify students.
Eligibility based on need: student within bottom 25% of district reading test score, teacher recommendation based on student performance, possible retention due to failure to meet reading standards
80-90%receive “free and reduced lunches” = most live in poverty, ethnic minorities mainly Asians and African-Americans, over 2 dozen languages represented in the pgm.

7. Who is Team Read? Site Coordinators Volunteers
Part-time, teachers or school staff
Meet informally with Tricia McKay for feedback
Volunteers
Needed due to amount of work
Partnership with VISTA/Americorps and U. of Washington Pipeline
Assist site coordinators
Site coordinators
Full time teachers or staff at school. Only work part-time
Opportunity to discuss issues or propose improvement to program magr.
Volunteers
Two agencies provided Team Read with 7 full time volunteer assistants by June 1999, one in each school.
Assist site coord. With a variety of tasks from transport logistics to working with coaches.

8. Team Read Session Two 1 hr session per week during school year
Standardized session format and structure
3.00 PM: coaches arrive
3.15 PM: snack with reader
3.30 pm: reading session begins
4.15 PM: reading session ends/ coaches fill out paperwork
4.30 PM: reading coaches leave
Opportunity for site coordinators to be creative
Overall very well planned sessions and preparation for the coaches. For ex, there are training sessions for reading coaches and additional training and guidance is provided throughout the school year
Mon-Wed or Tues-Thu depending on the school
Some site coordinators show creativity. For ex, Leschi site coord. color-coded books according to reading difficulty level.

9. Team Read Evaluation Results
1st year evaluation: not as positive as expected
2nd year: promising but Team Read’s impact not as promising as hoped
Tricia McKay disappointed by results and wonders if findings provide her with the information needed for improvements
Questions about the accuracy of results.
1st yr eval: strong evidence of benefit to the coaches, impact on readers was ambiguous. Only 5ht graders showed clear signs of reading improvement compared to other gp. 2nd and 3rd graders tests performance was lower than children not participating in the pgm. No 4th grade results available

10. Team Read Evaluations
Evaluation conducted by Margo Jones, statistician by trade.
17 schools in the program by June 2000 but only 10 used in the evaluation.
3 main objectives to show Team Read’s Impact:
Goal#1: Do the reading skills of the student readers improve significantly during their participation in the Team Read program?
Goal#2: How does the program affect the reading coaches?
Goal#3 What is working well, and what can be improved?
Margo Jones performed the eval the previous year as well. She was hired by Team Read as an independent evaluator.
10 schools were only chosen because they are in their 2nd year. The other 7 were added to the pgm later and less data is available. These 10 schools were the same one used for the 1st year evaluation

11. Evaluation Methods Goal #1
Goal #1: Reading skills. Two-fold approach
Grade
Pre-Test
Post-Test
Post-Test Metric 2
DRA Fall 1999
DRA Spring 2000
Test Level 3
DRA Spring 1999*
ITBS Spring 2000*
NCE 4
ITBS Spring 1999
WASL Spring 2000
Scale Score 5
WASL Spring 1999
ITBS Spring 2000
Jones uses a 2-fold approach to establish the improvement of reading skills. 1st approach AS SEEN HERE is to compare reading scores across the board. The pre-test is the one taken before being in the Team Read program while the post-test takes place about ¾ into the school year while students are enrolled in Team Read.
DRA = Developmental Reading Assessment;
ITBS = Iowa Test of Basic Skills;
WASL = Washington Assessment of Student Learning; and
NCE = Normal Curve Equivalent.
*Note in case…Note 3: 3rd grade tests are very different and may not be not comparable, because they measure different skills; tests for 4th and 5th graders are arguably statistically correlated, although they may measure different
skills.

12. Apples and oranges Grade Pre-Test Post-Test Post-Test Metric 2 3 NCE 45

13. Evaluation Methods Goal #1, Cont’d
2nd method: determine proportion of Team Read participants who moved from below grade level test score to at-or above level compared to all students from entire district
Note: one of the selection criteria to be in Team Read is to be within bottom 25% of district reading test score
Again, Jones is not comparing apples to apples. She is comparing the entire district (wide range of reading skills) to the TR kids who are in the bottom 25% of the district reading test score.
She should compare similar population

14. Evaluation Methods Goal #1
Both methods for goal 1 are flawed.
Note from Jones in report confirming problems with different tests.
“The question then becomes ‘Is it reasonable to assume that the pre- and post-tests measure the same skills?’ The evaluator adopted a correlation criterion for answering this question. If the correlation between pre- and post-tests was near or above .8 (see footnote 5) (as was the case for the 2nd grade, where pre- and post-tests were essentially the same), the pre- to post-test change score was interpreted as a gain score. If the correlation coefficient was much less than .8, no analysis was performed.”

15. Evaluation Methods, Goals #1- Assessment
Visible problems with tests:
Pre and post-tests do not measure the same skills except in 2nd grade
All Team Read participants are within the bottom quartile of the reading test scores and can’t be compared with students district-wide.
Analysis could not be carried out for some grades (3rd) due to incompatibility of reading tests

16. Evaluation Method, Goal #2
Method: questionnaire given to coaches during their last week of coaching.
Goal: to assess the impact of Team Read on coaches.
Identify 3 main areas of job satisfaction:
How positive is their experience with TR?
Extent to which the coaches feel that TR helps students readers
How supportive is TR’s site coordinators and staff?

17. Evaluation Method, Goal #2 - Assessment
Jones results showed that coaches were satisfied with their job and felt a sense of pride and accomplishment.
Test performed by Jones was successful in meeting its goals, unlike in Goal #1.
Some questions provided answers as to what to improve in the program.
Test is useful to the program manager

18. Evaluation Method, Goal #3
Method: interviews with major stakeholders, one site visit, review of last 5 years’ research literature on cross-age tutoring and last 3 years research on best practices.
Goal: assess the overall progress of the program.
Stakeholders: Tricia McKay, pgm mgr; Joan Dore: pgm reading specialist; site coordinators: Cathy Thompson and Susan Woodruff, a sample of volunteer assistants.
Site visit spent shadowing reading specialist and observing a reading session

19. After the Evaluation
Margo Jones makes her recommendations based on her findings to improve the efficacy of Team Read.
Tricia McKay reads the evaluation but fail to see what the results are telling her about the program

20. Evaluation Methods - Summary
Goals
Methodology
Reading improvement
among elementary
students
Analyzed standardized test scores for 10 of 17 Team Read schools.
Rate of increase on pre- and post- tests.
Percent of students moving from below reading level to at/above grade level.
Collected data by grade level and by school.
Compared Team Read results to district scores.
Coach development
Surveyed coaches by questionnaire
Program implementation
Interviewed stakeholders
Observed Team Read session
Reviewed other studies

21. Evaluation Problems Academic v. Practitioner approach.
Result presentation.
Research design flaws.
Not all the data was analyzed
Comparison with dissimilar items
Statistical analysis stretched
Evaluation only covers 3 goals.
Lack of communication between the evaluator and the program manager.
Evaluation is too technical and loses sight of whom the report is for. Jones uses a statistical approach to the program which is inadequate and somewhat unusable for McKay, as a pgm manager.
Results are not clear cut because items compared are not similar. Results were not presented to McKay in a way that is easy to understand. Picture provided by the evaluation is incomplete. The results do not provide Tricia McKay with clear answers or tools to improve problems.
Methods used do not match with the goals. Some of the data is not analyzed because no match can be established or because more recent data is not used in eval for yr 2.
Items compared are not similar which forces Jones to stretch her statistical analysis. Numbers become inaccurate.
Evaluation based only on 3 goals to represent the entire program. This seems insufficient to show how the program is performing.
No mention is the case that Jones and McKay discussed what they each expected to gain from the evaluation. The case clearly shows that McKay was eager to get the results from the evaluation to improve her program but the results fail to provide her with the answers she needs to do so.

22. What Could Have Been Done Better?

23. Impact: 2005-2006 Reading Results
From Team Read’s Annual Report.

24. Team Read Update
Program Evaluation Results for School Year   
70% of 2nd graders and 52% of 3rd graders were reading at/above or approaching grade level
54% of 2nd graders and 35% of 3rd graders gained greater than 1.5 grade levels in reading
98% of the parents of 2nd & 3rd graders and 92% of their teachers reported increased reading skills as a result of participation in Team Read
76% of 2nd & 3rd graders said that reading was more fun since joining Team Read