1. © CCSR 5Essentials: Background and methodology Rachel Levenstein Senior Manager for Survey Research rmlev@uchicago.edu

2. © CCSR  20 years of research in Chicago Public Schools (CPS)  Conversations with practitioners  Wide scan of the literature  Surveys  Organizing Schools for Improvement Origin of the 5Essentials

3. © CCSR Surveys of CPS

4. © CCSR Initial measure development process conducted in CPS Prior research findings indicate a link between construct and student outcomes 1.Search existing literature for survey questions 2.Focus groups  Draft questions 3.Cognitive interviews  Revise questions 4.Pilot survey questions  Check reliability, revise questions Validation study

5. © CCSR Development of survey questions: Focus groups Gather 5-10 practitioners or students together to talk about the concept. Moderated by a researcher, using a protocol. Identify themes and common nomenclature.  Get ideas for how to write survey questions

6. © CCSR Development of survey questions: Cognitive Interviews One-on-one interviews with practitioners, students “Tell me how you came to your answer” “What kinds of programs were you thinking about?”  Identify problems understanding or interpreting the questions.

7. © CCSR Development of survey questions: Pilot testing Small- or large-scale pilot Same conditions as the “regular” survey –May be conducted as part of the “regular” survey Analyze responses and calculate reliability.  Identify if the measure holds together (or not).

8. © CCSR Development of survey questions: Validation study Program Coherence example: Newman et al., 2001 –Qualitative data collection: (School improvement plans, classroom observations, staff meeting observations, assignment samples) Reports highly correlated with survey measure –Schools that improved on Program Coherence between 1994 and 1997 showed more improvement on ITBS than schools that did not improve—by 1/5 of a school year!

9. © CCSR  Practices change  Understanding of constructs change  General revisions for clarity  Clients outside of Chicago inform some changes  May add, delete, or modify questions every few years. Changes to content

10. © CCSR  To what extent do you agree or disagree with each of the following statements?  You can see real continuity from one program to another in this school.  Many special programs come and go in this school.  Once we start a new program, we follow up to make sure that it’s working.  We have so many different programs in this school that I can’t keep track of them all.  (Strongly disagree, disagree, agree, strongly agree) Changes to content: Program Coherence

11. © CCSR  To what extent do you agree or disagree with each of the following statements?  You can see real continuity from one program to another in this school.  Once we start a new program, we follow up to make sure that it’s working.  Many special programs come and go in this school.  Once we start a new program, we follow up to make sure that it’s working.  We have so many different programs in this school that I can’t keep track of them all.  (Strongly disagree, disagree, agree, strongly agree) Changes to content: Program Coherence

12. © CCSR  How much influence do teachers have over school policy in each of the areas below? -Hiring new professional personnel. -Planning how discretionary school funds should be used. -Determining books and other instructional materials used in classrooms. -Establishing the curriculum and instructional programs. -Setting standards for student behavior. (Not at all, A little, Some, To a great extent) Changes to content: Teacher Influence

13. © CCSR  How much influence do teachers have over school policy in each of the areas below? -Hiring new professional personnel. -Planning how discretionary school funds should be used. -Determining books and other instructional materials used in classrooms. -Establishing the curriculum and instructional programs. -Setting standards for student behavior. (Not at all, A little, Some, To a great extent) (No influence, A little influence, Some influence, A great deal of influence) Changes to content: Teacher Influence

14. © CCSR Putting a measure into the 5Essentials Two years of data before it becomes an Essential Fit within 5Essentials framework –Factor analysis