Presentation on theme: "Arizona State University"— Presentation transcript:
1 Arizona State University Inter-Rater and Test-Retest Reliability of the Early Language and Literacy Classroom Observation (ELLCO)Addie LaffertyKhawla ObeidatShelley GrayJeanne WilcoxArizona State UniversityTempe, Arizona
2 What is the ELLCO? A set of three interdependent research tools: Literacy Environment ChecklistClassroom Observation and Teacher InterviewLiteracy Activities Rating ScaleDeveloped by Miriam W. Smith & David K. Dickinson, with Angela Sangeorge, & Louisa Anastasopoulos (2002)
3 What is the purpose of the ELLCO? To identify practices and environmental supports that foster early language and literacy development.Provides objective ratings of the language & literacy environment of classrooms from preschool to third grade.
4 Literacy Environment Checklist Twenty four items in 5 conceptual categories:Book AreaBook SelectionBook UseWriting MaterialsWriting Around The Room
5 Classroom Observation and Teacher Interview Fourteen classroom observation items grouped into:General Classroom EnvironmentLanguage & Literacy CurriculumRated on a 5-point scale5=exemplary4=proficient3=basic2=limited1=deficient
6 Literacy Activities Rating Scale Nine questions in 2 categories:Book readingWriting
7 Method Purpose: Procedure: Assess inter-rater and test-retest reliabilityProcedure:Administered by two certified speech-language pathologists at the same timeOne was a bilingual Spanish-English speakerOne was a monolingual English speakerTwo to three week interval between first and second administrations
8 Characteristics of Participants 8 Head Start classroomsThree classes of 3- to 4-year-oldsFive classes of 4- to 5-year-olds135 children115 children identified as English Language LearnersTeachers & children spoke Spanish and English throughout the four hour dayTeacher demographics: 3 Latino, 3 White, 1 African American, 1 American Indian
10 Discussion The ELLCO had good inter-rater reliability. The Classroom Observation & Teacher Interview showed good test-retest reliability.The Literacy Environment Checklist showed adequate test-retest reliability. Scores were affected by the presence or absence of classroom materials from Time 1 to Time 2The Literacy Activities Rating Scale showed poor test-retest reliability. Scores were affected by the presence or absence of book reading and writing activities from Time 1 to Time 2.
11 Test-Re-test Reliability Concerns with the Literacy Activities Rating Scale The observations at test and re-test were scheduled at the same time of the day and when teachers reported that book reading and writing activities were scheduled to take place.However, the reliability coefficient was affected by scores on the presence of full-group book reading, number of minutes spent on book-reading, observing an adult model writing, and observing an adult help a child write.The observation of these behaviors at one time point and not another was the basis of the poor test-retest reliability found here.
12 Considerations When Using the ELLCO in Research Consider performing a minimum of two pre-intervention and two post-intervention observations to determine if the scores on the ELLCO are stable and truly representative of the classroom.Predetermine an amount of time to spend on observations to allow for consistent administration.When measuring change in a classroom, schedule the observations at the same time as the first observation and when the teacher reports book reading and writing activities will take place to maximize the possibility of observing assessed behaviors.
13 Suggestions for Future Research Determination of more controlled directions for performing ELLCO observation.Consideration of how many observations in a given classroom are needed before stability of scores is obtained.