The Evaluation of West Virginias Enhancing Education Through Technology Model Schools Program (PR#: S318A030014) Dale Mann, Ph.D. Jonathan D. Becker, J.D., Ph.D. Charol Shakeshaft, Ph.D. Interactive, Inc. September 19, 2005

The TMS Study n Research Questions: –Over this school year, how much change in the use/integration of computers (by 4 th and 5 th grade teachers) was there? –After the TIS specialists are done (June 2005), what will happen? Will teachers continue to use technology? (Year 2) n Using technology to study technology n Experimental/control + pre-post n 4-cell sampling framework (treatment/control x digital/non-digital)

Preview of Conclusions n Methods n Technology Integration Specialists (PD) n TMS program schools & teachers n TMS students n TMS program overall

Conclusions about Methods n Teachers and students cooperated with the use of technology to study technology. n The (classroom) desktop metering and the pager-triggered web-surveys worked reliably. n The digital data collection procedures did not affect classroom practice.

Conclusions about the TISs n The TIS services were within the boundaries of the TMS program. n But, rather than a single, consistent program, the variation in the TIS backgrounds and work arrangements delivered several program interventions, not a single intervention. n TISs did not become more proficient or more enthusiastic about technology over the year. n Over the course of the first year, TISs learned how to do their job and created helping relations with classroom teachers. They provided less direct instruction to their in-school colleagues, but built the necessary foundations for increased technology integration.

Conclusions about TMS Schools and Teachers n TMS teachers had more technology-related PD than others. n TMS teachers were twice as likely to be using computers more for their own work than the control group (web-survey data). n TMS teachers reported more student use of technology and more consistent use over the school year than the control group (web-survey data) n TMS teachers report a broader array of student- related applications than the reading-heavy focus of the control group (web-survey + TAM data)

Conclusions about TMS Schools and Teachers (contd) n TMS teachers reported more positive changes in more areas than the control group. n All teachers (TMS & control) were more likely to direct student use of computers if they had outside PD, one-to-one help, release time for tech. PD, and access to online PD – all potential features of the TMS program. n There are a number of areas of teacher proficiencies, attitudes and behaviors that did not change and that may be part of a program revision.

Conclusions about TMS students n Students in TMS schools consistently report using computers more frequently than the control group (web-survey data). n Students in TMS schools report more use of the Internet to do reading or research than the control group (web- survey data).

Conclusions about the TMS program overall n This is the first study to analyze more than self-report data about technology use – metered and random- interval web-survey data from teachers and students. Additionally, we do not yet have student achievement data. Therefore, we cannot conclude that the reported patterns are high, exemplary, successful or the opposite. n For the same reasons, we have not made policy or practice conclusions. n Unless the cadre of TISs can be staffed with people who are already expert with technology and with training adults, the first year of a turnkey sequence is more of a learning year for the TISs than a service- delivery year for teachers in schools.

Methods n We will follow 4 th and 5 th grade teachers who, last year, had TIS assistance. n Some will continue to have TIS services and that contaminates the turnkey logic. n Some 4 th or 5 th grade teachers will move to similar assignments in other schools; we will not be able to follow them. n We need to document that a Year One trained teacher follows through and tries to communicate new capability to colleagues. n Attrition in both treatment and control teachers. n Continued use of technology to study technology.

An Unanticipated Question n Following the TISs who work a second year in the same school (or new ones?): Are their second year services more aggressive, direct, mature? n That is central to the turnkey strategy but we did not know the issue would arise. n Similarly, dependence vs. independence: providing a service vs. building capacity.

Data Collection: USING TECHNOLOGY TO MEASURE TECHNOLOGY

TREATMENTCONTROL (Blue Team) DIGITAL Meters + Pagers (w/ Web surveys) Cheat Lake ES (Mon) Mason Dixon ES (Mon) Clay ES (Clay) Ellenboro ES (Ritchie) Harrisville ES (Ritchie) Doddridge ES (Doddridge) Burnsville ES (Braxton) Frametown ES (Braxton) Fairdale ES (Raleigh) New Haven ES (Mason) Fairplains ES (Wood) Madison ES (Wood) Franklin EC (Wood) Wilsonburg ES (Harrison) Woodsdale ES (Ohio) Riverside ES (Mon) Waverly ES (Wood) Kanawha ES (Wood) Neale ES (Wood) Valley Grove ES (Kanawha) Bridge ES (Kanawha) Clear Fork ES (Raleigh) North Pt. Pleasant ES (Mason) Worthington ES (Wood) Gihon ES (Wood) Lubeck ES (Wood) Van Horn ES (Harrison) Middle Creek ES (Ohio) NON-DIGITAL Pre + Post Paper/Pencil surveys only Cedar Grove Community (Kanawha) Tennerton ES (Upshur) Welch ES (McDowell) Smithville ES (Ritchie) Creed Collins ES (Ritchie) Flatwoods ES (Braxton) Burnsville ES (Braxton) Stratton ES (Raleigh) Beckley ES (Raleigh) Mason ES (Mason) Jefferson ES (Wood) Blennerhassett ES (Wood) Martin ES (Wood) Tucker County ES (Tucker) Richmond ES (Kanawha) French Creek ES (Upshur) Fall River ES (McDowell) Emerson ES (Wood) Chamberlain ES (Kanawha) Valley Grove ES (Braxton) Beale ES (Mason) Mineral Wells ES (Wood) Williamstown ES (Wood) McKinley ES (Wood) Davis Thomas ES (Tucker)

USING TECHNOLOGY TO MEASURE TECHNOLOGY n Meters in desktops –Software runs in the background 24/7 –Records file activity only –Installed on classroom computers only –Data ed directly to Interactive, Inc. –ZERO burden on teachers n Pagers on teachers + web surveys –Free pager rings once per day, every other week –When pager rings, or as soon as convenient thereafter, teacher and one randomly selected student take short web-survey –Pages are randomly scheduled during alternate weeks (not more than 5 per week)

Pagers + Web-Surveys n From November 22, 2004 through April , the pagers were activated 41 times. n In total, Interactive, Inc. received 1,563 usable pager-triggered web survey responses from 91 different teachers n The range in the number of responses from individual teachers is one to 40 (i.e. a couple of teachers have responded once, and one teacher responded to 40 pages). n We also received 1,311 usable pager-triggered web survey responses from students in 78 different classrooms.

TrueActive Monitors n Starting on October 1, 2004, TAM was installed on approximately 287 computers. n Interactive, Inc. ultimately received data at one point or another from 205 computers in 24 (of 26) schools. n As of April 1, Interactive, Inc. received over 33,000 s with data from TAM including over 550,000 files activated.

OTHER DATA n Technology Integration Specialists (TIS) –Each of the 18 technology integration specialists completed a paper-based questionnaire at the beginning of the school year; and 16 completed an end-of-year online QRE. –Over 42 weeks, Interactive, Inc. received right around 600 activity logs from the TISs. n Teachers –Baseline questionnaires were sent to 227 teachers; they were returned by 199 teachers, for a response rate of 88%. –A similar response rate was achieved for the end-of- year QRE.

Preliminary Findings

Documenting the Treatment: Who are the TISs? 10 were classroom-based elementary school teachers and 2 were secondary school teachers 3 were EETT-funded technology integration specialists (serving different schools than in AY 04-05) 2 were Title I literacy specialists and 1 was a librarian

Documenting the Treatment: A Range of Availability n ONLY 5 OF THE 18 TISs were contracted to work full-time (designated as at least 35 hours per week) in a single school n The other TISs were all part-time in one or more schools; The range of arrangements is important. n Most of the TISs who worked less than full- time as a TIS worked in another capacity within one of the schools they served as a TIS. For example, one TIS was a ½ time TIS and ½ time librarian in the same school.

Documenting the Treatment: What did they do? n During the early part of the school year, TISs devoted much of their time to setting up computers, installing software and trouble shooting. Group sessions were used to introduce software to teachers. n The rest of the year was spent doing mostly small-group work with teachers and individual mentoring/training. n The majority of the professional development occurred during school hours, not before or after school.

Web-survey Responses

At the time the pager was activated, were students using computers? (% of Yes Responses) TeachersStudents Treatment Control

Teacher responses over time

Student responses over time

At the time the pager was activated, were YOU (the teacher) using computers? % of Yes Responses Treatment (weighted n=933) 22.2 Control (weighted n=630) 9.2

What were you (the teacher) doing? TMS %Control % Word Processing PowerPoint 3.40 Spreadsheets MS Publisher 1.00 Kidspiration 03.5 Inspiration 00 Compass / Riverdeep Internet research Other

Summary n When asked at random times throughout the school year if they were using computers themselves, teachers who had been working with Technology Integration Specialists were much more likely to report that they were doing so (22% vs. 9%). n Furthermore, when using computers, those teachers were much more likely to be doing curriculum-related work (i.e. productivity software). n Students in schools where there was a TIS were only slightly (but statistically significantly) more likely to be using computers.

Teacher QRE Data

Time on Computers (labs) n SCALE: 1= not at all; 2= <15 minutes; 3= 15-45; 4= 46-90; 5=90+ Teacher End-of-Year Reports of Student Computer Use In Labs: Mean Scores by TMS vs Control and Type of Data Collection TMS SchoolsControl Schools Digital Data Collection Non-Digital Data Collection Digital Data Collection Non-Digital Data Collection

Time on Computers (classroom) n SCALE: 1= not at all; 2= <15 minutes; 3= 15-45; 4= 46-90; 5=90+ Teacher End-of-Year Reports of Student Computer Use In Labs: Mean Scores by TMS vs Control and Type of Data Collection TMS SchoolsControl Schools Digital Data Collection Non-Digital Data Collection Digital Data Collection Non-Digital Data Collection

Next September, I will be better able to use technology in my classroom than I was last September n Scale: 1= strongly agree to 4= strongly disagree TMS (digital) TMS (non-digital) Control (digital) Control (non-digital)

Next Sets of Analyses n TrueActive meter data n Technology use/integration as a function of school and TIS characteristics (regression analysis and multilevel modeling) n STUDENT ACHIEVEMENT (WESTEST) ANALYSIS (Sept. 05)

QUESTIONS & ANSWERS

Contact Us Interactive, Inc. 61 Green Street Huntington, New York phone: fax: