Technology Assisted Reading Assessment Exploring Assistive Technology Issues in Reading Assessment ATIA 2010 Orlando: Jason Altman, NCEO

Technology Assisted Reading Assessment Statement of Problem Little evidence that technology improves validity of tests for students w/ VI One reason is that students experience and exposure to AT varies TARA seeks to develop a test of student proficiency with AT For use prior to using AT on large-scale reading assessments

Technology Assisted Reading Assessment Preliminary Studies: The State of AT Use with VI Students Study #1: TVI survey (presented at ATIA 09) Study #2: TVI follow-up interview Study #3: Student Observational Interview

Technology Assisted Reading Assessment Revisiting Teacher Survey AT use is a large part of instruction for students with VI TVIs take a blended approach to teaching reading (e.g., using a variety of modalities) Standardization of assessments may be a challenge because students use different technology for different purposes

Technology Assisted Reading Assessment More on Teacher Survey Reading and technology are intertwined activities for students with VI The larger the caseload, the less likely students will have individualized AT experiences Teachers who focus on foundational reading skills tend to provide fewer technology opportunities for students.

Technology Assisted Reading Assessment Overview: Results of Teacher Interview Study (#2) 1 st study showed accommodations or modifications are common for students w/ VI Both high- and low-tech approaches to teaching reading are used by TVIs –Most likely for different purposes and at different times –Such flexible approaches to reading and situational use of technologies create challenges for standardization of assessments

Technology Assisted Reading Assessment Method Interviewed 27 TVIs At least two TVIs were interviewed in the following categories: –TVIs who work in Schools for the Blind –TVIs who are itinerant teachers –TVIs who teach students with low-vision –TVIs who teach students who are completely blind –TVIs who do not teach braille –TVIs who teach braille

Technology Assisted Reading Assessment Instrument and Procedures All interviews were conducted by telephone In most cases, there were two (and up to four) researchers present during telephone interviews Research participants were asked to answer a series of structured interview questions Interviews generally lasted 30 minutes, although some interviews lasted as long as one hour

Technology Assisted Reading Assessment Results All 27 TVIs indicated that they would use a technology-based assessment if one were readily available All TVIs who taught students with low-vision used at least some form of magnification equipment TVIs who taught students who use braille readers also reported using a variety of technologies such as audio devices and braille note takers

Technology Assisted Reading Assessment Specific Products Alpha Smart Bookport Braille and Speak DAISY Readers Extreme Reader Freedom Box IPod JAWS Kurzweil MAGIC OpenBook OutSpoken PacMate Zoomtext

Technology Assisted Reading Assessment Technology and reading Some TVIs used technology fluidly in instruction Those who had wide-ranging knowledge of the tools available (and had access to such tools) match instructional tasks with technology tools. 5 TVIs spent most of their time teaching students how to use CCTVs and were hesitant to engage in teaching students using computer-based tools (especially those that added voicing to files) In all cases, however, TVIs used some form of technology to assist teaching

Technology Assisted Reading Assessment Formal AT assessment protocols About 1 of 4 teachers were using formal assessment protocols/tools to assess students assistive technology use Among those most frequently used are the SETT Protocol and the Texas School for the Blind Assistive Technology Assessment Summary for Students with Visual Impairments One teacher who used the Texas assessment also mentioned using the Georgia Project for Assistive Technologys (n.d.) Assistive Technology Evaluation and the Wisconsin Assistive Technology Initiatives (2004) WATI Assistive Technology Checklist

Technology Assisted Reading Assessment Assessment: Proficient Reader Who Uses AT Teachers recognized that advanced proficiency in technology-assisted reading meant the ability to move fluidly between technologies in order to optimize the potential of various technologies A proficient user of technology-assisted reading would be a reader who read and understood grade level material –Most TVIs were concerned that their students be assessed on grade-level, but with a fair opportunity to demonstrate grade-level skills –TVIs were concerned about the validity of tests

Technology Assisted Reading Assessment Discussion There is a desire for a formal assessment or evaluation of student skills Should discriminate between various levels of proficiency There are a small number of assessments/frameworks in use

Technology Assisted Reading Assessment Activity TVIs in this room: Had you been participants would have findings changed

Technology Assisted Reading Assessment Student Observational Interview: Study #3

Technology Assisted Reading Assessment Method: Sample Students with visual impairments in grades In total, we interviewed 18 students for this study from 5 states across the US We sampled students from both general education school systems (n=9) and state schools (n=5) for the blind Four additional students were educated at state schools through general education classes in a nearby public school

Technology Assisted Reading Assessment Method: Instrument and Procedures Students participated in observational interviews facilitated by three researchers on the project –Observational interviews were a hybrid between verbal interviews, where respondents describe phenomenon (Bogdan & Biklen, 1998) and cognitive interviews, where interviewees participate in an activity and describe their thoughts and actions (Ericsson & Simon, 1993)

Technology Assisted Reading Assessment Procedures (cont.) During these interviews, students were asked to complete reading tasks using AT –Afterward, students were asked how they use AT in the reading process, including how to download files, retrieve information from printed material, and to explain preferences Each interview lasted between 30 and 60 minutes

Technology Assisted Reading Assessment Results of Student Study Demands of a rigorous high school curriculum at times left students without enough time to explore and become proficient in new and possibly more efficient technology strategies For students with visual impairments reading large print or Braille, the time it takes to complete a task is often far greater than that for their peers with full functioning vision

Technology Assisted Reading Assessment Results 11 of the 18 students could read large print – 3 of them regular print as well, and many were audio book users. The remaining 7 students all read braille – and 3 of the large print readers also read braille. 13 students used audio books to access print no matter what their primary method of print reading was

Technology Assisted Reading Assessment Results 8 students used technology for reading Braille – 5 used Braille Note For magnification, students used a variety of technologies, from simple handheld magnifiers to computer-based products Zoomtext (most often also with speech), CCTV Technology use for 7 depended on the reading situation

Technology Assisted Reading Assessment Results Technologies are sometimes unreliable (e.g., not all formats work for internet-based text), take time to learn, and do not always accompany the student beyond the school walls (e.g., many students cannot bring AT devices home to assist with homework)

Technology Assisted Reading Assessment Student characteristics and independence Students varied from completely relying on help from sighted teachers, para-professionals, and peers to being assertively independent Most students described their dependence on others as situational One student mentioned that he is not afraid to ask his friends or family to read things for him Another student, said that help is not needed

Technology Assisted Reading Assessment Technology Choices Depended on availability of technology in student location For example, students at a state school typically would have more access to technology than a student in a rural public school that did not serve a large population of students with visual impairments Access at home may differ from access at school –One student, who was on the academic honor roll at school, often stayed after school and came in before school every morning to finish homework

Technology Assisted Reading Assessment Choice of specific technology The students TVI or para-professional generally was very involved in the decision Some students attended technology conferences It is important to note that another person (teacher, parent, or para-professional) needed to learn the technology along with the student TVIs had to be careful not to over-extend themselves with multiple new technologies

Technology Assisted Reading Assessment Role of expected change in situation Decisions were impacted by the nature of the visual impairments Changes in student vision and forecasts for future change often called for a change in products –For example, one student used a CCTV up until 6 th grade but stopped using it because there was a decrease in his field of vision and it was no longer practical

Technology Assisted Reading Assessment Difficulty with AT The amount of time that must be dedicated to training and practice in using the equipment sometimes outweighed the potential benefit of the technology. Another complication was consistent access to technology Students were sometimes forced to share technology with several other students, including CCTVs

Technology Assisted Reading Assessment Other Hangups There was also difficulty in transporting, setting up, and stowing equipment Upon learning a particular technology platform, students were hesitant to move to other platforms Equipment malfunction, or general hang-ups and glitches also were sometimes troublesome for students, teachers, and paraprofessionals alike. One student stated that the Web sites that contained many graphics can be most problematic

Technology Assisted Reading Assessment Testing and Large-Scale Assessment Multiple students reported using enlarged text to access the items and then having the paraprofessional fill in the bubble answer sheet At least one student was able to take a math test using a computer; however, he was required to take the reading test using a paper and pencil version Other students used standard question and answer forms, accessing them via CCTVs

Technology Assisted Reading Assessment Discussion What do these findings mean for TVIs? Does the situation play out similar in your classroom? Would you agree that your students participation in large-scale assessment could be more meaningful?

Technology Assisted Reading Assessment Conclusions As statewide assessment policy is still in flux (Thurlow et al., 2006), it is important for research to continue in the area of technology-assisted reading to provide stakeholders with a clearer picture of the avenues that students with visual impairments take to access reading and demonstrate their reading skills

Technology Assisted Reading Assessment Also at this Conference Also see today from 2:45-3:45 Bonaire 8 EDU-15 Development of an Accountability Test to Measure a Students Ability to Access Text with AT Presenting: Laitusis, Hall

Technology Assisted Reading Assessment Next Steps Field Testing the new Technology Based Reading Assessment in school classrooms We need your help! –Administrators of the field test –Students who can participate

Technology Assisted Reading Assessment Thank you! For more information… National Center on Educational Outcomes University of Minnesota http:// Jason Altman 35