Presentation on theme: "Scalable and Sustainable Technologies for Reading Instruction"— Presentation transcript:
1 Scalable and Sustainable Technologies for Reading Instruction An Interagency Education Research Initiative: NSF - IES - NIHPrincipal Investigators:Walter Kintsch, Donna Caccamise, Ron Cole, Lynn Snyder, & Richard Olson
2 The Push to Improve Literacy Every child can learn to read, yetLiteracy is a national problemWe have good basic research, problem isTo develop practical implementationsTo get them used widely in schoolsIERI initiative
3 Colorado Literacy Tutor Tools for TeachersK-6th grade:emphasis on foundational reading skills: Interactive books for practice plus tutors for special problems6th-14th grade:emphasis on comprehension and learning from text
4 Underlying Cognitive Science Research Tutoring is an effective method of instruction:Human tutors produce gains between .4 and 2.3 standard deviations over classroom teachingMost human tutors have only modest domain knowledge and rarely use sophisticated tutoring strategiesExisting Artificial Intelligence tutors such as AUTOTUTOR (Graesser et a., 1999) and ATLAS (van Lehn et al., 2000) approach the effectiveness of human tutors (Graesser et al., 2001)
5 Underlying Cognitive Science Research Creating the Next Generation of Intelligent Animated Conversational Agents:Use of synthesized speech and speech recognitionConversation with a talking headLearners respond in English in addition to click-and-point
6 Underlying Cognitive Science Research Latent Semantic Analysis: Theory, Technology and PracticeLSA as a map of meaningHigh-dimensional semantic spaceText meaning as the sum of the wordsLimitations:Learns only from wordsNeglect of order and syntax
7 A map is a 2-dimensional representation of distance and direction data: Data: Map:
8 A semantic space is a k-dimensional representation of word co-occurrence data (k<<<m,n): Data: Semantic Space:
9 High-dimensional spaces are capable of representing complex semantic relations: TREEDOG.70BARK.06.66
10 The meaning of a text is the sum of the words in LSA: The accident victim was flown to the hospitalThis sentence is more similar to (1) than (2), although it shares two content words with (2) and none with (1):(1) A helicopter brought the man hurt in the crash to the emergency room cosine = .29(2) An accident occurred in the street where the victim could not get assistancecosine = .24
11 Interactive Books & Reading Tutors Ron Cole, Donna Caccamise, Lynn Snyder and the Colorado Literacy Tutor TeamCenter for the Study of Spoken LanguageUniversity Of Colorado
12 Components of the Reading Tutor: Foundational Skills TutorsTeach underlying reading skillsInteractive BooksTeach fluent reading & comprehensionManaged Learning EnvironmentEnroll students, track and display progress, manage individual study plans, etc.
17 Interactive Books Teacher or child selects appropriate book Child selects animated tutor or “coach”Animated coach guides child through bookCoach can read words, sentences paragraphs or whole pageCoach asks comprehension, inference and summary questions using speech recognition to determine accuracy of responsesEach level of a book has the capability of invoking specific reading tutors if a child needs more foundational skill training
22 Comprehension and Learning Walter Kintsch, Donna Caccamise, Lynn Snyder, and the LSA Research GroupInstitute of Cognitive ScienceUniversity of ColoradoTom LandauerPearson Knowledge Technologies & University of Colorado
23 Comprehension and Learning LSA computes how similar in meaning two texts areThe contents of a student’s essay can be compared with other essays and standardsStudents receive feedback about the content of their writing - guidance for revisingA tool for self-assessment: feedback allows students to judge how well they are doing and what needs more work
24 Summary Street®An existing, classroom-tested tested system that provides content-based feedback to middle-school students summarizing a text:indicates what content is missingwhat might be overemphasizedflags apparent problems sentenceshelps with the organization of the material
31 Provides hints about how the summary could be shortened: Sentences are flagged that are very similar in meaning:…...They also wrote books on paper. The books were made from bark paper that they folded together…..Sentences that appear unrelated to the topic are questioned:…..We also learned about the Incas…..
33 What makes Summary Street effective? Student Interviews: The computer is sometimes wrongStudents are never told what to doProblems are identified and hints are provided about how to solve themStudents always make their own decisionsAnonymity of the computerIt is not threatening to be corrected by a sub machine!
34 How effective is Summary Street? David Wade-Stein & Eileen Kintsch (2004) Summary Street: Interactive computer support for writing. Discourse Processes, 22,6th-graders write summaries, one with Summary Street, one on a word processor that provides only length and spelling feedback
35 Students working with Summary Street spend much more time on task:
36 Summaries written with Summary Street receive better teacher grades:
37 Of the 10 texts used in this study, the biggest improvements were observed for the most difficult textsThe best students did not need Summary Street; for the poorest students, Summary Street did not provide enough support; the middle group profited the most.
38 How effective is Summary Street? Marita Franzke, Eileen Kintsch, Donna Caccamise, Nina Johnson, & Scott Dooley (submitted) Summary Street®:Computer Support for Comprehension and Writing.8th-graders practice summary writing for four weeks, with and without Summary Street; four classrooms, same teacher
39 Teacher grades assigned to summaries written with and without Summary Street over the 4-week study period;texts get progressively more difficult.Summary StreetControl
40 Essentially identical results were observed for Content scoresInappropriate DetailsOrganizationStyleNo improvements forMechanicsPlagiarism was infrequent in both groups and did not increase with practice
41 Performance on a test patterned after the Colorado Student Assessment Program (CSAP): All students take the test under standard, paper-and-pencil conditions, so this is a test of distant transferStudents trained with Summary Street outperform students who only used a word processor on questions requiring summarization; p < .05, effect size is .42No statistically reliable difference between conditions on other types of test items:InferenceVocabularyFact FindingOther
42 Summary Street in Colorado schools: 7 school districts - urban, suburban, rural85 teachers77 classrooms2292 studentsabout 140 texts
43 A Valid Test of Reading Comprehension Current tests are not theoretically motivatedItems have been selected because they discriminate reliably between peopleLevels of comprehension are not distinguishedMultiple-choice questions are used to score tests objectively and automaticallyA valid test shouldDistinguish between deep and shallow comprehensionEmploy a free-response format
44 Beth Mulligan, Katherine Rawson, Praful Mangalath, & Walter Kintsch Designs for a comprehension test.In two experiments 241 college students read 6 expository texts ( words), recalled what they had written, and answered an extended inference questionRecall - shallow comprehensionAbility to reproduce the textInference - deep comprehensionInference questions required a paragraph length response. Information from the text as well as general world knowledge had to be combined to obtain the answerA variety of inference types were used, e.g.,what were the causes of an event described in the text?what is the general theme of the examples given in the text?
45 Recall and Inference are separate components of comprehension Recall performances sets upper bound for inference performance
46 21% of the students received a significantly higher grade on the recall test than on the inference testFor 58 students who received a memory score of B or better, inference scores ranged from F to B+A confirmatory factor analysis yielded two significant factors, a memory factor and an inference factor
47 Grading Essays with LSA: Classifying the vectors representing the students’ responses Support-vector regression segments the semantic space into areas according to the grades human raters assign to essaysNew essay is given a grade according to the area of the space it is in
48 LSA grading of protocols: MemoryHuman rater inter-reliability r = .83LSA-human correlation r = .80InferenceHuman rater inter-reliability r = .80LSA-human correlation r = .68
49 Tools for TeachersThe success and reputation of many professions is based on the use of powerful toolsTeachers employ few tools, mostly because versatile, powerful tools were not availableTools to empower teachers - not to substitute or replace them