Presentation is loading. Please wait.

Presentation is loading. Please wait.

Technology Based Assessment of Language and Literacy Patti Price - Intro - Decoding - Comprehension - Opportunities/Discussion PPRICE Speech & Language.

Similar presentations


Presentation on theme: "Technology Based Assessment of Language and Literacy Patti Price - Intro - Decoding - Comprehension - Opportunities/Discussion PPRICE Speech & Language."— Presentation transcript:

1

2 Technology Based Assessment of Language and Literacy Patti Price - Intro - Decoding - Comprehension - Opportunities/Discussion PPRICE Speech & Language Technology

3 Primates Gorillas Humans Stone tools FireShelter Spears Language: ~ 50,000 years ago Clothing Burial Hunting Art Farming, 10,000 years ago Bronze, 5500 years ago Written language: ~5 – 6,000 years ago Intro Decoding Comprehension Discussion

4 Pre-language: Information access by direct experience Oral language: Information access also from someone in same time and place with same language Written language: Information access not limited by time and place Multimedia: The power of direct experience, but not limited by time, place, language, learning style, sensory abilities… Intro Decoding Comprehension Discussion

5 Written and Spoken Language are Different These differences are important in learning to read. How difficult it is to spell (many letter combos to one sound) How difficult it is to read (many sounds for the same letter) How closely your dialect matches written conventions “Got to say this for you, you got guts. Guts and no brains. But guts alone don’t mean nothin’. “ “I gotta say this for you-- you have guts. Guts and no brains. But guts alone mean nothing.” I’ve got to say this for you: You have guts. You have guts and no brains, but guts alone mean nothing. Intro Decoding Comprehension Discussion

6 Differences Between Speech and Writing Vocabulary Distinct words Sentence length Sentence structure Grammar Prosody WrittenSpoken ‘things’… ‘stuff’ 20,000+ 2, – 30 words ? Complex shallow Conservative Liberal A., ! ? ; : () Rich Intro Decoding Comprehension Discussion

7 ~130 words (1) I’ve been talking – I’ve been multiplying matrices already, but certainly time for me to discuss the rules for matrix multiplication. (2) And the interesting part is the many ways you can do it, and they all give the same answer. (3) So it’s – and they’re all important. (4) So matrix multiplication, and then, uh, come inverses. (5) So we’re – uh, we – mentioned the inverse of a matrix, but there’s – that’s a big deal. (6) Lots to do about inverses and how to find them. (7) Okay, so I’ll begin with how to multiply two matrices. (8) First way, okay, so suppose I have a matrix A multiplying a matrix B and – giving me a result – well, I could call it C. (9) A times B. Okay. (10) Uh, so, l- let me just review the rule for w- for this entry. Transcript from a linear algebra lecture (From Glass, Hazen, Lee and Wang, Analysis and Processing of Lecture Audio Data, 2004) Example ~45 words We’ve been multiplying matrices. Now let’s discuss the rules for matrix multiplication. Note that there are many ways to do it and get the same answer. One way to multiply two matrices, A and B, to get a new matrix C is the following rule: Intro Decoding Comprehension Discussion

8 Overview of TBALL T echnology B ased A ssessment of L anguage & L iteracy Child-friendly testing system Measure & analyze child reading & pronunciation 5-8 year olds Native speakers of English & Spanish Multidisciplinary collaboration: o Engineering o Education o Computer science o Linguistics o Neuroscience UCLA, USC, UCB Intro Decoding Comprehension Discussion

9 8 TBall Goals and Components – Reliable and objective automatic reading assessments – Comprehensive instructional framework for Diagnosis Intervention – Three main components: Children’s interface Teacher interface Assessment module o Decoding (DAM) o Comprehension (BARLA) Intro Decoding Comprehension Discussion

10 9 Decoding Comprehension Need to be balanced – In alphabet-based languages, hints about pronunciation – If you know the language you are reading, the hints may be enough – If your first language is another language, and/or if you know the LTS rules in another language, there may be problems. – MOSTLY depends on successful decoding – Much more to learn than LTS rules… Intro Decoding Comprehension Discussion

11 Decoding Necessary (but not sufficient) for reading comprehension Necessary for reliable and efficient processing of text Strongly related to speed of initial reading acquisition A predictor of difficulties in comprehension development Accounts for much variance in reading ability at all ages Intro Decoding Comprehension Discussion

12 English Letter to Sound Rules Example: pronunciations of ‘o’ Pronunciations of the letter ‘o’ observed in the 100 most frequent words of English 9 OW: go, home, most, no, only, over, so (know, own) 6 UW: into, to, who, do, (too, you) 6 AH: from, of, other, another, (some, come) 6 AO: for, or, on, often, (your, more) 5 OU: (about, hour; down, how, now) 3 UH: (good, goodbye would) 3 -: (people, work, world) 2 AA: got, not 1 W AH: one Intro Decoding Comprehension Discussion

13 Written Language, Complexity LanguageLetterSound Englishc/s/, /k/ hot, car, walk, ah/aa/ Spanisha/aa/ Frencha/aa/ allo, pot, eau/ow/ SpellingSound ache + past = ached/ey k/ + /t/ aim + past = aimed/ey m/ + /d/ paint + past = painted/p ey n t/ + /ax d/ Intro Decoding Comprehension Discussion

14 Decoding Assessment Are the words read correctly? Decodability: closer to ‘one letter one sound’ is better In sample of 100 most frequent words in grade 1 material – m, j, b, d… pretty straight forward (always /m jh b d/) – Digraphs (2 letters -> 1 sound): -ng, -ck, (wh-)… – 2 letters -> 2 sounds (or): th -> /dh, th/ (thy, thigh) – 1 letter -> 2 sounds (and): x -> /k s/ (fox, box) – G: get, thing, laugh, large – E: because, get, been, they, great, often – O: go, do, from, for, how, good, people, got, one Intro Decoding Comprehension Discussion

15 Decoding Assessment Some words are more decodable than others High frequency words tend to be less decodable High frequency words tend to be ‘sight’ words (not ‘decoded’ but recognized instantly) (this is important) Decoding skills are also important If a word is well-known by a child, it may be treated as a sight word and NOT decoded (tyrannosaurus) Therefore, unless we know which words are sight words for a child, it’s hard to assess decoding skills Nonsense words must be decoded Decoding Assessment Measure Intro Decoding Comprehension Discussion

16 Decoding Issues: Dialect, Idiolect and Assessment If the child has trouble with the letter 'r' (common in English), and says 'cow' when shown 'car' is it a misreading? If the child is from Boston and reads 'Carl' as 'Call' is it a misreading? What if the teacher has never heard anyone from Boston? if the child sees the word 'ask' and says 'aks'? if the child is a native speaker of another language, e.g., Spanish, and says 'seat' when presented the word 'sit', is it a misreading? if the child is already learning to read, e.g., Spanish, and sees the word 'sore' and pronounces it 'sore-ay', is it a misreading? SOLUTION: – Map the child's system, whatever it is – Augment the dictionary to diagnostic labels. Intro Decoding Comprehension Discussion

17 Dialect Modeling in ASR Dialect is a system The system is not well modeled in ASR These are being merged These are being merged When the point is the Contrast system When the point is the Contrast system Head sections adapted from Gunnar Fant’s Acoustic Theory of Speech Production Intro Decoding Comprehension Discussion

18

19 Decoding Issues: Dialect, Idiolect and Assessment Grape g r ey p Grape g w ey p Grape g r aa p ey Bird b er d Bird b uh d Bird b iy r d … Rocks w aa k s … Horse h ow s … Three th w iy… Run w ah n… Intro Decoding Comprehension Discussion

20 Decoding Results nug: /n uw g/ - Sp. LTS, phonology, or short/long? jore, tay, bap: /hh ao r ey, t ay, b aa p/ - Sp. LTS yan, vag, hine: /zh aa n, b aa g, ay n/ - Sp. Phon. rin, bap, mot: /r iy n, b ey p, m ow t/ - short/long jore: /zh ow r ey/ - mixed Spanish phonology, LTS jick: /hh ih s k/ - Spanish LTS, misapplied English LTS Intro Decoding Comprehension Discussion

21 Comprehension Assessment GRADE 2: …Kim has a dog. The dog's name is "Dot." Dot is a very white dog, and he has a black dot on his leg. Sam, Ben, and Kim like to play with Dot. They play ball with Dot all day long. Ben lost the ball in the mud, and Dot went into the mud to get the ball. Now, Dot is a brown dog from the mud! … What does Dot look like? A Dalmatian Brown I don’t remember Like a dog A white dog A white dog with a black dot on his leg White… and a dot … on his leg Well he’s white and he has a brown dot on his leg Like a rat Please describe Dot’s physical appearance? Intro Decoding Comprehension Discussion

22 Motivation/ Desiderata ‘Equivalent’ measures for reading or listening (to compare) Separate understanding of probe from understanding of text (probe as picture adds interest and is not text or speech) Diagnose reading error patterns (by selection of alternate items) Easy to automate (compared to most comprehension tests) Items ascend in difficulty (several errors in a row stops the test) Many items (to allow retesting without repeating items) Equal number of plausible/implausible items (to mitigate guessing) PIAT-R (Peabody Individual Achievement Test – Revised) is best match, but… – Not automated – Reading only – They did not respond to our request Intro Decoding Comprehension Discussion

23 Intro Decoding Comprehension Discussion Vocabulary Syntax Morphology Accent adjustment Prosody Find words Print adjustment Punctuation Decoding (LTS) In your first language, this might be easier In a new language, this might be easier

24 BARLA Example 1. (pot and box reversed) 2.(dog in box) 3.(box on boy) 4.(correct) The pot is on a box. Hear OR Read: The pot is on a box. Intro Decoding Comprehension Discussion

25 BARLA Example Here is a fire. 1.‘tire’ (rhymes) 2.‘fire’ (correct) 3.‘fur’ (vowel differs) 4.‘fairy’(shares, f, i, r) Hear or read: Here is a fire. Intro Decoding Comprehension Discussion

26 Decoding Opportunity We sampled the space with 25 items – High Frequency Word: tell, nap – Low Frequency Word: fell, zap – Nonsense Word: zell, fap However, desirable would be – Frequent retesting (perhaps in a game environment) – Diagnostic of which decoding skills need more work – Items need to be generated and coded for diagnosis Intro Decoding Comprehension Discussion

27 An Opportunity! Intro Decoding Comprehension Discussion ContentGamePedagogy User Model Letter sounds, English: a /ae/ b /b/ c /k/, /s/ …z /z/ By font, audio files Levels: Lower case simple to complex Upper case Digraphs… Move up if: 100% correct Move down if: 3 wrong in a row Last played Sounds mastered Diagnosis Preferences… Engine b p d b p d

28 An Opportunity! Intro Decoding Comprehension Discussion ContentGamePedagogy User Model Vocabulary, English: … bat … Levels: Lower case simple to complex Upper case Digraphs… Move up if: 100% correct Move down if: 3 wrong in a row Last played Sounds mastered Diagnosis Preferences… Engine b p d b t a

29 A Comprehension Opportunity! Data collected for sentences analogous to PIAT-R But not analyzed Paradigm ready to hatch more uses For more information about the project: For more information about the data:  Intro Decoding Comprehension Discussion

30 An Opportunity! Intro Decoding Comprehension Discussion Natural language processing and reading augmentation

31 Simplify Vocabulary By "augmenting human intellect" we mean increasing the capability of a man to approach a complex problem situation, to gain comprehension to suit his particular needs, and to derive solutions to problems. Increased capability in this respect is taken to mean a mixture of the following: more-rapid comprehension, better comprehension, the possibility of gaining a useful degree of comprehension in a situation that previously was too complex, speedier solutions, better solutions, and the possibility of finding solutions to problems that before seemed insoluble. And by "complex situations" we include the professional problems of diplomats, executives, social scientists, life scientists, physical scientists, attorneys, designers--whether the problem situation exists for twenty minutes or twenty years. We do not speak of isolated clever tricks that help in particular situations. 1. Select Text2. Select View3. View By "augmenting human intellect" we mean increasing the capability of a man to approach a complex problem situation, to gain comprehension to suit his particular needs, and to derive solutions to problems. Increased capability in this respect is taken to mean a mixture of the following: more-rapid comprehension, better comprehension, the possibility of gaining a useful degree of comprehension in a situation that previously was too complex, speedier solutions, better solutions, and the possibility of finding solutions to problems that before seemed insoluble. And by "complex situations" we include the professional problems of diplomats, executives, social scientists, life scientists, physical scientists, attorneys, designers--whether the problem situation exists for twenty minutes or twenty years. We do not speak of isolated clever tricks that help in particular situations. Simplify vocabulary Simplify syntax Simplify both Simplify vocabulary Simplify syntax Simplify both By "adding to people's brains" we mean adding to the way we deal with difficult things, to add to understanding according to our needs, and to figure out solutions to problems. Intro Decoding Comprehension Discussion

32 Simplify Syntax By "augmenting human intellect" we mean increasing the capability of a man to approach a complex problem situation, to gain comprehension to suit his particular needs, and to derive solutions to problems. Increased capability in this respect is taken to mean a mixture of the following: more-rapid comprehension, better comprehension, the possibility of gaining a useful degree of comprehension in a situation that previously was too complex, speedier solutions, better solutions, and the possibility of finding solutions to problems that before seemed insoluble. And by "complex situations" we include the professional problems of diplomats, executives, social scientists, life scientists, physical scientists, attorneys, designers--whether the problem situation exists for twenty minutes or twenty years. We do not speak of isolated clever tricks that help in particular situations. 1. Select Text2. Select View3. View By "augmenting human intellect" we mean increasing the capability of a man to approach a complex problem situation, to gain comprehension to suit his particular needs, and to derive solutions to problems. Increased capability in this respect is taken to mean a mixture of the following: more-rapid comprehension, better comprehension, the possibility of gaining a useful degree of comprehension in a situation that previously was too complex, speedier solutions, better solutions, and the possibility of finding solutions to problems that before seemed insoluble. And by "complex situations" we include the professional problems of diplomats, executives, social scientists, life scientists, physical scientists, attorneys, designers--whether the problem situation exists for twenty minutes or twenty years. We do not speak of isolated clever tricks that help in particular situations. Simplify vocabulary Simplify syntax Simplify both Simplify vocabulary Simplify syntax Simplify both By "augmenting human intellect" we mean increasing the capability of a person to approach a complex problem situation. We also mean how a person gains comprehension to suit his particular needs and derives solutions to problems. Intro Decoding Comprehension Discussion

33 Simplify Vocab & Syntax By "augmenting human intellect" we mean increasing the capability of a man to approach a complex problem situation, to gain comprehension to suit his particular needs, and to derive solutions to problems. Increased capability in this respect is taken to mean a mixture of the following: more-rapid comprehension, better comprehension, the possibility of gaining a useful degree of comprehension in a situation that previously was too complex, speedier solutions, better solutions, and the possibility of finding solutions to problems that before seemed insoluble. And by "complex situations" we include the professional problems of diplomats, executives, social scientists, life scientists, physical scientists, attorneys, designers--whether the problem situation exists for twenty minutes or twenty years. We do not speak of isolated clever tricks that help in particular situations. 1. Select Text2. Select View3. View By "augmenting human intellect" we mean increasing the capability of a man to approach a complex problem situation, to gain comprehension to suit his particular needs, and to derive solutions to problems. Increased capability in this respect is taken to mean a mixture of the following: more-rapid comprehension, better comprehension, the possibility of gaining a useful degree of comprehension in a situation that previously was too complex, speedier solutions, better solutions, and the possibility of finding solutions to problems that before seemed insoluble. And by "complex situations" we include the professional problems of diplomats, executives, social scientists, life scientists, physical scientists, attorneys, designers--whether the problem situation exists for twenty minutes or twenty years. We do not speak of isolated clever tricks that help in particular situations. Simplify vocabulary Simplify syntax Simplify both Simplify vocabulary Simplify syntax Simplify both By "adding to people's brains" we mean adding to how we deal with difficult things. We also mean adding to understanding that is suited to our needs, and to how we can figure out solutions to problems. Intro Decoding Comprehension Discussion

34 Simplify Vocabulary & Syntax: Result By "adding to people's brains" we mean adding to how we deal with difficult things. We also mean adding to understanding that is suited to our needs, and to how we can figure out solutions to problems. By adding to what we can do, we mean a mix of several things. Faster and better understanding are examples. We also mean making it possible to get some understanding when before it was too difficult. And, we mean faster solutions, better solutions, and being able to find solutions that didn't seem possible before. By "difficult situations" we mean the problems people have in different kinds of jobs. The jobs can be in the government, running businesses, being scientists, lawyers, or designers. The problems can be there for twenty minutes or twenty years. We do not mean clever tricks that help in only some situations. We mean a way of life. In this new way of life many things are useful. Useful things include hunches, cut-and-try, things we can't touch, and our "feel for a things“. These things are more useful with great ideas, ways to talk about the ideas and ways to write them, intelligent ways of doing things, and powerful technologies to help us. The number of people and what they create are growing fairly quickly. But our problems are getting harder even more quickly. Also, the importance of finding solutions becomes greater because things are changing more quickly than they used to change. Things that before happened only in one place now are spreading to the whole world. This also makes it important to find solutions. If we could make our brains better in this way, any intelligent person would run as fast as possible to get it. To make them want to run, we need to show a way to do it. We also need to show what we might get out of it. Intro Decoding Comprehension Discussion

35 Summary of Views and Reading Ease Original Words239 Sentences7 Word/sentence34.1 Characters/ word 5.6 Reading Ease10.7% Flesch-Kincaid Grade Level 12.0 Vocabu -lary SyntaxBoth %28.2%71.7% Intro Decoding Comprehension Discussion

36 Questions? Intro Decoding Comprehension Discussion


Download ppt "Technology Based Assessment of Language and Literacy Patti Price - Intro - Decoding - Comprehension - Opportunities/Discussion PPRICE Speech & Language."

Similar presentations


Ads by Google