Time for Multi-State Models of Vocabulary Acquisition? Rob Waring

Assessing vocabulary acquisition Which components do we assess? receptive? productive? use? form? meaning? Which test type? m/c? translation? vocab knowledge scales? other? Which words? general? technical? low-hi frequency? Stakes high - for formal assessment, grades etc. low - for research?

Problems with our current test battery Translation L1 –> L2 or L2 –> L1 – Low volume– only a few dozen words at best before we kill the subjects :) – Notoriously difficult to score Inter-rater reliability issues Criteria for successful answer – award half points? Easy to make arbitrary choices about what is correct Not sensitive to partial knowledge Scores can be affected by test strategy use Lack of knowledge of L1 word for an L2 equivalent Not all words can be easily translated Etc.

Problems with our current test battery Multiple choice – Low volume– only a few dozen words at best before we kill the subjects :) – Notoriously difficult to make Which distractors? Sensitive or not? Contextualized or not? Equating vocabulary frequency of the distractors and the target Use of definitions, synonyms, or ???? – 25% is a giveaway (unless there’s ‘I don’t know) – Often need correction for guessing – Scores can be affected by test strategy use – Etc.

Problems with our current test battery Knowledge scales e.g. Wesche and Paribakht (1996)

Problems with our current test battery Knowledge scales e.g. Wesche and Paribakht (1993) A *?%$#>& mess! -using ordinal data nominally -multiple aspects of knowledge at the same level – internal inconsistency -productive and receptive all mixed up -totally arbitrary scoring -unclear what gains mean (e.g. t1 mean 2.5, t2 mean 2.7 t3 mean 2.8) -compare S1 + vs. S2 II +

Assumptions underlying scales We move from the receptive to productive receptiveproductive But this assumes receptive knowledge is complete before we can produce – Huh?

Assumptions underlying scales We could have separate scales receptive productive But any gains on the receptive are not seen on the receptive and vice versa – Huh?

Assumptions underlying scales We start with a threshold receptive knowledge productive receptive But any gains on the productive still aren’t seen in the receptive and vice versa – Huh?

A solution See each of the stages as states of knowledge not a scale Recognize the data are ordinal, not nominal Develop linear scales of a single aspect of vocabulary knowledge

Simple state model 0 I do not understand (the meaning of) this word 1I understand (the meaning of) this word a little 2I understand (the meaning of) this word quite well 3I understand (the meaning of) this word very well Test design Understand Can use in a sentence Apple Book Curtain

Build a matrix 3 xxxx 2 xxx 1 x 0 xxx 0123 Understand Use

Track data over time 3 h 2 efg 1 cd 0 ab 0123 Understand Use 3 ghe 2 cf 1 bd 0 a 0123 Understand Use t1t2

3d representations

Advantages of State Models Any words, phrases, collocations, etc. can be tested Fast data collection – hundreds per hour (esp. if digitally collected) Direct access to knowledge (subject reports what they know) – Knowledge is not mediated through assumptions for what a test is assessing – E.g. sensitive vs insensitive targets, with or without context Can track a single word or multiple words over time – E.g. verbs vs nouns vs adjectives – Can see how does derivative knowledge develop – Can see at what stage can learners use systemic knowledge e.g. inflectional Allows us to see changes or development over time Allows us to see patterns in development Allows us to look at whole lexicons, not just words Any variable (subject to declarative knowledge) can be used on the axes (meaning, use, pronunciation, etc.)

Issues with State Models Not suitable for high-stakes testing Assumes subjects have access to declarative knowledge Unclear what math to use for analysis (to me at least!) Adding levels to get finer detail leads to – massive increases in data needed for reliability – a need for clear labels for each state a three state model is crude (I don’t know, I think I know, I know) a 7 state model is too fine (I don’t know, ?. ?, ?, ?, ?, I know perfectly) Polygraphs need careful attention (the various meanings of bank might need contextualizing) Labels for states determine what you are testing – I can use it vs. I can use it in a sentence vs. I can use it in speech

Issues with State Models Hard to do for listening We may need to adjust the data for accuracy of reporting Hard to validate self-reports -Can use non-words to validate reports (splonk, merd, thyde) -Will need to validate any test instrument with a pilot population before mass-use e.g. Give oral check (e.g. m/c, translations) test with pilot populations to validate their rating of say state 2, is actually state 2

Issues with State Models Need to validate knowledge reports are not random -Give subjects several tests including a subset of test A words in test B a few days apart -Pilot the test instrument with some subjects first. We should find most data are orange Same knowledge t1 t2

Questions for you… What other ways could a state model of vocabulary be used? Is there an application in your own area? What math would be appropriate to use on these data?

Thanks for your time! Rob Waring