Mining Data from Randomized Within-Subject Experiments in an Automated Reading Tutor Joseph E. Beck and Jack Mostow Project LISTEN ( Carnegie Mellon Universitywww.cs.cmu.edu/~listen Funded by National Science Foundation and The Heinz Endowments Research question: What tutorial decision does the experiment investigate? Trial context: In what situation does the tutorial decision occur? Randomized decision: The Reading Tutor chooses at random among plausible alternative actions. Each such choice starts an experimental trial. Randomizing the decision allows causal attribution. Trial outcome: We define the outcome of each decision based on subsequent student behavior – a much finer-grained and more copious source of data than post-test scores. Analysis: Aggregating over many such trials can tell not only which choices work best, but when and for whom. Decoding: What type of help is most effective for helping students learn to decode words? Student is reading story and clicks on a word for help Examine student performance on a future encounter of the word. Does the student ask for help? Does the tutor accept the word as read correctly? N = 189,039 help events Rhyming help is most effective overall. For hard words, best to just tell the student the word. Vocabulary: Does a brief introduction to a word’s meaning before a story help the student to learn the word and comprehend the story? Before student starts to read story, Reading Tutor identifies vocabulary words in story. While student reads story, assess comprehension of story. After student finishes story, assess retention of vocabulary. N = 5,668 vocabulary words tested Explaining words helps for both within-story comprehension probes and after-story vocabulary questions – but effects interact with reading level. Comprehension: Does inserting generic wh- questions help students comprehend stories? Student is reading story N=15,187 cloze questions Logistic regression model Reading Tutor randomly selects which type of help to provide Reading Tutor randomly picks half of vocabulary words to explain Two outcome measures Independent variablesHelps/Hurtsp # preceding wh- questions # preceding cloze questions= # recent wh- questions  # recent cloze questions= Time since prior question (sec) ……… Reading Tutor randomly decides whether to insert a wh- question Student continues reading story During the story, student encounters cloze questions “Draw”Rhymes with “saw” … Experiments embedded in the Reading Tutor help evaluate its decisions in tutoring decoding, vocabulary, and comprehension This work was supported by the National Science Foundation under ITR/IERI Grant No. REC Any opinions, findings, and conclusions or recommendations expressed in this publication are those of the authors and do not necessarily reflect the views of the National Science Foundation or the official policies, either expressed or implied, of the sponsors or of the United States Government.