Toward a Theory Relating Text Complexity, Reader Ability and Comprehension AERA New Orleans April 10, 2011 Jackson Stenner Chairman & CEO, MetaMetrics jstenner@lexile.com

Three well researched constructs Reader ability Text Complexity Comprehension 3 well researched variables Let’s first put them together in a definition of reading, then put them in an equation. Our focus today will be on text complexity but it is one of three variables that cooperate in a causal model.

Reading is a process in which information from the text and the knowledge possessed by the reader act together to produce meaning. 50 years later we revisit one of Rasch’s favorite constructs Let’s put it all together – text, reader and comprehension Ternary Structure (temperature, pressure, volume) (force, mass, acceleration) Making meaning is comprehension Let’s build a foundation for the science underlying The Lexile Framework. Let’s start with a beautiful definition Text and Reader act together to make meaning How do we operationalize “act together” The Lexile Framework uses an equation that specifies how reader measure and text measures cooperate or “act together” to produce comprehension We use this equation to make text measures and reader measures. Anderson, R.C., Hiebert, E.H., Scott, J.A., & Wilkinson, I.A.G. (1985) Becoming a nation of readers: The report of the Commission on Reading Urbana, IL: University of Illinois

An Equation e Conceptual - = = 1 + e Text Reader Complexity Ability Comprehension Statistical e (RA – TC ) Raw Score i = 1 + e (RA – TC i ) i Operationalizes the definition *Imagining a text to be a test. Key innovation. Three inter-related but distinct constructs The familiar Rasch model with one important addition. Item calibrations come from theory. TCi’ s are either empirical or theoretical item calibrations (when making reader measures) or virtual item calibrations (when making text measures). Same equation is used to make reader measures and text measures!!! Major criticism is that this formulation is “just too simple.” We have amassed a staggering array of successful predictions using this “too simple” model. RA = Reading Ability TC = Text Calibrations 4

Eight Features of the Causal Model Relating Text Complexity, Reader Ability and Comprehension The model is individual centered. The focus is on explaining variation within persons over time. In this framework the measurement mechanism is well specified and can be manipulated to produce predictable changes in measurement outcomes (e.g. percent correct). Item parameters are supplied by substantive theory and, thus, person parameter estimates are generated without reference to or use of any data on other persons or populations. Therefore, effects of the examinee population have been completely eliminated from consideration in the estimation of person parameters for reader ability. Don’t need metaphysical baggage – I mean X causes Y if intervention on X produces predicable changes in Y. The trade-off Revisit equation slide. Explain intervention on RA-TC and nextemp

Eight Features of the Causal Model cont’d. The quantitivity hypothesis can be experimentally tested by evaluating the trade-off property for the individual case. A change in the person parameter can be off-set or traded-off for a compensating change in text complexity to hold comprehension constant. The trade-off is not just about the algebra. When uncertainty in item difficulties is too large to ignore, individual item difficulties may be a poor choice to use as calibration parameters in causal models. As an alternative we recommend, when feasible, averaging over individual item difficulties to produce “ensemble” means. For example empirical text complexities can be excellent dependent variables for testing causal theories.

Eight Features of the Causal Model cont’d. Causal Rasch models are individual centered and are explanatory at both within-subject and between-subject levels. The attribute on which I differ from myself a decade ago is the same attribute on which I differ from my brother today. When data fit a Rasch model, differences between person measures are objective. When data fit a causal Rasch model absolute person measures (reader abilities) are objective (i.e. independent of instrument). Causal Rasch models make possible the construction of generally objective growth trajectories. Each trajectory can be completely separated from the instruments used in its construction and from the performance of any other persons, whatsoever.

Text Demands for College and Career Student 1528 6th Grade Male Hispanic Paid Lunch May 2007 – Dec. 2009 284 Encounters 117,484 Words 2,894 Items 848 Minutes Text Demands for College and Career 1200 1000 1400 1600 May 2016 (12th Grade) 8

Mythology Text Complexity Theoretical: 1300L Empirical: 1357L Adapted from Oasis Article courtesy of EBSCO Publishing The study and interpretation of myth and the body of myths of a particular culture. Myth is a complex cultural phenomenon that can be approached from a number of viewpoints. As generally understood, a myth is a story or narrative that is traditional in a certain culture, having been passed down from early times and regarded as true. It may be said to 1 symbolically the origin of the basic elements and assumptions of a culture. Mythic narratives frequently revolve around the doings of gods or heroes, and may relate, for example, how the world began, how humans and animals came into being, or how certain customs, gestures, or forms of human activities 2. Almost all cultures possess or at one time possessed and lived in terms of myths. 1 immerse belittle portray contradict 2 originated adorned handicapped entwined 9999

Plot of Theoretical Text Complexity versus Empirical Text Complexity for 446 passages

What could account for the 10% unexplained variance? Missing Variables or Theory misspecification Better Criterion Variable Improved Proxies/Operationalizations Expanded Error Model – Treat Item Type as Random Rounding Error Imperfections in Theory Implementation Sarah Kershaw’s (FCRR) dissertation will check the most promising text variables not in the LF equation. Very large text corpora (Google 500 billion) will permit very accurate frequency measures for 10,000s of words and word families. Also empirical Lexile word difficulties will be available in the next five years. The task continuum promises new machine generated task types for measuring text complexity and reader ability. MM is bringing on-line a new analyzer that does not round text measures. Theory implementation compromises. Front to back vs. back to front text measurement gives us a way to estimate effects of theory implementation.

Closing No matter how it is sliced and diced, analyses of joint and conditional probability distributions yield no more than patterns of association. Nothing in the response data nor Rasch analyses of these data exposes the processes (features of the object of measurement) or mechanisms (features of the instrument) that are hypothesized to be conjointly causal on the measurement outcomes. In my view the agenda for a sixth decade of Rasch measurement practice should have at the top a focus on Causal Rasch Models.

Contact Info: A. Jackson Stenner CEO, MetaMetrics jstenner@Lexile.com