PISA 2009 – New Approaches to Assessing Reading Literacy Gerry Shiel Rachel Perkins Educational Research Centre

Overview Status of reading literacy in PISA 2009 Overview of the framework Engagement and metacognition Demonstration of electronic reading items Scaling reading literacy in PISA 2009 and reporting on trends

Reading Literacy as a Major Assessment Domain in PISA 2009

Definition of Reading Literacy in 2009 No change since 2000 Reading literacy is understanding, using and reflecting on written texts, in order to achieve one’s goals, to develop one’s knowledge and potential, and to participate in society. Written texts is understood to mean a range of texts – handwritten, printed and electronic

Dimensions of Reading Literacy Situation Text (medium, format, rhetorical structure) Aspect (strategy or cognitive approach)

Reading Situations Refers to uses for which the author constructed a text

Distribution of Reading Tasks by Situation

Texts Medium – printed vs. electronic (hypertext) Text format – continuous, non-continuous, mixed and multiple Rhetorical structure – description, narration, exposition, argumentation and instruction

Text - Medium Electronic texts – hypertext – a text or texts with access structures that make possible and encourage non-linear reading. Electronic texts exist in a number of environments, including web environment, desk top and e-mail. PISA 2009 excludes electronic texts outside computer environments (e.g., text messages)

Continuous and Non-continuous Texts – Print Medium

Continuous and Non-continuous Texts – Electronic Medium

Distribution of Continuous and Non-continuous Texts (Print)

Reading Processes (Aspects)

Three Broad Aspect Categories retrieving information; interpreting texts; reflection and evaluation.

Relationship between task, text and aspects in the print medium

Relationship between task, text and aspects in the electronic medium

PISA 2009 – Distributions of Items in Printed Text, by Aspect

PISA 2009 Reading Response Formats Print Texts Multiple choice Constructed Response Electronic Texts Multiple choice Constructed response Closed response (e.g., Drag and drop; copy and paste; select by clicking) Open response (e.g., generating and entering a search term)

Reading Engagement In PISA 2000, there was a stronger association between engagement in reading and reading performance than between SES and reading in all participating countries It was argued that reading engagement could reduce the gap in achievement between subgroups New definition of engagement, covering print and electronic texts, proposed for 2009.

Components of Reading Engagement interest in reading – disposition to read literature and information text for enjoyment and the satisfaction of curiosity; perceived autonomy – perceived control and self-direction of one’s reading activities, choices, and behaviours; efficacy – belief in one’s capacity to read well in the present and future; social interaction – social goals for reading and interactive competence; mastery goal – aim of reading to comprehend, understand, and explain text; reading practices – behavioural engagement referring to the amount and types of reading activities.

Measuring Engagement in Reading Concept of unified engagement Equal emphasis on reading motivation (interest, autonomy, efficacy, social interaction and mastery goal) and reading practices Contextualisation of engagement across a range of types of reading material (fiction, information and social communication).

Educational Context of Engagement Relevance: My teacher tells us how the reading relates to our experience. Autonomy support: This teacher lets me choose what to read. Self-efficacy support: My teacher shows me how to read the material for this course. Social interaction in literacy: In class, I talk with other students about what the text means. Mastery goal structure: The teacher explains how the topics fit together. Opportunity to read: In this class the teacher helps us with a lot of challenging reading.

Metacognition and Reading Metacognition in reading refers to the awareness of and ability to use a variety of appropriate strategies when processing texts in a goal oriented manner. It also involves an understanding of the differential processing demands associated with various kinds of tasks and how to apply this understanding. Metacognition has both a significant correlation with reading proficiency, and is amenable to teaching and learning

Measuring Metacognition

Assessing Metacognition

Pilot Assessment of Electronic Reading in the PISA 2009 Field Trial (2008) 22 countries participated in international option for reading of electronic texts Australia, Austria, Belgium, Canada, Chile, Denmark, France, Germany, Hong-Kong, Hungary, Iceland, Ireland, Japan, Korea, Macao-China, New Zealand, Norway, Poland, Russian Federation, Spain, Sweden and United Kingdom. Decision on participation in electronic reading component of main study to be made in October 2008.

Relationship between navigation and text processing for ERA tasks

Reading Literacy Scales in PISA 2009 Reading trend scale (based on 28 ‘common’ items) PISA 2009 reading literacy scale (print only) PISA 2009 electronic reading literacy scale PISA 2009 reading literacy scale (print and electronic combined)

Reading Literacy Sub-scales in PISA 2009 (Print) Access and retrieve Integrate and interpret Reflect and evaluate Continuous Non-continuous (Items based on mixed and multiple texts not to be scaled – too few items)

Reading Literacy Sub-scales in PISA 2009 (Electronic) Access and retrieve Integrate and interpret Reflect and evaluate Complex (processes)