PIAAC: OBJECTIVES AND METHODS New England Association of Colleges Conference, 2014 William Thorn, Senior Analyst, OECD

Give an overview of the objectives and methods of PIAAC – Why the survey was undertaken – Content – Methodology – Reporting Purpose

Origins 1980s - early 1990s – interest in literacy levels of workforce – ‘competence’ movement – Developments in large-scale testing Development of international adult literacy/skills surveys: – IALS (1994, 1996, 1998): 21 countries (including Ireland in 1994) – ALL (2003, 2006): 13 countries

Origins Work on PIAAC began in early 2000s – Updating measures to increase relevance to the digital world – Expansion of the range of skills about which information collected (e.g. ‘generic’ skills) – Interest in the ‘demand’ for skills in addition to supply – Measurement of ‘human capital’ rather than ‘literacy’

Objectives Design of PIAAC finalised in 2007 Broad objectives : – Provide high quality comparable information on the level and distribution of key information processing skills in the adult population – Show the relationship of these skills to individual and social ‘outcomes’ – Better understand the processes through which skills are gained, maintained and lost over the lifecycle

Design features: content Direct assessment of key information processing skills – Literacy (including reading components), numeracy, problem solving in technology-rich environments (PS-TRE) – Linked to IALS and ALL in domains of literacy and numeracy Information on the use of literacy, numeracy and problem solving at work and elsewhere Information on use of a range of other generic skills at work – Interaction, organisation (self and others), learning and physical skills Information on antecedents and outcomes of the assessed skills

Design features Target population – year olds resident in national territory Sample: probability sample representative of target population Household survey Computer delivery – BQ – CAPI – Assessment - CBA

Participation Round 1 ( ) – 24 countries – Australia, Austria, Belgium (Flanders), Canada, Czech Republic, Denmark, Estonia, Finland, France, Germany, Ireland, Italy, Japan, Korea, Netherlands, Norway, Poland, Slovak Republic, Spain, Sweden, UK (England, Nth Ireland), US, Cyprus, Russian Federation Round 2 ( ) – 10 countries – Chile, Greece, Indonesia, Israel, Lithuania, New Zealand, Singapore, Slovenia, Turkey

Innovative elements Proficiency in information processing in ICT environments – Reading of digital texts – Problem solving in technology rich environments Information regarding poor readers – Reading components Richer information regarding the use of skills – In particular, the use of generic skills Computer delivery

Literacy The ability to... Understand, evaluate, use and engage with written texts. In order to.. Achieve one’s goals, and to develop one’s knowledge and potential. Literacy encompasses a range of skills from.. The decoding of written words and sentences The comprehension, interpretation and evaluation of complex texts. Numeracy The ability to… Access, use, interpret and communicate mathematical information and ideas In order to.. Engage in and manage the mathematical demands of a range of situations in adults. Numeracy involves Managing a situation or solving a problem in a real context, by responding to mathematical content/information/ideas represented in multiple ways. Problem Solving In Technology-rich Environments The ability to… Use digital technology communication tools and networks to acquire and evaluate information, communicate with others and perform practical tasks. The assessment focuses on the abilities to… Solve problems for personal, work and civic purposes by setting up appropriate goals and plans, and accessing and making use of information through computers and computer networks. 10 The skills assessed

How the domains are defined Content – the artefacts, tools, knowledge, representations and cognitive challenges that constitute the corpus adults must respond to or use Cognitive strategies – the processes that adults must bring into play to respond to or use given content in an appropriate manner Contexts – the different situations in which adults have to read, display numerate behaviour, and solve problems

SectionContent ABasic demographics (age, sex) BEducational attainment and participation CLabour Force status and work history DCharacteristics of current job E Characteristics of last job (if unemployed and worked in last 5 years) FSkill use at work (generic skills) GSkill use at work (literacy, numeracy and ICTs) HSkill use in everyday life (literacy, numeracy and ICTs) I‘Social outcomes’ and learning strategies JBackground characteristics (social, linguistic and migration background and family composition) The background questionnaire

Reporting Results presented on a 500 point scale – test items and test-takers located on the same scale – Difficulty of items – Proficiency of persons

Proficiency levels To help interpret results the scale divided into proficiency levels Descriptors developed to summarise the underlying characteristics of items in each level in terms of the literacy framework Descriptive not normative 6 proficiency levels in literacy and numeracy and 4 levels in PSTRE

Proficiency levels: literacy and numeracy LevelScore Range Less than level 1Lower than 176 Level Level Level Level Level 5376 and above

Proficiency levels: numeracy LevelTask features Less than level 1Use simple processes (e.g. counting, basic arithmetic, recognising spatial representations; familiar contexts; little text and few distractors Level 1Use basic mathematical processes (e.g. involving one step): counting sorting, simple percentages, identifying elements of common graphical or spatial representations Level 2Use mathematical process involving two or more steps (calculations with whole numbers and decimals, percentages and fractions, simple measurement); common contexts; content explicit and few distractors Level 3Use mathematical processes involving several steps and choice of problem solving strategies; application of number and spatial sense; patterns and proportions, statistical data in texts tables and graphs Level 4Understand complex and abstract mathematical information; tasks require analysis and more complex reasoning Level 5Understand complex representations and mathematical and statistical ideas; integrate multiple types of mathematical information; draw inferences, work with models and evaluate choices and solutions

Level 4: numeracy Tasks at this level require the respondent to understand a broad range of mathematical information that may be complex, abstract or embedded in unfamiliar contexts. These tasks involve undertaking multiple steps and choosing relevant problem-solving strategies and processes. Tasks tend to require analysis and more complex reasoning about quantities and data; statistics and chance; spatial relationships; and change, proportions and formulas. Tasks at this level may also require understanding arguments or communicating well-reasoned explanations for answers or choices.

Proficiency levels: literacy LevelTask features Less than level 1Read short texts on familiar topics; few distractors; information requested is identical with that in text; basic vocabulary knowledge; no knowledge of text features required Level 1Read relatively short texts to locate information that is identical to or synonymous with information requested; few distractors; cycling though more than one piece of information; basic vocabulary; some knowledeg of text features Level 2Make matches between text and information, paraphrasing, low level inferences; some distractors; use strategies such a cycling through information, compare and contrast, navigate through digital texts Level 3Read relatively long and dense texts; understand text and rhetorical structures; evaluate information and make inferences; multi-step operations; distracting information present Level 4Read complex and lengthy texts; complex inferences and background knowledge required; conditional and competing information present Level 5Search for and integrate information across multiple and dense texts; construct syntheses of similar and contrasting ideas and points of view; evaluate evidentiary sources; awareness of rhetorical cues; make high level inferences and draw on specialise background knowledge.

Level1: Literacy Most of the tasks at this level require the respondent to read relatively short digital or print continuous, non- continuous, or mixed texts to locate a single piece of information that is identical to or synonymous with the information given in the question or directive. Some tasks, such as those involving non-continuous texts, may require the respondent to enter personal information onto a document. Little, if any, competing information is present. Some tasks may require simple cycling through more than one piece of information. Knowledge and skill in recognising basic vocabulary determining the meaning of sentences, and reading paragraphs of text is expected.

Find Out More about PIAAC at: All national and international publications The complete micro-level database Thank you