Sondra Stein & Katie Landeros American Institutes for Research 1 Findings from the Program for the International Assessment of Adult Competencies (PIAAC), 2012 November 12, 2014

About PIAAC PIAAC is an international large-scale assessment administered in in 23 countries It assessed 16 - to 65-year-olds, non-institutionalized, residing in each country, irrespective of nationality, citizenship, or language status Laptop computer or paper-and- pencil: In the U.S., 80% took the computer tests and 15% took the paper-and-pencil tests. Assessment subjects: Literacy Numeracy Problem Solving in Technology-Rich Environments (digital problem solving) Assessment was conducted only in English in the U.S.: The background survey was conducted in English or Spanish. About 4% could not complete the BQ because of language difficulties or learning or mental disabilities, and 1% could not complete it for other reasons. 2

Participating Countries Australia Austria Belgium Canada Cyprus Czech Republic Denmark Estonia Finland France Germany Ireland Italy Japan Korea, Rep of Netherlands Norway Poland Slovak Republic Spain Sweden United Kingdom United States Chile Greece Indonesia Israel Lithuania New Zealand Singapore Slovenia Turkey 3

The PIAAC Assessment was delivered to a nationally representative sample of households in every country.  In the U.S. the household sample was selected through a 4- stage stratified area sample: Counties (PSUs) Blocks Housing units with households Eligible persons within households  Resulted in 5,010 respondents  A U.S. supplement will add 3,600 more adults (report due late 2015) that represent key populations (young adults yrs, older adults yrs, unemployed adults,16-65 yrs)  A representative prison sample will include 1,200 inmates, yrs, in state, federal and private prisons (report due 2016). 4

PIAAC provides a rich source of data that tells us:  What skills adults actually have and can use rather than just the number of years of education they have completed or the degrees they have.  How adults acquire those skills, and what factors are related to skill acquisition and decline.  What the level and distribution of skills is within and across various subgroups within the population.  As a result. PIAAC data enables us to target our efforts to focus on raising the skills of adults with the greatest needs. 5

PIAAC collects data through it’s background questionnaire and module on skill use as well as through direct assessment of skills. Direct assessment of key information- processing skills Module on skill use Background questionnaire 6

PIAAC Background Questionnaire Focused on identifying:  Skills that are critical to functioning successfully in today’s society,  How participants acquire those skills, and  How those skills are distributed. Areas of BQ include:  Education and training, present and past,  Work experience,  Literacy, numeracy and ICT skill use at work and at home,  Other 21 st century skills used at work,  Personal traits, and background information. 7

5 min Background questionnaire Every country could add up to 5 minutes of unique questions and also make country-specific adaptations. U.S. changes include:  Additions Basic skills training Political Efficacy - Information Health Race/Ethnicity Language  Adaptations: Formal Education, Informal Training Occupation, Economic Sector, Earnings 8 8

The Direct Assessment focuses on four domains:  Literacy: both paper & pencil and computer versions  Numeracy: both paper & pencil and computer versions  Problem solving in technology-rich environments: only on computer  Reading components: only paper & pencil  All countries were required to administer literacy and numeracy assessments  The U.S. assessed all four domains 9

Definitions of PIAAC direct assessment subjects  Literacy is understanding, evaluating, using and engaging with written texts to participate in society, to achieve one’s goals, and to develop one’s knowledge and potential.  Numeracy is the ability to access, use, interpret, and communicate mathematical information and ideas, in order to engage in and manage the mathematical demands of a range of situations in adult life.  Problem solving in technology-rich environments involves using digital technology, communication tools and networks to acquire and evaluate information, communicate with others and perform practical tasks. 10

PIAAC results are reported in two ways:  Average Scores: reported on a scale of for all domains.  Proficiency Levels: reported as the percentages of adults scoring at six performance levels ( from below level 1 to level 5) in literacy and numeracy and at four performance levels in problem solving in technology-rich environments (from below level 1 to level 3). 11

These descriptions of the PIAAC Proficiency Levels for Literacy define what adults can do at each level. 12 Locate single piece of information in familiar texts. Read relatively short digital, print or mixed texts to locate single text. Make matches between text and information that may require low level para- phrasing and drawing low- level inferences. Identify, interpret, or evaluate one or more pieces of information and often require varying levels of inference. Perform multiple-step operations to integrate, interpret, or synthesize information from complex texts, and may require complex inferences. Integrate information across multiple, dense texts; construct syntheses, ideas or points of view; or evaluate evidence based arguments. Below Level 1 (0-175) Level 1 ( ) Level 2 ( ) Level 3 ( ) Level 4 ( ) Level 5 ( )

Election results Stimulus is a short report on the results of a union election. The report contains several brief paragraphs and a simple table identifying the three candidates in the election and the number of votes they received. The test-taker is asked to identify which candidate received the fewest votes. He or she needs to compare the number of votes that the three candidates received and identify the name of the candidate who received the fewest votes. The word “votes” appears in both the question and in the table and nowhere else in the text. 13 Below level 1 literacy item

Literacy item Level = 1 Literacy example – low difficulty 14

Literacy item Level = 4 (low) Literacy example – moderate difficulty 15

Overview of U.S. results PIAAC results tell a story about the systemic nature of the skills deficit among U.S. adults. PIAAC raises the question: What are we going to do to make sure that The U.S. has the workforce it needs to succeed in the global economy? U.S. citizens have the skills necessary to support a thriving democracy? 16

How did we do compared to other countries? 17 Japan Finland Netherlands Australia Sweden Norway Estonia Flanders-Belgium Slovak Rep. Germany France Czech Rep. Canada Italy Spain U.K. Denmark United States Ireland Poland Cyprus Austria Japan Finland Flanders-Belgium Netherlands Sweden Norway Denmark Slovak Rep. Czech Rep. Austria Estonia Germany Australia Canada Cyprus Korea, Rep. of U.K. Poland Ireland France United States Italy Spain Japan Finland Australia Sweden Norway Netherlands Austria Denmark Czech Rep. Korea, Rep. of Germany Canada Slovak Rep. Flanders-Belgium U.K. Estonia United States Ireland Poland Italy Spain Cyprus France Literacy Numeracy PS-TRE  The U.S. ranked lower than most other countries in all three domains.  The US ranked better in Literacy than in Numeracy or Problem Solving in technology-rich environments.

18 Japan Finland Netherlands Australia Sweden Norway Estonia Flanders-Belgium Slovak Rep. Germany France Czech Rep. Canada Italy Spain U.K. Denmark United States Ireland Poland Cyprus Austria Korea, Rep. of Literacy  Scores on literacy ranged from 296 (Japan) to 250 (Italy)  U.S. scores were: Lower than in 12 countries Not significantly different than in 5 countries Higher than in 5 countries The U.S. average literacy score (270) was lower than the international average (273).

The U.S. average is low because a higher proportion of U.S. adults are at the lowest levels (level 1 and below level 1) of literacy. 19

20 Who are the low-skilled adults in the U.S. in literacy? Educational attainment

21 Who are the low-skilled adults in the U.S. in literacy? Race/ethnicity

22 Who are the low-skilled adults in the U.S. in literacy? First language

23 Who are the low-skilled adults in the U.S. in literacy? Immigration status

Who are the low-skilled adults in the U.S. in literacy? Employment 24

25 Who are the low-skilled adults in the U.S. in literacy? Occupation

26 Who are the low-skilled adults in the U.S. in literacy? Industry

Source: Autor, D. H. and B.M. Price (2013), "The Changing Task Composition of the US Labor Market: An Update of Autor, Levy, and Murnane (2003)", MIT Mimeograph, June. Mean task input in percentiles of 1960 distribution Since 1970, there has been a shift in the U.S. economy away from routine and manual tasks and towards more analytic and interpersonal tasks that require higher skills. 27

Percentage Change in Earnings Since 1961 Tabulations of annual March Current Population Survey Data, by David Ellwood, Harvard University. Slide prepared by ETS. The percentage change in earnings since the 1970s mirrors that shift in occupations 28

Literacy skills in younger and older generations 29

Approximately 36 million U.S. adults have low skills. Roughly the same population of Minnesota, New York, and Michigan states combined. 30 MN NY MI + + Time for the U.S. to Reskill? What the Survey of Adult Skills Says (an OECD report)

Workforce Innovation and Opportunity Act (WIOA) and Ready to Work  Signed into law July 2014  Previously 1998 Workforce Investment Act  Ready to Work: Job-Drive Training and American Opportunity report also released  Both initiatives aim to “train Americans with the skills employers need, and match them to good jobs that need to be filled now.” 31

1. Well-connected and transparent education, training, credentials, and support services Increasing skills, competencies, and credential s e.g., license, industry credential 3 rd Job in Career Path 2 nd Job in Career Path 1 st Job in Career Path informed by industry/employer s e.g., certificate, diploma e.g., 2-year degree e.g., 4-year degree N th Job in Career Path 2. Multiple entry points – for both well-prepared students and targeted populations bridge(s) 3. Multiple exit points e.g., high school or CTE e.g., adult education or workforce system e.g., military or civilian workplace e.g., postsecondary system e.g., apprenticeship Three Essential Features of Career Pathways 32

 A higher percentage of Blacks and Hispanics  A higher percentage in manual and blue and white-collar semiskilled occupations, including the following sectors: Construction Administrative services Transportation Hospitality Retail Health Who are the low-skilled adults in the U.S. workforce? 33

Barriers to participation in Non-Formal Education (NFE) 34

What can we do to raise the skills of working age adults? In the long run, the best policies we have [to combat inequality] involve investing in our citizenry. … Higher education, and public education, is America’s best idea. -- David Autor 35

 Do a better job of assuring every American has a strong foundation of basic skills.  Do a better job of providing skills upgrading for immigrants to America.  Do a better job of providing skills upgrading to working adults at every level so they can keep pace with change.  Understand that improving the skills of our population is everybody’s business: we can educate our way to a better economy. What does the data suggest about what we need to do differently? 36

How PIAAC can help  Use PIAAC data to advocate for more investment in adult lifelong learning.  Use PIAAC data to strengthen program approaches to building lifelong career pathways.  Use PIAAC data to create and renew local, state, and national approaches to lifelong learning where learning is everybody’s business.  Use PIAAC data and tools to document successes. 37

 A test of Education and Skills Online – an online version of PIAAC that will be available in  Outreach Toolkits (coming soon) so you can share this data with others.  Access to the Data itself, through the NCES or OECD International Data Explorer.  Research reports that analyze the data.  Regular updates through the PIAAC Buzz. What else PIAAC offers at: 38

PIAAC Outreach Toolkits These Outreach Toolkits include all documents and power point modules you will need to share information about PIAAC with others. *Simply download the pieces you want to use!* PIAAC Overview Key Results National Supplement Education & Skills Online (ESO) Videos Infographics Toolkit Slides for Your Presentation Instructions: Start with “What is PIAAC” and “Results Overview”. Then add other modules to build a presentation that suits your audience. What is PIAAC Results Overview These can be downloaded with a focus on literacy, numeracy, or digital problem- solving Sample PIAAC Tasks Gateway & Other Resources National Supplement ESO Key U.S. Issues Low-skilled Workers The Future Workforce Health Status & Skills The Impact of Parent Education (Each of these can be downloaded with a focus on literacy, numeracy or digital problem-solving) Toolkit Documents 39

Visit the PIAAC Gateway at: 40

Questions and Discussion 41

For more information about PIAAC  Visit our website at  Or contact us at  Sign up for our regular newsletter, the PIAAC Buzz at

43 Links to everything you want to know about PIAAC, including:  The latest PIAAC reports and presentations  A calendar of PIAAC-related events  Infographics, brochures, and videos to share  Data tools and training resources  Press coverage from the U.S. and around the world  Links to release events, reports, and presentations What you can find on the PIAAC Gateway

How to Access PIAAC Data  The PIAAC Results Portal enables you to produce figures and tables that take an in-depth look at the U.S. results as well as compare the performance of U.S. adults to adults in other participating countries.  The US PIAAC International Data Explorer (IDE) is a web-based tool that produces customizable tables and graphs using data from PIAAC. The US IDE includes US national variables as well as data from all other participating countries from the administration of PIIAC in It is free of charge and does not require any advanced statistical software or knowledge.  PIAAC IDE Training offers online training for researchers, policymakers, and practitioners who want to use PIAAC data to answer questions they have about adult skills. To sign up for a webinar or arrange one for your group, contact 44

National Center for Education Statistics’ PIAAC International Data Explorer (IDE) 45

 What is the NCES IDE?  PIAAC Data Explorer Demonstration  Answering your research questions NCES PIAAC Data Explorer (IDE) 46

 A web-based application for accessing PIAAC data that does not require any advanced statistical knowledge or software  A point-and-click interface for: Creating statistical tables and charts Exploring levels of adult skills and demographics What is the NCES IDE? 47

 The PIAAC Data Explorer can: Compute simple descriptive statistics such as: – averages, percentages, and percentiles – percentages of students at pre-defined proficiency levels Run cross-tabulations Run significance tests on computed statistics Collapse response categories within a variable Create and export charts, graphs, and tables  It cannot: Run regression models or multi-level analysis Compute new variables What the NCES IDE can and can’t do 48

 Overall scale (e.g. literacy, numeracy, problem solving in technology-rich environments, reading components)  Continuous variables from international and U.S. national background questionnaire (e.g. earnings/hours of work per week)  Demographic variables (e.g. age, education background, employment history)  U.S. specific background questionnaire variables (e.g. self- reported health status, race/ethnicity)  Trend variables from PIAAC (2012), ALL (2003), IALS (1994)  Variables are organized into categories that have shared characteristics What’s in the NCES IDE? 49

Overall Process 2. Select Variables: Select at least one variable from the selection of categories and subcategories. 3. Edit Reports: Preview how your data will look, and edit your report format options and statistics options as desired. 4. Build Reports: Retrieve the data, make charts and graphs, save, and print reports Select Criteria: Choose your measure(s), year(s), and jurisdiction(s).

Select Criteria 51

Select Variables 52

Edit Reports 53

Edit Reports: Format Options and Statistics Options 54

Build Reports 55

56 Chart Options

Significance Test Options 57

Output: Data Table 58

Output: Significance Test 59

Output: Chart 60