Using Assessment Program Resources to Shape Effective ELA Instruction Amy F. Radikas, Literacy Specialist Renee A. Savoie, NAEP State Coordinator Connecticut Assessment Forum III Crowne Plaza, Cromwell August 13, 2012

Session Overview Key shifts in ELA/Literacy NAEP and PISA basics Evidence of the shifts illustrated through NAEP & PISA Discussion

CCSS Requires Three Shifts in ELA/Literacy #1Building knowledge through content-rich nonfiction #2Reading, writing and speaking grounded in evidence from text, both literary and informational #3Regular practice with complex texts and its academic language achievethecore.org

ACT Report In 2006, ACT released a report called Reading Between the Lines. The findings suggested that the ability to read complex texts is the “clearest differentiator between those ready for college-level reading and those not.”

ACT: Reading Between the Lines The most important implication of this study: “What students could read, in terms of its complexity, was at least as important as what they could do with what they read.” CCSS Appendix A. p. 2

What is NAEP? NAEP National Assessment of Educational Progress Nation’s Report Card Established by Congress in 1969 to answer questions about student academic performance The Nation’s ONLY monitor of what students know and can do ◦ “common yardstick” “Survey” design ◦ National, regional and state-level results

Overall NAEP Reading Results for Connecticut GRADE 4GRADE 8 YEAR AVG. SCALE SCORE % OF STUDENTS AT/ABOVE PROFICIENT AVG. SCALE SCORE % OF STUDENTS AT/ABOVE PROFICIENT *37* *34* *37* * 43X X 45X * indicates a statistically significant difference when compared to performance in 2011.

Grade 4 Reading in 2011: Average Scale Score Comparisons Grade 4 Reading in 2011: Average Scale Score Comparisons Higher avg. ss. than CT Avg. ss. not significantly different from CT Lower avg. ss. than CT

Grade 8 Reading in 2011: Average Scale Score Comparisons Higher avg. ss. than CT Avg. ss. not significantly different from CT Lower avg. ss. than CT Grade 8 Reading in 2011: Average Scale Score Comparisons Grade 8 Reading in 2011: Average Scale Score Comparisons

Program for International Student Assessment ( sponsored by the OECD) Program for International Student AssessmentOECD Administered every 3 years since 2000 Assesses 15-year-old students Measures application of knowledge and skills to problems in a real-life context What is PISA? Click to view YouTube video about PISA

PISA aims to answer… Are students well prepared to meet the challenges of the future? Are they able to analyze, reason, and communicate their ideas effectively? Do they have the capacity to continue learning throughout life?

Who Participates in PISA? OECD countries (34) Non-OECD education systems (33)

PISA 2009 Reading Literacy Results Data for 65 education systems (34 OECD and 31 non-OECD) U.S. not measurably different from OECD average U.S. scored below 9 education systems U.S. scored above 39 education systems 13

PISA Reading Literacy Trends

PISA and NAEP: Key Differences PISA includes a considerable amount of noncontinuous text. NAEP measures “meaning vocabulary.” NAEP Grade 8 and 12 passages are notably longer than PISA passages. In terms of readability and grade level, PISA is closer to Grade 12 NAEP.

The Origins of CCSS International Assessments NAEP CCSS

CCSS Requires Three Shifts in ELA/Literacy #1—Building knowledge through content-rich nonfiction #2—Reading, writing and speaking grounded in evidence from text, both literary and informational #3—Regular practice with complex texts and its academic language achievethecore.org

Informational Text Students are required to read very little informational text in elementary and middle school. Informational text is more difficult for students to comprehend than narrative text. Informational text makes up the vast majority of required reading in college/workplace.

NAEP Reading: Passage Types by Grade GradeLiteraryInformational 450% 845%55% 1230%70% CCSS ELA p.5

In the Early Years Children must:  Develop strong foundational cognitive skills (literacy/communication).

Grade Two  Informational texts use clear and consistent formats, use of simple headings to organize information into categories, illustrations extend the meaning, and simple graphics support understanding of content Elfrieda H. Hiebert, Fundamentals of Literacy Instruction and Assessment, Pre-K–6 Edited by Martha C. Hougen, Ph.D., & Susan M. Smart

An Example from NAEP NAEP 2011 Grade 4 Informational Passage: Marian’s Revolution by Sudipta Bardhan-Quallen Available through the NAEP Questions Tool

One of 10 Released Items Why is "A Voice for Civil Rights" a good heading for the section that follows it on page 3? Use information from the article to support your answer.

Student Responses: Full Comprehension

Student Responses: Partial Comprehension

Student Responses: Little or No Comprehension

Item-Level Performance Data Little/NoPartialFullOmitted Avg. SS% Avg. SS% Avg. SS% Avg. SS% NP CT ‡5 NP= National Public ‡ Reporting standards not met.

CCSS Requires Three Shifts in ELA/Literacy #1—Building knowledge through content-rich nonfiction #2—Reading, writing and speaking grounded in evidence from text, both literary and informational #3—Regular practice with complex texts and its academic language achievethecore.org

Evidence to Support Text Dependent Questions Can be literal (checking for understanding) but must also involve analysis, synthesis, evaluation. Focus on word, sentence, and paragraph, as well as larger ideas, themes, or events. Focus on difficult portions of text in order to enhance reading proficiency. Can also include prompts for writing (DOK) and discussion questions. achievethecore.org

Bloom’s Taxonomy [1956 ] & Bloom’s Cognitive Process Dimensions [2005] Knowledge -- Define, duplicate, label, list, name, order, recognize, relate, recall Remember Retrieve knowledge from long-term memory, recognize, recall, locate, identify Comprehension -- Classify, describe, discuss, explain, express, identify, indicate, locate, recognize, report, review, select, translate Understand -- Construct meaning, clarify, paraphrase, represent, translate, illustrate, give examples, classify, categorize, summarize, generalize, predict… Application -- Apply, choose, demonstrate, dramatize, employ, illustrate, interpret, practice, write Apply -- Carry out or use a procedure in a given situation; carry out or use /apply to an unfamiliar task Analysis -- Analyze, appraise, explain calculate, categorize, compare, criticize, discriminate, examine Analyze -- Break into constituent parts, determine how parts relate Synthesis -- Rearrange, assemble, collect, compose, create, design, develop, formulate, manage, write Evaluate -- Make judgments based on criteria, check, detect inconsistencies/fallacies, critique Evaluation -- Appraise, argue, assess, choose, compare, defend, estimate, explain, judge, predict, rate, core, select, support, value Create -- Put elements together to form a coherent whole, reorganize elements into new patterns/ structures Karin Hess

Webb’s Depth-of-Knowledge Levels DOK-1 – Recall & Reproduction - Recall of a fact, term, principle, concept, or perform a routine procedure DOK-2 - Basic Application of Skills/Concepts - Use of information, conceptual knowledge, select appropriate procedures for a task, two or more steps with decision points along the way, routine problems, organize/display data, interpret/use simple graphs DOK-3 - Strategic Thinking - Requires reasoning, developing a plan or sequence of steps to approach problem; requires some decision making and justification; abstract, complex, or non-routine; often more than one possible answer DOK-4 - Extended Thinking - An investigation or application to real world; requires time to research, problem solve, and process multiple conditions of the problem or task; non-routine manipulations, across disciplines/content areas/multiple sources Karin Hess

The Hess Cognitive Rigor Matrix integrates Bloom + Webb Different models used to describe cognitive rigor:  Bloom – What type of thinking (verbs) is needed to complete a task?  Webb – How deeply do you have to understand the content to successfully interact with it? How complex is the content? Karin Hess

Depth + Thinking Level 1 Recall & Reproduction Level 2 Skills & Concepts Level 3 Strategic Thinking/ Reasoning Level 4 Extended Thinking Remember - Recall, locate basic facts, details, events Understand - Select appropriate words to use when intended meaning is clearly evident - Specify, explain relationships - summarize – identify main ideas - Explain, generalize, or connect ideas using supporting evidence (quote, example…) - Explain how concepts or ideas specifically relate to other content domains or concepts Apply - Use language structure (pre/suffix) or word relationships (synonym/antonym) to determine meaning – Use context to identify meaning of word - Obtain and interpret information using text features - Use concepts to solve non-routine problems - Devise an approach among many alternatives to research a novel problem Analyze - Identify whether information is contained in a graph, table, text feature, etc. – Compare literary elements, terms, facts, events – analyze format, organization, & text structures - Analyze or interpret author’s craft (literary devices, viewpoint, or potential bias) to critique a text – Analyze multiple sources - Analyze complex/abstract themes Evaluate – Cite evidence and develop a logical argument for conjectures - Evaluate relevancy, accuracy, & completeness of information Create - Brainstorm ideas about a topic - Generate conjectures based on observations or prior knowledge - Synthesize information within one source or text - Synthesize information across multiple sources or texts The Hess Cognitive Rigor Matrix Applies Webb’s DOK to Bloom’s Cognitive Process Dimensions

DOK is about complexity—not difficulty! The intended student learning outcome determines the DOK level. What mental processing must occur? While verbs may appear to point to a DOK level, it is what comes after the verb that is the best indicator of the rigor/DOK level. ◦ Describe the process of photosynthesis. ◦ Describe how the two political parties are alike and different. ◦ Describe the most significant effect of WWII on the nations of Europe. Karin Hess

Let’s practice Your class has just read some version of Little Red Riding Hood. What is a basic comprehension question you might ask? What is a more rigorous question you might ask? Karin Hess

Depth + Thinking Level 1 Recall & Reproduction Level 2 Skills & Concepts Level 3 Strategic Thinking/ Reasoning Level 4 Extended Thinking Remember What color was Red’s cape? Who is this story about? Understand Apply Identify words/phrases that helped you to know the sequence of events in the story. Analyze Is this a realistic or fantasy story? Is this a realistic or fantasy story? Justify your interpretation using text evidence. Evaluate Create Write a telephone conversation between Red and her mother to explain the wolf incident.

How can we apply these ideas back in our schools AND CLASSROOMS?

Guiding Questions What skills & concepts are most important? Is the intended rigor of skills/concepts reflected in materials used (e.g., texts, tasks) Is there a range of DOK (rigor) within the learning activities/lesson?

Bands 11- CCR K-1 Increased Ability to Use Text Evidence Standards Two through Nine Bands 11- CCR K-1 Standard One Standard Ten 39 Increasing Range and Complexity achievethecore.org

PISA Reading Unit 3: Graffiti See Page 13

Question 4: Graffiti (p. 17) Full Credit Guidelines: Explain opinion with reference to style or form of one or both letters. Refers to criteria such as structure of argument, cogency of argument, strategies for persuading readers, etc. Example: “Helga’s. She gave you lots of different points to consider and she mentioned the environmental damage that graffiti artists do which I think is very important.”

Question 4: International Performance Data Overall Percent Correct Japan75> Canada63> United Kingdom62> New Zealand58> Finland58> Australia54= OECD Average53 Korea, Republic of52= Ireland51= Poland50= Norway48= United States46< Mexico45< Country average vs. OECD average Higher> Not Different= Lower<

CCSS Requires Three Shifts in ELA/Literacy #1—Building knowledge through content-rich nonfiction #2—Reading, writing and speaking grounded in evidence from text, both literary and informational #3—Regular practice with complex texts and its academic language achievethecore.org

Academic Vocabulary Words that add to students’ language ability (e.g., maintain, fortunate, required, tend, contrast/compare, insisted, summarize) Words that are needed in a content area (e.g., isotope, peninsula, photosynthesis, cubism, isosceles triangle)

More Examples from NAEP NAEP 2009 Grade 12 Literary Passage: The Open Window by Saki Available through the NAEP Questions Tool

NAEP’s “Meaning Vocabulary” ABCD* Avg. SS% Avg. SS% Avg. SS% Avg. SS% NP CT‡928923‡ On page 3, the narrator says Framton has the delusion that total strangers are interested in his nervous condition. This means Framton A.has a common illness related to his nerves. B.is worried about what people think of him. C.dreams that he is constantly being watched. D.has a mistaken idea about how others view him.

Grade 8 Example On page 2, when the author of the biographical sketch says that E. B. White's essays "appealed to an urbane crowd," he is describing the audience for White's writing as A.having sophisticated tastes. B.leading busy lives. C.being politically conservative. D.having a good sense of humor. A*BCD Avg. SS% Avg. SS% Avg. SS% Avg. SS% NP CT ‡

The Origins of CCSS International Assessments NAEP CCSS

Discussion What are the opportunities and challenges related to the shifts? How can districts support teachers in making these important shifts? Revisit your definition of rigor – has it changed? In what ways? What existing curriculum, assessment materials, or activities will you examine for a range of cognitive rigor? What types of supports and resources can CSDE provide?

Related Resources Many papers and presentations available at or contact Karin Hess Hess, K. (2004). “Applying Webb’s Depth-of-Knowledge (DOK) Levels in social studies and science” [online]: Little Red Riding Hood handout g%20Hood-CRM_KH11.pdf g%20Hood-CRM_KH11.pdf Achievethecore.org—Student Achievement Partners provides modules for professional development and other valuable CCSS resources. Achievethecore.org

NAEP Questions Tool Online access to released NAEP items in a variety of subjects over many years Teachers use the NQT … ◦ to see models of well-designed items measuring the same content taught in their classrooms; ◦ as a starting point when building common assessments; and ◦ to reflect on the performance of the students in their classrooms in comparison to students statewide and across the nation.

PISA Resources PISA Released Items from 2000, 2003 and 2006 PISA in FocusPISA in Focus: monthly policy-oriented notes PISA Assessment Framework

Please contact us to request further information. Renee Savoie phone: Amy Radikas phone: