Verona Public Schools Karin K. Hess, Ed.D. www.karin-hess.com Tools for Supporting Increased Rigor & High-Quality Assessment and Instruction Verona Public Schools Karin K. Hess, Ed.D. www.karin-hess.com

Track your reflections as we work… Ways I am refining my thinking about DOK/rigor… ? Scaffolding strategies for getting students to deeper thinking… ?

Before we begin… Take a minute to jot down words/phrases that come to mind when you think of “cognitive rigor” as it relates to instruction, learning, and/or assessment.

Let’s apply your rigor definitions Your class has just read some version of Little Red Riding Hood. What is a basic comprehension question you might ask? What is a more rigorous question you might ask? Set questions aside for now. Which type of question was easier to write? What kind of thinking was required by you to come up with it?

The Hess Cognitive Rigor Matrix integrates Bloom + Webb Different states/schools/teachers use different models to describe cognitive rigor. Each addresses something different. Bloom – What type of thinking (verbs) is needed to complete a task? Webb – How deeply do you have to understand the content to successfully interact with it? How complex is the content? The next few slides provide background of the Hess CRM development.

Merging Bloom + Webb: The thinking behind the development of the Hess Cognitive Rigor Matrix …

Bloom’s Taxonomy [1956 ] & Bloom’s Cognitive Process Dimensions [2001] Knowledge -- Define, duplicate, label, list, name, order, recognize, relate, recall Remember Retrieve knowledge from long-term memory, recognize, recall, locate, identify Comprehension -- Classify, describe, discuss, explain, express, identify, indicate, locate, recognize, report, review, select, translate Understand -- Construct meaning, clarify, paraphrase, represent, translate, illustrate, give examples, classify, categorize, summarize, generalize, predict… Application -- Apply, choose, demonstrate, dramatize, employ, illustrate, interpret, practice, write Apply -- Carry out or use a procedure in a given situation; carry out or use /apply to an unfamiliar task Analysis -- Analyze, appraise, explain calculate, categorize, compare, criticize, discriminate, examine Analyze -- Break into constituent parts, determine how parts relate Synthesis -- Rearrange, assemble, collect, compose, create, design, develop, formulate, manage, write Evaluate -- Make judgments based on criteria, check, detect inconsistencies/fallacies, critique Evaluation -- Appraise, argue, assess, choose, compare, defend, estimate, explain, judge, predict, rate, core, select, support, value Create -- Put elements together to form a coherent whole, reorganize elements into new patterns/ structures A little history of how Bloom has been revised. Notice how some verbs in 1956 version are at multiple levels and how some verbs have shifted to different levels in the 2001 taxonomy, now called cognitive process dimensions. There are still some issues with using Bloom, alone…

Webb’s Depth-of-Knowledge Levels DOK-1 – Recall & Reproduction - Recall of a fact, term, principle, concept, or perform a routine procedure DOK-2 - Basic Application of Skills/Concepts - Use of information, conceptual knowledge, select appropriate procedures for a task, two or more steps with decision points along the way, routine problems applying 2+ concepts, organize/display data, interpret/use simple graphs DOK-3 - Strategic Thinking - Requires reasoning, developing a plan or sequence of steps to approach problem; requires some decision making and justification; abstract, complex, or non-routine; often more than one possible answer or approach DOK-4 - Extended Thinking - An original investigation or application to real world; requires time to research, problem solve, and process multiple conditions of the problem or task; non-routine manipulations, across disciplines/content areas/multiple sources Overview of Webb Some examples of each of these: “right there questions” /defining a concept or term/routine math skills – add, measure (DOK 1); “think and search”/ using or organizing information/making observations/summarizing (DOK 2); “author and me”/author’s craft with justification for reasoning/science conclusions supported by data/proposition with support for reasoning; analyzing and synthesizing multiple sources or concepts/designing your own science investigation and conducting it/ research paper DOK 4)

DOK Misconception #1: All kids can’t do this; or Kids don’t need scaffolding to get “up” there. Engaging in “a complex task” with supports/ scaffolding is an essential step along the way to proficiency (Vygotsky’s ZPD) Do it with others first; DOK 3 and 4 are not meant to only be done alone/independently, especially at first Oral language & meaningful discourse support deeper thinking and increase initial exposures to the content and student engagement. This is NOT cheating! One strategy: Plan questioning & formative probes from DOK 1-2-3-4 over the course of a lesson or unit of study. Consider all DOK levels in your planning. Some workshop planning tools: DOK unit templates, quick tips for differentiation, unit planning for writing prompts

Vygotsky: Zone of Proximal Development (What a child can do with assistance today) What a child can do independently tomorrow/future What a child can do independently now: “ENTRY” Actual Development Area The ZONE Potential Development Area LEARNING PROGRESSIONS ZONE: Dynamic area Causes development to move forward Social interaction essential (scaffolding) Karin Hess (2008). Using learning progressions as a schema to monitor progress across grades.

DOK Misconception #2: Webb’s DOK model is a taxonomy Bloom’s is a taxonomy, intended to be a hierarchy Primary Weaknesses of Bloom: generic verbs (void of content); same verbs at different levels Webb’s DOK model is nominative: It names how you interact with content It differentiates varying levels of engagement with content and suggests what tasks might look like DOK 4 is not better than DOK 3, or DOK 2, or DOK 1

DOK Misconception #3: Bloom verbs & levels = Webb DOK The DOK “Wheel of Misfortune” implies that a DOK level is indicated by a particular verb or set of verbs. Norman Webb, “It’s what comes after the verb, that indicates the complexity of a task.”

DOK Misconception #4: DOK is about difficulty. The intended student learning outcome determines the DOK level. What mental processing must occur? DOK = Complexity, not difficulty! While verbs may appear to point to a DOK level, it is what comes after the verb that is the best indicator of the rigor/DOK level and complexity of the task. Describe the information contained in graphics or data tables in the text; or the rule for rounding a number Describe how the two story characters are alike and different. Describe the data or text evidence that supports your solution, reasoning, or conclusions Describe varying perspectives on global climate change using supporting scientific evidence, and identify the most significant effects it might have on the planet in 100 yrs. I have to remind people about this MANY times! The first is DOK 1, then DOK 2, DOK 3 and DOK 4 –discuss why the verb “describe” is not enough to determine complexity. “Hard” DOK 2 tasks, may not be deep, just really hard to do

Recall and Reproduction Bloom’s Taxonomy + Webb’s DOK = the Hess CRM DOK LEVEL 1 Recall and Reproduction DOK 2 Skills and Concepts DOK LEVEL 3 Reasoning DOK 4 Extended Thinking Remember Recall, locate basic facts, definitions, details and events Understand Apply Analyze Evaluate Create Select appropriate word when intended meaning is clear Explain relationships Summarize Central ideas Explain, generalize or connect ideas using supporting evidence (quote, text, evidence, data, etc.) Explain how concepts relate to other content domains Use language structure or word relationships (synonyms/anto-nyms) Use context to find meaning Obtain and use information in text features Use concepts to solve non-routine problems and justify solutions with evidence Devise an approach among alternatives to research a novel problem Identify information in a graphic, table, visual, etc. Compare literary elements, facts, terms and events. Analyze format, organization & text structures Analyze or interpret author’s craft (e.g., literary devices, viewpoint, or potential bias) to critique a text Analyze multiple sources or texts Analyze complex abstract themes. “UGs” Cite evidence and develop a logical argument for conjectures based on one text or problem. Evaluate relevancy, accuracy and completeness of information Brainstorm ideas, concepts, problems, or perspectives related to a topic Generate conjectures or hypotheses based on observations or prior knowledge Develop a complex model or approach for a given situation Develop an alternative solution Synthesize information across multiple sources Articulate a new voice, theme, perspective. Bloom’s Taxonomy + Webb’s DOK = the Hess CRM

“UG” = unsubstantiated generalization The Hess Cognitive Rigor Matrix Applies Webb’s DOK to Bloom’s Cognitive Process Dimensions Depth + Thinking Level 1 Recall & Reproduction Level 2 Skills & Concepts Level 3 Strategic Thinking Level 4 Extended Thinking Remember - Recall, locate basic facts, details, events Understand - Select appropriate words to use when intended meaning is clearly evident - Specify, explain relationships - summarize – identify main ideas - Explain, generalize, or connect ideas using supporting evidence (quote, example, data …) - Explain how concepts or ideas specifically relate to other content domains or concepts Apply - Use language structure (pre/suffix) or word relationships (synonym/antonym) to determine meaning – Use context to identify meaning of word - Obtain and interpret information using text features - Use concepts to solve non-routine problems - Devise an approach among many alternatives to research a novel problem Analyze - Identify whether information is contained in a graph, table, text feature, etc. – Compare literary elements, terms, facts, events – analyze format, organization, & text structures - Analyze or interpret author’s craft (literary devices, viewpoint, or potential bias) to critique a text – Analyze multiple sources - Analyze complex/abstract themes Evaluate “UG” = unsubstantiated generalization – Cite evidence and develop a logical argument for conjectures - Evaluate relevancy, accuracy, & completeness of information Create - Brainstorm ideas about a topic - Generate conjectures based on observations or prior knowledge - Synthesize information within one source, data set, or text - Synthesize information across multiple sources or texts Thus the Hess CRM was developed with descriptors for each content area

DOK Misconception #5: All DOK levels can be assessed with a multiple choice question That’s just dumb! Weak DOK 3 multiple choice items are possible; but does selecting the best option (e.g., locate supporting evidence for a theme) provide as much insight as seeing HOW a student formulates and reveals thinking? By their nature, DOK 3 and 4 questions/tasks are more open-ended, generally take longer to respond to/solve, and may have more than one “appropriate right answer”

Recall and Reproduction Reasoning and Thinking 2. The DOK Matrix Instructional Paths Instruction & Assessment Decisions… Selected Response Constructed Response Each standard has an assigned Depth of Knowledge. Performance Tasks The DOK determines the cognitive level of instruction. Recall, locate basic facts, definitions, details, events Select appropriate words for use when intended meaning is clearly evident. DOK 1 Recall and Reproduction Remember Understand DOK 2 Skills and Concepts Apply Explain relationships Summarize State central idea Use context for word meanings Use information using text features DOK 4 Extended Thinking Synthesize across multiple sources/ texts Articulate a new voice, theme, or perspective Evaluate relevancy, accuracy and completeness of information across texts or sources Analyze multiple sources or multiple text Analyze complex abstract themes Devise an approach among many alternatives to research a novel problem -Explain how concepts or ideas specifically relate to other content domains. Develop a complex model or approach for a given situation Develop an alternative solution . Create DOK 3 Reasoning and Thinking Analyze Analyze or interpret author’s craft (e.g., literary devices, viewpoint, or potential bias) to critique a text Explain, generalize or connect ideas using supporting evidence (quote, evidence, data) . Cite evidence and develop a logical argument for conjectures based on one text or problem Evaluate Use concepts to solve non-routine problems and justify

Let’s practice using the CRM-back to Little Red Riding Hood Your sample questions – a basic and more rigorous question Handout #2: Linking Research, Tool #1: CRM template for Close Reading (page 4) Handout:CRM template – where do your questions fall? Creative, but not deep, for example? No evidence required?

Depth + Thinking Remember Apply Analyze Evaluate Create Understand Level 1 Recall & Reproduction Level 2 Skills & Concepts Level 3 Strategic Thinking/ Reasoning Level 4 Extended Thinking Remember What color was Red’s cape? Who is this story about? Understand Who are the main characters? What was the story’s setting? Retell or summarize the story in your own words. What is the author’s message or theme? Justify your interpretation using text evidence. Apply Identify words/phrases that helped you to know the sequence of events in the story. Analyze Is this a realistic or fantasy story? Compare the wolf character to the character of Red. How are they alike-different? Is this a realistic or fantasy story? Justify your interpretation using text evidence. Are all wolves (in literature) like the wolf in this story? Support your response using evidence from this and other texts. Evaluate What is your opinion about the cleverness of the wolf? Justify your opinion using text evidence. Which version has the most satisfying ending? (establish criteria first, then locate evidence) Create Write text messages between Red & her mother explaining the wolf incident.

Some other content examples… Your class will be learning about… Fractions/decimals; Data use Data use; scientific investigation Elements of art & principles of design Come up with a basic understanding and more rigorous question you could pose. Use a CRM Template to Plan Units

Depth + Thinking Remember Apply Analyze Evaluate Create Understand Level 1 Recall & Reproduction Level 2 Skills & Concepts Level 3 Strategic Thinking/ Reasoning Level 4 Extended Thinking Remember What is slope? What is white space? Understand Read, write, and represent these fractions Describe why white space is used. Explain how you solved this problem. Why control variables in the investigation? Construct an argument to show equivalence using area, set, and linear models Apply Convert this fraction to a decimal Add these fractions Use these data to graph your solution Conduct the investigation, interpret results, and support conclusions with data Analyze What kind of graph or model is this? Which data point shows ____? Find examples… Compare these methods. Which graph shows how the data would be displayed? Interpret what was happening in the event? Justify your interpretation using what you know about slope. Analyze more than one product (same time period, medium, theme drawing from multiple contexts source materials for the analyses Evaluate UG - Which team is the best? How would you rank these ___? Justify your rankings using data that supports your criteria. Some say the NFL settlement for player brain injury is not adequate. Evaluate both sides using data to determine the validity of this claim. Create How would you demonstrate each technique? Create a card game using fractions. Create scenario explained by a data display. Integrate multiple source materials with intent to develop a product

DOK Misconception #6: Higher order thinking = deeper learning What we have thought of as “higher order” (analysis, evaluation, creative thinking) might only be engaging or fun…and not always deeper Many critical thinking examples do not go deep or get to DOK 3 or 4 (e.g., interpret/solve and justify) Shift our thinking from “higher order” to deeper learning, and that can mean: deeper understanding deeper application deeper analysis, etc. The Hess CRM illustrates this shift

Some general rules of thumb… If there is one correct answer, it is probably level DOK 1 or DOK 2 DOK 1: you either know it (can recall it, locate it, do it) or you don’t know it DOK 2 (conceptual): apply one concept, then make a decision before going on applying a second concept; express relationship (if-then; cause-effect) If more than one answer/approach, requiring evidence, it is DOK 3 or 4 DOK 3: Must interpret, provide supporting evidence and reasoning (not just HOW solved, but WHY it works– explain reasoning for each step/decision made) DOK 4: all of “3” + use of multiple sources/data/ texts; initiate & complete an investigation

DOK Misconception #7: Multi-step or longer tasks, multiple texts, or complex texts always means deeper thinking DOK 2 is not simply more than one step, it’s applying more than one concept; DOK 2 is still routine/typical (main idea, word problems, etc.) Simply reading more complex texts, but NOT delving deeply into the text’s meaning/style/etc., is likely to still be DOK 1 or 2 DOK 3 requires some aspect of open-endedness and interpretation with justification or support; DOK 4 = drawing from multiple sources

Reading Standards Depth + Thinking Remember Understand Apply Analyze Level 1 Recall & Reproduction Level 2 Skills & Concepts Level 3 Strategic Thinking Level 4 Extended Thinking Remember KEY DETAILS Decode, read orally Understand WORD MEANINGS- fill in CENTRAL IDEAS SUMMARIZE predict, infer REASONING & SUPPORT – interpret theme, purpose, pt of view/perspective REASONING & SUPPORT –multiple texts Apply WORD STRUCTURE -roots, affixes, structure, RELATIONSHIPS synonyms-antonyms WORD MEANINGS-use in context USE TEXT STRUCTURES & FEATURES TEXT STRUCTURES & FEATURES – how is the message structured/ presented? -compare treatments Analyze USE of TEXT STRUCTURE or FEATURES LANGUAGE USE-identify non- literal usage ANALYSIS & REASONING WITHIN TEXTS LANGUAGE USE-interpret non- literal usage ANALYSIS & REASONING ACROSS TEXTS Evaluate AUTHOR’s CRAFT WITHIN A TEXT (e.g., language use impact/intent; bias, reasoning) EVALUATE AUTHOR’s PURPOSE or CRAFT ACROSS TEXTS Create

Reading & Writing Depth + Thinking Remember Understand Apply Analyze Level 1 Recall & Reproduction Level 2 Skills & Concepts Level 3 Strategic Thinking Level 4 Extended Thinking Remember KEY DETAILS Understand WORD MEANINGS- fill in CENTRAL IDEAS SUMMARIZE predict, infer REASONING & SUPPORT – DEVELOP theme, or pt of view/ perspective topic REASONING & SUPPORT – use multiple texts - compare or elaborate Apply WORD STRUCTURE RELATIONSHIPS EDIT/CLARIFY USE TECHNOLOGY CITE SOURCES WORD MEANINGS-LANGUAGE USE USE OF TEXT STRUCTURES & FEATURES INTEGRATE TEXT STRUCTURES & FEATURES into Compositions Analyze USE of TEXT STRUCTURES or FEATURES LANGUAGE USE ANALYSIS & REASONING WITHIN TEXTS – RESEARCH for Writing Develop reasoning ANALYSIS & REASONING ACROSS TEXTS RESEARCH for Writing; Comparing themes Evaluate AUTHOR’s CRAFT WITHIN A TEXT Evaluate credibility of sources EVALUATE AUTHOR’s PURPOSE or CRAFT ACROSS TEXTS Create WRITE/EDIT BRIEF TEXTS COMPOSE /REVISE FULL TEXTS COMPOSE FULL TEXTS-sources

Math Content Standards & Math Practices Depth + Thinking Level 1 Recall & Reproduction Level 2 Skills & Concepts Level 3 Strategic Thinking (support with data, equations, models, etc.) Level 4 Extended Thinking (cross domains) Remember Know math facts, terms Understand Attend to precision Evaluate expressions, plot point Model with mathematics Estimate, predict, observe, explain relationships Construct viable arguments Geometry proof Integrate concepts across domains Apply Calculate, measure, make conversions Make sense of routine problems Make sense of non-routine problems Design & conduct a project Analyze Identify a pattern Locate information in table Use tools strategically Classify, organize data, extend a pattern Reason abstractly Generalize a pattern Analyze multiple sources of evidence Evaluate Critique the reasoning of others Create Design a complex model Most math content standards are really DOK 1 or DOK 2 – it is 4 of the 8 math practices applied to math content that can ramp it up to DOK 3 – e.g., construct arguments; DOK 4 is more of a longer-term project

For each assessment task (or rubric)…ask What is its purpose? (What content/skill is being assessed? is there a ‘right’ answer?) What is the implied/intended rigor? (What mental processing would you expect students to engage in? Use the CRM to find descriptors.) Which standards does it REALLY assess? (content + intended rigor) Does the scoring guide/rubric match content + intended rigor? What would student responses tell a teacher if students could/could not do all or part of this task? (open-ended tasks, reasoning used) – next instructional decisions are clear

Take-Away Messages: Cognitive Rigor & Some Implications for Assessment Begin with daily DOK3 classroom discourse! Assessing only at the highest DOK level (the “ceiling”) will miss opportunities to know what students do & don’t know – go for a range; end “high” in selected/prioritized content Performance assessments can offer varying levels of DOK embedded in a larger, more complex task Planned formative assessment strategies and tools can/should focus on differing DOK levels

Some resources worth exploring… www.nms.org – national math science initiative – gr 3 – HS; also has ELA and SS performance tasks with DOK designations www.readworks.org – gr k-8; short literary & informational texts with CC Qs http://www.louisianabelieves.com/resources/library/common-core-state-standards - LA gr 3-HS math & ELA tasks (some math samples are weak and not all math DOK levels are correct, but generally good examples) http://ccgpsmathematicsk-5.wikispaces.com/K-5+Formative+Assessment+Lessons+%28FALs%29 - GA gr K-5 math www.exemplars.com – sample math PAs for k-12; science PAs for k-8 Dan Meyer blog – math PAs for MS-HS; kids have to build the problems by deciding what’s needed to solve them – good strategic thinking required Hess & Gong (2014). Ready for college and career? Achieving the Common Core Standards and beyond through deeper, student-centered learning http://www.nmefoundation.org/resources/scl-2/ready-for-college-and-career Links on www.karin-hess.com: Karin’s YouTube video – “Text Complexity Tools” (qualitative text analysis) Karin’s vimeo DOK video – http://vimeo.com/18281415 Karin’s YouTube DOK video – excerpt from a CT workshop