Presentation on theme: "Karin K. Hess, Ed.D. www.karin-hess.com Tools for Supporting Increased Rigor & High-Quality Assessment and Instruction Karin K. Hess, Ed.D. www.karin-hess.com."— Presentation transcript:
1 Karin K. Hess, Ed.D. www.karin-hess.com Tools for Supporting Increased Rigor & High-Quality Assessment and InstructionKarin K. Hess, Ed.D.
2 Presentation Overview Develop a shared understanding of the concept of cognitive rigor/DOKDispel some common DOK misconceptionsUse DOK & the Hess Cognitive Rigor Matrices to:Examine what rigor/DOK looks like in actionSample videos, assessments/tasks, & rubricsKarin’s Coaching Tips: Analyzing assessment tasks/rubrics & instruction in light of current researchProvide tools & strategies to plan future work & support to teachers and students
3 Track your reflections as we work… Ways I am refining my thinking about DOK/rigor…?Scaffolding strategies for getting students to deeper thinking…?
4 Before we begin…Take a minute to jot down words/phrases that come to mind when you think of “cognitive rigor” as it relates to instruction, learning, and/or assessment.Share at tables – what’s the same? What’s different?
5 Let’s apply your rigor definitions Your class has just read some version of Little Red Riding Hood.What is a basic comprehension question you might ask?What is a more rigorous question you might ask?Set questions aside for now. Which type of question was easier to write? What kind of thinking was required by you to come up with it? Which is easier to “score”?
6 The Hess Cognitive Rigor Matrix integrates Bloom + Webb Different states/schools/teachers use different models to describe cognitive rigor. Each addresses something different.Bloom – What type of thinking (verbs) is needed to complete a task?Webb – How deeply do you have to understand the content to successfully interact with it? How complex is the content?The next few slides provide background of the Hess CRM development.
7 Merging Bloom + Webb: The thinking behind the development of the Hess Cognitive Rigor Matrix …
8 Bloom’s Taxonomy [1956 ] & Bloom’s Cognitive Process Dimensions  Knowledge -- Define, duplicate, label, list, name, order, recognize, relate, recallRemember Retrieve knowledge from long-term memory, recognize, recall, locate, identifyComprehension -- Classify, describe, discuss, explain, express, identify, indicate, locate, recognize, report, review, select, translateUnderstand -- Construct meaning, clarify, paraphrase, represent, translate, illustrate, give examples, classify, categorize, summarize, generalize, predict…Application -- Apply, choose, demonstrate, dramatize, employ, illustrate, interpret, practice, writeApply -- Carry out or use a procedure in a given situation; carry out or use /apply to an unfamiliar taskAnalysis -- Analyze, appraise, explain calculate, categorize, compare, criticize, discriminate, examineAnalyze -- Break into constituent parts, determine how parts relateSynthesis -- Rearrange, assemble, collect, compose, create, design, develop, formulate, manage, writeEvaluate -- Make judgments based on criteria, check, detect inconsistencies/fallacies, critiqueEvaluation -- Appraise, argue, assess, choose, compare, defend, estimate, explain, judge, predict, rate, core, select, support, valueCreate -- Put elements together to form a coherent whole, reorganize elements into new patterns/ structuresA little history of how Bloom has been revised. Notice how some verbs in 1956 version are at multiple levels and how some verbs have shifted to different levels in the 2001 taxonomy, now called cognitive process dimensions. There are still some issues with using Bloom, alone…
9 Webb’s Depth-of-Knowledge Levels DOK-1 – Recall & Reproduction - Recall of a fact, term, principle, concept, or perform a routine procedureDOK-2 - Basic Application of Skills/Concepts - Use of information, conceptual knowledge, select appropriate procedures for a task, two or more steps with decision points along the way, routine problems applying 2+ concepts, organize/display data, interpret/use simple graphsDOK-3 - Strategic Thinking - Requires reasoning, developing a plan or sequence of steps to approach problem; requires some decision making and justification; abstract, complex, or non-routine; often more than one possible answer or approachDOK-4 - Extended Thinking - An original investigation or application to real world; requires time to research, problem solve, and process multiple conditions of the problem or task; non-routine manipulations, across disciplines/content areas/multiple sourcesOverview of WebbSome examples of each of these: “right there questions” /defining a concept or term/routine math skills – add, measure (DOK 1); “think and search”/ using or organizing information/making observations/summarizing (DOK 2); “author and me”/author’s craft with justification for reasoning/science conclusions supported by data/proposition with support for reasoning; analyzing and synthesizing multiple sources or concepts/designing your own science investigation and conducting it/ research paper DOK 4)
10 How would you describe some of the differences between Bloom & Webb? Turn & talk…How would you describe some of the differences between Bloom & Webb?
11 DOK Misconception #1: All kids can’t do this; or Kids don’t need scaffolding to get “up” there. Engaging in “a complex task” with supports/ scaffolding is an essential step along the way to proficiency (Vygotsky’s ZPD)Do it with others first; DOK 3 and 4 are not meant to only be done alone/independently, especially at firstOral language & meaningful discourse support deeper thinking and increase initial exposures to the content and student engagement. This is NOT cheating!One strategy: Plan questioning & formative probes from DOK over the course of a lesson or unit of study. Consider all DOK levels in your planning.Some workshop planning tools: DOK unit templates, quick tips for differentiation, unit planning for writing prompts
12 Vygotsky: Zone of Proximal Development (What a child can do with assistance today) What a child can do independently tomorrow/futureWhat a child can do independently now: “ENTRY”Actual Development AreaThe ZONEPotential Development AreaLEARNING PROGRESSIONS ZONE:Dynamic areaCauses development to move forwardSocial interaction essential (scaffolding)Karin Hess (2008). Using learning progressions as a schema to monitor progress across grades.
13 DOK Misconception #2: Webb’s DOK model is a taxonomy Bloom’s is a taxonomy, intended to be a hierarchyPrimary Weaknesses of Bloom: generic verbs (void of content); same verbs at different levelsWebb’s DOK model is nominative:It names how you interact with content; is content-specificIt differentiates varying levels of engagement with content and suggests what tasks might look likeDOK 4 is not better than DOK 3, or DOK 2, or DOK 1
14 DOK Misconception #3: Bloom verbs & levels = Webb DOK The DOK “Wheel of Misfortune” implies that a DOK level is indicated by a particular verb or set of verbs.Norman Webb, “It’s what comes after the verb, that indicates the complexity of a task.”
15 DOK Misconception #4: DOK is about difficulty. The intended student learning outcome determines the DOK level. What mental processing must occur? DOK = Complexity, not difficulty!While verbs may appear to point to a DOK level, it is what comes after the verb that is the best indicator of the rigor/DOK level and complexity of the task.Describe the information contained in graphics or data tables in the text; or the rule for rounding a numberDescribe how the two story characters are alike and different.Describe the data or text evidence that supports your solution, reasoning, or conclusionsDescribe varying perspectives on global climate change using supporting scientific evidence, and identify the most significant effects it might have on the planet in 100 yrs.I have to remind people about this MANY times! The first is DOK 1, then DOK 2, DOK 3 and DOK 4 –discuss why the verb “describe” is not enough to determine complexity. “Hard” DOK 2 tasks, may not be deep, just really hard to do
16 Recall and Reproduction Bloom’s Taxonomy + Webb’s DOK = the Hess CRM DOK LEVEL 1Recall and ReproductionDOK 2Skills and ConceptsDOK LEVEL 3ReasoningDOK 4Extended ThinkingRememberRecall, locate basic facts, definitions, details and eventsUnderstandApplyAnalyzeEvaluateCreateSelect appropriate word when intended meaning is clearExplain relationshipsSummarizeCentral ideasExplain, generalize or connect ideas using supporting evidence (quote, text, evidence, data, etc.)Explain how concepts relate to other content domainsUse language structure or word relationships (synonyms/ antonyms)Use context to find meaningObtain and use information in text featuresUse concepts to solve non-routine problems and justify solutions with evidenceDevise an approach among alternatives to research a novel problemIdentify information in a graphic, table, visual, etc.Compare literary elements, facts, terms and events.Analyze format, organization & text structuresAnalyze or interpret author’s craft (e.g., literary devices, viewpoint, or potential bias) to critique a textAnalyze multiple sources or textsAnalyze complex abstract themes.“UGs”Cite evidence and develop a logical argument for conjectures based on one text or problem.Evaluate relevancy, accuracy and completeness of informationAt first, I thought that DOK would line up nicely with Bloom levels – NOT TRUE! Higher order is not always deep thinking!Brainstorm ideas, concepts, problems, or perspectives related to a topicGenerate conjectures or hypotheses based on observations or prior knowledgeDevelop a complex model or approach for a given situationDevelop an alternative solutionSynthesize information across multiple sourcesArticulate a new voice, theme, perspective.Bloom’s Taxonomy + Webb’s DOK = the Hess CRM
17 “UG” = unsubstantiated generalization The Hess Cognitive Rigor Matrix Applies Webb’s DOK to Bloom’s Cognitive Process DimensionsDepth + ThinkingLevel 1Recall & ReproductionLevel 2Skills & ConceptsLevel 3Strategic ThinkingLevel 4Extended ThinkingRemember- Recall, locate basic facts, details, eventsUnderstand- Select appropriate words to use when intended meaning is clearly evident- Specify, explain relationships- summarize– identify main ideas- Explain, generalize, or connect ideas using supporting evidence (quote, example, data …)- Explain how concepts or ideas specifically relate to other content domains or conceptsApply- Use language structure (pre/suffix) or word relationships (synonym/antonym) to determine meaning– Use context to identify meaning of word- Obtain and interpret information using text features- Use concepts to solve non-routine problems- Devise an approach among many alternatives to research a novel problemAnalyze- Identify whether information is contained in a graph, table, text feature, etc.– Compare literary elements, terms, facts, events– analyze format, organization, & text structures- Analyze or interpret author’s craft (literary devices, viewpoint, or potential bias) to critique a text– Analyze multiple sources- Analyze complex/abstract themesEvaluate“UG” = unsubstantiated generalization– Cite evidence and develop a logical argument for conjectures- Evaluate relevancy, accuracy, & completeness of informationCreate- Brainstorm ideas about a topic- Generate conjectures based on observations or prior knowledge- Synthesize information within one source, data set, or text- Synthesize information across multiple sources or textsDepth is on the “diagonal”Thus the Hess CRM was developed with descriptors for each content area – content-specific descriptors are important to determining task complexity
18 DOK Misconception #5: All DOK levels can be assessed with a multiple choice question That’s just dumb!Weak DOK 3 multiple choice items are possible; but does selecting the best option (e.g., locate supporting evidence for a theme) provide as much insight as seeing HOW a student formulates and reveals thinking?By their nature, DOK 3 and 4 questions/tasks are more open-ended, generally take longer to respond to/solve, and may have more than one “appropriate right answer”
19 Recall and Reproduction Reasoning and Thinking The DOK Matrix Instructional PathsInstruction & Assessment Decisions…Selected ResponseConstructed ResponseEach standard has an assigned Depth of Knowledge.Performance TasksThe DOK determines the cognitive level of instruction.Recall, locate basic facts, definitions, details, eventsSelect appropriate words for use when intended meaning is clearly evident.DOK 1Recall and ReproductionRememberUnderstandDOK 2Skills and ConceptsApplyExplain relationshipsSummarizeState central ideaUse context for word meaningsUse information using text featuresDOK 4Extended ThinkingSynthesize across multiple sources/ textsArticulate a new voice, theme, or perspectiveEvaluate relevancy, accuracy and completeness of information across texts or sourcesAnalyze multiple sources or multiple textAnalyze complex abstract themesDevise an approach among many alternatives to research a novel problem-Explain how concepts or ideas specifically relate to other content domains.Develop a complex model or approach for a given situationDevelop an alternative solution.CreateDOK 3Reasoning and ThinkingAnalyzeAnalyze or interpret author’s craft (e.g., literary devices, viewpoint, or potential bias) to critique a textExplain, generalize or connect ideas using supporting evidence (quote, evidence, data).Cite evidence and develop a logical argument for conjectures based on one text or problemEvaluateUse concepts to solve non-routine problems and justifySelected response works well for DOK 1 & 2 questions…move to constructed response and performance tasks for DOK 3 and 4
20 Let’s practice using the CRM-back to Little Red Riding Hood Your sample questions – a basic and more rigorous questionHandout: Linking Research with Practice, Tool #1: CRM template for Close Reading (page 4)Handout:CRM template – where do your questions fall? Creative, but not deep, for example? No evidence required?
21 Depth + Thinking Remember Apply Analyze Evaluate Create Understand Level 1Recall & ReproductionLevel 2Skills & ConceptsLevel 3Strategic Thinking/ ReasoningLevel 4Extended ThinkingRememberWhat color was Red’s cape?Who is this story about?UnderstandWho are the main characters?What was the story’s setting?Retell or summarize the story in your own words.What is the author’s message or theme? Justify your interpretation using text evidence.ApplyIdentify words/phrases that helped you to know the sequence of events in the story.AnalyzeIs this a realistic or fantasy story?Compare the wolf character to the character of Red. How are they alike-different?Is this a realistic or fantasy story? Justify your interpretation using text evidence.Are all wolves (in literature) like the wolf in this story? Support your response using evidence from this and other texts.EvaluateWhat is your opinion about the cleverness of the wolf? Justify your opinion using text evidence.Which version has the most satisfying ending? (establish criteria first, then locate evidence)CreateWrite text messages between Red & her mother explaining the wolf incident.
22 Some other content examples… Your class will be learning about…Fractions/decimals; Data useData use; scientific investigationElements of art & principles of designCome up with a basic understanding and more rigorous question you could pose.Use a CRM Template to Plan Units
23 Depth + Thinking Remember Apply Analyze Evaluate Create Understand Level 1Recall & ReproductionLevel 2Skills & ConceptsLevel 3Strategic Thinking/ ReasoningLevel 4Extended ThinkingRememberWhat is slope?What is white space?UnderstandRead, write, and represent these fractionsDescribe why white space is used.Explain how you solved this problem.Why control variables in the investigation?Construct an argument to show equivalence using area, set, and linear modelsApplyConvert this fraction to a decimalAdd these fractionsUse these data to graph your solutionConduct the investigation, interpret results, and support conclusions with dataAnalyzeWhat kind of graph or model is this?Which data point shows ____?Find examples…Compare these methods.Which graph shows how the data would be displayed?Interpret what was happening in the event? Justify your interpretation using what you know about slope.Analyze more than one product(same time period, medium,theme drawing from multiple contexts source materials for the analysesEvaluateUG - Which team is the best?How would you rank these ___? Justify your rankings using data that supports your criteria.Some say the NFL settlement for player brain injury is not adequate. Evaluate both sides using data to determine the validity of this claim.CreateHow would you demonstrate each technique?Create a card game using fractions.Create scenario explained by a data display.Integrate multiple source materials with intent to develop a product
24 DOK Misconception #6: Higher order thinking = deeper learning What we have thought of as “higher order” (analysis, evaluation, creative thinking) might only be engaging or fun…and not always deeperMany critical thinking examples do not go deep or get to DOK 3 or 4 (e.g., interpret/solve and justify)Shift our thinking from “higher order” to deeper learning, and that can mean:deeper understandingdeeper applicationdeeper analysis, etc.The Hess CRM illustrates this shift
25 Some general rules of thumb… If there is one correct answer, it is probably level DOK 1 or DOK 2DOK 1: you either know it (can recall it, locate it, do it) or you don’t know itDOK 2 (conceptual): apply one concept, then make a decision before going on applying a second concept; express relationship (if-then; cause-effect)If more than one answer/approach, requiring evidence, it is DOK 3 or 4DOK 3: Must interpret, provide supporting evidence and reasoning (not just HOW solved, but WHY it works– explain reasoning for each step/decision made)DOK 4: all of “3” + use of multiple sources/data/ texts; initiate & complete an investigation
26 DOK Misconception #7: Multi-step or longer tasks, multiple texts, or complex texts always means deeper thinkingDOK 2 is not simply more than one step, it’s applying more than one concept; DOK 2 is still routine/typical (main idea, word problems, etc.)Simply reading more complex texts, but NOT delving deeply into the text’s meaning/style/etc., is likely to still be DOK 1 or 2DOK 3 requires some aspect of open-endedness and interpretation with justification or support; DOK 4 = drawing from multiple sources
27 Where is more-less rigor implied in any set of curriculum standards?
28 Reading Standards Depth + Thinking Remember Understand Apply Analyze Level 1Recall & ReproductionLevel 2Skills & ConceptsLevel 3Strategic ThinkingLevel 4Extended ThinkingRememberKEY DETAILSDecode, read orallyUnderstandWORD MEANINGS- fill inCENTRAL IDEASSUMMARIZEpredict, inferREASONING & SUPPORT – interpret theme, purpose, pt of view/perspectiveREASONING & SUPPORT–multiple textsApplyWORD STRUCTURE -roots, affixes, structure,RELATIONSHIPSsynonyms-antonymsWORD MEANINGS-use in contextUSE TEXT STRUCTURES & FEATURESTEXT STRUCTURES & FEATURES – how is the message structured/ presented?-compare treatmentsAnalyzeUSE of TEXT STRUCTURE or FEATURESLANGUAGE USE-identify non- literal usageANALYSIS & REASONING WITHIN TEXTSLANGUAGE USE-interpret non- literal usageANALYSIS & REASONING ACROSS TEXTSEvaluateAUTHOR’s CRAFT WITHIN A TEXT (e.g., language use impact/intent; bias, reasoning)EVALUATE AUTHOR’s PURPOSE or CRAFT ACROSS TEXTSCreate
29 Reading & Writing Depth + Thinking Remember Understand Apply Analyze Level 1Recall & ReproductionLevel 2Skills & ConceptsLevel 3Strategic ThinkingLevel 4Extended ThinkingRememberKEY DETAILSUnderstandWORD MEANINGS- fill inCENTRAL IDEASSUMMARIZEpredict, inferREASONING & SUPPORT – DEVELOP theme, or pt of view/ perspective topicREASONING & SUPPORT– use multiple texts - compare or elaborateApplyWORD STRUCTURERELATIONSHIPSEDIT/CLARIFYUSE TECHNOLOGYCITE SOURCESWORD MEANINGS-LANGUAGE USEUSE OF TEXT STRUCTURES & FEATURESINTEGRATE TEXT STRUCTURES & FEATURES into CompositionsAnalyzeUSE of TEXT STRUCTURES or FEATURESLANGUAGE USEANALYSIS & REASONING WITHIN TEXTS – RESEARCH for WritingDevelop reasoningANALYSIS & REASONING ACROSS TEXTSRESEARCH for Writing;Comparing themesEvaluateAUTHOR’s CRAFT WITHIN A TEXTEvaluate credibility of sourcesEVALUATE AUTHOR’s PURPOSE or CRAFT ACROSS TEXTSCreateWRITE/EDIT BRIEF TEXTSCOMPOSE /REVISE FULL TEXTSCOMPOSE FULL TEXTS-sourcesThese also relate to SBAC Assessment targets for Reading and Writing
30 Math Content Standards & Math Practices Depth + ThinkingLevel 1Recall & ReproductionLevel 2Skills & ConceptsLevel 3Strategic Thinking (support with data, equations, models, etc.)Level 4Extended Thinking (cross domains)RememberKnow math facts, termsUnderstandAttend to precisionEvaluate expressions, plot pointModel with mathematicsEstimate, predict, observe, explain relationshipsConstruct viable argumentsGeometry proofIntegrate concepts across domainsApplyCalculate, measure, make conversionsMake sense of routine problemsMake sense of non-routine problemsDesign & conduct a projectAnalyzeIdentify a patternLocate information in tableUse tools strategicallyClassify, organize data, extend a patternReason abstractlyGeneralize a patternAnalyze multiple sources ofevidenceEvaluateCritique the reasoning of othersCreateDesign a complex modelMost math content standards are really DOK 1 or DOK 2 – it is 4 of the 8 math practices applied to math content that can ramp it up to DOK 3 – e.g., construct arguments; DOK 4 is more of a longer-term project
31 For each assessment task (or rubric)…ask What is its purpose? (What content/skill is being assessed? is there a ‘right’ answer?)What is the implied/intended rigor? (What mental processing would you expect students to engage in? Use the CRM to find descriptors.)Which standards does it REALLY assess? (content + intended rigor)Does the scoring guide/rubric match content + intended rigor?What would student responses tell a teacher if students could/could not do all or part of this task? (open-ended tasks, reasoning used) – next instructional decisions are clear
32 Take-Away Message: Cognitive Rigor & Some Implications for Assessment Begin with daily DOK3 classroom discourse!Assessing only at the highest DOK level (the “ceiling”) will miss opportunities to know what students do & don’t know – go for a range; end “high” in selected/prioritized contentPerformance assessments can offer varying levels of DOK embedded in a larger, more complex taskPlanned formative assessment strategies and tools can/should focus on differing DOK levels
33 Some resources worth exploring… – national math science initiative – gr 3 – HS; also has ELA and SS performance tasks with DOK designations– gr k-8; short literary & informational texts with CC Qs- LA gr 3-HS math & ELA tasks (some math samples are weak and not all math DOK levels are correct, but generally good examples)- GA gr K-5 math– sample math PAs for k-12; science PAs for k-8Dan Meyer blog – math PAs for MS-HS; kids have to build the problems by deciding what’s needed to solve them – good strategic thinking requiredHess & Gong (2014). Ready for college and career? Achieving the Common Core Standards and beyond through deeper, student-centered learningLinks and resources onKarin’s YouTube video – “Text Complexity Tools” (qualitative text analysis)Karin’s vimeo DOK video – (NYC Dept of Ed Common Core Library)Karin’s YouTube DOK video – excerpt from CT workshop