1. Dr. Karin Hess karinhessvt@gmail.com www.karin-hess.com
Are your students thinking more deeply or just working harder? Infusing Rigor into High-Quality Instruction & Assessment
Dr. Karin Hess

2. Session Overview Sample videos, assessments/tasks, & rubrics
Develop a shared understanding of the concept of cognitive rigor
Dispel some common DOK misconceptions
Use DOK & the Hess Cognitive Rigor Matrices to:
Examine what rigor/DOK looks like in action
Sample videos, assessments/tasks, & rubrics
Karin’s Coaching Tips: Analyzing assessment tasks/rubrics & instruction in light of current research
Provide tools & strategies for current & future work at
- Event Materials – Topic #1 –
Password: _______________________

3. Source: Teaching for Rigor: A Call for a Critical Instructional Shift Marzano & Toth (March 2014)

4. Track your reflections as we work…
Ways I am refining my thinking about DOK/rigor… ?
STRATEGIC scaffolding strategies for getting students to deeper thinking… ?

5. Before we begin…
Take a minute to jot down 2-3 words/phrases that come to mind when you think of “cognitive rigor” as it relates to instruction, learning, and/or assessment.

6. The Hess Cognitive Rigor Matrix integrates Bloom + Webb
Different states/schools/teachers use different models to describe cognitive rigor. Each addresses something different.
Bloom – What type of thinking (verbs) is needed to complete a task?
Webb – How deeply do you have to understand the content to successfully interact with it? How complex is the content?

7. Bloom’s Taxonomy [1956] & Bloom’s Cognitive Process Dimensions [2001]
Knowledge -- Define, duplicate, label, list, name, order, recognize, relate, recall
Remember Retrieve knowledge from long-term memory, recognize, recall, locate, identify
Comprehension -- Classify, describe, discuss, explain, express, identify, indicate, locate, recognize, report, review, select, translate
Understand -- Construct meaning, clarify, paraphrase, represent, translate, illustrate, give examples, classify, categorize, summarize, generalize, predict
Application -- Apply, choose, demonstrate, dramatize, employ, illustrate, interpret, practice, write
Apply -- Carry out or use a procedure in a given situation; carry out or use /apply to an unfamiliar task
Analysis -- Analyze, appraise, explain calculate, categorize, compare, criticize, discriminate, examine
Analyze -- Break into constituent parts, determine how parts relate
Synthesis -- Rearrange, assemble, collect, compose, create, design, develop, formulate, manage, write
Evaluate -- Make judgments based on criteria, check, detect inconsistencies/fallacies, critique
Evaluation -- Appraise, argue, assess, choose, compare, defend, estimate, explain, judge, predict, rate, core, select, support, value
Create -- Put elements together to form a coherent whole, reorganize elements into new patterns/ structures

8. Webb’s Depth-of-Knowledge Levels
DOK-1 – Recall & Reproduction - Recall of a fact, term, principle, concept, or perform a routine procedure
DOK-2 - Basic Application of Skills/Concepts - Use of information, conceptual knowledge, select appropriate procedures for a task, two or more steps with decision points along the way, routine problems applying 2+ concepts, organize/display data, interpret/use simple graphs
DOK-3 - Strategic Thinking - Requires reasoning, developing a plan or sequence of steps to approach problem; requires some decision making and justification; abstract, complex, or non-routine; often more than one possible answer or approach
DOK-4 - Extended Thinking - An original investigation or application to real world; requires time to research, problem solve, and process multiple conditions of the problem or task; OR non-routine manipulations, across disciplines/content areas/multiple sources

9. DOK Misconception #1: All kids can’t do this; or Kids don’t need scaffolding to get “up” there.
Engaging in “a complex task” with supports/ scaffolding is an essential step along the way to proficiency (Vygotsky’s ZPD)
Do it with others first; DOK 3 and 4 are not meant to only be done alone/independently, especially at first
Oral language & meaningful discourse support deeper thinking and increase initial exposures to the content and student engagement. This is NOT cheating!
One strategy: Plan questioning & formative probes from DOK over the course of a lesson or unit of study. Consider all DOK levels in your planning.

10. Vygotsky: Zone of Proximal Development (What a child can do with assistance today)
What a child can do independently tomorrow/future
What a child can do independently now: “ENTRY”
Actual Development Area
The ZONE
Potential Development Area
LEARNING PROGRESSIONS ZONE:
Dynamic area
Causes development to move forward
Social interaction essential (scaffolding)
Karin Hess (2008). Using learning progressions as a schema to monitor progress across grades.

11. DOK Misconception #2: Webb’s DOK model is a taxonomy
Bloom’s is a taxonomy, intended to be a hierarchy
Primary Weaknesses of Bloom: generic verbs (void of content); some of the same verbs at different levels
Webb’s DOK model is nominative:
It names how you interact with content; is content-specific
It differentiates varying levels of engagement with content and suggests what tasks might look like
DOK 4 is not better than DOK 3, or DOK 2, or DOK 1 11

12. Bloom verbs & levels = Webb DOK
DOK Misconception #3
Bloom verbs & levels = Webb DOK
The DOK “Wheel of Misfortune” implies that a DOK level is indicated by a particular verb or set of verbs.
Norman Webb, “It’s what comes after the verb, that indicates the complexity of a task (content).” 12

13. What does (cognitive) rigor mean?

14. DOK Misconception #4: DOK is about difficulty
The intended student learning outcome (not grade level) determines the DOK level. What mental processing must occur? DOK = Complexity, not difficulty!
While verbs may appear to point to a DOK level, it is what comes after the verb that is the best indicator of the rigor/DOK level and complexity of the task.
Describe the information contained in graphics or data tables in the text; or the rule for rounding a number
Describe how the two story characters are alike and different.
Describe the data or text evidence that supports your solution, your reasoning, or your conclusions
Describe varying perspectives on global climate change using supporting scientific evidence, and identify the most significant effects it might have on the planet in 100 yrs. 14

15. What is actually being assessed? What mental processing is required?
Can you define or spell erosion? (term/definitional)
Which one is an example/non-example of erosion? (conceptual)
Can you demonstrate/explain what happens during erosion? (procedural/descriptive)
Use the data to identify a trend … and some possible effects of erosion on this land mass over the next 5 years. Support your reasoning with data analysis.
What did the author mean when he said, “after that, her popularity began to erode”? Explain why you think so using your interpretation of the text? (contextual/multi- step, requiring analysis of supporting evidence)

16. “UG” = unsubstantiated generalization
The Hess Cognitive Rigor Matrix Applies Webb’s DOK to Bloom’s Cognitive Process Dimensions
Depth + Thinking
Level 1
Recall & Reproduction
Level 2
Skills & Concepts
Level 3
Strategic Thinking
Level 4
Extended Thinking
Remember
- Recall, locate basic facts, details, events
Understand
- Select appropriate words to use when intended meaning is clearly evident
- Specify, explain relationships
- summarize
– identify main ideas
- Explain, generalize, or connect ideas using supporting evidence (quote, example, data …)
- Explain how concepts or ideas specifically relate to other content domains or concepts
Apply
- Use language structure (pre/suffix) or word relationships (synonym/antonym) to determine meaning
– Use context to identify meaning of word
- Obtain and interpret information using text features
- Use concepts to solve non-routine problems
- Devise an approach among many alternatives to research a novel problem
Analyze
- Identify whether information is contained in a graph, table, text feature, etc.
– Compare literary elements, terms, facts, events
– analyze format, organization, & text structures
- Analyze or interpret author’s craft (literary devices, viewpoint, or potential bias) to critique a text
– Analyze multiple sources
- Analyze complex/abstract themes
Evaluate
“UG” = unsubstantiated generalization
– Cite evidence and develop a logical argument for conjectures
- Evaluate relevancy, accuracy, & completeness of information
Create
- Brainstorm ideas about a topic
- Generate conjectures based on observations or prior knowledge
- Synthesize information within one source, data set, or text
- Synthesize information across multiple sources or texts

18. DOK Misconception #5: All DOK levels can be assessed with a multiple choice question
That’s just dumb!
“Weaker” DOK 3 multiple choice items are possible; but does selecting the best option (e.g., locate supporting evidence for a theme) provide as much insight as seeing HOW a student formulates and reveals thinking?
By their nature, DOK 3 and 4 questions/tasks are more open-ended, generally take longer to respond to/solve, and may have more than one “appropriate right answer”
Research – when students construct an answer, they remember it longer 18 18

19. “Evidence-Based” Items may provide more insights than traditional MC

20. Performance Tasks/Projects
Instruction & Assessment Decisions…
The DOK Matrix Instructional Paths
Selected Response
Constructed Response
Each standard has an assigned Depth of Knowledge.
Performance Tasks/Projects
The DOK determines the cognitive level of instruction.
Recall, locate basic facts, definitions, details, events
Select appropriate words for use when intended meaning is clearly evident.
DOK 1
Recall and Reproduction
Remember
Understand
DOK 2
Skills and Concepts
Apply
Explain relationships
Summarize
State central idea
Use context for word meanings
Use information using text features
DOK 4
Extended Thinking
Synthesize across multiple sources/ texts
Articulate a new voice, theme, or perspective
Evaluate relevancy, accuracy and completeness of information across texts or sources
Analyze multiple sources or multiple text
Analyze complex abstract themes
Devise an approach among many alternatives to research a novel problem
-Explain how concepts or ideas specifically relate to other content domains.
Develop a complex model or approach for a given situation
Develop an alternative solution .
Create
DOK 3
Reasoning and Thinking
Analyze
Analyze or interpret author’s craft (e.g., literary devices, viewpoint, or potential bias) to critique a text
Explain, generalize or connect ideas using supporting evidence (quote, evidence, data) .
Cite evidence and develop a logical argument for conjectures based on one text or problem
Evaluate
Use concepts to solve non-routine problems and justify 20

21. DOK Misconception #6: Higher order thinking = deeper learning
What we have thought of as “higher order” (analysis, evaluation, creative) thinking might only be engaging or fun…and not always deeper
Many critical thinking examples do not go deep or ever get to DOK 3 or 4 (e.g., interpret/solve and justify)
Shift our thinking from “higher order” to deeper learning, and that can mean:
deeper understanding
deeper application
deeper analysis, etc.
The Hess CRM illustrates this shift 21

22. Some general rules of thumb…
If there is one correct answer, it is probably level DOK 1 or DOK 2
DOK 1: you either know it (can recall it, locate it, do it) or you don’t know it
DOK 2 (conceptual): apply one concept, then make a decision before going on applying a second concept; express relationship (if-then; cause-effect), making connections; HOW I did it
If more than one possible answer/approach, requiring evidence, it is DOK 3 or 4
DOK 3: Must interpret, provide supporting evidence and reasoning (not just HOW solved, but WHY it works– explain reasoning for each step/decision made)
DOK 4: all of “3” + use of multiple sources/data/ texts; initiate & complete an investigation

23. DOK is not cumulative! More = Deep [1+1+1 still =1!]
DOK Misconception #7: Multi-step or longer tasks, multiple texts, or complex texts always means deeper thinking
DOK 2 is not simply more than one step, it’s applying more than one concept; DOK 2 is still routine/typical (main idea, word problems, etc.)
Simply reading more complex texts, but NOT delving deeply into the text’s meaning/discourse style/etc., is likely to still be DOK 1 or 2
DOK 3 requires some aspect of open-endedness and interpretation with justification or support; DOK 4 = interpretation draws from analyzing multiple sources
DOK is not cumulative! More = Deep [1+1+1 still =1!] 23

24. Examining Rubrics What are the intended DOK levels?

25. Math Content Standards & Math Practices
Depth + Thinking
Level 1
Level 2
Level 3
Level 4
Remember
Know math facts, terms, principles
Understand
Attend to precision
Evaluate expressions, locate/plot points
Represent math relationships
Model with mathematics
Estimate, predict, observe, explain relationships
Construct viable arguments
Geometry proof
Integrate concepts across math domains or content areas
Apply
Calculate, measure, make conversions
Use formulas
Make sense of routine problems
Make sense of non-routine problems
Design & conduct a project
Analyze
Identify a pattern
Locate information in table, diagram
Use tools strategically
Classify, organize data, extend a pattern
Reason abstractly
Generalize a pattern
Analyze multiple sources of
Evidence to solve problem
Evaluate
Critique the reasoning of others
Create
Design a model for a new perspective
Design a complex model with multiple constraints

26. DOK examples from different content areas

27. UG - Which team, artist, play, etc. … is the best?
Depth + Thinking
Level 1
Level 2
Level 3
Level 4
Remember
What is slope?
What is negative space?
Understand
Read, write, and represent these fractions
Describe why negative space is used.
Which term ____?
Explain how you solved this problem.
Why control variables in the investigation?
Find examples of…
Construct a math argument to show equivalence using area, set, and linear models
Interpret the theme.
Apply
Convert fraction to a decimal
Add these fractions
Organize these data to support your solution (e.g., graph, diagram)
Conduct the investigation, interpret results, and support conclusions with data
Design & conduct an investigation, based on a new question raised.
Analyze
What kind of graph or model is this?
Which data point shows ____?
Is this a …
Compare these methods/mediums.
Which graph shows how the data would be displayed?
Are there design flaws? Justify your interpretation using what you know about fair tests.
Analyze more than one product, drawing from multiple source materials for the analyses
Evaluate
UG - Which team, artist, play, etc. … is the best?
How would you rank these ___? Justify your rankings using data that supports your criteria.
Some say the NFL settlement for player brain injury is not adequate. Evaluate both sides using data to determine the validity of this claim.
Create
How would you demonstrate this technique?
Create an equivalent fractions. card game
Create scenario supported by the data in display.
Integrate multiple source materials with intent to develop a novel product 27

28. For each assessment task (and rubric)…ask
What is its purpose? (What understandings - content/skill-are being assessed? is there a ‘right’ answer?)
What is the implied/intended rigor? (What mental processing would you expect students to engage in? Use the CRM Tools to find descriptors.)
Which standards does it REALLY elicit and assess? (content + intended rigor)
Does the scoring guide/rubric match content + intended rigor?
What would student responses tell a teacher if students could/could not do all or part of this task? (open-ended tasks, reasoning used) – next instructional decisions are clear

29. Hess Formative Assessment Analysis Tool #10 What will this assessment uncover?

30. Hess’ Performance Assessment Validation Tool #9A

31. Take-Away Messages: Cognitive Rigor & Some Implications for Assessment
Begin with daily DOK3 classroom discourse!
Assessing only at the highest DOK level (the “ceiling”) will miss opportunities to know what students do & don’t know – go for a range; end “high” in selected/prioritized content
Performance assessments can offer varying levels of DOK embedded in a larger, more complex (“rich”) tasks
Planned formative assessment strategies and tools can/should focus on differing DOK levels

32. Cognitive Rigor: Shifting Teacher & Student Roles © Karin Hess, 2013, Linking Research with Practice
DOK Levels
Teacher Role
Student Role 1
Questions to focus attention (Who? What? Where? How? When?)
Directs, leads, demonstrates, defines
Acquires vocabulary, facts, rules
Memorizes, recites, quotes
Practices, restates 2
Questions to differentiate/ classify, draw out inferences, check conceptual understanding (Why? What conditions? Give example/non-example?)
Explains relationships, sorts, classifies, compares, organizes
Makes predictions based on estimates, observations; proposes 3
Questions to probe reasoning and underlying thinking (How do you know? What is the hard evidence?
Uncovers relevant, accurate, credible information or flaws in a design
Develops supporting (hard) evidence for conclusions or claims
Tests ideas, solves non-routine problems 4
Questions to extend thinking, explore alternative sources, broaden perspectives (What are the potential biases? Can you propose an alternative model?
Initiates, transfers, and constructs knowledge
Modifies, creates, elaborates based on multiple sources
Investigates real-world problems and issues

33. Some Related Resources
New York City Department of Ed website – Common Core library: preK-12 assessments in ELA and mathematics, some of which Karin helped to develop and pilot
National Math Science initiative – gr 3 – HS; also has ELA and SS performance tasks with DOK designations -
Read Works - gr k-8; short literary & informational texts with CC Qs
Jim Burke – Middle & High School ELA resources
Dan Meyer blog – “3-act” math PAs for ES-MS-HS; kids build the problems by deciding what’s needed to solve them – good strategic thinking required;
Links and resources on
Norman Webb video clip – on how he developed the DOK model
Karin’s YouTube videos
Karin’s blog, newsletter, classroom poster set - “Rock the Rigor” & many other resources