1. Depth of Knowledge (DOK)
What are students being asked to do?
How complex is the student thinking?
Overview:
In this module, participants will explore Norman Webb’s Depth of Knowledge (DOK) levels and Cognitive Rigor (Karin Hess) to gain a better understanding of rigorous questions. Webb’s DOK Levels are adopted by Smarter Balanced to communicate complexity of CCSS Assessment items They will practice using these tools to evaluate the rigor of questions, adjust the level of rigor where appropriate, and consider implications for their daily practice.
Outcome:
Participants will understand rigor as defined in the Depth of Knowledge framework and will be able to use the Cognitive Rigor Matrix to evaluate and adjust the rigor of questions.
Guiding Questions:
What makes a question rigorous?
How do rigorous questions support the transition to the Common Core Learning Standards (CCLS)?
Approximate time: 1 ½ to 2 hours

2. Depth of Knowledge (DOK) is
A scale of cognitive demand (thinking) to align standards with assessments
Descriptive of the task students are being asked to do
Based on the research of Norman Webb, University of Wisconsin Center for Education Research and the National Institute for Science Education
Not the same as difficulty
Introduction
Introduce the presentation by stating that:
A key component of the transition to the Common Core State Standards (CCSS) is strengthening student work by examining and refining curriculum, assessment, and classroom instruction. Rigorous questions, tasks, and assessment are essential to driving the higher-order thinking and skills students need to be college- and career-ready.
The Depth of Knowledge is a framework that will guide educators in developing rigorous lessons and assessments to prepare students for accessing the Common Core. It is being used by Smarter Balanced Assessment Consortium (SBAC) in the development of CCSS assessments. The DOK Scale is from 1-4 with 4 being the most rigorous.
Participants will become familiar with the DOK framework, practice using it to evaluate questions, and consider how to incorporate rigorous questioning in classroom practice.

3. Why Depth of Knowledge?
To align instructional activities with the cognitive rigor needed to master the standards
Is used by Smarter Balanced to indicate the complexity of the assessment task/item
Applicable to all subject areas and at all grade levels
Smarter Balanced Assessment Consortium is using Webb’s DOK to develop the Common Core Assessment
The new CCSS assessments will have questions at all DOK levels
DOK is how deeply the content must be know in order to complete the task

4. Varying Level of Cognitive Demand
ELA
Retell or summarize Little Red Riding Hood in your own words.
What is your opinion about the intelligence of the wolf (in Little Red Riding Hood)? Justify using details/evidence from the story.
Ask participants why these two questions have different Cognitive Demand
What makes one question more rigorous than the other?
What information does this provide the teacher about student understanding?

5. Varying Level of Cognitive Demand
Mathematics
Is 4/5 closer to 1 than 5/4?
4/5 is closer to 1 than 5/4. Show why this is true on a number line.
Ask participants why these two questions have different Cognitive Demand
What makes one question more rigorous than the other?
What information does this provide the teacher about student understanding?

6. How are the new assessments changing?
Smarter Balanced Depth of Knowledge Goals for Items
MATHEMATICS
ELA / LITERACY
DOK 3
DOK 4
Current Assessments
<2% 0%
20% 2%
New SBAC Assessments
49%
21%
43%
25%
This shows how the new assessments will compare in Cognitive Rigor to the “old” CSTs. The percentage indicates the amount of time spent on questions, not the number of questions.
Yuan & Le (2012); Herman & Linn (2013),
from Linda Darling-Hammond Assembly Testimony,

7. Sample Item Specification
SBAC has item specifications that indicate the DOK level. The DOK is one of many specifications. Difficulty is another one – this example is Medium difficulty
This example is a DOK 3 ELA

8. Sample Item Specification
SBAC has item specifications that indicate the DOK level. The DOK is one of many specifications. Difficulty is another one – this example is Medium difficulty
This example is a DOK 2 Math

9. Share your questions with a partner
Write a definition of Cognitive Rigor on your DOK note-taking guide.
Read “The Ant and the Grasshopper”– develop one basic question and one “rigorous” question.
Share your questions with a partner
Materials:
DOK Note Taking Guide
Ant and the Grasshopper
Ask participants to write their own definitions of cognitive rigor in their note-taking guides and then share their definitions with the group. Participants should keep these definitions in mind throughout the activity and note any shifts in their understanding. ( minutes)
Instruct participants to read “The Ant and the Grasshopper” and to develop two questions based on the story: one basic comprehension question and one question they consider rigorous. Participants should record their questions in their note-taking guides. (5 – 10 minutes)
Ask participants to share their questions. Let them know they will examine these questions later in the activity using the Depth of Knowledge framework. (5 minutes)
INFORMATION for the Trainer:
Karen Hess defines cognitive rigor in this way: “Cognitive rigor encompasses the complexity of content, the cognitive engagement
with that content, and the scope of the planned learning activities”
Rigor and Depth of Knowledge
four myths about rigor in the classroom.
1. Lots of homework is a sign of rigor.
2. Rigor means doing more.
3. Rigor is not for everyone.
4. Providing support means lessening rigor.
Instead, “rigor is creating an environment in which each student is expected to learn at high levels, each student is supported so he or she can learn at high levels, and each student demonstrates learning at high levels.”

10. Rating Depth of Knowledge (DOK)
What is the Level of Cognitive Demand
Needed To Complete the Task?
Materials:
DOK Circle Chart
Stress to participants that DOK is the level of Cognitive Demand needed to complete the task (not the difficulty)
Have participants examine the DOK Chart – What DOK Level is the majority of the instructional time spent? Which DOK Level(s) are assessments focused on? What type of classroom activities do your students engage in for the DOK levels?
What level would you assign to your Ant and the Grasshopper questions?

11. In your classroom:
What part of your classroom instruction is spent in each DOK Level?
Which DOK levels are assessments focused on?
What lessons / activities engage students in the different DOK Levels?
What DOK level would you assign your “Ant and The Grasshopper” questions?
Have participants examine the DOK Chart – What DOK Level is the majority of the instructional time spent? Which DOK Level(s) are assessments focused on? What type of classroom activities do your students engage in for the DOK levels?

12. Webb’s 4 Levels of Cognitive Complexity
DOK 1: Recall & Reproduction
DOK 2: Basic Application of Skills/Concepts
DOK 3: Strategic Thinking and Reasoning
DOK 4: Extended Thinking
The Depth of Knowledge framework consists of four levels that define what students must know about a concept or topic to complete an assignment or answer a question.
The Depth of Knowledge levels do not necessarily reflect degrees of “difficulty,” but the rigor of thinking students must use to solve problems.
For example, memorizing the definitions of 100 vocabulary words might be difficult, but not complex. Completing 100 long division problems might be difficult, but not complex.

13. LEVEL 1: RECALL
Requires recall of information, such as a fact, definition, term, or simple procedure and/or performing simple procedures.
Involves only basic initial comprehension, not an analysis or interpretation.
Students are to work with specific facts or definitions. The activity requires a shallow understanding of the topic or text.
This slide gives examples of qualifiers for DOK Level 1

14. LEVEL 2: Skills and Concepts
Includes the engagement of some mental processing beyond recalling or reproducing a response
Items require students to make some decisions as to how to approach the question or problem
Actions imply more than one mental or cognitive process/step
This slide gives examples of qualifiers for DOK Level 2

15. Recall vs. Skills & Concepts
ELA
Describe the physical features of a place.
Identify and summarize the major events, problem, solution, and conflicts in a text.
DOK 1
DOK 2
This slide compares content at a DOK Level 1 and a DOK 2

16. Recall vs. Skills & Concepts
Selena’s
MATH
How many minutes did it take Selena to do 3 problems?
Selena continues to work at the same rate. How many problems will she do in 40 minutes?
DOK 1
DOK 2
This slide compares content at a DOK Level 1 and a DOK 2

17. Recall vs. Skills & Concepts
Science
Identify parts of the cell
Distinguish between a plant cell and an animal cell
DOK 1
DOK 2
This slide compares content at a DOK Level 1 and a DOK 2
For DOK 1 and DOK 2 there is usually one right answer

18. LEVEL 3: Strategic Thinking
Requires deep understanding exhibited through planning, using evidence, and more demanding cognitive reasoning
The cognitive demands are complex and abstract
An assessment item that has more than one possible answer and requires students to justify the response would most likely be a Level 3
This slide gives examples of qualifiers for DOK Level 3

19. Level 3 3. Strategic Thinking ELA
Analyze or evaluate the effectiveness of literary elements (plot, setting, conflict, point-of view …).
This slide gives an example of content at a DOK Level 3
Level 3

20. Level 3 3. Strategic Thinking Mathematics
Solve a multi-step problem, and provide support with a mathematical explanation that justifies the answer.
Beto and Jose were playing a game. At the end of each game, the loser gave the winner a penny. After a while, Jose won 3 games and Beto had 3 more pennies than he did when he began. How many games did they play.
This slide gives an example of content at a DOK Level 3
Level 3

21. Extended Thinking: Level 4
Requires high cognitive demand and is very complex
Students are expected to make connections, relate ideas within the content or among content areas, and select or devise one approach among many alternatives on how the situation can be solved
Due to the complexity of cognitive demand, DOK 4 often requires an extended period of time
This slide gives examples of qualifiers for DOK Level 4

22. Level 4 Extended Thinking ELA/Social Science
Write an analysis of an issue. Present multiple points of view and argue for a position in a way that could stimulate the read to consider new perspectives.
Given a problem situation, research and present multiple solutions. Be sure to provide alternative responses and reasons for adopting each.
This slide gives an example of content at a DOK Level 4

23. Level 4 Extended Thinking ELA/Science
Construct an explanation based on evidence for how the availability of natural resources, occurrence of natural hazards, and changes in climate have influenced human activity. (NGSS HS-ESS3-1.)
This slide gives an example of content at a DOK Level 4
This is a Next Generation Science Standard – NGSS – High School – Earth and Space Science

24. Extended Thinking Math
Level 4
Materials:
Packaging Cans SBAC Released Item
This slide gives an example of content at a DOK Level 4
This is a high school performance task released by Smarter Balanced
This full item is in your handouts

25. Extending the length of an activity alone does not necessarily create rigor
Task
Thinking
DOK Level
Collecting data samples over several months
Recall 1
Organizing data in a chart
Skills / Concept 2
Using charts and graphs to make predictions and justify conclusions with evidence
Strategic Thinking 3
Developing a generalized model from data, graphs and analysis and applying it to a new situation or to solve a new problem
Extended thinking 4
This slide demonstrates how an activity over time can be rated at each of the DOK Levels
Caution that time itself is not an indicator of a DOK Level 4

26. DOK an Introduction How does your original definition of
Take a few minutes to read through the descriptions or bullets for the levels of DOK 1-4
Materials: Choose appropriate reading for the participants
DOK Reading
DOK Writing
DOK Math
Math Table
DOK Science
Science Table
DOK Social Science
DOK Art
Have participants read the handouts in DOK and revise their definition on the note-taking guide
Have participants share their insights, thoughts or questions.
How does your original definition of
Cognitive Rigor Change?

27. Rate the Examples Level of DOK
Try it!
Rate the Examples Level of DOK
Compare and contrast desert and tropical climates.
Gather, analyze, organize and interpret data from multiple sources and draft a reasoned response.
Use a dictionary to define the meaning of words
Explain the cause-effect of historical events. 2 1 4 2
The DOK levels will appear one by one
Have participants read the prompt and choose a DOK Level
Participants should show their response so facilitator can assess understanding.
Discuss why the prompt is rated at that DOK level

28. Rate the Examples Level of DOK
Try it!
Rate the Examples Level of DOK
Identify the elements in a specific compound
Solve multi-step problems and provide support with mathematical explanation that justifies the answer. 1
Determine the area of a triangle given a drawing or labels.
Evaluate the effectiveness of different exercises and diets on the health of a population. 2 3 3
The DOK levels will appear one by one
Have participants read the prompt and choose a DOK Level
Participants should show their response so facilitator can assess understanding.
Discuss why the prompt is rated at that DOK level

29. Try it!
Rate the Level of DOK
What is your opinion about the intelligence of the Wolf in Little Red Ridding Hood? Justify your answer using evidence from the story.
Analyze and explain the multiple perspectives or issues within or across time periods, events, or cultures. 2
Classify plane and 3-D figures.
Specify a problem, identify solution paths, solve the problem and report the results.
What fruit did the witch give to Snow White? 3 4
The DOK levels will appear one by one
Have participants read the prompt and choose a DOK Level
Participants should show their response so facilitator can assess understanding.
Discuss why the prompt is rated at that DOK level 4 1

30. Sort the problems by DOK Levels
Choose a set of cards (ELA or Math)
With your partner, sort the 8 cards based on the DOK level
Compare your results with your table partners
Materials:
DOK Cards (ELA :K-5, ELA :6-8, Math:K-5, Math:6-8)
Answer key
Pass out the DOK Cards.
Have teams sort the cards (can be done individually) and justify why they choose specific DOK Levels
Have tables self correct their responses (pass out answer sheet after they complete the sort)
Have the tables debrief on “Aha’s” or difficulty they are still having
Note: The cards are SBAC Release items – questions may be DOK 1, but hard… Participants should note that the assessments are changing!

31. Have participants think about their own curriculum – how can they develop learning opportunities at each level?

32. Do it….
Look at the basic question and rigorous one that you developed on for “The Ant and The Grasshopper”
Can you create questions for all four DOK levels?

33. Depth of Knowledge
The depth of knowledge level is NOT determined by the verb, but by the context in which the verb is used and the complexity of thinking required.
How is this different from Bloom’s?
What comes after the verb is the best indicator of the rigor/DOK level
California’s CCSS: Toolkit adapted by the Stanislaus County Office of Education

34. Same Verb – 3 Different Levels of DOK
Caution 1
Same Verb – 3 Different Levels of DOK
DOK 1- Describe three characteristics of metamorphic rocks. (Requires simple recall)
DOK 2- Describe the difference between metamorphic and igneous rocks. (Requires cognitive processing to determine the differences in the two rock types)
DOK 3- Describe a model that you might use to represent the relationships that exist within the rock cycle. (Requires deep understanding of rock cycle and a determination of how best to represent it)
What comes after the verb is the best indicator of the rigor/DOK level
This example uses the word Describe in relationship to the rock cycle

35. One Statement Across DOK Levels
Recall & Reproduction
(DOK 1)
Skills & Concepts/Basic Reasoning (DOK 2)
Strategic Thinking/ Complex Reasoning (DOK 3)
Extended Thinking/ Reasoning
(DOK 4)
All four use the verb identify.
Students will identify information in a passage that is supported by fact.
Students will identify the appropriateness of an argument using supporting evidence.
Students will identify essential information needed to accomplish a task.
Students will identify interrelationships (themes, ideas, concepts) developed in more than one literary work.
Another example of one statement – different DOK

36. One Statement Across DOK Levels
Recall & Reproduction
(DOK 1)
Skills & Concepts/Basic Reasoning (DOK 2)
Strategic Thinking/ Complex Reasoning (DOK 3)
Extended Thinking/ Reasoning
(DOK 4)
Students will extend patterns, find the missing term(s) in a pattern or describe rules for patterns (numbers, pictures, tables, words from real-world and mathematical problems (Grade 5 – DOK 3).
Draw the next figure in the following pattern:
Find the next three terms in the pattern, determine the rule for finding the next number in the pattern and make or find a model for the pattern: 1, 1, 2, 3, 5, 8, 13, 21, 34, …
Find the next three terms in the following pattern: 2/7, 4/7, 6/7, 8/7, ….
Find the next three terms in the pattern and determine the rule for the following pattern of numbers: 1, 4, 8, 11, 15, 18, 22, 25, 29, …
Another example of one statement – different DOK

37. DOK is about intended outcome not about Difficulty
DOK is a reference to the complexity of mental processing that must occur to answer a question, perform a task, or generate a product.
Adding is a mental process
Knowing the rule for adding is the intended outcome that influences the DOK
Once someone learns the “rule” of how to add, is DOK 1 and is also easy
Adding 4,678, ,578,885 is still a DOK 1 but may be more “difficult”
Think Jeopardy – questions are difficult, but not mentally complex

38. Points to Consider DOK 1 + DOK 1 + DOK 1 = 1
Depths of knowledge classification is based on the task, not the student
DOK is different from task/item difficulty
DOK is not a grade level indicator
DOK ratings aid in alignment of standards and assessment, and therefore instruction

39. Remember DOK is... … descriptive
… about how deeply a student has to know the content in order to respond
… NOT the same as difficulty
… NOT the same as Bloom’s Taxonomy
… NOT the same as Costa’s Levels
“Although related through their natural ties to the complexity of thought, Bloom’s Taxonomy and Webb’s depth-of-knowledge differ in scope and application. Bloom’s Taxonomy categorizes the cognitive skills required of the brain to perform a task, describing the ‘type of thinking processes’ necessary to answer a question. Depth of knowledge, on the other hand, related more closely to the depth of content understanding and scope of a learning activity, which manifests in the skills required to complete the task from inception to finale (e.g., planning, researching, drawing conclusions). Both the thinking processes and the depth of content knowledge have direct implications in curricular designs, lesson delivery and assessment development and use.” Karen Hess
Difficulty is how many students answer the question correctly
Review Video

40. Compare & Contrast CST and SBAC
Choose 4 CST questions and assign a DOK level. Cite your evidence.
Choose 4 SBAC questions and assign a DOK level. Cite your evidence.
What is the most significant difference between the CST and the SBAC Assessment?
Discuss the implications for instruction.
Materials:
ELA 3-5 CST 3-5 SBAC
ELA 6-12 CST SBAC
Math 3-5 CST 3-5 SBAC
Math 6-12 CST SBAC
Pass out CST questions and SBAC questions. Have participants explore the CST and SBAC and respond to the questions on the slide

41. How Many of Each Type of Problem?
Colorado Department of Education
What percentage of your instructional time should be spent on different DOK levels?
Classroom instruction should include all DOK levels
The majority of classroom time should be spent at DOK Levels 2 & 3
Colorado Department of Education

42. Creating DOK Questions and Activities at all levels

43. DOK 1, 2, 3, 4 Level 1—Identify the tree Level 2—Explain the function
of the leaves
Level 3—Explain how a
drought might affect the growth
of the tree
Level 4—Design an
investigation of seedling growth
to determine the best fertilizer
for this type of tree.
Examples of activities at each level on the same topic
Share this slide with participants – have them create their own questions with the following slides...
Oak Tree

44. DOK 1, 2, 3 and 4 Level 1—Identify this utensil Level 2— Level 3—
Have participants write their own questions and share them out

45. I pledge allegiance to the flag of the United States of America, and to the republic for which it stands one nation under God, indivisible, with liberty and justice for all.
Level 1—
Level 2—
Level 3—
Level 4—
Materials:
Pledge of Allegiance
Have participants write their own questions and share them out

46. DOK in the Classroom
The following slides detail how DOK applies to Classroom Instruction

47. Applying DOK to Instruction
The intended student learning outcome determines the DOK level
Assessments, oral questions and classroom activities can all be assigned DOK levels
Instruction and classroom assessments must reflect the DOK level of the objective or intended learning outcome
Teachers should think about the DOK level as they plan units, lessons and assessments

48. As you plan instructional activities, ask:
What is its purpose?
What is the implied / intended rigor?
When in the (lesson / unit) could this be used?
Which standard(s) does it correlate with?
Will student responses tell a teacher what to do next in instruction?
From Karin Hess

49. DOK: Questions at lower levels are usually more appropriate for:
Evaluating students’ preparation and comprehension
Diagnosing students’ strengths and weaknesses
Reviewing and/or summarizing
Usually questions at upper DOK levels are appropriate for:
Encouraging students to think deeply and critically
Problem solving
Encouraging Discussions
Stimulating students to seek information on their own
When to use DOK 1 and/or 2 vs DOK 3 and/or 4
From Karin Hess

50. DOK is for all students
Often struggling students are denied access to higher level questions because they still have difficulty with skills
However, higher level thinking questions are sometimes easier for struggling students to answer because open-ended questions have more entry points and require more “think time” by the entire class
Caution participants that all students need access to all levels of DOK
From Karin Hess

51. You can ask higher DOK questions
Require students to manipulate prior information
Why do you suppose…?
What can you conclude from the evidence?
Ask students to state an idea or definition in their own words
Ask questions that require a solution to a problem
Involve students in observing and describing an event or object
What do you notice? Tell me about this. What do you see?
Ask students to Compare or contrast
From Karen Hess

52. Reflect on your learning…
Revisit your definition of rigor – has it changed or been refined? How?
What is one way you might apply these ideas to your work?
How might you shift your classroom instructional and assessment practices?
What existing materials could you examine for a range of cognitive rigor?
From Karin Hess

53. Questions and Thoughts
What questions and thoughts do you still have regarding DOK?