1. To do Check to see if they can read questions on screen Check Internet
Check videos

2. Creating Text-Dependent Constructed Response Questions
~ Training of Trainers ~
Penny Plavala, School Improvement Specialist
Multnomah ESD

3. Please access:

4. Training of Trainer Goals
Provide training resources to support two one-hour sessions: 1. Creating Text-Dependent CR Questions 2. Assessing Text-Dependent CR Questions
Resources are available on the PortaPortal:

5. Note
These are ideas I have used in trainings for teachers across the curriculum (not math). Use / modify activities. Create new strategies.
Training of Trainers:
Time to stop along the way and process how
you might use a piece with your staff.

6. If you think of a way to modify an activity,
Let’s learn together!
If you think of a way to modify an activity,
please share your idea.

7. Session One Goals
• Refresher: Smarter Balanced Assessment Components • Focus on Constructed Response Questions • Create Questions for a Classroom Formative Assessment
Instructional design:
direct instruction, work independently,
with a partner, and in small groups
Please locate a partner

8. Warm-Up Tell your partner what you know –
or have heard about Constructed Response or Text Dependent questions.

9. Why Are We Here?
This spring, students will answer Constructed Response questions on the Smarter Balanced Assessment. This is a new item type for Oregon students.
By integrating CR questions into our classroom assessments, students will be familiar with this type of item.
We will focus today on Text-Dependent
Constructed Response questions.

10. The SBAC Assessment System
Prepared by the Center for K-12 Assessment & Performance Management at ETS
4/21/2017
The SBAC Assessment System
English Language Arts and Mathematics, Grades 3 – 8 and High School
11th Grade: Last 6 weeks
Last 12 weeks of year*
DIGITAL CLEARINGHOUSE of formative tools, processes and exemplars; released items and tasks; model curriculum units; educator training; professional development tools and resources; an interactive reporting system; scorer training modules; and teacher collaboration tools.
Computer Adaptive Assessment and Performance Tasks
INTERIM ASSESSMENT
Computer Adaptive Assessment and Performance Tasks
INTERIM ASSESSMENT
PERFORMANCE
TASKS
Reading
Writing
Math
COMPUTER
ADAPTIVE ASSESSMENT
Scope, sequence, number, and timing of interim assessments locally determined
CR Questions on Both Parts of Test
Optional Interim assessment system —
no stakes
Summative assessment for accountability
* Time windows may be adjusted based on results from the research agenda and final implementation decisions.

11. What Will Be Tested? Place text here Claim 1 Reading Claim 2 Writing
Students can read closely and analytically to comprehend a range of increasingly complex literary and informational texts.
Claim 1
Reading
Students can produce effective and well-grounded writing for a range of purposes and audiences.
Claim 2
Writing
Students can employ effective speaking and listening skills for a range of purposes and audiences.
Claim 3
Speaking & Listening
Students can engage in research/inquiry to investigate topics, and to analyze, integrate, and present information.
Claim 4
Research
Place text here

12. Assessment Item Types Constructed Response (CR) Selected Response (SR)
Variety of multiple choice
Constructed Response (CR)
Short answer using textual evidence
Technology Enhanced (TE)
Technology embedded into items
Performance Tasks (PT)
Use higher level thinking skills; integrate reading, writing, and listening

13. The following items are from the SBAC Practice Test 2013.
Our Focus Today
Require the student to generate a response as opposed to selecting a response.
Include both short and extended responses.
Allow students to demonstrate their use of complex thinking skills consistent with the expectations for college and career readiness.
Constructed Response
The following items are from the SBAC Practice Test 2013.

14. Components of a Constructed Response Item
The Shepherd’s Boy and the Wolf
A Shepherd's Boy was tending his flock near a village, and thought it would be great fun to trick the villagers by pretending that a Wolf was attacking the sheep: so he shouted out, "Wolf! Wolf!" and when the people came running up he laughed at them because they believed him. He did this more than once, and every time the villagers found they had been tricked, for there was no Wolf at all. At last a Wolf really did come, and the Boy cried, "Wolf! Wolf!" as loud as he could: but the people were so used to hearing him call that they took no notice of his cries for help. And so no one came to help the boy, and the Wolf attacked the
In a few sentences, explain what lesson the reader can learn from the shepherd’s boy. Use details from the story to support your response.
STIMULUS
(Reading Passage)
STEM
In a few sentences, explain what lesson the reader can learn from the shepherd’s boy. Use details from the story to support your response.
All constructed response items are worth 2 to 4 points.
{+}
Let’s take a look at an example of a 2 point constructed response item. Like selected response items,
Constructed response items have a stimulus
and a stem.
SPACE FOR ANSWER

15. Constructed Response
What are some ways in which the Mexican free-tails are unique among bat species?
Use at least two details from the presentation to support your answer.
Type your answer in the space provided.
7th Grade

16. Constructed Response Read the sentence and the directions that follow.
The hermit crabs in the ocean have learned to adapt to the changing housing situation.
Using details from the text, define the word adapt and explain how the crabs have adapted.
Type your answer in the space provided.
6th grade

17. Constructed Response
There are physical health benefits associated with meditation.
Using at least two details from the text, support this conclusion
Type your answer in the space provided.
11th Grade

18. Constructed Response
Summarize the central idea in the section “Ups and Downs”
Use key evidence from the text to support your summary.
Type your answer in the space provided.
7th Grade

19. Scoring a Constructed Response Item
2-point Use Evidence Rubric 2
The response gives sufficient evidence of the ability to cite evidence to support arguments and/or ideas. 1
The response gives limited evidence of the ability to cite evidence to support arguments and/or ideas.
The response gets no credit if it provides no evidence of the ability to cite evidence to support arguments and/or ideas.
All constructed response items also must include
{+}
a scoring rubric,
scoring notes,
and sample responses.
The scoring rubric for each item distinguishes between characteristics of responses that provide evidence that the student has partially or fully developed the skill or knowledge defined by the assessment target.
The scoring notes detail the information that should be included in a correct response.
The sample responses provide concrete examples of what a response for each point value might look like. The top score sample should showcase a complete and thorough response. The language contained in samples should model what is expected from a student at the grade level being assessed.
Conventions are not scored on Constructed Response questions
Smarter Balanced Assessment Consortium

20. Constructed Response Questions
Students can read closely and analytically to comprehend a range of increasingly complex literary and informational texts.
Claim 1
Reading
Students can engage in research/inquiry to investigate topics, and to analyze, integrate, and present information.
Claim 4
Research
Place text here

21. Turn and Talk
What skills do students need to answer Constructed Response questions?
When do students complete short answer questions in your classroom?

22. Trainer Check-In Considering the prior knowledge of your staff…
What elements of the SBAC and CR overview might you use in the training introduction?
What may you revise?

23. After students read a passage in your classroom, how do you check for understanding?
Answer questions during whole-class discussion
Quick write – summarize key ideas (exit slip)
Thumbs up, sideways, down
Answer text-dependent constructed response questions

24. Text-Dependent Constructed Response Questions
• Definition
• Examples
• Components
• Practice creating questions for your classroom
CR = Constructed
Response

25. Text-Dependent CR Questions:
• are open ended, short answer questions that measure
application-level skills as well as content knowledge.
• can assess higher level thinking:
- comparisons, contrasts, causes, effects, changes
- identify patterns or conflicting points of view
- categorize or summarize information
- state a generalization, conclusion, explanation or
prediction……using evidence from the text.

26. Text-Dependent CR Questions:
• use a range of primary and secondary stimuli and
authentic "real world" examples including short
reading passages, time lines, maps, graphs, cartoons,
charts.
• are graded against specific criterion using a rubric.

27. Text-Dependent CR Question
Example: “Some animals would not survive if they were unable to use tools.” What two pieces of evidence from the article would you use to support this statement?

28. require Text-Based Answers Text-Dependent Questions
Teachers write questions that keep students in the text. Students shouldn’t have to go outside the text for answers.
“Stay within four corners of text”.
1 minute:
The ODE document has titled this shift Text-based Answers, and in the end we want students to give answers that are text-based and support their thinking using evidence from the text. That practice begins with asking good quality text-dependent questions that are interesting enough to keep kids in the four corners of the text. It takes much thought and careful planning.
We’re going to talk in a minute about what impact this might have on instruction and teacher preparation. What we know for sure…(next slide)

29. Quality Text-Dependent Questions
Can be answered by careful close reading
Do not depend on outside sources
Require more than recalling facts
Often require students to infer
Require time for students to process
1 minute:
SO, what are the parameters for text-based questions?
Text-dependent…students must have read the text to answer the questions.
Recalling facts is an important starting point, but most go beyond.
When a student makes an inference, they must support their finding with evidence from the text.

30. Constructed Response Questions:
Please take a moment with your partner to look over the examples from the Smarter Balanced Assessment Consortium.
These questions were taken from the Grade 6-12 Practice Performance Tasks:
What do you notice?
• Look closely at the verbs. Circle them!
• Locate verbs on Bloom’s list

31. Report Out: (I type their verbs) explain support with details describe use both sources analyze list Where do we see these landing on Bloom’s list? Encourage a variety of verbs across the levels.

32. What are Text Dependent Questions (2 min.)
Training Option
What are Text Dependent Questions (2 min.)

33. Outstanding New Resource
Text-Dependent Questions: Pathways to
Close and Critical Reading, Grades 6-12
• Creating Effective Close Reading Lessons
- What does the text say?
- How does the text work?
- What does the text mean?
• Sample passages with questions
• Classroom videos

34. Types of Text-Dependent Questions
Examples • Follow along on pink sheet + One side: English teachers ~ short story + Other side: all other teachers ~ historical speech • Examples increase in difficulty
Note: All questions would include a second sentence asking students to support their answer with evidence from the text.

35. Build Understanding of the Meaning
Sequence of information
Determine importance of ideas
Find details that support main ideas
Answers who, what, when, where, why, how much, or how many.

36. • Why is Susan B. Anthony delivering
this speech?
• Why was she arrested?
• Who is the narrator? How do
you know?

37. Vocabulary and Text Structure
Bridges literal and inferential meanings
Denotation
Connotation
Shades of meaning
Figurative language
How organization contributes to meaning

38. Examine the length of her speech. Is it sufficient for her purpose?
What does Poe mean by the word “mad”?
How does the author’s use of symbolism impact the story?

39. Author’s Purpose Genre: Entertain? Explain? Inform? Persuade?
• Narrator: Who tells the story?
Point of view: First-person, third-person limited, omniscient, unreliable narrator

40. What does the right to vote represent to Susan B. Anthony
What does the right to vote represent to Susan B. Anthony? Where specifically does she emphasize the importance of voting for women?
How does the narrator’s retelling of the events lead the reader to understand his state of mind?

41. Inference Questions
• Use details and examples in the text when drawing inferences. • Details in text + what you know = inference

42. Based solely on this speech, what type of person is Susan B. Anthony?
Why does the narrator declare his guilt
at the end of the story?

43. Are These Text-Dependent CR Questions?
Look up at screen to view the sample questions.
Read the question.
Talk to partner to determine if this is an example of a text-dependent question.
• Discuss why or why not?

44. From poem “Paul Revere’s Ride” by Henry W. Longfellow
How does the author express a rising call for action in the final four lines of the poem?
How does the author portray Revere as a protector of his community? Cite examples from the text.
Do you think it is fair for people to protest against the government?
• Many historians argue the accuracy of this poem. Do you think students should be expected to read this poem if it contains inaccurate details?

45. From other texts:
In “Letter from Birmingham Jail”, Dr. King discusses nonviolent protest. Discuss a time you wanted to fight against something that you felt was unfair.
After reading Dr. King’s letter, what can you infer about the content of the letter he received? Use details from the text to support your answer.
In the poem “Casey at the Bat”, how does the author make Casey’s experiences realistic in stanza four?
• In the poem, Casey strikes out. Describe a time you failed at something.

46. Trainer Check-In Look back at the last few slides reviewing:
Levels of questions
Sample questions
What elements might you use in a teacher training?
What might you revise?

47. How do we create Text-Dependent Constructed Response questions?

48. Stop!
The next slides and activities are for this training of trainers only.
We will get back to teacher training in a minute.

49. Visual Options Powtoon (3 min.) Writing Text-Dependent Questions•
Powtoon (2:26 min.) Creating Text-Dependent Questions: Five Easy Steps •

50. Practice Activity
• Engage NY Project: - reading passages - sample questions - scored student work • 8th grade passage: - “Brain Birds: Amazing Crows and Ravens” - Word Count Lexile 1150

51. SBAC Word Counts and Lexile Targets
Grade
Word Count Range
(short text)
(long text) 3
200 – 487
488 – 650 4
450 – 562
563 – 750 5 6
650 – 712
713 – 950 7 8 11
800 – 825
826 – 1100
Grade
CCSS Lexile Band 3
420 – 820
4 – 5
740 – 1010
6 – 8
925 – 1185
9 – 10
1050 – 1335
11-12
1185 – 1385

52. Individual & Partner Activity
On your own, read the article “Brain Birds”.
As you read, think of possible CR questions you
might write for students.
3. With your partner, write 2 T-D CR questions.
Resources:
SBAC sample questions
Pink sheet sample questions
Prompts for TDQ’s

53. Report Out in Pairs
• Share one of the two questions you created.

54. Back to the Teacher Session
We are moving into a min work session where teachers will create questions using the text they brought from their classroom.
They will give this text and questions to students before Session Two.

55. How do we create Text-Dependent Constructed Response questions?

56. A Constructed Response Question: - from handout -
• Clearly tells students what they are going to do.
• Uses simple, but authentic vocabulary and good sentence
structure.
• Identifies the information or materials that students
should use when preparing their response. (article, video, chart, etc.)
• Clearly indicates the process that should be demonstrated.
(explain, describe, analyze, evaluate, etc.)
• Identifies key words by using bold text as needed.
• Add: Requires students to use evidence from the text to support
their answers.

57. Facilitated Work Session
• Work alone or with a partner who has the same reading passage. • Work quietly to not disturb others. • Leave today with an assessment to use immediately in your classroom.

58. Directions for Creating CR Questions
1. Review the reading passage independently to get a good
understanding of the content.
2. Create two text-dependent Constructed Response
questions for this reading passage, and write them on
notebook paper/device.
Resources
• SBAC sample questions
• Levels of questions
• Prompts

59. Time to Share • Get together with another person or pair and share the
questions you created.
• Discuss the questions according to the criteria on your
handout.
Trainers:
Or, teachers can do the activity on the next slide... depending on remaining time and background knowledge.
Report Out!

60. Evaluating and Refining Your Questions
Does the question have value and is it worthy of students’ time?
Is the question text dependent, requiring understanding of the text and the use of textual evidence?
Is the question aligned to and reflective of the rigor of the CCSS?
Review questions according to the three qualities on your handout.
Make needed revisions.

61. Next Steps: Baseline Assessment
• Prepare the passage and questions for student use.
• Give the reading passage and questions to students in one
class by _______.
• Bring a set of student answers to our session on _______.
• We will create rubrics and score the answers.

62. Capturing Our Work • Please place all materials in your folder.
• Bring this folder to our next session

63. Please thank your partner and
small group members
for their good work.

64. Trainer Check-In Look back at the last few slides reviewing:
Creating questions for reading passage
Evaluating and refining questions
What elements might you use in a training?
What might you revise?

66. Trainer: Partner Talk • Share the passage you brought today.
• Would this passage render two TDQ’s?
Brainstorm:
What are the characteristics of a passage teachers should bring to the first session?
Report Out!

67. Session Two Goals
• Refresher: Text-Dependent Constructed Response Questions • Create CR Rubric and Expected Answer • Score Student Work
Please locate a partner

68. Smarter Balanced Assessment
Require the student to generate a response as opposed to selecting a response.
Include both short and extended responses.
Allow students to demonstrate their use of complex thinking skills consistent with the expectations for college and career readiness.
Constructed Response
CR questions are are Text-Dependent when students have to support their answer with evidence from the reading passage.
Trainers: After the first session, ask teachers to provide you with sample questions you can share at this second session.

69. Scoring a Constructed Response Item
Analyze / Integrate Information Rubric 2
The response gives sufficient evidence of the ability to analyze and integrate information within and among multiple sources. 1
The response gives limited evidence of the ability to analyze and integrate information within and among multiple sources.
The response gets no credit if it provides no evidence of the ability to analyze and integrate information within and among multiple sources.
• Only uses one source as support
All constructed response items also must include
{+}
a scoring rubric,
scoring notes,
and sample responses.
The scoring rubric for each item distinguishes between characteristics of responses that provide evidence that the student has partially or fully developed the skill or knowledge defined by the assessment target.
The scoring notes detail the information that should be included in a correct response.
The sample responses provide concrete examples of what a response for each point value might look like. The top score sample should showcase a complete and thorough response. The language contained in samples should model what is expected from a student at the grade level being assessed.
Sample Rubrics: SBAC Sample Question packet from Session One

70. Text-Dependent CR Question
Sample Question: How does the author use data to show sugar consumption is a problem to overall health? Provide two examples of evidence from the article to support your answer.
“Sweet Sacrifice”
(Article on curbing sugar intake)
Special Ed Teachers
Gladstone High School

71. Text-Dependent CR Question
Sample Question: Looking at Source #1 under the heading “We Can Make A Difference”, describe one action and explain how it can reduce the effects of climate change.
“Climate Change”
(Article from EPA)
Science Teachers
Gladstone High School

72. Text-Dependent CR Question
Sample Question 1: In the scene on pages 49-54, at the end of Act 1, analyze how the authors develop either the conflict between Bert Cates and Rachel or the conflict between Cates and the society of Hillsboro. Use examples of the text to support your answer.
Kris Schuberg, English Teacher
Gladstone High School

73. Text-Dependent CR Question
Sample Question 2: In the same scene, explain how the authors develop Drummond’s character as a pariah in the town of Hillsboro. Cite a line by Drummond or Rachel as evidence to support your answer.
Kris Schuberg, English Teacher
Gladstone High School

74. Steps to Creating a Rubric
• Choose one of the questions you wrote.
• On the template, create a rubric for the skill being
assessed. (analyze, explain, compare, etc.)
Look back at Sample Questions

75. Text-Dependent CR Question
Sample Question 1: In the scene on pages 49-54, at the end of Act 1, analyze how the authors develop either the conflict between Bert Cates and Rachel or the conflict between Cates and the society of Hillsboro. Use examples of the text to support your answer.
Kris Schuberg, English Teacher
Gladstone High School

76. Text-Dependent CR Question
Sample Question 2: In the same scene, explain how the authors develop Drummond’s character as a pariah in the town of Hillsboro. Cite a line by Drummond or Rachel as evidence to support your answer.

77. Rubric for Inherit the Wind Question
Analyze / Integrate Information Rubric 2
The response gives sufficient evidence of the ability to explain how the authors develop Drummond’s character as a pariah. 1
The response gives limited evidence of the ability to explain how the authors develop Drummond’s character as a pariah.
The response gets no credit if it provides no evidence of the ability to explain how the authors develop Drummond’s character as a pariah
• Gives general not specific information
All constructed response items also must include
{+}
a scoring rubric,
scoring notes,
and sample responses.
The scoring rubric for each item distinguishes between characteristics of responses that provide evidence that the student has partially or fully developed the skill or knowledge defined by the assessment target.
The scoring notes detail the information that should be included in a correct response.
The sample responses provide concrete examples of what a response for each point value might look like. The top score sample should showcase a complete and thorough response. The language contained in samples should model what is expected from a student at the grade level being assessed.

78. Create a Rubric • Choose one of the questions you wrote.
• On the template, create a rubric for the skill being
assessed. (analyze, explain, compare, etc.)
• Make notes on what would separate score points.

79. Expected Answer for Score of 2
Sample Question 2: The answer should have information that shows the qualities that make Drummond a pariah or make him understand what it means to be a pariah. The answer should also include directly quoted text to support the analysis.
Kris Schuberg, English Teacher
Gladstone High School

80. Record the Expected Answer
• What is the expected answer for a score of 2?
• Write down on template:
- the elements of the 2 score answer – or –
- a sample answer your student might write

81. Stop!
The next slides and activities are for this training of trainers only.
We will get back to teacher training in a minute.

82. From “Brain Birds” Article
Sample Question:
Explain how crows and ravens use their intelligence to help them find, capture, and eat food. Use two details from the article to support your answer.
Measures CCSS RI.8.1
Cite the textual evidence that most strongly supports an analysis of what the text says explicitly as well as inferences drawn from the text.

83. “Brain Birds” Article
1. On the template, create a rubric for the skill being
assessed. (analyze, explain, compare, etc.)
2. Create the expected answer for the score of 2.
Explain how crows and ravens use their intelligence to help them find, capture, and eat food. Use two details from the article to support your answer.

84. Note
I would only have the teachers create a rubric once during the session – not repeat for a second question – or during another training.
The idea is to have them see what the differences are between the score point. They would not necessarily have to create a rubric for every question they wrote – but recording the expected score of 2 might make sense.
How could the “rubric writing” activity be improved?

85. Scoring Student Work • Take out the student work you brought today.
• Use the rubric and expected answer as resources
when you score the CR question.
• Write the score of 2, 1, or 0 on the student’s paper or
on post-it notes.

86. Scoring Student Work
• After you score 5-8 papers, start jotting down the
trends you are seeing in the student work: (notebook
paper)
* What are students doing well?
* What skills need strengthening?
• Continue scoring and tracking trends.

87. Time to Share • Get together with another person or pair and share the
trends you noticed.
• What instruction is needed to boost the skills needed to
answer CR questions?
Report Out!

88. Trainers • Look at the ivory handout: Brain Birds • Student Work
• Imagine these are ½ sheets with student answers to
the question we reviewed.
• On your own, starting with Student E., score the
students work according to the rubric and expected
answer you created.
When you see your partner is finished scoring,
quietly start discussing scores.

89. Trainers Look at scores from Engage NY project.
During your training session, teachers with the same reading passage could exchange a few papers and cross-score, then discuss student work.
Teachers could reflect on Trends notes – and discuss how they might teach the key skills students need for success.

90. Baseline Formative Assessment
• Keep the scores of your student work as a baseline.
• Use results to determine the skills you need to teach
students.
• Create mini lessons with your team using text from your
curriculum.
• Create the next round of CR questions.
• Score student work and look for growth
from the baseline assessment.

91. Trainer Check-In Look back at the last few slides reviewing:
Creating rubrics and expected answer
Scoring student work
What elements might you use in a training?
What might you revise?

92. Trainers: Next Steps for CR Questions
Session 3 Ideas:
• Create mini lessons to teach key skills (modeling):
- putting part of question in answer
- supporting answer with evidence from the text
- analyzing, synthesizing, describing, explaining, etc.
What do these skills look like across the content areas?
• Create Mini Performance Task using
two pieces of text (see purple handout)

93. Please thank your partner and
small group members
for their good work.