Presentation on theme: "PARCC Core Leadership Group"— Presentation transcript:
1PARCC Core Leadership Group Item Review MeetingJanuary 2013Note: all items included in this presentation are for illustrative training purposes only. They are not representative of PARCC assessment items.
2Overview of Training Charge as Committee Member Purpose of PARCC Summative AssessmentsPARCC Summative Assessments (PBA, EOY)Item Review ProcessItem Review CriteriaApply Criteria 1-9 to Item Review
3Item Review Committee Charge Your role is to provide expert CONTENT review of items and tasks.You will use information provided in this item review training and will apply item review criteria to review the items.You should focus exclusively on PARCC’s item review criteria during your review.
4Item Review Committee Please also note the following: Passage review committees have already approved the passages according to PARCC content and bias/sensitivity guidelines.Bias/sensitivity item review committees will apply bias/sensitivity guidelines to all items at a different time.Other concerns identified will be placed in “the parking lot” for consideration by PARCC leadership.
5Purpose of PARCC Summative Assessments Determine whether students are college- and career-ready or on trackAssess the full range of the Common Core State Standards (CCSS) for reading, writing, and languageMeasure the full range of student performance, including the performance of high- and low-performing studentsProvide data for accountability, including measures of growthIncorporate innovative approaches throughout the system
6Performance-Based Assessment Composed of Three TasksLiterary Analysis Task (LAT)Research Simulation Task (RST)Narrative Task (NT)Eligible Item Types for Performance-Based Assessment (PBA)Evidence-Based Selected Response (EBSR)Technology-Enhanced Constructed Response (TECR)Prose-Constructed Response (PCR)
7Performance-Based Assessment A. Literary Analysis Tasks—The Literature Task plays an important role in honing students’ ability to read complex text closely, a skill that research reveals as the most significant factor differentiating college-ready from non-college-ready readers. This task will ask students to carefully consider literature worthy of close study and compose an analytic essay.B. Research Simulation Task—The Research Simulation Task is an assessment component worthy of student preparation because it asks students to exercise the career- and college- readiness skills of observation, deduction, and proper use and evaluation of evidence across text types.In this task, students will analyze an informational topic presented through several articles or multimedia stimuli, the first text being an anchor text that introduces the topic. Students will engage with the texts by answering a series of questions and synthesizing information from multiple sources in order to write two analytic essays.C. Narrative Writing Task—The Narrative Task broadens the way in which students may use this type of writing. Narrative writing can be used to convey experiences or events, real or imaginary. In this task, students may be asked to write a story, detail a scientific process, write a historical account of important figures, or to describe an account of events, scenes or objects, for example.Pg 13 Item Guidelines
8End-of-Year Assessment Focused on supporting Reading Comprehension ClaimsItem development must be informed by CCSS and Evidence StatementsEligible Item Types for End-of-Year (EOY)Evidence-Based Selected Response (EBSR)Technology-Enhanced Constructed Response (TECR)
9Evidence-Based Selected Response (EBSR) Items: Are designed to measure Reading Standard 1 and at least one other Reading Standard.May have one part.In grade 3, a one part EBSR is allowable because Reading Standard 1 evidence 1 is distinctly different from Reading Standard 1 in grades 4-11.In grades 4-11, a one part EBSR is allowable when there are multiple correct responses that elicit multiple evidences to support a generalization, conclusion or inference.Pgs Item Guidelines
10Evidence-Based Selected Response (EBSR) Items: May have multiple parts.The Item Guidelines document describes how PART A and PART B may function in a multiple part EBSR.Among things to consider:Each part must consist of a selected-response question with a minimum of four choices.In the first part, students select a correct answer among a minimum of four choices.(continued on next slide)Pgs Item Guidelines
11Evidence-Based Selected Response (EBSR) Items: In additional parts, one format requires students to select among a minimum of four choices to demonstrate the ability to locate and/or connect details/evidence from the text that explains, justifies, or applies the answer chosen for the first part of the item.There is no requirement for one-to-one alignment between Part A options and Part B options.EBSR items can meet evidence statements that specify “explain,” “provide an analysis,” etc., by selecting options that comprise explanation or analysis.
12Evidence-Based Selected Response (EBSR) Items: Variable Elements EBSR items may possess the following characteristics:For items with one correct response, four answer choices are requisite. For those items with two correct responses, six answer choices are requisite. For those items with three correct response (allowed only in grades 6-11), seven answer choices are requisite.Pgs Item Guidelines
13Evidence-Based Selected Response (EBSR) Items: For items with one correct response in Part A and one correct response in Part B, there is no partial credit. One part items do not offer partial credit.For those items with one or more correct responses in Part A and more than one correct response in Part B, partial credit should be available.To receive partial credit, students must answer Part A correctly AND select at least one correct response in Part B. This will earn the student 1 point. For these items, to receive full credit, students must answer both Part A and Part B correctly.When an item allows for more than one correct choice, each correct choice must be equally defensible.Pgs Item Guidelines
14Evidence-Based Selected Response (EBSR) Items: EBSR items can be written with an inference to be drawn in Part A and a requirement for students to find another example of how that inference applies in a different part of the text for Part B.Example 1:Part A – What character trait does Snow White reveal when Snow White does X?Part B—Which paragraph from the passage best shows additional evidence of this character trait?Example 2:Part A—What theme is revealed in the passage?Part B—Which paragraph from the passage best shows this same theme?Example 3:Part A—What is the point of view/perspective in this passage?Part B—Which paragraph from the passage best shows this same point of view/perspective (or the opposite point of view/perspective)?See the Item Guidelines document pages for other types of EBSR items.Note: all items included in this presentation are for illustrative training purposes only. They are not representative of PARCC assessment items.
15Additional Considerations for Distracters The primary purpose of a distracter is to provide evidence that a student is not able to meet the standard(s) assessed due to student misconceptions.Distracters must be plausible responses to item stems.The written style of all distracters in an item should be similar to that of the correct response(s), but need not be “perfectly parallel” in length, grammatical function, or in use of punctuation.The content of the distracters, rather than the parallelism of style, is the primary focus for distracter choices.Answer responses are not ordered alphabetically by first word or from short to long, etc. They may be ordered in any sequence as is appropriate to the content measured by the specific item.
16Additional Considerations for Distracters If answer responses are quotations or paraphrased textual evidence, place the answers in the order they appear in the passage.Particular care must be taken for Part B in EBSR items (where students are asked to select evidence from the text) such that achieving parallelism in distracters does not overly influence distracter wording.In Part B, when writing the distracters for evidences, all of the answer choices must be the same type of citation of evidence (e.g. all quotes or all paraphrases).All answer choices for Part B (distracters) must be accurate/relevant/from the passage (whether exact citations or paraphrases). All distracters must originate accurately from the text. In Part A, distracters may be written as plausible misreadings of the text.
17Sample Evidence-Based Selected Response (EBSR) Part A: What does the word “regal” mean as it is used in the passage?generousthreateningkingly*uninterestedPart B: Which of the phrases from the passage best helps the reader understand the meaning of “regal?”“wagging their tails as they awoke”“the wolves, who were shy”“their sounds and movements expressed goodwill”“with his head high and his chest out”*Here is one example of a multi-part EBSR. For other examples seeNote: all items included in this presentation are for illustrative training purposes only. They are not representative of PARCC assessment items.
18Technology-Enhanced Constructed Response (TECR) Items: TECR items must:Allow for machine scoringAward the student two (2) points for full creditBe delivered and responded to using technology, allowing for a variety of technology-enhanced student responses, including but not limited to the following:drag and drophighlighting the textannotating textother negotiated methodsPgs Item Guidelines
19Technology-Enhanced Constructed Response (TECR): TECR items may possess the following characteristics:When a TECR uses an EBSR structure (e.g. with Part A [measuring one or more of standards 2-9] and Part B [measuring standard 1]), use the same rules as applied for EBSR.Use the same guidelines for distractors for EBSR items for TECR items where applicable.For TECR items, partial credit may be offered when an item allows for partial comprehension of the texts to be demonstrated.Pgs Item Guidelines
20Sample Technology-Enhanced Constructed Response (TECR) Item Part A : Below are three claims that one could make based on the article “Earhart’s Final Resting Place Believed Found.”Highlight the claim that is supported by the most relevant and sufficient evidence within “Earhart’s Final Resting Place Believed Found.”Part B : Click on two facts within the article that best provide evidence to support the claim selected in Part A.ClaimsEarhart and Noonan lived as castaways on Nikumaroro Island.Earhart and Noonan’s plane crashed into the Pacific Ocean.People don’t really know where Earhart and Noonan died.Note: all items included in this presentation are for illustrative training purposes only. They are not representative of PARCC assessment items.
21Prose Constructed-Response (PCR): Required Elements PCR items must:Visibly align questions/tasks with specific Standards; that is, the actual language of the Standards should be used in the prompts/questionsElicit evidence(s) supporting the Sub Claim for Written Expression and the Sub Claim for Conventions and Knowledge of LanguagePgs Item Guidelines
22Prose Constructed-Response (PCR): Required Elements PCR items must:Establish a clear purpose for writing, modeling the language found in the Writing StandardsSpecify the audience to be addressedState clearly the topic, issue, or idea to be addressedReference the source text (or texts) serving as the stimulus (or stimuli) for a student responseSpecify the desired form or genre of the student responseNote: Standardized wording for PCRs is under discussion.Pgs Item Guidelines
23Prose Constructed-Response (PCR): Required Elements Elicit evidence(s) aligned with at least one Reading Standard (even when not scored for a sub claim associated with the Major Claim for Reading Complex Text)Allow students to earn partial creditPgs Item Guidelines
24Prose Constructed-Response (PCR): Required Elements In addition, prose constructed-response items must provide all students the opportunity to demonstrate a full range of sophistication and nuance in their responses.Prose constructed-response items must be designed to elicit meaningful responses on aspects of a text that may be discussed tangentially or in great detail and elaboration, thereby enabling measurement of the full range of student performance.Pgs Item Guidelines
25Prose Constructed-Response (PCR): Narrative Description Narrative writing takes two distinct forms in the PARCC assessment system:Narrative StoryNarrative Description*
26Prose Constructed-Response (PCR): Narrative Story Narrative Story – is about imagined situations and characters. It uses time as its deep structure.
27Prose Constructed-Response (PCR): Narrative Story- Scenarios Scenarios should be associated with the text and should culminate by stating the objective of the final prose constructed response for the task.Scenarios for Narrative Writing –Narrative Story Task Model :Today you will read [fill in the text type/title]. As you read, pay close attention to [fill in general focus of PCRs] as you answer the questions to prepare to write a narrative story.Note: all items included in this presentation are for illustrative training purposes only. They are not representative of PARCC assessment items.
28Sample Prose-Constructed Response (PCR) Item Use what you have learned from reading “Daedulus and Icarus” by Ovid and “To a Friend Whose Work Has Come to Triumph” by Anne Sexton to write an essay that analyzes how Icarus’s experience of flying is portrayed differently in the two texts.As a starting point, you may want to consider what is emphasized, absent, or different in the two texts, but feel free to develop your own focus for analysis.Develop your essay by providing textual evidence from both texts. Be sure to follow the conventions of standard English.Note: all items included in this presentation are for illustrative training purposes only. They are not representative of PARCC assessment items.
29Additional Considerations for Vocabulary Items Several styles for presenting vocabulary words/phrases are viable. In considering which means is best for presenting the vocabulary words/phrases, item writers should use the means that most efficiently directs students to the word/phrase in the text, while allowing students to see the full relevant context for determining the meaning of the word/phrase.The part of a vocabulary item that asks for word or phrase meaning should not use qualifiers (e.g., best, most likely: Which of the phrases from the excerpt best helps the reader understand the meaning of “XXX”?The part of a vocabulary item that asks for support may use qualifiers if needed: Which sentence from the excerpt best supports the response in Part A?Distractors should always be syntactically plausible. This is especially important in vocabulary items.
30Additional Considerations for Vocabulary Items When writing vocabulary items use the following formats:Part A - “What is the meaning of the word XXX as it is used in [paragraph 13, or line, or text]?—Part A wordingPart B - “Which of the phrases from the excerpt helps the reader understand the meaning of XXX? Unless referencing in this way creates a problem for the item. In this case, the item may require the use of a text box approach where the options for Part B come only from the excerpted text.Use of Technology Enhancement:To measure vocabulary using a drag/drop technology use the following format:“Drag the sentences from the passage into your notes that help create the meaning of the word “XXX” as it is used in the passage.”“Drag the words/phrases from the passage into your notes that help create the tone of the passage.”Note: Give a selection of sentences or words/phrases from which to choose where not all are correct.
31Item Review ProcessStep 1 - Each reviewer on your team will read the passage/text and the items independently.Step 2 - For each item, check to make sure the item meets the evidence statements/standards noted in the metadata for the item. If the item is aligned, continue to review the item. If the item is not aligned, but can be edited to align with the evidence statements/standards, note in comments how to align the item. If the item cannot be aligned, reject the item and move on to review the next item.Step 3 - Review the item in terms of other criteria on the criteria evaluation sheet. Make comments to reflect needs for revision/strengths noted.
32Item Review ProcessStep 4 - The facilitator will go item by item to determine which items are to be accepted, accepted with edits, or rejected. If all members of a group have marked an item “accepted” or “rejected,” no discussion occurs. Participants discuss those items where there is not 100% agreement.Step 5- Allow a maximum of 10 minutes per item for consensus building to determine whether to accept, accept with edits, or to reject an item. If consensus cannot be achieved, move on. The team will return to these items on the last day of the review.(continued)
33Item Review CriteriaDoes the item allow for the student to demonstrate the intended evidence statement(s) and to demonstrate the standard(s) to be measured?Is the wording of the item clear, concise, and appropriate for the intended grade level?Does the item provide sufficient information and direction for the student to respond completely?Is the item free from internal clueing and miscues?Do the graphics and stimuli included as part of the item accurately and appropriately represent the applicable content knowledge?
34Item Review CriteriaAre any graphics included as part of the item clear and appropriate for the intended grade level?If the item has a technology-based stimulus or requires a technology-based response, is the technology design effective and grade appropriate?Is the scoring guide/rubric clear, correct, and aligned with the expectations for performance that are expressed in the item or task?If the item is part of a PBA task, does it contribute to the focus and coherence of the task model?(continued)
35Criterion 1. Alignment to the CCSS and Evidence Statements Does the item allow for the student to demonstrate the intended evidence statement(s) and to demonstrate the CCSS to be measured?Refer participants to their hard copies of the evidence tables (all grade levels).The Common Core State Standards are so broad that there are multiple ways for students to demonstrate evidence that the students can meet them. These multiple ways are expressed in the evidence statements.For every grade level, there are three Evidence Tables: Reading for Information, Reading for Literature, and Vocabulary.
36Criterion 1. Alignment to the CCSS and Evidence Statements An item should:be aligned to more than one CCSS and corresponding evidence statements.Focus is no longer on a one-to-one relationship between an item and tested skillAll reading comprehension items will align to a focus standard as well as RL 1 or RI 1elicit the intended evidence(s) to demonstrate ability with the skills expressed in the standard(s).Items should focus on important aspects of the text and be worth answeringItems are intended to provide evidence to support the claims36
37Criterion 1. Alignment to the CCSS and Evidence Statements Vocabulary items should:target Tier 2 academic vocabularyWords with wide use across academic subjectsWords that are important for students to knowhave distracters in the same part of speech as the assessed wordelicit evidence from at least one language standard and from reading standard RI1 or RL1Pgs Item Guidelines
38Criterion 1. Alignment to the CCSS and Evidence Statements Note: all items included in this presentation are for illustrative training purposes only. They are not representative of PARCC assessment items.
39Criterion 2. Language Clarity and Appropriateness Is the wording of the item clear, concise,and appropriate for the intended grade level?
40Criterion 2: Language Clarity and Appropriateness Items should:convey a clearly defined task or problem in concise and direct languageuse the language of the evidence statements and standards when appropriatefocus on what is important to learn rather than on trivial contentuse a variety of approaches rather than a canned approach
41Criterion 2: Language Clarity and Appropriateness Example from Grade 10: Poor ClarityWhat was the problem that Galileo’s experiment did not solve?the influence of air pressure on the tubes’ water levelExample from Grade 10: Improved ClarityAccording to the excerpt, what problem did Galileo’s experiment fail to solve?the influence of air pressure on the glass tube’s water levelNOTE: Stress that the edits were for clarity and appropriateness. This is not preferential, but rather, focused with strong rationales.Note: all items included in this presentation are for illustrative training purposes only. They are not representative of PARCC assessment items.
42Criterion 3. Clarity of the Student Directions Does the item provide sufficient information and direction for the student to respond completely?
43Criterion 3: Clarity of the Student Directions Items should:pose the central idea in the stem and not in the answer choicesprovide directions that are specific and directrequire no background knowledge
44Criterion 3: Clarity of the Student Directions Example of Poor Clarity in Student DirectionsNotes: It’s not clear what the student is supposed to do – “choose” and “select” don’t tell the student how to answer. Should it say “click on” instead?Note: all items included in this presentation are for illustrative training purposes only. They are not representative of PARCC assessment items.
45Criterion 3: Clarity of the Student Directions Example of Improved Clarity in Student DirectionsNote: all items included in this presentation are for illustrative training purposes only. They are not representative of PARCC assessment items.
46Criterion 4: Avoiding Clueing and Miscues Is the item free from internal clueing and miscues?
47Criterion 4: Avoiding Clueing and Miscues Options should NOT echo a stem word.Distractors should be plausible so that students will consider them carefully.Items should not be answerable by reading other items in the set.
48Criterion 4: Avoiding Clueing and Miscues Example of Miscue:How is the life cycle of a frog similar to the life cycle of the butterfly?Both turn into a pupa.Both breathe with gills.Both begin life as an egg.*Both break out as a chrysalis.Notes: The stem asks test-takers to focus on “life cycles,” but the options focus on “frog” and “butterfly.” What is being asked in stem differs from what is being provided in options list, and test-takers will not know what they’re being asked to do.Note: all items included in this presentation are for illustrative training purposes only. They are not representative of PARCC assessment items.
49Criterion 5: Accuracy of Content Represented by Graphics Do the graphics and stimuli included as part of the item accurately and appropriately represent the applicable content knowledge?
50Criterion 5: Accuracy of Content Represented by Graphics Items should:not include graphics simply for the sake of including graphics.use graphics that allow students to understand the problem or task or demonstrate ability to solve a problem or task.elicit the intended evidence(s) to demonstrate ability with the skills expressed in the standard(s).
51Criterion 6: Clarity and Appropriateness of Graphics Are any graphics included as part of the item clear and appropriate for the intended grade level?
52Criterion 6: Clarity and Appropriateness of Graphics Items should:include graphics that are not too complex in structure or language for the intended grade levelutilize graphics that allow students to understand the problem or task or demonstrate ability to solve a problem or taskhave figures, graphs, charts, and diagrams precisely labeled
53Criterion 7: Effectiveness and Grade Appropriateness of Technology If the item has a technology-based stimulus or requires a technology-based response, is the technology design effective and grade appropriate?
54Criterion 7: Effectiveness and Grade Appropriateness of Technology Items should:include the number of options and correct answers that are appropriate for the grade level.use a format that is sufficiently simple and interesting for students.include directions that are clear and detailed.
55Criterion 7: Effectiveness and Grade Appropriateness of Technology Grade 3: Example of Effective and Grade-Appropriate Use of Technology Drag the words from the word box into the correct locations on the graphic to show the life cycle of a butterfly as described in “How Animals Live.” Words:PupaAdultEggLarvaNote: all items included in this presentation are for illustrative training purposes only. They are not representative of PARCC assessment items.
56Criterion 8: Clarity and Alignment of Performance Expectations to Scoring Guide Is the scoring guide/rubric clear, correct, and aligned with the expectations for performance that are expressed in the item or task?NOTE: Talk about how we are going to develop scoring notes for PCR items together during the item review.
57Criterion 8: Clarity and Alignment of Performance Expectations to Scoring Guide Scoring Guidelines for EBSRs and TECRs:Partial credit is allowable and desirable. See the specifics for scoring in the Item Guidelines document.Pgs Item Guidelines
58Criterion 9: Alignment of PBA Task to Model If the item is part of a PBA task, does it contribute to the focus and coherence of the task model?
59Criterion 9: Alignment of PBA Task to Model Sample Task Generation ModelELA Task Generation Model 5A.4Task Focus: Comparing themes and topicsTask TypeLiterary AnalysisGrade5Number and type of Texts1 Extended Literature Text1 Additional Literature TextBoth texts must be stories from the same genre (e.g., mysteries and adventure stories) with similar themes and topics.Number and type of Prose Constructed Response Items1 Analytic PCRMeasures reading literature sub-claim using standards 1 and 9Measures all writing claimsNumber and type of EBSR and/or TECR reading items6 total items = 12 points2 of 6 items(4 points) to measure the reading sub-claim for vocabulary (one per text)4 of 6 items (8 points) measuring standards RL 2,3 and 5Items that do not measure reading sub-claim for vocabulary are designed to measure reading literature sub-claimTask Complexity (including text, item, and task complexity)To be determinedTotal # of Items for the Task Model:7Order of Student Actions:Students read extended literature textStudents respond to 1 item to measure the reading sub-claim for vocabularyStudents respond to 2 EBSR or TECR itemsStudents read 1 additional literature textsStudents respond to 1 PCRNOTE: PCR addresses the standards that are the focus of this task model. EBSR and TECR items will address other standards and be ordered in such a way as to build toward the culminating PCR
60Criterion 9: Alignment of PBA Task to Model Task Model Review for PBA TasksTask Alignment with the TGMDoes the provided task match the task generation model designated in the metadata provided?Task DirectionsAre the directions for the task clear, coherent, and appropriate for the intended grade level?NOTE:1. Go through the requirements in the task generation model2. This is where the scenario and purpose setting statements will be reviewed.