CREATING LOCAL ASSESSMENTS May 8, :00-3:00 p.m.

Desired Outcomes  Identify the purpose and reasons for building local assessments  Identify the criteria for selecting target standards  Understand the purpose and process of creating an assessment blueprint  Understand the purpose and process of item validation

Local Assessment Development Process

CCSS 1664 VAM NEA FEA SEEC 736 AIR FCAT FL Standards TIF

Project Update: Creating Local Assessments What? When? Where? Why?

Creating a Blueprint: Suggested Resources Course Description Course Standards Blank or electronic copies of the blueprint Multiple content experts Any available FL DOE test design summaries that are from current or future assessments Additional Resources: Pacing Guides Curriculum Maps

What is an assessment blueprint?  A blueprint is your guide to assessment creation.  It is used to see the content and structure of an assessment.  The blueprint should always be based on the standards or benchmarks listed in the course description on C-PALMS.

Take A Closer Look Upper Level Lower Level

Before you begin: Before drafting the blueprint, determine the purpose and type of assessment. Example: Purpose: to gauge student mastery of course standards Type of Assessment: Kindergarten math fall interim assessment Timeframe for Assessment: 40, 60, 90 minutes?

Approximate Time Allocations Item TypeMinutes AllocatedPoint Value Selected Response (SR)1–3 minutes1 Gridded Response (GR)1–3 minutes1 Short Response (SHR)3 minutes1 Constructed Response (CR) 5 minutes2 Extended Response (ER)10–15 minutes4 Essay Response (ESR)20–30 minutes6 Performance Task (PT)Varies1–10 (depending on complexity and content area)

Step 1: Category Breakdown Upper Columns A-D of Blueprint Based on the instructional time spent on each strand of standards or benchmarks (reporting category) For Example, if your pacing guide directs you to spend 90 days on Organisms, Populations, and Ecosystems and 36 days on Classification, Heredity, and Evolution, then there should be approximately 3 times as many Organisms as Classification. Determine the percentage of items by reporting category. Biology Interim Assessment: Molecular and Cellular Biology = 30% Classification, Heredity, and Evolution = 20% Organisms, Populations, and Ecosystems = 50%

Step 2: Total Number of Items for Each Category Upper Level E Reporting CategoryPercentage of Instructional Time Percentage of Items by Category Total Number of Items for Category Molecular and Cellular Biology 30% Classification, Heredity, and Evolution 20% Organisms, Populations, and Ecosystems 50% Total100% Biology 1 Interim Assessment 80 Minute Assessment, 40 Items Multiply the number of test questions by the percentage of questions by category.

Step 3: What Standards will make up the reporting category? Lower Level Columns A-B Not all standards are created equal! Choose approximately 25 targeted standards.

How Do I Determine Targeted Standards? Power Standards Endurance Leverage Readiness Appropriate for the item types available Supports future EOC’s or other standardized tests Meets time constraints Required resources to test the standard are available

Step 4: Identify DOK Lower Level Column C Once standards have been chosen, use C-PALMS to determine the cognitive complexity of the standards. At least 50% or more of the assessment items developed for a standard or benchmark should meet or exceed the DOK level of that standard.

Step 5: Percentage of Items by DOK Level Upper Level Columns F-H on Blueprint Determine the percentage of items by DOK level. DOK level: 1, 2, 3, 4 Should total 100% Give minimum range of 5–10 percentage points (i.e., 20–30% DOK level 1) Biology Interim Assessment DOK level 1 = 10-20% DOK level 2 = 30-40% DOK level 3-4 = 50-60%

Reporting Category Percentage of Instruction al Time Percentage of Items by Category Total Number of Items for Category Percentage of items by DOK level 1: Percentage of items by DOK level 2: Percentage of items by DOK level 3-4: Molecular and Cellular Biology 30% %30-40%50-60% Reporting Category (Broadest Level) Standard or Benchmark (Instructional Level) DOK Level of Standard or Benchmark DOK Level of Question Molecular and Cellular Biology Standard 133 Molecular and Cellular Biology Standard 133 Molecular and Cellular Biology Standard 132 Molecular and Cellular Biology Standard 223 Molecular and Cellular Biology Standard 221 Molecular and Cellular Biology Standard 222 Molecular and Cellular Biology Standard 333 Molecular and Cellular Biology Standard 332

Step 6: Number of Items by DOK Level Lower Level Columns I-K on Blueprint Multiply the cognitive complexity level percentage by the total number of items to get the item range for each DOK level. Start with the low complexity items. Biology Interim Assessment example: 10-20% DOK level 1 x 40 items = 4-8 DOK level 1 items Multiply the cognitive complexity level percentage by the total number of items to get the item range for the DOK level 2 items. Biology Interim Assessment example: 30-40% DOK level 2 x 40 items = DOK level 2 items

Multiply the cognitive complexity level percentage by the total number of items to get the item range for the DOK level 3 or 4 items. Biology Interim Assessment example: 50-60% DOK level 3–4 x 40 items = DOK level 3–4 items Step 6 continued: Number of Items by DOK Level Lower Level Columns I-K on Blueprint

Can only be completed after giving the assessment.

Now you try… Grade 8 U.S. History (Social Studies)

Using a Blueprint to Create a Test Map  District-selected test developers Must have access to item banks Secure process  Add a column for the actual item number for each standard and item on your blueprint.

Item Vetting

Security Assessment security is of the upmost importance to ensure fidelity and validity. Our preventative measures: Security agreements/logs for the item binders Protocols No phone/cameras No items leaving the room No copying items All materials stay at the location in a locked cabinet. Materials are printed on brightly colored paper.

Item Validation Process

Anatomy of a Test Item  Item: stem + response options  Stem: section of the item that poses a problem or gives context  Stimulus: passage, text, or graphic that is an introductory part of an item or items (optional- should only be used if it contains necessary information to answer the stem)  Responses: possible answer options Order responses logically (e.g., by length, chronology, or value)  Distractors: incorrect responses that distract those without knowledge

Example

Universal Design Checklist A test item meets Universal Design requirements if it: Applies to all levels of learning Defines ideas precisely Is free from bias Accommodates all backgrounds and abilities Offers simple, clear instructions and procedures Can be read easily and understandably Has a correct answer Investigate the item specifications handout Digestible Bites…Chunks… One Criteria at a Time…

Identifying Bias Read the following statement: Policemen are important to our city. They keep all of us safe. How should this item be changed to eliminate bias?

Identifying Bias Read the following scenario: The administrators are planning a school field trip to a yachting club in Fernandina Beach. How does this scenario show bias? How might the item be improved?

Example of a Strong Selected Response Item NGSSS Benchmark: SC.8.E.5.5: Describe and classify specific physical properties of stars: apparent magnitude (brightness), temperature (color), size, and luminosity (absolute brightness). One physical property of a star is apparent magnitude. Which of the following is used in determining the apparent magnitude of a star? A.the constellation the star is in B.the distance of the star from Earth C.the number of times the star rotates D.the number of prominences the star makes

Activity For the next 10 minutes, read through items investigating only the following criteria: Bias/Sensitivity Correct Responses Does it meet some or most of the requirements of universal design (handout)? Is it a bad item (handout) We will discuss any hot topics or questions after time has expired.

Standards/Benchmarks FL DOE “In these courses, the NGSSS benchmarks provide the learning targets for instruction and assessment…” Item specification criteria from FL DOE: “Each item should be written to measure primarily one NGSSS benchmark; however, other benchmarks may also be addressed for some item types.”

Activity For the next 10 minutes, read through items and correlate them to one NGSSS benchmark: We will discuss any hot topics or questions after time has expired.

Cognitive Complexity Since these are all selected response items, we will be rating cognitive complexity on a 3 tiered system Webb’s DOK LevelsFlorida’s Original DOK Levels Level 1: RecallLow: Recall Level 2: Basic Application of Skills & Concepts Moderate: Basic Application of Skills & Concepts Level 3: Strategic Thinking & Complex Reasoning High: Strategic Thinking & Complex Reasoning Level 4: Extended Thinking & Complex Reasoning

Cognitive Complexity Sourced from “Content Cognitive Complexity Initiative” on CPALMS ( “Content complexity relates to the cognitive demands inferred from the language of a content standard.” What factors must be considered in content complexity? Prior knowledge Processing of concepts and skills Sophistication Number of parts Application of content structure required to meet an expectation or attain an outcome

DOE on DOK FL DOE “Florida uses a Depth of Knowledge Content Complexity Model to identify the cognitive demands of learning targets and corresponding assessment items.” “In contrast to item difficulty, which is based upon the analysis of student responses to specific assessment items, content complexity ratings reflect the level of cognitive demand associated with learning targets and corresponding instruction and assessment.”

Activity For the next 10 minutes, read through items and correlate them to one tier on the Cognitive Complexity Scale. We will discuss any hot topics or questions after time has expired.

TSC Summer PD June 24 th – Connecting Your Role to the Teacher Leader Standards June th – PLC’s, Instructional Coaching, and Peer Coaching with Steve Barkley July 9 th – Using Fierce Conversations to Drive the Balanced Coaching Frame July nd – Summer Leadership Conference (St. Augustine WGV)