ALS Design, Build, Review: Using PDE’s Online Tools to Implement the SLO Process SAS Portal:

Navigate to the homeroom page: RIA Homeroom site.

Log in and if not a user then register for the site: Pause until entire room is registered or with a partner:

Home Page for information: Open ALS

The ALS Box expands…………..

Assessment Literacy Series 6 -Orientation Module-

What is Assessment Literacy? 7 A process of designing, building, and reviewing performance measures for use in evaluating student achievement.

1. Develop high-quality performance measures for use within the greater educator effectiveness system. 2. Increase assessment literacy of participants. 3. Gain a deeper understanding of academic content standards. 8 Series Objectives

Assessment Life Cycle Establish Assessment Purpose(s) and Design Build Test Specifications & Blueprint Develop Items Develop Scoring Keys and Rubrics Create Operational Forms & Administrative Guidelines Review Forms Administer Test and Report Results Examine Validity Evidence 9

The Assessment Literacy training contains six (6) modularized components within three (3) action strands: ◦ DESIGN 1.M1-Design & Purpose Statement 2.M2-Test Specifications & Blueprints ◦ BUILD 3.M3-Item Specifications 4.M4-Scoring Keys & Rubrics 5.M5-Operational Forms & Administrative Guidelines ◦ REVIEW 6.M6-Quality Assurance & Form Reviews 10 Overview of the Development Process

Purpose Statement Explain why measures are made, what they will measure, and how the results will be used. Targeted Standards Select content standards associated with the performance measure. 11 Development Process

Test Specifications and Blueprints Outline the types of items being used, point values, and depth of knowledge (DoK) distribution. Guide item development by focusing items/tasks on discrete aspects of the content standards. Item/Tasks Specifications Articulate the type and characteristics of items/tasks that align to the test specifications and blueprints. 12 Development Process (cont.)

Scoring Keys & Rubrics Provide answers for multiple choice items and detailed scoring rubrics for constructed response tasks. Operational Forms Organize and sequence items/tasks within an outlined format that includes item tags (to uniquely identify each item/task). 13 Development Process (cont.)

Administrative Guidelines Provide detailed instructions for how to prepare, administer, and collect results. Quality Assurance & Form Reviews Review all operational forms, scoring keys/rubrics, and guidelines to ensure pre- established quality expectations have been met. Determine the overall rigor and alignment at the item/task-level and for the overall operational form. 14 Development Process (cont.)

Guides Handout #1 – Purpose Statement Examples Handout #2 – Targeted Standards Example Handout #3 – Test Specifications & Blueprint Example Handout #4 – Depth of Knowledge (DoK) Chart Handout #5 – Item Examples Handout #6 – Scoring Key Example Handout #7 – Rubric Examples Handout #8 – Operational Form Example Handout #9 – Administrative Guidelines Example Handout #10 – Quality Assurance Checklist 15 Helpful Materials

Templates Template #1-Purpose Statement Template #2-Targeted Standards Template #3-Test Specifications & Blueprint Template #4-Scoring Key Template #5-Operational Form Template #6-Administrative Guidelines 16 Helpful Materials (cont.)

Stuff “Smart Book” ◦ Provides techniques and technical criteria for developing measures. Performance Rigor Checklist ◦ A quick screening tool used to examine item/task quality. 17 Helpful Material (cont.)

Principles of Well-developed Measures Measures must: Be built to achieve the designed purpose Produce results that are used for the intended purpose Align to targeted content standards Contain a balance between depth and breadth of targeted content Be standardized, rigorous, and fair Be sensitive to testing time and objectivity Be valid and reliable 18

Evaluating Measures Quality Assurance Rubrics Three (3) versions- ◦ Teacher ◦ District ◦ State or Vendor Comprehensive tools that evaluate the technical process used in developing performance measures. Organized into strands and provides an overall rating. 19

Decision-Making Approaches Build Consensus Approach All opinions expressed Ideas of all participants gathered and synthesized Group agreement reached Team Leader Approach Guiding questions Group discussion about the pros and cons Create a “best fit” course of action 20

Think-Pair-Share 5 minutes to think about the statement below and write down your thoughts 5 minutes to share your ideas with a partner 10 minutes to share with your group “The Assessment Literacy process will help me as an educator by _______________.” 21

Summary Orientation Module Introduced the Assessment Literacy process used to create high quality measures of student achievement. Next Steps Module 1: Design & Purpose Statement Designing performance measures, including creating purpose statements and selecting academic content standards. 22 Summary & Next Steps

Assessment Literacy Series 23 -Module 1- Design & Purpose Statement

Participants will: 1. Develop a Purpose Statement that states the “what”, “how”, and “why” aspects of the performance measure being developed. 2. Determine the targeted content standards to be measured. 24 Objectives

Participants may wish to reference the following: Guides ◦ Handout #1 – Purpose Statement Examples ◦ Handout #2 – Targeted Standards Example Templates ◦ Template #1 – Purpose Statement ◦ Template #2 – Targeted Standards Other “Stuff” ◦ Content standards associated with the applicable grade-levels/spans; Common Core Standards ◦ Textbooks, teacher guides 25 Helpful Tools

26 Outline of Module 1 Module 1: Design & Purpose Statement Purpose Statement Targeted Standards

27 Purpose Statement

Statement outlining what the performance measure is measuring Statement about how the results (scores) can be used Statement of why the performance measure was developed 28 Purpose Statement [Handout #1]

Purpose Statement (WHAT) The Social Studies assessment is intended to measure student proficiency of grade-level expectations in the sequence of the district’s curriculum. (HOW) This grade-level assessment is provided to all students as a post-test. (WHY) Scores are reported to the district and used as a part of a comprehensive teacher evaluation program. 29

Process Steps [Template #1] 1. Each team member will work independently to create a statement about the performance measure in terms of the content standards it will purport to measure. 2. Build consensus by focusing on each aspect of the statement: What, How, Why 3. Draft a sentence reflecting the group’s consensus for each aspect and review as a group. 4. Merge each sentence to create a single paragraph “statement”. Again, review to ensure the statement reflects the group’s intent. 5. Finalize the statement and double-check for editorial soundness. 30

Statement is clear and concise; Free of technical jargon. Statement identifies what the performance measure is designed to measure; Grade-level/subject area/course. Statement articulates how the information from the performance measure is intended for use; Provides insight about what the scores mean. 31 QA Checklist

32 Targeted Content Standards

Targeted Standards [Handout #2] Choosing targeted standards means: Selecting certain standards for use with the performance measure being developed. Identifying standards representing the “big ideas” within the content area. 33

Targeted Standards… are a refined list of the content standards. represent the essential knowledge and skills that students are expected to acquire. are the standards that educators will spend the most time on. create transparency for families and the community about what is most important for student success. become the identified content standards used to create the measures. 34

Selection Criteria Endurance - Will this standard provide students with knowledge and skills that will be of value beyond a single test date? Leverage - Does this standard provide knowledge and skills that will be of value in multiple disciplines? Readiness for the next level of learning - Will this standard provide students with essential knowledge and skills that are necessary for success in the next level of instruction? 35

Targeted Standards Example [Handout #2] 36 Targeted Standards ID Num ber Content StatementDraftFinal A- SSE.1 Interpret expressions that represent a quantity in terms of its context. A- SSE.3 Choose and produce an equivalent form of an expression to reveal and explain properties of the quantity represented by the expression. X A- CED.1 Create equations and inequalities in one variable and use them to solve problems. XX A- REI.2 Solve simple rational and radical equations in one variable, and give examples showing how extraneous solutions may arise. XX

Process Steps [Template #2] 1. First, copy and paste the content standards into the table (or use an Excel spreadsheet) and then each team member will work independently to apply the three criteria (endurance, leverage, preparation) to his/her targeted content standards. 2. Build consensus about which identified content standards should be on the team’s draft list. Remember to balance the number of distinct standards selected and the time needed to sufficiently measure them. Place a checkmark in the draft box if the team agrees. 3. Compare the draft of targeted standards to the “conceptual” blueprint indicating what is likely going to be emphasized on the performance measure. 4. Review the list of targeted standards and look for gaps and redundancies. Apply the checklist to each standard. 5. Finalize the list. Place checkmarks in the final box of the standards that will be used to guide the blueprint development. 37

QA Checklist  Do my targeted standards have endurance?  Do my targeted standards have leverage?  Do my targeted standards prepare students for the next level of learning? 38

Summary Module 1: Design & Purpose Statement Developed a purpose statement and chose the targeted standards that the performance measure will be based upon. Next Steps Module 2: Test Specifications and Blueprints Given the purpose and targeted standards, create a specification table and associated blueprint that will guide item/task development. 39 Summary & Next Steps

Assessment Literacy Series 40 -Module 2- Test Specifications & Blueprints

Participants will: 1. Develop test specifications that articulate: Number of items by type Item point value Depth of Knowledge (DoK) levels 2. Develop a blueprint that designates: Items per content standard across DoK levels 41 Objectives

Participants may wish to reference the following: Guides Handout #3 – Test Specifications and Blueprint Example Handout #4 – Depth of Knowledge (DoK) Chart Templates Template #3 – Test Specifications and Blueprint 42 Helpful Tools

43 Outline of Module 2 Module 2: Test Specifications & Blueprints Test Specifications Blueprints Process Steps Item Type Depth of Knowledge (DoK) Depth of Knowledge Content Standards

44 Specification Tables

Test Specifications When developing test specifications consider: Sufficient sampling of targeted content standards Aim for a 3:1 items per standard ratio Developmental readiness of test-takers Type of items Multiple Choice (MC) Short Constructed Response (SCR) Extended Constructed Response (ECR)/Complex Performance tasks Time burden imposed on both educators and students 45

Test Specifications (cont.) When developing test specifications consider: Cognitive load Aim for a balance of DoK levels Objectivity of scoring Each constructed response item/task will need a well- developed rubric Weight of items (point values) Measures should consist of total points; points for high school Item cognitive demand level/DoK level Measures should reflect a variety of DoK levels as represented in the targeted content standards 46

Test Specifications Example [Handout #3] 47 Content Strand(s)MCSCRECRTotal Expressions & Equations4004 Creating Equations5005 Structure in Expressions3003 Ratios & Proportions3205 Reasoning with Equations & Inequalities4105 Interpreting Functions3216 Real Number System5117 Grand Totals *Performance measure contains 35 items/tasks. Content Strand(s)MC (1 pt.) SCR (2pts.) ECR (4pts.) Total Expressions & Equations4004 Creating Equations5005 Structure in Expressions3003 Ratios & Proportions3407 Reasoning with Equations & Inequalities4206 Interpreting Functions34411 Real Number System52411 Grand Totals *Performance measure score based upon 47 points.

Test Specifications Example (cont.) [Handout #3] 48 Content Strand(s)DoK 1DoK 2DoK 3Total Expressions & Equations1214 Creating Equations1225 Structure in Expressions2013 Ratios & Proportions0505 Reasoning with Equations & Inequalities 3205 Interpreting Functions0246 Real Number System1427 Grand Totals *Performance measure contains items/tasks with the following Level/DoK distribution: DoK 1 = 23% DoK 2 & 3 = 77%

Stem (question) with four (4) answer choices Typically worth one (1) point towards overall score Generally require about one (1) minute to answer Pros Easy to administer Objective scoring Cons Students can guess the correct answer No information can be gathered on the process the student used to reach answer 49 Multiple Choice Items

Requires students to apply knowledge, skills, and critical thinking abilities to real-world performance tasks Entails students "constructing" or developing their own answers in the form of a few sentences, a graphic organizer, or a drawing/diagram with explanation Worth 1-3 points Pros Allows for partial credit Provides more details about a student’s cognitive process Reduces the likelihood of guessing Cons Greater scoring subjectivity Requires more time to administer and score 50 Short Constructed Response Items

Requires students to apply knowledge, skills, and critical thinking abilities to real-world performance tasks by developing their own answers in the form of narrative text with supporting graphic organizers and/or illustrations Worth 4 or more points Entails more in-depth explanations than SCR items Pros Allows for partial credit Provides more details about a student’s cognitive process Reduces the likelihood of guessing Cons Greater scoring subjectivity Requires more time to administer and score 51 Extended Constructed Response Items

Depth of Knowledge is… The complexity of mental processing that must occur in order to construct an answer A critical factor in determining item/task rigor 52

Depth of Knowledge Chart [Handout #4] 53

Think-Pair-Share A pencil costs 5 cents at the school store. Matt wants to buy 3 pencils. How much will they cost? During a 12.5-hour period, the water level at a certain harbor started at mean sea level, rose to 9.3 feet above sea level, dropped to 9.3 feet below sea level, and then returned to mean sea level. Find a simple, harmonic motion equation that models the height h of the tide above or below mean sea level for this 12.5-hour period. 54

Think-Pair-Share 55 After reading Refrigeration by Evaporation, write an informative essay explaining how Abba’s invention responded to the needs of Nigeria’s people and changed their lives. Use evidence from the text to support your ideas.

Process Steps [Template #3] 1. Review content standards from completed Targeted Content Standards Template and insert content strand(s) into specification table. 2. Determine the number of items by item type (i.e., Multiple Choice, Short Constructed Response, Extended Constructed Response) for each content strand. 3. Ensure item type and cognitive level (I, II, III)/depths of knowledge (DoK) are assigned. 4. Assign item weights to each item type. 5. Assign number of passages (by type) when using literary works. 56

QA Checklist There is a sufficient sampling of targeted standards. The specifications reflect a balance between developmental readiness and time constraints. Time is considered for both educators and students. The cognitive demands reflect those articulated in the targeted standards. The measure allows for both objective and subjective scoring procedures. The measure consists of points with the Level I/DoK I limited to one-third of the items/tasks. 57

58 Blueprints

Content ID # Content Statement Item Depth of Knowledge (DoK) Performance measures should reflect a variety of DoK levels. Sufficient sampling of content standards Aim for a 3:1 item to standard ratio (3 items for every standard). Cognitive load Aim for a balance of DoK levels among standards. Design measures with at least 50% DoK 2 or higher. 59

Blueprint Example [Handout #3] 60

Process Steps 1. List the standards by number and statement in the appropriate columns. Remember to aim for a 3:1 item to standard ratio. 2. Decide on the item count for each standard and fill in the appropriate column. 3. Determine the number of DoKs for each standard following the specified guidelines for “rigor”. 4. Repeat Steps 1-3 ensuring that item and DoK counts meet the specification requirements. 61

The blueprint lists the content standard ID number. The blueprint lists or references the targeted content standards. The blueprint designates item counts for each standard. The blueprint reflects a range of DoK levels. The blueprint item/task distribution reflects that in the specification tables. 62 QA Checklist

Summary Module 2: Test Specifications and Blueprints Developed test specifications and a blueprint to guide the item development process. Next Steps Module 3: Item Specifications Given the specifications and blueprint, develop items to measure aspects of the targeted content standards. 63 Summary & Next Steps

Joining the SLO Professional Learning Community on SAS. Go to the SAS home page( ) Log in with your user name and password. If you do not have an account with SAS you will have to create one.

Enter your information on the log in page and submit.

Once you have successfully logged in and are at the SAS home page, go to Teacher Tools in the upper right corner.

Click on Teacher Tools, this will provide you with various tools. Locate the button labeled “My Communities.”

This will open your membership to various Professional Learning Communities. If you are not a member of the Student Learning Objectives PLC, type SLO in the search bar.

Once a member of the SLO community you will have access to communication with all other members and a calendar of upcoming events.

Along with posting questions to the entire community you have access to the Digital Repository, in which SLO training materials and supporting documents are located. (This is located at the bottom of the SLO community page.)

Contact Info PDE POC: Mr. O David Deitz RIA POC: Dr. JP Beaudoin

SAS Institute