1. ALS Design, Build, Review: Using PDE’s Online Tools to Implement the SLO Process SAS Portal:

2. Navigate to the homeroom page: RIA Homeroom site.

3. Log in and if not a user then register for the site: Pause until entire room is registered or with a partner:

4. Home Page for information:
Open ALS

5. The ALS Box expands…………..

6. Assessment Literacy Series
-Orientation Module-
IMT Orientation Draft 02Sept11-CS

7. What is Assessment Literacy?
A process of designing, building, and reviewing performance measures for use in evaluating student achievement.

8. Series Objectives
Develop high-quality performance measures for use within the greater educator effectiveness system.
Increase assessment literacy of participants.
Gain a deeper understanding of academic content standards.

9. Assessment Life Cycle Examine Validity Evidence
Establish Assessment Purpose(s) and Design
Build Test Specifications & Blueprint
Develop Items
Develop Scoring Keys and Rubrics
Create Operational Forms & Administrative Guidelines
Review Forms
Administer Test and Report Results
Examine Validity Evidence

10. Overview of the Development Process
The Assessment Literacy training contains six (6) modularized components within three (3) action strands:
DESIGN
M1-Design & Purpose Statement
M2-Test Specifications & Blueprints
BUILD
M3-Item Specifications
M4-Scoring Keys & Rubrics
M5-Operational Forms & Administrative Guidelines
REVIEW
M6-Quality Assurance & Form Reviews

11. Development Process Purpose Statement Targeted Standards
Explain why measures are made, what they will measure, and how the results will be used.
Targeted Standards
Select content standards associated with the performance measure.

12. Development Process (cont.)
Test Specifications and Blueprints
Outline the types of items being used, point values, and depth of knowledge (DoK) distribution.
Guide item development by focusing items/tasks on discrete aspects of the content standards.
Item/Tasks Specifications
Articulate the type and characteristics of items/tasks that align to the test specifications and blueprints.

13. Development Process (cont.)
Scoring Keys & Rubrics
Provide answers for multiple choice items and detailed scoring rubrics for constructed response tasks.
Operational Forms
Organize and sequence items/tasks within an outlined format that includes item tags (to uniquely identify each item/task).

14. Development Process (cont.)
Administrative Guidelines
Provide detailed instructions for how to prepare, administer, and collect results.
Quality Assurance & Form Reviews
Review all operational forms, scoring keys/rubrics, and guidelines to ensure pre- established quality expectations have been met.
Determine the overall rigor and alignment at the item/task-level and for the overall operational form.

15. Helpful Materials Guides Handout #1 – Purpose Statement Examples
Handout #2 – Targeted Standards Example
Handout #3 – Test Specifications & Blueprint Example
Handout #4 – Depth of Knowledge (DoK) Chart
Handout #5 – Item Examples
Handout #6 – Scoring Key Example
Handout #7 – Rubric Examples
Handout #8 – Operational Form Example
Handout #9 – Administrative Guidelines Example
Handout #10 – Quality Assurance Checklist

16. Helpful Materials (cont.)
Templates
Template #1-Purpose Statement
Template #2-Targeted Standards
Template #3-Test Specifications & Blueprint
Template #4-Scoring Key
Template #5-Operational Form
Template #6-Administrative Guidelines

17. Helpful Material (cont.)
Stuff
“Smart Book”
Provides techniques and technical criteria for developing measures.
Performance Rigor Checklist
A quick screening tool used to examine item/task quality.

18. Principles of Well-developed Measures
Measures must:
Be built to achieve the designed purpose
Produce results that are used for the intended purpose
Align to targeted content standards
Contain a balance between depth and breadth of targeted content
Be standardized, rigorous, and fair
Be sensitive to testing time and objectivity
Be valid and reliable

19. Quality Assurance Rubrics
Evaluating Measures
Quality Assurance Rubrics
Three (3) versions-
Teacher
District
State or Vendor
Comprehensive tools that evaluate the technical process used in developing performance measures.
Organized into strands and provides an overall rating.

20. Decision-Making Approaches
Build Consensus Approach
All opinions expressed
Ideas of all participants gathered and synthesized
Group agreement reached
Team Leader Approach
Guiding questions
Group discussion about the pros and cons
Create a “best fit” course of action

21. Think-Pair-Share
5 minutes to think about the statement below and write down your thoughts
5 minutes to share your ideas with a partner
10 minutes to share with your group
“The Assessment Literacy process will help me as an educator by _______________.”

22. Summary & Next Steps Summary Orientation Module Next Steps
Introduced the Assessment Literacy process used to create high quality measures of student achievement.
Next Steps
Module 1: Design & Purpose Statement
Designing performance measures, including creating purpose statements and selecting academic content standards.

23. Assessment Literacy Series
-Module 1-
Design & Purpose Statement
IMT Orientation Draft 02Sept11-CS

24. Objectives Participants will:
Develop a Purpose Statement that states the “what”, “how”, and “why” aspects of the performance measure being developed.
Determine the targeted content standards to be measured.

25. Helpful Tools Participants may wish to reference the following: Guides
Handout #1 – Purpose Statement Examples
Handout #2 – Targeted Standards Example
Templates
Template #1 – Purpose Statement
Template #2 – Targeted Standards
Other “Stuff”
Content standards associated with the applicable grade-levels/spans; Common Core Standards
Textbooks, teacher guides

26. Module 1: Design & Purpose Statement
Outline of Module 1
Module 1: Design & Purpose Statement
Purpose Statement
Targeted Standards

27. Purpose Statement

28. Purpose Statement [Handout #1]
Statement outlining what the performance measure is measuring
Statement about how the results (scores) can be used
Statement of why the performance measure was developed

29. Purpose Statement
(WHAT) The Social Studies assessment is intended to measure student proficiency of grade-level expectations in the sequence of the district’s curriculum. (HOW) This grade-level assessment is provided to all students as a post-test. (WHY) Scores are reported to the district and used as a part of a comprehensive teacher evaluation program.

30. Process Steps [Template #1]
Each team member will work independently to create a statement about the performance measure in terms of the content standards it will purport to measure.
Build consensus by focusing on each aspect of the statement: What, How, Why
Draft a sentence reflecting the group’s consensus for each aspect and review as a group.
Merge each sentence to create a single paragraph “statement”. Again, review to ensure the statement reflects the group’s intent.
Finalize the statement and double-check for editorial soundness.

31. QA Checklist Statement is clear and concise; Free of technical jargon.
Statement identifies what the performance measure is designed to measure; Grade-level/subject area/course.
Statement articulates how the information from the performance measure is intended for use; Provides insight about what the scores mean.

32. Targeted Content Standards

33. Targeted Standards [Handout #2]
Choosing targeted standards means:
Selecting certain standards for use with the performance measure being developed.
Identifying standards representing the “big ideas” within the content area.

34. Targeted Standards… are a refined list of the content standards.
represent the essential knowledge and skills that students are expected to acquire.
are the standards that educators will spend the most time on.
create transparency for families and the community about what is most important for student success.
become the identified content standards used to create the measures.

35. Selection Criteria
Endurance - Will this standard provide students with knowledge and skills that will be of value beyond a single test date?
Leverage - Does this standard provide knowledge and skills that will be of value in multiple disciplines?
Readiness for the next level of learning - Will this standard provide students with essential knowledge and skills that are necessary for success in the next level of instruction?

36. Targeted Standards Example [Handout #2]
ID Number
Content Statement
Draft
Final
A-SSE.1
Interpret expressions that represent a quantity in terms of its context.
A-SSE.3
Choose and produce an equivalent form of an expression to reveal and explain properties of the quantity represented by the expression. X
A-CED.1
Create equations and inequalities in one variable and use them to solve problems.
A-REI.2
Solve simple rational and radical equations in one variable, and give examples showing how extraneous solutions may arise.

37. Process Steps [Template #2]
First, copy and paste the content standards into the table (or use an Excel spreadsheet) and then each team member will work independently to apply the three criteria (endurance, leverage, preparation) to his/her targeted content standards.
Build consensus about which identified content standards should be on the team’s draft list. Remember to balance the number of distinct standards selected and the time needed to sufficiently measure them. Place a checkmark in the draft box if the team agrees.
Compare the draft of targeted standards to the “conceptual” blueprint indicating what is likely going to be emphasized on the performance measure.
Review the list of targeted standards and look for gaps and redundancies. Apply the checklist to each standard.
Finalize the list. Place checkmarks in the final box of the standards that will be used to guide the blueprint development.

38. QA Checklist Do my targeted standards have endurance?
Do my targeted standards have leverage?
Do my targeted standards prepare students for the next level of learning?

39. Summary & Next Steps Summary Module 1: Design & Purpose Statement
Developed a purpose statement and chose the targeted standards that the performance measure will be based upon.
Next Steps
Module 2: Test Specifications and Blueprints
Given the purpose and targeted standards, create a specification table and associated blueprint that will guide item/task development.

40. Assessment Literacy Series
-Module 2-
Test Specifications & Blueprints
IMT Orientation Draft 02Sept11-CS

41. Objectives Participants will:
1. Develop test specifications that articulate:
Number of items by type
Item point value
Depth of Knowledge (DoK) levels
2. Develop a blueprint that designates:
Items per content standard across DoK levels

42. Helpful Tools Participants may wish to reference the following: Guides
Handout #3 – Test Specifications and Blueprint Example
Handout #4 – Depth of Knowledge (DoK) Chart
Templates
Template #3 – Test Specifications and Blueprint

43. Module 2: Test Specifications & Blueprints Depth of Knowledge (DoK)
Outline of Module 2
Module 2: Test Specifications & Blueprints
Content Standards
Depth of Knowledge
Test Specifications
Blueprints
Process Steps
Item Type
Depth of Knowledge (DoK)

44. Specification Tables

45. Test Specifications When developing test specifications consider:
Sufficient sampling of targeted content standards
Aim for a 3:1 items per standard ratio
Developmental readiness of test-takers
Type of items
Multiple Choice (MC)
Short Constructed Response (SCR)
Extended Constructed Response (ECR)/Complex Performance tasks
Time burden imposed on both educators and students

46. Test Specifications (cont.)
When developing test specifications consider:
Cognitive load
Aim for a balance of DoK levels
Objectivity of scoring
Each constructed response item/task will need a well- developed rubric
Weight of items (point values)
Measures should consist of total points; points for high school
Item cognitive demand level/DoK level
Measures should reflect a variety of DoK levels as represented in the targeted content standards

47. Test Specifications Example [Handout #3]
Content Strand(s) MC
SCR
ECR
Total
Expressions & Equations 4
Creating Equations 5
Structure in Expressions 3
Ratios & Proportions 2
Reasoning with Equations & Inequalities 1
Interpreting Functions 6
Real Number System 7
Grand Totals 27 35
*Performance measure contains 35 items/tasks.
(1 pt.)
(2pts.)
(4pts.) 11 12 8 47
*Performance measure score based upon 47 points.

48. Test Specifications Example (cont.) [Handout #3]
Content Strand(s)
DoK 1
DoK 2
DoK 3
Total
Expressions & Equations 1 2 4
Creating Equations 5
Structure in Expressions 3
Ratios & Proportions
Reasoning with Equations & Inequalities
Interpreting Functions 6
Real Number System 7
Grand Totals 8 17 10 35
*Performance measure contains items/tasks with the following Level/DoK distribution: DoK 1 = 23% DoK 2 & 3 = 77%

49. Multiple Choice Items Stem (question) with four (4) answer choices
Typically worth one (1) point towards overall score
Generally require about one (1) minute to answer
Pros
Easy to administer
Objective scoring
Cons
Students can guess the correct answer
No information can be gathered on the process the student used to reach answer

50. Short Constructed Response Items
Requires students to apply knowledge, skills, and critical thinking abilities to real-world performance tasks
Entails students "constructing" or developing their own answers in the form of a few sentences, a graphic organizer, or a drawing/diagram with explanation
Worth 1-3 points
Pros
Allows for partial credit
Provides more details about a student’s cognitive process
Reduces the likelihood of guessing
Cons
Greater scoring subjectivity
Requires more time to administer and score

51. Extended Constructed Response Items
Requires students to apply knowledge, skills, and critical thinking abilities to real-world performance tasks by developing their own answers in the form of narrative text with supporting graphic organizers and/or illustrations
Worth 4 or more points
Entails more in-depth explanations than SCR items
Pros
Allows for partial credit
Provides more details about a student’s cognitive process
Reduces the likelihood of guessing
Cons
Greater scoring subjectivity
Requires more time to administer and score

52. Depth of Knowledge is…
The complexity of mental processing that must occur in order to construct an answer
A critical factor in determining item/task rigor
Level
Example of Verb
Example of Task
DoK Level 1
Recall
List three characteristics of metamorphic rocks.
DoK Level 2
Compare/Contrast
Describe the difference between metamorphic and igneous rocks.
DoK Level 3
Create
Develop a model to represent the rock cycle.
DoK Level 4
Construct
Using multiple sources, develop an essay on the rise of the Industrial Revolution.

53. Depth of Knowledge Chart [Handout #4]
DoK
Level
Definition
Verbs
Examples 1
Involves recall and the response is automatic. Activities require students to demonstrate a rote response, follow a set of procedures, or perform simple calculations.
define, duplicate, list, memorize, recall, repeat, reproduce, state, classify, describe, discuss, explain, identify, locate, recognize, report, select, paraphrase
Identify the main character.
Subtract the numbers.
Label the rivers on the map.
Measure the length of your desk.
List the steps in the water cycle. 2
Activities are more complex and require students to engage in mental processing and reasoning beyond a habitual response.
These activities make students decide how to approach a problem.
choose, demonstrate, dramatize, employ, illustrate, interpret, operate, schedule, sketch, solve, use, write, appraise, compare, contrast, criticize, differentiate, discriminate, distinguish, examine, experiment, question, test
Summarize the events in the story.
Describe the cause/effect of an event.
Organize the data using a bar graph.
Formulate a problem given data.
Compare and contrast the main characters from the stories. 3
Activities necessitate higher cognitive demands. Students are providing support and reasons for conclusions they draw.
Typically, Level 3 activities have more than one correct response or approach to the problem.
appraise, argue, defend, judge, select, support, value, evaluate, assemble, construct, create, design, develop, formulate, write
Support your ideas with details and examples.
Design investigations for a scientific problem.
Construct a model of the solar system.
Using the graph, predict how many teeth would be lost by all the 2nd grade classes in the school and justify your answer.

54. Think-Pair-Share
A pencil costs 5 cents at the school store. Matt wants to buy 3 pencils. How much will they cost?
During a 12.5-hour period, the water level at a certain harbor started at mean sea level, rose to 9.3 feet above sea level, dropped to 9.3 feet below sea level, and then returned to mean sea level. Find a simple, harmonic motion equation that models the height h of the tide above or below mean sea level for this 12.5-hour period. 

55. Think-Pair-Share
After reading Refrigeration by Evaporation, write an informative essay explaining how Abba’s invention responded to the needs of Nigeria’s people and changed their lives. Use evidence from the text to support your ideas. 

56. Process Steps [Template #3]
Review content standards from completed Targeted Content Standards Template and insert content strand(s) into specification table.
Determine the number of items by item type (i.e., Multiple Choice, Short Constructed Response, Extended Constructed Response) for each content strand.
Ensure item type and cognitive level (I, II, III)/depths of knowledge (DoK) are assigned.
Assign item weights to each item type.
Assign number of passages (by type) when using literary works.

57. QA Checklist There is a sufficient sampling of targeted standards.
The specifications reflect a balance between developmental readiness and time constraints.
Time is considered for both educators and students.
The cognitive demands reflect those articulated in the targeted standards.
The measure allows for both objective and subjective scoring procedures.
The measure consists of points with the Level I/DoK I limited to one-third of the items/tasks.

58. Blueprints

59. Blueprints Content ID # Content Statement
Item Depth of Knowledge (DoK)
Performance measures should reflect a variety of DoK levels.
Sufficient sampling of content standards
Aim for a 3:1 item to standard ratio (3 items for every standard).
Cognitive load
Aim for a balance of DoK levels among standards.
Design measures with at least 50% DoK 2 or higher.

60. Blueprint Example [Handout #3]
Standard/
Content ID
Content Statement
Item Count
DoK 1 2 3
8.EE.1
Know and apply the properties of integer exponents to generate equivalent numerical expressions.
8.EE.2
Use square root and cube root symbols to represent solutions to equations of the form x2 = p and x3 = p, where p is a positive rational number. Evaluate square roots of small perfect squares and cube roots of small perfect cubes.
A-CED.1
Create equations and inequalities in one variable and use them to solve problems. 5

61. Process Steps
List the standards by number and statement in the appropriate columns. Remember to aim for a 3:1 item to standard ratio.
Decide on the item count for each standard and fill in the appropriate column.
Determine the number of DoKs for each standard following the specified guidelines for “rigor”.
Repeat Steps 1-3 ensuring that item and DoK counts meet the specification requirements.

62. QA Checklist The blueprint lists the content standard ID number.
The blueprint lists or references the targeted content standards.
The blueprint designates item counts for each standard.
The blueprint reflects a range of DoK levels.
The blueprint item/task distribution reflects that in the specification tables.

63. Summary & Next Steps Summary
Module 2: Test Specifications and Blueprints
Developed test specifications and a blueprint to guide the item development process.
Next Steps
Module 3: Item Specifications
Given the specifications and blueprint, develop items to measure aspects of the targeted content standards.

64. Joining the SLO Professional Learning Community on SAS.
Go to the SAS home page(
Log in with your user name and password.
If you do not have an account with SAS you will have to create one.

65. Enter your information on the log in page and submit.

66. Once you have successfully logged in and are at the SAS home page, go to Teacher Tools in the upper right corner.

67. Click on Teacher Tools, this will provide you with various tools.
Locate the button labeled “My Communities.”

68. This will open your membership to various Professional Learning Communities.
If you are not a member of the Student Learning Objectives PLC, type SLO in the search bar.

69. Once a member of the SLO community you will have access to communication with all other members and a calendar of upcoming events.

70. (This is located at the bottom of the SLO community page.)
Along with posting questions to the entire community you have access to the Digital Repository, in which SLO training materials and supporting documents are located.
(This is located at the bottom of the SLO community page.)

71. PDE POC: Mr. O David Deitz
Contact Info
PDE POC: Mr. O David Deitz
RIA POC: Dr. JP Beaudoin

72. SAS Institute