Assessment Literacy Series -Orientation Module- IMT Orientation Draft 02Sept11-CS
What is Assessment Literacy? A process of designing, building, and reviewing performance measures for use in evaluating student achievement.
Series Objectives Develop high-quality performance measures for use within the greater educator effectiveness system. Increase assessment literacy of participants. Gain a deeper understanding of academic content standards.
Assessment Life Cycle Examine Validity Evidence Establish Assessment Purpose(s) and Design Build Test Specifications & Blueprint Develop Items Develop Scoring Keys and Rubrics Create Operational Forms & Administrative Guidelines Review Forms Administer Test and Report Results Examine Validity Evidence
Overview of the Development Process The Assessment Literacy training contains six (6) modularized components within three (3) action strands: DESIGN M1-Design & Purpose Statement M2-Test Specifications & Blueprints BUILD M3-Item Specifications M4-Scoring Keys & Rubrics M5-Operational Forms & Administrative Guidelines REVIEW M6-Quality Assurance & Form Reviews
Development Process Purpose Statement Targeted Standards Explain why measures are made, what they will measure, and how the results will be used. Targeted Standards Select content standards associated with the performance measure.
Development Process (cont.) Test Specifications and Blueprints Outline the types of items being used, point values, and depth of knowledge (DoK) distribution. Guide item development by focusing items/tasks on discrete aspects of the content standards. Item/Tasks Specifications Articulate the type and characteristics of items/tasks that align to the test specifications and blueprints.
Development Process (cont.) Scoring Keys & Rubrics Provide answers for multiple choice items and detailed scoring rubrics for constructed response tasks. Operational Forms Organize and sequence items/tasks within an outlined format that includes item tags (to uniquely identify each item/task).
Development Process (cont.) Administrative Guidelines Provide detailed instructions for how to prepare, administer, and collect results. Quality Assurance & Form Reviews Review all operational forms, scoring keys/rubrics, and guidelines to ensure pre- established quality expectations have been met. Determine the overall rigor and alignment at the item/task-level and for the overall operational form.
Helpful Materials Guides Handout #1 – Purpose Statement Examples Handout #2 – Targeted Standards Example Handout #3 – Test Specifications & Blueprint Example Handout #4 – Depth of Knowledge (DoK) Chart Handout #5 – Item Examples Handout #6 – Scoring Key Example Handout #7 – Rubric Examples Handout #8 – Operational Form Example Handout #9 – Administrative Guidelines Example Handout #10 – Quality Assurance Checklist
Helpful Materials (cont.) Templates Template #1-Purpose Statement Template #2-Targeted Standards Template #3-Test Specifications & Blueprint Template #4-Scoring Key Template #5-Operational Form Template #6-Administrative Guidelines
Helpful Material (cont.) Stuff “Smart Book” Provides techniques and technical criteria for developing measures. Performance Rigor Checklist A quick screening tool used to examine item/task quality.
Principles of Well-developed Measures Measures must: Be built to achieve the designed purpose Produce results that are used for the intended purpose Align to targeted content standards Contain a balance between depth and breadth of targeted content Be standardized, rigorous, and fair Be sensitive to testing time and objectivity Be valid and reliable
Quality Assurance Rubrics Evaluating Measures Quality Assurance Rubrics Three (3) versions- Teacher District State or Vendor Comprehensive tools that evaluate the technical process used in developing performance measures. Organized into strands and provides an overall rating.
Decision-Making Approaches Build Consensus Approach All opinions expressed Ideas of all participants gathered and synthesized Group agreement reached Team Leader Approach Guiding questions Group discussion about the pros and cons Create a “best fit” course of action
Think-Pair-Share 5 minutes to think about the statement below and write down your thoughts 5 minutes to share your ideas with a partner 10 minutes to share with your group “The Assessment Literacy process will help me as an educator by _______________.”
Summary & Next Steps Summary Orientation Module Next Steps Introduced the Assessment Literacy process used to create high quality measures of student achievement. Next Steps Module 1: Design & Purpose Statement Designing performance measures, including creating purpose statements and selecting academic content standards.