Getting the Most Value for Your Assessment Dollar – Designing Adapting and Maintaining Quality Assessment Programs During Tough Economic Times To Consortia, or not to Consortia CCSSO National Conference on Student Assessment June 23, 2010

Joining a Consortium Implementing a new, innovative assessment program in a consortium as a way to save costs. Or, maintaining a current program without having to make drastic cuts Is it doable? Can a consortia of states implement a new assessment at a significantly reduced cost than a single state acting alone? How large does the consortia need to be? Where are the cost savings opportunities?

Joining a State Assessment Consortium Joining a state assessment consortium can have its advantages but... Requires a lot of planning, coordination & desire Several successful examples: NECAP WIDA Achieve Algebra 2 PARCC & SBAC (responses due today) June 23,

Stanford/Nellie Mae Study Purpose of the study was to see if it is possible to create an affordable “high quality” assessment Step one – Model a current typical assessment in ELA and Math – Cost $19-$20 Step two – Model a high quality assessment for the same state – Cost $55-$56 a student Step three – Implement several cost savings strategies June 23, 2010 Assessment Solutions Group

Cost Reduction Strategies Participation in a consortium Looked at 10, 20 and 30 state sizes Cost reduction - $15 per student Uses of technology for online test delivery, distributed human scoring of some of the open-ended items, and automated scoring for certain constructed response items Together, these innovations account for cost savings of about $3 to $4 per student Likely to account for more as efficiencies are developed in programming and using technology for these purposes Two approaches to the use of teacher-moderated scoring. Teacher-moderated scoring can net both substantial cost reductions as well as potential professional development benefits. We used two different models for teacher-moderated scoring 5

Cost Reduction Strategies Two different models for teacher-moderated scoring: Professional development model - no additional teacher compensation beyond that supported by the state or district for normal professional development days (NY Regents) Stipend model - assume a $125/day stipend for teachers to score the performance items. Note: teachers were assumed to score all performance items in a distributed scoring model These strategies for using teachers as scorers reduce costs by an additional $10 to $20 per pupil (depending on whether teachers are engaged as part of professional development or are paid) Adopting all cost reduction strategies while paying teachers a $125/day stipend to score all performance tasks results in an assessment cost of $21 June 23, 2010 Assessment Solutions Group

Consortia Size How big do you have to be? Stanford/Nellie Mae study found that 80% of the cost benefits of joining a consortium are realized at the 10 state size. Rough estimate is that a 5 state consortium could achieve 75%+ of the cost savings of a 10 state consortia Perhaps $3 - $6 per student $2.7 M/year for the average sized state (600K students) June 23,

Where are the Cost Savings? Big cost savings opportunity in development Largely a fixed cost function Increase in forms cost partially offsets the savings Other fixed cost functions such as IT, Quality Assurance and Psychometrics provide savings Even functions that are largely variable in nature also have a fixed cost component Some functions like program management allow for economies of scale June 23,

Assessment Costs by Consortium Size June 23,

Where are the Cost Savings? Consortia size can make assessment technology more affordable Online test delivery (CBT and CAT) Artificial intelligence scoring of CRs More states/students more bargaining power A common assessment with common standards and operational methods s/b more efficient Need to weigh this against potential additional collaboration costs and risks June 23,

PARCC & SBAC Support We recently assisted both consortia in preparing their cost estimates for the NIA responses Both consortia had innovative ideas for new assessments and a wide variety of design and operational decisions to make Each idea/design choice came with unique cost implications June 23, 2010www.assessmentgroup.org11

PARCC & SBAC Support Initially, each consortia’s design was deemed too expensive in both the operational and ongoing periods. Each needed adjustments: The number of choices and variables can be daunting as there are many variables and moving parts Ultimately, each consortia created innovative assessment systems with the designs they wanted June 23,

Assessment Design Decision Tree Delivery Method Paper based Computer (linear or CAT) Mixed (both CBT and PPT) Assessment Types Summative, through course summative Interim/benchmark, End of Course, Formative Domains, special populations Indicates a major cost element for either PARCC or SBAC June 23,

Decisions and Cost Variables (cont.) Development Types of items (SR, CR, Computer enhanced, PE, PT) Mix of item types Number of forms, CAT algorithm ( items per grade), number of attempts Release rates (by item type) Breach form (develop?, print?) Grades/domains tested Item bank development June 23,

Decisions and Cost Variables Paper based testing/cutover to CBT How long to cut over (operating in both modes is very expensive)? Different production strategies Minimize print page “signatures” Use of color (B/W, grey scale, 4 color) Breach form (print?) Security measures (# of forms, labels, seals, student ID) June 23, 2010www.assessmentgroup.org15

Decisions and Cost Variables Logistics Transportation mode (ground, air) Carrier selection Ship from/to locations (consolidated shipping) Meetings and Travel (online vs. live) Scoring Computer vs. Human (incl. scanning and editing) June 23,

Design Decisions & Costs Scoring (cont.) Human Method (teacher or 3 rd party) Holistic vs. analytic scoring Requires a lot of work to develop innovative items that can be scored in a timely manner Alternatively, a test design where these items are scored during a classroom period may make sense (PEs) AI scoring for open ended items Math vs. ELA Items requiring inference can’t easily be scored using AI System training fees (fixed cost); per score costs June 23,

Design Decisions & Costs Open-Ended Scoring (cont.) Double scoring/Read behind rates (by grade) Distributed vs. on-site Reporting Paper vs. online reporting Number and complexity of reports June 23, 2010www.assessmentgroup.org18

Conclusion Even a small consortium of states can achieve significant reductions in assessment cost Such a strategy can be useful in developing a new, high quality assessment or maintaining a current one during times of budgetary stress Participating in a consortium also allows for the implementation of innovative technologies that can improve assessment quality and reduce costs Teacher scoring of open-ended items is critical for implementing a high quality assessment There are a myriad of design and operational decisions that have significant cost impacts 19

Conclusion “You can’t always get what you want; but if you try sometime you just might find you get what you need.” - Mick Jagger

Questions? Barry Topol John Olson Ed Roeber June 23, 2010www.assessmentgroup.org21