1. Managing Multi-State Assessment Consortia Lessons From American Diploma Project Network Assessment Consortium

2. 2 AMERICAN DIPLOMA PROJECT NETWORK n Background: ADP Assessment Consortium n Consortium Organization and Governance n Potential Application Requirements: Demonstrating Capacity for Success n Increasing Odds of Success

3. 3 AMERICAN DIPLOMA PROJECT NETWORK American Diploma Project Assessment Consortium n Initiated in 2005 by 9 states in Achieve-led ADP Network l consortium has grown to 15 states n State share broader college- and career-ready policy agenda n Started with Algebra II EOC exam, expanded to include Algebra I n Agreed-upon policy purposes l Improve curriculum and instruction and ensure consistency of content and rigor l Enable states to compare results across and within states l Indicate readiness for college credit bearing mathematics n Agreement to college-ready cut score set by Achieve

4. 4 AMERICAN DIPLOMA PROJECT NETWORK ADP Assessment Consortium Arkansas, Indiana, Kentucky, Maryland, Massachusetts, New Jersey, Ohio, Pennsylvania, Rhode Island Ohio acted as “lead state” in unprecedented multi-state procurement arrangement Pearson awarded contract in 2007 with addition of Algebra I in 2008 Since the consortium began, six additional states have joined: Arizona, Florida, Hawaii, Minnesota, North Carolina, and Washington. More states are expressing interest in the exam and joining the consortium … In summer 2006, nine states issued RFP for the development of an Algebra II EOC Exam:

5. 5 AMERICAN DIPLOMA PROJECT NETWORK Consortium Organization and Governance n ADP Consortium Member States n Ohio – lead state n Coordination and Direction Team (CDT) n Achieve n Center for Assessment – technical advisor n Pearson – test developer selected competitively

6. 6 AMERICAN DIPLOMA PROJECT NETWORK 6 New State Members n Membership is extended to any ADP Network state wanting to join n New member states must: l Agree to terms of Partnership Agreement and MOU l Agree to terms stated in the contract (but may add language) l Participate in all CDT meetings, item development and review meetings l Make firm, long range commitments u Contractual u Personnel u Financial

7. 7 AMERICAN DIPLOMA PROJECT NETWORK 7 Role of the Lead State: Ohio n Appoint a procurement officer n Conduct procurement n Provide guidance to the CDT n Administer the contract l Manage contract amendments l Incorporate new states into the contract

8. 8 AMERICAN DIPLOMA PROJECT NETWORK 8 The Coordination and Direction Team n Includes assessment directors or other high-ranking policy-making officials from each member state in the consortium n Oversees production and implementation of the Algebra I & II program n Ensures that legal and policy needs of each state are addressed during team deliberations and decision making

9. 9 AMERICAN DIPLOMA PROJECT NETWORK 9 The CDT and Decision Making n The CDT states have: l Multiple representatives l One vote n Process of consensus, collaboration, and compromise: l Relating to the policies of each state l Repeated as necessary l Less discussion time needed as process matures n For contractual matters, decisions must be unanimous

10. 10 AMERICAN DIPLOMA PROJECT NETWORK Achieve’s Role n Convener and consensus builder l Brought chiefs, state content and assessment directors, governors & postsecondary leaders to the table n Consortium Management l Communications with and among states l Consensus building l Project manager/oversee development contract l Assembled additional expertise (technical, legal) n Content expertise l ADP Benchmark standards l Model course descriptions

11. 11 AMERICAN DIPLOMA PROJECT NETWORK Achieve’s Role n Promote Postsecondary participation l Math faculty at 2- and 4-year institutions l System leadership l With key national higher education organizations n Define college readiness l Conceptually l Set achievement levels and cut scores

12. 12 AMERICAN DIPLOMA PROJECT NETWORK Key Lessons for States: Participating in a Consortium Takes Time and Communication n Active involvement of chief state school officer in start up phase is essential, so that state policy objectives and improvement strategy drive assessment design n Essential but difficult to sustain their engagement over time, as project moves to development and implementation phases n Governors and legislatures play critical roles in state testing policy; they must be engaged and informed along the way n The transaction costs for SEA staff are high compared with single-state assessment development

13. 13 AMERICAN DIPLOMA PROJECT NETWORK Key Lessons for States: It’s Not Just About the Test n Developing common summative assessments is difficult n Summative assessments not enough – coherent approach that includes formative and interim assessments, curriculum and instructional materials, professional development is necessary to support instructional improvement n That requires more ambitious effort, and broader and deeper consensus among participating states n Common standards provides critical foundation for such an approach

14. 14 AMERICAN DIPLOMA PROJECT NETWORK Key Lessons for States: Shared Vision is Essential n States must have common standards and must hold firm to a shared vision: l of policy objectives l of uses and purposes of assessments l of assessment system design n Scope of the vision will determine other practices must also be held in common. The more comprehensive and coherent the vision, the more issues states must address in common n Within a consortium, willingness to compromise on behalf of larger vision and shared goals is essential l To develop summative tests alone, states must still be willing to compromise on scores of details (e.g. testing windows, testing modes, test security, test guidelines, calculator policy for mathematics, accommodations, etc.) l Additional compromise will be needed in more ambitious approaches

15. 15 AMERICAN DIPLOMA PROJECT NETWORK Potential Application Requirements: Consortia Should be Asked to Demonstrate: High degree of consensus around: n Policy Objectives l Instructional Improvement l Vision of assessment system n Leadership Capacity in each state l Support of Governor l Active Leadership by and commitment of Chief State School Officer (not just state testing directors) and Postsecondary leaders (for college-readiness) l Engagement of State Board of Education and Legislature as appropriate in each state

16. 16 AMERICAN DIPLOMA PROJECT NETWORK Potential Application Requirements: Consortia Should be Asked to Demonstrate: n Technical capacity across the consortium, matched to assessment system design n Consortium Governance and Decision-Making Structure n Consortium Procurement Strategy n Roles and Responsibilities of Consortium members and partners, including supporting organizations

17. 17 AMERICAN DIPLOMA PROJECT NETWORK Increasing the Odds of Success Over Time Applications should include a sustainability plan that addresses: n Anticipated (and unanticipated) leadership turnover, especially for governors and chiefs n Strategies for building and sustaining support among key stakeholders l Ongoing communications with teachers, students and parents about purposes and uses of assessments n Annual assessment costs, especially if: l Projected annual costs expected to exceed current costs l Continuous development (e.g., new items) is part of design, beyond life of federal grant n Ongoing commitment of partner organizations through grant period

18. Managing Multi-State Assessment Consortia Lessons From American Diploma Project Network Assessment Consortium