Automated Scoring Yes, It Is Ready for Prime Time Julie Quinn Coordinator Test Administration and Data Assessment and Accountability Utah State Office.

Slides:

Advertisements

Similar presentations

GMC National Trainer Survey What did you say in 2010? (Royal Gwent Hospital)

Advertisements

Teacher Effectiveness Evaluation Pilot September 1, 2011 – September 30, 2012 NJ State Board of Education, July 13, 2011.

Instructional Materials: The Future is Here.. What is Happening Nationally?

DELAWARE COMPREHENSIVE ASSESSMENT SYSTEM Presented to District Test Coordinators February 17, 2010.

Transitioning to On-Line Field Testing for the PATs and the Diploma Exams.

WV Writes: A Roadmap to Writing Success Dr. Vaughn G. Rhudy, NBCT, WESTEST 2 RLA/Online Writing Coordinator West Virginia Department of Education Office.

Holistic Rating Training Requirements Texas Education Agency Student Assessment Division.

SAA UPDATE ON SCHOOL ASSESSMENT Presenting on behalf of the SAA Student Assessment Project Team: Peggy Weber, Principal, Osceola Tim Schell, Director of.

Measuring Performance within School Climate Transformation Grants

Monica Madden October 25,  Superintendent has taken cuts as well. Because of the charter amendment, Supt. office has seen a 25% cut.  There will.

Utah Effective Teaching Standards-based Jordan Performance Appraisal System Orientation (UETS-based JPAS)

Understanding Grade Level and School Growth Reports Office of Assessment and Accountability.

Iowa Assessment Update School Administrators of Iowa November 2013 Catherine Welch Iowa Testing Programs.

Grade 3 Pre-test Test Administrator Training NHCS Testing and Accountability Department September 2012.

1 Judy W. Park, Ed.D., Associate Superintendent, Utah State Office of Education.

Michigan Council for Educator Effectiveness Toward an Improvement-Focused System of Educator Evaluation Jennifer Hammond OCTE Meeting November 7, 2013.

Race to the Top Technology and Innovation in Assessments Boston, MA Tony Alpert Oregon Department of Education.

The Delaware Performance Appraisal System II for Teachers Training Module 4 Component Five – Student Improvement.

Utah Test Item Pool Service UTIPS January 17, 2007 Julie Quinn Accountability/UTIPS Specialist Utah State Office of Education.

Introduction to Assessment Basic Terms and Concepts.

CALIFORNIA DEPARTMENT OF EDUCATION Tom Torlakson, State Superintendent of Public Instruction 2014 Assessment and Accountability Information Meeting Smarter.

How do we document evidence of student growth in Healthful Living?

This document is an editable template. Please make sure to customize it for your district before distributing. Copyright 2014 by The Colorado Education.

UTIPS Core Roll-Out Fall, 2011 Julie Quinn, Data and Delivery Coordinator Assessment and Accountability Utah State Office of Education.

Critical Information SAGE Critical Information 1 Judy Park, Ed.D. Associate Superintendent Utah State Office of Education.

SMARTER Balanced Assessment Consortium Smarter Balanced Assessment Consortium.

Consortia of States Assessment Systems Instructional Leaders Roundtable November 18, 2010.

PARCC Update June 6, PARCC Update Today’s Presentation:  PARCC Field Test  Lessons Learned from the Field Test  PARCC Resources 2.

KEEP And Student Growth Measures for Building Leaders Lawrence School District, May 14, 2014 Bill Bagshaw, Assistant Director, TLA, KSDE Kayeri Akweks,

Company: Account Requests FMCSA Portal Prioritization Phase I Release, December 2010 v1.4.

Automated Scoring: Smarter Balanced Studies CCSSO- NCSA San Diego, CA June, 2015.

Artificial Intelligence Scoring of Student Essays: West Virginia’s Experience Vaughn G. Rhudy, Ed.D., NBCT Office of Assessment West Virginia Department.

Evaluation Tools & Approaches for Engaging Stakeholders: The AREA Approach Presentation at the Canadian Evaluation Society Dr. Mariam Manley, Principal.

Julie Quinn Computer-Based Assessments Specialist Utah State Office of Education 1.

Systems Review: Schoolwide Reading Support Cohort 5: Elementary Schools Winter, 2009.

PARCC Assessments Updates Updates Arrived 2/6/13! general specifics.

South Dakota Department of Education WriteToLearn Gay Pickner Director of Assessment June 22, 2011.

Assessing The Next Generation Science Standards on Multiple Scales Dr. Christyan Mitchell 2011 Council of State Science Supervisors (CSSS) Annual Conference.

CALIFORNIA DEPARTMENT OF EDUCATION Tom Torlakson, State Superintendent of Public Instruction Santa Clara COE Assessment Accountability Network September.

Migrant Education Program Updates for Coordinators Chris McLaughlin Virginia Department of Education Office of Program Administration and Accountability.

Automated Scoring is a Policy and Psychometric Decision Christina Schneider The National Center for the Improvement of Educational Assessment

Future Ready Schools English Language Proficiency Testing in North Carolina: English Language Proficiency Testing in North Carolina: A Brief Introduction.

ASSESSMENT, STANDARDS, & ACCOUNTABILITY. WHY IS ASSESSMENT IMPORTANT? Used to gather information and make decisions about student learning Formal tests.

1 Standard Test Administration Testing Ethics Training PowerPoint Spring 2011 Utah State Office of Education.

CALIFORNIA DEPARTMENT OF EDUCATION Tom Torlakson, State Superintendent of Public Instruction Overview of the Smarter Balanced Assessment System CTA Pre-Good.

Ort workshop on e-assessing ICT  By Terry Freedman.

Holistic Rating Training Requirements Texas Education Agency Student Assessment Division ©2010 TEXAS EDUCATION AGENCY. ALL RIGHTS RESERVED.

Race to the Top General Assessment Session Atlanta, Georgia November 17, 2009 Louis M. (Lou) Fabrizio, Ph.D. Director of Accountability Policy & Communications.

Grant Proposal Jeffrey Linton.  Engagement  Well-Planned Instruction  Time?  Story.

SD WriteToLearn Assessment Pilot TIE Conference April 20, 2010.

Human Society and its Environment K-6 (HSIE) Using the syllabus for consistency of assessment © 2006 Curriculum K-12 Directorate, NSW Department of Education.

TOMS and Digital Library ACCOUNTS 1 REF-6578 Getting Ready for Your TOMS Accounts Principals 1.Designate CAASPP Coordinator in the Principal’s Portal.

Standards Instruction Assessment Feedback. Session Outcomes Participants will:  Develop an understanding of the Test and Quizzes application in BCPS.

2011–2012 Holistic Rating Training Requirements Texas Education Agency Student Assessment Division.

Educator Effectiveness Summit School District’s Recommendation for the School Year.

Updates on State Assessments Steven Katz Director Office of State Assessment.

29 States $176,000,000 for development Includes formative, interim & summative Governed and controlled by states Co-chairs, Judy Park, Utah; Tony Alpert,

Assessment and the Quality of Learning  Summative assessment: Gives a numerical mark or grade at the conclusion of a term of study.  Formative assessment:

1 Rose Hermodson Assistant Commissioner Minnesota Department of Education December 13, 2011 Teacher Evaluation Components in Legislation.

Risk Management: Assessment, Education, & Action Katie Wilson | Director, Armstrong Student Center | Miami University Ryan O’Connell | Product Manager.

GRADING n n Grading that reflects the actual ability of the student n n Grading as a personal communication.

Missouri’s Experience with Automated Scoring National Conference on Student Assessment June 2014.

SAGE Spring Summative Training Assessment and Accountability Utah State Office of Education.

Kristen A Condella MCSD Mathematics Coordinator. Mathematics Formative Assessment System (MFAS) Formative assessment is a process used by teachers and.

New State Law goes into effect July 1, 2018 (HB1091)

Updates on State Assessments

North Carolina Observation Calibration Process (OCT)

Access to HE Standardisation Event Science, Engineering and Maths Thursday, 10th December 2015, 9.30 am to pm ONE Awards offices, Peterlee.

Writing Performance Assessment

TOMS and Digital Library ACCOUNTS

Presentation transcript:

Automated Scoring Yes, It Is Ready for Prime Time Julie Quinn Coordinator Test Administration and Data Assessment and Accountability Utah State Office of Education

2009 Utah Legislature required an online writing assessment for grades 5 and 8 (53A-1-603) Contract awarded for formative and summative online writing assessment, exclusive AI scoring, grades 5 and 8 Many LEAs provide formative online writing for other grade levels at their own cost 2

In the Beginning Formative Educator skepticism about AI scoring validity and reliability Positive response to immediate scoring and increased opportunities for writing feedback Vendor and teacher prompts

In the Beginning Summative Misunderstandings about AI engine training and scoring rules Positive response to online administration and AI identification of alert papers Contract for AI scoring only, vendor provided human read-behind

AI vs. Human Scoring 2010 Grade 5 Agreement Rates N=98 Reader 1 vs. Reader % PEG vs. Reader % PEG vs. Reader %

AI vs. Human Scoring 2010 Grade 8 Agreement Rates N=99 Reader 1 vs. Reader 2 100% PEG vs. Reader % PEG vs. Reader %

Outcomes Students are more engaged in writing with immediate feedback The number of writing events with feedback has greatly increased AI is consistent and accurate Utah will continue to use AI exclusively for summative writing assessments