1. Automated Scoring Yes, It Is Ready for Prime Time Julie Quinn Coordinator Test Administration and Data Assessment and Accountability Utah State Office of Education julie.quinn@schools.utah.gov julie.quinn@schools.utah.gov

2. 2009 Utah Legislature required an online writing assessment for grades 5 and 8 (53A-1-603) Contract awarded for formative and summative online writing assessment, exclusive AI scoring, grades 5 and 8 Many LEAs provide formative online writing for other grade levels at their own cost 2

3. In the Beginning Formative Educator skepticism about AI scoring validity and reliability Positive response to immediate scoring and increased opportunities for writing feedback Vendor and teacher prompts

4. In the Beginning Summative Misunderstandings about AI engine training and scoring rules Positive response to online administration and AI identification of alert papers Contract for AI scoring only, vendor provided human read-behind

5. AI vs. Human Scoring 2010 Grade 5 Agreement Rates N=98 Reader 1 vs. Reader 2 98.3% PEG vs. Reader 198.98% PEG vs. Reader 297.96%

6. AI vs. Human Scoring 2010 Grade 8 Agreement Rates N=99 Reader 1 vs. Reader 2 100% PEG vs. Reader 197.81% PEG vs. Reader 299.67%

7. Outcomes Students are more engaged in writing with immediate feedback The number of writing events with feedback has greatly increased AI is consistent and accurate Utah will continue to use AI exclusively for summative writing assessments