1. Spring 2014 Caveon Data Interpretation & Appeal Procedures Jeremy Billington Gabriel Colorado 1

2. Introduction Pearson, currently contracted by FLDOE for the Statewide FCAT/FCAT 2.0/EOC Assessments, subcontracts with Caveon Test Security to perform analyses on all data to ensure reliability of results. –Similarity: answer copying, collusion –Erasures: tampering –Gains: pre-knowledge, coaching The purpose of these analyses is only to release results that were achieved independently. 2

3. Glossary of Terms Similarity Index: A probability value (expressed as a number) that measures whether the number of correct and incorrect matches shared by the tests with the greatest agreement is anomalous. Questions in Common: The number of test items a student’s test form has in common with the form(s) of the student(s) he/she is flagged with. Correct Matches: The number of questions in common that were correctly answered and are identical between a student’s test and the test(s) of the student(s) he/she is flagged with. Incorrect Matches: The number of questions in common that were incorrectly answered and are identical between a student’s test and the test(s) of the student(s) he/she is flagged with. Percent Match: The percentage (out of questions in common) of identical responses (correct and incorrect) found between a student’s test and the test(s) of the student(s) he/she is flagged with. 3

4. Student Level Analysis Students are ONLY flagged based on similar answer patterns. –Caveon calculates frequencies based on the answers students selected for each question on the test by administration. –Based on those frequencies, Caveon creates an expected number of correct and incorrect responses students should have in common with one another. –Students are measured against the statewide expectation. The more anomalous a test’s similarity is with another test as compared to results across the state, the higher the similarity index will be. 4

5. Similarity Index Caveon creates an index for each student. Examples: 0.9, 14.0, 2.4, 5.7, 33.1 –For interpretation, the index tells the probability that the student took the test independently. Example: 2.0 = 1 in 100 chance of tests being taken independently. Example: 3.0 = 1 in 1,000 chance of tests being taken independently. Example: 2.4 = Between a 1 in 100 and a 1 in 1,000 chance of tests being taken independently. Example: 14.0 = 1 in 100,000,000,000,000 chance of tests being taken independently. –The higher the index, the larger or more egregious the anomaly. 5

6. Similarity Index The test results of any student with a similarity index of 12.0 or greater is invalidated and not released. –Probability of taking tests independently with an index of 12.0 = 1 in 1,000,000,000,000 (one in one trillion). Students with invalidated scores do not receive a score or score report during the first round of reporting. Invalidations are not always final. Each district has the opportunity to submit an appeal for any student whose score they believe should be released. 6

7. Student Invalidation Spreadsheets After each test administration, the Assessment Office will post spreadsheets on PearsonAccess listing each student flagged by Caveon. –The spreadsheets are posted when the first round of data is released to districts. –If multiple subjects are released at the same time, the spreadsheets may contain more than one subject or grade. This will be reflected in the name of the file. Once the spreadsheets are available, the Assessment Office will notify districts by email. The Appeals Guide will be attached to the email. This guide helps districts with the procedures for appealing invalidations. –See Slide 21 for more information about the Caveon Appeals Guide. 7

8. Spreadsheet Overview Student IDLast Name First Name MIGradeDistrictSchool District Name School NameSubject Test Group Passed Similarity Index Cluster ID Questions in Common Correct Matches Incorrect Matches Percent Match 1234567890BYRDMACK N12001234SAND SUNSHINE HIGH SCHOOL ALG1234NO21.512350103080.00 999999999XBLOSSOMDORANG12001234SAND SUNSHINE HIGH SCHOOL ALG1234NO21.512350103080.00 111111111XPALMSYBIL8001001SAND BEACH MIDDLE SCHOOL BIO2235YES14.25666511498.48 111223333XLOVEMANNYT 8001001SAND BEACH MIDDLE SCHOOL BIO2225YES14.25666511498.48 555555555XSTONEMOEN8001001SAND BEACH MIDDLE SCHOOL BIO2225YES12.35666401177.27 Here is an example of the spreadsheet that will be posted on PearsonAccess for districts with student invalidations. Each district should have login information for PearsonAccess to download the spreadsheet. If not, please contact the Assessment Office. The file name will tell you the district and test administration of invalidations. Example: 2013_Winter_EOC_Invalidations_District00.xls 8

9. Student Information Student IDLast Name First Name MIGradeDistrictSchool District Name School NameSubject Test Group Passed Similarity Index Cluster ID Questions in Common Correct Matches Incorrect Matches Percent Match 1234567890BYRDMACK N12001234SAND SUNSHINE HIGH SCHOOL ALG1234NO21.512350103080.00 999999999XBLOSSOMDORANG12001234SAND SUNSHINE HIGH SCHOOL ALG1234NO21.512350103080.00 111111111XPALMSYBIL8001001SAND BEACH MIDDLE SCHOOL BIO2235YES14.25666511498.48 111223333XLOVEMANNYT 8001001SAND BEACH MIDDLE SCHOOL BIO2225YES14.25666511498.48 555555555XSTONEMOEN8001001SAND BEACH MIDDLE SCHOOL BIO2225YES12.35666401177.27 Variables include: Student ID number, Last Name, First Name, Middle Initial, Grade, District, School, District Name, School Name Example: Mack tested at Sunshine High School in Sand County. Example: Sybil is an 8 th grade student at Beach Middle School. 9

10. Subject Student IDLast Name First Name MIGradeDistrictSchool District Name School NameSubject Test Group Passed Similarity Index Cluster ID Questions in Common Correct Matches Incorrect Matches Percent Match 1234567890BYRDMACK N12001234SAND SUNSHINE HIGH SCHOOL ALG1234NO21.512350103080.00 999999999XBLOSSOMDORANG12001234SAND SUNSHINE HIGH SCHOOL ALG1234NO21.512350103080.00 111111111XPALMSYBIL8001001SAND BEACH MIDDLE SCHOOL BIO2235YES14.25666511498.48 111223333XLOVEMANNYT 8001001SAND BEACH MIDDLE SCHOOL BIO2225YES14.25666511498.48 555555555XSTONEMOEN8001001SAND BEACH MIDDLE SCHOOL BIO2225YES12.35666401177.27 Indicates EOC subject, R (Reading), M (Mathematics), or S (Science). – Subject for Retake students is listed as R (Reading) or M (Mathematics). Example: Mack and Doran took the Algebra 1 EOC test. Example: Sybil, Manny, and Moe took the Biology 1 EOC test. 10

11. Test Group Student IDLast Name First Name MIGradeDistrictSchool District Name School NameSubject Test Group Passed Similarity Index Cluster ID Questions in Common Correct Matches Incorrect Matches Percent Match 1234567890BYRDMACK N12001234SAND SUNSHINE HIGH SCHOOL ALG1234NO21.512350103080.00 999999999XBLOSSOMDORANG12001234SAND SUNSHINE HIGH SCHOOL ALG1234NO21.512350103080.00 111111111XPALMSYBIL8001001SAND BEACH MIDDLE SCHOOL BIO2235YES14.25666511498.48 111223333XLOVEMANNYT 8001001SAND BEACH MIDDLE SCHOOL BIO2225YES14.25666511498.48 555555555XSTONEMOEN8001001SAND BEACH MIDDLE SCHOOL BIO2225YES12.35666401177.27 Test Group Code identifies groups of students that tested together. – Example: Mack and Doran tested in Test Group 1234. Because this is typically a student-entered field, the test group code may not always be accurate, but can be confirmed during the investigation. – Example: Sybil tested in Test Group 2235, Manny and Moe tested in Test Group 2225. Because these three students were flagged together, it is likely that Sybil made an error when gridding her Test Group Code. 11

12. Passed Student IDLast Name First Name MIGradeDistrictSchool District Name School NameSubject Test Group Passed Similarity Index Cluster ID Questions in Common Correct Matches Incorrect Matches Percent Match 1234567890BYRDMACK N12001234SAND SUNSHINE HIGH SCHOOL ALG1234NO21.512350103080.00 999999999XBLOSSOMDORANG12001234SAND SUNSHINE HIGH SCHOOL ALG1234NO21.512350103080.00 111111111XPALMSYBIL8001001SAND BEACH MIDDLE SCHOOL BIO2235YES14.25666511498.48 111223333XLOVEMANNYT 8001001SAND BEACH MIDDLE SCHOOL BIO2225YES14.25666511498.48 555555555XSTONEMOEN8001001SAND BEACH MIDDLE SCHOOL BIO2225YES12.35666401177.27 This column indicates whether the student passed the test. The Passed variable will only be populated for Grade 10 Reading, Reading Retake, Mathematics Retake, and the Algebra 1, Biology 1, Geometry, and U.S. History EOCs. Example: Mack and Doran did not pass the test. Example: Sybil, Manny, and Moe passed the test. 12

13. Similarity Index Student IDLast Name First Name MIGradeDistrictSchool District Name School NameSubject Test Group Passed Similarity Index Cluster ID Questions in Common Correct Matches Incorrect Matches Percent Match 1234567890BYRDMACK N12001234SAND SUNSHINE HIGH SCHOOL ALG1234NO21.512350103080.00 999999999XBLOSSOMDORANG12001234SAND SUNSHINE HIGH SCHOOL ALG1234NO21.512350103080.00 111111111XPALMSYBIL8001001SAND BEACH MIDDLE SCHOOL BIO2235YES14.25666511498.48 111223333XLOVEMANNYT 8001001SAND BEACH MIDDLE SCHOOL BIO2225YES14.25666511498.48 555555555XSTONEMOEN8001001SAND BEACH MIDDLE SCHOOL BIO2225YES12.35666401177.27 Similarity Index: A probability value (expressed as a number) that measures whether the number of correct and incorrect matches shared by the tests with the greatest agreement is anomalous. Example: Sybil and Manny have a Similarity Index of 14.2, which means they have over a 1 in 100,000,000,000,000 probability of having taken the Biology 1 EOC test independently. 13

14. Cluster ID Student IDLast Name First Name MIGradeDistrictSchool District Name School NameSubject Test Group Passed Similarity Index Cluster ID Questions in Common Correct Matches Incorrect Matches Percent Match 1234567890BYRDMACK N12001234SAND SUNSHINE HIGH SCHOOL ALG1234NO21.512350103080.00 999999999XBLOSSOMDORANG12001234SAND SUNSHINE HIGH SCHOOL ALG1234NO21.512350103080.00 111111111XPALMSYBIL8001001SAND BEACH MIDDLE SCHOOL BIO2235YES14.25666511498.48 111223333XLOVEMANNYT 8001001SAND BEACH MIDDLE SCHOOL BIO2225YES14.25666511498.48 555555555XSTONEMOEN8001001SAND BEACH MIDDLE SCHOOL BIO2225YES12.35666401177.27 New field starting in Spring 2014. Arbitrary number (one to three digits), unique by subject. Identifies groups of students that were flagged with each other. Example: Mack and Doran are in cluster 123. Example: Sybil, Manny, and Moe are all flagged together in cluster 56. – Moe doesn’t have as many similar responses as Sybil and Manny do, but he still has enough matches with Sybil and Manny to cause him to be flagged with them. 14

15. Questions in Common Student IDLast Name First Name MIGradeDistrictSchool District Name School NameSubject Test Group Passed Similarity Index Cluster ID Questions in Common Correct Matches Incorrect Matches Percent Match 1234567890BYRDMACK N12001234SAND SUNSHINE HIGH SCHOOL ALG1234NO21.512350103080.00 999999999XBLOSSOMDORANG12001234SAND SUNSHINE HIGH SCHOOL ALG1234NO21.512350103080.00 111111111XPALMSYBIL8001001SAND BEACH MIDDLE SCHOOL BIO2235YES14.25666511498.48 111223333XLOVEMANNYT 8001001SAND BEACH MIDDLE SCHOOL BIO2225YES14.25666511498.48 555555555XSTONEMOEN8001001SAND BEACH MIDDLE SCHOOL BIO2225YES12.35666401177.27 Questions in Common: The number of test items a student’s test form has in common with the form(s) of the student(s) he/she is flagged with. – Just because two students have different forms does not mean they can not be flagged together for similarity. In order to equate the different test forms, there are common items across them. Example: Mack’s test form has 50 questions in common with Doran’s test form. Example: Sybil’s test form, Manny’s test form, and Moe’s test form all have 66 questions in common with one another. 15

16. Correct Matches Student IDLast Name First Name MIGradeDistrictSchool District Name School NameSubject Test Group Passed Similarity Index Cluster ID Questions in Common Correct Matches Incorrect Matches Percent Match 1234567890BYRDMACK N12001234SAND SUNSHINE HIGH SCHOOL ALG1234NO21.512350103080.00 999999999XBLOSSOMDORANG12001234SAND SUNSHINE HIGH SCHOOL ALG1234NO21.512350103080.00 111111111XPALMSYBIL8001001SAND BEACH MIDDLE SCHOOL BIO2235YES14.25666511498.48 111223333XLOVEMANNYT 8001001SAND BEACH MIDDLE SCHOOL BIO2225YES14.25666511498.48 555555555XSTONEMOEN8001001SAND BEACH MIDDLE SCHOOL BIO2225YES12.35666401177.27 Correct Matches: The number of questions in common that were correctly answered and are identical between a student’s test and the test(s) of the student(s) he/she is flagged with. Example: Mack and Doran have 10 questions in common that were answered with the identical correct response. Example: Sybil and Manny have 51 questions in common that were answered with the identical correct response. Sybil, Manny, and Moe have 40 questions in common that were answered with the identical correct response. 16

17. Incorrect Matches Student IDLast Name First Name MIGradeDistrictSchool District Name School NameSubject Test Group Passed Similarity Index Cluster ID Questions in Common Correct Matches Incorrect Matches Percent Match 1234567890BYRDMACK N12001234SAND SUNSHINE HIGH SCHOOL ALG1234NO21.512350103080.00 999999999XBLOSSOMDORANG12001234SAND SUNSHINE HIGH SCHOOL ALG1234NO21.512350103080.00 111111111XPALMSYBIL8001001SAND BEACH MIDDLE SCHOOL BIO2235YES14.25666511498.48 111223333XLOVEMANNYT 8001001SAND BEACH MIDDLE SCHOOL BIO2225YES14.25666511498.48 555555555XSTONEMOEN8001001SAND BEACH MIDDLE SCHOOL BIO2225YES12.35666401177.27 Incorrect Matches: The number of questions in common that were incorrectly answered and are identical between a student’s test and the test(s) of the student(s) he/she is flagged with. Example: Mack and Doran have 30 questions in common that were answered with the identical incorrect response. Example: Sybil and Manny have 14 questions in common that were answered with the identical incorrect response. Sybil, Manny, and Moe have 11 questions in common that were answered with the identical incorrect response. 17

18. Percent Match Student IDLast Name First Name MIGradeDistrictSchool District Name School NameSubject Test Group Passed Similarity Index Cluster ID Questions in Common Correct Matches Incorrect Matches Percent Match 1234567890BYRDMACK N12001234SAND SUNSHINE HIGH SCHOOL ALG1234NO21.512350103080.00 999999999XBLOSSOMDORANG12001234SAND SUNSHINE HIGH SCHOOL ALG1234NO21.512350103080.00 111111111XPALMSYBIL8001001SAND BEACH MIDDLE SCHOOL BIO2235YES14.25666511498.48 111223333XLOVEMANNYT 8001001SAND BEACH MIDDLE SCHOOL BIO2225YES14.25666511498.48 555555555XSTONEMOEN8001001SAND BEACH MIDDLE SCHOOL BIO2225YES12.35666401177.27 10 Correct Matches+ 30 Incorrect Matches 50 Questions in Common = 80 Percent Match 18 Percent Match: The percentage (out of questions in common) of identical responses (correct and incorrect) found between a student’s test and the test(s) of the student(s) he/she is flagged with. 40 Identical Responses ( )

19. Informing Parents and Guardians The Assessment Office provides a courtesy parent/guardian letter template for districts to use to communicate test score invalidations to parents or guardians. –The parent/guardian letter has been translated into Spanish and Haitian-Creole. –The parent/guardian letter may be provided on your district letterhead and customized to meet your district’s needs. The parent/guardian letter template is found in the Appeals Guide. 19

20. Timeline for Appeals Districts are given a deadline to appeal any invalidations. –This date is included in the initial email sent to notify districts that Caveon invalidations are available in PearsonAccess. After it has completed its review, the Assessment Office will post letters for each appeal the district submitted. –These letters will have the final appeal decision (whether the score will be invalidated or released). –The Assessment Office will notify districts by email when these letters are available. Released scores will be reflected in the corresponding late reporting data files and reports. –Districts will receive student reports for their records and for the student/parent. 20

21. 21 Appeals Guide The Appeals Guide is updated regularly. –Please review this guide for each administration as it may contain updates. The guide outlines helpful documentation that leads to a successful appeal. –Providing as much requested information as possible helps the Assessment Office make the most informed decision. –If there is any piece of documentation that is unobtainable, EXPLICITLY state this in your appeal with the reason.  Example: We were unable to contact Mack because he has moved to a different state. Attempts to contact him by using the phone number on file were unsuccessful.

22. 22 Interviewing Students Interviewing ALL flagged students is very important. –Most scores that are released are released because one of the flagged students admits to guilt. –Even if a district/school is strongly advocating for one student, give ALL flagged students the opportunity to provide their side of the story. If you believe other students (e.g. other students in the testing room, friends of flagged students) may have insight into an investigation, interview them as well. –However, we do not require interviews from every student in a testing room. Only provide interview results that inform the investigation. See the information in the Interviewing Students, Test Administrators, and Proctors section of the Appeals Guide on conducting student interviews.

23. 23 Advocating for Students If the investigation results are inconclusive, the district/school may advocate for a specific student. –This occurs when one student is thought to be innocent of collusion. –Example: Mack has a high GPA, especially in the subject area tested, teachers speak highly of Mack, seating chart shows Mack seated in front of the other student, and Mack has little to no discipline history. When advocating for a student, please provide documentation for ALL flagged students. –Example: Transcripts that show Mack is high-performing in the subject area, and Doran is low-performing in the subject area. –Give ALL students’ information for a fair and accurate appeal decision.

24. 24 Graduating Seniors Appeals are less restrictive for students who are on track to graduate, and require only the invalidated score to graduate. –As with other appeals, test security protocols must have been observed, with no indications of suspicious activity. –The student must have passed the exam. This information is reflected on the spreadsheet. –Appeal should detail all earned requirements for graduation and/or that student is on track to meet requirements to graduate in the CURRENT semester.  GPA  Credits earned –Reference Florida Statute 1003.428 for high school graduation requirements.Florida Statute 1003.428

25. Questions/Discussion? Jeremy Billington 850-245-5158 jeremy.billington@fldoe.org Gabriel Colorado 850-245-0779 gabriel.colorado@fldoe.org 25