WDE Test Security Training: Statewide Assessments February 2016

Understanding the importance of test security Common test security issues How WDE detects possible test security issues Possible consequences of test security issues Questions Session Overview

Statewide assessments rely on the accurate measurement of individual achievement. Any deviation from procedures meant to ensure test validity and security (e.g., group work, teacher coaching, teaching or release of any test items, photocopying, recording, etc.) is a violation of test security. Security Statement

To ensure that assessment results are accurate and meaningful To truly reflect a valid and reliable measure of each student’s knowledge To provide useful data to guide school improvement To protect the enormous investments of resources, time, and energy that go into assessments We have a responsibility to the public to ensure the accuracy of test scores. Disregarding test security can lead to investigations, invalidations, or in extreme circumstances, even jail! Why is Test Security Important?

Consequences of Violating Test Security An unfair advantage may be provided to some students, making it difficult to draw meaningful conclusions about their performance. This may result in: – Parents receive no scores on their child’s report if a test is invalidated – Public reporting will reflect all invalidated tests as “not tested.” This may reduce the percentage of students meeting proficiency and counts against a school’s participation rate. – Schools may have a more difficult time meeting the accountability requirements as a result.

An Infamous Test Security Case…

The Atlanta Cheating Scandal District-wide falsification of answer sheets by teachers and administrators Discovered after newspaper reported unusual jump in test scores on the Criterion-Referenced Competency Tests (CRCT) Investigation uncovered rampant, widespread cheating 30+ Atlanta school employees charged under RICO (Racketeer Influenced and Corrupt Organizations) statute for conspiring in order to receive bonuses tied to high test scores

Test Irregularities (security breaches) may occur before, during, or after testing. Testing irregularities may compromise the reliability and validity of the test, and could affect a school’s accountability ratings All testing irregularities, including test security breaches, must be reported to the Building Coordinator immediately Building Coordinator should initiate the completion of the Irregularity Report Form and notify the District Test Coordinator All testing irregularities must be reported Testing Irregularities

Common Test Security Issues Before, During, and AfterPaper/PencilComputer-Based Testing Lost or stolen test bookletsX Obtaining unauthorized access to secure test materials XX Educators logging in to view test materials or change responses X Hacking in to computersX BeforeP/PCBT Educators, parents, or students engaging others to take a test on a student’s behalf XX

Common Test Security Issues Cont’d DuringP/PCBT Students giving or receiving unauthorized assistance from other students during test administration XX Teachers providing answers to the studentsXX Students accessing non-allowable resourcesXX Taking photos of test items and sharing on social media XX Accommodations being used inappropriately to cheat XX Keystroke loggingX

Common Test Security Issues Cont’d AfterP/PCBT Erasing wrong answers, or changing responsesXX Reconstructing assessment materials through memorization XX Posting memorized test items or answers onlineXX Printing, ing, or storing test information in a computer outside the test delivery system XX Accessing or altering assessment materials or scores during the transfer of data XX

Social Media and Test Security Pictures of, and/or discussion about test items on social media steadily on the rise No age barrier Not limited to just students

Questions?

Detecting Possible Issues WDE conducting announced visits to review administration and test security procedures in schools Monitoring social media sites for test security breaches, e.g., photos of test items, photos of classrooms testing, discussions of specific live test items

During the PAWS administration WDE staff will be conducting site visits to school districts. We will be using the Quality Assurance Checklist for Secure Testing (PAWS BCM, pg. 37). We will be looking for secure test storage, testing schedules, test materials tracking information, Test Administrator training, proper movement of materials, signed Test Security Agreements. If you do not have a materials tracking process in place, WDE has provided a form to assist (PAWS BCM, pg. 36). School Observations

There are test security agreements that must be signed for by all Building Coordinators and Test Administrators. Test security agreements must be signed annually Test Security Agreements can be found PAWS in the BC manual Wy-ALT in the BC manual ACT during establishment BCs agree to be the responsible party Test Security Agreements

WDE is not: The testing police Searching for or asking users to adjust opinions about testing or the assessments Invading privacy on social media accounts WDE is: Working to enhance understanding of appropriate test preparation and administration practices Proactively addressing potential security breaches on public-facing websites Sharing best practices around the state Dispelling Myths

Unusual Scores Gains (for PAWS extreme group/cohort gains) Unusual gains occur when an unusually large number of high gain scores are present within a classroom or school A high gain score is measured when a student’s test score is substantially higher than predicted based upon prior achievement using an appropriate statistical model. Unusual gains are very unlikely but may indicate a testing irregularity has occurred, such as inappropriate coaching or inappropriate answer changing. A Method of Detection

Erasure Analysis (For PAWS) WDE initiated reviews of erasure data in all schools statewide in the summer of 2013 Where very high erasure rates are associated with very high year-to-year test score gains, WDE will contact the district Superintendent for follow up on internal procedures for test preparation and administration Another Method of Detection

Following a reported incident or complaint, WDE may ask the district to investigate the issue and submit a report and signed statements, along with a detailed chain of events and any steps taken to remediate the issue. How WDE Responds to TS Issues

After reviewing all information and evidence, WDE makes one of the following determinations: There was no irregularity A self-correction was completed – WDE determines that the school properly resolved the issue The irregularity led to a breach in test security but did not affect validity of test results The irregularity resulted in an invalid test administration Determinations

Provide training for staff on appropriate test security, test preparation, test administration procedures, and accommodations Supervise assessment materials closely – they must be secure at all times Follow standardized test administration procedures and directions explicitly, according to manuals Closely monitor students and enforce a strict “no cell phone” policy Document everything – chains of custody of materials, seating charts, etc. Easy Ways to Avoid Breaches in TS

WDE may invalidate any test if: There is a breach of test security The test is administered in a manner inconsistent with the Test Administration Manual Invalidation

Standard protocol dictates that all secure materials be returned to the vendor. In instances of contamination of test materials, ETS prefers that those PAWS materials not be returned, or materials containing PII (labels), please ensure that those items are securely destroyed. The best method for the destruction of secure materials is secure shredding. Destruction of Secure Materials

WDE is currently partnering with the Wyoming Attorney General’s Office and the Professional Teaching Standards Board to develop formal administrative rules around test security to provide clear guidance for best testing practices. We will also be establishing consequences for test security violations. Test Security Rules

All test items and materials are secure and must be appropriately handled. Secure handling protects the integrity, validity, and confidentiality of assessment items, prompts, and student information. Any deviation in test administration must be reported as a test irregularity to ensure the validity of the assessment results. Failure to honor security jeopardizes student information and puts the operational test at risk. Conclusion

QUESTIONS?

Contact us! Deb Lindsey – Jessica Steinbrenner – Michelle Carroll – Emily Brantz – Or give us a call: