1. WDE Test Security Training: Statewide Assessments February 2017
I have put everyone on mute. If you would like to ask a question please do so in the chat box. There are question breaks in the presentation, and we will allow time for questions at the end.

2. Session Overview Understanding the importance of test security
Common test security issues
How WDE detects possible test security issues
Possible consequences of test security issues
Questions
The goal is to protect the integrity of the assessments by securing the test materials at all times, and ensuring appropriate administration of the test

3. Security Statement
Statewide assessments rely on the accurate measurement of individual achievement. Any deviation from procedures meant to ensure test validity and security (e.g., group work, teacher coaching, teaching or release of any secure test items, photocopying, recording, etc.) is a violation of test security.

4. Why is Test Security Important?
To ensure that assessment results are accurate and meaningful
To truly reflect a valid and reliable measure of each student’s knowledge
To provide useful data to guide school improvement
To protect the enormous investments of resources, time, and energy that go into assessments
We have a responsibility to the public to ensure the accuracy of test scores.
Disregarding test security can lead to investigations, invalidations, or in extreme circumstances, even jail!
Anytime we assess students we do so to make inferences about student achievement, it protocols are not followed, we may not be able to make those inferences.

5. Consequences of Violating Test Security
An unfair advantage may be provided to some students, making it difficult to draw meaningful conclusions about their performance.
This may result in:
Parents receive no scores on their child’s report if a test is invalidated
Public reporting will reflect all invalidated tests as “not tested.” This may reduce the percentage of students meeting proficiency and counts against a school’s participation rate.
Schools may have a more difficult time meeting the accountability requirements as a result.
When the security of an item is breached, this may result in making the item unusable on future tests. The state then incurs additional costs associated with item development, which is already a very expensive process.

6. An Infamous Test Security Case…

7. The Atlanta Cheating Scandal
District-wide falsification of answer sheets by teachers and administrators
Discovered after newspaper reported unusual jump in test scores on the Criterion-Referenced Competency Tests (CRCT)
Investigation uncovered rampant, widespread cheating
30+ Atlanta school employees charged under RICO (Racketeer Influenced and Corrupt Organizations) statute for conspiring in order to receive bonuses tied to high test scores

8. Testing Irregularities
Test Irregularities (security breaches) may occur before, during, or after testing.
Testing irregularities may compromise the reliability and validity of the test, and could affect a school’s accountability ratings
All testing irregularities, including test security breaches, must be reported to the Building Coordinator immediately
Building Coordinator should initiate the completion of the Irregularity Report Form and notify the District Test Coordinator
All testing irregularities must be reported
PAWS on the Test Irregularity Form
Test Security breaches are a form of irregularity, but irregularities are not necessarily security breach
District Coordinators should contact the vendor helplines in the event of test irregularities.
We have a respon

9. Common Test Security Issues
Before, During, and After
Paper/Pencil
Computer-Based Testing
Lost or stolen test booklets X
Obtaining unauthorized access to secure test materials
Educators logging in to view test materials or change responses
Hacking in to computers
Before
P/P
CBT
Educators, parents, or students engaging others to take a test on a student’s behalf X

10. Common Test Security Issues Cont’d
During
P/P
CBT
Students giving or receiving unauthorized assistance from other students during test administration X
Teachers providing answers to the students
Students accessing non-allowable resources
Taking photos of test items and sharing on social media
Accommodations being used inappropriately to cheat
Keystroke logging

11. Common Test Security Issues Cont’d
After
P/P
CBT
Erasing wrong answers, or changing responses X
Reconstructing assessment materials through memorization
Posting memorized test items or answers online
Printing, ing, or storing test information in a computer outside the test delivery system
Accessing or altering assessment materials or scores during the transfer of data

12. Social Media and Test Security
Pictures of, and/or discussion about test items on social media steadily on the rise
No age barrier
Not limited to just students
Twitter wins the “Social Media Incident Award” for 2015 – 72% of Michigan’s social media incidents  Instagram the “runner up” – 12% of incidents  Facebook and Snapchat get “honorable mention
In Wyoming, we have seen test items appear on Facebook.

13. Device Awareness
Does your school have a cell phone policy?
How do you make sure cell phones are not used during testing?
Do your students have apple watches? Do your teachers?

14. Questions?

15. Detecting Possible Issues
WDE conducting announced visits to review administration and test security procedures in schools
Monitoring social media sites for test security breaches, e.g., photos of test items or booklets, photos of classrooms testing, discussions of live test items
Visits will be contacted on a rotation, in all districts across the state.

16. School Observations
During the test administrations WDE staff will be conducting site visits to school districts.
We will be using the Quality Assurance Checklist for Secure Testing (PAWS BCM, pg. 37 and WDE website).
We will be looking for secure test storage, testing schedules, test materials tracking information, Test Administrator training, proper movement of materials, signed Test Security Agreements.
If you do not have a materials tracking process in place, WDE has provided a form to assist (PAWS BCM, pg. 36).
This is not an audit.
Test Security Agreements should be stored for 2 years. It is ok to scan them and store them digitally as long as they are accessible.
Also allows us to view the testing environment, and to get information on how improvements can be made in assessment protocols and trainings.
For Act it’s in the establishment documents.
For Wy-Alt it’s on TIDE

17. Test Security Agreements
There are test security agreements that must be signed for by all Building Coordinators and Test Administrators.
Test security agreements must be signed annually
Test Security Agreements can be found
PAWS in the BC manual
Wy-ALT in the BC manual
ACT during establishment BCs agree to be the responsible party
WDE website (‘2017 Administration Materials’ page)
Test Security Agreements should be stored for 2 years. It is ok to scan them and store them digitally as long as they are accessible.
All persons handling secure test materials must be trained and have signed a Test Security Agreement

18. Test Security Agreements must be signed by all staff members who handle secure materials
Signed ‘Test Security’ agreements must be kept for 2 years
They may be stored hard-copy or digitally.

19. Dispelling Myths WDE is not: The testing police
Searching for or asking users to adjust opinions about testing or the assessments
Invading privacy on social media accounts
WDE is:
Working to enhance understanding of appropriate test preparation and administration practices
Proactively addressing potential security breaches on public-facing websites
Sharing best practices around the state
This is our way of avoiding issues that have occurred in other areas across the country. We want to model best practices while being cognizant of Wyoming specific scenarios.

20. A Method of Detection
Unusual Scores Gains (for PAWS extreme group/cohort gains)
Unusual gains occur when an unusually large number of high gain scores are present within a classroom or school
A high gain score is measured when a student’s test score is substantially higher than predicted based upon prior achievement using an appropriate statistical model.
Unusual gains are very unlikely but may indicate a testing irregularity has occurred, such as inappropriate coaching or inappropriate answer changing.
Alternative explanations of unusual gains must always be considered and include laudable education efforts such as excellent teaching and improved instructional resources or access.

21. Another Method of Detection
Erasure and High Gains Analyses (For PAWS)
WDE initiated reviews of erasure data in all schools statewide in the summer of 2013
Where very high erasure rates are associated with very high year-to-year test score gains, WDE will contact the district Superintendent for follow up on internal procedures for test preparation and administration

22. How WDE Responds to TS Issues
Following a reported incident or complaint, WDE may ask the district to investigate the issue and submit a report and signed statements, along with a detailed chain of events and any steps taken to remediate the issue.
We will also provide information on how to avoid future security breaches, and hopefully stay out of the newspapers. This is a pro-active stance rather than reactive. We don’t like scandals even though we live in the capitol city.

23. Determinations
After reviewing all information and evidence, WDE makes one of the following determinations:
There was no irregularity
A self-correction was completed – WDE determines that the school properly resolved the issue
The irregularity led to a breach in test security but did not affect validity of test results
The irregularity resulted in an invalid test administration
All of this information needs to be documented to verify that we have done our due diligence and that appropriate measures were taken to resolve any concerns.

24. Easy Ways to Avoid Breaches in TS
Provide training for staff on appropriate test security, test preparation, test administration procedures, and accommodations
Supervise assessment materials closely – they must be secure at all times
Follow standardized test administration procedures and directions explicitly, according to manuals
Closely monitor students and enforce a strict “no cell phone” policy
Document everything – chains of custody of materials, seating charts, etc.
If you see something, say something!
There is a powerpoint for Test Administrator training on the WDE website that provide a framework for PAWS test security training. Building Coordinators are responsible for training all Tas for PAWS. ACT also has trainings available for administrators for their assessments. Wy-ALT has trainings available as well on their TIDE platform.

25. Invalidation
WDE may invalidate any test if: • There is a breach of test security • The test is administered in a manner inconsistent with the Test Administration Manual
ACT, AIR and ETS may also invalidate scores if their tests are compromised.
ACT and Wy-ALT are not custom Wyoming tests and are used more broadly. Where it’s costly for a single test item to be invalidated in Wyoming, it is exponentially costly for an item to be invalidated for multiple states. That is a lot of test forms!

26. Destruction of Secure Materials
Standard protocol dictates that all secure materials be returned to the vendor.
In instances of contamination of test materials, ETS prefers that those PAWS materials not be returned, or materials containing PII (labels), please ensure that those items are securely destroyed. The best method for the destruction of secure materials is secure shredding.
AIR requires all secure Wy-ALT materials returned, regardless of condition.
ACT requires all materials to be returned, regardless of condition.

27. Test Security Rules
WDE is currently partnering with the Wyoming Attorney General’s Office and the Professional Teaching Standards Board to develop formal administrative rules around test security to provide clear guidance for best testing practices. We will also be establishing consequences for test security violations.

28. Conclusion
All test items and materials are secure and must be appropriately handled. Secure handling protects the integrity, validity, and confidentiality of assessment items, prompts, and student information. Any deviation in test administration must be reported as a test irregularity to ensure the validity of the assessment results. Failure to honor security jeopardizes student information and puts the operational test at risk.

29. QUESTIONS?

30. Contact us! Laurie Hernandez – Standards and Assessment Director
Jessica Steinbrenner – Assessment Supervisor
Michelle Carroll – Assessment Consultant
Catherine Palmer – Assessment Consultant
Or give us a call: