1. Local School Audits Virginia Association of Test Directors Fall 2009 Conference Fairfax County Public Schools

2. Fairfax County Public Schools Office of Student Testing Presenters Lenore Callahan Manager: Test Analysis & Administration Jeremy Shughart Test Administration Specialist

3. Local School Audits The Fairfax County Public Schools (FCPS) Office of Student Testing staff members created the process for local school audits to assist schools with test preparation and for possible VDOE audits. The audits were never intended as a gotcha.

4. Goals To build and test our own model for audits To determine the best approach to use for the auditsinterviews, observations, examiners responses, or documentation To implement that model To develop local capacity to effectively maintain, update, enhance, and use the resulting system To improve trainings to meet school needs

5. Background Input from: VDOE audit packet provided with each test administration The test security agreement Documentation related to test security Feedback from focus groups

6. Standards Standards Verifiable by Documentation Standards Verifiable by Observation Standards Verifiable by Statements Examiners Responses

7. Standards Verifiable by Documentation 28 questions Test security agreement Testing materials to/from schools Inventory process Transmittal documents Signature log in front office Teacher/student test preparation packets Emergency response plans

8. Standards Verifiable by Observation 26 questions Test materials storage area Check in/out process Curriculum cleared from walls Electronic device policy Examiner/proctor –kept test materials secure –stayed with script –monitored room –collected materials at end of test

9. Standards Verifiable by Statements 17 interview questions Interviews conducted with office administrative assistants, custodians, test coordinators, teachers, principals, etc. Topics: –security of materials –training –electronic device policy –accommodations –possible testing irregularities

10. Examiners Responses 32 questions Anonymous Self-survey Yes/No/NA Completed by teachers who tested on day of audit Topics reflect information from other standards such as training and test administration

11. Training the Auditors Discuss and sign the test security agreement Discuss the look-fors curriculum on walls, electronic devices, monitors, etc. Discuss procedures for anomalies observed during testing Review questions from the Standards Verifiable by Observation packet Provide Examiners Manual and discuss use

12. The Audit Arrived at school one hour prior testing Met at hub or location of test material pickup/return Followed an examiner back to site from check out of materials to return Completed Standards Verifiable by Observations Returned all documents to lead auditor

13. After the Audit Compiled results from all sections into one booklet. Copies provided to schools (2), school cluster assistant superintendent, and DDOT. Cover letter included positives and any areas to review and improve upon prior to next test administration.

14. What We Learned Not easy and very time consuming! Schools apprehensive about what we will find. Varied by test and by grade level. –Not invited back to discuss with high school staff. –Elementary staff most appreciative and wanted to meet right away to discuss.

15. Limitations By invite only Limited staff Could make no recommendations or suggestions for improvements Not invited back to discuss

16. Feedback from Schools Good to use as a list to prepare. Reminded me of things that I had not thought about. Used with examiners as part of their training. Gave credibility to what I have been telling staff all along!

17. Next Steps Continue with audits. Streamline the process to possibly use one section per school to include more schools. Use also for non-SOL assessments. Potential desk audit.

18. Fairfax County Public Schools Office of Student Testing Presenters Lenore Callahan, Manager: Test Analysis & Administration, 703-208-7784, lfcallahan@fcps.edu lfcallahan@fcps.edu Jeremy Shughart, Test Administration Specialist, 703-208-7780, jshughart@fcps.edu jshughart@fcps.edu