1. Stayin’ Alive Surviving the Dreams and Realities of Assessing Full-time, Web-based Students’ Learning Copyright [Tracy A. Chapman, M.Ed.] [205]. This work is the intellectual property of the author. Permission is granted for this material to be shared for non-commercial, educational purposes, provided that this copyright statement appears on the reproduced materials and notice is given that the copying is by permission of the author. To disseminate otherwise or to republish requires written permission from the author.

2. Tracy A. Chapman, M.Ed. Executive Director, eLearning and Technology School of Pharmacy and Health Professions Creighton University Medical Center tchapman@creighton.edu

3. Objectives Describe a 7-step process for developing models of delivering online assessments. Match online testing tools to exam question types. Describe several frameworks for delivering faculty and student training in the use of online testing tools. Describe strategies for ensuring exam integrity. Describe a model for formative and summative evaluation of the online assessment process.

4. Entry Level Web-based Pharmacy Pathway Shortage of pharmacists $1,000,000 grant from Institute for the Advancement of Community Pharmacy Nation’s first web-based entry-level doctor of pharmacy pathway development First entering class August 2001 (57 students) Graduated May 2005 (47 students with 6 more to graduate within the next year)

5. Preparation Accrediting body Intellectual property Faculty development – technology, pedagogy of teaching online Back end support – servers, network Student support – technology, services Course development Distance education policy Assessing student learning

7. Course objectives Assessment strategies Exams Question types Other test construction features required Security required Select tool Development & delivery Formative and Summative Evaluation Projects Papers CATs Standardized Patient

8. Question Types QMSecure Exam Bb M/Cxxx Essayxxx Short answerxxx Multiple response xxx T/Fxxx Orderingxxx Matchingxxx Drawingx Numericalxxx Click and Dragx Text matchxx Likert Scalexx

9. Other Features QMSecure Exam Bb Adaptive construction x Question banks xx Scramble Q&A xx Feedback options xx Control # attempts xxx Timed/start and stop times xxx Self grading xx

10. Exam Security Use of Proctors –Selection –Policy –Management –When suspicions arise Secure Browser –Access to resources –Caching –Printing Use of Passwords

11. Exam Security [cont.] Exam Windows –Generally 24 hours –1 course, 6 hour window Move to testing centers –Based on desire to increase assurance to external stakeholders of program's academic rigor –30 mile radius from student’s home –Centers identified by web-based pathway administrative assistant Academic Misconduct (Exams) –Fewer than on campus

12. Exam Security and Tools Available QMSecure Exam Bb Proctor xxx Secure browser xxx Timed xx No caching xx No printing xxx Password xxx

13. Course objectives Assessment strategies Exams Question types Other test construction features required Security required Select tool Development & delivery Formative and Summative Evaluation Projects Papers CATs Standardized Patient

14. Development and Delivery Faculty development Exam development Support – faculty and student Hardware/software for delivery Reliability

15. Faculty Development Online delivery of exams –Security processes –Work flow to get exam online Software used to create exams Venues –Seminars –Individual tutorials

16. Support – Faculty and Student Pharmacy department assistant –50% time spent transferring exams from Word to QM Online testing manager –Dedicated to managing QM exams –~145 exams and quizzes per semester –Review each for accuracy and view-ability –Post and assign student groups –Assist with exam related help desk calls and emails Instructional Designers assist with Secure Exam and Bb Student help desk –Monday-Thursday, 8:00-6:00 CT, Fridays 8:00 4:30 CT –800 testing ‘Hotline’

17. Hardware/software/reliability Hardware –2 servers dedicated to QM –SQL –University manages Blackboard Software –QM annual licenses –Software Secure annual licenses –Bb annual license (University IT) Reliability –First 2 years - UGLY… VERY UGLY! –This year - implement load balancing –Bb – discovered known issue that affects lengthy exams

18. Course objectives Assessment strategies Exams Question types Other test construction features required Security required Select tool Development & delivery Formative and Summative Evaluation Projects Papers CATs Standardized Patient

19. Evaluation of Online Assessment Process Data sources –End of course student feedback –Focus groups during summer sessions –Anecdotal information from email, discussions, chats –Student participation in committees –Graduate exit surveys

20. Formative Evaluation

21. Summative Evaluation

22. What We Have Learned Student training is a must! Staff support for faculty is crucial –Assist with exam development –Keep students informed and on track –Keep proctors informed/send materials to proctors –Provide help desk support –Immediate response to students experiencing technical problems during an exam Staff must be able to communicate calmness and competence when students panic Manage assigning and posting of exams must be centralized

23. So…. Are the web based students learning? On average, the web-based and campus-based exam scores are very similar Assessment of web-based students clinical experiences – scores, on average, same as campus-based students North American Pharmacist Licensure Examination (NAPLEX) pre-test results, web- based students’ average score slightly higher that campus-based students NAPLEX results - web students scores statistically equal to those of campus students

24. Acknowledgements Ms. DeDe Hedlund – Online Testing Manager, School of Pharmacy and Health Professions Dr. Patrick Malone – Director, Entry-level Web- based Pharmacy Pathway Ms. Andrea Winters – Administrative Assistant, Entry-level Web-based Pharmacy Pathway Dr. Victoria Roche – Senior Associate Dean Dr. Naser Alsharif – Associate Professor Mr. Arvind Jagannathan – Web Developer

25. QUESTIONS? THANK YOU!