1. Jackie Stapleton Sandra Keys University of Waterloo January 30, 2008

2. Overview Part 1 CMS and the online quiz – review of the literature On-line quiz options – UW ACE / Angel Part 2 Features of good question design Hands on exercise 2OLA Super Conference 2008: Creating On-line Quizzes

3. Advantages/Disadvantages Advantages : Easy (relatively) to create and distribute in CMS environment Feedback – instant, automatic, consistent, Digital format preferred by new generation of students/active learners (Johnson, GM, 2006) Disadvantages: Lack of quiz security Surface understanding assessment only Technical problems Johnson, G. M., & Johnson, J. A. (2006). Learning style and preference for online learning support: Individual quizzes versus study groups. 18th Annual World Conference on Educational Multimedia, Hypermedia, and Telecommunications, Orlando, FL. 1861-1868. 3OLA Super Conference 2008: Creating On-line Quizzes

4. What does the literature say? The next series of slides will highlight some interesting case studies and outcomes of research studies in the library and education literature. 4OLA Super Conference 2008: Creating On-line Quizzes

5. Assess beyond surface learning Dr. Diana Quinn and Ian Reid, University of South Australia Quizzes can be developed in ways that they play less of an assessment role, and more of a teaching role. Provide an outline of several quiz designs which claim to: Support personal construction of understanding Simulate group feedback Promote self assessment Quinn, D and Reid, I. (2003). Using innovative online quizzes to assist learning, Accessed on October 21, 2007 at http://ausweb.scu.edu.au/aw03/papers/quinn/paper.html.. 5OLA Super Conference 2008: Creating On-line Quizzes

6. CMS and other quiz software: RH Locklin and PR Howell, Pennsylvania State University Used Angel for practice and pre test quizzes, Questionmark for end of unit quizzes Open book vs closed book quizzes (different delivery options) Student satisfaction survey revealed: 80% of students preferred open book quizzes over closed book exams Majority believed that they would obtain a higher grade with an open book format (this did not prove to be true) 90% of students claimed to read the feedback Locklin, R. H., & Howell, P. R. (2005). The design, implementation and assessment of an on- line, open-book quizzing environment for an introductory materials science course. Materials Research Society Symposium Proceedings, 909, 23-41. 6OLA Super Conference 2008: Creating On-line Quizzes

7. Innovative training opportunities Kathryn Skhal and Catherine Thureson, University of Iowa Library Used CMS to create an online training program for library science students who staff the reference desk. Included article discussion forum, online quizzes, challenging questions module Outcomes: Social community for reference students who often work in isolation Students posted many more questions to the challenging question forum. Librarians have become much more efficient in the use of the CMS and able to promote their skills to faculty Skhal, K. J., & Thureson, C. (2007). ICONic training: Use of a course management system to provide continual reference student education. Medical reference services quarterly, 26(2), 15-25. 7OLA Super Conference 2008: Creating On-line Quizzes

8. Not cheating – group collaboration! Amanda Fales-Williams et al, Iowa State University College of Veterinary Medicine Online pre exam quiz in WebCT for veterinary anatomy course Questions were designed to be challenging, did not expect student to be able to answer the question on their first attempt Instructor noted an increased rate of identical answers Not cheating, but group collaboration Fales-Williams, A., Kramer, T., Heer, R., & Danielson, J. (2005). A quiz becomes a multidirectional dialogue with web-based instructional tools for an anatomical pathology rotation. Journal of veterinary medical education, 32(1), 144-149. 8OLA Super Conference 2008: Creating On-line Quizzes

9. Creating a quiz in CMS I consider it a three step process: 1. Think about the learning objectives of your instructional material and then the purpose of your quiz. Assessment of learned material? Learning/teaching opportunity? 2. Create the questions Many different question types. 3. Decide on delivery options 9OLA Super Conference 2008: Creating On-line Quizzes

10. Question types Fill in the blank, multiple fill in the blank True/False Drop down list Ordering Matching Multiple choice, multiple select 10OLA Super Conference 2008: Creating On-line Quizzes

11. Other quiz design features Section heading Images, links, other html features can appear in a question stem or answer 11OLA Super Conference 2008: Creating On-line Quizzes

12. Feedback Advantages: instant, automatic General feedback for an entire question Specific feedback for each individual answer option 12OLA Super Conference 2008: Creating On-line Quizzes

13. Delivery options: Creating the questions is only the first step. There are many other decisions to make before you can implement the quiz. Access – Who? When? Question delivery Submission options Quiz security Marking Does your CMS offer different features? 13OLA Super Conference 2008: Creating On-line Quizzes

14. Accessibility considerations: Results of test study with JAWS reader: Technical issues: Multiple choice questions were the most successful, both in clarity and technical issues Drop down boxes were not readable Fill in the blank (edit boxes) caused problems moving to the next question Quiz Design issues: Place fill in the blank at end of question Stem should be a complete question. 14OLA Super Conference 2008: Creating On-line Quizzes

15. My experiences: What I have learned: Spend more time on question design View quiz more as a learning tool Expect technical problems and student issues Next steps: Study how quiz delivery affects various factors such as: How many students attempt to complete quiz Student achievement Student satisfaction 15OLA Super Conference 2008: Creating On-line Quizzes

16. Writing Questions No matter what assessment tool is chosen, the key is to write sound test items Nothing new; Long-standing tradition Plethora of information on writing good multiple choice questions But do we do it right? 16OLA Super Conference 2008: Creating On-line Quizzes

17. Types of Questions True/False Multiple Choice Fill in the Blanks General Advantages and Guidelines Type-Specific Advantages, Disadvantages, and Guidelines Examples 17OLA Super Conference 2008: Creating On-line Quizzes

18. General Advantages Versatility Scoring accuracy and economy Reliability Efficiency 18OLA Super Conference 2008: Creating On-line Quizzes

19. General Guidelines Test significant material and specialized knowledge, not trivial details Focus on a single idea in each question Be precise and correct in your language Rewrite or use examples to discourage rote memorization Avoid negatively worded statements and particularly double negatives Ignore any guideline if you have a good reason to do so! 19OLA Super Conference 2008: Creating On-line Quizzes

20. True/False Questions Advantages Only two reasonable answers Especially useful for testing misconceptions Expressible in a few words Disadvantages Subject to guessing Less discriminating Lead to verbatim memorization due to tendency to write trivial items 20OLA Super Conference 2008: Creating On-line Quizzes

21. True/False Questions Guidelines Make use of popular misconceptions or beliefs Provide sufficient information to judge the truth If expressing a relationship, give the correct part first and vary the second part Do not use qualifiers (always, never, usually, often, etc.) Keep items of approximately equal length throughout the test Use somewhat more false than true statements Randomize the sequence The Dana Porter Library is sinking e.g., According to... 21OLA Super Conference 2008: Creating On-line Quizzes It never rains in California

22. True/False Questions Problems Negative (not) Clarity (which article?) Vocabulary (reference list) Solutions Write as a positive Specify which article or other clarification Ensure that the vocabulary is consistent with the lesson Question: Krebs article does not have a reference list New question: Krebs article, discussed in Lesson 3, has a reference list 22OLA Super Conference 2008: Creating On-line Quizzes

23. Multiple Choice Questions Advantages Diagnosis Control of difficulty Reduction of guessing Freedom from response sets Disadvantages Difficult and time-consuming to write Subject to clueing Tendency to write items requiring factual knowledge rather than higher- level skills and understanding Performance can be influenced by unrelated factors 23OLA Super Conference 2008: Creating On-line Quizzes

24. Multiple Choice Questions General Guidelines Must have a definite answer Dont give answer away by including irrelevant clues Dont use language which may be offensive Have items reviewed by other knowledgeable people 24OLA Super Conference 2008: Creating On-line Quizzes

25. Multiple Choice Questions Two parts Item stem (the question) Item options (the responses) Item Stem Guidelines Single, concise statement Write as a question or an incomplete statement Most of the reading should be in the stem Include in the stem any words that would be repeated in each option 25OLA Super Conference 2008: Creating On-line Quizzes

26. Multiple Choice Questions Item Options Guidelines Keep the number of options consistent List options in logical order Ensure each option is mutually exclusive Keep length of options the same Keep options parallel in form and grammatically consistent with the stem All distracters should be plausible and equally attractive Use distracters which are correct but do not answer the question Use distracters obtained from short answer/completion responses 26OLA Super Conference 2008: Creating On-line Quizzes

27. Multiple Choice Questions Problems Negative (not) Vocabulary (synonomous) Too many options(7) Improbable option(c) all of the above; A and D Question: Which of these is not synonomous with Krebs Cycle? A.Tricarboxylic acid cycle B.Citric acid cycle C.Rinse cycle D.Trioxaloacetic acid cycle E.Citrate cycle F.All of the above G.A and D 27OLA Super Conference 2008: Creating On-line Quizzes

28. Multiple Choice Questions Solutions Write as a positive Change synonomous Eliminate extraneous distracters Include select all that apply statement New question: The Krebs cycle is a common name for which of the following terms? (select all that apply) A.Tricarboxylic acid cycle B.Citric acid cycle C.Trioxaloacetic acid cycle D.Citrate cycle 28OLA Super Conference 2008: Creating On-line Quizzes

29. Fill in the Blanks Advantages Difficult to guess answers Relatively easy to construct Useful for definitions Disadvantages Not as easy to score Guidelines Avoid long or complex sentences Omit only one or two key words at the end of the sentence 29OLA Super Conference 2008: Creating On-line Quizzes

30. Fill in the Blanks Problems WAY too many blanks! More than one idea The ____ cycle, also known as the ____ or the _____, is a series of _____ of central importance in all _____ that use _____ as part of _______. 30OLA Super Conference 2008: Creating On-line Quizzes

31. Fill in the Blanks Solution Rewrite question with just one or two blanks, preferably at the end Make it two questions to cover both the name and a main idea 1.The citric acid cycle is also known as the tricarboxylic acid or the _____ cycle. 2.The citric acid cycle is a series of enzyme-catalysed chemical reactions in all living cells that use _____ as part of cellular respiration. 31OLA Super Conference 2008: Creating On-line Quizzes

32. Hands on exercise Each table has been given a sample quiz question. Please review as a group and make any changes which you think will improve the question. Things to consider: Is there a better way to phrase the question? Is library jargon an issue? Does the question follow the suggested format for that question type? 32OLA Super Conference 2008: Creating On-line Quizzes

33. Sample quiz question #1: What activities are allowed in the library? Please check all that apply. a) Silent study b) Low conversation at group tables c) Birthday party with hats and whistles d) Eating your lunch from the cafeteria Answer: a and b Note: Is humour appropriate/inappropriate in this question/in any question? 33OLA Super Conference 2008: Creating On-line Quizzes

34. Sample quiz question #2 The boolean operator ______ should be used to combine two concepts in a search box. Answer: AND 34OLA Super Conference 2008: Creating On-line Quizzes

35. In conclusion Whether you choose online or in print, make your quiz a good one! Cyrus Wu, UW co op student, 2008 35OLA Super Conference 2008: Creating On-line Quizzes