1. Narrowing the Gulf Annual Conference 2010
Mastering the Art of
Writing Objective Test Items
Narrowing the Gulf
Annual Conference 2010
March 2010

2. Writing Objective Test Items
Presenter
Dr. James Coraggio, Director, Academic Effectiveness and Assessment
Former Life…
Director of Test Development , SMT
Director of Measurement and Test Development, Pearson
Taught EDF 4430 Measurement for Teachers, USF
March
Academic Effectiveness and Assessment

3. Academic Effectiveness and Assessment
Purpose
Are your students learning the course content or are they just good test takers?
This presentation will explain how to create effective multiple choice test questions.
The presentation will provide item-writing guidelines as well as best practices to prevent students from just guessing the correct answers.
March
Academic Effectiveness and Assessment

4. Academic Effectiveness and Assessment
Agenda
Propose of a Test
Advantages of Objective Tests
Types of Objective tests
Writing Multiple Choice Items
The Test-wise Student
Test Instructions
Test Validity
March
Academic Effectiveness and Assessment

5. Academic Effectiveness and Assessment
Purpose of a Test
‘Clearly delineate between those that know the content and those that do not.’
Purpose of an assessment is to determine whether the student knows the content, not whether the student is a good test-taker.
Likewise, confusing and tricky questions should be avoided to prevent incorrect responses from students who know (and understand) the material.
March
Academic Effectiveness and Assessment

6. Academic Effectiveness and Assessment
Objective Tests
Measure several types of learning (also levels)
Wide content, short period of time
Variations for flexibility
Easy to administer, score, and analyze
Scored more reliability and quickly
What type of learning cannot be measured?
March
Academic Effectiveness and Assessment

7. Types of Objective Tests
Written-response
Completion (fill-in-the-blank)
Short answer
Selected-response
Alternative response (two options)
Matching
Keyed (like matching)
Multiple choice
March
Academic Effectiveness and Assessment

8. Academic Effectiveness and Assessment
Written-response
Single questions/statements or clusters (stimuli)
Advantages
Measure several types of learning
Minimizes guessing
Points out student misconceptions
Disadvantages
Time to score
Misspelling and writing clarity
Incomplete answers
More than one possible correct response (novel answers)
Subjectivity in grading
March
Academic Effectiveness and Assessment

9. Academic Effectiveness and Assessment
Completion
A word that describes a person, place or thing is a ________.
Remove only ‘key’ words
Blanks at end of statement
Avoid multiple correct answers
Eliminate clues
Paraphrase statements
Use answer sheets to simplify scoring
March
Academic Effectiveness and Assessment

10. Academic Effectiveness and Assessment
Short Answer
Briefly describe the term proper noun. ____________________________
Terminology – Stimulus and Response
Provide an appropriate blank (word (s) or sentence).
Specify the units (inches, dollars)
Ensure directions for clusters of items and appropriate for all items
March
Academic Effectiveness and Assessment

11. Academic Effectiveness and Assessment
Selected-response
Select from provided responses
Advantages
Measure several types of learning
Measures ability to make fine distinctions
Administered quickly
Cover wide range of material
Reliably scored
Multiple scoring options (hand, computer, scanner)
Disadvantages
Allows guessing
Distractors can be difficult to create
Student misconceptions not revealed
March
Academic Effectiveness and Assessment

12. Academic Effectiveness and Assessment
Alternative Response
T F 1. A noun is a person place or thing.
T F 2. An adverb describes a noun.
Explain judgments to be made
Ensure answers choices match
Explain how to answer
Only one idea to be judged
Positive wording
Avoid trickiness, clues, qualifiers
March
Academic Effectiveness and Assessment

13. Academic Effectiveness and Assessment
Matching Item
Column A Column B
__Person, place, or thing a. Adjective
__Describes a person, place, or thing b. Noun
Terminology – premises and responses
Clear instructions
Homogenous premises
Homogenous responses (brief and ordered)
Avoid one-to-one
March
Academic Effectiveness and Assessment

14. Academic Effectiveness and Assessment
Keyed Response
Responses
a. A noun
b. A pronoun
c. An adjective
d. An adverb
___Person, place, or thing.
___Describes a person, place, or thing.
Like matching items, more response options
March
Academic Effectiveness and Assessment

15. Academic Effectiveness and Assessment
MC Item Format
What is the part of speech that is used to name a person, place, or thing?
A) A noun*
B) A pronoun
C) An adjective
D) An adverb
March
Academic Effectiveness and Assessment

16. Academic Effectiveness and Assessment
MC Item Terminology
Stem: Sets the stage for the item; question or incomplete thought; should contain all the needed information to select the correct response.
Options: Possible responses consisting of one and only one correct answer.
Key: correct response
Distractor: wrong response, plausible, but not correct, attractive to an under-prepared student
March
Academic Effectiveness and Assessment

17. Academic Effectiveness and Assessment
Competency
Items should test for the appropriate or adequate level of knowledge, skill, or ability (KSA) for the students.
Assessing lower division students on graduate level material is an ‘unfair’ expectation.
The competent student should do well on an assessment, items should not be written for only the top students in the class.
March
Academic Effectiveness and Assessment

18. Academic Effectiveness and Assessment
Clarity
Clear, precise item and instruction
Correct grammar, punctuation, spelling
Address one single issue
Avoid extraneous material (teaching)
One correct or clearly best answer
Legible copies of exam
March
Academic Effectiveness and Assessment

19. Academic Effectiveness and Assessment
Bias
Tests should be free from bias…
No stereotyping
No gender bias
No racial bias
No cultural bias
No religious bias
No political bias
March
Academic Effectiveness and Assessment

20. Academic Effectiveness and Assessment
Level of Difficulty
Ideally, test difficulty should be aimed a middle level of difficulty. This can not always be achieved when the subject matter is based on specific expectations (i.e, workforce area).
March
Academic Effectiveness and Assessment

21. Academic Effectiveness and Assessment
Level of Difficulty
To make a M/C item more difficult, make the stem more specific or narrow and the options more similar.
To make a M/C item less difficult, make the stem more general and the options more varied.
March
Academic Effectiveness and Assessment

22. Trivial and Trick Questions
Avoid trivia and tricks
Avoid humorous or ludicrous responses
Items should be straight forward, they should cleanly delineate those that know the material from those that do not
Make sure every item has value and that it is contributing to the final score
March
Academic Effectiveness and Assessment

23. Test Taking Guidelines
Test Taking Guidelines
When you don’t know the answer
As with all exams, attempt the questions that are easiest for you first. Come back and do the hard ones later. Unless you will lose marks for an incorrect response, never leave a question blank. Make a calculated guess if you are sure you don’t know the answer. Here are some tips to help you guess ‘intelligently’.
Use a process of elimination
Try to narrow your choice as much as possible: which of the options is most likely to be incorrect? Ask: are options in the right range? Is the measurement unit correct? Does it sound reasonable?
March
Academic Effectiveness and Assessment

24. Test Taking Guidelines
Test Taking Guidelines
Look for grammatical inconsistencies
In extension-type questions a choice is nearly always wrong if the question and the answer do not combine to make a grammatically correct sentence. Also look for repetition of key words from the question in the responses. If words are repeated, the option is worth considering. e.g.:
The apparent distance hypothesis explains…
b) The distance between the two parallel lines appears…
March
Academic Effectiveness and Assessment

25. Test Taking Guidelines
Test Taking Guidelines
Be wary of options containing definitive words and generalizations
Because they can’t tolerate exceptions, options containing words like ‘always’, ‘only’, ‘never’, ‘must’ tend to be incorrect more often. Similarly, options containing strong generalizations tend to be incorrect more often.
Favor look-alike options
If two of the alternatives are similar, give them your consideration. e.g.:
A. tourism consultants
B. tourists
C. tourism promoters
D. fairy penguins
March
Academic Effectiveness and Assessment

26. Test Taking Guidelines
Test Taking Guidelines
Favor numbers in the mid-range
If you have no idea what the real answer is, avoid extremes.
Favor more inclusive options
If in doubt, select the option that encompasses others. e.g.:
A. an adaptive system
B. a closed system
C. an open system
D. a controlled and responsive system
E. an open and adaptive system.
Please note: None of these strategies is foolproof and they do not apply equally to the different types of multiple choice questions, but they are worth considering when you would otherwise leave a blank.
March
Academic Effectiveness and Assessment

27. Academic Effectiveness and Assessment
Test-wise Students
Are familiar with item formats
Use informed and educated guessing
Avoid common mistakes
Have testing experience
Use time effectively
Apply various strategies to solve different problem types
March
Academic Effectiveness and Assessment

28. Academic Effectiveness and Assessment
Test-wise Students
Vary your keys: ‘Always pick option ‘C’ ’
Avoid ‘all of the above’ and ‘none of the above’
Avoid extraneous information: It may assist in answering another item
Avoid item ‘bad pairs’ or ‘enemies’
Avoid clueing with the same word in the stem and the key
March
Academic Effectiveness and Assessment

29. Academic Effectiveness and Assessment
Test-wise Students
Make options similar in terms of length, grammar, and sentence structure. Different options stand out. Avoid ‘clues’.
March
Academic Effectiveness and Assessment

30. Item Format Considerations
Information in the stem
Avoid negatively stated stem, qualifiers
Highlight qualifiers if used
Avoid irrelevant symbols (“&”) and jargon
Standard set number of options (Prefer only four)
Ideally, you should tie an item to reference
March
Academic Effectiveness and Assessment

31. Academic Effectiveness and Assessment
Test Directions
Highlight Directions
State the skill measured.
Describe any resource materials required.
Describe how students are to respond.
Describe any special conditions.
Time limits, if any
March
Academic Effectiveness and Assessment

32. Academic Effectiveness and Assessment
Ensure Test Validity
Congruence between items and course objectives
Congruence between item and student characteristics
Clarity of items
Accuracy of the measures
Item formatting criteria
Feasibility-time, resources
March
Academic Effectiveness and Assessment

33. Academic Effectiveness and Assessment
Questions
March
Academic Effectiveness and Assessment

34. Narrowing the Gulf Annual Conference 2010
Mastering the Art of
Writing Objective Test Items
Narrowing the Gulf
Annual Conference 2010
March 2010