1. Developing Tests and Test Questions/Items

2. Why do we evaluate students?
Motivate students
Measure achievement
Identify areas for review
Check effectiveness of materials, teaching methods, and teacher
Differentiate between different levels of achievement
Assign grades

3. Basic Principles of Testing
Validity
Does the test measure what it is supposed to measure?
Does the test require student performance as described by a corresponding objective?

4. Basic Principles of Testing
Reliability
Would the test generate the same results if it were given again (assuming no change in students)?
Is the test affected by misinterpretation?
Poorly written directions or instructions
Vague or tricky questions
Misleading words
Not sure of student response

5. Agenda Aligning evaluation with level of learning
Developing a test blueprint or “table of specifications”
Common pitfalls when writing test questions
Writing questions at various levels of performance

6. Defining objectives using Bloom’s Cognitive Taxonomy
CATEGORY/level
Evaluation
Synthesis
Analysis
Application
Comprehension
Knowledge
SKILL (what students do)
Judge
Combine
Break apart
Apply principles
Understanding
Recall

7. Sample Verbs for Different Categories/Levels of Cognition
Knowledge
Cite, define, identify, label, list, match, name, recognize, reproduce, select, state
Comprehension
Classify, convert, describe, distinguish between, explain, extend, give examples, illustrate, interpret, paraphrase, summarize, translate
Application
Apply, arrange, compute, construct, demonstrate, discover, modify, operate, predict, prepare, produce, relate, show, solve, use

8. Sample Verbs continued
Analysis
Analyze, associate, determine, diagram, differentiate, discriminate, distinguish, estimate, infer, order, outline, point out, separate, subdivide
Synthesis
Combine, compile, compose, construct, create, design, develop, devise, formulate, integrate, modify, organize, plan, propose, rearrange, reorganize, revise, rewrite, tell, write
Evaluation
Appraise, assess, compare, conclude, contrast, criticize, discriminate, evaluate, judge, justify, support, weigh

9. Agenda Developing a test blueprint or “table of specifications”
Aligning evaluation with level of learning
Developing a test blueprint or “table of specifications”
Common pitfalls when writing test questions
Writing questions at various levels of performance

10. Developing a Test Blueprint or a Table of Specifications
Develop a matrix (much like a rubric) of number of test questions (or percentage of test) by topic and level of objective, and by amount of emphasis on each topic and objective
Helps you plan your test
Do before writing items

11. Example Test Blueprint or Table of Specifications
Number of Objectives (Amount of Effort*}
Adjust
Level of Objective
Topic A
Topic B
Topic C
Topic D
Total
Knowledge 1 2
5 (20%)
Comprehension
6 (24%)
Application
Analysis
Synthesis*
2 (8%)
Evaluation*
1 (4%)
*Adjustment factor 6
(24%) 7
(28%) 25
(100%)
PAGE 10

12. Levels of performance
Lower levels (knowledge, comprehension, and application)
Recognize the principle in altered format.
Match component parts with vocabulary.
Identify example seen before.
Recognize an example never seen before.
Solve a problem using familiar format, but different specifics.
Higher levels (analysis, synthesis, evaluation)
Pick out components of a situation.
Problem solve by combining elements.
Make judgments of value.
PAGE 25

13. Preparing to write test items
Create test items while preparing class lessons.
Make note of questions asked frequently by students.
Make note of misconceptions made often by students during class or homework.
Invite students to submit items at the end of each class or at other times.
Adapt questions from formative assessments.

14. Categories of Test Questions
Test items can roughly be broken into
Objective (“fixed” or single correct response)
True-false
Multiple choice
Matching
Completion or fill-in-the-blank.
Subjective (extended response/multiple correct)
Short answer and/or Essay
Oral test
Case Study

15. Common pitfalls in writing test directions—and avoiding them!
Giving unclear directions: “Choose the best answer.”
What is meant by “best”? Funniest? Simplest? Cheapest? Easiest?
“Choose the correct answer” is a much better statement of directions—and then ensure that there is a single correct answer!
Providing unintended cues such as making the correct answer longer than the other options

16. Writing GOOD test directions
Indicate carefully and specifically how students should respond. For example, on a TRUE/FALSE section on a test:
Circle “V” for statements that are VERDAD and “F” for statements that are FALSO.
Do NOT have students write V or F in a blank.
Give point values for items and sections.
Allow students to EXPLAIN on any fixed-response (true-false, multiple choice, etc.) item.
Providing unintended cues such as making the correct answer longer than the other options

17. Writing True / False Items for Tests and Exams
Can be used to check for knowledge and comprehension.
Keep statements simple.
Make entirely true or entirely false.
Avoid double negative items.
In fact, best to avoid negative items.

18. Common pitfalls in writing true-false test items
Using NOT or EXCEPT in the statement
Using cues such as “ALWAYS” or “USUALLY”
Using unclear, vague terms such as “OFTEN” or “MANY” or “FEW”
Using double negatives in a statement
Using compound statements with one indpendent clause true and one, false.
Providing unintended cues such as making the correct answer longer than the other options

19. Writing True / False Test Items
For each of the statements below, circle “T” if true and circle “F” if false.
T F The length of gestation in swine is 33 days.
For each of the statements below, circle “T” if true and circle “F” if false. Also, explain in the blank why it is false.
__________________________________

20. Writing multiple choice items for tests and exams
Use distractors that are plausible
Do NOT use “all of the above” or “none of the above” or “only A and B” or “B and C on alternate Thursdays” as distractors
Use 4 or 5 distractors, but use the SAME number of distractors (4 or 5) for all items
Vary randomly where you put the correct answer.
Allow students to explain!
Providing unintended cues such as making the correct answer longer than the other options

21. Preparing for students who are test-wise to some kinds of items
Longest answer
“C” as correct answer
Process of elimination
Use grammatical cues
Scientific sounding
Avoid simple and obvious
Related word
Guess by figuring out your pattern
Pick the longest answer
make sure the longest answer is only correct a part of the time
try to make options equal length
b. When in doubt pick “c”
make sure the correct answer choice letter varies
c. Never pick an answer which uses the word ‘always’ or ‘never’ in it
make sure this option is correct part of the time or avoid using always and never in the option choices
d. If there are two answers which express opposites, pick one or the other and ignore other alternatives
sometimes offer opposites when neither is correct or offer two pairs of opposites
e. If in doubt, guess
use five alternatives instead of three or four to reduce guessing
f. Pick the scientific-sounding answer
use scientific sounding jargon in wrong answers
g. Don’t pick an answer which is too simple or obvious
sometimes make the simple, obvious answer the correct one
h. Pick a word which you remember was related to the topic
when creating the distractors use terminology from the same area of the text as the right answer, but in distractors use those words incorrectly so the wrong answers are definitely wrong

22. Writing Matching Items for Tests and Exams
Appropriate only for knowledge level testing
Use a reasonable number of items: 7 – 12
All premises in one list, all responses in another list
Premises can be long, but try to keep responses short
Use more responses than premises
Can approximate fill-in-the-blank items

23. Writing Clear Directions for Matching Items
Write the LETTER of the correct Response in the
Blank preceding the numbered premise.
____ 1. List premise
____ 2. List premise
____ 3. List Premise
____ 4. List Premise
____ 5. List Premise
____ 6. List Premise
Response

24. Writing Fill-in-the-Blank or Completion Test Items
Most useful for lower order levels of learning
Use your own words
Try to ensure that only one answer is correct, but, if that is too hard to do, …
You may consider “matching” using a word bank
Keep all blanks the same length and at END of statement
Avoid grammatical clues

25. Writing Directions for Completion Test Items Complete each statement by writing the correct word(s) in the blank(s) provided.
1. A thermometer should be rinsed in _____________ water. Good or bad item?
2. The letters “N-P-K” in fertilizers stand for, respectively, _____________, _____________, and _____________.

26. Extended Response Test Items
Short answer and/or Essay
Oral test
Case Study or “Word Problems/Situations”
These kinds of test questions require criteria and pre-determined descriptions of levels of performance to be graded reliably and fairly. We call that what?

27. Short Answer and/or Essay
May require many different behaviors:
Comparison, for or against, cause and effect, explanation, summary, analysis, describe relationships, apply rules, or discussion

28. Oral Test
All students must receive the same test item stated in a clear, concise manner designed to elicit correct answers arranged in logical order.
Determine beforehand how you may lead the students if they stumble.

29. Case Study Problem should be practical and realistic
Requires recall and comprehension of prior information and, if well-written, may require application, analysis, synthesis and/or evaluation of that information
“Word problems” are similar to a case study.

30. Which Type of Question to Use?
Select the appropriate technique to test the student competence at the level of cognition required by the objective.
Avoid using just one type of test question.
We know students have different strengths:
Visual
Audio
Kinesthetic

31. Materials in this presentation developed in part by…
Jason Peake
University of Georgia
Brian Parr
Auburn University
Dawn Zimmaro
University of Texas
Gary Briers
Texas A&M University