1. Testing to Evaluate Learning New Faculty Orientation 2005 Dr. Diane Wink Professor School of Nursing College of Health and Public Affairs ©Diane Wink 2005

2. Welcome To UCF

3. Why test?? Achievement of objectives of learning measured with evaluation

4. Why test?? Achievement of objectives of learning measured with evaluation

5. Why test?? Achievement of objectives of learning measured with evaluation Class activities must reflect objectives and what will be tested Class Experience and Activities

6. Objectives All objectives should lead directly to evaluation through the learning experience itself. Evaluation should be of the objectives. Class Experience and Activities

7. Questions to ask What is the best evaluation method to determine student mastery and application of this knowledge?

8. Questions to ask - Test Alternatives Term paper Class activity –Debate and bib development Problem set completion Service Learning activity Course project –Case Study –Development of tool Non written evaluation –Performance of the competency being developed

9. For Testing: Questions to ask How will this test help me know if learning objectives are being met? Is the test addressing essential information? Does the test reflect the readings, class activities and/or web discussion?

10. Questions to ask How many questions will it take to determine if student has met objectives/ has the knowledge? Do a test blueprint!

11. Test Blueprint

12. Written Tests Multiple Choice True-False Matching Essay and Short answer essay

13. For all types of exams Have a colleague review your items Consider the possibility of cheating –Alternative forms of tests –Random seating –3X5 key points card –New exams –No cell phones, PDAs –Open book if on-line –In person for on-line

14. Multiple Choice Items The stem should: –Contain a self-contained question or problem –Present as much of the item's content as possible –Be precise - no extraneous information

15. Multiple Choice Items The stem should: –Be independent of all other stems. s If part of a situation, say this clearly s Do not make answer dependent on getting a prior item correct –Be a positive rather than a negative statement or question Should not be a reading test!

16. Multiple Choice Items The answer choices should: –Contain only one best or correct answer –Be grammatically consistent with stem –Be similar in complexity, detail, length and plausibility –Avoid "all of the above" or "none of the above"

17. Multiple Choice Items The answer choices should: –All be plausible –Be arranged logically s Alphabetical, numerical, length –Randomly use each alternative position for correct answers –Contain action verbs from Bloom's Taxonomy

18. Multiple Choice Items The answer choices should: –Be on the same page as the stem –Not have obscure or esoteric language

19. Sample Item- Poor If all of the following heart valves are defective but only one can be repaired, which should it be? a. Aortic b. Mitral c. Pulmonic d. Tricuspid

20. Sample Item Narrowing and tightness of which heart valve is most likely to result in left ventricular enlargement? a. Aortic b. Mitral c. Pulmonic d. Tricuspid

21. True False Items State the item as simply and clearly as possible Keep statements equal in length Write each item so that it is clearly true or false

22. True False Items Avoid qualifications such as "often" or "usually" Randomize the true and false items Test important information and facts

23. Sample T/F Item (Hard to score) 15. For each of the following, choose A if true and B if false. a. The major pump chamber of the heart is the ventricle b. Valves control blood flow into the atria c. The pulmonary artery carries fully oxygenated blood to the lungs

24. Sample T/F Item (Easy to score) For each of the following, choose A if true and B if false. 15. The major pump chamber of the heart is the left ventricle 16. Valves control blood flow into the atria 17. The pulmonary artery carries fully oxygenated blood

25. Matching Items Put the basis of relationship for matching in the directions Put the entire set of matching items on a single page Arrange items in a systematic order Number premises on the left and letter the responses on the right a-e s Allows computer grading

26. Matching Items Use a larger or smaller set of responses and allow responses to be used more than once or not at all Use homogeneous lists that concern the same content Number each item separately to allow computer grading

27. Sample Matching Item - Poor 31. Match each item in Column A with an historical event listed in Column B. Events in Column B can be used more than once. Hoovera. Implemented emancipation proclamation Rooseveltb. Established the CCC Lincolnc. Major war ends Clintond. Stock market crash f. Was impeached

28. Sample Matching Item - Poor 31. Match each item in Column A with an historical event listed in Column B. Events in Column B can be used more than once. Hoovera. Implemented emancipation proclamation Rooseveltb. Established the CCC Lincolnc. Major war ends Clintond. Stock market crash f. Was impeached All questions (there are actually four) have same item number! Confusion over which Roosevelt, which stock market crash?

29. Sample Matching - Better Match each of the presidents listed in Column A with the historical event listed in Column B. Events in Column B can be used more than once. Column AColumn B 31.Hoovera. Post civil way reconstruction in progress in US 32.F.D. Rooseveltb. Establishment of the CCC 33.Lincolnc. WWII ends 34. Clintond. Stock market crashed starting depression f. Impeached

30. Exam grading support at UCF University testing service support for multiple choice/true false/matching exams WebCT Disability Services

31. Grading issues What if the item is bad??!!

32. Essay/Short Answer Items What do they test? Higher cognitive levels Application Analysis Synthesis Evaluation Not for simple recall of facts

33. Essay/Short Answer Items First ask –Is item clearly written? –If extended response needed, is it too broad? –If restricted response needed, would the content be assessed better with an objective item?

34. Sample Item Poor: List the components of a persuasive letter to a government official. Better: Write a letter to a government official in support of or opposed to gun control.

35. Sample Item Poor: What are the themes in War and Peace? Better: Relate a theme in War and Peace to world events in 2005

36. Grading essay/short essay items Write out the answer s Does your answer match the intent of the item? s What problems did you encounter in answering the item? s Can the student answer in the time allowed?

37. Grading essay/short answer items Analytical method Points for each essential element Holistic method Look for essential elements Evaluate overall quality of response Little feedback

38. Analytical Grading Method Identify criteria for evaluating the content of the response and assign points for each: –Essential elements required –Accuracy of content –Logical organization –Clarity –Depth and complexity of analysis –Use of references and specific citations * * Usually only used in formal papers

39. Analytical Grading Method Decide on the extent to which writing ability and use of style format will be included in the scoring –Spelling –Grammar –Sentence structure –Writing style –APA citation and format * * Usually only used in formal papers

40. Holistic Grading Method Sample “A” answer: Covers all key points with citations to major resources. Clearly written with analysis of problem clear and persuasive. Minimal to no grammatical or spelling errors.

41. Holistic Grading Method Sample “B” answer: Covers most key points with citations to most major resources. Clearly written with good analysis of problem. Few grammatical or spelling errors. Sample “C” answer: Covers few key points with few references to major resources. Written poorly with limited analysis of problem and a non persuasive argument Multiple grammatical or spelling errors.

42. Holistic Grading Method Sample “D” Answer: Omits most key points with few/no citations to resources. Writing difficult to follow with consistent grammatical and spelling errors Sample “F” answer: Omits all key points, does not answer question

43. Grading essay and short answer items Before scoring s Read a random sample of papers s Get a sense of how students answered s Revise evaluation criteria if needed s Review grading criteria with any other graders - practice!!! s Do key for commonly made comments (give to students and all graders) 1. Great job. Shows excellent analysis of your main points 2. Great ideas but needed to be developed more 3. Unclear what you mean 4. Needs citation

44. Grading essay and short answer items Ideal grading technique: –Score answers to one question at a time for all students –Read each answer twice s First for omissions of essential elements s Second reread and score

45. Grading essay and short answer items Ideal grading technique: –Cover up prior scores to items or record them on a separate answer sheet –Read papers anonymously; have students use a code number

46. Grading essay and short answer items If fatigued, stop and rest!

47. Grading essay/short answer items GTA Help from peers Consider time for grading when designing exams

48. Questions???

49. Bloom’s Taxonomy Cognitive Knowledge Identification and recall of information Comprehension Organization and selection of facts and ideas Application Use of facts, rules principles

50. Bloom’s Taxonomy Cognitive (continued) Analysis Separation of a whole into component parts Synthesis Combination of ideas to form a new whole Evaluation Development of opinions, judgments, or decisions

51. Bloom’s Taxonomy Affective Domain Feeling tone Emotion Degree of acceptance or rejection Psychomotor Domain

52. Resources Used Oermann, M. (1999). Developing and scoring essay tests. Nurse Educator, 24(2), 29-32. Tomey, A. (1999). Preparing a testing blueprint from levels of course objectives. Nurse Educator, 24(4), 11-12. Tomey, A. (1999). Selected response test items. Nurse Educator, 24(5), 9- 10. Tomey, A. (1999). Completion testing items. Nurse Educator, 24(6), 6-7.