1. Preparing Effective Essay Questions Bryan Bradley Center for Teaching and Learning January 21, 2010 http://ctl.byu.edu/home/about/contact/

2. Focus: Focus on essays as a way to assess thinking and reasoning; (we will Not be focusing on essays as a way to assess writing ability.)

3. Objectives: Review definition of an essay question Evaluate when an essay question should be used Construct effective essay questions Discuss how to prepare students to take essay exams Review methods for scoring student responses

4. Learning-Centered Objectives FOCUS ON HIGHER-LEVEL LEARNING GOALS LEVEL 1 Remember Retrieve relevant knowledge from long- term memory LEVEL 2 Understand Construct meaning from instructional messages, including oral, written and graphic LEVEL 3 Apply Carry out or use a procedure in a given situation LEVEL 4 Analyze Break material into parts and relate to one another and to larger structure / purpose LEVEL 5 Evaluate Make judgments based on criteria and standards LEVEL 6 Create Put elements together to form a coherent / functional whole; reorganize into new patterns L.W. Anderson and D. R. Krathwohl (eds). A Taxonomy for Learning, Teaching and Assessing, 2001.

5. 5 Effective Alignment of Course Design Components Learning Outcomes (Verb) Learning Activities (Verb) Learning Assessment (Verb) 1. Remember 2.Understand 2.Understand 3. Apply 4. Analyze 5. Evaluate 6. Create

6. Categories of In-Class Assessments Which level of cognitive activity would be most valuable for students and instructors during class? Remember Tell, list, describe, name, repeat, remember, recall, identify, state, select, match, know, locate, report, recognize, observe, choose, who, what, where, when Understand Change, explain, restate, find, describe, review, relate, define, clarify, illustrate, diagram, outline, summarize, interpret, paraphrase, transform, compare similarities and differences, derive main idea Apply Apply, practice, employ, solve, use, demonstrate, illustrate, show, report, paint, draw, collect, dramatize, classify, put in order Analyze Analyze, dissect, detect, test, deconstruct, discriminate, distinguish, examine, focus, survey, compare, contrast, classify, investigate, outline, structure, categorize, same/different, solve, determine evidence and conclusions Evaluate Coordinate, judge, select/choose, decide, debate, evaluate, justify, recommend, verify, monitor, measure, the best way, what worked, what could have been different, what is your opinion, test Create Create, hypothesis, design, construct invent, imagine, compose, pretend, predict, organize, plan, modify, improve, suppose, produce, set up, what if, propose, formulate, solve (more than one answer)

7. What is an Essay Question? It reflects the cognitive (thinking) behaviors noted in the course goals and modeled in the readings, discussions and other course activities. It clearly defines and establishes the task within the problem situation. It presents a reasonable task to students. It is generally written as an imperative statement or as one or more questions. It reflects the major areas of emphasis for which students will be scored.

8. Evaluate the following essay question. Does it meet the criteria? List the Alcoholics Anonymous 12-steps for alcoholism recovery. List them in their correct progressive order.

9. Evaluate the following essay question. Does it meet the criteria? Explain in what ways a person’s failure to apply step 5 of the 12-steps for alcoholism recovery will impact his/her ability to continue on the path to sustained sobriety. Provide an example that illustrates this impact.

10. Common Misconceptions Essay questions assess higher-order or critical thinking skills regardless of how the responses are written. Essay Questions are easy to construct. The use of essay questions eliminates the problem of guessing. Essay questions benefit all students by placing emphasis on the importance of written communication skills. Essay questions encourage students to prepare (study) more thoroughly.

11. When Should You Use Essays? It is appropriate to use essay questions for the following purposes: To assess students' understanding of subject-matter content. To assess students' abilities to reason with their knowledge of a subject.

12. When Should You Use Essays? Use essay questions for intended learning outcomes that require complex thinking and constructed responses.

13. When Should You Use Essays? If an intended learning outcome could be either assessed through objective items or essay questions, use essay questions for the following situations: Your skill in writing objective items is poor, but your resources and time for grading are high (e.g., small classes, grading assistants). Student reasoning needs to be evaluated.

14. How to Construct Essay Questions 1.Clearly define the intended learning outcome to be assessed by the item. 2.Avoid using essay questions for intended learning outcomes that are better assessed with other kinds of assessment. 3.Clearly define the task (thinking processes and content to use in answering the question) and situate it in a novel problem situation. a.Clearly define the task (thinking processes and content). b.Delimit the scope of the task. c.Clearly develop the problem or problem situation. 4.Present a reasonable task to students. 5.The tasks can be written as a statement or a question. 6.Specify the relative point value and the approximate time limit in clear directions.

15. How to Construct Essay Questions 7.State the criteria for grading. 8.Know the trade-offs between focused and less-focused essay questions. 9.Avoid the use of optional questions. 10.Improve the essay question through preview and review. a.Predict the student responses. b.Write a model answer. c.Ask a knowledgeable colleague to critically review the essay question, the model answer and the intended learning outcome for alignment. d.Review student responses to the essay questions.

16. Match the Assessment to the Context of the Learned Skill

17. Supporting Students in Preparing for Essay Questions Emphasize Alignment Highlight the required skills students need Ask probing questions Model test taking strategies Link learning goals and class activities Give feedback to students –Frequent –Immediate –Discriminating –Lovingly

18. Two Approaches to Scoring Essays Holistic Prepare an ideal response as a standard. Essay responses representing different degrees of lesser quality may also be used as anchor points The grader rapidly reads each essay and forms a general impression of it. The general impression can be recorded in pencil next to the response. The grader focuses on the whole essay as an integrated unit rather than focusing on specific features that a good essay should include The essay as a whole is compared to the model answer (and other anchors, if any) and given a single overall rating Analytical List the components that should be contained in an ideal answer. This should include major ideas but also specific features such as accuracy and factual information, pertinent examples, relevant reasons, unified organization etc. The grader reads each essay and searches for component ideas or desired features Points are assigned based upon the degree to which each desired component is present Each component should be scored independently

19. Scoring Student Responses to Essay Questions Decide whether to use the holistic or the analytic approach. Develop the scoring key/model answer(s). Decide in advance how to handle irrelevant and inaccurate info, bluffing, or technical problems. Improve the scoring key/model answer. Evaluate responses. Don't look at the student's name. Apply the answer key to all students. Grade only one question at a time. Be aware of typical student errors. Look for reoccurring mistakes in student responses. Randomly shuffle the order of papers each time before grading a new question. If possible score responses twice. If possible ask another person to score a few of the items to check for reliability.

20. Rubrics http://sat.collegeboard.com/scores/sat-essay-scoring-guide http://www.pz.harvard.edu/Research/RubricsSelfPE.htm http://www.rcampus.com/rubricshowc.cfm?sp=true&code=B37C97&

21. Conclusion Questions Reflections Final Thoughts