1. Effective grading (and teaching) through the use of rubrics & use of rubrics for assessment Sarah Murnen Kenyon College

2. Personal Background Although a psychologist, no particular background in this topic As Kenyon’s “Assessment Coordinator” from 2002-2006 I learned that: Some faculty resent the assessment process and see it as extra burden Departments with faculty who feel burdened by assessment don’t get much out of it Departments that find a way to integrate assessment with what they already do benefit from the process – they discuss what students are learning and change their teaching methods, assignments, and sometimes their entire curricula to help students learn

3. Benefits of Ohio-5 Project To help improve student learning (and faculty teaching) by encouraging the use of rubrics (Many other benefits to the use of rubrics) To specifically encourage a focus on creativity and critical thinking, important aspects of a liberal education To use rubrics for assessing critical thinking and creativity To show faculty how all of the work they put into grading can be used for assessment purposes

4. Background Information Presentation by Dr. Douglas Eder on “Primary Trait Analysis” Presentation by Barbara Walvoord on the use of rubrics for effective grading Walvoord, B. E., P& Anderson, J. A. (1998). Effective Grading: A tool for learning and assessment. Jossey- Bass: San Francisco, CA. Stevens & Levi (2005). Introduction to rubrics: An assessment tool to save grading time, convey effective feedback, and promote student learning. Stylus Publishing: Sterling, Virginia.

5. Our primary concern is student learning Good practice in undergraduate education (Walvoord, 1998, p. 15): Encourages student-faculty contact Encourages cooperation among students Encourages active learning Gives prompt feedback Emphasizes the time that students devote to the task Communicates high expectations Respects diverse talents and ways of learning

6. Walvoord asks (1998, p. 15): “How many of these principles of good practice in some way involve the grading system in your class, the tests and assignments on which that system is based, and your ways of communicating with students about their work and their grades?”

7. Walvoord’s Argument for Establishing Clear Criteria and Standards for Grading Saves time in grading process Allows you to make the process consistent and fair Helps you explain to students what you expect Shows you what to teach Identifies essential relationship between discipline information and processes Help students evaluate their own and each other’s work Saves you from having to explain your criteria to students after they have handed in their work, as a way of justifying the grades they are contesting Helps student peers give each other constructive feedback on plans and drafts Helps team teachers or teaching assistants grade student papers consistently Helps teachers of sequenced courses communicate with each other about standards and criteria Form the basis for departmental or institutional assessment

8. How to Establish Clear Criteria: Primary Trait Analysis Developed to score essays on the National Assessment of Educational Progress by Lloyd- Jones in1977 Creates a scoring rubric that can be used to assess any student performance Is assignment specific Can be used for grading For this project, we will develop rubrics using PTA to measure the critical thinking and creativity we are encouraging in our students

9. Why take the time to do PTA for grading? Makes grading more consistent and fair Saves time in the grading process once the rubric is developed Can diagnose students’ strengths and weaknesses very specifically in order to teach more effectively Can track changes in student performance

10. Process of PTA (Eder) Identify the “primary traits” – essential or central components of the discipline to be learned by the student Build a scale for scoring the student’s performance on the trait Evaluate the student’s performance against those criteria

11. Key Stages in Constructing a Rubric (Stevens & Levi) 1. Reflecting - What do we want from our students? Why did we create the assignment? What happened the last time we used the assignment? What are our expectations for the assignment? 2. Listing – Focus on particular details of the assignment and what specific learning objectives we hope to see completed (Sometimes helps to imagine the best and the worst performance on the assignment) 3. Grouping and Labeling the goals together - Organize the results of our reflections in Stages 1 and 2, grouping similar expectations together in what will probably become the dimensions of the rubric 4. Application – Transfer groupings to a rubric grid

12. Example: Research Article Critique 1. Reflecting - Want students to learn to be critical evaluators of psychological research The research article critique should start them thinking critically, will follow up with class discussion on each critique, and use multiple critiques throughout the semester to help develop their skills I expect students will move from description to analysis 2. Listing – I have seen students move through Bloom’s taxonomy in a semester using this assignment: Most can show “knowledge” and “comprehension” at the beginning of the course, move to “application,” “analysis,” “synthesis,” and “evaluation” (hopefully). I want to try to capture this process. 3. Grouping and Labeling the goals together – see the questions that follow 4. Application – Transfer groupings to a rubric grid See the rubric that follows

13. Research Article Critique Assignment For each article analysis you are to answer the following questions: What is the primary question posed by the study? Is there a hypothesis stated? If so, what is it? What is the theoretical explanation for the proposed hypothesis? Briefly describe the way the independent and dependent variable(s) were manipulated or measured How do the results of the study affect the originally posed hypothesis (or purpose of study)? Two strengths of the study? Two weaknesses of the study? What is a logical extension of the study? Briefly describe a study you could conduct to extend the research

14. Sample Rubric – Designed for Assessment ACC = Accomplished, AVG = average, DEV = developing, BEG = beginning Understands primary purpose of research Clearly labels the independent and dependent variable without extraneous material provided and shows understanding of theoretical basis of study ACCAVGDEVBEG Describes measurement issues clearly Hones in on how the variables are manipulated/measured without providing extraneous material ACCAVGDEVBEG Understands significance and implication of the results E.g., does research represent a methodological improvement, novel hypothesis, important test of a theory, etc. ACCAVGDEVBEG Chooses relevant and important strengths of study Discusses strengths that are very relevant to judging the internal validity of the study (or possibly external validity) ACCAVGDEVBEG Chooses relevant and important weaknesses of study Discusses weaknesses that are very relevant to judging the internal validity of the study (or possibly external validity) ACCAVGDEVBEG Proposed study an important extension of the research E.g., methodological improvement, advancement of theory, tests an alternative explanation, etc. ACCAVGDEVBEG Creativity e.g., Unusual association of ideas in responses, proposed study is novel and represents a significant advancement ACCAVGDEVBEG

15. Rubric Used for Grading – weight each component 5% Understands primary purpose of research Clearly labels the independent and dependent variable without extraneous material provided and shows understanding of theoretical basis of study ACCAVGDEVBEG 5% Describes measurement issues clearly Hones in on how the variables are manipulated/measured without providing extraneous material ACCAVGDEVBEG 10% Understands significance and implication of the results E.g., does research represent a methodological improvement, novel hypothesis, important test of a theory, etc. ACCAVGDEVBEG 25% Chooses relevant and important strengths of study Discusses strengths that are very relevant to judging the internal validity of the study (or possibly external validity) ACCAVGDEVBEG 25% Chooses relevant and important weaknesses of study Discusses weaknesses that are very relevant to judging the internal validity of the study (or possibly external validity) ACCAVGDEVBEG 25% Proposed study an important extension of the research E.g., methodological improvement, advancement of theory, tests an alternative explanation, etc. ACCAVGDEVBEG 5% Creativity e.g., Unusual association of ideas in responses, proposed study is novel and represents a significant advancement ACCAVGDEVBEG

16. Describing the level of performance on the rubric Anywhere from 2 to 5+ levels of performance on each trait Terms used to describe level of performance might be positive and active to encourage motivation in students (suggestions from Stevens & Levi): Mastery, partial mastery, progressing, emerging high level, middle level, beginning level Sophisticated, competent, partly competent, not yet competent Exemplary, proficient, marginal, unacceptable Advanced, intermediate high, intermediate, novice Distinguished, proficient, intermediate, novice Accomplished, average, developing, beginning

17. Using Rubrics for Assessment We all need to attend to the issue of assessment, and Walvoord argues that we can make use of what we already do in the grading process

18. American Association for Higher Education’s “Principles of Good Practice for Assessing Student Learning” Answer questions that people care about Lead directly to improvement in teaching and learning Be embedded in the context of learning Take place repeatedly over time

19. The rubrics we use for grading can be used for assessment “If we as faculty do not make our learning goals, tests, criteria, and standards explicit and understandable to legislatures, boards, accrediting agencies, and other audiences in ways that meet their needs and concerns, we face the very real possibility that some of the control we currently exercise in the classrooms will be taken away from us. We must deal with assessment, but we need not construct a parallel assessment structure that ignores the assessment we already conduct.” (Walvoord, 1998, p. 5).

20. How to turn this into assessment Paper 1Paper 2Paper 3Total Prof A Creative1 4127 Creative24127 Critical13137 Critical22136 Prof B Creative1 3227 Creative24138 Critical13137 Critical22136 Prof C Creative1 4127 Creative23238 Critical12237 Critical22136 Prof D Creative1 4239 Creative24239 Critical13238 Critical23249 Total502345

21. How have faculty been influenced by this process? “I was able to clarify my expectations for the course by articulating the purpose behind each graded exercise. This helped dramatically in easing anxieties about performance and final grades. Students seemed to appreciate understanding the logic behind the assignments and seeing that each focused on building and assessing a particular skill, rather than merely providing me with another grade.” “Best of all, I’m looking forward to sharing the process of developing PTA-based rubrics with my students. Specifically, students will work in groups to identify their own PTA-based critical thinking rubrics before they embark on writing a particular essay. I can hardly think of a better way to develop the students’ meta-cognition., i.e., how they think about their own thinking!” “Yes, we’ve been disappointed with some aspects of student performance (for example synthetic ability) and using the rubric has helped us communicate better with students, and we’re thinking about ways to improve courses to help teach the skills needed to succeed on this senior exercise objective.” “I have learned a great deal, and I think that my teaching has been positively influenced by this important work. I can also say that my research and scholarship have been impacted by these foci, as well.”

22. Go to Website for Rubric Samples: http://www.wooster.edu/teagle/ default.html