1. ASSESSMENT AND EFFECTIVE GRADING PRACTICES DAVID W. KALE, PH.D. DIRECTOR OF ASSESSMENT MOUNT VERNON NAZARENE UNIVERISTY

2. ACKNOWLEDGEMENTS Most of the ideas in this workshop come from two excellent books on the topic. One is Effective Grading: A Tool for Learning and Assessment by Barbara Walvoord and Virginia Johnson Anderson. Any really good ideas you hear in this session probably came from that book or Knowing What Students Know by the National Research Council. Most of the ideas in this workshop come from two excellent books on the topic. One is Effective Grading: A Tool for Learning and Assessment by Barbara Walvoord and Virginia Johnson Anderson. Any really good ideas you hear in this session probably came from that book or Knowing What Students Know by the National Research Council.

3. WORKSHOP OBJECTIVES 1. Faculty will be able to relate principles of effective grading to their courses; 1. Faculty will be able to relate principles of effective grading to their courses; 2. Faculty will be able to discern various levels of learning in their courses; 2. Faculty will be able to discern various levels of learning in their courses; 3. Faculty will be better able to use grades for motivation, organization and evaluation; 3. Faculty will be better able to use grades for motivation, organization and evaluation; 4. Faculty will be able to use grading to more effectively reflect student learning. 4. Faculty will be able to use grading to more effectively reflect student learning.

4. WORKSHOP OUTLINE 1. Principles of effective grading 2. Motivating students with grades 3. The Assessment Triangle 4. Linking Assessment with Grades 5. Linking Assessment, Grading and Classroom Activities

5. GUIDING PRINCIPLES FOR EFFECTIVE GRADING SUPERIOR

6. Principle #1 Appreciate the complexity of grading No grading system is immutably right by some external, eternal standard. No grading system is immutably right by some external, eternal standard. Collaborate with students to make the grading process meaningful to them and to you. Collaborate with students to make the grading process meaningful to them and to you. The goal is to construct a system that will lead to meaningful change for both you and the students. The goal is to construct a system that will lead to meaningful change for both you and the students.

7. Principle #1 Appreciate the complexity of grading (con’t) The purpose of grades is to provide a means of: The purpose of grades is to provide a means of: (1) Evaluation, (2) Communication, (3) Motivation, and (4) Organization. This is a large order for any grading process to achieve well.

8. Principle #2 Substitute Judgments for Objectivity Recognize that grading is a professional judgment influenced by a wide variety of factors. Recognize that grading is a professional judgment influenced by a wide variety of factors. These factors are meaningful to you and you will work to make them meaningful to your students. These factors are meaningful to you and you will work to make them meaningful to your students.

9. Principle #3 Distribute time effectively Recognize the time necessary to make a consistent, thoughtful, professional judgment. Recognize the time necessary to make a consistent, thoughtful, professional judgment.

10. Principle #4 Be Open to Change Be ready to make changes when students are not demonstrating the learning you want to see in them. Be ready to make changes when students are not demonstrating the learning you want to see in them. If changes in the grading system are to be made in the middle of a course (which hopefully will not often be necessary), be very clear with students as to what is being changed, why it is being changed and when the changes take effect. If changes in the grading system are to be made in the middle of a course (which hopefully will not often be necessary), be very clear with students as to what is being changed, why it is being changed and when the changes take effect.

11. Principle #5 Communicate and collaborate with your students The more actively students are involved in the learning process, the better they will understand your grading system. The more actively students are involved in the learning process, the better they will understand your grading system. Clearly explain your criteria and your standards for effective performance. (Rubrics really help here, but that is the topic for another workshop.) Clearly explain your criteria and your standards for effective performance. (Rubrics really help here, but that is the topic for another workshop.)

12. Principle #6 Remember, Student learning is your primary goal. Give students a picture of how a professional would judge their work. Give students a picture of how a professional would judge their work. Support your judgments with clear and full explanations. Support your judgments with clear and full explanations. Provide information on how students can improve. Provide information on how students can improve. Grades, when used effectively, can be a powerful motivator of student learning. Grades, when used effectively, can be a powerful motivator of student learning.

13. Principle #7 Be a Teacher First and a Gatekeeper Last Use grades for both formative and summative purposes. Use grades for both formative and summative purposes. Help guide students through the process, using grades as a way of helping you and the students track what they have learned. Help guide students through the process, using grades as a way of helping you and the students track what they have learned.

14. Principle #8 Encourage Learning Centered Motivations Counter student perceptions that: Counter student perceptions that: 1. Hard work doesn’t matter; 2. They are powerless to affect their own welfare; 3. Failure is due to circumstances beyond their control; 4. Grades are mostly an indication of who the professor likes or does not like.

15. DAILY GRADING I am convinced some students enter courses asking themselves how they can get through the course with the least effort possible. I am convinced some students enter courses asking themselves how they can get through the course with the least effort possible. Others have a sincere desire to learn, but want their learning to be recognized and rewarded. Others have a sincere desire to learn, but want their learning to be recognized and rewarded. Daily grading both keeps the pressure on the first student and motivates the second student. It treats learning as something that is continuous throughout the course. Daily grading both keeps the pressure on the first student and motivates the second student. It treats learning as something that is continuous throughout the course.

16. DAILY GRADING The motivational aspect of grading suggests that students should feel that learning is something that is expected in each class period and not just when the time comes to cram for a test. The motivational aspect of grading suggests that students should feel that learning is something that is expected in each class period and not just when the time comes to cram for a test. Provide motivation to learn as often as you can, in every class session if possible. Provide motivation to learn as often as you can, in every class session if possible. Perhaps there is a grade for participation in discussion which could be as simple as a plus, check or minus for each class. Perhaps there is a grade for participation in discussion which could be as simple as a plus, check or minus for each class.

17. DAILY GRADING In Senior Colloquium I have questions posted on Blackboard at the beginning of the course for all assigned readings. In Senior Colloquium I have questions posted on Blackboard at the beginning of the course for all assigned readings. I tell students their class participation grade comes largely from whether they know the answers to those questions when I call on them. I tell students their class participation grade comes largely from whether they know the answers to those questions when I call on them. I make an effort to call on every student in every class period and with 25 students in the class I generally accomplish that. I make an effort to call on every student in every class period and with 25 students in the class I generally accomplish that.

18. MOTIVATING STUDENTS Barbara Walvoord has demonstrated that student motivation can change from one course to another and within a particular course. Barbara Walvoord has demonstrated that student motivation can change from one course to another and within a particular course. Some students who enter a course with a feeling that there is little they can do to affect their outcome can get highly motivated when they see that their initial impression was not true. Some students who enter a course with a feeling that there is little they can do to affect their outcome can get highly motivated when they see that their initial impression was not true.

19. MOTIVATING STUDENTS A sociologist, talking about teaching a general education course, “You always get the students who are interested right at the start, but the ones I really like, the ones that get my adrenalin going, are the ones who are slouching back, thinking, ‘What a jerky course this is.’ Then you show them what sociologists do, and how much fun it is, and sometimes, wow, They get it.” A sociologist, talking about teaching a general education course, “You always get the students who are interested right at the start, but the ones I really like, the ones that get my adrenalin going, are the ones who are slouching back, thinking, ‘What a jerky course this is.’ Then you show them what sociologists do, and how much fun it is, and sometimes, wow, They get it.”

20. SCAFFOLDING Establish a building relationship among the objectives and the assignments in the course. Create a course outline (some call it a course map) which makes clear both: Establish a building relationship among the objectives and the assignments in the course. Create a course outline (some call it a course map) which makes clear both: – how the whole course structure holds together, and – how each assignment or test builds on what the student has already learned.

21. THE ASSESSMENT TRIANGLE OBSERVATION COGNITION INTERPRETATION PELLEGRINO, ET. AL., KNOWING WHAT STUDENTS KNOW

22. COGNITION “In any particular assessment application, a theory of learning in the domain is needed to identify the set of knowledge and skills that is important to measure for the task at hand, whether that be characterizing the competencies students have acquired thus far or guiding instruction to increase learning.” Pellegrino, et. Al., Knowing What Students Know, p. 44. “In any particular assessment application, a theory of learning in the domain is needed to identify the set of knowledge and skills that is important to measure for the task at hand, whether that be characterizing the competencies students have acquired thus far or guiding instruction to increase learning.” Pellegrino, et. Al., Knowing What Students Know, p. 44.

23. COGNITION What are the dominant learning theories in your area about how students learn material in that discipline? What are the dominant learning theories in your area about how students learn material in that discipline?

24. COGNITION Instructional strategies should then lead students through the learning process in the way which most closely matches the way in which they learn this material best. Instructional strategies should then lead students through the learning process in the way which most closely matches the way in which they learn this material best. For example, my experience has been that students do not learn good speechmaking by reading sample speeches nearly as well as they learn by giving speeches and getting prompt feedback on how to improve. For example, my experience has been that students do not learn good speechmaking by reading sample speeches nearly as well as they learn by giving speeches and getting prompt feedback on how to improve.

25. COGNITION As a matter of fact, having students read speeches mixes the written with the oral media which could lead students to believe that as long as they are good writers they can be good speakers. This could be an instructional strategy which leads students directly away from the objective I am trying to achieve. As a matter of fact, having students read speeches mixes the written with the oral media which could lead students to believe that as long as they are good writers they can be good speakers. This could be an instructional strategy which leads students directly away from the objective I am trying to achieve.

26. COGNITION For most disciplines, students must master a first exposure level of learning which is then followed by a higher processing level. For most disciplines, students must master a first exposure level of learning which is then followed by a higher processing level.

27. LEVELS OF LEARNING In teaching Advanced Public Speaking, I always found that skill in audience analysis was an important processing objective in that course. In teaching Advanced Public Speaking, I always found that skill in audience analysis was an important processing objective in that course. In the first exposure level, students learned the factors that were important in audience analysis, (e.g., age, culture, level of education, etc.) but (1) using those factors to do an actual analysis and then (2) building a speech based on that analysis were two important higher level processing course objectives. In the first exposure level, students learned the factors that were important in audience analysis, (e.g., age, culture, level of education, etc.) but (1) using those factors to do an actual analysis and then (2) building a speech based on that analysis were two important higher level processing course objectives.

28. LEVELS OF LEARNING The reading for homework would provide a review of audience analysis categories students were exposed to in Public Speaking, providing a more in depth discussion than they had in the earlier course. The reading for homework would provide a review of audience analysis categories students were exposed to in Public Speaking, providing a more in depth discussion than they had in the earlier course. My in-class assessment was to have students watch a speech and then work in groups to identify the evidences that the speaker had adapted the speech to that particular audience. My in-class assessment was to have students watch a speech and then work in groups to identify the evidences that the speaker had adapted the speech to that particular audience.

29. LEVELS OF LEARNING In Senior Colloquium, familiarizing students with the decision making model for moral and ethical decisions was the first exposure level. In Senior Colloquium, familiarizing students with the decision making model for moral and ethical decisions was the first exposure level. I would then have them read a case study involving a moral or ethical decision as a homework assignment and work through the case study in class to see if they were at the processing level. I would then have them read a case study involving a moral or ethical decision as a homework assignment and work through the case study in class to see if they were at the processing level.

30. LEVELS OF LEARNING This would give me an idea as to how many case studies I would need to do in class to help them get to the level of analysis I wanted them to achieve. This would give me an idea as to how many case studies I would need to do in class to help them get to the level of analysis I wanted them to achieve. It would also tell me what aspects of the model I needed to emphasize in the case studies. It would also tell me what aspects of the model I needed to emphasize in the case studies.

31. LEVELS OF LEARNING Hopefully by mid-term exam time they were ready to be shown a case study for the first time when they walked into class and then use the decision making model to analyze the case as the mid-term exam. Hopefully by mid-term exam time they were ready to be shown a case study for the first time when they walked into class and then use the decision making model to analyze the case as the mid-term exam. I remember well how distressed I was the first time I did this and in scoring the tests realized that the weakest part of the students answers was in using Christian principles as the basis for their decision making. (:<) I remember well how distressed I was the first time I did this and in scoring the tests realized that the weakest part of the students answers was in using Christian principles as the basis for their decision making. (:<) I knew I had to make some changes. I knew I had to make some changes.

32. COGNITION What is the “exposure level” learning and what is the “higher processing level” learning you want your students to acquire in your course? What is the “exposure level” learning and what is the “higher processing level” learning you want your students to acquire in your course? Bloom’s Taxonomy would be a good starting place for identifying the types of learning students are likely to go through in coming to grips with the material of your discipline. Bloom’s Taxonomy would be a good starting place for identifying the types of learning students are likely to go through in coming to grips with the material of your discipline.

33. BLOOM’S TAXONOMY Bloom’s taxonomy is a means of identifying the level of cognitive processing you are expecting from students. Bloom’s taxonomy is a means of identifying the level of cognitive processing you are expecting from students. This can be applied to the objectives you are setting for your course as well as to the means you are using to determine the degree to which those objectives are being achieved. This can be applied to the objectives you are setting for your course as well as to the means you are using to determine the degree to which those objectives are being achieved.

34. Original Terms New Terms Evaluation Evaluation Synthesis Synthesis Analysis Analysis Application Application Comprehension Comprehension Knowledge Knowledge CreatingCreating EvaluatingEvaluating AnalysingAnalysing ApplyingApplying UnderstandingUnderstanding RememberingRemembering (Based on Pohl, 2000, Learning to Think, Thinking to Learn, p. 8)

35. THE ASSESSMENT TRIANGLE OBSERVATION COGNITION INTERPRETATION PELLEGRINO, ET. AL., KNOWING WHAT STUDENTS KNOW

36. OBSERVATION In this stage you are identifying the evidence you would expect to observe that indicates that a particular level of learning has taken place. In this stage you are identifying the evidence you would expect to observe that indicates that a particular level of learning has taken place. For each of your course’s objectives, identify the evidence that would indicate to you and the student that the objective has been achieved. For each of your course’s objectives, identify the evidence that would indicate to you and the student that the objective has been achieved.

37. THE ASSESSMENT TRIANGLE OBSERVATION COGNITION INTERPRETATION PELLEGRINO, ET. AL., KNOWING WHAT STUDENTS KNOW

38. INTERPRETATION Here is where the grading process comes in because this is where you determine the degree to which the objective has been achieved. Here is where the grading process comes in because this is where you determine the degree to which the objective has been achieved. The grade should therefore be closely tied to the evidence and to your theory about how the learning takes place. The grade should therefore be closely tied to the evidence and to your theory about how the learning takes place.

39. LINKING ASSESSMENT WITH GRADING MAKE YOUR GRADES HIT THE BULL’S EYE OF STUDENT LEARNING

40. GRADING Once the evidence of learning pertaining to a particular objective has been identified, you are then ready to attach grades to the various levels. Once the evidence of learning pertaining to a particular objective has been identified, you are then ready to attach grades to the various levels. See the examples which are included in your handout. See the examples which are included in your handout.

41. GRADING Link your grading to the level of cognitive processing in your objectives and your assignments. Assignments and tests that require higher level thinking skills should have greater weight in your grading scheme. Link your grading to the level of cognitive processing in your objectives and your assignments. Assignments and tests that require higher level thinking skills should have greater weight in your grading scheme.

42. LINKING ASSESSMENT WITH OBJECTIVES AND GRADING Review the number of questions on multiple choice tests in light of the course objectives to which they are linked. Review the number of questions on multiple choice tests in light of the course objectives to which they are linked. Are all questions related to important course objectives for a unit or the course? Are all questions related to important course objectives for a unit or the course? Do the most important objectives have the most questions associated with them or at least the greatest weight in the test grade? Do the most important objectives have the most questions associated with them or at least the greatest weight in the test grade?

43. LEVELS OF LEARNING AND COURSE ASSESSMENTS Make sure your assessments are lined up with the level of learning at which you expect students to perform, whether at the first exposure level or the higher processing level. Make sure your assessments are lined up with the level of learning at which you expect students to perform, whether at the first exposure level or the higher processing level. Don’t ask first exposure type questions if you are expecting students to be processing at a higher level and don’t ask higher level questions if students are only at the first exposure level. Don’t ask first exposure type questions if you are expecting students to be processing at a higher level and don’t ask higher level questions if students are only at the first exposure level.

44. LINKING ASSESSMENT, GRADING AND CLASS ACTIVITIES OBJECTIVES ASSESS- MENTS CLASS ACTVITIES GRADING

45. LEVELS OF LEARNING What are the evidences you will look for to indicate that the student has moved from the first exposure to the processing level of learning? What are the evidences you will look for to indicate that the student has moved from the first exposure to the processing level of learning? What activities will you have planned for your students to help them move from the exposure level to the processing level? Is there a reading to be done before class, a case to be read and analyzed, a set of problems to be solved, etc.? What activities will you have planned for your students to help them move from the exposure level to the processing level? Is there a reading to be done before class, a case to be read and analyzed, a set of problems to be solved, etc.?

46. LEVELS OF LEARNING What built in class activity can you design to see whether students have moved from the first exposure level to the higher level processing level? This becomes an important in-class assessment. The students may consider the activity to be solely instructional, while in fact it may also have an important assessment purpose. What built in class activity can you design to see whether students have moved from the first exposure level to the higher level processing level? This becomes an important in-class assessment. The students may consider the activity to be solely instructional, while in fact it may also have an important assessment purpose. We call this course-embedded assessment which may be a case study to analyze, a problem to solve, a group activity to complete, etc. We call this course-embedded assessment which may be a case study to analyze, a problem to solve, a group activity to complete, etc.

47. LEVELS OF LEARNING Note you may have an activity to help students move to a higher level and a subsequent activity to determine whether students have moved to the higher level. Note you may have an activity to help students move to a higher level and a subsequent activity to determine whether students have moved to the higher level. On the other hand, you may have one activity serve both purposes. On the other hand, you may have one activity serve both purposes.

48. LEVELS OF LEARNING The key is to use class time as much as possible for process oriented teaching and use outside of class time for first exposure to the concepts involved. The key is to use class time as much as possible for process oriented teaching and use outside of class time for first exposure to the concepts involved. By the end of the class period, students should be demonstrating evidence as to the level of learning at which they are performing. By the end of the class period, students should be demonstrating evidence as to the level of learning at which they are performing.

49. THE CLASSROOM “FLIP” Jeremy Strayer demonstrated in his doctoral dissertation research that students learned just as well when the first exposure learning was done outside of class, rather than in an in-class lecture, and the application of the learning was done during the class period. This has been called the classroom “flip.” Jeremy Strayer demonstrated in his doctoral dissertation research that students learned just as well when the first exposure learning was done outside of class, rather than in an in-class lecture, and the application of the learning was done during the class period. This has been called the classroom “flip.”

50. LEVELS OF LEARNING Sometimes this will require a demonstration by the faculty member of the process oriented learning you are expecting of the student. Sometimes this will require a demonstration by the faculty member of the process oriented learning you are expecting of the student. You will need to decide whether that demonstration or example can be in reading they do before class or whether it needs to be an in-class demonstration. You will need to decide whether that demonstration or example can be in reading they do before class or whether it needs to be an in-class demonstration.

51. LEVELS OF LEARNING Walvoord gives an example of a physics professor who said students could not understand a physics text without the guidance of a faculty member so he had to lecture in class. Walvoord gives an example of a physics professor who said students could not understand a physics text without the guidance of a faculty member so he had to lecture in class. Colleagues suggested that he have his lectures videotaped so students could review them as they read the material before class and then use class time for application purposes. Colleagues suggested that he have his lectures videotaped so students could review them as they read the material before class and then use class time for application purposes.

52. LEVELS OF LEARNING He has students work in groups to do homework problems so they taught each other. He has students work in groups to do homework problems so they taught each other. If a group got stuck, they could get help from the professor right when they needed it. If a group got stuck, they could get help from the professor right when they needed it. Once that demonstration tape was made, it could be re-used with future classes. Once that demonstration tape was made, it could be re-used with future classes.

53. LEVELS OF LEARNING Even in large classes, the class could be broken up into groups to work on problems and call for help from the professor when they needed it. Even in large classes, the class could be broken up into groups to work on problems and call for help from the professor when they needed it. The degree to which groups were able to arrive at the correct answers on their own was an important in-class assessment of student learning. The degree to which groups were able to arrive at the correct answers on their own was an important in-class assessment of student learning.

54. LARGE CLASSES For large classes, break up the class into on-line discussion groups on Blackboard to discuss readings outside of class. For large classes, break up the class into on-line discussion groups on Blackboard to discuss readings outside of class. Blackboard will give you the opportunity to “drop in” on their discussions to see if they are being productive and give comments to them if they are “off track.” Blackboard will give you the opportunity to “drop in” on their discussions to see if they are being productive and give comments to them if they are “off track.”

55. SCIENCE LABS Science professors often complain that students do not apply the concepts learned in the lecture when they get to the laboratory. Science professors often complain that students do not apply the concepts learned in the lecture when they get to the laboratory. Some science professors require students to have a “ticket” to get into the lab. The “ticket” is an explanation, in their own words, of the concepts from the lecture or reading the lab is intended to demonstrate. (Walvoord and Anderson, p. 62-63.) Some science professors require students to have a “ticket” to get into the lab. The “ticket” is an explanation, in their own words, of the concepts from the lecture or reading the lab is intended to demonstrate. (Walvoord and Anderson, p. 62-63.)

56. LEVELS OF LEARNING My experience is that giving students questions to be answered based on reading assignments outside of class is an excellent way of preparing them to move from the first exposure to the process oriented level of learning. My experience is that giving students questions to be answered based on reading assignments outside of class is an excellent way of preparing them to move from the first exposure to the process oriented level of learning. These questions should be carefully crafted so they do not assume a higher level of learning than what should be expected after first exposure, but yet cause some cognitive dissonance with staying at that level. These questions should be carefully crafted so they do not assume a higher level of learning than what should be expected after first exposure, but yet cause some cognitive dissonance with staying at that level.

57. LEVELS OF LEARNING I always tried to include at least one question that students could not answer strictly based on the reading, but which could be answered if the student gathered some additional information or did some critical thinking about the issue involved. I always tried to include at least one question that students could not answer strictly based on the reading, but which could be answered if the student gathered some additional information or did some critical thinking about the issue involved.

58. LEVELS OF LEARNING In Senior Colloquium, students also know that debates are coming up in which they will be expected to debate both sides of a controversial issue. In Senior Colloquium, students also know that debates are coming up in which they will be expected to debate both sides of a controversial issue. I emphasize to them that developing the ability to use the decision making model will help them as they prepare for the debates. I emphasize to them that developing the ability to use the decision making model will help them as they prepare for the debates.

59. PRIMARY TRAIT ANALYSIS Barbara Walwoord has suggested that we think about the primary traits we are attempting to develop in our students. Barbara Walwoord has suggested that we think about the primary traits we are attempting to develop in our students. These traits should grow directly out of the course objectives. These traits should grow directly out of the course objectives.

60. PRIMARY TRAITS IN WORK RELATED PERFORMANCE Comprehension Comprehension Problem identification and solution Problem identification and solution Organization Organization Creativity Creativity Analysis Analysis Synthesis Synthesis

61. PRIMARY TRAITS IN WORK RELATED INTERACTIONS WITH OTHERS Collaboration Collaboration Listening skills Listening skills Leadership skills Leadership skills Team work Team work Group facilitation Group facilitation

62. PRIMARY TRAIT ANALYSIS AND LEVELS OF LEARNING For each primary trait you identify, work out a hierarchy of student learning which you hope to see students achieve. For each primary trait you identify, work out a hierarchy of student learning which you hope to see students achieve. What are the levels of learning within each particular trait? What are the levels of learning within each particular trait? What evidence would you expect to see that students were performing at a particular level? What evidence would you expect to see that students were performing at a particular level?

63. SUMMARY 1. Grading is most effective when it provides as accurate a reflection of student learning as we can possible achieve. 2. For that to happen, we need to have as close an association as possible among course objectives, assessment strategies and classroom activities.