Grading and Analysis Report For Clinical Portfolio 1

Analysis for Formal Traditional Assessment All grades and names recorded in this example are from the accompanying handouts. 1. After grading, Rank order the students according to overall grade. 2. Record the results of each item in a chart like your worksheet and the following slide. 1 is used to indicate a correct answer and a 0 is used to indicate an incorrect answer. If a student were to earn half credit for an item, a 0.5 would be used. 2

Analysis for Formal Traditional Assessment 3

3. Once all scores are recorded for each student in each item, compute the sum for each item (sum the column for each item). 4

Analysis for Formal Traditional Assessment 5

6

4. Next compute the p value (item difficulty index) for each of the items using the equation below. All answers will range from 0.0 to 1.0. p = # of students who answered the item correctly Total # test takers OR p = Σ (column x ) n 7

Analysis for Formal Traditional Assessment 8

9

p Value Interpretation for Classroom Teachers: A p value below 0.75 is considered problematic. That tells us that at least 25% of the students answered the item correctly… That could mean that: 1)the students do not understand the information tested by the item; 2)the item is not well written, but the students do understand the information; 3)the item is miss-keyed or there is more than one right answer; 4)the students did not study. 10

Analysis for Formal Traditional Assessment 5. Divide the students into 3 groups, based on their grades. Put the upper 1/3 in group 1(g1), the middle 1/3 in group 2(g2), the lower 1/3 in group 3 (g3). Groups 1 and 3 must have an equal number of students in the groups. 11

Analysis for Formal Traditional Assessment 12

Analysis for Formal Traditional Assessment 6. Compute the D values (item discrimination index) for the items using the formula below. Notice the difference in the denominator of this equation. All answers will range from -1.0 to 1.0. D =# correct upper group - # correct lower group Total # test takers in each group OR D = Σ u – Σ l n g 13

Analysis for Formal Traditional Assessment 14

Analysis for Formal Traditional Assessment 7. Determine which items are problematic according to p and D values. 15

Analysis for Formal Traditional Assessment D Value Interpretation for Classroom Teachers: D values are really something that Norm referenced test writers are concerned with. Typically, any item with a D value below 0.40 or negative in value is considered to not discriminate between those with the highest grades and those with the lowest grades. Really, in the classroom, we want all students to do well!!, so typically we REALLY want the students with the lowest grades to correctly answer as many items as possible! 16

Analysis for Formal Traditional Assessment D Value Interpretation for Classroom Teachers: Classroom teachers are concerned with an item that has a D = or approaches 1.0 Means that all of the top/highest scoring group (G1) answered the item correctly and all or almost all of the bottom/lowest group (G3) answered the item incorrectly. Tells us that at least the lowest 1/3 rd of the class has missed the concept of that item completely …Definite need for remediation. 17

Analysis for Formal Traditional Assessment D Value Interpretation for Classroom Teachers: Also concerned with any item that has a D = some negative value, Means that more of the students with the lowest scores answered correctly than those with the highest scores. First and foremost, the item has to be checked to see if it is miss-keyed. Negative value could also be the result of poor directions; poor item construction; or the common occurrence of those who score well to typically rush through assessments and not read directions or items completely. 18

Analysis for Formal Alternative Assessment Analysis of the function of the Grading Rubric How well does the Rubric actually grade the product/presentation, the students’ knowledge and skill capability? 1.After grading, Record the score received from each criterion for each student in a chart like the one on the next slide. This will be done for each criterion that you measure with the rubric. An X indicates the points earned by each student for the criterion. 19

Analysis for Formal Alternative Assessment 20

Analysis for Formal Alternative Assessment 2. Once all scores are recorded for each student in each criterion, calculate the sum for each column for each point value. 3. Next compute the relative frequency of each score by dividing the sum of each column by the total number of students. 4. To create a number that is easier to interpret, multiply frequency decimal by 100. You will do this for each point value for each criterion. 21

Analysis for Formal Alternative Assessment 22

Analysis for Formal Alternative Assessment 23

Analysis for Formal Alternative Assessment 5. Once this has been done for each of the criteria, analyze the frequencies to determine how appropriate the criteria graded the performance assessment. For the previous chart,Aesthetics, 72% of the students scored a 3 and above, and 27% of the students scored a 1 or 2. The missing 1% is due to rounding errors. This criteria is an appropriate measure of the aesthetics of the student work b/c more than 70% of the students scored a 3 or higher (meaning that at least 70% of the students scored a D or higher on that criteria). 24

Analysis for Formal Alternative Assessment If the percentage of 3 or higher is below 70%, then the criteria may not be grading effectively because it does not match the expectations; or is grading too harshly against the quality of work the students are capable of. If either is true, the criteria would need to be adapted before final grades are returned to the students. It may also mean that the students did not work up to their full potential, in that case the rubric can stand, but the responses must be returned with extensive constructive feedback. My analysis of the Aesthetics criteria is that it is grading the aesthetic quality of student work effectively b/c 72% of the student scored a passing grade or higher. 25

Analysis for Formal Alternative Assessment 6. Do this analysis for each criterion and based on the results, determine if the entire rubric is appropriate or not. This is done by looking at the analysis of each criterion. If more of the criteria are effective than ineffective, the rubric is an effective grading tool. 26

Analysis for Formal Alternative Assessment 27

Analysis for Formal Alternative Assessment 28

Grading and Analysis Report for the Clinical Portfolio See Wiki for Example of Grading and Analysis Report For Traditional Assessments - Create a chart like the one on slide 3 - Fill in your information - Complete the statistics to find the p and D values - Answer the following questions - Which items fall below the p value of 0.75? Why do you think this occurred? What would need to be done about these items? - Which items have a D value of 1.0 or a value that is negative? Why do you think this occurred? What would need to be done about these items? 29

Item Analysis for the Clinical Portfolio For Performance Assessments - Create a chart like the one on slide 20 for each criteria (maximum of 4) - Fill in your information - Complete the statistics to find the frequency values - Answer the following questions - How do each of the criteria rate? How does this overall rubric rate? What, if anything, would need to be done before this rubric is used for grading? 30

Final Reflection 31 A reflection commenting on the assessment’s effectiveness is required after you analyze student responses. This is a brief reflection (max 1 page) discussing the experience of administering your traditional and alternative assessments. Please discuss good and bad items/experiences, suggestions for changes, and theories for why certain items/techniques did or did not work as well. Also include how the student responses reflected what was taught following the assessment.