1. Win May USC Keck School of Medicine
RATING SCALES
Win May
USC Keck School of Medicine

2. CRITIQUING RATING SCALES
Does the rating scale have a title describing the performance/trait to be measured?
Does the performance/trait to be measured reflect the objectives/
Is the performance/ trait to be measured clearly defined in operational terms?
Are the reference points on the scale clearly defined?

3. CRITIQUING RATING SCALES (cont.)
How many categories/rating positions are utilized? (Optimum 5-9)
Is each category/rating position described in specific observable behavioral terms?
If there is an option not to assess the behavior, is a concrete reason given for using that option?

4. CRITIQUING RATING SCALES (cont.)
Are the statements in the form of single sentences?
Are the statements phrased in a positive manner?
How long is the rating scale? (Optimum behaviors)
Are there directions for use of the scale?
Is there a key with the directions?
Is there provision for training of raters?

5. CRITIQUING RATING SCALES (cont.)
Test the rating scale in a focus group, by having it read aloud. This will enable you to make sure that every person understands each item in the same way.

6. WHY TRAIN RATERS?
RATERS DO BETTER,
IF TRAINED

7. COMMON RATING ERRORS
Error of central tendency - the hesitancy to give extreme judgments
Error of leniency - the tendency of a rater to rate too high or too low regardless of what trait is being tested
The halo effect - the contamination of the rating of a specific trait by the overall impression of the person.

8. COMMON RATING ERRORS (continued)
The logical error - the tendency to apply the same rating to two traits or skills that the rater feels are logically related.
The contrast error - the contamination of the rating of a specific trait by the rater’s possession of that trait.
The proximity error - the tendency to be influenced by traits or persons rated just previous to the present rating

9. (Habitual Performance)
COMMON RATING ERRORS
(Habitual Performance)
Generalization error - making ratings at the end of a designated period of time, based on samples, which are too small or not representative
Rater bias - anything which causes the rater to make a less than totally objective assessment of a learner’s performance.

10. SELECTION OF RATERS
Important to determine the ability of a rater to rate a specific trait
Raters are more accurate (do better) :
when they are interested in the ratings they make
if they have educational and professional backgrounds similar to those of the ratees

11. RATER TRAINING Train with respect to: Distribution of abilities
Nature of the scale
Halo effect
Logical errors, etc.

12. OTHER INFLUENCING FACTORS
Raters tend to have greater leniency error when they must confront ratee with the ratings
Raters produce more accurate ratings when they are aware that they will be checked
Raters must have sufficient time to complete the ratings

13. OTHER INFLUENCING FACTORS (cont.)
Accuracy is influenced by the number or categories or the fineness of discrimination called for by the rating task.
The more judgment is required, the lower the rater accuracy
Raters rate their colleagues, or fellow teachers or fellow students higher than they rate others.
Length of acquaintance is highly correlated with errors of leniency, but can be reduced by training.

14. OTHER INFLUENCING FACTORS (cont.)
Two ratings by the same rater on the same content are no more valid than one.
Raters disagree more when they observe individuals in different situations, than when they observe the same situation.
Different raters use different criteria in judging the same trait.