2 Please send comments or suggestions to firstname.lastname@example.org This presentation is provided as part of our commitment to serving our clientsPlease send comments or suggestions toCritiCall® is a registered trademark ofBiddle Consulting Group, Inc.
3 General Notes Cutoff scores in C4 are set based on AutoTest Codes AutoTest Codes are a series of test modules that an employer groups together for testing for a specific job or job categoryYou must create an AutoTest Code using the Test Creation Wizard (or the AutoTest Setup feature) before cutoff scores for that AutoTest code can be setCutoff scores are typically determined by an employer’s own “subject matter experts” using C4’s Validation Wizard featureThey are then entered using the “Set Cutoff Scores” feature in the C4 Administrator’s Program
4 Table of Contents Quick Links Click here with your mouse to continue the presentation about setting cutoff scoresClick here to jump ahead in the presentation to the section on using Cutoff Scores and Printing Candidate Score Reports with Pass or Fail Status Indicated
5 Before you set cutoff scores … We strongly recommend that you review the validation report you created before you set any cutoff scores to make certain all of the information has been entered appropriately by your job experts
6 Before you set your cutoff scores you should check to see whether you have used the Validation Wizard feature for the AutoTest Code in questionTo do this, click on the “Validation Report” link in the C4 Administrator’s Program
7 Select the AutoTest Code you wish to view from the Validation Report menu
8 If you have not validated this AutoTest Code using the Validation Wizard you will see this message
9 You are now ready to set your cutoff scores. If you have validated this AutoTest Code using the Validation Wizard you will see this screen7After 7 to 10 Job Experts have used the Validation Wizard, it should say “Yes” for all of the tests. If not, you will need to create a new AutoTest Code without the invalid testsIt is recommended that a minimum of 7 to 10 Job Experts use the Validation Wizard to validate the tests, if possibleYou are now ready to set your cutoff scores.Hint: It may be helpful to print the Validation Report before you set the cutoff scores
10 Click hereTo set the cutoff scores, you first click on the “Set Cutoff Scores” link on the C4 Administrator’s Screen
11 initially appears here – for other AutoTest Codes Default AutoTest Codeinitially appears here –Use this menu to viewcutoff score settingsfor other AutoTest CodesThis is the screen you see if the listed AutoTest Code has been validated using the “Validation Wizard” feature
12 The message below appears on the screen if no Validation Report has been run for the chosen AutoTest Code“Call Center”Note: This message will also appear if Subject Matter Experts have responded to the Validation Wizard survey for the listed AutoTest Code, but the Validation Report feature has not been run in the Administrator’s Program for that AutoTest Code
13 The first column shows the tests contained in this AutoTest Code
14 The second column is where you set the cutoff scores for each of the tests
15 First choice: Keep the cutoff score at “none,” which is the default. Default is “None”If a Validation Report has been run for this AutoTest Code you have five choices when setting cutoff scores…First choice: Keep the cutoff score at “none,” which is the default.This means the cutoff score is set at zero – everyone passes no matter what they score.
16 Less 1 SEMSecond choice: Set the cutoff score at the cutoff score suggested by your subject matter experts less one standard error of measurement
17 This is a sample of the Report Summary section of a C4Validation Report Where do the cutoff scores in the “Cutoff Score Setup” screen come from?The cutoff scores listed in the “Cutoff Score Setup” area match the cutoff scores shown in the Report Summary section of the C4 Validation Report Summary
18 U.S. v. South Carolina, 434 US 1026 (1978) Question: Why is there more than one recommend cutoff score for each test?Answer: Because no tests are perfect. They include "measurement error" and the U.S. Supreme Court has supported adjusting minimum passing (cutoff) scores to account for this.U.S. v. South Carolina, 434 US 1026 (1978)
19 The U.S. Supreme Court has supported that when a test is used for selection (i.e., hiring) purposes, employers consider adjusting the cutoff score downward one, two, or three standard error of measurements (SEMs) considering the five factors below
20 What is a Standard Error of Measurement? The standard error of measurement for each test is shown in the Detailed Validation ReportWhat is a Standard Error of Measurement?If a person were to take the same test repeatedly, with no change in his or her level of knowledge and preparation, it is possible that some of the resulting scores would be slightly higher or slightly lower than the score that precisely reflects the person’s actual level of knowledge and ability. The Standard Error of Measurement sets a boundary around where these hypothetical scores will lie. For example, a person who scores 45 on a typing test that has a SEM of 5 would most likely score between 40 and 50 about 70% of the time.
21 SEM refers to the standard error of measurement If you had to compute the standard error of measurement yourself, you would use this formulaSEM refers to the standard error of measurementSD refers to the standard deviation of test scoresrxy refers to the reliability of the testNote: More sophisticated methods are available for computing conditional SEMs (SEMs that change throughout the test score range) for tests that have more test items than are typically contained on CritiCall tests.
22 If you had to compute the standard error of measurement, you would use this formula SEM refers to the standard error of measurementSD refers to the standard deviation of test scoresrxy refers to the reliability of the testThe C4 Systemautomatically computes the standard error of measurement and adjusted cutoff scores so you don’t have toFor more details, refer to the “Test Validation” chapter, “Cutoff Scores” section, of the CritiCall Administrator’s Manual
23 For example, the average less one SEM for this test is the average of suggested cutoffs (65%) minus one SEM (1.718), which is rounded out to 63%.Calculating the adjusted cutoff scores using the standard error of measurementThe average less two SEMs for this test isthe average of suggested cutoffs (65%) minus two SEMs (3.435), which is rounded out to 62.We follow the same logic for the average less three SEMs
24 So, the U.S. Supreme Court recommends you consider using one of these three cutoff scores instead of the average of suggested cutoffsPlease contact Biddle Consulting Group, Inc. at if you have additional questions about how cutoff scores are indicated in the Validation Report
25 Less 2 SEMsThird choice: Set the cutoff score at the cutoff score suggested by your subject matter experts less two standard error of measurements
26 Less 3 SEMsFourth choice: Set the cutoff score at the cutoff score suggested by your subject matter experts less three standard error of measurements
27 Fifth choice: Set a custom cutoff score. Add cutoff scoreFifth choice: Set a custom cutoff score.Be extremely careful when setting custom cutoff scores since you may increase your liability exposure if you set an inappropriate cutoff score
28 First choice: Keep the cutoff score at “none,” which is the default. Default is “None”If no Validation Report has been run for this AutoTest Codeyou will have only two choicesFirst choice: Keep the cutoff score at “none,” which is the default.This means the cutoff score is set at zero – everyone passes no matter what they score.
29 If no Validation Report has been run for this AutoTest Code Add cutoff scoreWe recommend entering one of the adjusted cutoff scores (average cutoff score minus one, two, or three standard errors of measurement) listed in the Validation Report you have produced for this AutoTest CodeIf no Validation Report has been run for this AutoTest CodeSecond choice: Set the cutoff score at the cutoff score suggested by your subject matter experts or using some custom cutoff score
30 The third column shows how the cutoff score will be used Rule: Pass/FailIf a test taker scores below the cutoff score that has been set for ANY test that uses the “Pass/Fail” rule, that test taker fails that test. This rule currently cannot be changed or modified.
31 “Low SME Warning” is shown if less than 7 SMEs used the The final column indicates the total number of the subject matter experts (SMEs) who have used the Validation Wizard for this AutoTest CodeNumber of SubjectMatter Experts“Low SME Warning” is shownif less than 7 SMEs used theValidation Wizard for this testThe U.S. Courts indicate that a minimum of seven to ten Subject Matter Experts should provide validation information, if possible
32 After all of the cutoff scores are set, click on the “Save” and then the “Close” buttons If you wish to exit without saving any changes, just click on the “Close” button
33 Using Cutoff Scores and Printing Candidate Score Reports with Pass or Fail Status Indicated
34 Click hereTo print a Candidate Score Report showing test takers Pass or Fail status, click on the “Candidate Score Report” link on the C4 Administrator’s Screen
35 First, select the candidate(s) whose Candidate Score Report you wish to print by clicking on the check box by their namesHint: You can limit the candidates who appear on this screen by creating “filters” in the area provided (see above).After filtering, click on the “Update List” button to apply the filter. (See CritiCall Manual for more details)
36 To create a report that indicates whether a person passes or fails an AutoTest Code you have set cutoff scores for, check () the “Pass/Fail Report” option on the Candidate Score Report dialogue box before printing the report
37 The dialogue box shown here will appear on the screen even if you have used the Validation Wizard for this AutoTest Code. It is merely an advisory message reminding users that they must set the cutoff scores before they will accurately appear on the Pass/Fail version of the Candidate Score Report.
38 This is what the Pass/Fail Candidate Score Report looks like IMPORTANT! Check the AutoTest Code name here to make sure this is the AutoTest Code you wanted to scoreIf this is not the AutoTest Code you wanted to score, it may be beneficial to go back to the Candidate Score Report dialogue box and limit the test takers by filtering
39 This is what the Pass/Fail Candidate Score Report looks like Test Taker’s score for each testThe tests in the Candidate Score Report are listed in the order in which they were administered to the test taker
40 This is what the Pass/Fail Candidate Score Report looks like Candidates’ Pass/Fail status for each testCutoff scores you have set using the Set Cutoff Score feature
41 This is what the Pass/Fail Candidate Score Report looks like Pass in this column means the test taker scored at or above the cutoff score set for each of the C4 test administeredPass/Fail status for each test module and for the overall scoresFail in this column means the test taker scored below the cutoff score set for a C4 test
42 This is what the Pass/Fail Candidate Score Report looks like In this example the test taker failed the “Customer Service Interaction” test since they scored (61%), which is lower than the cutoff score (63%) that had been set
43 In our experience…Many employers find it beneficial to consider only those candidates who have passed all of the required tests in the AutoTest CodeThose who fail one or more tests may have difficulties in those areas if hiredThose who fail one or more tests are more likely to need additional supervision and training if hired
44 Frequently Asked Question Question: Can I change the cutoff scores for an AutoTest Code after they have been set and used to score some test takers?Answer: Yes. You can change cutoff scores at any time.However, every Candidate Score Report you print out reflects the most recently set cutoff scores, even if some of those candidates have been graded using previously set cutoff scores
45 If you need assistance or more information Technical Support – ext 127Sales – ext 150Other Assistance – ext 151OrC4 (CritiCall® for Contact Centers) is a product of Biddle Consulting Group, Inc. 193 Blue Ravine Road, Suite 270, Folsom, CA 95630CritiCall® is a registered trademark of Biddle Consulting Group, Inc.