Presentation is loading. Please wait.

Presentation is loading. Please wait.

Fire & Police Selection, Inc. Dan Biddle, Ph.D. Stacy L. Bell, M.S. August 26, 2011 A National Fire Chief Survey Reveals the Key Ingredients to Include.

Similar presentations


Presentation on theme: "Fire & Police Selection, Inc. Dan Biddle, Ph.D. Stacy L. Bell, M.S. August 26, 2011 A National Fire Chief Survey Reveals the Key Ingredients to Include."— Presentation transcript:

1 Fire & Police Selection, Inc. Dan Biddle, Ph.D. Stacy L. Bell, M.S. August 26, 2011 A National Fire Chief Survey Reveals the Key Ingredients to Include in Your Fire Department Test

2 Copyright © 2011 Fire & Police Selection, Inc. Contact Information Dan A. Biddle, Ph.D. CEO, Biddle Consulting Group, Inc./Fire & Police Selection, Inc. Stacy L. Bell, M.S. Executive Vice President, Fire & Police Selection, Inc. 193 Blue Ravine, Ste. 270 Folsom, CA | | | | s

3 Copyright © 2011 Fire & Police Selection, Inc. Biddle Consulting Group, Inc. Assisted hundreds of clients with cases involving Equal Employment Opportunity (EEO) / Affirmative Action (AA) (both plaintiff and defense) EEO Litigation Support / OFCCP (federal contracting) Audit Support Compensation Analyses / Test Development and Validation Published: Adverse Impact and Test Validation, 2 nd Ed., as a practical guide for HR professionals Editor & Publisher: EEO Insight an industry e-Journal Creator and publisher of a variety of productivity Software/Web Tools: –OPAC ® (Administrative Skills Testing) –CritiCall ® (9-1-1 Dispatcher Testing) –AutoAAP (Affirmative Action Software and Services) –C 4 (Contact Center Employee Testing) –Encounter (Video Situational Judgment Test) –Adverse Impact Toolkit (free online at –AutoGOJA ® (Automated Guidelines Oriented Job Analysis ® ) Industry Leader

4 Copyright © 2011 Fire & Police Selection, Inc. Fire & Police Selection, Inc. Over 80 Physical Ability Test installations at fire departments across the U.S. Over 80 Physical Ability Test installations at fire departments across the U.S. Over 800 personnel and fire and police clients across the U.S. and Canada Over 800 personnel and fire and police clients across the U.S. and Canada Entry-level written, physical, interview assessments, manipulative skills tests, cognitive ability tests, and personality tests available for selection of public safety personnel Entry-level written, physical, interview assessments, manipulative skills tests, cognitive ability tests, and personality tests available for selection of public safety personnel Post-Academy tests Post-Academy tests Fire Promotional tests and assessment exercises Fire Promotional tests and assessment exercises Online applicant testing Online applicant testing Custom job analysis and validation services Custom job analysis and validation services

5 Copyright © 2011 Fire & Police Selection, Inc. Presentation Overview Personnel Testing Overview Personnel Testing Overview What is Validation and Why Do I Need It? What is Validation and Why Do I Need It? Test Usage Test Usage The National Fire Select TestA Case Study The National Fire Select TestA Case Study Benefits to a Balanced Testing Process Benefits to a Balanced Testing Process

6 Copyright © 2011 Fire & Police Selection, Inc. Approach to Test Validation The Quickly Changing Climate of Personnel Testing… EEOC DOJ DOL (OFCCPOffice of Federal Contract Compliance Programs) State EEO Offices of Fair Employment Private Plaintiff Groups

7 Copyright © 2011 Fire & Police Selection, Inc. Adverse Impact & Test Validation Uniform Guidelines & Civil Rights Act of 1991 Amends Section 703 of the 1964 Civil Rights Act (Title VII) (k)(1)(A). An unlawful employment practice based on disparate impact is established under this title only if: A(i) a complaining party demonstrates that a respondent uses a particular employment practice that causes a disparate impact on the basis of race, color, religion, sex, or national origin, and the respondent fails to demonstrate that the challenged practice is job-related for the position in question and consistent with business necessity; OR, A(i) a complaining party demonstrates that a respondent uses a particular employment practice that causes a disparate impact on the basis of race, color, religion, sex, or national origin, and the respondent fails to demonstrate that the challenged practice is job-related for the position in question and consistent with business necessity; OR, A(ii) the complaining party makes the demonstration described in subparagraph (C) with respect to an alternate employment practice, and the respondent refuses to adopt such alternative employment practice. A(ii) the complaining party makes the demonstration described in subparagraph (C) with respect to an alternate employment practice, and the respondent refuses to adopt such alternative employment practice.

8 Copyright © 2011 Fire & Police Selection, Inc. How Can Testing Practices be Challenged? Title VII Disparate Impact Discrimination Flowchart OR

9 Copyright © 2011 Fire & Police Selection, Inc. Validation Standards Court Precedence Court Precedence Title VII: Uniform Guidelines on Employee Selection Procedures (1978): EEOC, DOJ, CSB, DOL Title VII: Uniform Guidelines on Employee Selection Procedures (1978): EEOC, DOJ, CSB, DOL Principles for the Validation and Use of Personnel Selection Procedures (2003) Principles for the Validation and Use of Personnel Selection Procedures (2003) Division 14 of the American Psychological Association (the Society for Industrial and Organizational Psychology, or SIOP)Division 14 of the American Psychological Association (the Society for Industrial and Organizational Psychology, or SIOP) Standards for Educational and Psychological Testing, published by: Standards for Educational and Psychological Testing, published by: American Educational Research AssociationAmerican Educational Research Association American Psychological Association, andAmerican Psychological Association, and National Council on Measurement in EducationNational Council on Measurement in Education

10 Copyright © 2011 Fire & Police Selection, Inc. What is Validation? Traditionally, it is making sure a selection procedure (i.e., test) measures what it is designed to measure Traditionally, it is making sure a selection procedure (i.e., test) measures what it is designed to measure In a legal realm, a selection procedure is valid if it can be proven by an employer that it is …job related for the position in question and consistent with business necessity. In a legal realm, a selection procedure is valid if it can be proven by an employer that it is …job related for the position in question and consistent with business necessity.

11 Copyright © 2011 Fire & Police Selection, Inc. Great Job Performance

12 Copyright © 2011 Fire & Police Selection, Inc. Reliable… but not Valid Consistently hiring the wrong people

13 Copyright © 2011 Fire & Police Selection, Inc. Neither Reliable or Valid Hit or Miss…

14 Copyright © 2011 Fire & Police Selection, Inc. Reliable and Valid! Better Productivity Less Liability Employee Satisfaction Increased Reduced Turnover & Absenteeism Less Time to Train More Satisfied Clients

15 Copyright © 2011 Fire & Police Selection, Inc. Uniform Guidelines Style Validation… Content Validity Content Validity Section14C Section14C Criterion Validity Criterion Validity Section 14B Section 14B Construct Validity Construct Validity Section 14D Section 14D

16 Copyright © 2011 Fire & Police Selection, Inc. Job Duties Operationally defined KSAOs Other KSAOs Selection Devices (e.g., application form, tests, interviews) Content Validity Essential #3: Connecting the Test to the Job Content Valid!

17 Copyright © 2011 Fire & Police Selection, Inc. Test Usage Pass/Fail Cutoffs: Pass/Fail Cutoffs: –Normal Expectations of Acceptable Proficiency in the Workplace (Guidelines, 5H) –Modified Angoff (U.S. v. South Carolina, USSC) Banding: Banding: –Substantially Equally Qualified Applicants –Statistically Driven (use Std. Error of Difference) Ranking: Ranking: –Is there adequate score dispersion? –Does the test have high reliability? (e.g., >.85) –Is the KSA performance differentiating? Weighted/combined with other tests Weighted/combined with other tests –How are the weights related to the job –Do they come from the job analysis or SME ratings?

18 Fire & Police Selection, Inc Using Test Scores Ranking assumes one applicant is reliably more qualified than the other Banding considers the unreliability of the test battery and ties applicants Pass/fail cutoffs treat all applicants as either qualified or not qualified

19 Copyright © 2011 Fire & Police Selection, Inc. Pass/Fail Cutoffs Some major considerations: Some major considerations: NOT ADVISED:NOT ADVISED: Using an arbitrary cutoff (e.g., 70%)Using an arbitrary cutoff (e.g., 70%) Using applicant scores to benchmark (e.g., setting cutoff scores at mean-SD of applicant scores)Using applicant scores to benchmark (e.g., setting cutoff scores at mean-SD of applicant scores) ADVISED:ADVISED: Consider Normal expectations of acceptable proficiency in the workplace (Guidelines, 5H)Consider Normal expectations of acceptable proficiency in the workplace (Guidelines, 5H) Usually requires SME-level data or ratingsUsually requires SME-level data or ratings # Applicants that can be feasibly processed# Applicants that can be feasibly processed CONSIDER:CONSIDER: Is the test supported by content validity information or criterion- related information?Is the test supported by content validity information or criterion- related information? How critical are the KSAs measured?How critical are the KSAs measured? Does the test measure baseline or differentiating KSAs?Does the test measure baseline or differentiating KSAs? How would current incumbents perform on this test?How would current incumbents perform on this test?

20 Copyright © 2011 Fire & Police Selection, Inc. Comparison Between Test Use Options FactorRankingBanding Pass/Fail Cutoffs Validation Requirements HighModerateLow Adverse Impact HighModerateLow DefensibilityLowHighHigh Litigation "Red Flag" HighModerateLow UtilityHighModerateLow CostLowModerateHigh Applicant Flow Restrictive/ControllableModerate/ControllableHigh Development Time LowModerateHigh Reliability Requirements HighModerateLow # Item Requirements HighModerateLow

21 Copyright © 2011 Fire & Police Selection, Inc. The Value of a Standardized Written Test Candidates bounce from one department to the next taking entry-level firefighter tests that measure different skills and abilities when in fact the job requirements rarely differ from department to department. Candidates bounce from one department to the next taking entry-level firefighter tests that measure different skills and abilities when in fact the job requirements rarely differ from department to department. There is value to a standardized national firefighter written test similar to the concept of the standardized physical ability test (e.g., C-PAT). There is value to a standardized national firefighter written test similar to the concept of the standardized physical ability test (e.g., C-PAT). How can we measure critical skills and abilities that are standard entry-level requirements across fire departments in a way that is job-related and fair to candidates? How can we measure critical skills and abilities that are standard entry-level requirements across fire departments in a way that is job-related and fair to candidates?

22 Copyright © 2011 Fire & Police Selection, Inc. Selecting the Ideal Firefighter Critical Skills & Critical Skills & Abilities Abilities Test A Test B Reading Ability Mathematical Ability Spatial Abilities Mechanical Aptitudes Writing Ability Interpersonal Skills Decision Making Test C Map Reading NFST

23 Copyright © 2011 Fire & Police Selection, Inc. Development & Validation of the National Fire Select Test Leveraged cumulative firefighter job analysis data from the last decade to identify which skills and abilities are consistently rated as being critical to successful firefighter job performance. Leveraged cumulative firefighter job analysis data from the last decade to identify which skills and abilities are consistently rated as being critical to successful firefighter job performance. Administered the Buy the Perfect Firefighter survey to build the test content and test weights. Administered the Buy the Perfect Firefighter survey to build the test content and test weights. Used Subject-Matter Experts (SMEs) and job performance ratings to identify a job-related cutoff. Used Subject-Matter Experts (SMEs) and job performance ratings to identify a job-related cutoff.

24 Copyright © 2011 Fire & Police Selection, Inc. Development & Validation of the National Fire Select Test FPSI sent surveys to over 1,000 chief officers from fire departments across the U.S. who were asked to weigh the importance of key critical skills and abilities required for successful entry-level firefighter job performance. FPSI sent surveys to over 1,000 chief officers from fire departments across the U.S. who were asked to weigh the importance of key critical skills and abilities required for successful entry-level firefighter job performance. Respondents were asked to evaluate both cognitive (e.g., Academic Success) and non-cognitive (e.g., Personal Characteristics) skills and abilities. Respondents were asked to evaluate both cognitive (e.g., Academic Success) and non-cognitive (e.g., Personal Characteristics) skills and abilities.

25 Copyright © 2011 Fire & Police Selection, Inc. Buy the Perfect Firefighter There are two parts to this survey. The first section asks you to spend $ to buy the perfect Entry-Level Firefighter by spending this money within two major categories: Cognitive/Academic and Personal Characteristics. The second section asks you to spend $100 within both of these categories. Section One: Distribute $100 Between Two Qualification Areas Section One: Distribute $100 Between Two Qualification Areas Cognitive/Academic Skills/Abilities (see detailed list below) ____% Personal Characteristics (see detailed list below) ____% National SurveyPart I

26 Copyright © 2011 Fire & Police Selection, Inc. National SurveyPart II Cognitive/Academic Skill/Ability $ to Spend Skill/AbilityDescription Math Basic mathematics to add, subtract, multiply, divide. Reading Ability to read, retain, recall, and comprehend written material. Verbal Communication Verbally communicate in English. Writing Writing ability including spelling, grammar, sentence structure. Map Reading Ability to read maps and mapped information. Problem-Solving Problem solving and analysis Strategic Decision-Making Decision-making ability to carry out assignments. Mechanical Ability Sufficient to operate, use, and/or repair equipment. Total must sum to $100

27 Copyright © 2011 Fire & Police Selection, Inc. National SurveyPart II Personal Characteristics $ to Spend Personal Characteristic Description Teamwork Sufficient to work and live harmoniously with co-workers and to consistently follow-through on routine duties Working Under Stress Ability to perform duties under stressful emergency situations. Allegiance/Loyalty Have allegiance to departments core values and mission. Truthfulness/Integrity demonstrate truthfulness/integrity to self and others even in the face of severe consequences. Public Relations Willing to be resourceful, goal-oriented, and proactive in bringing tasks to completion. Emotional Stability Willing to remain calm under pressure or heavy workloads, not easily given to hostility, anxiousness, or vulnerability. Sensitivity Ability to maintain respect for others emotional needs following a crisis. Proactive/Goal-Oriented Represent public relations and public perception of departments image. Thoroughness/Attention to Detail Thoroughness and attention to detail in performing all duties. Following Orders Receives and responds to orders or instructions from superiors. Total must sum to $100

28 Copyright © 2011 Fire & Police Selection, Inc. U.S. Fire Department Survey Results Personal Characteristics 55% Cognitive/Academic Skills & Abilities 45% National Fire Select Test Cognitive/Academic Component Strategic Decision- Making 14% Problem Solving 14% Map Reading 10% Writing Ability 15% Reading Ability 18% Mathematical Reasoning 15% Mechanical Ability 14% Personal Characteristics Component Thoroughness/ Attention To Detail 9% Public Relations 9% Sensitivity 8% Emotional Stability 10% Proactive/ Goal-Oriented 8% Truthfulness/ Integrity 13% Allegiance/ Loyalty 10% Working Under Stress 10% Teamwork 12% Following Orders 11%

29 Copyright © 2011 Fire & Police Selection, Inc. U.S. Fire Department Survey Results Test Section # of Items Weighted Value Cognitive/ Academic Skills 5545% Personal Characteristics 2643% Work Styles Inventory8512%

30 Copyright © 2011 Fire & Police Selection, Inc. Cutoff Score Determination 12 Fire Chiefs reviewed the academic (cognitive) test items and were asked to identify the difficulty level. The difficulty levels of the remaining cognitive and non-cognitive test items, which were validated using criterion-related validity, were calculated from the original validation studies. 12 Fire Chiefs reviewed the academic (cognitive) test items and were asked to identify the difficulty level. The difficulty levels of the remaining cognitive and non-cognitive test items, which were validated using criterion-related validity, were calculated from the original validation studies. These score bands were derived using the composite Standard Error of Difference (SED) from the validation sample scores (N = 728). The composite SED is calculated by taking the composite SEM and multiplying it by the square root of 2. Band A was calculated by taking the highest score received on the NFST validation administration and reducing it by one composite SED. Band B was calculated by taking the next highest score below the lowest score in Band A and reducing it by one composite SED. This process was continued until the composite adjusted critical score was reached (e.g., a % on the test). The SED was used to create these score bands because it effectively provides meaningful score breaks that represent different levels of the attributes measured. If ranking is used for the NFST, FPSI recommends utilizing the statistically derived bands shown in Table 1 (rather than top-down ranking). These score bands were derived using the composite Standard Error of Difference (SED) from the validation sample scores (N = 728). The composite SED is calculated by taking the composite SEM and multiplying it by the square root of 2. Band A was calculated by taking the highest score received on the NFST validation administration and reducing it by one composite SED. Band B was calculated by taking the next highest score below the lowest score in Band A and reducing it by one composite SED. This process was continued until the composite adjusted critical score was reached (e.g., a % on the test). The SED was used to create these score bands because it effectively provides meaningful score breaks that represent different levels of the attributes measured. If ranking is used for the NFST, FPSI recommends utilizing the statistically derived bands shown in Table 1 (rather than top-down ranking). Band Highest Score Lowest Score A100%86.246% B86.245%80.297% C80.296% D74.335%70.704%

31 Copyright © 2011 Fire & Police Selection, Inc. Failure to Use a Balanced Test No validity No validity Academy failure rates increase and/or job performance suffers Academy failure rates increase and/or job performance suffers Less minority representation (i.e., adverse impact) in the final eligibility list Less minority representation (i.e., adverse impact) in the final eligibility list Might hire mostly book smart but not people smart firefighters Might hire mostly book smart but not people smart firefighters Greater potential for lawsuits which almost always results in lowering standards Greater potential for lawsuits which almost always results in lowering standards

32 Copyright © 2011 Fire & Police Selection, Inc. Fire & Police Selection, Inc. Contact us at: Mailing Address Mailing Address 193 Blue Ravine Rd. Suite Blue Ravine Rd. Suite 270 Folsom, CA Folsom, CA Website Website Telephone Telephone x x


Download ppt "Fire & Police Selection, Inc. Dan Biddle, Ph.D. Stacy L. Bell, M.S. August 26, 2011 A National Fire Chief Survey Reveals the Key Ingredients to Include."

Similar presentations


Ads by Google