Presentation is loading. Please wait.

Presentation is loading. Please wait.

Leveraging Technology in the Selection Process Alfredo Rocha and Eric Rowen Personnel Commission Los Angeles County Office of Education Presented to: PTC-SC,

Similar presentations


Presentation on theme: "Leveraging Technology in the Selection Process Alfredo Rocha and Eric Rowen Personnel Commission Los Angeles County Office of Education Presented to: PTC-SC,"— Presentation transcript:

1 Leveraging Technology in the Selection Process Alfredo Rocha and Eric Rowen Personnel Commission Los Angeles County Office of Education Presented to: PTC-SC, February 25, 2015

2 Current Process Improvements 1. Interview Rater Briefing Video Standardizes briefing process Length: 12 minutes

3 Current Process Improvements 1. Interview Rater Briefing Video Standardizes briefing process Length: 12 minutes 2. Video-Rated Interviews (VRI) Reduces length of interview schedule Increases control and standardization 3. Written Exam Pass Point Excel Workbook (RQS) 4. Rater App

4 RQS Version 1 Written Exam Pass Point Excel Workbook Automated statistical information − Measures of central tendency − Measures of variability − Percentile scores − Absolute floor − Correlation coefficients − Data-driven histograms Pass Point Survival tool Adverse Impact Analysis tool

5 RQS Version 1 Disclaimer This workbook is provided as a professional courtesy Any use in an operational context is solely the responsibility of the user agency Unknown “bugs” or deficiencies may exist or develop with continued use LACOE makes no performance assurance or guarantee for this Excel-based tool

6 RQS Version 1 Candidate Results

7 RQS Version 1

8 16 17 12 45 12 14 6 32 14 19 52 14 1817 49 15 1918 52 13 18 16 47 Candidate 1 Candidate 2 3 4 5 6

9 RQS Version 1 Test Characteristics

10 RQS Version 1 15 20 5 Test Characteristics

11 RQS Version 1 Section/Test Reliability KR-20 Cronbach’s Alpha Test Characteristics

12 RQS Version 1 Section/Test Reliability KR-20 Cronbach’s Alpha Test Characteristics

13 RQS Version 1 Exam Statistics

14 RQS Version 1 Exam Statistics Math ReadingSafety Overall

15 RQS Version 1 Exam Statistics Math ReadingSafety Overall

16 RQS Version 1 Exam Statistics Math ReadingSafety Overall

17 RQS Version 1 Exam Statistics Absolute floor can rule out someone passing by just guessing! Math ReadingSafety Overall

18 RQS Version 1 Correlations

19 RQS Version 1 Correlation Coefficients

20 RQS Version 1 Correlation Coefficients 0 - 0.29 = 0.3 - 0.49 = 0.5 - 0.69 = 0.7 - 1 = Relationship Strength None – Weak Moderate Strong Very strong – Perfect

21 RQS Version 1 Data-Driven Histograms Normal Distribution Skewed Distribution Bimodal Distribution z Look at the distribution of scores to establish norm-referenced pass points.

22 RQS Version 1 Angoff Method Can use SMEs to establish criterion-referenced pass points !

23 RQS Version 1 Pass Point Survival

24 RQS Version 1 Pass Point Survival Math ReadingSafety Overall

25 RQS Version 1 Pass Point Survival

26 RQS Version 1 Pass Point Survival

27 RQS Version 1 Pass Point Survival

28 RQS Version 1 Adverse Impact Analysis

29 RQS Version 1 Adverse Impact Analysis Example #1: Out of 26 males taking an exam, 16 pass. Out of 19 females, 14 pass.

30 RQS Version 1 Adverse Impact Analysis Example #1: Out of 26 males taking an exam, 16 pass. Out of 19 females, 14 pass. Example #2: White : 4 pass, 2 fail Black : 2 pass, 10 fail Hispanic : 3 pass, 8 fail Asian : 8 pass, 3 fail Native American : 2 pass, 4 fail

31 RQS Version 1 Adverse Impact Analysis Example #2: White : 4 pass, 2 fail Black : 2 pass, 10 fail Hispanic : 3 pass, 8 fail Asian : 8 pass, 3 fail Native American : 2 pass, 4 fail What if another group had a higher selection ratio? Instead of 8 passing, only 5 had passed?

32 RQS Version 1 Adverse Impact Analysis Example #2: White : 4 pass, 2 fail Black : 2 pass, 10 fail Hispanic : 3 pass, 8 fail Asian : 8 pass, 3 fail Native American : 2 pass, 4 fail What if another group had a higher selection ratio? This results in different calculations

33 RQS Version 1 Limitations Not all encompassing…yet Allows for only 5 test sections and 999 candidates May require scrolling to view RQS sections Data-driven histograms may not work well with some types of tests Possible issues with workbook after continued use (Keeping a master copy is necessary!) All features may not be supported by previous versions of Excel User feedback is welcome!

34 Rater App Microsoft Access database application Used with any competency-based examination (T&E, Performance, Writing Projects, Interviews, etc.) Goal: Simple, yet powerful tool to collect & store ratings Created by Eric Rowen & LACOE PC staff, 2014

35 Microsoft Access Basics Initial Challenges Learning how Access works Finding relationships between Access objects and how data links together Invaluable resources – GCFLearnFree.org GCFLearnFree.org – Microsoft Support Microsoft Support – MSDN MSDN – Stack Overflow Stack Overflow – YouTube Image Credit to GCF LearnFree.org

36 Microsoft Access Basics (cont.) Most difficult parts: setting table relationships and queries for the data Image Credit to GCF LearnFree.org

37 Database Design

38 Development Goals Minimize preparation and output time – Also remove potential transfer errors from the process Increased access to information for Analysts Reduce paper use in administration and archiving Application Limitations Limited to use with LACOE rating procedures – Comp Model 3G, Rating Scale, Weighted Sum Scores, NEOGOV Designed for use on our servers & hardware NOT available to share (yet) – contact our director

39 The Rater App Demonstration

40 LACOE Experience Takeaways Consistently positive feedback Trouble with user error and network connections – Many issues should be fixed with new hardware Requires agency-wide training (analysts & specialists) Continuous improvement is essential – ALWAYS adding new features based on user feedback – Application bugs are inevitable Multiple administrators if possible

41 Thank you! Questions? Connect with us on !


Download ppt "Leveraging Technology in the Selection Process Alfredo Rocha and Eric Rowen Personnel Commission Los Angeles County Office of Education Presented to: PTC-SC,"

Similar presentations


Ads by Google