1. Crowd Sourcing Tests Warren Bobrow, Ph.D. All About Performance, LLC warren@allaboutperformance.biz

2. What is Crowd Sourcing? Obtaining services or content by asking for input from an online community. Obtaining services or content by asking for input from an online community. Software code Software code KickStarter KickStarter Marketing Campaigns Marketing Campaigns Product Development Product Development

3. What Are the Potential Applications Test Questions Test Questions Survey Items Survey Items Recruitment Ideas Recruitment Ideas

4. Test Development Where appropriate for content valid tests Where appropriate for content valid tests No in-house SMEs No in-house SMEs Frequently changing content Frequently changing content Unique jobs where there is not published content Unique jobs where there is not published content In reality, it’s a very scaled up version of item banking from previous exams In reality, it’s a very scaled up version of item banking from previous exams

5. Concerns Test Security Test Security Risks are the same as with unproctored tests. Risks are the same as with unproctored tests. Perhaps less so if the test is adaptive. Perhaps less so if the test is adaptive. Quality of the Items Quality of the Items Is the online expert community better or worse at writing items than your SMEs? Is the online expert community better or worse at writing items than your SMEs? Norming Norming Averages likely to be lower, but better item statistics due to larger samples. Averages likely to be lower, but better item statistics due to larger samples. Standardization Standardization Likely to be adaptive Likely to be adaptive

6. Application Virtual software company Virtual software company Low number of SMEs and geographically spread out Low number of SMEs and geographically spread out Growing fairly rapidly Growing fairly rapidly Need for programmers and professional staff (marketing, etc) Need for programmers and professional staff (marketing, etc)

7. Outcomes Cost Cost It’s FREE (for now) It’s FREE (for now) Integrates into ATS Integrates into ATS Candidate Reaction Candidate Reaction Meets the face validity test Meets the face validity test Still some dissatisfaction as they items are multiple choice Still some dissatisfaction as they items are multiple choice

8. Test Statistics Tests are normed Tests are normed Not a presumption of a normal curve. Not a presumption of a normal curve. Master = 780-800 Expert = 700-779 Proficient = 450-699 Familiar = 300-449 Beginner = < 300

9. Observed Statistics Type of Test MeanSD Median Intercorr IT598.2162.6.32 Office639.1134.1.11 Scores are above the midpoint of the middle range (450 to 699), which indicates that there are some non-experts in the crowd. Items have large SDs, indicating that there is good variance. Not all tests are given in combination with others, but the intercorrelations appear to be low-positive.

10. Outcomes Too early for criterion related study. Too early for criterion related study. The automation and integration with the ATS is a huge time saver for a company with a small HR staff. The automation and integration with the ATS is a huge time saver for a company with a small HR staff. Have not lost any candidates to the process. Have not lost any candidates to the process. Used in combination with personality/aptitude testing and structured interviews. Used in combination with personality/aptitude testing and structured interviews.

11. What Does the Future Hold? Unlikely that this will take over content validation testing. Unlikely that this will take over content validation testing. Not enough control over the process for man organizations. Not enough control over the process for man organizations. Likely to be adopted by tech startups and small firms Likely to be adopted by tech startups and small firms Meets an important need Meets an important need Is a good cultural fit Is a good cultural fit

12. Q&A