RTI International is a trade name of Research Triangle Institute 3040 Cornwallis Road ■ P.O. Box ■ Research Triangle Park, North Carolina, USA Phone The O*NET Data Collection Program: Improving Efficiency in a Multistage Complex Establishment Survey Marcus Berzofsky, Brandon Welch, Susan McRitchie, and Rick Williams Third International Conference on Establishment Surveys Montreal, Canada June 21, 2007

2 Outline O*NET Background and Goals Design Challenges Initial Design Introduction of Wave Design Introduction of Model Assisted Sampling Conclusions

3 O*NET Study Goals Produce nationally representative estimates on 810 occupations plus new and emerging occupations Deductive approach which utilizes a common set of prespecified items to determine the critical characteristics for each occupation Respondents are incumbents in the occupation Estimates produced for four domains Skills Work Context Work activities Knowledge

4 O*NET Background Conducted by the National Center for O*NET Development and RTI International Sponsored by the U.S. Department of Labor Data collection began in June 2001 To date Over 143,000 establishments selected Over 100,000 employee respondents

5 Design Challenges: Basic Question How do we identify incumbents in all 810 occupations?

6 Design Challenges: Simple Solution Generate list of incumbents for each occupation Will work for some occupations, but not all Lawyers – list from ABA Secretaries no such list exists

7 Design Challenges: More Complete Solution Take advantage of the likelihood of occupations to be employed within the same industries and, hence, establishments Conduct general population survey of establishments and then sample incumbents within selected establishments

8 Design Challenges: Challenge of Linking Occupations to Industries How to determine which occupations are found in which industries?

9 Design Challenges: Solution to Linking Occupations to Industries U.S. Bureau of Labor Statistics (BLS) conducts the Occupational Employment Statistics (OES) survey Obtains estimates on the number of employees in an occupation that are found in a particular industry Identifies industries that employ each occupation Dun and Bradstreet (D&B) provides a frame of establishments by industry Used D&B frame to select establishments based on information provided by OES

10 Initial Design Target Population: All non-military, non- institutionalized incumbents in the 50 United States plus DC Traditional probability based design Targeted a large set of diverse occupations for initial sample Sample size of 12,000 establishments targeting 210 occupations

11 Initial Design Two-stage cluster design First stage: Select establishments Second stage: Select employees Linked up to 10 occupations to a single establishment Used PPS random sampling to select up to 10 occupations employed in the establishment’s industry Reduces cost and number of establishments sampled by identifying more than one occupation at a time compared to sampling one occupation at a time under a list design Reduces total burden by only asking about up to 10 occupations likely employed by establishment

12 Initial Design: Selection of Establishments Establishments stratified by size (# of employees), industry groupings Establishments selected by Sequential PPS sampling

13 Initial Design: Selection of Employees Ask point of contact (POC) at establishment about occupations on list Selection of employees For occupations present, POC rosters employees SRS of employees selected from each occupation present

14 Initial Design Occupation s Select Establishments Select Employees OES/D&B Occupation-Industry linkage

15 Introduction of Wave Design: Motivation Large single sample: Inefficiently covered all occupations Limited the information available to inform future follow-up samples Was not able to adequately target all industries which meant that some occupations were not linked to many of the selected establishments

16 Introduction of Wave Design: Wave Design Split occupations into smaller sets of approximately 50 occupations Released sample in sub-waves Used information from prior sub-waves to inform sample allocation of future sub-waves Smaller sample size per sub-wave (approx. 3,000 establishments) More easily allowed sample to be allocated to more difficult to find occupations

17 Introduction of Wave Design: Clustering of Occupations Desired smaller set of occupations to contain occupations found in common industries Cluster analysis conducted to determine the most efficient manner to group occupations Clustered occupations by the distribution of industries associated with occupation from OES survey Reduced the dimensionality by excluding industries that contained less than 1% of the occupation for all occupations Uses SAS clustering procedures to group occupations Combined groups of occupations into sets of approximately 50 occupations

18 Wave Design: Targeting Industries and Coverage OES lists all industries in which an occupation is found Initial targeting of all occupations leads to a more efficient sample Coverage requirements ensure that all industry groupings that employ an occupation are sampled Based on the cluster of occupations in a wave, industries were targeted based on their likelihood of finding the occupations Substantive experts (I/O psychologists) assign concentration level to each industry to which occupation is linked Industries with a higher level concentration of the occupations were given a greater chance of selection

19 Wave Design: Targeting Industries and Coverage Coverage analysis indicated that coverage requirements could be relaxed from a high coverage level while not introducing additional coverage bias and simultaneously reducing costs by eliminating occupations with little likelihood of being found in the establishment’s industry

20 Wave Design: Burden Two types of burden on O*NET Establishment burden:  Time spent by POC at establishment determining if occupations are employed by establishment  Rostering employees as needed by POC  Passing out questionnaires to selected employees Employee burden:  Time spent by employee completing questionnaire (approx. 30 min)

21 Wave Design: Burden Methods used to minimize burden Establishment burden:  POC asked to roster only the first five occupations found at the establishment  Selected establishments (physical location) not eligible to be selected again for 12 months  Selection algorithm  Select no more than 20 employees from an establishment  Select no more than 8 employees per occupation Employee burden  Only asked respondents to complete one domain questionnaire (skills, work context, knowledge, or work activities)

22 Wave Design Occupations Wave Wave n Select Employees Select Estabs Targeting industries Cluster Analysis Select Employees Sample Size Completed ?

23 Model Assisted Sampling: Motivation Level of effort to obtain desired number of questionnaires varied greatly across occupations E.g., Secondary school teachers were easy to find and responded at very high levels E.g., Roustabouts are more difficult to identify and respond at a lower level Wanted method to control level of effort while ensuring that employee sample was representative of the occupation

24 Model Assisted Sampling: Definition Uses known information about the occupation to model how it is distributed by Census region (4 domains) Establishment size (4 domains) Industry division (12 domains) Sampling of an occupation is completed only after minimum targets are met for each of the occupation’s domains

25 Model Assisted Sampling: Application Sample yield monitored by domain Employee sample selection ceased for an occupation in domains where model parameters have been achieved Future sub-wave sample establishment selection eliminated occupations in completed domains Analysis indicated that estimates under MAS are not substantively different than estimates produced under traditional sampling paradigm

26 Model Assisted Sampling Occupations Wave Wave n Controlling Sample Allocation Select Employees Define MAS distribution Select Estabs Targeting industries Cluster Analysis Define MAS distribution Select Employees Controlling Sample Allocation MAS Domains Complet e?

27 Conclusions Multi-year studies often require modifications over time O*NET illustrates how relatively small changes can greatly improve the efficiency of a sample Often helpful to test or simulate changes to determine their impact on a study

28 Questions?