Presentation is loading. Please wait.

Presentation is loading. Please wait.

Whats Up DOT? IARP Occupational Database Committee IARP Forensic Conference Hyatt Regency Bonaventure Weston, Florida October 31, 2008.

Similar presentations


Presentation on theme: "Whats Up DOT? IARP Occupational Database Committee IARP Forensic Conference Hyatt Regency Bonaventure Weston, Florida October 31, 2008."— Presentation transcript:

1 Whats Up DOT? IARP Occupational Database Committee IARP Forensic Conference Hyatt Regency Bonaventure Weston, Florida October 31, 2008

2 Scope of Project IODC formed spring 2007 IODC formed spring 2007 Purpose: Purpose: Identify replacement of DOTIdentify replacement of DOT Focus: Focus: research past efforts to update DOTresearch past efforts to update DOT current uses of DOT & O*NETcurrent uses of DOT & O*NET identify & evaluate other occupational databasesidentify & evaluate other occupational databases develop support to update/replace DOTdevelop support to update/replace DOT

3 Scope of Project Identified two occupational databases: eDOT, a product of Economic Research Institute; McDOT, a product of Vocationology, Inc. Identified two occupational databases: eDOT, a product of Economic Research Institute; McDOT, a product of Vocationology, Inc. Third database, based on the Common Metric Questionnaire (CMQ) may be ready in few years Third database, based on the Common Metric Questionnaire (CMQ) may be ready in few years Arranged to have training and use of these databases for study purposes. Arranged to have training and use of these databases for study purposes. First, needed to learn more about DOT First, needed to learn more about DOT

4 History of the DOT Following Civil War, U.S. and Census Bureau began to classify occupations Following Civil War, U.S. and Census Bureau began to classify occupations First occupational dictionary: 1927, A Dictionary of Occupational Terms, Great Britain Ministry of Labour First occupational dictionary: 1927, A Dictionary of Occupational Terms, Great Britain Ministry of Labour Wagner Peyser Act, 1933: United States Employment Service (USES)-- match workers with jobs. Wagner Peyser Act, 1933: United States Employment Service (USES)-- match workers with jobs.

5 History of the DOT Work on the DOT began around the same time; 1 st Edition 1939 Work on the DOT began around the same time; 1 st Edition 1939 Subsequent editions published 1949, 1965, 1977 Subsequent editions published 1949, 1965, 1977 Supplements came out 1982, 86, 91 Supplements came out 1982, 86, 91 20 new occupations added in 1998 20 new occupations added in 1998 Intended purpose: to assist USES in placing workers in jobs. Intended purpose: to assist USES in placing workers in jobs. 1 st Edition: skilled, semi-skilled or unskilled 1 st Edition: skilled, semi-skilled or unskilled

6 History of the DOT 1965 edition: worker traits, worker functions (Data, People, Things), Occupational Group Arrangements 1965 edition: worker traits, worker functions (Data, People, Things), Occupational Group Arrangements 1977 edition: ~75,000 job analyses; 2,100+ new occupations; eliminated gender bias in job titles and descriptions; included detailed worker characteristics (SCO) 1977 edition: ~75,000 job analyses; 2,100+ new occupations; eliminated gender bias in job titles and descriptions; included detailed worker characteristics (SCO) 1986 ed. added 761 new occupations 1986 ed. added 761 new occupations 1991 added 844, deleted 208; added GOE codes, GED, SVP, DLU 1991 added 844, deleted 208; added GOE codes, GED, SVP, DLU

7 History of the DOT Each definition includes: code, title, industry designation, alternate titles, body of definition, undefined related titles, and definition trailer (GOE, strength, GED-RML, SVP, DLU) Each definition includes: code, title, industry designation, alternate titles, body of definition, undefined related titles, and definition trailer (GOE, strength, GED-RML, SVP, DLU) Basic concepts described in the DOT occupational definitions: element, task, position, job, and occupation. Basic concepts described in the DOT occupational definitions: element, task, position, job, and occupation.

8 History of the DOT Major elements of an occupation: Major elements of an occupation: What the worker does (data, people, things)What the worker does (data, people, things) What gets done (work fields-WF)What gets done (work fields-WF) Skill level to perform this work (SVP)Skill level to perform this work (SVP) End product (materials, products, subject matter and services-MPSMS)End product (materials, products, subject matter and services-MPSMS)

9 History of the DOT DOT 4 classification systems: DOT 4 classification systems: by job content (OGA)by job content (OGA) by worker function (DPT)by worker function (DPT) by industry affiliation (Industry Designation)by industry affiliation (Industry Designation) alphabetically by titlealphabetically by title

10 History of the DOT Job analysis basis of the DOT Job analysis basis of the DOT National Research Council (1999) defines occupational [job] analysis: "the tools and methods used to describe and label work, positions, jobs and occupations" National Research Council (1999) defines occupational [job] analysis: "the tools and methods used to describe and label work, positions, jobs and occupations" DOT defines an occupation as: a collective description of individual jobs performed, with minor variations, in many establishments DOT defines an occupation as: a collective description of individual jobs performed, with minor variations, in many establishments

11 History of the DOT The Handbook for Analyzing Jobs job analysis methodology used in creation of DOT. The Handbook for Analyzing Jobs job analysis methodology used in creation of DOT. HAJ first published in 1944; 4 th Edition in 1991 (RHAJ). HAJ first published in 1944; 4 th Edition in 1991 (RHAJ). 1972 Edition published halfway through completion of 1977 DOT. 1972 Edition published halfway through completion of 1977 DOT. Two primary formats in job analysis: work- orientedwhat gets done; and worker- orientedwhat the worker does. HAJ/RHAJ uses both formats Two primary formats in job analysis: work- orientedwhat gets done; and worker- orientedwhat the worker does. HAJ/RHAJ uses both formats

12 History of the DOT The Selected Characteristics of Occupations Defined in the Revised Dictionary of Occupational Titles (1993) The Selected Characteristics of Occupations Defined in the Revised Dictionary of Occupational Titles (1993) 1st edition 1966, updates in 1968, 1981, and 1993. 1st edition 1966, updates in 1968, 1981, and 1993. Includes DOT titles arranged by: GOE; SVP; strength; physical demands; environmental conditions; index of titles with DOT codes; definitions of the worker traits and functions Includes DOT titles arranged by: GOE; SVP; strength; physical demands; environmental conditions; index of titles with DOT codes; definitions of the worker traits and functions

13 History of the DOT Positive factors of the DOT: Positive factors of the DOT: use of skilled job analystsuse of skilled job analysts task level descriptionstask level descriptions foundation built upon data people thingsfoundation built upon data people things Worker traits, characteristicsWorker traits, characteristics Useful for TSA: MPSMS, WF, SVPUseful for TSA: MPSMS, WF, SVP Attempt to cover national economyAttempt to cover national economy Depth of informationDepth of information

14 Development of the DOT Sample for DOT: all jobs in the US economy Sample for DOT: all jobs in the US economy County Business Patterns/Thomas Registry used to identify # establishments in each industry. County Business Patterns/Thomas Registry used to identify # establishments in each industry. Industries assigned to one of 11 field centers average 42 industries/field center Industries assigned to one of 11 field centers average 42 industries/field center any industry designation given to one field office any industry designation given to one field office

15 Development of the DOT Each analyst identified which establishments to contact Each analyst identified which establishments to contact Attempt to include one small, one medium and one large size establishment for each industry, and to focus on new and emerging occupations. Attempt to include one small, one medium and one large size establishment for each industry, and to focus on new and emerging occupations. No clear supervision during process. No clear supervision during process. Appeared primary criteria for selecting establishment proximity to field office Appeared primary criteria for selecting establishment proximity to field office Employers: right of refusal; no incentives Employers: right of refusal; no incentives

16 Development of the DOT Employers controlled which jobs were analyzed, and which employees were chosen for analysis. Employers controlled which jobs were analyzed, and which employees were chosen for analysis. Analyst chose which jobs to observe. Analyst chose which jobs to observe. If job analyzed for the 3 rd Edition, may complete abbreviated analysis or none at all. If job analyzed for the 3 rd Edition, may complete abbreviated analysis or none at all. If another analyst had recently completed a similar analysis, none would be done. If another analyst had recently completed a similar analysis, none would be done. Observe 1-2 workers per job. Observe 1-2 workers per job.

17 Development of the DOT Variety methods to record data: HAJ format; abbreviated format; or simple notes. Variety methods to record data: HAJ format; abbreviated format; or simple notes. Analysts not allowed to bring in tools or equipment to measure certain aspects of jobs estimation and observation only. Analysts not allowed to bring in tools or equipment to measure certain aspects of jobs estimation and observation only. Methods for rating worker traits were vague particularly for aptitudes, temperaments and interests. Methods for rating worker traits were vague particularly for aptitudes, temperaments and interests.

18 Updates of the DOT Much time has passed since any onsite job analyses of DOT occupations Much time has passed since any onsite job analyses of DOT occupations Date last analyzed N%Age 1977 and previous 10,28880.6 30 years 19783552.8 1979-19825204.1 >25 years 1984-19881,1879.3 >20 years 1989-19913903.1 >15 years 1993200.2 15 years

19 How the DOT is used Basis for transferable skills analysis Basis for transferable skills analysis Primary uses: career & voc counseling, SSD Primary uses: career & voc counseling, SSD Secondary uses: library reference, personnel management, employee placement, govt. uses, research, others Secondary uses: library reference, personnel management, employee placement, govt. uses, research, others

20 O*NET Outgrowth of the Advisory Panel on the DOT (APDOT) from the early 1990s Outgrowth of the Advisory Panel on the DOT (APDOT) from the early 1990s Dept. of Labor replacement for the DOT Dept. of Labor replacement for the DOT Skills-based database rather than task-based dictionary Skills-based database rather than task-based dictionary Preliminary version released Dec. 1997 with first version (O*NET 98) out Dec. 1998 Preliminary version released Dec. 1997 with first version (O*NET 98) out Dec. 1998

21 O*NET Utilized SOC coding rather than DOT Utilized SOC coding rather than DOT Current version is O*NET 13 Current version is O*NET 13 The 12, 761 DOT occupations have been aggregated/collapsed to 812 groups The 12, 761 DOT occupations have been aggregated/collapsed to 812 groups Composite information from many jobs; not intended to describe a particular job Composite information from many jobs; not intended to describe a particular job Uses mean data rather than mode used by DOT Uses mean data rather than mode used by DOT

22 Position Analysis Questionnaire (PAQ) PAQs business: create useful work and labor economic reports PAQs business: create useful work and labor economic reports Structured job analysis; 187 items (plus 8 items re: comp.) called job elements Structured job analysis; 187 items (plus 8 items re: comp.) called job elements Job elements are worker-oriented;6 domains: information input; work output (physical activities and tools); mental processes; relationships with others; job context (physical and social environment); and other work characteristics (such as pace and structure) Job elements are worker-oriented;6 domains: information input; work output (physical activities and tools); mental processes; relationships with others; job context (physical and social environment); and other work characteristics (such as pace and structure)

23 PAQ PAQ dedicated solely to job analysis services since 1972 PAQ dedicated solely to job analysis services since 1972 Terminology, definitions and rating scales different than that used in DOT Terminology, definitions and rating scales different than that used in DOT Requires post-college graduate reading level Requires post-college graduate reading level Job analysts or supervisors usually complete PAQ Job analysts or supervisors usually complete PAQ

24 PAQ PAQ trains subscriber employers in job analysis techniquesin person and online PAQ trains subscriber employers in job analysis techniquesin person and online Limited coverage of managerial, supervisory, executive and professional work led to development of separate questionnaire Limited coverage of managerial, supervisory, executive and professional work led to development of separate questionnaire Average reliabilitiesitem and re-raterange.68-.80 Average reliabilitiesitem and re-raterange.68-.80

25 PAQ/DAQ DAQ: Disability Analysis Questionnaire. Developed by PAQ DAQ: Disability Analysis Questionnaire. Developed by PAQ Includes 99 questions from the PAQ that are most closely related to items from the DOT/HAJ Includes 99 questions from the PAQ that are most closely related to items from the DOT/HAJ ERI modified DAQ slightlymade certain the questions best-matched those of the DOT ERI modified DAQ slightlymade certain the questions best-matched those of the DOT The DAQ questions are the ones asked of incumbents when visiting websites The DAQ questions are the ones asked of incumbents when visiting websites

26 eDOT The enhanced DOT -- database and software program developed by Economic Research Institute (ERI) The enhanced DOT -- database and software program developed by Economic Research Institute (ERI) ERI began as compensation information provider in 1987 ERI began as compensation information provider in 1987 Started the eDOT Skills Project in 2002 to collect data and update the DOT Started the eDOT Skills Project in 2002 to collect data and update the DOT In 2004, ERI purchased PAQ; PAQ operates eDOT Skills Project under a license with ERI In 2004, ERI purchased PAQ; PAQ operates eDOT Skills Project under a license with ERI Database has two parts: archived DOT and new eDOT which includes the old DOT Database has two parts: archived DOT and new eDOT which includes the old DOT

27 eDOT Includes 64 SCO characteristics + 35 new factors such as keyboarding, education, mental and cognitive factors, etc. Includes 64 SCO characteristics + 35 new factors such as keyboarding, education, mental and cognitive factors, etc. Includes 20 measures from SSAs Mental Residual Capacity paper Includes 20 measures from SSAs Mental Residual Capacity paper Various people contribute data: subject matter experts (job analysts trained in the use of the PAQ; voc rehab counselors; major disability carriers; workers compensation analysts; and primarily incumbents visiting websites Career Builder, SalaryExpert, SalariesReview. Various people contribute data: subject matter experts (job analysts trained in the use of the PAQ; voc rehab counselors; major disability carriers; workers compensation analysts; and primarily incumbents visiting websites Career Builder, SalaryExpert, SalariesReview.

28 eDOT Each data point has associated reliability, standard error and deviation calculated Each data point has associated reliability, standard error and deviation calculated 1,000-1,500 job analyses done per year, including those by incumbents 1,000-1,500 job analyses done per year, including those by incumbents Over 1 million PAQ job analyses included in eDOT (completed since 1974) Over 1 million PAQ job analyses included in eDOT (completed since 1974) Sample is one of convenience Sample is one of convenience Control for sampling error by using multiple sources of data Control for sampling error by using multiple sources of data

29 eDOT Differences between eDOT and DOT Differences between eDOT and DOT Different rating scales, definitions, termsDifferent rating scales, definitions, terms eDOT uses revised version of SIC which it developed, called eSICeDOT uses revised version of SIC which it developed, called eSIC eDOTfluid database; DOTfixedeDOTfluid database; DOTfixed eDOTconvenience sample; DOTattempt to capture all jobs in the national economyeDOTconvenience sample; DOTattempt to capture all jobs in the national economy Electronic v. paperElectronic v. paper PAQ interviewing worker, not always observing; DOTobservationPAQ interviewing worker, not always observing; DOTobservation

30 eDOT Examples of new jobs added to eDOT Examples of new jobs added to eDOT computer sys admin030.162-500 call center rep299.357-201 asst. mgr retail store185.167-505 executive asst169.167-911 sales assistant209.562-800 maint. helper806.687-011 CAD/CAM Tech017.262-519

31 eDOT ERI/PAQ masks jobs ERI/PAQ masks jobs Criteria for exclusion: Criteria for exclusion: job analyses over 15 years oldjob analyses over 15 years old face validity (abalone diver)face validity (abalone diver) 10 or less requests for the job10 or less requests for the job not listed on job boards or salary surveysnot listed on job boards or salary surveys not mentioned on any loan applicationsnot mentioned on any loan applications not on any labor/proxy/form 990not on any labor/proxy/form 990 not mentioned on Calif. state work compnot mentioned on Calif. state work comp

32 eDOT Examples of masked jobs in eDOT: Examples of masked jobs in eDOT: animal breeder410.161-010 military recruiter166.267-026 pres., financial inst.186.117-054 police commissioner188.117-118 feed blender520.685-094 collator operator208.685-010

33 eDOT As of January 2008, ERI had As of January 2008, ERI had Added 717 new occupationsAdded 717 new occupations Removed 4,103 occupations (no/low frequency)Removed 4,103 occupations (no/low frequency) Verified the existence of 8,658 occupationsVerified the existence of 8,658 occupations Total of 9,375 occupations in eDOT, compared to 12,761 in the DOT and 812 in O*NETTotal of 9,375 occupations in eDOT, compared to 12,761 in the DOT and 812 in O*NET As of July 2008, have added WF and MPSMS to all jobs in eDOTAs of July 2008, have added WF and MPSMS to all jobs in eDOT

34 McDOT History McCroskey Dictionary of Occupational Titles (McDOT) McCroskey Dictionary of Occupational Titles (McDOT) Part of the McCroskey Vocational Quotient System (MVQS) Part of the McCroskey Vocational Quotient System (MVQS) Based on VDARE Based on VDARE McDOT includes both the old DOT and O*NET. McDOT includes both the old DOT and O*NET. Data has been fused from both sources, identifying 24 most significant worker traits Data has been fused from both sources, identifying 24 most significant worker traits

35 UNUM: Project with eDOT Methodology for selecting eDOT Methodology for selecting eDOT Findings Findings Future Directions Future Directions

36

37 Issues with DOT Miller et al. Study 1980 Work, Jobs and Occupations: A Critical Review of the Dictionary of Occupational Titles Work, Jobs and Occupations: A Critical Review of the Dictionary of Occupational Titles Comprehensive review & evaluation of the 1977, 4 th Ed. DOT, conducted on behalf of the National Research Council, at the request of the Department of Labor Comprehensive review & evaluation of the 1977, 4 th Ed. DOT, conducted on behalf of the National Research Council, at the request of the Department of Labor

38 Issues with DOT Miller et al. Study 1980 The comprehensiveness, reliability and accuracy of the DOT are in large part a function of the data collection and analysis procedures used to produce it. p.114 The comprehensiveness, reliability and accuracy of the DOT are in large part a function of the data collection and analysis procedures used to produce it. p.114

39 Issues with DOT Miller et al. Study 1980 Data Collection/Methodology Issues: Data Collection/Methodology Issues: Lack of written procedures on how DOT produceLack of written procedures on how DOT produce Majority of principles used to create DOT were established in 1939 & 1949Majority of principles used to create DOT were established in 1939 & 1949 Sampling plan complicated and indirect, did not include all jobs in US economySampling plan complicated and indirect, did not include all jobs in US economy

40 Issues with DOT Miller et al. Study 1980 Data Collection/Methodology Issues: Data Collection/Methodology Issues: Heavy orientation to manufacturing sectorHeavy orientation to manufacturing sector Limited review of jobs requiring cognitive skills over physical skillsLimited review of jobs requiring cognitive skills over physical skills Inadequate update scheduleInadequate update schedule

41 Issues with DOT Miller et al. Study 1980 Found analysts often unable to provide concrete explanation for how they rated worker traits besides subjective means or past experience Found analysts often unable to provide concrete explanation for how they rated worker traits besides subjective means or past experience Some occupations not reviewed at all; others reviewed excessively: material handler 652 job analyses Some occupations not reviewed at all; others reviewed excessively: material handler 652 job analyses 1979 study random job titles found 81% of 4 th Ed. descriptions exactly the same as the 3 rd Edition 1979 study random job titles found 81% of 4 th Ed. descriptions exactly the same as the 3 rd Edition

42 Issues with DOT Miller et al. Study 1980 Data Collection/Methodology Issues: Data Collection/Methodology Issues: 16% occupational descriptions completed without a single job analysis16% occupational descriptions completed without a single job analysis 29% based on one job analysis29% based on one job analysis 19% based on two job analyses19% based on two job analyses 36% based on three or more analyses36% based on three or more analyses

43 Issues with DOT Miller et al. Study 1980 Reliability/Validity Issues: Reliability/Validity Issues: Validity: measurements of worker traits/functions found to be fairly unreliable: vague and ambiguously defined. Not readily apparent what the variables are intended to measure (p. 164)Validity: measurements of worker traits/functions found to be fairly unreliable: vague and ambiguously defined. Not readily apparent what the variables are intended to measure (p. 164) Worker traits/functions developed in the 1950s questionable validity for todays labor marketWorker traits/functions developed in the 1950s questionable validity for todays labor market Question whether GED and SVP measure prestige or social status of occupationsQuestion whether GED and SVP measure prestige or social status of occupations

44 Issues with DOT Miller et al. Study 1980 Reliability/Validity Issues: Reliability/Validity Issues: GED scale validated against curriculum content, not validated for occupational performanceGED scale validated against curriculum content, not validated for occupational performance Working conditions & physical demands appear to be developed for unskilled, physical jobsWorking conditions & physical demands appear to be developed for unskilled, physical jobs Reliability: items are scored subjectively; raters had trouble assigning some factors, particularly SVP and aptitudesReliability: items are scored subjectively; raters had trouble assigning some factors, particularly SVP and aptitudes Reliabilities higher for manufacturing jobs than for service jobsReliabilities higher for manufacturing jobs than for service jobs

45 Issues with DOT Cain et al. (1983) study on reliabilities of different ratings. Cain et al. (1983) study on reliabilities of different ratings. Two job descriptions per 24 occupations; experienced analysts rated the factorsTwo job descriptions per 24 occupations; experienced analysts rated the factors Found acceptable reliabilities for: data, people, GED reasoning, GED language and SVPFound acceptable reliabilities for: data, people, GED reasoning, GED language and SVP Modest reliabilities for things, GED math, strength factors and locationModest reliabilities for things, GED math, strength factors and location Reliabilities higher for manufacturing jobsReliabilities higher for manufacturing jobs

46 Issues with DOT Botterbusch (1993): DOT weaknesses: Botterbusch (1993): DOT weaknesses: Data people things not actually a hierarchydata is, people is not and things is two hierarchiesData people things not actually a hierarchydata is, people is not and things is two hierarchies GED not directly related to educationGED not directly related to education SVP does not distinguish between formal and informal trainingSVP does not distinguish between formal and informal training Difficulty defining, using and defending temperamentsDifficulty defining, using and defending temperaments Too many titlesToo many titles

47 Issues with DOT Dunn & Growick (2000): weaknesses: Dunn & Growick (2000): weaknesses: Failure to include variables such as org. setting and worker responsibility levelFailure to include variables such as org. setting and worker responsibility level Low reliability of worker traitsLow reliability of worker traits Redundant classification systemRedundant classification system Skills that are psychometrically questionable, such as GED and SVPSkills that are psychometrically questionable, such as GED and SVP Ambiguous definitions of worker traitsAmbiguous definitions of worker traits Questionable validity of worker traitsQuestionable validity of worker traits

48 Issues with DOT Harvey (Fine et al. 2004) addressed weaknesses of DOT: Harvey (Fine et al. 2004) addressed weaknesses of DOT: Legal and psychometric defensibility in the use of holistic scales to rate worker traits tend to show low inter-rater reliability and low discriminant validityLegal and psychometric defensibility in the use of holistic scales to rate worker traits tend to show low inter-rater reliability and low discriminant validity Lack of reasonable update scheduleLack of reasonable update schedule

49 Issues with DOT National Research Council (1999) National Research Council (1999) Unwieldy sizeUnwieldy size Growing differences between job descriptions and jobs as they actually occur in the labor market todayGrowing differences between job descriptions and jobs as they actually occur in the labor market today Too much detail in each definitionToo much detail in each definition Jobs described by taskJobs described by task

50 Issues with DOT National Research Council cont. (1999) National Research Council cont. (1999) Lack of information on factors such as skills, abilities, and knowledge requirements that are either not collected or not reportedLack of information on factors such as skills, abilities, and knowledge requirements that are either not collected or not reported Little or no reported information on cognitive factorsLittle or no reported information on cognitive factors Expense of updatingExpense of updating Difficulty of linking it with other databasesDifficulty of linking it with other databases

51 Issues with DOT APDOT (1993): need to reinvent the DOT APDOT (1993): need to reinvent the DOT Representative of economyRepresentative of economy Develop new JA methodology: found existing ones to be lacking, particularly with cognitive skillsDevelop new JA methodology: found existing ones to be lacking, particularly with cognitive skills Create one better at job matching, skills transferCreate one better at job matching, skills transfer

52 Issues with DOT 1980 Miller et al. study issued several conclusions and a number of recommendations (p.214) Conclusion #1 – There is a strong and continuing need both within and outside the U.S. Employment Service for the kind of information provided by the Dictionary of Occupational Titles and certain other products based on it. Conclusion #2 – Substantial improvements in the procedures and products of the occupational analysis program are required in order to meet the national need for occupational information.

53 Issues with O*NET …O*NET as currently implemented arguably fails to provide (a) occupational data expressed at the appropriate level of detail and defensibility needed for many functions and (b) occupational title taxonomy that is sufficiently detailed and phrased at the desired level of analysis (Fine et al., 2004, p1). …O*NET as currently implemented arguably fails to provide (a) occupational data expressed at the appropriate level of detail and defensibility needed for many functions and (b) occupational title taxonomy that is sufficiently detailed and phrased at the desired level of analysis (Fine et al., 2004, p1). Depth of material in O*NET: telling someone which state you live in but not giving your address (ERI) Depth of material in O*NET: telling someone which state you live in but not giving your address (ERI)

54 Issues with O*NET Terminology and measurement scales are poorly defined Terminology and measurement scales are poorly defined Does not provide correct type of information on physical demands, training requirements and other information essential for TSA Does not provide correct type of information on physical demands, training requirements and other information essential for TSA Job demands difficult to measure (e.g. cognitive) Job demands difficult to measure (e.g. cognitive) Use of numerical scales over interval scales Use of numerical scales over interval scales

55 Issues with O*NET Completed by incumbents, except for abilities section Completed by incumbents, except for abilities section Problem with incomplete survey instruments Problem with incomplete survey instruments Uses mean data rather than modal data as found in the DOT Uses mean data rather than modal data as found in the DOT

56 On the + Side: O*NET Many new variables to describe work-related characteristics and behavior Many new variables to describe work-related characteristics and behavior Includes relevant aspects of work not in the DOT, i.e. work context Includes relevant aspects of work not in the DOT, i.e. work context Expanded knowledge requirements, training and preparation needs Expanded knowledge requirements, training and preparation needs

57 IODC Evaluation of eDOT ERI provided complimentary eDOT software plus two training sessions ERI provided complimentary eDOT software plus two training sessions Met with ERI staff at 2007 Forensic Conference to discuss issues/concerns Met with ERI staff at 2007 Forensic Conference to discuss issues/concerns ERI staff available to answer questions throughout process ERI staff available to answer questions throughout process Separate training provided by user of eDOT at UNUM Separate training provided by user of eDOT at UNUM Provided beta versions of OccuBrowse database with integrated eDOT Provided beta versions of OccuBrowse database with integrated eDOT

58 IODC Evaluation of eDOT Intention to update DOT positive Intention to update DOT positive Creative in data gathering processes Creative in data gathering processes Inclusion of cognitive elements Inclusion of cognitive elements Staffed with I/O psychologists Staffed with I/O psychologists Presentation of raw data and data sources for professional study Presentation of raw data and data sources for professional study Separation of physical demands Separation of physical demands

59 IODC Evaluation of eDOT New ratings gathered from convenience sample New ratings gathered from convenience sample Industries and occupations do not always match up Industries and occupations do not always match up Coding of occupations different than DOT Coding of occupations different than DOT Uses mean trait ratings rather than mode used in DOT Uses mean trait ratings rather than mode used in DOT Method for removing occupations unsatisfactory Method for removing occupations unsatisfactory

60 IODC Evaluation of eDOT Method of data aggregation to define new occupations are unclear Method of data aggregation to define new occupations are unclear Method used to translate PAQ data into DOT terms is unclear Method used to translate PAQ data into DOT terms is unclear Several listings for same job--confusing Several listings for same job--confusing Unclear and excessive number of rating scales Unclear and excessive number of rating scales Based on the flawed DOT Based on the flawed DOT

61 IODC Evaluation of McDOT IODC provided with complementary software IODC provided with complementary software One training session One training session Vocationology available for questions, issues throughout process Vocationology available for questions, issues throughout process

62 IODC Evaluation of McDOT The new job analysis information in McDOT comes directly from O*NETpositive and negative The new job analysis information in McDOT comes directly from O*NETpositive and negative Analyses of inter-correlations among various worker trait characteristics Analyses of inter-correlations among various worker trait characteristics Reliability and validity data Reliability and validity data No independent job analyses were conducted by Vocationology, Inc. No independent job analyses were conducted by Vocationology, Inc.

63 IODC Evaluation of McDOT Since O*NET was initially derived from DOT data, O*NET is a confounded data source as an update to the DOT. Since O*NET was initially derived from DOT data, O*NET is a confounded data source as an update to the DOT. Incorporates all problems of both DOT and O*NET Incorporates all problems of both DOT and O*NET SSA has reported it will not use a database with O*NET in it SSA has reported it will not use a database with O*NET in it Difficult for many to understand or explain to judges, juries, attorneys, etc. Difficult for many to understand or explain to judges, juries, attorneys, etc.

64

65 IODC Findings IODC was unable to identify and endorse a ready-to-use database that fits members needs and needs of SSA IODC was unable to identify and endorse a ready-to-use database that fits members needs and needs of SSA eDOT has potential to be that database, but many methodological problems which need to be solved before it is usable for transferable skills analysis and in court settings eDOT has potential to be that database, but many methodological problems which need to be solved before it is usable for transferable skills analysis and in court settings IODC recommends continued use of DOT until better product is available IODC recommends continued use of DOT until better product is available

66 Moving toward a better DOT Job Analysis Issues: Job Analysis Issues: Identification of an agreed-upon job analysis format: reliable & valid, standardized formatIdentification of an agreed-upon job analysis format: reliable & valid, standardized format Re-evaluation and redefinition of worker characteristics; e.g. what constitutes a physical demandRe-evaluation and redefinition of worker characteristics; e.g. what constitutes a physical demand Develop improved definitions and indicators for attributes; scales for eachDevelop improved definitions and indicators for attributes; scales for each Use of decomposed rather than holistic ratingsUse of decomposed rather than holistic ratings

67 Moving toward a better DOT Job Analysis Issues: Job Analysis Issues: Expanded physical demand ratings: sitting, standing, walking, reaching (above shoulder v. forward v. downward), vehicle operation, lift, carry, push, pull, use of controls (hands, feet), bilateral/unilateral upper extremity use, exposure to chemical agents, biological hazards, latex, animals, potential food allergens, photic triggers (seizure potential)Expanded physical demand ratings: sitting, standing, walking, reaching (above shoulder v. forward v. downward), vehicle operation, lift, carry, push, pull, use of controls (hands, feet), bilateral/unilateral upper extremity use, exposure to chemical agents, biological hazards, latex, animals, potential food allergens, photic triggers (seizure potential)

68 Moving toward a better DOT Job Analysis Issues: Job Analysis Issues: Addition of basic skills including: keyboarding, technology use (computer user, familiarity with multiple applications, train others, troubleshooting, etc.)Addition of basic skills including: keyboarding, technology use (computer user, familiarity with multiple applications, train others, troubleshooting, etc.) Addition of attributes, including cognitive requirements; deletion of othersAddition of attributes, including cognitive requirements; deletion of others

69 Moving toward a better DOT Job Analysis Issues: Job Analysis Issues: Preferred personal qualities of workers for successful performance: attitude, initiative, persistence, resourcefulness, honesty, flexibility, team orientation, communication skills, tact, organization, leadership, efficiency, versatility, reliability, quality orientation, attentiveness, etc.Preferred personal qualities of workers for successful performance: attitude, initiative, persistence, resourcefulness, honesty, flexibility, team orientation, communication skills, tact, organization, leadership, efficiency, versatility, reliability, quality orientation, attentiveness, etc.

70 Moving toward a better DOT Job Analysis Issues: Job Analysis Issues: Acceptable methods of preparation for entry into occupations: formal education, vo-tech, apprenticeship, in-service training, OJT, certification, licensure, registration, prior experienceAcceptable methods of preparation for entry into occupations: formal education, vo-tech, apprenticeship, in-service training, OJT, certification, licensure, registration, prior experience Barriers to hiring for a specific occupation: criminal history, appearance, personal hygiene habits, monocular vision, etc.Barriers to hiring for a specific occupation: criminal history, appearance, personal hygiene habits, monocular vision, etc.

71 Moving toward a better DOT Data Collection/Methodology: Data Collection/Methodology: Comprehensive, multi-level stratified sampling planComprehensive, multi-level stratified sampling plan Audit to identify where occupations are found within businesses /industries of all sizes (e.g. small employers with <25 employees, self- employment)Audit to identify where occupations are found within businesses /industries of all sizes (e.g. small employers with <25 employees, self- employment) Use only trained/qualified professionals as job analystsUse only trained/qualified professionals as job analysts

72 Moving toward a better DOT Data Collection/Methodology: Data Collection/Methodology: Determine how job analysts would be trained and retrained: what, who, where, whenDetermine how job analysts would be trained and retrained: what, who, where, when Identify whether JAs would be done solely by resurrected Field Offices, public sector or with help from private.Identify whether JAs would be done solely by resurrected Field Offices, public sector or with help from private. Improvement in technical quality of data and methodologies used, must be ongoing in order to ensure its defensibilityImprovement in technical quality of data and methodologies used, must be ongoing in order to ensure its defensibility

73 Moving toward a better DOT Data Collection/Methodology: Data Collection/Methodology: Coding methodology used must make senseCoding methodology used must make sense 4 raters produces good reliability; 4 raters should be goal4 raters produces good reliability; 4 raters should be goal Avoid use of incumbent ratings due to reliability problems with this dataAvoid use of incumbent ratings due to reliability problems with this data Online system for input of job analysisOnline system for input of job analysis

74 Moving toward a better DOT Data Collection/Methodology Data Collection/Methodology Ensure database reflects the economy/labor market with at least 90% (preferred 95%) confidence – tied to number of analyses performed and variability in results obtainedEnsure database reflects the economy/labor market with at least 90% (preferred 95%) confidence – tied to number of analyses performed and variability in results obtained Attributes must have multiple measuresAttributes must have multiple measures Appropriate scaling for each attribute and sub- measures must make senseAppropriate scaling for each attribute and sub- measures must make sense

75 Moving toward a better DOT Data Collection/Methodology Data Collection/Methodology Proper instrumentation/equipment must be available to conduct an objective, measurable rather than strictly observational job analysisProper instrumentation/equipment must be available to conduct an objective, measurable rather than strictly observational job analysis Total number and range of attributes should be limited to what an analyst can handle without diminishing quality of data being gatheredTotal number and range of attributes should be limited to what an analyst can handle without diminishing quality of data being gathered

76 Moving toward a better DOT Data Collection/Methodology Data Collection/Methodology Identify variables needed in order to do a TSA-job match; once identified, develop scales, use accepted psychometric practices this would increase reliabilityIdentify variables needed in order to do a TSA-job match; once identified, develop scales, use accepted psychometric practices this would increase reliability

77 Moving toward a better DOT Use of Data/DOT Use of Data/DOT Methodology needs to be explained in as simple terms as possible for ease of customer useMethodology needs to be explained in as simple terms as possible for ease of customer use Processes & methodologies, as well as results of data usage, must be easily explainable to ALJs/judges, juries, attorneys, etc.Processes & methodologies, as well as results of data usage, must be easily explainable to ALJs/judges, juries, attorneys, etc. Software used to access data and perform TSAs, etc., should be simplified wherever possible to reduce errors/improve understandabilitySoftware used to access data and perform TSAs, etc., should be simplified wherever possible to reduce errors/improve understandability

78 Moving toward a better DOT Use of Data/DOT Use of Data/DOT Needs to be available as stand-alone product, not solely as part of a transferable skills analysis productNeeds to be available as stand-alone product, not solely as part of a transferable skills analysis product Allow end-users to search, compare, and retrieve informationAllow end-users to search, compare, and retrieve information Development of crosswalks should be well- explainedDevelopment of crosswalks should be well- explained

79 Moving toward a better DOT Use of Data/DOT Capable of generating printed reportsCapable of generating printed reports Data available to end-users in variety of formats, including online and in print; particularly important that data not require Internet connection so it can be accessed during Social Security hearingsData available to end-users in variety of formats, including online and in print; particularly important that data not require Internet connection so it can be accessed during Social Security hearings

80 Moving toward a better DOT Use of Data/DOT Author should have vocational rehabilitation person(s) on staff for development phase and customer supportAuthor should have vocational rehabilitation person(s) on staff for development phase and customer support

81 Moving toward a better DOT Updates: Updates: Continuously updatedContinuously updated Monitor changes in labor market; reflect changes within the databaseMonitor changes in labor market; reflect changes within the database Work closely with OES Long Term Projections Survey to learn of new emerging occupationsWork closely with OES Long Term Projections Survey to learn of new emerging occupations Sufficient funding must be provided to develop improved database and ensure maintenance into the futureSufficient funding must be provided to develop improved database and ensure maintenance into the future

82 Moving toward a better DOT Integration: Integration: Method for integrating new DOT with existing occupational classification systems (SOC, O*NET, NAICS, etc.)Method for integrating new DOT with existing occupational classification systems (SOC, O*NET, NAICS, etc.) Decisions must be made on other Labor Market Information databases that should be tied into an improved occupational database to replace the D.O.T.Decisions must be made on other Labor Market Information databases that should be tied into an improved occupational database to replace the D.O.T.

83 The IODC Committee Angela Heitzman, MA, CRC, CLCP, MSCC Angela Heitzman, MA, CRC, CLCP, MSCC John Meltzer, MS, CRC, CDMS, LPC John Meltzer, MS, CRC, CDMS, LPC Sonia Paquette, OTD, OTR/L, CPE, ABVE-D Sonia Paquette, OTD, OTR/L, CPE, ABVE-D Gerald Schneck, PhD, CRC-MAC, FVE, NCC Gerald Schneck, PhD, CRC-MAC, FVE, NCC Jeffrey Truthan, MS, CVE Jeffrey Truthan, MS, CVE With assistance from Kelly Beery-Marsiano, MEd, CRC; Betty Morris, MS, CRC; G. Shannon OKelley, MEd, CRC; and Sharon Shou, CRC, LCPC, ABVE, all of UNUM With assistance from Kelly Beery-Marsiano, MEd, CRC; Betty Morris, MS, CRC; G. Shannon OKelley, MEd, CRC; and Sharon Shou, CRC, LCPC, ABVE, all of UNUM

84 References Botterbusch, K.F. (1993, Fall). Suggestions for revisions in the dictionary of occupational titles. Vocational Evaluation and Work Adjustment Bulletin, 101-110. Cain, P.S. & Green, B.F. (1983). Reliabilities of selected ratings available from the Dictionary of Occupational Titles. Journal of Applied Psychology, 68(1), 155-165. Dunn, P.L. & Growick, B.S. (2000). Transferable skills analysis in vocational rehabilitation: Historical foundations, current status, and future trends. Journal of Vocational Rehabilitation, 14, 79-87.

85 References Economic Research Institute. (April 2008). Methodology/Disclaimer: Occupational Assessor and Survey (OA/OA+) -- eDOT®, Software and Databases (Professional and Consultant Editions). Retrieved from: www.paq.com/pdf/e-dot-methodology.pdf. www.paq.com/pdf/e-dot-methodology.pdf Fine, S.A., Harvey, R.J. & Crenshaw, S.F. (2004, April). FJA strategies for addressing O*NET limitations in a post-DOT environment. In Fleischman, E.A. (Chair), Things, Data, and People: Fifty Years of a Seminal Theory. Symposium presented at the Annual Conference of the Society for Industrial and Organizational Psychology, Chicago.

86 References Mariani, M. (1999, Spring). Replace with a database: O*NET replaces the Dictionary of Occupational Titles. Occupational Outlook Quarterly, 43(1), 3-9. McCroskey, B.J. (2008). Volcano 8.0. Brooklyn Park, MN: Vocationology., Inc. Miller, A.R., Treiman, D.J., Cain, P.S. & Roos, P.A. (Eds). (1980). Work, Jobs and Occupations: A Critical Review of the Dictionary of Occupational Titles. Washington, DC: National Academy Press.

87 References National Research Council. (1999). The Changing Nature of Work: Implications for Occupational Analysis. Committee on Techniques for the Enhancement of Human Performance: Occupational Analysis, Commission on Behavioral and Social Sciences and Education. National Academy of Sciences: Washington, DC. United States Department of Labor. (1991). Dictionary of Occupational Titles (4th Ed.). Washington, DC: United States Government Printing Office. United States Department of Labor. (1991). The Revised Handbook for Analyzing Jobs. Washington, DC: United States Government Printing Office.

88 References United States Department of Labor. (1993). Selected Characteristics of Occupations Defined in the Revised Dictionary of Occupational Titles. Washington, DC: United States Government Printing Office. U.S. Department of Labor (1993). The New DOT: A Database of Occupational Titles for the 21st- Century. Advisory Panel for the Dictionary of Occupational Titles (APDOT). United States Department of Labor. (2008). The O*NET Content Model. Retrieved from http://www.onetcenter.org/dl_files/ContentMode l_DetailedDesc.pdf. http://www.onetcenter.org/dl_files/ContentMode l_DetailedDesc.pdf http://www.onetcenter.org/dl_files/ContentMode l_DetailedDesc.pdf


Download ppt "Whats Up DOT? IARP Occupational Database Committee IARP Forensic Conference Hyatt Regency Bonaventure Weston, Florida October 31, 2008."

Similar presentations


Ads by Google