Presentation is loading. Please wait.

Presentation is loading. Please wait.

© 2010 by ACT, Inc. All rights reserved. 1 Creating an Aligned Assessment System to Promote Work Training Readiness and Career Success Steve Robbins Vice.

Similar presentations


Presentation on theme: "© 2010 by ACT, Inc. All rights reserved. 1 Creating an Aligned Assessment System to Promote Work Training Readiness and Career Success Steve Robbins Vice."— Presentation transcript:

1 © 2010 by ACT, Inc. All rights reserved. 1 Creating an Aligned Assessment System to Promote Work Training Readiness and Career Success Steve Robbins Vice President, Research ACT, Inc. Though designed to meet a wide array of needs, all ACT programs and services have one guiding purpose: Helping people achieve education and workplace success.

2 © 2010 by ACT, Inc. All rights reserved. 2 The Challenges Create a comprehensive assessment system that predicts success in education and work Tailor assessments to critical transition points and context Understand the complexity and interplay of cognitive and noncognitive factors across time and jobs

3 © 2010 by ACT, Inc. All rights reserved. 3 Do noncognitive factors matter? What we know from the world of work: –Project A from the military –Training and work outcomes

4 © 2010 by ACT, Inc. All rights reserved. 4 Combining Personality & Cognitive Ability Tests Correlations between general cognitive ability and personality tests and measures of job performance in Project A CogPersBothCriteria Core technical proficiency General Soldiering Proficiency Effort and Leadership Personal Discipline Physical fitness and military bearing (McHenry, Hough, Toquam, Hanson, & Ashworth, 1990) Creating opportunity for incremental validity especially as criteria vary

5 © 2010 by ACT, Inc. All rights reserved. 5 Training and Work Outcomes Noncognitive factors are predictive of important job performance criteria (e.g., Rotundo & Sackett, 2002; Casillas et al., 2009) –Task Performance –Organizational Citizenship –Counterproductive Work Behaviors –Safety Noncognitive factors meet a variety of employer needs –Screening –Selection –Training

6 © 2010 by ACT, Inc. All rights reserved. 6 Validity of Cognitive and Noncognitive Measures for Training and Work Outcomes Training Outcomes Work Performance General Mental Ability.54 a.62 b Math.48 a.52 b Reading.44 a.35 b Conscientiousness.17 c.22 c Emotional Stability.10 c.11 c Notes. a Brown, Le, & Schmidt (2006). b Salgado et al. (2003). c Schmidt et al. (2007) corrected for indirect range restriction.

7 © 2010 by ACT, Inc. All rights reserved. 7 No Unified Theory of Key Characteristics However, the literature suggests that several factors are important: –Motivation, Social Engagement, and Self-Regulation –Personality characteristics (the Big Five) Role of behavior ratings vs. self-report –Different perspectives –Coaching/training –Monitoring progress

8 © 2010 by ACT, Inc. All rights reserved. 8 Common and Specific Factors Are the same characteristics (i.e., common factors) associated with educational and work outcomes? Are there unique (i.e., specific) characteristics that contribute to the prediction of educational and work outcomes? In particular, how do age and setting impact the outcomes of interest?

9 © 2010 by ACT, Inc. All rights reserved. 9 Behavioral Development (Motivation, Social Engagement, & Self-Regulation) Cognitive Development & Acquisition of Foundational Skills (Academic Learning & Achievement) Social Capital School/Work Factors Integrated Pyramid for Success Career Development (Exploration, Crystallization, Choice & Match) Allen & Robbins (2010)

10 © 2010 by ACT, Inc. All rights reserved. 10 The human side of the pyramid

11 © 2010 by ACT, Inc. All rights reserved. 11 ACT’s noncognitive programs are part of a comprehensive and integrated set of solutions, which address the three broad areas essential for academic and work success across the lifespan:  Academic achievement & foundational skills  Behavior (noncognitive)  Career planning (noncognitive) Set of Solutions

12 © 2010 by ACT, Inc. All rights reserved. 12 Achievement & Foundational Skills Grades CollegeWork EXPLORE ® PLAN ® ACT ® COMPASS ® Comprehensive & Integrated Academic & Work Solutions: Grade 6 - Work SRI and BMS SRI Behavior WorkKeys ® Performance, Talent & BMS ACT Interest Inventory Career Planning Fit College Fit

13 © 2010 by ACT, Inc. All rights reserved. 13 Creating a Comprehensive Set of Behavior (Self & Other Report) Assessments Role of meta-analysis and validity generalization –Robbins et al. (2004). Do psychosocial and study skill factors predict college outcomes? A meta-analysis. Psychological Bulletin, 130, –Robbins et al. (2009). Intervention effects on college performance and retention as mediated by motivational, emotional, and social control factors: Integrated meta- analytic path analyses. Journal of Applied Psychology, 94,

14 © 2010 by ACT, Inc. All rights reserved. 14 Behavior: Self-Report DomainMiddle SchoolHigh School & CollegeWork* Motivation (Getting work done)  Academic Discipline  Commitment to School  Optimism  Academic Discipline  Commitment to College  Goal Striving  General Determination  Study Skills  Communication Skills  Carefulness  Discipline  Order Social Engagement (Getting along)  Family Attitude toward Education  Family Involvement  Relationships w/ School Personnel  School Safety Climate  Social Activity  Social Connection  Cooperation  Goodwill  Influence  Sociability  Striving Self-Regulation (Keeping your cool)  Managing Feelings  Thinking Before Acting  Orderly Conduct  Academic Self-Confidence  Steadiness  Optimism  Stability Note. *Creativity and Savvy do not fully map onto these domains.

15 © 2010 by ACT, Inc. All rights reserved. 15 Behavior: Other Report DomainMiddle School & High SchoolWork Motivation (Getting work done)  Initiative  Planning & Organizing  Sustained Effort  Performance  Initiative  Planning & Organizing  Persistence  Responsibility Social Engagement (Getting along)  Communication  Working with Others  Communication  Working with Others Self-Regulation (Keeping your cool)  Managing Feelings  Conduct  Organizational Citizenship  Stress Management  Following Rules  Adaptability

16 © 2010 by ACT, Inc. All rights reserved. 16 Career Planning DomainMiddle School & High SchoolWork Interests  ACT Interest Inventory  DISCOVER Career Information Decision System  Fit Inventory Values  DISCOVER Career Information Decision System  Fit Inventory - Neuman et al. (2009). Job congruence, academic achievement, and earnings - Tracey & Robbins (2006). The interest-major congruence and college success relation: A longitudinal study.

17 © 2010 by ACT, Inc. All rights reserved. 17 Tailoring Assessments to Critical Transition Points and Context The role of noncognitive measures in promoting work training readiness and career success –Talent supply chain –Meeting various needs

18 © 2010 by ACT, Inc. All rights reserved. 18 Talent Supply Chain Noncognitive factors are important at all of the key transitions that lead individuals to the world of work. Pre-K Primary (K-8) Secondary (9-12) Post Secondary (2 & 4-yr) Work

19 © 2010 by ACT, Inc. All rights reserved. 19 Noncognitive Measures Meet Various Workforce Needs PurposeNeedsACT Noncognitive Solutions Pre-Selection-- Screen people in most cost effective way -- Find honest/dependable employees -- Save time in the screening process WorkKeys Performance Recruitment-- Identify people who fit the work environment -- Identify people with skills that match the job WorkKeys Fit Selection-- Select employees with skills that best fit the job -- Save time in selection process -- Select people in most cost-effective way -- Find honest/dependable employees -- Certifying employees WorkKeys Performance WorkKeys Talent WorkKeys Fit

20 © 2010 by ACT, Inc. All rights reserved. 20 Noncognitive Measures Meet Various Workforce Needs (cont.) PurposeNeedsACT Noncognitive Solutions Coaching/ Development -- Identify other jobs that an employee can fit -- Develop employees for future company needs -- Employees identify areas of improvement WorkKeys Talent WorkKeys Fit Behavioral Monitoring Scales Succession/ Leadership Planning -- Identify candidates for top-level positions -- Develop employees for future needs of company -- Retain top performers WorkKeys Talent WorkKeys Fit Training/ Development -- Identifying work readiness -- Identify basic workplace skill levels -- Educating about career planning -- Job Placement WorkKeys Talent WorkKeys Fit Behavioral Monitoring Scales

21 © 2010 by ACT, Inc. All rights reserved. 21 Critical Issues Self vs. other report Changing attitudes, behaviors, and skills Complexity of personality & career relationships –Different personality characteristics are important for different jobs (whether across or within jobs) –Moderation effects of personality on ability –Curvilinear effects of personality on performance

22 © 2010 by ACT, Inc. All rights reserved. 22 Self vs. Other Report Connelly & Ones (2010) conducted meta-analyses of self-other ratings based on 44,000+ individuals across 263 independent samples. –Other ratings are strong predictors of behaviors, particularly for academic achievement and job performance. –In some cases, other ratings yielded predictive validities substantially greater than self-ratings. –When other-ratings are added to self-ratings, results show considerable increases in validity (with gains more pronounced for Conscientiousness, Emotional Stability, and Extraversion).

23 © 2010 by ACT, Inc. All rights reserved. 23 Changing Attitudes, Behaviors, and Skills Behavior modeling training (BMT), based on Bandura’s social learning theory, has become a widely used psychologically-based approach to training in work environments. A recent meta-analysis by Taylor et al. (2005) summarizing 117 studies found substantial effects of BMT on a variety of training outcomes. Outcome d Attitudes.56 Behaviors.33 Declarative knowledge.69 Procedural knowledge-skills 1.21

24 © 2010 by ACT, Inc. All rights reserved. 24 “It was about here, wasn’t it, Ed, when you came on board as sales manager?” Harvard Business Review. March p. 90

25 © 2010 by ACT, Inc. All rights reserved. 25 Complexity of Personality & Career Different personality characteristics are important for different jobs (whether across or within fields). –Conscientiousness appears to be important for all jobs. –Extraversion is relevant to some jobs (e.g., sales) but not others (e.g., computer programmers).

26 © 2010 by ACT, Inc. All rights reserved. 26 Complexity of Personality & Career Across Fields Position Computer Programmers O-NET Job Zone Four: Considerable Preparation Needed Education Most occupations in this zone require a four-year bachelor's degree, but some do not. Position Sales Representatives O-NET Job Zone Four: Considerable Preparation Needed Education Most occupations in this zone require a four-year bachelor's degree, but some do not.

27 © 2010 by ACT, Inc. All rights reserved. 27 Complexity of Personality & Career: Comparing Scores Across Fields

28 © 2010 by ACT, Inc. All rights reserved. 28 Complexity of Personality & Career Moderation Postlethwaite et al. (2009) found that conscientiousness was a stronger predictor of safety behavior for individuals with lower levels of cognitive ability.

29 © 2010 by ACT, Inc. All rights reserved. 29 Complexity of Personality & Career Curvilinear Relationships Le et al. (in press) found curvilinear relationships between Emotional Stability and job performance (task, OCB, CWB), as well as between Conscientiousness and task performance.

30 © 2010 by ACT, Inc. All rights reserved. 30 Questions? Correspondence regarding this presentation should be addressed to: Steve Robbins Vice President, Research ACT, Inc.

31 © 2010 by ACT, Inc. All rights reserved. 31 References Allen, J. & Robbins, S. (2010). Effects of interest–major congruence, motivation, and academic performance on timely degree attainment. Journal of Counseling Psychology, 57, 23–35. Brown, K. G., Le, H., & Schmidt, F. L. (2006). Specific aptitude theory revisited: Is there incremental validity for training performance? International Journal of Selection and Assessment, 14, Casillas, A., Robbins, S. B., McKinniss, T., Postlethwaite, B., & Oh, I.S. (2009). Using narrow facets of integrity to predict safety: A test validation study. International Journal of Selection and Assessment, 17, Connelly, B.S. & Ones, D.S. (2010). An Other Perspective on Personality: Meta-Analytic Integration of Observers’ Accuracy and Predictive Validity. Psychological Bulletin, 136, 1092–1122. Le, H., Oh, I.-S., Robbins, S. B., Ilies, R., Holland, E., & Westrick, P. (in press). Too Much of a Good Thing: Curvilinear Relationships Between Personality Traits and Job Performance. Journal of Applied Psychology. McHenry, J. J., Hough, L. M., Toquam, J. L., Hanson, M. A., & Ashworth, S. (1990). Project A validity results: The relationship between predictor and criterion domains. Personnel Psychology, 43, Neumann, G., Olitsky, N., & Robbins, S. (2009). Job congruence, academic achievement, and earnings. Labour Economics, 16, Postlethwaite, B., Robbins, S., Rickerson, J., & McKinniss, T. (2009). The moderation of conscientiousness by cognitive ability when predicting workplace safety behavior. Personality and Individual Differences, 47,

32 © 2010 by ACT, Inc. All rights reserved. 32 References (cont.) Robbins, S. B., Lauver, K., Le, H., David, D., Langley, R., & Carlstrom, A. (2004). Do psychosocial and study skill factors predict college outcomes? A meta-analysis. Psychological Bulletin, 130, Robbins, S. B., Oh, I., Le, H., & Button, C. (2009). Intervention effects on college performance and retention as mediated by motivational, emotional, and social control factors: Integrated meta-analytic path analyses. Journal of Applied Psychology, 94, Rotundo, M. & Sackett, P. R. (2002). The relative importance of task, citizenship, and counterproductive performance to global ratings of job performance: A policy- capturing approach. Journal of Applied Psychology, 87, 66–80. Salgado, J. F., Anderson, N., Moscoso, S., Bertua, C., & de Fruyt, F. (2003). International validity generalization of GMA and cognitive abilities: A European community meta-analysis. Personnel Psychology, 56, Schmidt, F. L., Shaffer, J., & Oh. I. (2007). Reassessing the Relative Importance of Cognitive Ability and Personality in Job Performance and Training Performance: Some Surprising New Research Findings. Paper presented at the 2007 ATP conference, Palm Springs, CA. Feb. 6. Taylor, Paul J., Russ-Eft, D.F., & Chan, D.W.L. (2005). A Meta-Analytic Review of Behavior Modeling Training. Journal of Applied Psychology, 90, Tracey, T.J.G., & Robbins, S.B. (2006). The interest-major congruence and college success relation: A longitudinal study. Journal of Vocational Behavior, 69,


Download ppt "© 2010 by ACT, Inc. All rights reserved. 1 Creating an Aligned Assessment System to Promote Work Training Readiness and Career Success Steve Robbins Vice."

Similar presentations


Ads by Google