We think you have liked this presentation. If you wish to download it, please recommend it to your friends in any social system. Share buttons are a little bit lower. Thank you!
Presentation is loading. Please wait.
Published byKorbin Lambeth
Modified over 2 years ago
© 2010 by ACT, Inc. All rights reserved. 1 Creating an Aligned Assessment System to Promote Work Training Readiness and Career Success Steve Robbins Vice President, Research ACT, Inc. Though designed to meet a wide array of needs, all ACT programs and services have one guiding purpose: Helping people achieve education and workplace success.
© 2010 by ACT, Inc. All rights reserved. 2 The Challenges Create a comprehensive assessment system that predicts success in education and work Tailor assessments to critical transition points and context Understand the complexity and interplay of cognitive and noncognitive factors across time and jobs
© 2010 by ACT, Inc. All rights reserved. 3 Do noncognitive factors matter? What we know from the world of work: –Project A from the military –Training and work outcomes
© 2010 by ACT, Inc. All rights reserved. 4 Combining Personality & Cognitive Ability Tests Correlations between general cognitive ability and personality tests and measures of job performance in Project A CogPersBothCriteria Core technical proficiency General Soldiering Proficiency Effort and Leadership Personal Discipline Physical fitness and military bearing (McHenry, Hough, Toquam, Hanson, & Ashworth, 1990) Creating opportunity for incremental validity especially as criteria vary
© 2010 by ACT, Inc. All rights reserved. 5 Training and Work Outcomes Noncognitive factors are predictive of important job performance criteria (e.g., Rotundo & Sackett, 2002; Casillas et al., 2009) –Task Performance –Organizational Citizenship –Counterproductive Work Behaviors –Safety Noncognitive factors meet a variety of employer needs –Screening –Selection –Training
© 2010 by ACT, Inc. All rights reserved. 6 Validity of Cognitive and Noncognitive Measures for Training and Work Outcomes Training Outcomes Work Performance General Mental Ability.54 a.62 b Math.48 a.52 b Reading.44 a.35 b Conscientiousness.17 c.22 c Emotional Stability.10 c.11 c Notes. a Brown, Le, & Schmidt (2006). b Salgado et al. (2003). c Schmidt et al. (2007) corrected for indirect range restriction.
© 2010 by ACT, Inc. All rights reserved. 7 No Unified Theory of Key Characteristics However, the literature suggests that several factors are important: –Motivation, Social Engagement, and Self-Regulation –Personality characteristics (the Big Five) Role of behavior ratings vs. self-report –Different perspectives –Coaching/training –Monitoring progress
© 2010 by ACT, Inc. All rights reserved. 8 Common and Specific Factors Are the same characteristics (i.e., common factors) associated with educational and work outcomes? Are there unique (i.e., specific) characteristics that contribute to the prediction of educational and work outcomes? In particular, how do age and setting impact the outcomes of interest?
© 2010 by ACT, Inc. All rights reserved. 9 Behavioral Development (Motivation, Social Engagement, & Self-Regulation) Cognitive Development & Acquisition of Foundational Skills (Academic Learning & Achievement) Social Capital School/Work Factors Integrated Pyramid for Success Career Development (Exploration, Crystallization, Choice & Match) Allen & Robbins (2010)
© 2010 by ACT, Inc. All rights reserved. 10 The human side of the pyramid
© 2010 by ACT, Inc. All rights reserved. 11 ACT’s noncognitive programs are part of a comprehensive and integrated set of solutions, which address the three broad areas essential for academic and work success across the lifespan: Academic achievement & foundational skills Behavior (noncognitive) Career planning (noncognitive) Set of Solutions
© 2010 by ACT, Inc. All rights reserved. 12 Achievement & Foundational Skills Grades CollegeWork EXPLORE ® PLAN ® ACT ® COMPASS ® Comprehensive & Integrated Academic & Work Solutions: Grade 6 - Work SRI and BMS SRI Behavior WorkKeys ® Performance, Talent & BMS ACT Interest Inventory Career Planning Fit College Fit
© 2010 by ACT, Inc. All rights reserved. 13 Creating a Comprehensive Set of Behavior (Self & Other Report) Assessments Role of meta-analysis and validity generalization –Robbins et al. (2004). Do psychosocial and study skill factors predict college outcomes? A meta-analysis. Psychological Bulletin, 130, –Robbins et al. (2009). Intervention effects on college performance and retention as mediated by motivational, emotional, and social control factors: Integrated meta- analytic path analyses. Journal of Applied Psychology, 94,
© 2010 by ACT, Inc. All rights reserved. 14 Behavior: Self-Report DomainMiddle SchoolHigh School & CollegeWork* Motivation (Getting work done) Academic Discipline Commitment to School Optimism Academic Discipline Commitment to College Goal Striving General Determination Study Skills Communication Skills Carefulness Discipline Order Social Engagement (Getting along) Family Attitude toward Education Family Involvement Relationships w/ School Personnel School Safety Climate Social Activity Social Connection Cooperation Goodwill Influence Sociability Striving Self-Regulation (Keeping your cool) Managing Feelings Thinking Before Acting Orderly Conduct Academic Self-Confidence Steadiness Optimism Stability Note. *Creativity and Savvy do not fully map onto these domains.
© 2010 by ACT, Inc. All rights reserved. 15 Behavior: Other Report DomainMiddle School & High SchoolWork Motivation (Getting work done) Initiative Planning & Organizing Sustained Effort Performance Initiative Planning & Organizing Persistence Responsibility Social Engagement (Getting along) Communication Working with Others Communication Working with Others Self-Regulation (Keeping your cool) Managing Feelings Conduct Organizational Citizenship Stress Management Following Rules Adaptability
© 2010 by ACT, Inc. All rights reserved. 16 Career Planning DomainMiddle School & High SchoolWork Interests ACT Interest Inventory DISCOVER Career Information Decision System Fit Inventory Values DISCOVER Career Information Decision System Fit Inventory - Neuman et al. (2009). Job congruence, academic achievement, and earnings - Tracey & Robbins (2006). The interest-major congruence and college success relation: A longitudinal study.
© 2010 by ACT, Inc. All rights reserved. 17 Tailoring Assessments to Critical Transition Points and Context The role of noncognitive measures in promoting work training readiness and career success –Talent supply chain –Meeting various needs
© 2010 by ACT, Inc. All rights reserved. 18 Talent Supply Chain Noncognitive factors are important at all of the key transitions that lead individuals to the world of work. Pre-K Primary (K-8) Secondary (9-12) Post Secondary (2 & 4-yr) Work
© 2010 by ACT, Inc. All rights reserved. 19 Noncognitive Measures Meet Various Workforce Needs PurposeNeedsACT Noncognitive Solutions Pre-Selection-- Screen people in most cost effective way -- Find honest/dependable employees -- Save time in the screening process WorkKeys Performance Recruitment-- Identify people who fit the work environment -- Identify people with skills that match the job WorkKeys Fit Selection-- Select employees with skills that best fit the job -- Save time in selection process -- Select people in most cost-effective way -- Find honest/dependable employees -- Certifying employees WorkKeys Performance WorkKeys Talent WorkKeys Fit
© 2010 by ACT, Inc. All rights reserved. 20 Noncognitive Measures Meet Various Workforce Needs (cont.) PurposeNeedsACT Noncognitive Solutions Coaching/ Development -- Identify other jobs that an employee can fit -- Develop employees for future company needs -- Employees identify areas of improvement WorkKeys Talent WorkKeys Fit Behavioral Monitoring Scales Succession/ Leadership Planning -- Identify candidates for top-level positions -- Develop employees for future needs of company -- Retain top performers WorkKeys Talent WorkKeys Fit Training/ Development -- Identifying work readiness -- Identify basic workplace skill levels -- Educating about career planning -- Job Placement WorkKeys Talent WorkKeys Fit Behavioral Monitoring Scales
© 2010 by ACT, Inc. All rights reserved. 21 Critical Issues Self vs. other report Changing attitudes, behaviors, and skills Complexity of personality & career relationships –Different personality characteristics are important for different jobs (whether across or within jobs) –Moderation effects of personality on ability –Curvilinear effects of personality on performance
© 2010 by ACT, Inc. All rights reserved. 22 Self vs. Other Report Connelly & Ones (2010) conducted meta-analyses of self-other ratings based on 44,000+ individuals across 263 independent samples. –Other ratings are strong predictors of behaviors, particularly for academic achievement and job performance. –In some cases, other ratings yielded predictive validities substantially greater than self-ratings. –When other-ratings are added to self-ratings, results show considerable increases in validity (with gains more pronounced for Conscientiousness, Emotional Stability, and Extraversion).
© 2010 by ACT, Inc. All rights reserved. 23 Changing Attitudes, Behaviors, and Skills Behavior modeling training (BMT), based on Bandura’s social learning theory, has become a widely used psychologically-based approach to training in work environments. A recent meta-analysis by Taylor et al. (2005) summarizing 117 studies found substantial effects of BMT on a variety of training outcomes. Outcome d Attitudes.56 Behaviors.33 Declarative knowledge.69 Procedural knowledge-skills 1.21
© 2010 by ACT, Inc. All rights reserved. 24 “It was about here, wasn’t it, Ed, when you came on board as sales manager?” Harvard Business Review. March p. 90
© 2010 by ACT, Inc. All rights reserved. 25 Complexity of Personality & Career Different personality characteristics are important for different jobs (whether across or within fields). –Conscientiousness appears to be important for all jobs. –Extraversion is relevant to some jobs (e.g., sales) but not others (e.g., computer programmers).
© 2010 by ACT, Inc. All rights reserved. 26 Complexity of Personality & Career Across Fields Position Computer Programmers O-NET Job Zone Four: Considerable Preparation Needed Education Most occupations in this zone require a four-year bachelor's degree, but some do not. Position Sales Representatives O-NET Job Zone Four: Considerable Preparation Needed Education Most occupations in this zone require a four-year bachelor's degree, but some do not.
© 2010 by ACT, Inc. All rights reserved. 27 Complexity of Personality & Career: Comparing Scores Across Fields
© 2010 by ACT, Inc. All rights reserved. 28 Complexity of Personality & Career Moderation Postlethwaite et al. (2009) found that conscientiousness was a stronger predictor of safety behavior for individuals with lower levels of cognitive ability.
© 2010 by ACT, Inc. All rights reserved. 29 Complexity of Personality & Career Curvilinear Relationships Le et al. (in press) found curvilinear relationships between Emotional Stability and job performance (task, OCB, CWB), as well as between Conscientiousness and task performance.
© 2010 by ACT, Inc. All rights reserved. 30 Questions? Correspondence regarding this presentation should be addressed to: Steve Robbins Vice President, Research ACT, Inc.
© 2010 by ACT, Inc. All rights reserved. 31 References Allen, J. & Robbins, S. (2010). Effects of interest–major congruence, motivation, and academic performance on timely degree attainment. Journal of Counseling Psychology, 57, 23–35. Brown, K. G., Le, H., & Schmidt, F. L. (2006). Specific aptitude theory revisited: Is there incremental validity for training performance? International Journal of Selection and Assessment, 14, Casillas, A., Robbins, S. B., McKinniss, T., Postlethwaite, B., & Oh, I.S. (2009). Using narrow facets of integrity to predict safety: A test validation study. International Journal of Selection and Assessment, 17, Connelly, B.S. & Ones, D.S. (2010). An Other Perspective on Personality: Meta-Analytic Integration of Observers’ Accuracy and Predictive Validity. Psychological Bulletin, 136, 1092–1122. Le, H., Oh, I.-S., Robbins, S. B., Ilies, R., Holland, E., & Westrick, P. (in press). Too Much of a Good Thing: Curvilinear Relationships Between Personality Traits and Job Performance. Journal of Applied Psychology. McHenry, J. J., Hough, L. M., Toquam, J. L., Hanson, M. A., & Ashworth, S. (1990). Project A validity results: The relationship between predictor and criterion domains. Personnel Psychology, 43, Neumann, G., Olitsky, N., & Robbins, S. (2009). Job congruence, academic achievement, and earnings. Labour Economics, 16, Postlethwaite, B., Robbins, S., Rickerson, J., & McKinniss, T. (2009). The moderation of conscientiousness by cognitive ability when predicting workplace safety behavior. Personality and Individual Differences, 47,
© 2010 by ACT, Inc. All rights reserved. 32 References (cont.) Robbins, S. B., Lauver, K., Le, H., David, D., Langley, R., & Carlstrom, A. (2004). Do psychosocial and study skill factors predict college outcomes? A meta-analysis. Psychological Bulletin, 130, Robbins, S. B., Oh, I., Le, H., & Button, C. (2009). Intervention effects on college performance and retention as mediated by motivational, emotional, and social control factors: Integrated meta-analytic path analyses. Journal of Applied Psychology, 94, Rotundo, M. & Sackett, P. R. (2002). The relative importance of task, citizenship, and counterproductive performance to global ratings of job performance: A policy- capturing approach. Journal of Applied Psychology, 87, 66–80. Salgado, J. F., Anderson, N., Moscoso, S., Bertua, C., & de Fruyt, F. (2003). International validity generalization of GMA and cognitive abilities: A European community meta-analysis. Personnel Psychology, 56, Schmidt, F. L., Shaffer, J., & Oh. I. (2007). Reassessing the Relative Importance of Cognitive Ability and Personality in Job Performance and Training Performance: Some Surprising New Research Findings. Paper presented at the 2007 ATP conference, Palm Springs, CA. Feb. 6. Taylor, Paul J., Russ-Eft, D.F., & Chan, D.W.L. (2005). A Meta-Analytic Review of Behavior Modeling Training. Journal of Applied Psychology, 90, Tracey, T.J.G., & Robbins, S.B. (2006). The interest-major congruence and college success relation: A longitudinal study. Journal of Vocational Behavior, 69,
1 © 2007 Clemson University – All rights reserved Georgias Graduation/ Dropout Prevention Project Building Systems to Help Students with Disabilities Graduate:
1 © Development Dimensions Intl, Inc., MMV. All rights reserved. 1 Succession Planning Overview STATE ACTION FOR EDUCATION LEADERSHIP PROJECT Succession.
1 Best of 2008: Research findings you may have missed. Ryan Watkins, Ph.D. George Washington University
1 Dropout Prevention for Students with Disabilities :What the Research Tells Us Urban Special Education Leadership Collaborative Fall Meeting October 26-27,
1 What Is Organizational Behavior? 1. 2 The Importance of Interpersonal Skills Understanding OB helps determine manager effectiveness –Technical and quantitative.
Planning for Successful Transition from School to Post School Life Presented by Ed P r o o o Development 1.
National Association of School Psychologists NASP Model for Comprehensive & Integrated School Psychological Services.
OUSD New Small Schools External Evaluation New Small Schools Evaluation Board Presentation September 26, 2007.
Chapter 7 / Slide 1 Copyright © 2008 Pearson Education Canada Part 3 Groups and Teamwor Copyright © 2008 Pearson Education Canada Social Behaviour and.
1 Cultural Considerations with Response to Intervention (RTI) Models and Literacy Instruction Janette Klingner Michael Orosco University of Colorado at.
Copyright © 2007 by The McGraw-Hill Companies, Inc. All rights reserved. McGraw-Hill/Irwin Chapter Work Analysis and Design 4.
Connecting PBIS, Student Support Teams, and School Mental Health Margaret Grady Kidder William Flook Joan Ledvina Parr Erika Wood Baltimore County Public.
Processes and Practices for Postsecondary Transition Planning A Focus on Age-Appropriate Transition Assessment Amy Szymanski, M.Ed. Consultant February/March.
2 Welcome Todays course Your facilitator The EQ-360 The facilities.
Robbins et al., Fundamentals of Management, 4th Canadian Edition ©2005 Pearson Education Canada, Inc. Chapter 1 Managers and Managing in the 21st Century.
1 ICS 462 Information Systems Strategy & Implementation 2.1 Analytical Tools in Strategic Management: Strategic Analysis.
Communicating Effectively About Liberal and General Education AAC&U Network for Academic Renewal March 2011Chicago, IL.
Young People in England An evidence discussion paper Young People Analysis & Strategic Analysis Department for Children, Schools and Families This document.
Training Session #5 Youth Development & Youth Leadership.
Debra Humphreys Association of American Colleges & Universities Communicating Effectively About the Aims and Outcomes of.
Are you experienced? Translating Experiential Learning into Credit for Non-Traditional Students Amy Sherman The Council for Adult and Experiential Learning.
Systems of Professional Learning Module 3 Grades 6–12: Focus on Teaching and Learning.
© John Beveridge CobiT Update NSAA IT Conference Richmond, VA John W. Beveridge September 27, 2007.
Introduction to Project Management 1 August 24 & 25, 2010 Presented by: Joanne Cobb Ginny Montgomery Dan Druliner.
AP Program Update Midwest Regional Forum February 11, 2008 Marcia Wilbur Executive Director, Curriculum & Content Development AP Program.
Head Start and Public Schools Strengthening Birth to PK-3 Partnerships Approaches to Linking PK-3 in Massachusetts: Activities to Support Continuity for.
INCREASING Engagement BROADENING Diversity OPTIMIZING Development Position Yourself for Success Using The Four Stages® of Contribution Increase Your Impact,
Transitioning to the Common Core Trinity County Office of Education August 15, 2012 Mathematics.
© 2011 # 1 רשות ארצית למדידה והערכה בחינוך ראמ"ה טיוטא Introduction Large Scale Assessments (LSA) International Large Scale Assessments (ILSA) Recent Developments.
Leading Change in the 21 st Century Larry D. Coble School Leadership Services.
© 2016 SlidePlayer.com Inc. All rights reserved.