1. To Use the Teamwork Test -- Or Not? A Psychometric Evaluation Janet L. Kottke California State University, San Bernardino Kimberly A. French University of South Florida Rhiannon J. Kirchner California State University, San Bernardino Presented to PTC of Southern California, July 23, 2013

2. TKSA Development Stevens and Campion (1994; 1999) develop the Teamwork KSA Test – 35 items, multiple choice format (4 options, dichotomous scoring) Reviewed teamwork literature – Identified 14 KSAs identified with effective teamwork Kottke, French, & Kirchner PTC July 23, 2013 2

3. 14 KSAs 5 facets These 14 KSAs represent – Five facets Conflict resolution Collaborative problem solving Communication Goal setting and performance management Planning and task coordination Kottke, French, & Kirchner PTC July 23, 2013 3

4. 5 facets 2 dimensions Two higher-order dimensions: – Interpersonal KSAs Conflict resolution Collaborative problem solving Communication – Self-management KSAs Goal setting and performance management Planning and task coordination Kottke, French, & Kirchner PTC July 23, 2013 4

5. Teamwork KSA Structure Kottke, French, & Kirchner PTC July 23, 2013 5 ONeill, Goffin, & Gellatly, 2012, p. 37

6. 6

7. Sample Question When you set work goals for yourself or your work team, what are the best goals to set? A. Set goals to "do your best." B. Set general and broad goals. C. Set specific and detailed goals. D. Set easy and simple goals. Kottke, French, & Kirchner PTC July 23, 2013 7

8. Criterion validity Several studies have found predictability of supervisor and observer ratings of team effectiveness (rs range.20 to.56) – Ellis, Bell, Ployhart, Hollenbeck, & Ilgen, 2005 – Leach, Wall, Rogelberg, & Jackson, 2005 – McClough & Rogelberg, 2003 – Morgeson, Reider, & Campion, 2005 Mixed results in predicting team member peer ratings Kottke, French, & Kirchner PTC July 23, 2013 8

9. Questions about predictive validity Miller, 2001 – Issues raised about Stevens & Campion validation studies Team performance was predicted less well (r =.44) than task work (r =.56) TKSA predicted team performance only slightly better than did an aptitude test – But: TKSA added incremental variance beyond GMA Aptitude data suggest individual aptitude, not teamwork aptitude measured – Millers own study finds no significant relationship between team performance and TKSA scores Kottke, French, & Kirchner PTC July 23, 2013 9

10. Questions about psychometrics ONeill, Goffin, & Gellatly (2012) examine the TKSA – Comprehensive analysis Reviewed all known studies using TKSA Classic test theory analysis Confirmatory factor analysis Exploratory factor analysis Convergent, discriminant validation Criterion validity analysis – Disappointing results all around Kottke, French, & Kirchner PTC July 23, 2013 10

11. Our plan Replicate ONeill CTT & CFA analyses with fresh samples Data sources – Had used the TKSA Test for a group project in several classes (sample 1) – Collected new data to address unexpected finding from sample 1 that aptitude was negatively related to teamwork interest (sample 2) – Collected data for a thesis project on teamwork and task interdependence (sample 3) Kottke, French, & Kirchner PTC July 23, 2013 11

12. The 3 samples Students from a mid-size southwestern University. – Sample 1 = 251 college students Cohort groups, upper level undergraduate-level psychology course, term-length group project. – Sample 2 = 279 college students Recruited from psychology and business courses – Sample 3 = 404 college students From upper-level undergraduate courses in both psychology and business departments, either a long-term (i.e., several weeks) or term-length group project Kottke, French, & Kirchner PTC July 23, 2013 12

13. Work experience of samples Although students, vast majority were employed – 96% in sample 1 – 82.4% in sample 2 – 88.4% in sample 3 Work experience ranged from means of 5.5 to 7.3 years – SDs ranged from 5.6 to 6.3 years Kottke, French, & Kirchner PTC July 23, 2013 13

14. Table 1. Demographic Descriptive Statistics Sample 1Sample 2Sample 3 Demographic CharacteristicMSDM M Age25.16.623.1 7.023.96.1 Work experience (years) 7.35.9 5.5 6.3 5.95.6 Number of term-length group projects-- 1.8 2.0-- Number of work group projects-- 4.411.9-- Gender (% Female)82.1%74.9%65.1% Ethnicity African American 9.2%11.5% 8.2% Asian American 4.4% 4.7 % 4.5% Hispanic45.4%47.3%45.0% White35.1%26.9%29.7% Other 6.0% 8.2%11.9% Education Freshman 0.4%16.5% 0.0% Sophomore 2.8%14.3% 4.7% Junior43.4%33.3%50.2% Senior51.0%34.1%42.6% Graduate 2.0% 0.4% 0.2% Other 0.4% 0.5% Respondents with work experience96.0%82.4%88.4% Respondents with supervisor experience40.2%24.7%- Kottke, French, & Kirchner PTC July 23, 2013 14

15. Analyses conducted Analytic Approach – In each sample we evaluated Classical test theory – Item, subscale, and total measure properties, » Means, standard deviations, and item-total correlations – Subscale and total measure means, standard deviations, and internal reliability Structure using categorical confirmatory factor analysis (Mplus) – One factor model, two factor model, and a five factor model » For multi-factor models, both a covaried model (all factors were allowed to correlate), as well as a higher-order model (includes higher-order general teamwork KSA factor) Kottke, French, & Kirchner PTC July 23, 2013 15

16. Supplemental Supplemental analysis – To ensure psychometric issues are not a result of student samples, ran classical test analyses on the subset of working respondents from each sample. – Results from working samples were not substantially different than full samples For example, average absolute difference in factor and total means was.11 for sample 1 [SD =.07],.21 for sample 2 [SD =.19], and.14 for sample 3 [SD =.21]. The average difference in inter-item correlations was.02 [SD =.01]) Thus, only the results from the full samples are presented here. Kottke, French, & Kirchner PTC July 23, 2013 16

17. Table 2. Teamwork KSA Test item-level means, standard deviations, and item-total correlations for all samples. Sample 1Sample 2Sample 3 Subscale Item Statistics M(p)SDr i-f r i-h r i-t M(p)SDr i-f r i-h r i-t M(p)SDr i-f r i-h r i-t Conflict resolution mean0.690.430.140.220.260.640.45 0.080.220.17 0.670.440.050.160.17 Conflict resolution SD0.160.070.090.140.150.190.06 0.050.110.14 0.200.070.050.110.12 Communication mean0.540.480.130.17 0.540.48 0.080.130.15 0.530.480.110.140.16 Communication SD0.130.020.06 0.070.130.02 0.090.11 0.150.020.070.080.07 Collaborative problem solving mean0.420.470.130.180.200.420.48 0.140.160.18 0.400.470.110.140.16 Collaborative problem solving SD0.140.020.090.160.170.130.02 0.090.10 0.130.020.090.080.09 Goal setting & performance management mean0.600.470.200.220.230.580.48 0.070.130.19 0.590.480.190.250.28 Goal setting & performance management SD0.120.040.070.040.050.120.01 0.07 0.05 0.110.010.060.02 Planning and task coordination mean0.640.470.150.180.240.630.47 0.130.170.23 0.640.470.150.220.24 Planning and task coordination SD0.100.030.02 0.070.120.03 0.050.09 0.130.040.030.020.05 Grand mean0.550.470.140.190.21 0.540.480.100.150.17 0.540.470.120.170.19 Grand SD0.160.040.080.100.12 0.150.030.080.10 0.160.030.070.08 Notes. M(p) = item means, SD = item standard deviations, r i-f = corrected item-factor correlation,, r i-h = corrected item-higher order factor correlation, r i-t = corrected item-total correlation. RESULTS: Item means, SDs, item total correlations Kottke, French, & Kirchner PTC July 23, 2013 17

18. Table 3. Teamwork KSA Test facet and total means, standard deviations, reliability, and item correlation summary for all samples Sample 1Sample 2Sample 3 No. of ItemsMSDK-R 20 Inter-item correlation MSDK-R 20 Inter-item correlation MSDK-R 20 Inter-item correlation Teamwork KSA Test Score MRangeM M Interpersonal KSAs 2513.053.55.58.06-.12,.3212.843.36.52.05-.17,.2312.723.29.51.04-.15,.19 Conflict Resolution 42.75.97.25.09-.06,.262.55.95.15.05-.01,.142.67.91.10.03-.11,.16 Communication 126.492.04.36.05-.11,.266.531.89.24.03-.10,.226.411.99.33.04-.11,.19 Collaborative Problem Solving 93.821.71.34.05-.12,.233.761.72.35.06-.08,.223.631.64.28.04-.07,.18 Self-Management KSAs 106.231.99.49.09.00,.276.061.87.39.06-.08,.226.112.10.54.11-.01,.26 Goal Setting & Performance Management 53.011.29.40.12.04,.242.901.16.17.03-.06,.142.931.29.38.11.03,.23 Planning & Task Coordination 53.221.20.30.08.00,.183.171.19.27.07-.05,.223.191.21.32.09-.01,.26 Teamwork KSA Overall 3519.294.84.69.06-.14,.3218.814.69.65.05-.17,.2518.824.70.67.05-.15,.26 More Results: Scale means, SDs, inter item correlation means, KR 20s Kottke, French, & Kirchner PTC July 23, 2013 18

19. Confirmatory Factor Analysis Results on following slide Kottke, French, & Kirchner PTC July 23, 2013 19

20. Table 4. Summary of confirmatory factor analysis results SampleModel 2 dfCFIRMSEA Absolute Loading MeanLoading Range Sample 1Five factor, covaried b 198.12*158.85.032.34-.10 to.83 Revised five factor, covaried a 309.23**152.41.064.41-.05 to.92 Five factor, higher orderDid not converge Revised five factor, higher order a 359.39**154.24.072.44-.03 to 1.00 Two factor, covaried202.701*159.84.033.34-.51 to.82 Two factor, higher orderDid not converge Revised two factor, higher order a 202.95*156.36.066.36-.09 to 1.00 Single factor202.701*159.84.033.34-.10 to.82 Sample 2Five factor, covaried b 211.12183.88.022.29-.08 to.58 Revised five factor, covaried a 308.61**182.47.048.38-.09 to.71 Five factor, higher orderDid not converge Revised five factor, higher order a 368.40**182.22.058.38-.09 to 1.00 Two factor, covaried212.79184.88.023.30-.49 to.62 Two factor, higher orderDid not converge Revised two factor, higher order a 363.73**182.24.057.32-.08 to 1.00 Single factor212.54184.88.022.30-.08 to.62 Sample 3Five factor, covaried b 327.42**213.78.036.33-.01 to.61 Revised five factor, covaried a 471.13**211.51.055.40.01 to.71 Five factor, higher orderDid not converge Revised five factor, higher order a 565.28**209.32.065.40.01 to 1.00 Two factor, covaried345.70**214.75.039.32-.55 to.52 Two factor, higher orderDid not converge Revised two factor, higher order a 527.38**211.40.061.32.03 to 1.00 Single factor353.40**214.74.040.30-.01 to.52 *p <.01, **p <.001 Kottke, French, & Kirchner PTC July 23, 2013 20

21. Discussion To be fair to the test authors – They recommend the subscales be used for training and developmental purposes only – In all of their publications, they use only total scores – So, whether the test authors intended a hierarchical structure is not a certainty Yet, with their careful development that categorized the 14 KSAs into specific facets, one would expect meaningful structure Criterion validity has been found, but issues here as well – TKSA test correlates highly with general mental ability – Some incremental predictability Kottke, French, & Kirchner PTC July 23, 2013 21

22. Conclusion Very limited evidence for adequate psychometric properties Use with caution – Recognize the tests limitations Kottke, French, & Kirchner PTC July 23, 2013 22

23. Thank you Kottke, French, & Kirchner PTC July 23, 2013 23

24. Questions? Kottke, French, & Kirchner PTC July 23, 2013 24

25. References Kottke, French, & Kirchner PTC July 23, 2013 25