Presentation is loading. Please wait.

Presentation is loading. Please wait.

To Use the Teamwork Test -- Or Not? A Psychometric Evaluation Janet L. Kottke California State University, San Bernardino Kimberly A. French University.

Similar presentations


Presentation on theme: "To Use the Teamwork Test -- Or Not? A Psychometric Evaluation Janet L. Kottke California State University, San Bernardino Kimberly A. French University."— Presentation transcript:

1 To Use the Teamwork Test -- Or Not? A Psychometric Evaluation Janet L. Kottke California State University, San Bernardino Kimberly A. French University of South Florida Rhiannon J. Kirchner California State University, San Bernardino Presented to PTC of Southern California, July 23, 2013

2 TKSA Development Stevens and Campion (1994; 1999) develop the Teamwork KSA Test – 35 items, multiple choice format (4 options, dichotomous scoring) Reviewed teamwork literature – Identified 14 KSAs identified with effective teamwork Kottke, French, & Kirchner PTC July 23,

3 14 KSAs 5 facets These 14 KSAs represent – Five facets Conflict resolution Collaborative problem solving Communication Goal setting and performance management Planning and task coordination Kottke, French, & Kirchner PTC July 23,

4 5 facets 2 dimensions Two higher-order dimensions: – Interpersonal KSAs Conflict resolution Collaborative problem solving Communication – Self-management KSAs Goal setting and performance management Planning and task coordination Kottke, French, & Kirchner PTC July 23,

5 Teamwork KSA Structure Kottke, French, & Kirchner PTC July 23, ONeill, Goffin, & Gellatly, 2012, p. 37

6 6

7 Sample Question When you set work goals for yourself or your work team, what are the best goals to set? A. Set goals to "do your best." B. Set general and broad goals. C. Set specific and detailed goals. D. Set easy and simple goals. Kottke, French, & Kirchner PTC July 23,

8 Criterion validity Several studies have found predictability of supervisor and observer ratings of team effectiveness (rs range.20 to.56) – Ellis, Bell, Ployhart, Hollenbeck, & Ilgen, 2005 – Leach, Wall, Rogelberg, & Jackson, 2005 – McClough & Rogelberg, 2003 – Morgeson, Reider, & Campion, 2005 Mixed results in predicting team member peer ratings Kottke, French, & Kirchner PTC July 23,

9 Questions about predictive validity Miller, 2001 – Issues raised about Stevens & Campion validation studies Team performance was predicted less well (r =.44) than task work (r =.56) TKSA predicted team performance only slightly better than did an aptitude test – But: TKSA added incremental variance beyond GMA Aptitude data suggest individual aptitude, not teamwork aptitude measured – Millers own study finds no significant relationship between team performance and TKSA scores Kottke, French, & Kirchner PTC July 23,

10 Questions about psychometrics ONeill, Goffin, & Gellatly (2012) examine the TKSA – Comprehensive analysis Reviewed all known studies using TKSA Classic test theory analysis Confirmatory factor analysis Exploratory factor analysis Convergent, discriminant validation Criterion validity analysis – Disappointing results all around Kottke, French, & Kirchner PTC July 23,

11 Our plan Replicate ONeill CTT & CFA analyses with fresh samples Data sources – Had used the TKSA Test for a group project in several classes (sample 1) – Collected new data to address unexpected finding from sample 1 that aptitude was negatively related to teamwork interest (sample 2) – Collected data for a thesis project on teamwork and task interdependence (sample 3) Kottke, French, & Kirchner PTC July 23,

12 The 3 samples Students from a mid-size southwestern University. – Sample 1 = 251 college students Cohort groups, upper level undergraduate-level psychology course, term-length group project. – Sample 2 = 279 college students Recruited from psychology and business courses – Sample 3 = 404 college students From upper-level undergraduate courses in both psychology and business departments, either a long-term (i.e., several weeks) or term-length group project Kottke, French, & Kirchner PTC July 23,

13 Work experience of samples Although students, vast majority were employed – 96% in sample 1 – 82.4% in sample 2 – 88.4% in sample 3 Work experience ranged from means of 5.5 to 7.3 years – SDs ranged from 5.6 to 6.3 years Kottke, French, & Kirchner PTC July 23,

14 Table 1. Demographic Descriptive Statistics Sample 1Sample 2Sample 3 Demographic CharacteristicMSDM M Age Work experience (years) Number of term-length group projects Number of work group projects Gender (% Female)82.1%74.9%65.1% Ethnicity African American 9.2%11.5% 8.2% Asian American 4.4% 4.7 % 4.5% Hispanic45.4%47.3%45.0% White35.1%26.9%29.7% Other 6.0% 8.2%11.9% Education Freshman 0.4%16.5% 0.0% Sophomore 2.8%14.3% 4.7% Junior43.4%33.3%50.2% Senior51.0%34.1%42.6% Graduate 2.0% 0.4% 0.2% Other 0.4% 0.5% Respondents with work experience96.0%82.4%88.4% Respondents with supervisor experience40.2%24.7%- Kottke, French, & Kirchner PTC July 23,

15 Analyses conducted Analytic Approach – In each sample we evaluated Classical test theory – Item, subscale, and total measure properties, » Means, standard deviations, and item-total correlations – Subscale and total measure means, standard deviations, and internal reliability Structure using categorical confirmatory factor analysis (Mplus) – One factor model, two factor model, and a five factor model » For multi-factor models, both a covaried model (all factors were allowed to correlate), as well as a higher-order model (includes higher-order general teamwork KSA factor) Kottke, French, & Kirchner PTC July 23,

16 Supplemental Supplemental analysis – To ensure psychometric issues are not a result of student samples, ran classical test analyses on the subset of working respondents from each sample. – Results from working samples were not substantially different than full samples For example, average absolute difference in factor and total means was.11 for sample 1 [SD =.07],.21 for sample 2 [SD =.19], and.14 for sample 3 [SD =.21]. The average difference in inter-item correlations was.02 [SD =.01]) Thus, only the results from the full samples are presented here. Kottke, French, & Kirchner PTC July 23,

17 Table 2. Teamwork KSA Test item-level means, standard deviations, and item-total correlations for all samples. Sample 1Sample 2Sample 3 Subscale Item Statistics M(p)SDr i-f r i-h r i-t M(p)SDr i-f r i-h r i-t M(p)SDr i-f r i-h r i-t Conflict resolution mean Conflict resolution SD Communication mean Communication SD Collaborative problem solving mean Collaborative problem solving SD Goal setting & performance management mean Goal setting & performance management SD Planning and task coordination mean Planning and task coordination SD Grand mean Grand SD Notes. M(p) = item means, SD = item standard deviations, r i-f = corrected item-factor correlation,, r i-h = corrected item-higher order factor correlation, r i-t = corrected item-total correlation. RESULTS: Item means, SDs, item total correlations Kottke, French, & Kirchner PTC July 23,

18 Table 3. Teamwork KSA Test facet and total means, standard deviations, reliability, and item correlation summary for all samples Sample 1Sample 2Sample 3 No. of ItemsMSDK-R 20 Inter-item correlation MSDK-R 20 Inter-item correlation MSDK-R 20 Inter-item correlation Teamwork KSA Test Score MRangeM M Interpersonal KSAs , , ,.19 Conflict Resolution , , ,.16 Communication , , ,.19 Collaborative Problem Solving , , ,.18 Self-Management KSAs , , ,.26 Goal Setting & Performance Management , , ,.23 Planning & Task Coordination , , ,.26 Teamwork KSA Overall , , ,.26 More Results: Scale means, SDs, inter item correlation means, KR 20s Kottke, French, & Kirchner PTC July 23,

19 Confirmatory Factor Analysis Results on following slide Kottke, French, & Kirchner PTC July 23,

20 Table 4. Summary of confirmatory factor analysis results SampleModel 2 dfCFIRMSEA Absolute Loading MeanLoading Range Sample 1Five factor, covaried b * to.83 Revised five factor, covaried a ** to.92 Five factor, higher orderDid not converge Revised five factor, higher order a ** to 1.00 Two factor, covaried * to.82 Two factor, higher orderDid not converge Revised two factor, higher order a * to 1.00 Single factor * to.82 Sample 2Five factor, covaried b to.58 Revised five factor, covaried a ** to.71 Five factor, higher orderDid not converge Revised five factor, higher order a ** to 1.00 Two factor, covaried to.62 Two factor, higher orderDid not converge Revised two factor, higher order a ** to 1.00 Single factor to.62 Sample 3Five factor, covaried b ** to.61 Revised five factor, covaried a ** to.71 Five factor, higher orderDid not converge Revised five factor, higher order a ** to 1.00 Two factor, covaried345.70** to.52 Two factor, higher orderDid not converge Revised two factor, higher order a ** to 1.00 Single factor353.40** to.52 *p <.01, **p <.001 Kottke, French, & Kirchner PTC July 23,

21 Discussion To be fair to the test authors – They recommend the subscales be used for training and developmental purposes only – In all of their publications, they use only total scores – So, whether the test authors intended a hierarchical structure is not a certainty Yet, with their careful development that categorized the 14 KSAs into specific facets, one would expect meaningful structure Criterion validity has been found, but issues here as well – TKSA test correlates highly with general mental ability – Some incremental predictability Kottke, French, & Kirchner PTC July 23,

22 Conclusion Very limited evidence for adequate psychometric properties Use with caution – Recognize the tests limitations Kottke, French, & Kirchner PTC July 23,

23 Thank you Kottke, French, & Kirchner PTC July 23,

24 Questions? Kottke, French, & Kirchner PTC July 23,

25 References Kottke, French, & Kirchner PTC July 23,


Download ppt "To Use the Teamwork Test -- Or Not? A Psychometric Evaluation Janet L. Kottke California State University, San Bernardino Kimberly A. French University."

Similar presentations


Ads by Google