Presentation is loading. Please wait.

Presentation is loading. Please wait.

Assessing Cross- Cultural Competence: How Good Are the Available Instruments? Institute for Cross- Cultural Management Florida Institute of Technology.

Similar presentations


Presentation on theme: "Assessing Cross- Cultural Competence: How Good Are the Available Instruments? Institute for Cross- Cultural Management Florida Institute of Technology."— Presentation transcript:

1 Assessing Cross- Cultural Competence: How Good Are the Available Instruments? Institute for Cross- Cultural Management Florida Institute of Technology Institute for Cross- Cultural Management Florida Institute of Technology

2 2 Florida Tech – Institute for Cross-Cultural Management Authors William Gabrenya, PhD - Cross-Cultural Psychology (USA) Rana Moukarzel, MS - Industrial/Organizational Psychology (Lebanon) Marne Pomerance, MS – I/O (USA) Richard Griffith, PhD – I/O (USA) William Gabrenya, PhD - Cross-Cultural Psychology (USA) Rana Moukarzel, MS - Industrial/Organizational Psychology (Lebanon) Marne Pomerance, MS – I/O (USA) Richard Griffith, PhD – I/O (USA)

3 3 Florida Tech – Institute for Cross-Cultural Management Support DEOMI Contract FA2521-10-T-0087 Dan McDonald, PhD Patrice Reid, PhD DEOMI Contract FA2521-10-T-0087 Dan McDonald, PhD Patrice Reid, PhD

4 4 Florida Tech – Institute for Cross-Cultural Management Cross-Cultural Competence and the U.S. Military Cross-Cultural Psychology watches Iraq disintegrate –Case study in worst practices –Sidelined culture experts –How could this happen? Cross-Cultural Psychology watches Iraq disintegrate –Case study in worst practices –Sidelined culture experts –How could this happen?

5 5 Florida Tech – Institute for Cross-Cultural Management Culture Expertise Rediscovered Defense Language Office (DLO) DEOMI Army Research Institute Services DLO project to try to keep track of it all… Defense Language Office (DLO) DEOMI Army Research Institute Services DLO project to try to keep track of it all…

6 6 Florida Tech – Institute for Cross-Cultural Management Three Issues in Cross- Cultural Competence 1.Identify competencies 2.Measure competencies 3.Train competencies 4.Utilize capabilities acquired at considerable cost 1.Identify competencies 2.Measure competencies 3.Train competencies 4.Utilize capabilities acquired at considerable cost

7 7 Florida Tech – Institute for Cross-Cultural Management DLO Framework “Framework for Cross-Cultural Competency” Competency model –Organized set of requirements for acceptable performance –Higher level generality to lower level, hierarchy –Expressed behaviorally Two parts –3C competencies –Antecedent variables (enablers) Revised “Framework for Cross-Cultural Competency” Competency model –Organized set of requirements for acceptable performance –Higher level generality to lower level, hierarchy –Expressed behaviorally Two parts –3C competencies –Antecedent variables (enablers) Revised

8 8 Florida Tech – Institute for Cross-Cultural Management Competencies Name (# elements) Example C1Culture-General Concepts and Knowledge (3) Comprehends and navigates intercultural dynamics C3Cultural Perspective- Taking (3) Understands how one’s own group is viewed by members of another group C4Communication (2)Acquires and applies knowledge and concepts of intercultural communication skills C5Interpersonal Skills (2) Builds relationships in support of mission performance C6Cultural Adaptability (2) Understands the implications of one’s actions and adjusts approach to maintain relationships with other groups, or cultures

9 9 Florida Tech – Institute for Cross-Cultural Management Antecdents (Enablers) Name (# elements) Example E1Cognitive Bias Resilience (3) Accepts, or does not feel threatened by, ambiguous situations and uncertainty. E2Emotional Resilience (4) Tolerates emotionally shocking, frustrating, or exhausting circumstances… E3Self-Identity Resilience (3) Demonstrates ability to maintain personal values independent of situational factors E4Learning Motivation (4) Is motivated to make sense of inconsistent information about social rules and norms… E5Social Interaction (4)Actively seeks out and explores unfamiliar cross-cultural interactions …

10 10 Florida Tech – Institute for Cross-Cultural Management Validation* of the Framework Content validity –The right competencies? Criterion validity –Are these competencies related to performance and adjustment? *Analogously Content validity –The right competencies? Criterion validity –Are these competencies related to performance and adjustment? *Analogously

11 11 Florida Tech – Institute for Cross-Cultural Management Content Evaluation of the DLO Framework Developed using subject matter experts (SMEs): –Returned soldiers –Experts in expatriate assignments –Military psychologists We compared it to: –Military models –Civilian models –New SME research Short answer: content is good Developed using subject matter experts (SMEs): –Returned soldiers –Experts in expatriate assignments –Military psychologists We compared it to: –Military models –Civilian models –New SME research Short answer: content is good

12 12 Florida Tech – Institute for Cross-Cultural Management Criterion Evaluation Difficult… –The research is civilian –The measurement is poor Imprecise… –Mapping a competency model to common constructs Short answer: Middling support Difficult… –The research is civilian –The measurement is poor Imprecise… –Mapping a competency model to common constructs Short answer: Middling support

13 13 Florida Tech – Institute for Cross-Cultural Management This Report: Evaluate the quality of instruments that can assess Framework competencies Using “off the shelf” instruments Using non-cultural instruments –Personality, cognitive, etc. Using “off the shelf” instruments Using non-cultural instruments –Personality, cognitive, etc.

14 14 Florida Tech – Institute for Cross-Cultural Management Overview of the Procedure 1.Deconstruct the competency model to elemental form 2.Map elements to commonly studied constructs 3.Find “all” plausibly useful instruments 4.Evaluate the quality of the instruments 5.Map instruments to Framework elements 1.First: All instruments 2.Second: Only the good ones 6.Tally it up 1.Deconstruct the competency model to elemental form 2.Map elements to commonly studied constructs 3.Find “all” plausibly useful instruments 4.Evaluate the quality of the instruments 5.Map instruments to Framework elements 1.First: All instruments 2.Second: Only the good ones 6.Tally it up

15 15 Florida Tech – Institute for Cross-Cultural Management 1. Deconstruct the Framework Element Aka “component” in competency modeling C5: Interpersonal Skills “[C5.1] Develops and maintains rapport. Builds relationships in support of mission performance. // [C5.2] Manage and resolve conflict in support of mission objectives.” Element Aka “component” in competency modeling C5: Interpersonal Skills “[C5.1] Develops and maintains rapport. Builds relationships in support of mission performance. // [C5.2] Manage and resolve conflict in support of mission objectives.”

16 16 Florida Tech – Institute for Cross-Cultural Management 2. Map the elements to constructs What existing, researched construct does this competency seem to be related to?

17 Competency 1 Competency 2 Enabler 2 FrameworkConstructsMeasures Competency element 1.1 Competency element 2.1 Competency element 2.2 Enabler element 2.1 Enabler element 2.2 Personality Construct 1 Personality Construct 2 Competency element 1.2 Enabler 1 Enabler element 1.1 Enabler element 1.2 Attitude Construct 1 Attitude Construct 2 Cognition Construct 1 Cognition Construct 2 Abilities Construct 1 Abilities Construct 2 Instrument/Subscale 1 Instrument/Subscale 2 Instrument/Subscale 1 Instrument/Subscale 2 Instrument/Subscale 1 Instrument/Subscale 2 Instrument/Subscale 1 Instrument/Subscale 2

18 Competency 3: Cultural Perspective Taking Competency element 3.1: Demonstrates an awareness of one’s own worldview… and how one’s own group is viewed… Competency element 3.3: Takes the cultural context into consideration when interpreting situational cues. Knowledge of attributed stereotypes Self-insight Perspective taking Metacognition Situational Awareness MAKSS- Awareness ICC-Awareness SEE-Empathic perspective taking SEE-Empathic Awareness BEVI- Sociocultural Closure (Perspective taking) CQS- Metacognition

19 19 Florida Tech – Institute for Cross-Cultural Management 3. Find ALL Instruments Hunting and gathering in a vast literature Two kinds: –Single-construct –Batteries Three business models (why this matters) –Open –Restricted –Proprietary Hunting and gathering in a vast literature Two kinds: –Single-construct –Batteries Three business models (why this matters) –Open –Restricted –Proprietary

20 20 Florida Tech – Institute for Cross-Cultural Management Many Instruments Don’t trust the published lists We unfairly divided scales into “primary” and “secondary” Don’t trust the published lists We unfairly divided scales into “primary” and “secondary”

21 21 Florida Tech – Institute for Cross-Cultural Management Primary Instruments NameSubscales CCAICross-Cultural Adaptability Inventory Flexibility/Openness (FO); Emotional Resilience (ER); Perceptual Acuity (PAC) ; Personal Autonomy (PA) CQSCultural Intelligence Scale Metacognition ; Cognition ; Motivation ; Behavior ICAPSIntercultural Adjustment Potential Scale Emotion Regulation (ER); Openness (OP); Flexibility (FL); Creativity (CT) IDIIntercultural Development Inventory Denial/Defense (DD); Reversal [R]; Minimization (M); Acceptance/Adaptation (AA); Cultural Disengagement (CD) MPQMulticultural Personality Questionnaire Cultural Empathy; Openmindedness; Social Initiative; Emotional Stability; Flexibility ICCIntercultural Communicative Competence Complex structure: 3 domains; 5 dimensions; 4 developmental levels INCAIntercultural Competence Assessment 6 subscales

22 22 Florida Tech – Institute for Cross-Cultural Management NameSubscales CCAICross-Cultural Adaptability Inventory Flexibility/Openness (FO); Emotional Resilience (ER); Perceptual Acuity (PAC) ; Personal Autonomy (PA) IESIntercultural Effectiveness Scale Continuous Learning; Interpersonal Engagement; Hardiness ISSIntercultural Sensitivity Scale Interaction Engagement; Respect for Cultural Differences; Interaction Confidence; Interaction Enjoyment; Interaction Attentiveness SEEScale of Ethnocultural Empathy Empathic Feeling and Expression (EFE); Empathic Perspective Taking (EP); Acceptance of Cultural Differences (AC); Empathic Awareness (EA)

23 23 Florida Tech – Institute for Cross-Cultural Management Secondary Instruments Instrument ADSAdjustment Difficulties Subscale AICBeliefs, Events, and Values Inventory ASSISAcculturative Stress Scale for International Students BASICBehavioral Assessment Scale for Intercultural Communication Effectiveness BEVIBeliefs, Events, and Values Inventory CCSICross-Cultural Social Intelligence CGAICCulture-Generic Approach to Intercultural Competence CWQThe Culture in the Workplace Questionnaire EMMICEuropean Multidimensional Models of Intercultural Competence GAP Test Global Awareness Profile ICSIIntercultural Sensitivity Inventory Instrument IRCIntercultural Readiness Checklist IRIInterpersonal Reactivity Index ISASThe Inventory of Student Adjustment Strain MAKS S Multicultural Competence Scale MASQ UE Munroe Multicultural Attitude Scale Questionnaire MCIMulticultural Competence Scale MCKA S Multicultural Competence Scale PCATPeterson Cultural Awareness Test PCSIPeterson Cultural Style Indicator SCSSocial Connectedness Scale WDSWorkplace Diversity Survey

24 24 Florida Tech – Institute for Cross-Cultural Management Validation of the Instruments Primary –Face –Construct –Criterion Secondary –Criterion Primary –Face –Construct –Criterion Secondary –Criterion

25 25 Florida Tech – Institute for Cross-Cultural Management Face Validity Good: Most instruments Bad: ICAPS (Intercultural Adjustment Potential Scale) Moderate: CQS (Cultural Intelligence Scale) Good: Most instruments Bad: ICAPS (Intercultural Adjustment Potential Scale) Moderate: CQS (Cultural Intelligence Scale)

26 26 Florida Tech – Institute for Cross-Cultural Management Construct Validity Convergent validity –Similar to what it should be similar to? Divergent validity –Different than what it should be different than? Structure –If subscales: support for dimensionality? Convergent validity –Similar to what it should be similar to? Divergent validity –Different than what it should be different than? Structure –If subscales: support for dimensionality?

27 27 Florida Tech – Institute for Cross-Cultural Management Construct Validity Findings (Primary scales only) Missing information Diverse outcomes, moderate validity Mushy Method variance not accounted for (Primary scales only) Missing information Diverse outcomes, moderate validity Mushy Method variance not accounted for

28 28 Florida Tech – Institute for Cross-Cultural Management Construct Validity Findings Good: MPQ (Multicultural Personality Questionnaire) Mixed: most instruments Bad: ICAPS, ISS (Intercultural Sensitivity Scale) Unknown: GCI, IES, INCA, ICC (mainly proprietary) Good: MPQ (Multicultural Personality Questionnaire) Mixed: most instruments Bad: ICAPS, ISS (Intercultural Sensitivity Scale) Unknown: GCI, IES, INCA, ICC (mainly proprietary)

29 29 Florida Tech – Institute for Cross-Cultural Management Criterion Validity Does the scale predict something real? –Adjustment: psychological or sociocultural –Performance –Response to training in experiments Known-groups –Do groups differ on the scale the way they should? Does the scale predict something real? –Adjustment: psychological or sociocultural –Performance –Response to training in experiments Known-groups –Do groups differ on the scale the way they should?

30 30 Florida Tech – Institute for Cross-Cultural Management Criterion Validity Findings Good: MPQ (Multicultural Personality Questionnaire) Moderate: CCAI, CQS, SEE Mixed: ICAPS Poor: ISS Unknown: GCI, IES, INCA Good: MPQ (Multicultural Personality Questionnaire) Moderate: CCAI, CQS, SEE Mixed: ICAPS Poor: ISS Unknown: GCI, IES, INCA

31 31 Florida Tech – Institute for Cross-Cultural Management Conclusion: Validity Few scales with overall high validity Many highly desirable scales have little support Few scales with overall high validity Many highly desirable scales have little support

32 32 Florida Tech – Institute for Cross-Cultural Management 5. Map (good) instruments to Framework elements So: can the Framework be assessed? Strategy: Estimate extent to which each element is assessed using high and moderate validity instruments So: can the Framework be assessed? Strategy: Estimate extent to which each element is assessed using high and moderate validity instruments

33 Competency 1 Competency 2 Enabler 2 FrameworkConstructsMeasures Competency element 1.1 Competency element 2.1 Competency element 2.2 Enabler element 2.1 Enabler element 2.2 Personality Construct 1 Personality Construct 2 Competency element 1.2 Enabler 1 Enabler element 1.1 Enabler element 1.2 Attitude Construct 1 Attitude Construct 2 Cognition Construct 1 Cognition Construct 2 Abilities Construct 1 Abilities Construct 2 Instrument/Subscale 1 Instrument/Subscale 2 Instrument/Subscale 1 Instrument/Subscale 2 Instrument/Subscale 1 Instrument/Subscale 2 Instrument/Subscale 1 Instrument/Subscale 2

34 Competency 3: Cultural Perspective Taking Competency element 3.1: Demonstrates an awareness of one’s own worldview… and how one’s own group is viewed… Competency element 3.3: Takes the cultural context into consideration when interpreting situational cues. Knowledge of attributed stereotypes Self-insight Perspective taking Metacognition Situational Awareness MAKSS- Awareness ICC-Awareness SEE-Empathic perspective taking SEE-Empathic Awareness BEVI- Sociocultural Closure (Perspective taking) CQS- Metacognition

35 35 Florida Tech – Institute for Cross-Cultural Management Score Card: Competencies CompetencyName (# elements)Rating C1Knowledge (3)* C3Perspective-Taking (3)*** C4Communication (2)** C5Interpersonal Skills (2)** C6Cultural Adaptability (2)****

36 36 Florida Tech – Institute for Cross-Cultural Management Score Card: Enablers EnablerNameRating E1.1Ambiguity tolerance(****) E1.2Need for closure(****) E1.3Suspending judgment* E1.4Inclusiveness** E2.1Stress resilience** E2.2Emotion regulation**** E3.1Self-confidence(****) E3.2Self-identity**** E3.3Optimism(**)

37 37 Florida Tech – Institute for Cross-Cultural Management More Enablers EnablerName (# elements)Rating E4.1Learning through observation (3) 0 E4.2Inquisitiveness(*) E5.1Social flexibility (3)**** E5.2Willingness to engage***

38 38 Florida Tech – Institute for Cross-Cultural Management So How Do We Measure a Competency? Plan A: off the shelf self-report measures –Mixed value –Various problems with self-report measures Faking, response biases, cognitive/declarative knowledge, etc. Plan B: behavior-based assessment centers –Designed for each competency –Difficult, expensive, slow Plan A: off the shelf self-report measures –Mixed value –Various problems with self-report measures Faking, response biases, cognitive/declarative knowledge, etc. Plan B: behavior-based assessment centers –Designed for each competency –Difficult, expensive, slow

39 39 Florida Tech – Institute for Cross-Cultural Management Another Way… 1. Rewrite the Framework as a blended competency/causal model Competency model: –Just the right competencies for the job Causal model: –Related competencies and enablers in models –Add moderators; mediation 1. Rewrite the Framework as a blended competency/causal model Competency model: –Just the right competencies for the job Causal model: –Related competencies and enablers in models –Add moderators; mediation

40 40 Florida Tech – Institute for Cross-Cultural Management 2. Assess the model, not just the competencies For each competency, assess several variables in the model –“Triangulate” on the individual’s competency –Relieves the pressure on assessing the competency itself 2. Assess the model, not just the competencies For each competency, assess several variables in the model –“Triangulate” on the individual’s competency –Relieves the pressure on assessing the competency itself

41 41 Florida Tech – Institute for Cross-Cultural Management Blended Model Openness Multicultural Attitudes E4.2: Culture Inquisitiveness E4.1.2: Learning Through Observation C1.1a: Culture- General Knowledge Need for Cognition C: Competencies Requiring Knowledge Situational & External Antecedents

42 42 Florida Tech – Institute for Cross-Cultural Management fin


Download ppt "Assessing Cross- Cultural Competence: How Good Are the Available Instruments? Institute for Cross- Cultural Management Florida Institute of Technology."

Similar presentations


Ads by Google