Presentation on theme: "Institute for Cross-Cultural Management"— Presentation transcript:
1 Assessing Cross-Cultural Competence: How Good Are the Available Instruments? Institute for Cross-Cultural ManagementFlorida Institute of Technology
2 Authors William Gabrenya, PhD - Cross-Cultural Psychology (USA) Rana Moukarzel, MS - Industrial/Organizational Psychology (Lebanon)Marne Pomerance, MS – I/O (USA)Richard Griffith, PhD – I/O (USA)
3 Support DEOMI Contract FA2521-10-T-0087 Dan McDonald, PhD Patrice Reid, PhD
4 Cross-Cultural Competence and the U.S. Military Cross-Cultural Psychology watches Iraq disintegrateCase study in worst practicesSidelined culture expertsHow could this happen?
5 Culture Expertise Rediscovered Defense Language Office (DLO)DEOMIArmy Research InstituteServicesDLO project to try to keep track of it all…
6 Three Issues in Cross-Cultural Competence Identify competenciesMeasure competenciesTrain competenciesUtilize capabilities acquired at considerable cost
7 DLO Framework “Framework for Cross-Cultural Competency” Competency modelOrganized set of requirements for acceptable performanceHigher level generality to lower level, hierarchyExpressed behaviorallyTwo parts3C competenciesAntecedent variables (enablers)Revised
8 Competencies Name (# elements) Example C1 Culture-General Concepts and Knowledge (3)Comprehends and navigates intercultural dynamicsC3Cultural Perspective-Taking (3)Understands how one’s own group is viewed by members of another groupC4Communication (2)Acquires and applies knowledge and concepts of intercultural communication skillsC5Interpersonal Skills (2)Builds relationships in support of mission performanceC6Cultural Adaptability (2)Understands the implications of one’s actions and adjusts approach to maintain relationships with other groups, or cultures
9 Antecdents (Enablers) Name (# elements)ExampleE1Cognitive Bias Resilience (3)Accepts, or does not feel threatened by, ambiguous situations and uncertainty.E2Emotional Resilience (4)Tolerates emotionally shocking, frustrating, or exhausting circumstances…E3Self-Identity Resilience (3)Demonstrates ability to maintain personal values independent of situational factorsE4Learning Motivation (4)Is motivated to make sense of inconsistent information about social rules and norms…E5Social Interaction (4)Actively seeks out and explores unfamiliar cross-cultural interactions …
10 Validation* of the Framework Content validityThe right competencies?Criterion validityAre these competencies related to performance and adjustment?*Analogously
11 Content Evaluation of the DLO Framework Developed using subject matter experts (SMEs):Returned soldiersExperts in expatriate assignmentsMilitary psychologistsWe compared it to:Military modelsCivilian modelsNew SME researchShort answer: content is good
12 Criterion Evaluation Difficult… Imprecise… The research is civilianThe measurement is poorImprecise…Mapping a competency model to common constructsShort answer: Middling support
13 This Report: Evaluate the quality of instruments that can assess Framework competencies Using “off the shelf” instrumentsUsing non-cultural instrumentsPersonality, cognitive, etc.
14 Overview of the Procedure Deconstruct the competency model to elemental formMap elements to commonly studied constructsFind “all” plausibly useful instrumentsEvaluate the quality of the instrumentsMap instruments to Framework elementsFirst: All instrumentsSecond: Only the good onesTally it up
15 1. Deconstruct the Framework ElementAka “component” in competency modelingC5: Interpersonal Skills“[C5.1] Develops and maintains rapport. Builds relationships in support of mission performance. // [C5.2] Manage and resolve conflict in support of mission objectives.”
16 2. Map the elements to constructs What existing, researched construct does this competency seem to be related to?
17 Framework Constructs Measures Competency 1Competency element 1.1Personality Construct 1Personality Construct 2Instrument/Subscale 1Instrument/Subscale 2Competency element 1.2Competency 2Competency element 2.1Attitude Construct 1Attitude Construct 2Instrument/Subscale 1Instrument/Subscale 2Competency element 2.2Enabler 1Cognition Construct 1Cognition Construct 2Instrument/Subscale 1Instrument/Subscale 2Enabler element 1.2Enabler element 1.1Enabler 2Abilities Construct 1Abilities Construct 2Enabler element 2.1Instrument/Subscale 1Instrument/Subscale 2Enabler element 2.2
18 Competency 3: Cultural Perspective Taking MAKSS-AwarenessICC-AwarenessSEE-Empathic perspective takingSEE-Empathic AwarenessBEVI-Sociocultural Closure(Perspective taking)Competency element 3.1: Demonstrates an awareness of one’s own worldview… and how one’s own group is viewed…Knowledge of attributed stereotypesSelf-insightPerspective takingCompetency element 3.3: Takes the cultural context into consideration when interpreting situational cues.MetacognitionSituational AwarenessCQS-Metacognition
19 3. Find ALL Instruments Hunting and gathering in a vast literature Two kinds:Single-constructBatteriesThree business models (why this matters)OpenRestrictedProprietary
20 Many Instruments Don’t trust the published lists We unfairly divided scales into “primary” and “secondary”
22 Name Subscales CCAI Cross-Cultural Adaptability Inventory Flexibility/Openness (FO); Emotional Resilience (ER); Perceptual Acuity (PAC)； Personal Autonomy (PA)IESIntercultural Effectiveness ScaleContinuous Learning; Interpersonal Engagement; HardinessISSIntercultural Sensitivity ScaleInteraction Engagement; Respect for Cultural Differences; Interaction Confidence; Interaction Enjoyment; Interaction AttentivenessSEEScale of Ethnocultural EmpathyEmpathic Feeling and Expression (EFE); Empathic Perspective Taking (EP); Acceptance of Cultural Differences (AC); Empathic Awareness (EA)
23 Secondary Instruments ADSAdjustment Difficulties SubscaleAICBeliefs, Events, and Values InventoryASSISAcculturative Stress Scale for International StudentsBASICBehavioral Assessment Scale for Intercultural Communication EffectivenessBEVICCSICross-Cultural Social IntelligenceCGAICCulture-Generic Approach to Intercultural CompetenceCWQThe Culture in the Workplace QuestionnaireEMMICEuropean Multidimensional Models of Intercultural CompetenceGAP TestGlobal Awareness ProfileICSIIntercultural Sensitivity InventoryInstrumentIRCIntercultural Readiness ChecklistIRIInterpersonal Reactivity IndexISASThe Inventory of Student Adjustment StrainMAKSSMulticultural Competence ScaleMASQUEMunroe Multicultural Attitude Scale QuestionnaireMCIMCKASPCATPeterson Cultural Awareness TestPCSIPeterson Cultural Style IndicatorSCSSocial Connectedness ScaleWDSWorkplace Diversity Survey
24 Validation of the Instruments PrimaryFaceConstructCriterionSecondary
25 Face Validity Good: Most instruments Bad: ICAPS (Intercultural Adjustment Potential Scale)Moderate: CQS (Cultural Intelligence Scale)
26 Construct Validity Convergent validity Divergent validity Structure Similar to what it should be similar to?Divergent validityDifferent than what it should be different than?StructureIf subscales: support for dimensionality?
27 Construct Validity Findings (Primary scales only)Missing informationDiverse outcomes, moderate validityMushyMethod variance not accounted for
29 Criterion Validity Does the scale predict something real? Known-groups Adjustment: psychological or socioculturalPerformanceResponse to training in experimentsKnown-groupsDo groups differ on the scale the way they should?
31 Conclusion: Validity Few scales with overall high validity Many highly desirable scales have little support
32 5. Map (good) instruments to Framework elements So: can the Framework be assessed?Strategy: Estimate extent to which each element is assessed using high and moderate validity instruments
33 Framework Constructs Measures Competency 1Competency element 1.1Personality Construct 1Personality Construct 2Instrument/Subscale 1Instrument/Subscale 2Competency element 1.2Competency 2Competency element 2.1Attitude Construct 1Attitude Construct 2Instrument/Subscale 1Instrument/Subscale 2Competency element 2.2Enabler 1Cognition Construct 1Cognition Construct 2Instrument/Subscale 1Instrument/Subscale 2Enabler element 1.2Enabler element 1.1Enabler 2Abilities Construct 1Abilities Construct 2Enabler element 2.1Instrument/Subscale 1Instrument/Subscale 2Enabler element 2.2
34 Competency 3: Cultural Perspective Taking MAKSS-AwarenessICC-AwarenessSEE-Empathic perspective takingSEE-Empathic AwarenessBEVI-Sociocultural Closure(Perspective taking)Competency element 3.1: Demonstrates an awareness of one’s own worldview… and how one’s own group is viewed…Knowledge of attributed stereotypesSelf-insightPerspective takingCompetency element 3.3: Takes the cultural context into consideration when interpreting situational cues.MetacognitionSituational AwarenessCQS-Metacognition
36 Score Card: Enablers Enabler Name Rating E1.1 Ambiguity tolerance (****)E1.2Need for closureE1.3Suspending judgment*E1.4Inclusiveness**E2.1Stress resilienceE2.2Emotion regulation****E3.1Self-confidenceE3.2Self-identityE3.3Optimism(**)
37 More Enablers Enabler Name (# elements) Rating E4.1 Learning through observation (3)E4.2Inquisitiveness(*)E5.1Social flexibility (3)****E5.2Willingness to engage***
38 So How Do We Measure a Competency? Plan A: off the shelf self-report measuresMixed valueVarious problems with self-report measuresFaking, response biases, cognitive/declarative knowledge, etc.Plan B: behavior-based assessment centersDesigned for each competencyDifficult, expensive, slow
39 Another Way…1. Rewrite the Framework as a blended competency/causal modelCompetency model:Just the right competencies for the jobCausal model:Related competencies and enablers in modelsAdd moderators; mediation
40 2. Assess the model, not just the competencies For each competency, assess several variables in the model“Triangulate” on the individual’s competencyRelieves the pressure on assessing the competency itself
41 Blended Model C1.1a: Culture-General Knowledge Openness E4.1.2: Learning Through ObservationE4.2: Culture InquisitivenessMulticultural AttitudesNeed for CognitionC: Competencies Requiring KnowledgeSituational & External Antecedents