1. Assessing Impact: Study Abroad at Two Institutions Elaine Meyer-Lee Saint Mary’s College Carol Warfield Auburn University 2006 AIEA Annual Conference February, 2006

2. Why Assess Outcomes? 1,022 US institutions reported information on their study abroad programs 1,022 US institutions reported information on their study abroad programs 174,629 U.S. students studied abroad 174,629 U.S. students studied abroad Data from Open Doors Report on International Educational Exchange 2003-2004

3. Inform continuous quality improvement Inform continuous quality improvement Advocate for international education Advocate for international education Satisfy regional accrediting bodies Satisfy regional accrediting bodies Add to basic understanding of student growth in this area Add to basic understanding of student growth in this area

4. Young, wide-open field Young, wide-open field Much need for better methods and more data on many different dimensions Much need for better methods and more data on many different dimensions This presentation is not a step-by-step overview, but two examples This presentation is not a step-by-step overview, but two examples (See Forum’s upcoming Guide to Outcomes Assessment in Education Abroad)

5. Multi-method Longitudinal Assessment of Intercultural Engagement, IC Sensitivity, Identity, and Goals: Preliminary Results Elaine Meyer-Lee Director of International & Intercultural Learning Joy Evans Research Analyst-Center for Women’s InterCultural Leadership Saint Mary’s College Notre Dame, Indiana 206 AIEA Annual Conference February 25, 2006

6. Presentation Overview Context Context Assessment Methods Assessment Methods Review of Instruments Review of Instruments Preliminary Results Preliminary Results Contributions to Study Abroad Assessment Contributions to Study Abroad Assessment

7. Saint Mary’s College Catholic Women’s Liberal Art Catholic Women’s Liberal Art Established 1844 Established 1844 About 1600 enrollment About 1600 enrollment 98% full-time98% full-time 82% on-campus82% on-campus 85% Catholic85% Catholic 9% students of color9% students of color 20% of students study abroad

8. Mission of the Center for Women’s InterCultural Leadership Saint Mary's College's mission includes preparing women to make a difference in an increasingly complex and interdependent world. The role of the Center for Women's InterCultural Leadership (CWIL) within this mission is to foster the intercultural knowledge and competence critical to educating the next generation of women leaders.

9. CWIL at Saint Mary’s Encompasses much more than study abroad (see brochures) Encompasses much more than study abroad (see brochures) Is adopting Byram’s definition of intercultural competence Is adopting Byram’s definition of intercultural competence Is launching a portfolio-based certificate program in Women’s Intercultural Leadership Is launching a portfolio-based certificate program in Women’s Intercultural Leadership

10. Study Abroad Programs Year/ Semester Ireland (Year only) India (Semester only) Rome Spain France Australia Austria South Africa

11. 4-2 Weeks European Studies (4) Internships in Europe (4+) Honduras(3) Ecuador(3-2) Greece (2 ½) S. Korea (2) 1 Week Prague/Poland Jamaica Mexico Study Abroad Programs

12. Assessing Study Abroad… Going Beyond Student Satisfaction

13. Evaluation & Assessment Goals Evaluate programs’ effectiveness Evaluate programs’ effectiveness Measure Students’: Measure Students’: Changes in sensitivity to cultural differencesChanges in sensitivity to cultural differences American identity developmentAmerican identity development Own sense of growth toward goalsOwn sense of growth toward goals Explore correlations of change with: Explore correlations of change with: Intercultural engagement while abroad (interaction and reflection)Intercultural engagement while abroad (interaction and reflection) Program or demographic characteristicsProgram or demographic characteristics

14. Assessment Methods/Design Longitudinal Research Design Longitudinal Research Design Pre-test/Post-test (some grad follow-up)Pre-test/Post-test (some grad follow-up) Procedures – 70% response rate so farProcedures – 70% response rate so far Control Group Control Group 30 First-year students30 First-year students Focus Group Focus Group Study Abroad Re-entry CourseStudy Abroad Re-entry Course Written reflection and Kolb’s LSIWritten reflection and Kolb’s LSI

15. Assessment Instruments: Pre-Test Study Abroad Participants Intercultural Development Inventory (IDI) Intercultural Development Inventory (IDI) American Identity Questionnaire American Identity Questionnaire Goals/motivations from Saint Mary’s College Study Abroad Survey Goals/motivations from Saint Mary’s College Study Abroad Survey Control Group Intercultural Development Inventory (IDI) American Identity Questionnaire

16. Assessment Instruments: Post-Test Study Abroad Participants Intercultural Development Inventory (IDI) Intercultural Development Inventory (IDI) American Identity Questionnaire (AIQ) American Identity Questionnaire (AIQ) SMC Study Abroad Survey Program Impact & Measure of Intercultural Engagement (MIE) SMC Study Abroad Survey Program Impact & Measure of Intercultural Engagement (MIE) Control Group Intercultural Development Inventory (IDI) American Identity Questionnaire

17. Data Collection Timeline 2004-20052005-20062006-2007 FallSpringFallSpringFallSpring Pre-testN=16----Pre-testN=44---- ----------  Pre-test N=83----  Control N=32----  Post-testN=(16)Post-testN=(44)Post-test N=(6)---------  ----------------  Pre-test N=(?)---------  Spring Semester Post-test N=(48?)--  Fall Semester -----------  Pre-test? N=(?)----  SummerPost-testN=(29?)YearControlN=(32)Post-testN=(?)Post-test?N=(?)

18. Review of Instruments

19. Intercultural Development Inventory (IDI) 50-item Self-Assessment 50-item Self-Assessment Valid & Reliable Psychometric Measure Valid & Reliable Psychometric Measure Culture general in focus Culture general in focus Quantifies the subjective experience of cultural difference Quantifies the subjective experience of cultural difference Most widely used currently Most widely used currently Based on Milton Bennett’s Developmental Model of Intercultural Sensitivity (constructivist stage model) Based on Milton Bennett’s Developmental Model of Intercultural Sensitivity (constructivist stage model)

20. Slide Available Online:http://www.coe.int/T/E/North-South_Centre/Programmes/3_Global_Education/c_GENE/GENE%20Web- page%20-%20(3)%20Hesse%20article.asp

21. IDI Measurement Scales Denial/Defense Denial/Defense Reversal Reversal Minimization Minimization Acceptance/Adaptation Acceptance/Adaptation Encapsulated Marginality Encapsulated Marginality

22. Overall Profile

23. American Identity Questionnaire 10-item Likert-style Self Assessment 10-item Likert-style Self Assessment Adaptation of Jean Phinney’s Multigroup Ethnic Identity Measure (good track record) Adaptation of Jean Phinney’s Multigroup Ethnic Identity Measure (good track record) Based on Eriksonian identity development theory. Measures 2 Factors: Based on Eriksonian identity development theory. Measures 2 Factors: identity search (e.g. I have often talked to other people about what it means to be an American.)identity search (e.g. I have often talked to other people about what it means to be an American.) affirmation, belonging, and commitment (e.g. Being American plays an important part in my life.)affirmation, belonging, and commitment (e.g. Being American plays an important part in my life.) One goal of study abroad is more complexity in understanding own culture, and students in pilot focus group felt it captured an important dimension One goal of study abroad is more complexity in understanding own culture, and students in pilot focus group felt it captured an important dimension Currently refining adaptation’s psychometric properties Currently refining adaptation’s psychometric properties

24. Saint Mary’s College Study Abroad Survey Semester/Year-Long Programs Core Survey Core Survey 111 Items111 Items 4 open-ended questions4 open-ended questions Customized Addendum Customized Addendum Summer/Short-term Programs Core Survey Core Survey 58 Items 1 open-ended questions Customized Addendum Customized Addendum

25. Saint Mary’s College Study Abroad Survey Semester/Year-Long Programs Question Sections Include: Question Sections Include: Demographics/BackgroundDemographics/Background GoalsGoals Pre-DeparturePre-Departure Academic ProgramAcademic Program Cultural Tours & ExcursionsCultural Tours & Excursions On-Site Student ServicesOn-Site Student Services Intercultural EngagementIntercultural Engagement Overview of ProgramOverview of Program Program Impact (Goals)Program Impact (Goals) Volunteer InterestsVolunteer Interests Summer/Short-term Programs Question Sections Include: Question Sections Include: Demographics Pre-Departure Living Arrangements Intercultural Engagement Overview of Program Volunteer Interests

26. Self-reported Growth in: Understanding of different cultures Understanding of different cultures Different perspective on American culture Different perspective on American culture Greater understanding of the US in world affairs and history Greater understanding of the US in world affairs and history Different perspective on your ethnic heritage Different perspective on your ethnic heritage Understanding of my faith Understanding of my faith Self-awareness Self-awareness Maturity and self-confidence Maturity and self-confidence Interpersonal skills and the ability to adapt Interpersonal skills and the ability to adapt Intellectual development in the liberal arts Intellectual development in the liberal arts Ability to communicate in a foreign language Ability to communicate in a foreign language

27. Measure of Intercultural Engagement Asks students to quantify the frequency of their engagement through: Interaction with cultural resources Interaction with cultural resources Interaction with local people Interaction with local people Explicit reflection on these interactions Explicit reflection on these interactions

28. Interaction with cultural resources Theater/opera/symphony/concerts Theater/opera/symphony/concerts Museums/ historical sites Museums/ historical sites Forests/parks/beaches Forests/parks/beaches Traveling outside of the host country Traveling outside of the host country Traveling within the host country Traveling within the host country

29. Interaction with local people Shopping/visiting local markets Shopping/visiting local markets Movies/entertainment catering to locals Movies/entertainment catering to locals Restaurants/pubs/cafes catering to locals Restaurants/pubs/cafes catering to locals Visiting local families/friends in their homes Visiting local families/friends in their homes Participating in worship services Participating in worship services On-campus informal conversations On-campus informal conversations Events sponsored by the study abroad program Events sponsored by the study abroad program Extra-curricular activities (student clubs, intramurals, etc.) Extra-curricular activities (student clubs, intramurals, etc.) Volunteering, service-learning, internships, shadowing Volunteering, service-learning, internships, shadowing Reading or listening to local news Reading or listening to local news

30. Explicit reflection on these interactions Journaling/Blogging Journaling/Blogging Photography Photography Other artistic expression (painting, drawing, collages, etc.) Other artistic expression (painting, drawing, collages, etc.) Class discussions Class discussions Informal conversations with locals Informal conversations with locals Informal conversations with students studying abroad Informal conversations with students studying abroad Calling/e-mailing family/friends/faculty in America about experience Calling/e-mailing family/friends/faculty in America about experience

31. Pre-test Results IDI Worldview Profile: Pre-Abroad

32. IDI Worldview Profile “Pre”: Controls

33. Pre-test Comparison to Control: American Identity Questionnaire Affirmation, belonging and commitment Affirmation, belonging and commitment  no significant difference Identity search Identity search  statistical significance (at the.05 level)  statistical significance (at the.05 level)

34. Summary of Pre-test Findings IDI: Students’ scores do not support developmental model IDI: Students’ scores do not support developmental model IDI: Low Minimization Score IDI: Low Minimization Score Catholic beliefs ….We’re all God’s childrenCatholic beliefs ….We’re all God’s children Control on both: Heritage Seeker Factor Control on both: Heritage Seeker Factor Are study abroad participants a different population than other SMC students?Are study abroad participants a different population than other SMC students?

35. Preliminary Analysis of Change Data for Matched Set No significant change in means for IDI or AIQ yet (on paired T-test on pre-post mean scores, n=42 and 27) No significant change in means for IDI or AIQ yet (on paired T-test on pre-post mean scores, n=42 and 27) BUT, high self-reported growth (1.7 when 1 is high) and wide range in change scores and program means BUT, high self-reported growth (1.7 when 1 is high) and wide range in change scores and program means

36. What about those who do change? Correlation of AIQ with IDI scales (bivariate non-parametric correlations) Correlation of AIQ with IDI scales (bivariate non-parametric correlations) Correlations with Intercultural Engagement: Correlations with Intercultural Engagement: Cultural resource scale with AIQ changeCultural resource scale with AIQ change Overall and reflection with program impact (especially interpersonal items)Overall and reflection with program impact (especially interpersonal items) 50% did grow on at least one scale (defined as 2 standard deviations of positive change, n=21) 50% did grow on at least one scale (defined as 2 standard deviations of positive change, n=21)

37. How do the “growers” compare? CharacteristicGrowersNon-Growers From N.America 91%95% White67%86% Latina24%4.5% Never been abroad53%63% Abroad < 3 months24%26% Whole year program10%46% Semester program76%46% Summer program14%8% Rome program24%32% Seville program24%5% Mexico14%9% France 14%9% Ireland10%46%

38. Methodological Findings Importance and Challenges of Control Group Importance and Challenges of Control Group Procedural effects (e.g. on-line vs. paper) Procedural effects (e.g. on-line vs. paper) Signs of internal consistency in measures (inter-item correlations) Signs of internal consistency in measures (inter-item correlations)

39. Intercultural Development Inventory: Intercultural Development Inventory: Standardized instrument Standardized instrument Ability to compare study abroad program outcomes with domestic intercultural programs Ability to compare study abroad program outcomes with domestic intercultural programs ? Intercultural Development Inventory: ?Requires qualifying training to administer ?Highly rigorous and proprietary ?Labor- and cost- intensive for large scale use Benefits to Study Abroad Assessment & Challenges to Institutional Research

40. Program Impact & MEIM Program Impact & MEIM Comparable & program specific data on students’ intercultural interactions Comparable & program specific data on students’ intercultural interactions AIQ AIQ Examine identity dimension, connect to literature on this Examine identity dimension, connect to literature on this ? Program Impact & MEIM ?Labor-intensive development and administration ? AIQ ?Reliability and Validity of adaptation ?Needs refining Benefits to Study Abroad Assessment & Challenges to Institutional Research

41. Next Steps Analysis Analysis More on change and changers with full datasetMore on change and changers with full dataset More on correlations with intercultural engagement, program variables (including Engle & Engle’s category), and student variables (including original motivation)More on correlations with intercultural engagement, program variables (including Engle & Engle’s category), and student variables (including original motivation) QualitativeQualitative Psychometrics on AIQ, Program Impact, MIEPsychometrics on AIQ, Program Impact, MIE Data collection Data collection Add 1 year follow-upAdd 1 year follow-up Draw on Women’s Intercultural Leadership Portfolios?Draw on Women’s Intercultural Leadership Portfolios? Applications Applications Many improvements to (and rationales for) programs already. Overhauling re-entry course.Many improvements to (and rationales for) programs already. Overhauling re-entry course.

42. For more handouts or bibliography on assessing intercultural outcomes of study abroad programs, go to http://www.saintmarys.edu/~cwil/php/intercultural.learning/ IILOutcomes.php http://www.saintmarys.edu/~cwil/php/intercultural.learning/ IILOutcomes.php http://www.saintmarys.edu/~cwil/php/intercultural.learning/ IILOutcomes.php http://www.saintmarys.edu/~cwil/php/intercultural.learning/ IILOutcomes.php or email me: meyerlee@saintmarys.edu