Assessing a Campus-wide Internationalization Initiative: Aligning Assessments with Institutional Outcomes M. David Miller Timothy S. Brophy University of Florida Presented at Annual Meeting of the Association for the Assessment of Learning in Higher Education, Albuquerque, NM, June, 2014.

Quality Enhancement Plan - SACSCOC Learning Without Borders: Internationalizing the Gator Nation Based on institutional assessment Focuses on learning outcomes and enhances student learning environment Includes broad-based involvement of institutional constituencies in development and implementation Identifies initiatives to promote learning outcomes Assesses their achievement for the institution

Development of the Initiative – Broad Based Involvement Internationalization Task Force (ITF) formed in January 2011 ITF has representatives nominated by the Deans of units across campus ITF involves all stakeholders – faculty, students, administration, staff and others

Alignment Student Learning Outcomes Assessment Campus Initiatives

Internationalization Internationalization is the conscious integration of intercultural and global competencies into student learning. GLOBAL CITIZENSHIP Students comprehend the trends, challenges, and opportunities that affect our communities and communities worldwide.. CONTENT: SLO 1. Students identify, describe, and explain global and intercultural conditions and interdependencies CRITICAL THINKING: SLO 2. Students analyze and interpret global and intercultural issues COMMUNICATION: SLO 3. Students communicate effectively with members of other cultures INTERCULTURAL COMMUNICATIONS Students interact effectively with members of other cultures.

Initiatives Curriculum EnhancementStudy AbroadCampus LifeInternational Calendar and International Event SupportInternational Scholars Program

For students: new and exciting courses for the International Scholars Program For faculty: support and professional development opportunities to develop International Scholar courses Curriculum Enhancement

For students: more scholarships and exciting new programs, especially in underrepresented disciplines For faculty: support and professional development opportunities to develop Study Abroad programs Study Abroad

Increase the number of events that bring together international and domestic students Campus Life

Develop a plan for the university to showcase the new international calendar and specific events through various media Develop suggestions for synergies to harness UF’s current sources of international events International Calendar

Students earn medallions at graduation to recognize their international expertise International Scholars Program

Assessment Principle s Direct and indirect assessments must measure progress on the SLOs Outputs must measure participation in the initiatives Each measure must have a planned use or interpretation (Validity) Each measure must be reliably measured (minimize error)

Using Existing Instruments It is easier to adopt an existing instrument than to develop a new instrument Easier is not better if you violate your principles! That is, existing instruments must: measure the SLOs; be valid for its planned uses and interpretations; and be reliable for your purposes

Experts for Judging Alignment (Content Validation) International Task Force Assessment Committee (5-6 with expertise on content and assessment) – Curriculum experts – International Administrators – Psychometricians

Defining Review Criteria for Existing Assessments Measures the SLOs Validity Evidence Reliability Evidence Feasibility to Use in Our Context

Factors in our decision to create our own assessments Most commercial products had good reliability and validity evidence for their stated purposes None matched our SLOs (Alignment problem)Little evidence of feasibility for large scale use Final Decision: Need to develop our own assessments

What Assessments do we need to Develop? Direct Assessments focusing on SLO achievement Must allow flexibility to measure learning in any discipline Needs to be aligned with Curriculum Indirect Assessments of SLOs 2 and 3 only Not appropriate for SLO 1 since Content is discipline-based

Developing the Assessments Writing the Assessment Items to measure SLOs Ongoing review for fidelity with the SLOs and Initiatives (Validity) Piloting to establish reliability and psychometric properties of items

Indirect Assessment Item Development Alternate forms Critical Thinking & Communication Retain best items for single form Item Banks Critical Thinking 70 Likert items (agreement/self rating) 11 Behavioral questions 7 open-ended Communication 62 Likert items (agreement/self rating) 14 open-ended Item Specifications Defining Outcomes Aligning Items to operational definition

IntCRIT 1.I consider different perspectives before making conclusions about the world. 2.I am able to manage when faced with multiple cultural perspectives. 3.I am open to different cultural ways of thinking in any international context. 4.I can make effective decisions when placed in different cultural situations 5.Knowing about other cultural norms and beliefs is important to me. 6.I am able to think critically to interpret global and intercultural issues. 7.I actively learn about different cultural norms. 8.Understanding different points of view is a priority to me. 9.I can recognize how different cultures solve problems. 10.I can contrast important aspects of different cultures with my own. 11.Knowing about other cultural beliefs is important. 12.I am able to recognize how members of other cultures make decision s

IntCOMM 1.I demonstrate flexibility when interacting with members of another culture. 2.I am confident that I can adapt to different cultural environments 3.I am able to communicate effectively with members of other cultures 4.I like working in groups with students from other countries. 5.I feel comfortable in conversations that may involve cultural differences. 6.When working on a group project, I enjoy collaborating with students from other countries. 7.I often ask questions about culture to members of other cultures. 8.I enjoy learning about other cultures 9.I appreciate members of others cultures teaching me about their culture. 10.I am able to interact effectively with members of other cultures. 11.I appreciate differences between cultures 12.I feel comfortable discussing international issues. 13.I can clearly articulate my point of view to members of other cultures

Direct Assessment Rubrics developed for each of the three SLOs Rubrics to be implemented in Curriculum and Study Abroad portion of the Initiative Rubrics allow for discipline based content

Developing Rubrics The Association of American Colleges and Universities developed 15 Rubrics that can be used across programs and courses VALUE (Valid Assessment of Learning in Undergraduate Education) rubrics were developed by faculty and assessment expert teams across the country -- used by more than 2000 institutions (

QEP Content Rubric SLO Components Outstanding 3 Satisfactory 2 Unsatisfactory 1 Not Applicable 0 Concepts/ Principles Consistently and effectively demonstrates sophisticated understanding of the complexity of factors important to members of another culture in relation to its history, values, politics, communication styles, economy, and beliefs and practices. Usually demonstrates understanding of the complexity of factors important to members of another culture in relation to its history, values, politics, communication styles, economy, or beliefs and practices. Rarely or never understands the complexity of factors important to members of another culture in relation to its history, values, politics, communication styles, economy, or beliefs and practices. Not Applicable to Assignment or Course Terminology Consistently recognizes and effectively utilizes important and relevant terminology regarding intercultural and global issues in the appropriate environmental context. Usually identifies and implements important and relevant terminology regarding intercultural and global issues in the appropriate environmental context. Rarely or never understands important and relevant terminology regarding intercultural and global issues in the appropriate environmental context. Not Applicable to Assignment or Course Methodologies Consistently comprehends and effectively utilizes diverse and appropriate methodologies for understanding complex intercultural and global issues. Usually comprehends and utilizes diverse and appropriate methodologies for understanding intercultural and global issues. Rarely or never comprehends and utilize diverse and appropriate methodologies for understanding intercultural and global issues. Not Applicable to Assignment or Course

Rubric Use Used for courses (curriculum) and study abroad Use must be approved by International Scholars Course Approval Committee Must adopt SLO(s) from program Must show assignments and specific tests that measure the rubrics Otherwise, not part of International Scholars program

Assessment Plan University wide impact Indirect Assessments SERU Biennial, odd years IntCrit and IntComm – Annual – Fall - Sampling Outputs Monitor participation and implementation - Annual Impact on those taking courses or Study Abroad Direct Assessments - Rubrics Indirect Assessments – IntCrit and IntComm Output – Number of students and options

For further information: See QEP online: Contact us: