

Reaffirmation 2014Off-site review is completedOn-site review is February 2014QEP is ready to be submitted in January 2014

The institution has developed an acceptable Quality Enhancement Plan that includes an institutional process for identifying key issues emerging from institutional assessment and focuses on learning outcomes and/or the environment supporting student learning and accomplishing the mission of the institution. (Quality Enhancement Plan)

The Institution has developed a Quality Enhancement Plan that (1) demonstrates institutional capability for the initiation, implementation and completion of the QEP; (2) includes broad-based involvement of institutional constituencies in the development and proposed implementation of the QEP; and (3) identifies goals and a plan to assess their achievement.

Purpose Uses and interpretations Student Learning (Outcomes) QEP Assessment

Step 1. Establish topic and student learning outcomes (SLOs) Step 2. Identify direct and indirect assessments to measure SLOs (and outputs to measure initiatives) Step 3. If assessments are not available and sufficient, develop your own assessments Step 4. Plan Assessment Schedule and follow it

Direct – Measures of Student Learning Indirect – Measures of Attitudes, Behaviors, Beliefs, … Outputs – Counts of the numbers of students participating in events, number of events, number of classes, etc.

Direct and indirect assessments must measure progress on the SLOs Outputs must measure participation in the initiatives Each measure must have a planned use or interpretation (Validity) Each measure must be reliably measured (with minimum error)

It is easier to adopt an existing instrument than to develop a new instrument Easier is not better if you violate your principles! That is, existing instruments must: measure the SLOs; be valid for its planned uses and interpretations; and be reliable for your purposes

 UF Theme : Internationalization Internationalization is the conscious integration of global awareness and intercultural competence into student learning.

SLO1 (Content):Students identify, describe, and explain global and intercultural conditions and interdependencies. SLO2 (Critical Thinking): Students analyze and interpret global and intercultural issues. SLO3 (Communication): Students communicate effectively with members of other cultures.

 Internationalization Measures Identified – Commercially Available ◦ Global Perspectives Inventory (GPI) ◦ Global Competence Aptitude Assessment (GCAA) ◦ Intercultural Development Inventory (IDI) ◦ Global Competencies Inventory ◦ Cross Cultural Adaptability Inventory ◦ Global Awareness Profile ◦ Intercultural Effectiveness Scale

 International Task Force ◦ Representation from all 16 Colleges ◦ Student Representation ◦ Representation from Faculty, Staff, Students, Administration  Assessment Committee (5-6 with expertise on content and assessment)

Measures the SLOs Validity Evidence Reliability Evidence Feasibility to Use in Our Context

Most commercial products had good reliability and validity evidence for their stated purposes None matched our SLOsLittle evidence of feasibility for large scale useFinal Decision: Need to develop our own assessments

 Consider Options Already Being Used as Part of Reporting System  Student Experience in the Research University (SERU) - biennial survey used by multiple universities that includes items on internationalization

 Items measure behaviors (e.g., courses taken, participation in Study Abroad, and other types of international experiences) and attitudes toward other cultures  We elected to add 10 items designed to measure attitudes related to SLOs 2 and 3

Direct Assessments focusing on SLO achievement Must allow flexibility to measure learning in any discipline Needs to be aligned with Curriculum Indirect Assessments of SLOs 2 and 3 only Not appropriate for SLO 1 since Content is discipline-based

Writing the Assessment Items Ongoing review for fidelity with the SLOs (Validity) Piloting to establish reliability and psychometric properties of items

International Critical Thinking (IntCRIT) Attitudes and Beliefs International Communication (IntCOMM) Attitudes and Beliefs International Content not feasible with Disciplinary Differences Two Assessments Developed

1.Development of item specifications based on the two SLOs and a literature review. 2.Writing items based on the item specifications.  Approximately 70 items were written for each SLO

3.Review of the items by the ITF, the Assessment Committee and other experts in assessment. (Validity and Match to SLOs) 4. Revision of items based on feedback from expert review.  Revisions were minor changes in wording.

5. Pilot testing with undergraduate students at UF and eliminating items with poor discriminations. ◦ Initial piloting was completed with four forms to minimize the testing burden for students. ◦ Forms overlapped with ten items that expert agreed helped to define the construct. ◦ Each form was pilot tested with undergraduates. 6. Item analysis of pilot data. ◦ The scale reliabilities exceeded.95 for all four forms. ◦ Items were retained that had an item discrimination of.25 or higher.

7. Pilot testing (N=70-80) the retained items on a single form for each SLO. 8. Item analysis of pilot data. ◦ Recommended retaining items with the highest item discriminations that would result in a scale with a reliability of at least.90.  For IntCRIT, the recommendation was to retain 12 items.  For IntCOMM, the recommendation was to retain 14 items. 9. Review of the items by the ITF, the Assessment Committee and other experts in assessment. (Validity and Match to SLOs)

1. I consider different perspectives before making conclusions about the world. 2. I am able to manage when faced with multiple cultural perspectives. 3. I am open to different cultural ways of thinking in any international context. 4. I can make effective decisions when placed in different cultural situations 5. Knowing about other cultural norms and beliefs is important to me. 6. I am able to think critically to interpret global and intercultural issues. 7. I actively learn about different cultural norms. 8. Understanding different points of view is a priority to me. 9. I can recognize how different cultures solve problems. 10. I can contrast important aspects of different cultures with my own. 11. Knowing about other cultural beliefs is important. 12. I am able to recognize how members of other cultures make decisions

1. I demonstrate flexibility when interacting with members of another culture. 2. I prefer to socialize with people of my culture. 3. I am confident that I can adapt to different cultural environments 4. I am able to communicate effectively with members of other cultures 5. I like working in groups with students from other countries. 6. I feel comfortable in conversations that may involve cultural differences. 7. When working on a group project, I enjoy collaborating with students from other countries. 8. I often ask questions about culture to members of other cultures. 9. I enjoy learning about other cultures 10. I appreciate members of others cultures teaching me about their culture. 11. I am able to interact effectively with members of other cultures. 12. I appreciate differences between cultures 13. I feel comfortable discussing international issues. 14. I can clearly articulate my point of view to members of other cultures

Use with curriculum Must be flexible to allow faculty to define internationalization in discipline Must provide a standard measure

Define Rubric Instructors develop or identify assessments in courses Instructors will provide evidence of use and scores to University with examples of assessments

 The Association of American Colleges and Universities developed 15 Rubrics that can be used across programs and courses  VALUE (Valid Assessment of Learning in Undergraduate Education) rubrics were developed by faculty and assessment expert teams across the country -- used by more than 2000 institutions (

UF QEP SLOVALUE RubricAdaptation SLO 1: Content Intercultural Knowledge and Competence Limit criteria to knowledge, dropping skills and attitudes. Modify descriptions for consistency across levels and ease of use. SLO 2: Critical Thinking Critical Thinking Add language to reflect emphasis on international context for critical thinking. Modify descriptions for consistency across levels and ease of use. SLO 3: Communication Written Communication Oral Communication Combine rubrics to measure communication in multiple modes, and add language to reflect emphasis on international context. Modify descriptions for consistency across levels and ease of use.

QEP Content Rubric SLO Components Outstanding 3 Satisfactory 2 Unsatisfactory 1 Not Applicable 0 Concepts/ Principles Consistently and effectively demonstrates sophisticated understanding of the complexity of factors important to members of another culture in relation to its history, values, politics, communication styles, economy, and beliefs and practices. Usually demonstrates understanding of the complexity of factors important to members of another culture in relation to its history, values, politics, communication styles, economy, or beliefs and practices. Rarely or never understands the complexity of factors important to members of another culture in relation to its history, values, politics, communication styles, economy, or beliefs and practices. Not Applicable to Assignment or Course Terminology Consistently recognizes and effectively utilizes important and relevant terminology regarding intercultural and global issues in the appropriate environmental context. Usually identifies and implements important and relevant terminology regarding intercultural and global issues in the appropriate environmental context. Rarely or never understands important and relevant terminology regarding intercultural and global issues in the appropriate environmental context. Not Applicable to Assignment or Course Methodologies Consistently comprehends and effectively utilizes diverse and appropriate methodologies for understanding complex intercultural and global issues. Usually comprehends and utilizes diverse and appropriate methodologies for understanding intercultural and global issues. Rarely or never comprehends and utilize diverse and appropriate methodologies for understanding intercultural and global issues. Not Applicable to Assignment or Course

Review by Internationalization Task Force, Assessment Committee, and Experts for Match to SLOs and Validity Piloting (in progress) in curriculum and Study Abroad

 Number of participants at specific campus events with an international focus (QEP events).  Number of International Scholar courses.  Number of students enrolled in International Scholar courses.  Number of Study Abroad courses offered.  Number of students studying abroad.

University wide impact Indirect Assessments SERU Biennial, odd years IntCrit and IntComm – Annual – Fall - Sampling Outputs Monitor participation and implementation - Annual Impact on those taking courses or Study Abroad Direct Assessments - Rubrics Indirect Assessments – IntCrit and IntComm Output – Number of students and options

Annual analysis, review, and interpretation of data to inform: The implementation of the program The degree of SLO achievement When adjustments to program are needed The benefit of the program for underrepresented populations

M. David Miller Timothy S. Brophy