1. Developing Valid Rubrics for Assessing Global Awareness and Global Perspective Dr. Stephanie Doscher Florida International University AAC&U General Education and Assessment Conference Boston, Massachusetts, March 1 st, 2013

2. Session Outline Global Learning @ FIU Rubric anatomy Developing the rubrics Scorer training Rubric for rubrics Gathering evidence of validity and reliability

3. Text

4. FIUs Global Learning SLOs Global Awareness: knowledge of the interrelatedness of local, global, international and intercultural issues, trends, and systems Global Perspective: ability to develop a multi- perspective analysis of local, global, international, and intercultural problems Global Engagement: willingness to engage in local, global, international, and intercultural problem solving

5. Active Learning Strategies/ Performance Assessments Essay, story, or poem Research paper Literary analysis Book or article review Case study Speech Journal response Art exhibit Portfolio Musical composition Lab report Strategic planning In-class discussion Editorials Peer editing Poster presentation Video Podcast Mock trial Oral presentation Debate Role play Online discussion Blogs Advertising campaign Building a prototype Modeling Experiments Service learning

8. Developing the Rubrics Faculty learning community Pilot case studies, questions, scoring criteria Revisions based on response trends Pilot study Benchmark responses Anchor papers Faculty feedback, expert judge feedback Two field tests Scorer training Rubric language, response minimum

9. Scorer Training Full-time and adjunct faculty Pre-training packet Open discussion and review of cases, questions, and rubrics Norming session with anchor papers Sample scoring session, 10% of total papers

14. Validating the Rubrics Validity: Can the rubrics detect the differences in students development of global awareness and global perspective? Reliability: Can trained faculty raters agree on rubric scores 80% of the time or more?

15. Validating the Rubrics PretestTreatmentPosttest Global learning O 1 X 1 O 1 - - - - - - - - - - - - - - - Non-global learningO 2 X 2 O 2 Research Design

16. Data Collection Emails to chairs and potential faculty Pretestwithin first two weeks of class Posttestwithin last two weeks of class Trained faculty raters Two raters scored each question, third rater for discrepant scores

17. Results Inter-rater reliability Global awareness rubric Pretest (.89, p <.0001) Posttest (.95, p <.0001) Global perspective rubric Pretest (.92, p <.0001) Posttest (.91, p <.0001)

18. Results Validity – Global Awareness Rubric No significant main effects Post-hoc analysis: Groups differed significantly on pretest scores, p =.003 Global learning (M = 1.51) Non-global learning (M = 1.85) Significant interaction between global awareness pretest score and the treatment in predicting global awareness posttest score, p =.005

19. Global Awareness Rubric

20. Results Validity – Global Perspective Rubric No significant main effects Post-hoc analysis: Groups differed significantly on pretest scores, p =.003 Global learning (M =.90) Non-global learning (M = 1.2) Significant interaction between global perspective pretest score and the treatment in predicting global perspective posttest score, p =.005

21. Results

22. Interpretation of Results Reliability Both rubrics highly reliable Consistent with literature on rubric development and rater training Empirical support for structural validity

23. Interpretation of Results Validity Rubric detects differences between global learning and non-global learning students Rubric detects differences within the group of global learning students

24. Thank You! For more information, please contact me: Stephanie Doscher, Office of Global Learning sdoscher@fiu.edu Visit our web site: goglobal.fiu.edu