Mapping RSD-informed marking rubrics to USP Graduate Outcomes in undergraduate courses Shazna Buksh, School of Social Sciences Heena Lal, Planning & Quality Office

Introduction Background Implementation of the Research Skills Development (RSD) Framework commenced under the Strategic Total Academic Review (STAR) project in 2012. In response to the need for a systematic approach to embed research literacy and skills in the curricula of undergraduate and postgraduate programmes. Working Group 5 of the STAR project in year 2011 recommended that the RSD Framework be used to explicitly embed research literacy and skills in the curricula from undergraduate to postgraduate levels (Vanualailai et al., 2011). Since implementation, approximately 80 USP undergraduate courses have developed RSD-informed marking rubrics for specific course assessments.

Overlap between the RSD Framework and USP Graduate Outcomes Introduction Overlap between the RSD Framework and USP Graduate Outcomes STAR Working Group 5 in 2011 highlighted the substantial overlap between RSD and the USP Graduate Outcomes. In 2017, the definition of the USP Graduate Outcomes were revised to ensure clarity and consistency in language, developmental progression across the undergraduate levels, feasibility of use for assessment, and applicability across disciplines. Following the revision, broad level re-mapping carried out by the Research Office has maintained that there is substantial overlap between the RSD framework and USP Graduate Outcomes. Communication Creativity Critical thinking and Quantitative reasoning Ethics Pacific Consciousness Teamwork Professionalism

Introduction Why is the overlap between the RSD and USP graduate outcomes significant? Evidence of authentic assessment of graduate outcomes is critical for quality assurance of academic programmes. For example, evidence of assessment and development of graduate outcomes is part of the requirements for institutional accreditation by the Western Association of Schools and Colleges (WASC) Senior College and University Commission (WSCUC) (WSCUC, 2013). The USP Seeking Accreditation Visit 1 Institutional Report: highlighted that embedding of the RSD framework in courses contributes to the achievement and measurement of the USP Graduate outcomes (USP, 2018). Hence, the need for a deeper level analysis to see whether the RSD-informed marking rubrics used in specific assessments in fact assess the individual criteria from the seven USP Graduate Outcomes.

Research Question Can RSD informed marking rubrics assess the revised criteria of USP Graduate Outcomes?

Sample Method 12 RSD-informed rubrics from the Research webpage. Four compulsory undergraduate courses UU100, UU114, UU200, and UU204. Purposive Sampling Availability of the assessment question, alongside the rubrics. Included as many disciplines as possible within each faculty and selected four first year courses, five second year courses and three third year courses.

Method Procedure Qualitative Content Analysis is becoming increasingly popular as a method of analysis of textual data. Deductive qualitative analysis approach whereby the criteria for each of the seven USP GOs were used as the coding categories in the categorization matrix. RSD-informed rubrics were then mapped for the appropriate level i.e. benchmark, milestone and capstone in the GO rubric. Focus on explicit description of the GO criteria and sub-criteria in the rubrics. Assessment question was used to provide contextual information especially in determining at what level each GO was being assessed in a rubric

Results

Methods Mapping to Graduate Outcome Critical thinking and Quantitative reasoning Graduates will be able to evaluate multiple perspectives and arrive at a reasonable independent judgement based on evidence Course PD301 Pacific Population & Urbanisation Issues Assignment: Research Report All criteria covered at Capstone level of the University GO rubric. Criterion 1: Information Literacy Criterion 2: Management of multiple perspectives Criterion 3: Judgement based on evidence Criterion 4: Quantitative reasoning Some of the graduate outcome criteria are covered by more than one criteria in the RSD Framework, for example, Information literacy is covered under Evaluate & Reflect and Communicate & Apply.

Results Graduate Outcome Sub-criteria not covered Number of assessment rubrics Percentage of assessment rubrics (N = 12) Communication c1: Context and audience 1 8.3% c2: Structure c3: Clarity of expression and coherence 2 16.7% Creativity c1: Formulation of new approaches or methods of inquiry c2: Innovative application of knowledge and skills in an entrepreneurial context where appropriate Critical Thinking and Quantitative Reasoning c1: Information literacy c2: Management of multiple perspectives c4: Quantitative reasoning 5 41.7%

Results Graduate Outcome Sub-criteria not covered Number of assessment rubrics Percentage of assessment rubrics (N = 12) Ethics   Recognition of ethical principles and practices 3 25.0% Pacific Consciousness c1: Respect for the cultural heritage and diversity of Pacific societies 7 58.3% c2: Relationship between one’s culture and one’s position in the world 9 75.0% c3: Integration of traditional and contemporary practices to sustain Pacific societies Professionalism Professional practice (Recognition of the roles and responsibilities of the profession and occupation) 1 8.3% Teamwork Effective performance in teams

Conclusion & Recommendations RSD-informed marking rubrics can be used to assess USP Gos. Consistent with the USP report for the WSCUC accreditation. Consistent with broad level mapping of the revised GOs definitions with the facets of the RSD framework GOs such as Communication, Critical Thinking and Quantitative Reasoning, Ethics and Professionalism were covered very well. GOs such as Teamwork and Pacific Consciousness were poorly covered in the sample. Programme level mapping is required to identify gaps in the assessment of graduate outcomes in the programme.

Conclusion & Recommendations Some GOs were assessed by the RSD-informed rubrics, not all of the GO sub-criteria could be mapped for the revised Gos Critical Thinking and Quantitative Reasoning GO which was mapped across all 12 rubrics, however, the Quantitative Reasoning sub-criteria was not assessed by 41.7% of the rubrics. Sub-criteria for Pacific Consciousness were poorly covered in the selected rubrics. Respect for the cultural heritage and diversity of Pacific societies was not covered by 58.3% of the rubrics Relationship between one’s culture and one’s position in the world and Integration of traditional and contemporary practices to sustain Pacific societies were not covered by 75% of the rubrics in the sample. Rubrics for courses aiming to assess this GO will need to be further modified to address this sub-criteria

Limitations In some cases the assessment question was not detailed enough and while more guidance would have been provided through other platforms, the assignment question did not specify these. Therefore these rubrics may have scored a different ranking (benchmark, milestone and capstone) for the sub-criteria as we only looked for explicit description of the GO criteria and sub- criteria in the rubrics. Mapping issues with GO rubrics as we found some some GO statements (e.g. Creativity) as being too high pitched for undergraduate programmes and therefore having low applicability to vast majority of USP courses, at the appropriate level. Future research will benefit from examining the design and phrasing of GO rubrics and assess applicability to USP courses. Our study none-the-less showed that the RSD-informed rubrics can be effectively used for assessing the revised USP GOs.