Arts education is a seriously funny business. We demand that students conform to the formalities of the university and yet we secretly hope they will practise wild, if subtle rebellion. We require them to be versed in inherited theoretical vocabularies, but need them to energise us with some previously unseen thing... The very fact that so many students survive the contradictions is in itself wonderfully encouraging. Robert Clark, The Guardian, 7thJuly 1998 High Anxiety? – assessing on the edge of chaos

High Anxiety? assessing on the edge of chaos

Close to certainty Close to agreement Far from agreement Far from certainty (Based on Stacey 2000) Assessing on the edge of chaos

Case Study reports Dissertations Essays Examinations – unseen, seen, open-book, etc. Group projects or assignments Individual projects or assignments Multiple Choice Peer and/or Self assessments Performances – individual, group Portfolios Presentations – oral, poster, etc. Reflective Journals Reports Vivas Habeshaw, Gibbs, & Habeshaw (1993) list 48 discrete items [e.g. project exam, note-form essay, computer-based assessment] High Anxiety?

the problem The more assessment involves qualitative information, the more subjectivity is involved Stricter assessment criteria + more structured and proscribed content = improved reliability BUT….. above would “obliterate the essence of qualitative assessment in terms of flexibility, personal orientation and authenticity”. Driessen et al (2005) Medical Education 39: 214–220 High Anxiety?

Positivist There’s a reality ‘out there’ that can be studied, captured and understood Interpretive ‘Truth’ is a matter of consensus amongst informed and sophisticated constructors, not of correspondence with an objective reality. Understanding, Subjectivity, Contextualised, Value dependent, Multiple-realities, negotiated Teacher as Participant Explanation, Control Objectivity, Measurable, Value-free, Universal, Generalisable, External, Quantifiable, Can be transmitted and acquired Teacher as Expert

Therefore: Find and Replace Replace Validity with Credibility, Coherence, Consistency, Trustworthiness, Authenticity Replace Certainty with Relativity Replace Generalised Explanation with Local Understanding Replace Source Data with Empirical Materials (which can become data…. and then evidence) Replace Is it true? with Does it work? Replace Single Point Perspective with Multiple Perspective Replace the Triangle with the Crystal

Ad sedere / to sit down together "You've got to involve students actively, not just view them as objects of assessment, but as agents of assessment. This can be done in many ways. One is that you ask students systematically what they have learned. It's a simple idea; it's rarely done.....You find that students say some remarkable things.” Walt Haney, Professor of Education, Center for the Study of Testing, Evaluation and Educational Policy, Boston, USA. negotiating assessment

assessing portfolios At the heart of the approach …..criteria from qualitative research might be more appropriate than criteria from quantitative research, like reliability. E Driessen et al (2005) Medical Education 2005; 39: 214–220

Strategies for reaching credibility: triangulation (combining different information sources) prolonged engagement (sufficient time investment by the researcher) member checking (testing the data with the members of the group from which they were collected) Strategies for realising dependability: establishing an audit trail (i.e. documentation of the assessment process to enable external checks) carrying out a dependability audit (i.e. quality assessment procedures with an external auditor). assessing portfolios

Driessen et al (2005) All portfolios (n = 237) were judged at the end of the academic year and given a grade of fail, pass or distinction. Criteria: the quality of the analyses of strengths and weaknesses; the quality of the evidence; the extent to which the evidence reflected the analyses of strengths and weaknesses; the clarity and feasibility of the learning objectives the extent to which the learning objectives were achieved. assessing portfolios E Driessen et al (2005) Medical Education 2005; 39: 214–220 assessing portfolios

Assessment in all phases of the portfolio process: 1 during the compilation of the portfolio in regular meetings of mentor and student; 2 in the end-of-year meeting when mentor and student recommended the final grade; 3 after submission of the portfolio to the portfolio assessment committee (PAC) for final grading. E Driessen et al (2005) Medical Education 2005; 39: 214–220

assessing portfolios Procedures and precautions 1.Compiling the portfolio At least 2 review/feedback sessions with mentor Oral and written feedback Mentor meetings – sharing experiences 2.Recommendations Student and mentor discuss mentor’s recommendation. Student signs agreement or disagreement 3.Portfolio Assessment Committee Sequential judgement procedure All mentors on PAC Mentors do not grade own students Judgements of full committee only required if available information not unanimous. E Driessen et al (2005) Medical Education 2005; 39: 214–220

assessing portfolios 233 portfolios considered for assessment Portfolios with mentor/student agreement scanned by single assessor. If assessor agrees, then grade finalised (202 / 85%). If 1 st assessor disagrees (24 / 9%), then 2 nd assessor brought in. If both assessors agree, then grade finalised. If mentor/student disagreement (7 / 3%) then 2 assessors assess independently. If disagreement, then to full committee. 31 portfolios (24 + 7) assessed by 2 nd assessor. 22 agreed at this stage. Full committee considered remaining 9 / 4% cases. Student and mentor present case. Agreement by consensus. E Driessen et al (2005) Medical Education 2005; 39: 214–220

“Nine students failed, 147 received a pass and 81were given a distinction. A total of 226 portfolios (96%) were graded without being reviewed by the full committee. The entire procedure was completed in the relatively short time of 42 hours (i.e. 11 minutes per portfolio), with the committee meeting lasting 1 hour. The participants did not perceive the process as particularly stressful.” assessing portfolios E Driessen et al (2005) Medical Education 2005; 39: 214–220

Ad sedere / to sit down together "You've got to involve students actively, not just view them as objects of assessment, but as agents of assessment. This can be done in many ways. One is that you ask students systematically what they have learned. It's a simple idea; it's rarely done.....You find that students say some remarkable things.” Walt Haney, Professor of Education, Center for the Study of Testing, Evaluation and Educational Policy, Boston, USA. negotiating assessment

Presentation and Performance: Negotiated assessment Students engaged in creative practice work at different levels AND different ways The products they create will be different, as will the processes and methods utilised, and the disciplines they represent That assessment should operate and be perceived as an integral part of the learning process rather than 'bolted-on' to the end of that process. That the form, content and implementation of the assessment process should be commensurable with the discourse and practices of the field ‘Ad sedere’

Six assessment fields: Presentation/Production i.e. the finished product presented to an audience Process i.e. the journey that led to the product Idea i.e. the ideas that informed both the process and the product. Technical i.e. the quality and utility of the technical features of the product and the skills with which they were assembled and/or operated Documentation i.e. research, design, sketches, planning, evaluation, analyses, portfolio, etc. Interview i.e. the student's ability to articulate their understanding, utilisation and application and use of any of the above. negotiating assessment

Learning contract – negotiated Regular meetings/tutorials Assessors see the performance/presentation + student compiles ‘portfolio of evidence’ Assessment tutorial c. 30 min; individual / group + at least 2 assessors Work through the criteria - moving upwards Reach a point of maximum information, optimal achievement Agree a grade band Sense of ‘completeness’, ‘accomplishment A learning experience for all concerned negotiating assessment

Both cases demonstrate the feasibility of a qualitative approach to achieve reliable summative judgement using an inherently complex and non- standardised assessment instrument, which relies - to a greater or lesser extent – on holistic professional judgement.

? ?