1. Designing assessment with the ‘Assessment Design Decisions’ framework Phillip Dawson Office of the Vice-Provost (Learning & Teaching) Monash University

2. The ‘Assessment Decisions’ team Dr Phillip Dawson, Monash (Co-lead) A/Prof Margaret Bearman, Monash (Co- lead) A/Prof Liz Molloy, Monash Prof David Boud, UTS A/Prof Gordon Joughin, UQ A/Prof Sue Bennett, UOW Dr Matt Hall, Monash

3. Agenda 1.“Assessment design decisions?” 2.Your assessment designs 3.The framework 4.Applying the framework 5.Summary and close (and feedback!)

4. 1. “Assessment design decisions?”

6. We know a lot about ideal practice Hattie, 2009

7. Actual practice is different

8. Research suggests … Changing assessment ‘thinking’ in academics doesn’t necessarily change practice (Offerdahl & Tomanek, 2011) “…achieving a balance between summative and formative assessment requires complex, contextual thinking” (Price, et al., 2011)

9. Focus on assessment design Assessment policy and procedures Decisions in the design and implementation of assessment Judgements about student work ‘assessment design decisions’

10. 2. Your assessment designs

11. Your designs Think of an assessment design – It might be new – You might be thinking of revising it What is the impetus for change? Pair-share

12. Absorb the complexity, but don’t try to read

13. 3. The framework

14. Framework design Joined research data with conceptual frameworks, and literature Drew from own experience as expert practitioners Oriented towards educators’ agency rather than prescriptive

16. Engaging with the framework One-page summary Online ‘guide’ with detailed information, resources and educator experiences 55-page document version of ‘guide’ (as well as a ‘Five insights’ document to give to L&T leaders) http://assessmentdecisions.org

17. 4. Applying the framework

18. Applying the framework Focus on a specific unit We will work through the six framework components Address key questions on one-pager Use hard copy or online versions

25. assessmentdecisions.org phillip.dawson@monash.edu philldawson.com 5. Summary and close (and feedback) Support for this project/presentation/resource has been provided by the Australian Government Office for Learning and Teaching. The views expressed here do not necessarily reflect the views of the Australian Government Office for Learning and Teaching.

26. References Carless, D. (2009). Trust, distrust and their impact on assessment reform. Assessment & Evaluation in Higher Education, 34(1), 79-89. doi: 10.1080/02602930801895786 Hattie, J., The Black Box of Tertiary Assessment: An Impending Revolution, in Tertiary Assessment & Higher Education Student Outcomes: Policy, Practice & Research, L.H. Meyer, et al., Editors. 2009, Ako Aotearoa: Wellington, New Zealand. Offerdahl, E. G., & Tomanek, D. (2011). Changes in instructors' assessment thinking related to experimentation with new strategies. Assessment & Evaluation in Higher Education, 36(7), 781- 795. doi: 10.1080/02602938.2010.488794 Price, M., Carroll, J., O'Donovan, B., & Rust, C. (2011). If I was going there I wouldn't start from here: a critical commentary on current assessment practice. Assessment & Evaluation in Higher Education, 36(4), 479-492. doi: 10.1080/02602930903512883

27. Extra slides after this point

28. How do university teachers make decisions about assessment?

29. Research design Arts - Professions Arts – Non- profession Sciences - Professions Sciences – Non- profession 30 semi-structured interviews Gritty, coalface, ‘actual’ not ‘ideal’ Thematic analysis; coding against framework Meaning-making from coded data

30. What we can say Improving assessment is more than just a problem of knowledge transmission/translation Rarely about rationally selecting from options Assessment decisions are complex; situated; pragmatic

31. Impetus Opportunity or driver for change Influences Environment Educator Activities Making it work

35. “I think a lot of us have good intentions, we just don't have the time” – science lecturer

36. Time

37. “I was redeveloping a unit, it had already had a particular format of assessment. I elected to run with that rather than to go though the processes of trying to alter it … I wasn't gonna [jump through] those other hoops.” – humanities lecturer “I was redeveloping a unit, it had already had a particular format of assessment. I elected to run with that rather than to go though the processes of trying to alter it … I wasn't gonna [jump through] those other hoops.” – humanities lecturer

38. Committees and paperwork

39. “I don't think an assessment should be painful for the students or painful for the staff that assess it” – science lecturer

40. Beliefs

41. “technology becomes really critical where assessment is concerned. If you set something up and it doesn't work, they don't trust you. Getting them on board again is a killer … students can be very hostile to you making mistakes. They're not very forgiving” – arts lecturer

42. Technology

43. Improving assessment Requested supports Exemplars Time, money, sessionals Someone to help Involvement from senior academics Our analysis adds Understanding of freedom to move

44. More information http://assessmentdecisions.org A short paper: sclr.li/19 sclr.li/19 This research is supported by an Australian government Office for Learning and Teaching grant titled “Improving assessment: understanding educational decision-making in practice” (ID12-2254)

45. References Carless, D. (2009). Trust, distrust and their impact on assessment reform. Assessment & Evaluation in Higher Education, 34(1), 79-89. doi: 10.1080/02602930801895786 Hattie, J., The Black Box of Tertiary Assessment: An Impending Revolution, in Tertiary Assessment & Higher Education Student Outcomes: Policy, Practice & Research, L.H. Meyer, et al., Editors. 2009, Ako Aotearoa: Wellington, New Zealand. Offerdahl, E. G., & Tomanek, D. (2011). Changes in instructors' assessment thinking related to experimentation with new strategies. Assessment & Evaluation in Higher Education, 36(7), 781- 795. doi: 10.1080/02602938.2010.488794 Price, M., Carroll, J., O'Donovan, B., & Rust, C. (2011). If I was going there I wouldn't start from here: a critical commentary on current assessment practice. Assessment & Evaluation in Higher Education, 36(4), 479-492. doi: 10.1080/02602930903512883

46. Extra slides

47. Research and ideas from the literature “there's some interesting papers that look at the use of wikis and peer learning and those sorts of things” – science lecturer (early- career)

48. Research and ideas from the literature “That was really quite confronting to me and reading education literature, and I'd still find it difficult because it's completely different to my discipline … I just find it really, really difficult. Because I’m going, "My god, that's just their feelings."” – science lecturer (late- career)

49. Time and workload “I think a lot of us have good intentions, we just don't have the time” – science lecturer “But the main thing is that, it has to be feasible from a resource point of view as well. And if at a resource point of view, as well, as that educational perspective as well.” – health professions lecturer

50. Being strategic one of the other things that we learned as well was not to put too much data into the FEC documents, which I remember in the early days, again, we had 1500-word essay on topic X and then afterwards when you go 'that's crazy' we need to change it, we've got to go back to FEC and that load of paperwork and things. So, we do... You know, we are much more general about what we're putting into the FEC documents

51. Policy and flexibility “I was redeveloping a unit, it had already had a particular format of assessment. I elected to run with that rather than to go though the processes of trying to alter it … I wasn't gonna [jump through] those other hoops.” – humanities lecturer “there's supposed to be between 10 and 20% HDs and no fewer... No more than 5% fails, or 10%, I think … or more than probably 10% fails. I don't know would you call that a bell curve or what, but it is a prescribed range of results that you should have.” – humanities lecturer

52. Policy and flexibility “In regards to formal channels of approval, like education committees and the like, turns out that the, what the paperwork that currently exists with the education committees, is actually very, extremely non- specific.” - health professions lecturer “But I don't feel that the committee structures are really designed to be able to give... feedback or you know contribute, it's really more tick-cross. So in some of these cases, if what I was doing wasn't a change to the paperwork they already had, then we just carried on.” – health professions lecturer

53. Teamwork (or not) “The process of how this was developed was kind of the most amazing process, because basically, said, "Would you develop the unit, write a unit guide", and I did, with no help at all. I just thought ‘what would be needed?’, and I invented it… …the process has been a bit out of necessity, less consultative than what it needs to be. I mean, we had to get a course up and running really quickly.” – humanities lecturer

54. Teamwork (or not) “Yeah, I had to learn a fair bit, with that, which I learned off colleagues. So, again, yeah, that the sort of the theoretical background to teaching. Which sort of was provided to me. Through exposure to colleagues. And, that made a big change. That helped me do assessments better.” – health professions lecturer

55. Technology technology becomes really critical where assessment is concerned. If you set something up and it doesn't work, they don't trust you. Getting them on board again is a killer … students can be very hostile to you making mistakes. They're not very forgiving – arts lecturer