1. Client - meeting - _____ - 1 Student perceptions of assessment at Peninsula Medical School Dr. Karen Mattick Senior Lecturer, Peninsula College of Medicine and Dentistry

2. Background Modern undergraduate education aims to provide an assessment environment that promotes high quality learning Modern undergraduate education aims to provide an assessment environment that promotes high quality learning This is not easy to achieve - student perceptions of learning and assessment environments are hard to predict This is not easy to achieve - student perceptions of learning and assessment environments are hard to predict Experienced educators have introduced changes in assessment designed to promote high quality learning that have had the exact opposite effect! Experienced educators have introduced changes in assessment designed to promote high quality learning that have had the exact opposite effect!

3. Need to understand the impact of assessment on what is learnt and how, and use this to promote high quality learning and constructive alignment between teaching and assessment. Need to understand the impact of assessment on what is learnt and how, and use this to promote high quality learning and constructive alignment between teaching and assessment. Great complexity of factors that might influence high quality learning. Barriers to achieving it almost inevitable. Great complexity of factors that might influence high quality learning. Barriers to achieving it almost inevitable. In 2002, we designed an evidenced based learning and assessment environment that we believed should promote high quality learning In 2002, we designed an evidenced based learning and assessment environment that we believed should promote high quality learning Could it stand the ultimate test…? Could it stand the ultimate test…?

4. Our students…!

5. Our context – learning environment In years 1 and 2, two week case units In years 1 and 2, two week case units  PBL sessions  clinical placements and jigsaw sessions  biomedical science tutorials  clinical skills training  small number of lectures. Spiral curriculum Spiral curriculum Emphasis on self-directed learning Emphasis on self-directed learning Two week Special Study Units (SSUs) Two week Special Study Units (SSUs)  research ‘non-core’ topics in depth  publication-style report.

6. Our context – assessment environment Assessment characterised by a ‘little but often’ approach. Assessment characterised by a ‘little but often’ approach. Applied medical knowledge assessed by a ‘progress test’ Applied medical knowledge assessed by a ‘progress test’  sat quarterly  featuring clinical vignettes and questions at graduation level  any topic covered in any test. Clinical skills Clinical skills  regular competency assessments  ISCE at the end of the second year. Team working and professional behaviours Team working and professional behaviours  tutor and peer judgements  reflective portfolios.

7. What we did Created a definition of high quality learning for undergraduate medical students Created a definition of high quality learning for undergraduate medical students Interviewed 15 second year medical students in 2004 about their approaches to learning, and how the PMS learning and assessment environment affected this Interviewed 15 second year medical students in 2004 about their approaches to learning, and how the PMS learning and assessment environment affected this Thematic analysis of interview data Thematic analysis of interview data

8. Definition of high quality learning The latest conceptions of a deep approach to studying (learning through interest with an intention to understand the material covered) The latest conceptions of a deep approach to studying (learning through interest with an intention to understand the material covered) Combined with broader metacognitive attributes that include aspects of the strategic approach described previously (e.g. personal organisation, effort monitoring, effort regulation) Combined with broader metacognitive attributes that include aspects of the strategic approach described previously (e.g. personal organisation, effort monitoring, effort regulation) And characteristics perceived as desirable for trainee doctors (e.g. reflective learning). And characteristics perceived as desirable for trainee doctors (e.g. reflective learning).

9. What we found Facilitating high quality learning Inhibiting high quality learning Learning in context Perceived quantity of information Self directed learning 1. Perceived lack of feedback from assessment Spiral curriculum Disease frameworks 2. Progress test Anatomy 3. Summative ISCE

10. 1. Perceived lack of assessment feedback ‘Little but often’ approach sometimes perceived as an absence of useful feedback on progress. ‘Little but often’ approach sometimes perceived as an absence of useful feedback on progress.  I don’t know if I’m doing enough now. I seem to be doing all right, so I think I’m learning enough. But there’s nothing that tells you whether you are or not, I find that quite difficult. Participant 13; female; school leaver. This was despite (per academic year) This was despite (per academic year)  4 progress test results  4 SSU project reports  numerous skills competencies  10 PPD judgements and 2 portfolio analyses scores. So why the mismatch in perception? So why the mismatch in perception?  format, timing or level of detail of feedback?  Importance of feedback within a curriculum that promotes SDL and features reduced contact time with teachers?

11. Whilst the perceived lack of assessment feedback seemed anxiety-inducing, it deterred a focus on assessment. Whilst the perceived lack of assessment feedback seemed anxiety-inducing, it deterred a focus on assessment.  I kind of had to change the way I motivate myself… I’ve had to do it by myself rather than let other people motivate me I think. Participant 14; female; school leaver. Participants reported becoming motivated by intrinsic factors - a desirable development for lifelong learning where there will be no formal assessment. Participants reported becoming motivated by intrinsic factors - a desirable development for lifelong learning where there will be no formal assessment.

12. 2. Progress test – for the majority Most students reported that understanding and application of information was needed due to the clinical vignettes. Most students reported that understanding and application of information was needed due to the clinical vignettes. Students believed the test rewarded those working consistently. Students believed the test rewarded those working consistently. The fact that it could test any topic at any time deterred short-term memory strategies and ‘cramming’. The fact that it could test any topic at any time deterred short-term memory strategies and ‘cramming’.  I think that’s what makes progress tests so difficult is because you’ve got to be learning constantly, you can’t like have a cram. Participant 5; female; school leaver. Progress test already known to reward students with a deep approach when starting medical school (Mattick et al. 2004). Progress test already known to reward students with a deep approach when starting medical school (Mattick et al. 2004). Students that were highly sensitive to task demands and able to adapt to the learning environment tended to succeed. Students that were highly sensitive to task demands and able to adapt to the learning environment tended to succeed.

13. 2. Progress test – for a minority But, when students became excessively anxious about it, the progress test was sometimes a barrier to high quality learning. But, when students became excessively anxious about it, the progress test was sometimes a barrier to high quality learning. Anxiety seemed to drive students to do more work in terms of quantity but not necessarily the right kind of work Anxiety seemed to drive students to do more work in terms of quantity but not necessarily the right kind of work  If you’re tensed up you need to really rote learn a lot of it. Participant 8; male; school leaver. Previous work indicates that underperforming students often adopted or retained undesirable approaches to learning in response to poor grades (Mattick, Dennis and Bligh, unpublished data). Previous work indicates that underperforming students often adopted or retained undesirable approaches to learning in response to poor grades (Mattick, Dennis and Bligh, unpublished data). Underperforming students need help and support to become accomplished learners and successful in HE and beyond. Underperforming students need help and support to become accomplished learners and successful in HE and beyond. Remediation should aim to challenge why, what and how individuals study. Remediation should aim to challenge why, what and how individuals study.

14. 3. Summative ISCE The ISCE constituted a single exam to pass for progression to year 3 (i.e. not ‘little but often’). The ISCE constituted a single exam to pass for progression to year 3 (i.e. not ‘little but often’). High stakes and anxiety inducing. High stakes and anxiety inducing.  I ended up studying a couple of things that weren’t in the, in the ISCE because of the general panic that was out there. It’s hard not to be influenced by that. Participant 10; male; mature entry. But, many participants said they liked this - the ISCE was a good motivator for learning and consolidating the skills learned and practiced over the last two years. But, many participants said they liked this - the ISCE was a good motivator for learning and consolidating the skills learned and practiced over the last two years.

15. Participants felt a cramming-type approach was an appropriate way to prepare for the ISCE. Participants felt a cramming-type approach was an appropriate way to prepare for the ISCE.  The ISCE is like a final exam, as such, so I think if you wanted to you could really cram for that. Participant 8; male; school leaver. Was this because: Was this because:  it was a practical exam for which they could practice by repetition  of the high anxiety it invoked. Participants perceived the advice from teachers about preparing for skills-based examinations (‘practice, practice, practice’) to conflict with that for knowledge-based assessment (‘avoid rote learning’). Participants perceived the advice from teachers about preparing for skills-based examinations (‘practice, practice, practice’) to conflict with that for knowledge-based assessment (‘avoid rote learning’).

16. Questions, reflections. These barriers to high quality learning were interesting since they were unintended consequences of the curriculum. These barriers to high quality learning were interesting since they were unintended consequences of the curriculum. We don’t think we could predict them without studies such as these. We don’t think we could predict them without studies such as these. We have now introduced some measures to address the barriers identified (but possibly also introduced more?!). We have now introduced some measures to address the barriers identified (but possibly also introduced more?!). Any questions about our study? Any questions about our study? What about your own settings? What about your own settings?