1. www.monash.edu.au Neil Spike Director of Training & Assessment School of Primary Health Care Monash University, Australia New challenges in assessment of clinical competence

2. www.monash.edu.au 2 Answers? We’re not actually going to tell you how to play the game; that’s something you’ll have to figure out for yourself. After all the object of the game is to discover The Meaning of Life, and life does not come with an instruction book. If it did, you’d be wasting all your days in the troubleshooting section, hoping to figure out where you went wrong. That would be no fun. Manual for Monty Python’s Meaning of Life CD-Rom game.

3. www.monash.edu.au 3 Answers? If you have the same ideas as everybody else but have them one week earlier than everyone else then you will be hailed as a visionary. But if you have them five years earlier, you will be called a lunatic. Barry Jones

4. www.monash.edu.au 4 Questions? Confidence never comes from having all the answers; it comes from being open to all the questions. – Earl Gary Stevens

5. www.monash.edu.au 5 Overview Other parameters of utility Changes in students Changes in academic roles Performance assessment General professional competencies

6. www.monash.edu.au 6 Overview Other parameters of utility Changes in students Changes in academic roles Performance assessment General professional competencies

7. www.monash.edu.au 7 Utility function U = Utility R = Reliability V = Validity E = Educational impact A = Acceptability C = Cost W = Weight U = w r R x w v V x w e E x w a A x w c C Adapted from Cees van der Vleuten

8. www.monash.edu.au 8 Utility function U = Utility R = Reliability V = Validity E = Educational impact A = Acceptability C = Cost W = Weight U = w r R x w v V x w e E x w a A x w c C

9. www.monash.edu.au 9 Utility function Introduce a new factor: –Feasibility

10. www.monash.edu.au 10

11. www.monash.edu.au 11

12. www.monash.edu.au 12

13. www.monash.edu.au 13 Overview Other parameters of utility Changes in students Changes in academic roles Performance assessment General professional competencies

14. www.monash.edu.au 14 Universiteit Maastricht Student-centred assessment -self-assessment -peer-assessment -co-assessment self-evaluation skills assessing the unmeasurable peer pressure? peer-evaluation skills self, peer and co-evaluation skills peer pressure? joint responsibility

15. www.monash.edu.au 15 Student changes Increasing diversity –Ethnicity –Socio-economic status –Previous school/educational experience –Equity categories –Basis of admission Adapted from Frank Tesoriero

16. www.monash.edu.au 16 Student changes Student diversity means that previous assumptions on student characteristics may no longer hold true –Common values –Preferred learning styles –Levels of language proficiency

17. www.monash.edu.au 17 Student centredness Early needs assessment Individual learning plans & portfolios Changes to staffing ratios Professional development for students Increased time for increased assessments Collating numerous assessments Translating the standard into a cut score

18. www.monash.edu.au 18

19. www.monash.edu.au 19 Overview Other parameters of utility Changes in students Changes in academic roles Performance assessment General professional competencies

20. www.monash.edu.au 20 Universiteit Maastricht Suggestions Assessment is not mainly a measurement problem but more an educational design problem. - built-in quality control procedures and bodies - sampling in stead of (over) structuring - variation of methods, compensation, regulations - saturation of information approach - training of students, teachers, examiners, board members Masterplan of the assessment system

21. www.monash.edu.au 21 Academic changes In addition to traditional roles: –Research –Clinical service –Teaching –Consultancy Management of casualised teaching & assessment teams

22. www.monash.edu.au 22 Academic changes Need to maintain program integrity when teachers are casual and teach outside of a full program context. Need to adequately select, induct, support and manage of casual staff both individually and collectively. Need to engage casual staff who are often at margins of academic life, in the teaching and learning debates which have significant impact on them, their teaching and their students. Adapted from Frank Tesoriero

23. www.monash.edu.au 23 Overview Other parameters of utility Changes in students Changes in academic roles Performance assessment General professional competencies

24. www.monash.edu.au 24 Universiteit Maastricht Broaden the measurement time frame Specific test occasions Longitudinal testing - specific preparation of students - inflexibility - unwanted influence on learning - certain aspects unmeasurable - no specific preparation - congruent with flexible curricula - wanted influence on learning - more performance assessment

25. www.monash.edu.au 25 Performance Assessment Indirect methods –Chart audit –Practice audits –Prescription data……… Direct methods –Clinical (supervisor) ratings –Masked standardized patients –Video assessment –Patient reports –Peer reports –Clinical work samples………

26. www.monash.edu.au 26 Overview Other parameters of utility Changes in students Changes in academic roles Performance assessment General professional competencies

27. www.monash.edu.au 27 Universiteit Maastricht General professional competencies dealing with tasks dealing with others dealing with oneself planning and time management information gathering co-operation communicating giving feedback and criticising self-responsibility self-evaluation self-reflection

28. www.monash.edu.au 28 The aim of education is the knowledge not of facts but of values. William Ralph Inge

29. www.monash.edu.au 29 Professional competencies Quality assurance focus Traditionally includes: –The curriculum –The teaching and learning process –The assessment of competence Should also include: –Monitoring –Evaluation –Staff development –“Fitness for purpose”

30. www.monash.edu.au 30 Where do we start? ‘ Would you tell me please, which way I ought to go from here?’ ‘That depends a good deal on where you want to get to,’ said the Cat. ‘I don’t much care where’ said Alice. ‘Then it doesn’t matter which way you go,’ said the Cat. Alice’s Adventures in Wonderland - Lewis Carroll

31. www.monash.edu.au 31