1. Universiteit Maastricht Barcelona, 6 – 9 July 2004 11 th Ottawa conference on Medical Education

2. Universiteit Maastricht where do we go from here? Where do we go from here? Lambert Schuwirth new challenges in assessment of clinical competence

3. Universiteit Maastricht overview of the presentation -why this presentation Where do we go from here? - developments in competence assessment - implications for research - developments in performance assessment

4. Universiteit Maastricht overview of the presentation Where do we go from here? -why this presentation - developments in competence assessment - implications for research - developments in performance assessment

5. Universiteit Maastricht experts vs novices: context vs concept Where do we go from here? FzFz FfFf 1,25 kg

6. Universiteit Maastricht overview of the presentation Where do we go from here? -why this presentation - developments in competence assessment - implications for research - developments in performance assessment

7. Universiteit Maastricht Where do we go from here? simple isn’t it? test pass fail competent incompetent

8. Universiteit Maastricht Where do we go from here? simple, it isn’t competent incompetent pass fail

9. modern approaches to assessment Universiteit Maastricht 1re-evaluate the nature of medical competence 2shift towards more student-centred assessment 3include general professional competencies 4broaden the measurement time frame Where do we go from here?

10. 1 trait orientated view on medical competence Universiteit Maastricht knowledge instrument skills instrument problem solving instrument attitudes instrument Where do we go from here?

11. one instrument for one trait Universiteit Maastricht Where do we go from here? instruments traits ABCD 1 2 3 4

12. one instrument for one trait Universiteit Maastricht Where do we go from here?

13. old model of expertise: problems Universiteit Maastricht Where do we go from here? more variation within than between methods

14. 1 trait orientated view on medical competence Universiteit Maastricht knowledge instrument skills instrument problem solving instrument attitudes instrument Where do we go from here?

15. 1 trait orientated view on medical competence Universiteit Maastricht knowledge instrument skills instrument problem solving instrument attitudes instrument Where do we go from here? instrument

16. old model of expertise: problems Universiteit Maastricht Where do we go from here? more variation within than between methods content determines the assessed competence

17. Universiteit Maastricht knows knows how shows how does Miller's pyramid Where do we go from here?

18. Universiteit Maastricht 2 shift towards student-centred assessment -self-assessment -peer-assessment -co-assessment self-evaluation skills assessing the unmeasurable peer pressure? peer-evaluation skills self, peer and co-evaluation skills peer pressure? joint responsibility Where do we go from here?

19. Universiteit Maastricht 3 include general professional competencies dealing with tasks dealing with others dealing with oneself planning and time management information gathering co-operation communicating giving feedback and criticising self-responsibility self-evaluation self-reflection Where do we go from here?

20. Universiteit Maastricht 4 broaden the measurement time frame specific test occasions - specific preparation of students - inflexibility longitudinal testing - unwanted influence on learning - certain aspects unmeasurable - no specific preparation - congruent with flexible curricula - wanted influence on learning Where do we go from here?

21. Universiteit Maastricht lessons learnt Where do we go from here? good assessment draws the whole picture no single method is a panacea the content is important, not the format too much structure does not work sampling strategies are essential

22. Universiteit Maastricht concept and context: subjectivity Where do we go from here? experts LambertMozart

23. Universiteit Maastricht Where do we go from here? sampling strategies are essential how do you feel now how did you feel 5’ ago how did you feel 10’ ago how will you feel in 5’ how will you feel in 10’ what does your partner about think how you feel LambertNeil Dale 7 7 7 7 7 7 5 5 5 5 5 5 9 9 9 9 9 9

24. Universiteit Maastricht lessons learnt Where do we go from here? sampling strategies are essential good assessment shows the whole picture no single method is a panacea the content is important, not the format too much structure does not work

25. Universiteit Maastricht overview of the presentation Where do we go from here? -why this presentation - developments in competence assessment - implications for research - developments in performance assessment

26. Universiteit Maastricht definitions Where do we go from here? competence what professionals can do in controlled representations of professional practice, performing at the maximum of their ability performance what professionals actually do in their professional practice

27. Universiteit Maastricht Where do we go from here? 1 clinical expertise 2 communication skills 3 collaboration skills 4 management skills 6 educational skills 7 professional attributes 8 personal health & self care 5 personal development current approaches: still trait orientated

28. Universiteit Maastricht Where do we go from here? instruments chart review data review interviews rating scales surveys video portfolios incognito simulated patiens critical incident analyses

29. Universiteit Maastricht Where do we go from here? but……. so……. no direct link between instrument and trait domains are not discrete entities trait-orientated approach did not work in competence assessment modern approaches define competence and performance in terms of professional roles

30. teacher Universiteit Maastricht Where do we go from here? patient care person professional role-orientated goals scientist health care management

31. Universiteit Maastricht Where do we go from here? 1 role – 1 instrument? instruments roles ABCD 1 2 3 4

32. Universiteit Maastricht Where do we go from here? multi-modal assessment roles ABCD 1 2 3 4 sources of information

33. Universiteit Maastricht Where do we go from here? how: seeing the whole picture

34. Universiteit Maastricht Where do we go from here? how: approaches - variation in instruments needed - performance often involves subjective aspects -collation of information based on content-equal bits and not on method-equal bits no flight into structuring, but in sampling

35. Universiteit Maastricht Where do we go from here? how: organisation plan purposes elements of defensibility quality control measures pathways implementation administration consequences assessee

36. Universiteit Maastricht Where do we go from here? observations assessment is not mainly a measurement problem but more an (educational) design problem. instrument is not synonymous with a competence no single instrument is the panacea, a variety is needed: an assessment programme content is more important than format: it is the concept that counts, not the format

37. Universiteit Maastricht Where do we go from here? suggestions assessment is not mainly a measurement problem but more an educational design problem. - built-in quality control procedures and bodies - sampling in stead of (over) structuring - variation of methods, compensation, regulations - saturation of information approach - training of students, teachers, examiners, board members masterplan of the assessment system

38. Universiteit Maastricht overview of the presentation Where do we go from here? -why this presentation - developments in competence assessment - implications for research - developments in performance assessment

39. Universiteit Maastricht Where do we go from here? trait orientated assessment as a psychological testing problem - highly quantitative methodologies - reliability of methods - construct validity +++

40. Universiteit Maastricht Where do we go from here? trait orientated assessment as a instructional design problem - quantitative methodologies - reproducibility of methods - construct validity - qualititative methodologies - consequential validity - research into assessment systems - direct validity + + + ± ++ +++ ++++++

41. Universiteit Maastricht Where do we go from here? observations research of assessment systems is more important now than research into single methods. methodology is not synonymous to research question no single one study/methodology will provide the definitive answer, a variety is needed the methodological concept is important not the specific analysis or method

42. Universiteit Maastricht Where do we go from here? research agenda what constitutes good assessment programmes? which building blocks should be included? how should information be collated: qualitatively / quantitatively? assessment as a programme: who should do what?

43. Universiteit Maastricht Where do we go from here? research agenda g-analyses reproducibility of (portfolio) judgments? other statistical approaches sequential testing? organisational approaches?

44. Universiteit Maastricht Where do we go from here? Research agenda how to “score” qualitative judgments? quality elements of multi-modal assessment identifiers for profile scores what influences the thinking of judges? fraud/plagiarism (internet) flexible adaptation of teaching to assessment outcomes

45. Universiteit Maastricht Where do we go from here? research agenda roles of format, content, scheduling and regulations consequential validity / educational impact what influences teacher behaviour

46. Universiteit Maastricht Where do we go from here? research agenda consequential validity / educational impact

47. Universiteit Maastricht Where do we go from here? research agenda how to change a programme? research into assessment programmes stakeholder involvement how are the instruments combined? how to use parts in stead of whole instruments? rules of design

48. Universiteit Maastricht Where do we go from here? SUMMARY overview of the presentation -why this presentation - developments in competence assessment - implications for research - developments in performance assessment

49. Universiteit Maastricht Where do we go from here? conclusions: in (education), assesment and research strength comes from collation of results the method is not synonymous with the concept no single method is a panacea translation of concepts into methods is useful

50. Universiteit Maastricht Where do we go from here? Thank you