1. Brian Jolly Centre for Medical and Health Sciences Education Monash University Train the Trainer Workshop October 31, 2003 Hong Kong International Consortium Blueprinting

2. Hong Kong International Consortium Objectives Describe basic ideas behind blueprinting Develop 2 examples Show how this does/does not articulate with IDEAL database

3. Hong Kong International Consortium Principle The PURPOSE of the assessment will define the content, strategy, & technique used.

4. Hong Kong International Consortium Curriculum and Assessment: ‘Ideal’

5. Hong Kong International Consortium Curriculum and Assessment: Real

6. Hong Kong International Consortium H. C. Curriculum and Assessment: Real

7. Hong Kong International Consortium Curriculum and Assessment: Blueprinting

8. Hong Kong International Consortium  Face validity  Content  Construct  Criterion  Effects on learning Aspects of Validity - Constructs

9. Hong Kong International Consortium  Face validity  Content  Construct  Criterion  Effects on learning Aspects of Validity - Constructs

10. Hong Kong International Consortium  Face validity  Content  Construct  Criterion  Effects on learning Aspects of Validity - Constructs

11. Hong Kong International Consortium  Define what is to be tested  Clinical problems at some resolution  Tasks for each problem  Blueprint  Select test methods  Authenticity appropriate to clinical tasks  Clinical tasks determine the method  Titrate against practical constraints Content Steps

12. Hong Kong International Consortium Competence Categories CVSRS…GI…..MSSEndo/ Metab Haem / Oncology …. History Taking Physical Exam Tests and Procedures ….. Management Communication and Patient Education Simple Blueprint

13. Hong Kong International Consortium The Hospital SettingThe Community Setting Competence Categories CVSRS…GI…..CVSRS…GI History Taking Physical Exam xx Tests and Procedures xx ….. Other Parameters - Hierarchical

14. Hong Kong International Consortium InpatientOutpatient/Ambulatory Competence Categories CVSRS…GI…..CVSRS…GI History Taking Physical Exam xx Tests and Procedures xx ….. Hierarchical - IDEAL ® Related

15. Hong Kong International Consortium The ChildThe Elderly Competence Categories CVSRS…GI…..CVSRS…GI History Taking xx Physical Exam Tests and Procedures ….. Other Dimensions

16. Hong Kong International Consortium The ChildThe Elderly Cognitive /Affective Categories CVSRS…GI…..CVSRS…GI Recall xx Reasoning Affect xx Skill ….. Other Dimensions - Response

17. Hong Kong International Consortium The ChildThe Elderly Cognitive /Affective Categories CVSRS…GI…..CVSRS…GI Recall KFI Reasoning Affect CISP Skill ….. Test – Response Matrix

18. Hong Kong International Consortium Steps: A plausible test of Content Validity   Chart distribution of questions by level & topic   Compare this distribution with weighting of topics in curriculum or course documents   Note to establish content validity we need a demonstrated relationship Jolly 2000, Medical Education,

19. Hong Kong International Consortium  Some researchers use judgments of clinicians or successful test-takers  Dimensions of blueprint not always easy to define c.f. the items  How clinical problems, tasks or other 'concrete examples' relate to content often taken for granted in blueprinting Jolly 2000, Medical Education, Steps: A plausible test of Content Validity

20. Hong Kong International Consortium Summary Before selecting evaluation tools, need to create an evaluation “blueprint” The blueprint = matrix that outlines parameters for the assessment in 3 major areas: domains to be assessed, content & assessment instruments

21. Hong Kong International Consortium Blueprint matrix ensures balance across assessment formats - conversely Multiple methods usually required to achieve blueprint objectives Different assessment instruments allow balance of strengths and weaknesses of each method Summary

22. Hong Kong International Consortium  Content  In Australia RACS has sole responsibility for goals, training delivery & standards  See CANMEDS, Duties of a Doctor,  Investigate ways of addressing these issues  Get profession & public to assist  Responsibility for the ‘hard to measure’  RACS blueprint – are we a surgeon first & a specialist second? Blueprinting Issues: e.g., RACS Assessment

23. Hong Kong International Consortium  Judgement (80%)  Literature on judgement  Measures of judgement (as yet undeveloped)  Manual dexterity (20%)  Simulation and testing in training  Reznick, Paisley vs. Beard et al RACS Assessment - Content