1. 1 Meeting Milestone Teaching and Assessment Requirements for Problem-Based Learning and Improvement (PBLI) Allen F. Shaughnessy, PharmD, MmedEd Tufts University School of Medicine Department of Family Medicine Boston, Massachusetts David C. Slawson, MD The University of Virginia, Department of Family Medicine Charlottesville, Virginia

2. Objectives for today Describe the different aspects of the PBLI milestones List the curriculum elements necessary to develop resident competence in these areas Describe several resources available to teach the content or that provide faculty development Compare the available assessment methods available for documenting competence. 2

3. So... Where are we with Milestone assessment and documentation? PBLI teaching? 3

4. Overview of the family medicine PBLI milestones, their rationale, and their limitations 4

5. PBLI milestones 1. EBM: “Locates, appraises, and assimilates evidence from scientific studies related to the patients’ health problems” 2. Information Mastery: “Demonstrates self- directed learning” 3. Population Management: “Improves systems in which the physician provides care” 5

6. Anatomy of a milestone 6 Entering residencyEnd of residencyAspirationalDuring residency Not progressive; each level describes separate skills

7. Curriculum elements needed to teach PBLI content 7

8. EBM: Critical appraisal, which includes basics of clinical epidemiology, statistics Recognizing study design and quality, threats to validity, and critical appraisal of primary research, reviews (secondary literature), and clinical practice guidelines (tertiary literature). Relative generalizability of research (levels of evidence) and other issues of relevance and applicability. Demonstration of knowledge of different study designs, sources of bias in clinical research, and the concept of an evidence hierarchy/levels of evidence.

9. Information Mastery Lifelong learning skills: Identifying need for information Answering questions at the point of care Keeping up with changes in medicine

10. System/practice improvement Self-analysis of one’s practice System improvement and population management

11. So... Where are we with Milestone assessment and documentation? PBLI teaching? 11

12. Tour of Internet resources available for teaching PBLI, including FMDRL, STFM resources, online curricula, and blended curricula 12

13. Family Medicine Digital Resources Library (www.FMDRL.org)

14. Curricula Family Medicine 2012;44(4):259-264. Acad Med. 2008 Jun;83(6):581-7. doi: 10.1097/ACM.0b013e3181723a5c.

15. BMC Medical Education 2009, 9:59 http://www.biomedcentral.com/1472- 6920/9/59 BMC Medical Education 2004, 4:19 http://www.biomedcentral.com/1472-6920/4/19

16. STFM Resource Library R1

17. STFM Resource Library R2, R3

18. Online Curricula

19. Michigan State University http://omerad.msu.edu/ebm/

20. 20

21. 21

22. 22

23. 23

24. 24

25. Level 5 (your own doc) Know enough to provide basic system- based care Know one’s limits Demonstrate the ability to stay up to date find and apply the best information in the care of patients at the point of care 25

26. Assessment Methods EBM Fresno Test of Evidence-Based Medicine Berlin Questionnaire Knowledge, attitudes, access and confidence Online certification Information Mastery Online certification Documentation of actions Population management: Documentation similar to the MOC process

28. Fresno Test of EBM Basic EBM knowledge Short-answer; very difficult to grade and reproducibility is difficult Composite score does not link directly to milestones Ramos KD, Schafer S, Tracz SM: Validation of the Fresno test of competence in evidence based medicine. BMJ (Clinical research ed) 2003;326(7384):319.

29. Berlin Questionnaire Basic EBM knowledge and search skills Unable to get the actual questionnaire Adapted to psychiatry Fritsche L, Greenhalgh T, Falck-Ytter Y, Neumayer HH, Kunz R: Do short courses in evidence based medicine improve knowledge and skills? Validation of Berlin questionnaire and before and after study of courses in evidence based medicine. BMJ (Clinical research ed) 2002; 325(7376):1338.

30. KACE (Knowledge Attitudes Confidence Access) Measure of implementation Have not used Hendricson WD, Rugh JD, Hatch JP, Stark DL, Deahl T, Wallmann ER:Validation of an instrument to assess evidence-based practice knowledge, attitudes, access and confidence in the dental environment. J Dent Educ 2011; 75(2):131-144.

31. Online Assessment: Information Mastery Curriculum and Assessment Program (our stuff) Assessment in development Mix of knowledge assessment, reflections, self-reported behavior Option: onsite assessment

33. Group discussion: reflection on content, sharing of ideas, planning for networking and support going forward 33