1. 1 Ambulatory Pediatric Association Educational Guidelines for Pediatric Residency Tutorial 6 Source: Kittredge, D., Baldwin, C. D., Bar-on, M. E., Beach, P. S., Trimm, R. F. (Eds.). (2004). APA Educational Guidelines for Pediatric Residency. Ambulatory Pediatric Association Website. Available online: www.ambpeds.org/egweb. ©2004 Ambulatory Pediatrics Association. All Rights Reserved.

2. 2 Part 1 Principles of Program Evaluation

3. 3 Ambulatory Pediatric Association Educational Guidelines for Pediatric Residency Why Evaluate a Program? Evaluation helps a program ensure that it is:  fulfilling its mission  meeting accreditation requirements  satisfying stakeholders  meeting learners’ needs Evaluation of past performance informs and invigorates future program planning

4. 4 Ambulatory Pediatric Association Educational Guidelines for Pediatric Residency Types of Program Evaluation Program evaluations can take the form of:  Needs assessment* ( What should we teach our learners?)  Implementation monitoring* ( Are we conducting the program as planned? )  Summative evaluation ( Are we accomplishing our long-term objectives?) * Topics for Tutorial 2: Program Planning. This tutorial focuses on summative evaluation

5. 5 Ambulatory Pediatric Association Educational Guidelines for Pediatric Residency These Processes Are Interrelated Needs assessment creates context for later evaluations Monitoring data are indispensable to summative evaluation Summative evaluation can indicate need for reassessment of program needs and redesign of activities

6. 6 Ambulatory Pediatric Association Educational Guidelines for Pediatric Residency What is Summative Evaluation? Summative evaluation is retrospective It summarizes where the program has been and asks:  Have we fulfilled our mission?  Are our graduates competent?  Are the needs of our patients being met?  Did unintended consequences occur?

7. 7 Ambulatory Pediatric Association Educational Guidelines for Pediatric Residency Components of Summative Evaluation A comprehensive program evaluation:  Gathers data on learning outcomes  Evaluates the content and process of each program component  Assesses impact of the program as a whole More targeted evaluations are common when programs have limited time and resources Set realistic targets and choose methods that your program can afford

8. 8 Ambulatory Pediatric Association Educational Guidelines for Pediatric Residency Examples of Small-Scale Evaluation Targets Cardiology rotation’s evaluation processes Residents’ teaching of students on the wards Communications role modeling in the PICU EBM learning outcomes of a sample of residency graduates

9. 9 Part 2 Key Steps to Evaluate an Educational Program

10. 10 Ambulatory Pediatric Association Educational Guidelines for Pediatric Residency Key Steps 1.Plan in advance 2.Choose sound evaluation methods 3.Evaluate learner outcomes 4.Evaluate major program components 5.Evaluate program as a whole 6.Conduct continuing review and program modification

11. 11 Ambulatory Pediatric Association Educational Guidelines for Pediatric Residency Step 1: Plan in Advance Pre-planning is essential to provide adequate data for a later evaluation Plan establishes the primary evaluation questions. E.g.: Are program objectives being met? Are essential processes working? Are learners and faculty satisfied? Have unintended consequences occurred? Pre-planning also defines the audience for the evaluation report and their unique priorities.

12. 12 Ambulatory Pediatric Association Educational Guidelines for Pediatric Residency Step 1: Plan in Advance, con’d Pre-planning also addresses:  Logistical issues : who, what, where, when, how?  Resources available : time, effort, money, compliance from faculty and residents  Information sources: learners, faculty, other stakeholders An efficient plan devotes the most resources to the highest priority educational issues  Easily measured  highest priority!

13. 13 Ambulatory Pediatric Association Educational Guidelines for Pediatric Residency Example of a Cost-efficient Evaluation Strategy “Counting data” (cheap and easy) are collected on a systematic sampling of program activities Process evaluations (more costly) are conducted in higher priority areas Learner outcome assessments (most resource-intensive) are reserved for top priority goals of program

14. 14 Ambulatory Pediatric Association Educational Guidelines for Pediatric Residency Step 2: Choose Sound Evaluation Methods Impact on program: How will the evaluation affect the faculty, learners, stakeholders? Validity: Does the evaluation assess important issues and provide data useful for program improvement? Are all important information sources included? Reliability: Would we get the same results if we repeated the evaluation? Is the sample representative, and do the methods avoid error and bias?

15. 15 Ambulatory Pediatric Association Educational Guidelines for Pediatric Residency Step 2: Choose Sound Evaluation Methods, con’d Feasibility: Does the evaluation make the best use of limited resources? Will teacher and learner compliance be a problem? Generalizability: Will your findings transfer to other settings? What really made your intervention successful, and could that be replicated elsewhere?

16. 16 Ambulatory Pediatric Association Educational Guidelines for Pediatric Residency Step 2: Choose Sound Evaluation Methods, con’d The most important consideration in selecting evaluation methods is: “ Do they address your highest priority questions? ” Not all important questions can be reduced to numerical responses! Often qualitative methods (e.g. focus groups) are very informative. In the final analysis, our evaluation processes need to be driven not by convenience or comfort, but by our ultimate educational goals.

17. 17 Ambulatory Pediatric Association Educational Guidelines for Pediatric Residency Step 3: Evaluate Learner Outcomes Aim for a generous sampling of performance from multiple residents in diverse settings Collect data from past graduates as well as current trainees The clinical setting is the best place to evaluate learner outcomes In the future, we hope better methods for practice-based evaluations will be developed!

18. 18 Ambulatory Pediatric Association Educational Guidelines for Pediatric Residency Step 4: Evaluate Major Program Components Most programs collect mainly attendance and learner satisfaction data on their teaching and learning activities. More objective sources are also needed. E.g.:  Pre- and post-intervention measures of learners’ knowledge, skills and attitudes  Expert evaluations of faculty teaching effectiveness

19. 19 Ambulatory Pediatric Association Educational Guidelines for Pediatric Residency Step 4: Evaluate Major Program Components, con’d Outcome assessments of clinical teaching and learning ideally should also review patient satisfaction and health outcomes Future development of improved evaluation methods may make such outcome assessments more feasible than they are today

20. 20 Ambulatory Pediatric Association Educational Guidelines for Pediatric Residency Step 5: Evaluate Program as a Whole All residency programs participate in program-wide evaluations during accreditation reviews. Such evaluations are very thorough and reasonably objective. A program may also conduct evaluations for its own purposes. E.g.:  Is it meeting patients’ needs?  Is it meeting learners’ needs?  Is it being run in a cost-effective manner?  Is it successful in attracting strong applicants?

21. 21 Ambulatory Pediatric Association Educational Guidelines for Pediatric Residency Step 5: Evaluate Program as a Whole, con’d A key question: Are your methods sound? To obtain objective data, programs often review mean test scores of residents and graduates; although fairly objective, such tests do not evaluate performance. More comprehensive data on skills and attitudes comes from multi-source evaluations of residents or graduates in practice settings. New methods are needed, especially to assess patient outcomes as an indicator of graduate competence.

22. 22 Ambulatory Pediatric Association Educational Guidelines for Pediatric Residency Step 6: Conduct Continuing Review and Modification Despite methodological shortfalls, program evaluation is often useful to programs in:  Indicating general areas of strength or weakness  Highlighting unusually strong or weak teachers  Identifying focal points of friction or discontent that need attention

23. 23 Ambulatory Pediatric Association Educational Guidelines for Pediatric Residency Step 6: Conduct Continuing Review and Modification Such data, if interpreted with caution, can help the program to:  Discover activities that need redesign  Target resources to critical remediation efforts  Establish priorities for continuous quality improvement As the RRC progresses toward better defined program evaluation criteria, programmatic quality improvement will become a key focus of future effort.