Pieces of the Puzzle Perspectives on the Accreditation Visit……

Central Theme Accreditation and Assessment are critically linked in contemporary accreditation models. They were parallel themes in recent past but have collided to form a new accountability in higher education. Ratcliff JL et al, How Accreditation Influences Assessment, Josey Bass 2001

How does accreditation and assessment come together? Two key questions: How should learner outcomes be demonstrated in the accreditation process? How should assessment information be used to show improvement in programs, services, and student/resident learning?

Accreditation Purposes Fostering excellence through the development of criteria and guidelines for assessing effectiveness Encouraging improvement through ongoing self-study and planning Ensuring external constituents that a program has clearly defined goals and appropriate objectives, maintains faculty and facilities to attain them, demonstrates it is accomplishing them, and has the prospect for continuing to do so Ratcliff JL et al, How Accreditation Influences Assessment, Josey Bass 2001

Accreditation Purposes Provides advice and counsel to new and established programs in the accrediting process Ensures that programs receive sufficient support and are free from external influence that may impede their effectiveness and the freedom of inquiry Ratcliff JL et al, How Accreditation Influences Assessment, Josey Bass 2001

Program Accreditation Needs to fulfill two dimensions of program quality: There should be some evidence that the programs have clear goals and courses of study to attain them The process should demonstrate that responsibilities associated with the goals are being carried out. Peers interpret quality in the context of the programs own aims and activities Central issue: Program review is conducted as a self study process. Central assumption: Evaluators from outside cannot effectively examine and judge until those inside have reached a judgment about its quality. Ratcliff JL et al, How Accreditation Influences Assessment, Josey Bass 2001

Program Review – Audit Style Issues identified by outside party Buy-in from program director? Reactive – Fear about the inspection Approach – cover the basics, get by this one and on to the next Not stimulated to think about points of excellence – only points of deficiency Outcome from Inspection: Fear relieved – get on to the next one. Yardstick: STANDARDS – Basic/Minimal

Program Review Assessment Style Self Study and Peer assessment against target goals: Minimal standards = musts Growth goals Improvement goals Outcome goals Buy-in for Program Director Proactive – ownership of goals Outcome of Inspection External validation of internal assessment (agree or not) How much progress has been made against goals? Yardstick: Goals achieved or improvement beyond minimal standards – Points of Excellence

Federal and Local Rules OPTI Standards Basic Standards For I R Specialty College Standards Practicality? Last Inspection? Time? Salary? Patient Care? Productivity? Family? What about the Residents? How am I Doing as a PD? Malpractice?

Federal and Local Rules OPTI Standards Basic Standards For I R Specialty College Standards YES!! ??? What role? What am I supposed To do with this?

Federal and Local Rules OPTI Standards Basic Standards For I R Specialty College Standards Do they agree??? ? ? ? ? Which one takes precedence?

Experience with the Audit Program Review…. Attempts to help program directors and DMEs to prepare for upcoming inspections. Attempts to find common areas of support need across OPTI programs of same discipline Based on BASIC STANDARDS – developed Crosswalks to assist in review of individual standards and collective standards.

Standards Crosswalk and Worksheet Specialty:______________ Completed by:_____________ Date:_____

Program Pre-Inspection Findings Hospital_______________ Completed by:________________

Standards often do not: Consider practical issues A complete research paper per year of residency (human subject approvals take almost half that time). Ability of community hospital to afford out rotations since the BBA 1997 Ability of trainer to (again) teach subjects which were dropped from practice in time past. Consider federal rules or procedures Institutional Review Board and HIPPA approval for human subject research Documentation standards

Standards often do not: Reflect the need to include the following in the curriculum and assess effectiveness: Ethics Communication skills Require the assessment of outcomes of study Clinical skill development or OSCEs In Service Examination and Analysis (individual and program wide). Progress exams or oral exams Require that a program develop an ongoing quality assessment program and benchmarks for process and learning outcomes goals.

Experiences with Accreditation Visits – an OPTI view..example: Program Review 40 areas of concern – standards not met in crosswalk Affiliation agreements for major components not completed (for over 10 years) Manual in disarray…never distributed to residents for past three years PD mostly practices at alternate hospitals (where residents not welcome) for >70% time Etc. Inspector Wonderful program with dedicated PD and hospital No major deficiencies Recommend increase in number of trainees Commendable PD attention to program Recommend full five year accreditation. Net result:: OPTI is too tough…. see, We are OK!!

Experiences with Accreditation Visits – an OPTI view..example: Program Review 36 areas of concern on Standards Crosswalk No resident manual in print/no written curriculum Resident work hours = 2 weeks on call continuously 2 yr program required by PD/1 yr by specialty college….few recent trainees complete the two years. Certificates difficult to obtain. Certification rates = <%50 pass rate on first attempt. Inspector exit notes: Excellent program which should be a model for others This program should be expanded to train more residents. No major deficiencies Net result:: PD who had been working with OPTI to correct findings has stopped. We are doing fine….see how well we did on the inspection? Why wont you sponsor more positions?

Experiences with Accreditation Visits – the Inspector Factor Inspectors Often not familiar with Basic Standards for Interns and Residents or OPTI standards Often spend as little as 4-6 hours in the entire process Do not dig deep enough to see issues Accept inspectors workbook without verifying content. Inspectors workbooks and standards often do not reflect the same intent or priority for the issue in question – produce confusion Ranking of standards is evident by threat of automatic probation if question answered NO. Inter-rater reliability seems low

Conclusion – Audit Review Dissatisfaction with end result Gets through the current audit inspector approach Fails to stimulate long term goals or real improvement Something is missing May promote a punitive rather than a supportive atmosphere –

Begin with the end in mind- What is the goal? Get from one accreditation visit to the next? Minimal level thinking Threshold is the lowest possible point to keep the program afloat (?) Continuous improvement over time against declared goals. Determine a base line Set realistic but challenging goals Measure progress….set new goals (meeting minimal thresholds is never a problem)

Goal –get from one accreditation visit to the next? Data driven Minimalist thinking audit mentality Promotes bashing and blaming Sets interested parties at odds Promotes a pessimistic environment and inhibits creative thinking Goals are at the minimal threshold – Basic

Goal – Assessment for CQI and Outcomes Markers of Excellence Internalized goals born in self study and assessment Buy-in for all stakeholders

Goals for the Workshop Recreate the Basic Standards to allow and encourage growth, goal setting and a reach for higher goals. Revise and recreate basic standards to ensure basic elements are present and to foster agreement between: External standards/Internal standards External validation/Internal validation To foster better understanding of the fit between the AOA standards and laws which apply to residency training.

Federal and Local Rules OPTI Standards Basic Standards For I R Specialty College Standards Goal