1. Program Administrators
University of Alberta
Pre-survey Visit
March 16, 2017

2. Objectives Overview of accreditation
Review the process and organization of onsite surveys
The role of the program administrator in onsite surveys
In preparation for the review
The day of the review
Following the review
Provide the program administrator with practical tools and tips
To facilitate a smooth survey
For the organization of program materials
Opportunity to network with colleagues

3. What is Accreditation? IS a process to -
Improve the quality of postgraduate medical education
Provide a means for objective assessment of postgraduate medical education programs against minimum standards
Assist program directors in planning program developments
IS NOT a process to determine –
Excellence of departments
Excellence and reputation of research
Quality and expertise of clinical faculty
Moral worth of program directors
Performance of residents
DENIS

4. General Standards of Accreditation
“A” Standards:
Apply to the university
Verified by the postgraduate review team at the onsite survey (chair/deputy chair/various representatives)
“B” Standards:
Apply to EACH residency program
Verified by surveyors (one per program)
“C” Standards
Apply to EACH Areas of Focused Competence (AFC) programs
DENIS

5. “B” Standards
B1 Administrative Structure B2 Goals and Objectives B3 Structure and Organization of Program B4 Resources B5 Clinical, Academic and Scholarly Content of Program B6 Assessment of Resident Performance
DENIS

6. Quiz Question 1 Quiz Question 1 How often does your RPC meet?
Do you know how often the RPC must meet according to Standard B1?
Must meet at least quarterly (four times) a year.

7. Specialty-Specific Requirements
Objectives of Training (OTR)
Specialty Training Requirements (STR)
Specialty-specific Standards of Accreditation (SSA)
DENIS

8. Quiz Question 2
Do you know where to find the specialty-specific documents (OTR, STR, SSA) for your program?
royalcollege.ca/accreditation-standards
DENIS
Residents may or may not attend your review – don’t be alarmed if they are there – might not mean anything at all

9. The Regular Onsite Survey
Every six years
All standards are reviewed for PGME office (“A” standards)
All programs are reviewed
All standards are reviewed for each program (“B” standards)
LAURIE
Regular onsite survey
Every six years
All programs are reviewed – a few exceptions which will be reviewed later
Mid-Cycle Internal Review
- Integral component of the accreditation process – monitoring of programs. Conducted at least 2 years prior to the regular College visit. Format is frequently modelled by the regular College visit and is a helpful trial run
Enables the medical school to make corrective measures prior to the regular College visit
Not evaluated for the content but come time for the next onsite, the survey chair will ask for them to ensure that they’re happening

10. The Onsite Survey Process
Completing the pre-survey questionnaire
Preparing for the onsite survey
Schedule
Documents
The onsite survey
Following the survey
From a program administrator’s perspective, four key stages:
LAURIE
Focus of today’s presentation will be onsite survey –
Onsite surveys will affect all of you at some point
Other types of reviews (one-offs) may or may not affect you but if you understand the onsite process, other reviews are variations of this
PA’s role is key the accreditation process and to keeping everyone on track.
We’re going to look at the onsite survey from a PA’s perspective. Four key stages that we’ll look at in more detail:
Completing the pre-survey questionnaire
Preparing for the visit, including developing the schedule for the visit as well as preparing the required documents
The actual onsite visit
What happens after the visit
Not going to look at review of A standards (university level) or C standards (AFC)

11. Which Programs will be Reviewed?
All programs that are “active” at the time of the onsite
There is a resident registered for the complete program (participates in all components of the accredited program, including all of the CanMEDS teaching)
Resident is at least six months into the specialty-specific portion of the program
Exceptions:
Off-cycle (depends on program’s status)
Active less than six months
Inactive programs
No resident at the time of the survey
Program is not reviewed
Inactive across two regular surveys – program is withdrawn
LAURIE
Regular onsite survey
Every six years
All programs are reviewed – a few exceptions which will be reviewed later
Mid-Cycle Internal Review
- Integral component of the accreditation process – monitoring of programs. Conducted at least 2 years prior to the regular College visit. Format is frequently modelled by the regular College visit and is a helpful trial run
Enables the medical school to make corrective measures prior to the regular College visit
Not evaluated for the content but come time for the next onsite, the survey chair will ask for them to ensure that they’re happening

12. The Pre-survey Questionnaire (PSQ)
Two parts:
Generic
Specialty-specific
Appendices:
Additional documents that must be submitted with the PSQ
List of required documents provided with PSQ
Additional attachments
Other relevant documents (e.g. ToR for RPC)
LAURIE
Sent approx 12 months prior to the review
Effective July 2016 – New format
Appendices
RPC agendas
Resident safety policy
Objectives, teaching sessions, teaching staff across relevant disciplines (radiology, pathology)
Conferences, half days schedules
Terms of reference for RPC, structure of RPC

13. Completing the PSQ Ensure all sections are complete
Include current information:
Dates, # residents, faculty, sites, etc.
Data for the last 12 months
Faculty publications (12 months); resident publications (six years)
The PSQ is the first impression of the program:
Clear and concise
No typos
Spell out abbreviations
LAURIE

14. Submitting the PSQ The PSQ is submitted to the PGME office
Deadline provided by your PGME office
Follow the Royal College guidelines for submitting the PSQ and appendices
PDF versions
Files labeled appropriately
E.g. PSQ_Part_I, Appendix_A
Guidelines will provide a checklist of what to include
Ensure all additional attachments are labelled and indicated in Part I of PSQ
LAURIE
Deadline to PGME office is usually 6 months prior to the survey – the PGME will review and submit to the RC on your behalf.
Importance of format when submitting:
- looks how you would like it presented (no changes or formatting)
organized in format that you would like it presented
- You may wish to include a title page for each of the appendices – it will be clear when the surveyor is reviewing the documents what they’re looking for

15. What Happens to the PSQ? University Royal College Specialty Committee
Questionnaires
Specialty Committee
Questionnaires
Royal College
Comments
Questionnaires and
Comments
Comments
Program Director
Surveyor

16. Quiz Question 3
Who should you contact if you have any questions at any point during the accreditation process?
Your PGME office
The Royal College does not communicate directly with program administrators or program directors
LAURIE
All questions should be directed to the PGME office:
- They may have the answers
- They are the liaison to the RC – if they don’t know the answer, they will contact us
The Royal College does not communicate directly with PAs or PDs
- Need to ensure the PGME office is always in the loop

17. The Master Schedule Length and date of each program review, based on:
the number of residents in each program
travel to different sites
the previous survey schedule
Coordinated by the Royal College and PGME Office
Announced as soon as possible after the pre-survey visit
Inform PGME Office quickly if you foresee a scheduling conflict
e.g.: residents/faculty attending specialty conference
Surgical Foundations will be reviewed typically on Monday morning
Final exit meeting with university will take place on the last day
Royal College/CFPC survey team chairs present summary of survey week to Faculty
DENIS

18. The Onsite Survey Team Chair – Dr. Glen Bandiera
Deputy Chair – Dr. Alan Chaput
Surveyors
will be a specialist from another discipline
one surveyor per program
Resident representatives (RDoC)
one to four representatives (depends on the number of programs)
accompany surveyors on selected program reviews
Regulatory authorities representative (FMRAC)
Teaching hospital representative (HealthCareCAN)
Other observers
DENIS

19. Quiz Question 4
If one of the resident representatives accompanies the surveyor for your program, does that indicate major concerns with the program?
No – the resident reps determine which programs they will visit
There may also be observers with the surveyor
DENIS
Residents may or may not attend your review – don’t be alarmed if they are there – might not mean anything at all

20. Creating a Program Schedule
The PGME office will provide you with a copy of the template for the schedule for your program
The template will specify:
Which meetings will need to be scheduled
The duration of each meeting
The order of the meetings
Any questions/concerns with the schedule should be directed to your PGME office
DENIS RC
Always through the postgrad office – you don’t contact the RC
You must follow the sequence that the RC sends you
The PGME office will provide you with your date
and you report any extraordinary circumstances to the PGME, and only if the conflict is major, then the PGME transmits this conflict to the RC.
And the PGME transmits the final schedules to the RC
As soon as you learn the date, book your involved staff, residents, etc right away.

21. The Survey Schedule Must include (in sequence):
Document review
Program director
Department/division chairs**
Residents** (groups of 20)
Teaching faculty**
Residency Program Committee
Exit meeting (15 min) – the next morning at the hotel (7:30am)
Include as appropriate:
Lunch (30 min)
Breaks (15 min) – mid-morning and mid-afternoon
** Scheduled anytime after PD but before RPC
DENIS
There is some flexibility – the meetings highlighted in grey can be moved around but other meetings needs to follow given sequence

22. Other Information to Include
Contact information
Names, titles, locations, phone numbers
List of all participants for each meeting
Names, titles, PGY levels for residents
Indicate if in person or via videoconference
Indicate if breakfast, snacks, or lunch are provided
Details of transportation
Name and contact information
DENIS

23. The Schedule Template DENIS
Sample template – you will receive a blank copy of the template to fill in as well as a instructions go along with it and tell you more specific details about each meeting.

24. Quiz Question 5
How do I adjust the schedule template if I have a large program and the survey takes more than one day?
Same template applies
Each day must finish by 4:00 p.m.
The rules for the sequence of meetings is the same
Day 1 – Document review, Program Director
Last day – Ends with RPC
DENIS

25. The Survey Schedule Pick up at the hotel – 7:45 a.m.
Document review (45 min)
Scheduled at the beginning of the review
Available in the room:
RPC meeting minutes (last six years)
Resident assessment files
If automated assessments (e.g. one45), provide access / assistance to a computer
Sampling
Additional files may be requested by surveyor
LAURIE

26. The Survey Schedule Program Director (75 min)
Always scheduled after the document review
To discuss:
overall view of program
how program addressed previous weaknesses
current strengths and challenges
evaluation of each standard
address specialty committee concerns
LAURIE

27. The Survey Schedule Department Head/Division Chair (30 min)
Scheduled after the program director but before the RPC meeting
To discuss:
support for program
resources available to program
LAURIE

28. The Survey Schedule Residents One hour per group of 20 residents
If more than one group, divide by PGY level or junior/senior
Scheduled after the program director but before the RPC meeting
To discuss
How the program is meeting the Standards from their perspective
Strengths and weaknesses of the program
LAURIE
One of the most important meetings of the day
Reassure your residents – they will likely be anxious
- Their training will not be affected by the review
- They will not be named in the survey report – feedback will be anonymous as much as possible
- They will all be asked to participate

29. The Survey Schedule
Residents (cont’d)
ALL residents invited to attend/participate in their resident meeting(s)
Residents who cannot attend in person?
Arrange link by videoconference or teleconference
Are visa trainees or (clinical) fellows invited to attend?
Different funding / title across Canada
For accreditation purposes only, a resident is a person who is following the same academic program/training as a resident eligible to write College exams – funding doesn’t matter!
LAURIE
One of the most important meetings of the day
Reassure your residents – they will likely be anxious
- Their training will not be affected by the review
- They will not be named in the survey report – feedback will be anonymous as much as possible
- They will all be asked to participate

30. The Survey Schedule Suggestions to helping residents prepare:
Residents (cont’d)
Suggestions to helping residents prepare:
Complete the ‘confidential’ RDoC questionnaire
Meet as a group to discuss strengths and challenges prior to visit
LAURIE
One of the most important meetings of the day
Reassure your residents – they will likely be anxious
- Their training will not be affected by the review
- They will not be named in the survey report – feedback will be anonymous as much as possible
- They will all be asked to participate

31. The Survey Schedule Faculty (60 min)
Scheduled after the program director but before the RPC meeting
To discuss:
Involvement with residents
Communication with program director
Teaching faculty who sit on RPC need only attend RPC meeting. If all faculty sit on RPC, separate faculty meeting is not required.
LAURIE

32. Program director attends first-half of meeting
The Survey Schedule
Residency Program Committee (1 hour)
MUST be the LAST meeting of the review
All members to attend, including resident representative(s)
Smaller programs
Teaching faculty and RPC meetings can be combined when most of teaching faculty are also members of RPC
** Important**
Program director attends first-half of meeting
LAURIE
Program director and program admin will both be asked to leave part-way through the meeting. Standard protocol – allows for honest discussion and to observe how engaged the RPC is in the program.

33. The Survey Schedule Exit meeting - with program director
The morning following the review – 7:30 a.m.
very brief (15 minutes)
At the hotel
Surveyor informs program director
recommendation on the category of accreditation
the strengths and weaknesses of the program
No written report is given
details will be included in the survey report
LAURIE
Surveyor is presenting a recommendation only – the final decision rests with the Accreditation Committee

34. Things to Consider Availability of meeting rooms Transportation
Responsibility of the program
Pick-up at hotel at 7:45 a.m.; during review if required; return to hotel by 4:00 p.m.
Confirm attendance of participants
If some participants are unable to attend in person, set up a teleconference or videoconference
Ensure technician is available if required
If resident will be on off-service rotation, inform that service as soon as possible and request that resident be released for the meeting(s)
Arrange catering as necessary
LAURIE
Many reviews happening at the same time so space for meetings might be limited - less and less meeting rooms and more offices
Don’t want to move a review between rooms partway through the day
Surveyors meet with the program director from the previous day at 7:30 am
So to have someone from your dept pick them up at 7:45 is important – they can’t be losing time trying to figure out where to go

35. Tips for Organizing the Schedule
As soon as you learn your date, book your residents and involved staff right away
Book meeting rooms well in advance
Book any video-conference equipment
Review draft schedule with PD before submitting to PGME
Respect recommended time limits
Notify all participants about the review
Their role
Their attendance
Send out a reminder to all attendees a few days before the review to attend their specific meeting and to arrive on time
LAURIE
Role of residents in the review – they may not have any experience so will need to know what accreditation is all about

36. Submitting the Schedule
Submit your draft schedule to the PGME office
They will review it and submit it to the Royal College
The Royal College may request revisions to the schedule
LAURIE

37. Quiz Question 6
Why does the exit meeting not happen until the morning after the review?
Survey team meets the evening after the review
Surveyor presents their findings to the team
The team determines:
The strengths and weaknesses of the program (always tied to standards)
The recommended category of accreditation
LAURIE

38. Documentation - Pre-survey
Surveyors will have access to a number of documents prior to the survey:
The pre-survey questionnaire (including appendices)
The specialty committee comments
The background of the program
Previous survey report; program response; decision letter
Standards of accreditation
Specialty-specific requirements and standards
OTR
STR
SSA

39. Documentation - Pre-survey
Resident representatives on the survey team will have access to the results of the resident questionnaire
Confidential survey sent to ALL residents
Sent prior to the visit by RDoC
Encourage your residents to complete the survey
Only the resident representatives on survey team have access to results
Royal College never has access to the results
PGME Office never has access to the results

40. Documentation - Onsite
Required documents that must be made available to the reviewers during onsite survey:
Residency Program Committee (RPC) minutes for the past six years
Tip: Update your RPC minutes binder following each meeting
Resident files (one or two for a resident from each year) and computer access to One45 to view online resident assessment forms. Also include files of residents in difficulty or requiring remediation.
Tip: Ensure that you have the resident sign a release of information letter to allow the reviewers to view their file. (as per your University policy)

41. Documentation - Onsite
Documents sent along with the questionnaire
All educational schedules for the past two years (i.e.: teaching sessions, journal clubs, grand rounds, professor rounds, visiting professors, attendance, evaluations, etc.)
Overall goals and objectives and rotation specific goals and objectives
Any documents referenced in your questionnaire (i.e.: rotation specific assessment forms, list of faculty publications, list of resident publications, etc.)

42. Preparing the Documentation
Ensure that all documentation is well organized and up-to-date
Resident files (sampling)
Use binders with dividers
Label the cover of each binder
Recommended breakdown:
Contact information
Summary of rotations
Documentation of meetings with PD
Contracts, leaves, remediation contracts w/schedules, objectives, assessments, etc.
Royal College documentation
Assessments
Application forms

43. Preparing the Documentation
Residency Program Committee (RPC) Minutes
RPC must meet regularly, at least quarterly
Any subcommittees? Terms of Reference
Attendance
Resident participation

44. Tips for Review Day
Be there and be available to assist the surveyors as needed
Ensure that surveyor is comfortable
private room, free of interruption
table and enough chairs
beverages, healthy snacks for breaks, lunch
identify closest washroom
Organize documents in the room
computer available if assessments are electronic
be available if additional documents are requested or if they run into technical difficulties
Arrange for retrieval of confidential documents at the end of the document review

45. Tips for Review Day Follow the schedule
Ensure that participants attend and arrive on time
Send friendly reminders
Those who will be driving surveyors
Participants
If meeting room is unfamiliar, directional signs in the hallway
If using videoconference
Ensure that equipment is in place and support is available
If multiple sites, signage indicating location
‘Do Not Disturb – Meeting in Progress’ sign on door

46. Tips for Review Day
If guest Wi-Fi available, suggest having a sign with user name and password in meeting rooms
Power available for laptops
Suggest having paper copies of schedule available
For larger groups, suggest having attendance sheets
Break it down by expected attendees and regrets

47. Quiz Question 7
During the onsite visit, what should you do if the surveyor requests a resident file and the resident has not given permission for their file to be shared?
File cannot be provided to the surveyor
Advise the surveyor that resident has not consented
Tip: When obtaining consent prior to the survey, reassure residents that their files are being reviewed for process:
Are meetings with PD happening?
Are assessments performed and documented in a timely way?
What is process for remediation?

48. What happens after the survey?
survey team
Reports
specialty committee
royal college
university
Report and
Response
Reports
Responses
Recommendation
Reports and Responses
RESIDENCY accreditation committee

49. Post-survey Timelines
Survey reports
sent to PGME office
six to eight weeks after survey visit
will address each of the standards and outline recommendation for the category of accreditation
Program response
program has two weeks to respond to errors of fact
sent to the PGME office for transmittal to Royal College

50. Post-survey Timelines
Final decision is made by the Residency Accreditation Committee (Res-AC)
Res-AC will discuss June 2018
Dean and postgrad dean will attend
Decision letter will be issued that specifies:
Strengths and weaknesses
Category of accreditation
Required follow-up and deadline
Appeal period (60 days after decision letter is issued)

51. Categories of Accreditation
Approved by the Royal College, CFPC and CMQ
Accredited program
Follow-up:
Next regular survey
Progress report
Internal review within 24 months
External review within 24 months
Accredited program on notice of intent to withdraw accreditation
External review conducted within 24 months

52. Quiz Question 8
Do residents lose their training if the program receives a status of ‘accredited program on notice of intent to withdraw accreditation’?
No – notice of intent is determined by the types of weaknesses identified at the time of the survey and results in an external review within 24 months
If accreditation is withdrawn at the subsequent review, the residents receive credit for their training to the end of that academic year
The university is responsible for placing the residents at another university to complete their training

53. Following the Survey It’s not “over”
Celebrate your program’s strengths
Use the weaknesses to identify opportunities for improvement
Accreditation is a cycle of continuous quality improvement
Keep copies of all documents pertaining to the survey:
PSQ
Survey report
Program response
Decision of the Residency Accreditation Committee

54. Lessons Learned Be familiar with the standards
Keep documents up-to-date in between visits
Start early, do a little often
Reminders are essential – participants are busy people
Get organized early
Checklists are helpful

55. Things to Remember
The program administrator plays a key role in the accreditation process
Your knowledge and organizational skills will be an asset to the program throughout the accreditation process
You are not alone
Every program across the country goes through accreditation
Don’t be afraid to ask questions
PGME office
Colleagues (within university or from same discipline)

56. Conjoint Residency Education Accreditation System Reform

57. The Canadian Residency Accreditation Consortium (CanRAC)
Collaborative initiative between the Royal College, CFPC, and CMQ
New system of residency accreditation:
21st century best practices in accreditation;
Digitized; and,
Aligned with the shift towards competency-based medical education (CBME). 58
9/21/2018

58. 59

59. Shadow Surveyors Objective Role
To leverage the knowledge and expertise of experienced surveyors acting as “shadow surveyors” at an onsite survey and collect the necessary feedback on the new standards and process in accordance to the accreditation reform objectives
Role
Evaluate and validate new standards and process during an onsite survey visit
An “overlay” to the visit for selected programs or the institution to test the process with no impact to accreditation status / recommendations or decisions 60

60. Shadow Surveyor Code of Conduct
The shadow surveyor:
will act strictly as an observer during the onsite visit
will not engage in the discussions at various meetings, will not ask questions or seek clarification
will not influence in any way the recommendation of the surveyor and the survey team during the evening discussions
will not have voting privileges on the program‘s accreditation status recommendation 61

61. University of Alberta Onsite Survey
November 26 to December 1, 2017