1. Tips & Tricks for Developing or Improving Your Auditing Function Leslie M. Howes, MPH, CIP Director, Office of Human Research Administration Harvard School of Public Health Harvard Medical School & Dental School

2. Today’s Outline Audit Basics Key Elements Audit Process Preparation Onsite Engagement Report & Resolution Follow-Up Metrics & Trends Questions 2

3. Auditing Basics One working definition… A systematic and independent examination of study- related activities and regulatory documentation to evaluate protocol adherence and compliance with regulation, institutional policy, ethical guidance, and applicable guidelines Audit types Not for cause (“Routine”) For cause (“Directed”) 3

4. Creating a Strong Audit Function: Key Elements 4

5. Key Elements Understand your “client” Define Scope and Authority Obtain Institutional Support Obtain Research Community Buy-in Establish Goals/Focus Develop SOPs and Templates 5

6. Who’s Your Client Institutional culture Principal Investigators’ involvement in their studies Organizational/Departmental structure/hierarchy Who’s likely to take advantage Research portfolio FDA vs non-FDA regulated Biomedical vs Socio-behavioral Full Board vs Expedited review Study populations Domestic vs International 6

7. Scope & Authority Auditing (for cause/not for cause) Investigators only IRB only Both Investigators and IRB Auditing + education Auditing + education + human research support IRB submission assistance External audit prep Study consultation On-call research coordinator, etc. 7

8. Scope & Authority, cont. Reportable Information, Noncompliance, Suspensions & Terminations Identify Refer/Report to IRB (with recommendation) Make official determination Reprimand Investigators Conduct further investigation Assist PI in reporting 8 Tip: If the IRB is responsible for making official determinations, develop an escalation plan –be transparent with your research community

9. Institutional Support Identify Key Constituents Institutional Official, VP of Research, Department Chairs/Chiefs, IRB Director/Chairs Convince the leadership with data Specific “business plan” for your audit function If available, your own audit findings Relevant FDA Warning Letters, OHRP Determination Letters Borrow “data” from other/neighboring institution Less is more Don’t ask for “too much” in terms of resources (in the beginning) 9

10. Research Community Buy-In Provide Incentives Human research training credit IRB Submission Assistance support Not for cause audit “free pass” for defined time period Pilot different approaches Voluntary 360 o Review Seek feedback from current “clients” and share with the research community Share relevant “common findings” with investigators 10

11. Establish Goals S.M.A.R.T Specific Measurable Attainable Realistic Time-related Flexible/Creative Anything is better than nothing Regular adjustment Maximizing efficiency and effectiveness 11 Consider the following goals, what’s more realistic? A. Offer 3 departments the opportunity for an onsite review vs B. Conduct 30 onsite reviews

12. SOPs and Templates Develop Standard Operating Procedures Easy to create and update User-friendly Develop templates Audit report Standard communications, e.g., letters/emails Compile Common Observations 12

13. An Overview of the Audit Process 13

14. Preparation Phase 14

15. Preparation Identify Protocol (not for cause) Notify Investigator et al., as appropriate When to communicate, if at all (unannounced visit) What to communicate (What, Who, How) How to communicate, e.g., email, phone, website Schedule Date/Time All at once vs separate components, e.g., interview, onsite review How much time -for investigators, support staff Secure private space 15

16. Preparation, cont. Review IRB File, etc. Protocol Funding source Study tools, e.g., survey Consent documents (all versions) External/Ancillary reviews, if any Modifications to date Reportable Information to date, e.g., AEs, Safety Reports, etc. IRB determinations and concerns, if any Directed/For-cause/Area of focus, e.g., participant payment Prepare Tools, e.g., notes vs audit worksheet/checklist 16

17. Onsite Engagement Phase 17

18. Onsite Engagement Introduction Set the tone QA/I Program (Scope, Authority) Overall process, purpose Type of audit, e.g., for-cause vs not-for-cause Questions Discussion vs Interview PI to share challenges with study implementation, IRB 18 Not For Cause Audit Tip: Ease into the review; develop rapport with the PI/study team. This will set the stage for an educational, not adversarial approach.

19. Onsite Engagement, cont. Orientation to organization system Ensure access to all materials Review of Regulatory Documentation Document observations Make photocopies, when necessary Review of Participant files File selection –how to determine sample size? Be flexible –you may need to expand the sample 19

20. How to Determine Sample Size Consider the goal of the QA/I Program 100% Compliance vs “Sampling” of Noncompliance May depend on… Specific institutional concerns Audit trigger, e.g., Routine vs For-Cause audit; General vs Directed/Focused audit 20

21. Sample Size, cont. 100% Compliance – no sample plan needed “Sampling” of Noncompliance – must identify a representative sample Random – selection process occurs at random, with equally likelihood that one protocol would be identified vs another; lacks any pattern E.g., http://www.random.org/sequences/http://www.random.org/sequences/ Online tool that generates a sequence of numbers that can be matched to a protocol number 21

22. Sample Size, cont. Non-Random – You decide! Informed by… Institutional Environment/Policy Prior Noncompliance/Trends Other Industry Standards, e.g., 10% of total sample Key stakeholders, e.g., IO, IRB, or investigators 22 Tip: Whatever you chose, be transparent with the research community about your selection process

23. Onsite Engagement, cont. Exit Interview What is the PI/study team doing well What does the PI/study team need to work on Summary of noncompliance Explore “root cause” 23

24. Report & Resolution Phase 24

25. Report & Resolution Compile Report Provide in a timely manner, e.g., 5 business days Recommended elements Introduction (who, what, when) Observations Provide examples, frequencies Corrective Actions Best Practice Recommendations Regulatory citations Summary, e.g., “…regulatory document well organized…/…could use improvement…” 25 TIP: Be concise, keep it factual, make it easy for investigators to understand what you found and how to correct it, use tables & bullets (avoid narrative)

26. Report & Resolution Create Report Template Create common observations Improves efficiency and consistency Facilitates tracking and trending Consider including General Observations Best Practice Recommendations Corrective Actions Regulatory citations 26 TIP: Consider your institutional research portfolio, e.g., SBER, IND/IDE, etc., and seek input from stakeholders, e.g., IRB

27. Report Distribution: A Case Study 27

28. Report Distribution Example 28 Not for Cause/Routine For Cause/ Directed Potential serious or continuing noncompliance observed No potential serious or continuing noncompliance observed PI; key study staff IRB Institutional Official Others, e.g., VP of Research, department chairs, Associate Dean of Students PI; key study staff TIP: Share aggregate findings from not for cause audits with key stakeholders

29. Follow-Up Phase 29

30. Follow-Up Options Obtain confirmation that correction actions have been implemented QA/I staff conduct 2 nd, 3 rd, etc., audit PI attestation, e.g., formal letter, eIRB system verification Additional audits of investigator/department QA/I staff provide support to PI to implement corrective actions/best practice recommendations Education General education offerings Focused in-serve for PI and study team PI evaluation of audit 30 TIP: Consider your resources; conduct a risk-benefit analysis

31. Metrics & Trends 31

32. Audit Metrics & Trends Collect data Not-for-cause/for-cause audits Common observations Post-audit evaluation data Basic demographics/characteristics Analyze data Track frequencies Identify trends, e.g., isolated, pattern, systemic Inform general education offerings Share findings with key stakeholders 32

33. Questions? 33