1. Effectiveness in Review & Oversight of Human Subjects Research Steven Joffe, MD, MPH Assistant Professor of Pediatrics

2. Human Subjects Research (HSR) Review & Oversight Complex system for ensuring high-quality, safe, respectful research – Investigators, sponsors, institutions, IRBs, DSMBs, etc. Primary responsibility for independent research review & oversight lies with IRB & its staff

3. Defining Quality (or Effectiveness) in HSR – Health care (Institute of Medicine): “the degree to which health services for individuals & populations increase the likelihood of desired health outcomes & are consistent with current professional knowledge” – Research oversight (by analogy): “the degree to which research review & oversight activities increase the likelihood that HSR will satisfy essential scientific & ethical requirements, & are consistent with current professional knowledge & best practice” JAMA 1998;280:1000

4. What Are The “Essential Requirements?” Belmont Principles Respect for persons – Informed consent – Protections for vulnerable subjects Beneficence – Minimization and justification of research risk Justice – Fairness in distribution of burdens and benefits Respect for persons – Informed consent – Protections for vulnerable subjects Beneficence – Minimization and justification of research risk Justice – Fairness in distribution of burdens and benefits Six Substantive Requirements Social value Scientific validity Fair distribution of benefits & burdens Favorable risk-benefit ratio Informed consent Respect for potential & enrolled subjects Social value Scientific validity Fair distribution of benefits & burdens Favorable risk-benefit ratio Informed consent Respect for potential & enrolled subjects JAMA 2000;283:2701

5. Conceptual Model of Quality/Effectiveness Structure Process Outcome JAMA 1988;260:1743

6. Definitions of Terms Structure – Attributes of settings in which review is conducted Material & human resources; organizational structure; written policies & procedures Process – Activities of review & oversight bodies Outcome – Ultimate ends that review and oversight system seeks to achieve

7. Example Consent form editor on IRB staff Editing of consent form for clarity, readability Understanding among individuals considering research

8. Priorities for the Quality & Effectiveness Agenda 1.Define key outcomes 2.Establish relationships among structure, process & outcome 3.Decide what will be assessed, & how 4.In parallel, begin integrating quality improvement & assessment of effectiveness into IRB culture

9. 1. Define Key Outcomes Six substantive requirements for ethical research – E.g., social value  value of individual studies & of research portfolio – E.g., informed consent  participant understanding, voluntariness of decisions – E.g., favorable risk-benefit ratio  avoidance of preventable research-related harm JAMA 2000;283:2701

10. 1. Define Key Outcomes Also: – Procedural & interactional fairness for investigators – Respect for review & oversight system among investigators – Community & public trust – Efficiency? – Other? Consider stakeholder perspectives Substantive & procedural quality JERHRE 2006;67 BMC Med Eth 2008;9:6

11. 1. Define Key Outcomes Need to decide what the primary objective of the quality/effectiveness agenda is? – Seeking of optimal outcomes? – Avoidance of bad outcomes?

12. 2. Establish Structure-Process- Outcome Relationships Structure-process – E.g., how do IRB resources or written policies & procedures impact quality or comprehensiveness of review? Structure-outcome – E.g., how do members’ expertise & training impact likelihood that scientifically poor-quality protocols will be improved or disapproved? Process-outcome (most important) – E.g., how does attention to clarity of consent form impact likelihood that prospective participants will make informed decisions?

13. 3. Decide What Will Be Assessed Structure? – Easiest to quantify – Easiest to modify – But relationship to outcome must be validated, and is likely attenuated

14. 3. Decide What Will Be Assessed Process? – Most direct measure of IRB performance – But first need to validate which processes are associated with desired outcomes – Proximal vs. distal process

15. 3. Decide What Will Be Assessed Outcome? – Most concrete object of measurement – But numerous challenges and confounders: Normative agreement on what counts Valid measurement instrument “Case mix” Multiple causal influences on outcomes; research review and oversight explains only part of variability

16. 3. Decide What Will Be Assessed Ultimate goal – Assess process variables that are known to be associated with outcome “Best practice” Evidence-based review and oversight – Assess selected outcome measures directly Limited to those most causally related to review and oversight

17. 4. In Parallel, Integrate QI & Evaluation into IRB Practice Self-evaluation Continuous or rapid-cycle QI – Plan, do, study, act Root cause/systems analysis – E.g., investigations into TGN1412 disaster Peer review JERHRE 2008;3:25 http://www.patientsafety.gov/CogAids/RCA/index.html http://www.patientensicherheit.ch/de/projekte/londonprotocol_e.pdf BMJ 2006;333:270

18. What is the Evidence Base Today? Not much Primarily related to informed consent and participant understanding * – Use of consent form templates; presence of 3 rd party in consent conversation; time to consider decision; use of nurse educators; structured consent process – Use of real-time tests of comprehension – Formal assessment of decision-making capacity Ethical evaluation of approved protocols is methodologically possible * See last slide for references

19. Summary Effectiveness agenda needs to start with conceptual model Long-term project: select outcomes, validate relationships to structure and process, and develop measures of key indicators Short-term project: begin to create culture of QI and of evidence-based research oversight Limited evidence base exists on which to build

20. References Lancet 2001;358:1772 J Clin Oncol 1996;14:984 JAMA 2004;291:470 Control Clin Trials 1989;10:83 Pediatr 2007;119:e849 Schizophr Res 2005;80:1 Arch Gen Psychiatry 2000;57:533 Am J Psychiatr 2006;163:1323 Advisory Committee on Human Radiation Experiments. Final Report. New York:OUP, 1996 (Chapter 15)