FDA and Pharmaceutical Manufacturing Research Projects Jeffrey T. Macher Jackson A. Nickerson Co-Principal Investigators

Presentation Overview  Executive summary  Project goals  Data collection and synthesis  Analysis methodology  Findings  Development opportunities and constraints

Executive Summary  We develop statistical models that predict the:  Probability of a facility being chosen for inspection.  Effect of investigator training, experience, and individual effects on the probability of investigational outcomes.  Characteristics and identities of facilities that correlate with the probability of non-compliance.  We present initial results for each of these analyses.  We identify additional opportunities and next steps to create value along with some constraints.

FDA Research Project History  Research project idea emerged in Fall,  Approached FDA in late Spring,  Formalized relationship with FDA in Fall,  Began receiving data September, 2004.

FDA Research Project Goals  Risk-based assessment of FDA cGMP outcomes.  Identify underlying ability of investigators and their training.  Identify underlying compliance of each facility.  Identify attributes (currently recorded by the FDA) that impact inspection outcomes.  Transfer “learning” to FDA.

Progress to Date  Just as new drugs go through  Discovery  Development and  Commercialization….  Our model and this presentation concludes the discovery phase of our project.  Please think of our model as a “platform” that can be developed to assess a variety of compliance issues.

FDA Project Approach  Compile and link FDA databases.  Estimate the likelihood of various outcomes:  NAI, VAI, OAI; Warning Letters; Field Alerts; Product Recalls.  based on…  compound/product, facility, firm, FDA district, investigator and training derived factors.  in order to …  evaluate the allocation of investigational resources.  inform effectiveness of investigator training and management.

FDA Databases  DQRS (Field alerts)  EES  FACTS (Inspections) – CDER only  Product Listing  Product Recalls  Product Shortages  Facility Registration (DRLS)  ORA Training database  Warning letter database

Data Preparation  Started with FACTS ( ).  Manufacturing facilities only.  Assembled investigator training database:  Identified corporate ownership by plant by year and firms operating at a specific facility each year.  Constructed facility-year data  Added observations for years NOT inspected.  Corrected FEI/CFN mismatches.  Constructed numerous other variables.

Some basic “facts” about the FDA data  Years covered: FY  Total number of facilities inspected: 3753  Total number of “Pac codes”: 38,341  Total number of “Inspections”: 14,162  Total number of investigators: 783

, Number of FDA Facility Visits per Year:

Empirical Methodology  Inspection  Probability of choosing a facility to inspect.  Detection  Probability of a non-compliance inspection outcome.  Noncompliance  Probability of noncompliance, inspection, and detection.  Detection control estimation.

Inspection  Groups of variables:  Technology variables RxPrompt ReleaseExt or Delayed Rel Gel CapSoft Gel CapOintment LiquidPowderGas ParenteralLg. Vol. Parent.Aerosol BulkSterileSuppositories  Industry variables Vitamins (IC 54) Necessities (IC 55) Antibiotics (IC 56) Biologics (IC 57)  Inspection decision variables Ln(Days between inspections) Surveillance = reason for inspection (0 = Compliance) Last inspection outcome (1 = OAI, 0 = NAI, VAI)  Years (binary variables for each year)

Inspection: Explained Variance  R 2 Cumulative R 2 Technology variables 12% 12% Industry variables 9 21 Inspection Decision variables Year dummy variables ~0 51 Omitted categories: Human Drugs (IC 60-66), select technologies, Year dummies Foreign inspection included in analysis but uniquely identifies many inspections and is dropped from the analysis.  Probit analysis of decision to inspect.

Rx0.13** Promp Rel.-0.19** Ext/del Rel.-0.19** Gel Cap-0.25** Soft Gel Cap-0.36** Ointment-0.32** Liquid-0.30** Powder-0.37** Technology Variables: Change in Probability of Inspection Gas-0.68** Parenteral-0.32** Lg Vol Parent Aerosol-0.26** Bulk-0.37** Sterile-0.07** Suppositories-0.23** ** 99% confidence interval * 95% confidence interval +90% confidence interval Omitted categories: Not Classified, Bacterial antigens, Bacterial vaccines, Modified bacterial vaccines, Blood serum, Immune serum.

Industry and Inspection Variables: Change in Probability of Inspection Antibiotics (IC 56)0.19** Vitamins (IC 54)0.11** Necessities (IC 55)-0.06** Biologics (IC 57)-0.07** Industry VariablesInspection Variables Ln(Days btwn Insp)-0.28** Surveillance-0.84** Last outcome0.13** Omitted category: Human drugs ** 99% confidence interval * 95% confidence interval +90% confidence interval

Days Between Inspections Probability of Inspection Years Since Last Inspection

Detection  Groups of variables  Technology  Industry  Training Total training days prior to inspection (other than 5 main drug courses) Drug course 1: Basic drug school Drug course 2: Advanced drug school Drug course 3: Pre-approval inspections Drug course 4: Active Pharmaceutical Ingrediant Mfg. Drug course 5: Industrial sterilization  Investigator Experience Number of inspections in the prior 12 months Number of inspections in the prior months  ORA District Office  Investigator Classification A consolidation of position classifications

Detection: Explained Variance  R 2 Cumulative R 2 Technology variables 0.9 % 0.9 % Industry variables Training and Experience vars Office and Position variables Investigator effect  Probit analysis of decision to inspect.

Training and Experience Variables: Change in Probability of Detection 1 1 Without investigator fixed effects. Total training days prior to inspection (less 1-5)-2.2E-03 Drug course 1: Basic drug school0.07* Drug course 2: Advanced drug school-0.05 Drug course 3: Pre-approval inspections-0.23** Drug course 4: Activ. Ingred. Mfg.-0.15* Drug course 5: Industrial sterilization0.08* No. of inspections in the prior 12 months4.8E-03+ No. of inspections in the prior months-1.4E-03

ORA Office and Classification Variables: Change in Probability of Detection 2 ORA LOS0.07+ ORA KAN ORA NYK-0.07* ORA SJN-0.09** ORA SRL-0.10* ORA ATL-0.10** ORA DAL-0.10** ORA SAN-0.11** ORA DET-0.13** ORA NWE-0.15** All other ORA off. insignificant. Compliance0.04 Microbiologist-0.02 Investigator-0.04 Chemist-0.05 Eng/Sci-0.07 Dist/Reg. Admin FDA Bureau-0.15* Technician-0.18 ORA Office VariablesPosition Variables 2 With investigator fixed effects.

425 Investigators

Non-compliance  Detection Control Estimation  Relatively new procedure used in academic literature.  Used for assessing tax evasion, EPA compliance, and other applications.  FDA application more complicated than other applications.  Assume three actors:  Facility decides level of compliance.  Inspection decision-maker chooses when to inspect.  Investigator chooses detection or not.  Estimate all three processes simultaneously.

Non-compliance model  Assume inspection decisions are non-random.  Assumption is different from other applications.  Construct a likelihood function that models the probabilities of:  a plant being selected for inspection and  the outcome of the inspection.

Constructing a Likelihood Function L 1i = 1 L 1i = 0 L 2i = 1 L 3i = 1 L 2i = 0 The likelihood that facility i is non- compliant The likelihood that facility i is compliant The likelihood that facility i is inspected The likelihood that facility i is not inspected The likelihood that facility i is found non-compliant The likelihood that facility i is found compliant L 3i = 0

Likelihood Function  Three probabilities are combined to form the function:  Probability that a non-compliant facility is inspected and detected: L 1i =1, L 2i =1, L 3i =1  Probability of inspecting and not detecting noncompliance: probability that the facility is compliant: L 1i =0, L 2i =1 probability that noncompliance goes undetected: L 1i =1, L 2i =1, L 3i =0  Probability that a facility is not inspected in a given year: L 2i =0

“Simple” Likelihood Function LL = log { F(x 1i  1 ) G(x 2i  2 ) H(x 3i  3 ) } + log { G(x 2i  2 ) [ F(-x 1i  1 ) + F(x 1i  1 ) H(-x 3i  3 ) ] } + log { G(-x 2i  2 ) } Where A = facilities inspected and found noncompliant B = facilities inspected and found compliant C = facilities not chosen for inspection

Estimating the Likelihood Function  Select covariates associated with non-compliance, selection, and detection.  Non-compliance: facility-related characteristics.  Selection: factors currently used in selecting facilities.  Detection: investigator-related factors.  Use a maximum likelihood estimation to find coefficient estimates that maximize the function.  Initialize parameter estimates with results from inspection and detection analyses.

Change in Probability of Non-compliance Rx-0.10* Prompt rel Ext/Del rel Gel cap Soft gel cap-7.E Ointment Liquid0.21* Powder4.E Gas Parenteral Lg. vol Parent Aerosol Bulk-0.18** Sterile Suppositories Number of obs

Vitamins Necessary Antibiotics0.23**0.22* Biologics No. Thera. Classes/Plant2.E-03-3.E-03 No. Products/Plant-2.E-03-1.E-03 No. Dose forms/Plant-4.E No. D.F. Routes/Plant-3.E No. Sponsor Appl./Plant0.02* ** Ownership change (t=0)0.16 Ownership change (t=1)-0.13 Ownership change (t=2)-0.09 Ownership change (t=3)0.34+ Firms per plant-0.07 InspectionTechnologyYes Plant SelectNoYes No DetectionTrainingYes No. of obs

“Facility-fixed” Effects  Construct binary variables for the facilities with the Greatest number of inspections.  Re-estimate non-compliance model using binary variables for these 50 facilities.  Identify those facility more or less likely than average to be non-compliant.

Predicted Level of Facility Non-compliance For 50 Most Inspected Facilities Statistically more noncompliant than the mean facility. Statistically not different from the mean facility. Statistically more compliant than the mean facility.

Immediate Implications  Inspection and Non-compliance  New suggestions for inspection choices. Use non-compliance analysis to assess risk of any given facility, firm, or technology. –Increase focus on particular facilities and attributes. –Ownership changes.  Mixed strategy inspection plan.  Detection  Use detection analysis to assess quality of investigators and their training.  Focus investigator activities to build and maintain short-run experience.

Broader Implications  Our statistical methods provide a test-bed for asking and answering management and oversight questions.  Further development is needed.  DCE has potentially broad applicability to CDER and other centers at the FDA including CBER, food, etc..  What facilities are most at risk of non-compliance? Base-line non-compliance Technology Ownership changes, etc.  What manufacturers are more/less prone to non-compliance.  DCE has implications for the type, format, and processing of data to be collected and analyzed.

Development Opportunities  Additional variables can and are being constructed to examine additional issues.  Recall, shortages, supplement filings.  More fine-grain information on technology, manufacturing knowledge, organizational capabilities.  Evaluate manufacturer data collected in our study.  More heavily weight more recent investigations.  Expand to full set of investigators and facilities (requires additional computational resources).  Evaluate endogeneity concerns.

Development Constraints  Software/computer limitation.  Data preparation/man-power.  Funding resources are nearly exhausted.  Teaching.

Current Plan  Document current progress in a white paper.  Further develop data in hand (EES, Shortages, etc.).  We received cooperation from the gold sheets.  Work with you to develop plan for transferring results to FDA.  Look for additional funding sources.