1. CDRH Review Practice for Over-the-Counter in Vitro Diagnostic Tests
Center for Devices and Radiological Health (CDRH)
Office of in Vitro Diagnostic Device Evaluation and Safety
Arleen Pinkos
Scientific Reviewer
I’ll be presenting a brief overview of CDRH review practices for OTC IVDs.
As a frame of reference……these products are…..
CDRH OTC Review Practice, BPAC,

2. Overview of Presentation
A. Background
B. Three Elements of OTC Review
Accuracy and Reliability
Labeling
Risk vs. Benefit
C. Conclusion

3. OTC History 817 OTC products cleared for home use Diabetic Products
Hormones
Drugs of Abuse
Urine Dipsticks
Fecal Occult Blood
Lipids, PT
There are 817….
Diabetic products have the highest volumeThey are listed in decreasing order of the number that we’ve cleared
Of note is the absence of any ID devices

4. OTC Infectious Disease Devices
None cleared or approved
Devices reviewed for OTC- Self diagnosis:
Group A Streptococcus, and Influenza A and B
In professional hands:
Sensitivity averaged 55-80%, dependent on specimen matrix
Specificity averaged 80-90%
False negatives – risks are life threatening
False positives – risk of masking serious underlying disease
In addition to some other review issues, CDRH felt- risks associated with both false negatives and false positives were too great.

5. Indications for Use: What Drives the Review
Need to know who, what, when, and why
Special conditions for use that impact performance
Contraindications
Drives:
Protocol requirements
Questions raised
Data required
Before a review can begin, it is critical to know the who, what, where and whens of a device.
The Indications drive the review process……

6. Three Review Elements
1. Is the device accurate and reliable in the hands of lay-users?
2. Is the device adequately labeled?
3. Do benefits outweigh the risks?
As I stated earlier, there are three main review elements, and we’ll go through each of them…. by CDRH….

7. First Review Element
Is the device accurate and reliable in the hands of the user?
Can the test provide the right answer?
Can the lay user get the right answer?
Human Factors (HF)
Risk Mitigation & Safeguards
Stress Studies
We’ll cover a number of subjects as we talk about this element…….

8. Can the test provide the right answer?
Analytical and Clinical Performance
From initial professional-use submission
Analytical and Clinical Sensitivity and Specificity
Accurate (true) result
Quantitative -Bias, total error
Qualitative- Clinical Sensitivity and Specificity, Predictive Values
Typically – first thing- the performance characteristics of a device under optimal conditions.
Because almost all of our OTC applications were previously cleared for professional use, this information is already available.
So- not going to spend. But will run thru the primary parameters……

9. Can the test provide the right answer?
Other Important Parameters
Cross-reactivity & Interference
Reproducibility
Reference intervals
Environmental factors, stability, reportable range, calibration, linearity, lot to lot variability, etc.
Here are some of the other parameters we look at:
Cross-Rx and interference are especially important with OTC, false pos and neg
Environmental conditions are also important with OTC devices, both in terms of storage req. and testing req.
Not unusual for consumers to leave a device in a hot car, or leave a kit out when it should be refrigerated.

10. Can the lay user get the right answer?
Consumer Study Goal
Demonstrate comparable performance when used as intended
Representative Lay Users
Enrollment criteria, demographics
“Home” environment
Evaluate entire test system
Consumers perform all steps, unassisted
Labeling – Representative labeling, or according to marketing plan.
Once we know what a test is capable of doing under optimal conditions,
We want to see if lay users can achieve a comparable level of performance at home.
Usually- Consumer Study
To evaluate a test being used as intended , look at three ……
A home is not always a home-
Mall
isolated area - dr office or clinic
ENTIRE test system, from beginning to end
Unassisted
only PI.
But the key is that the device is used according to how it will be used when its marketed. So….
educational information,
training (e.g., video)
help line
important - consumers don’t get anything more

11. Consumer Study Sample Considerations
Clinical or spiked –
Challenging and span expected range
Prospectively collected if:
establish clinical sensitivity and specificity
demonstrate ability to collect the sample, or
sample is not stable
Biohazard
Properly characterized- WHO, CDC, reference method, or to another device
Samples used in the study- perhaps one of the areas we struggle with the most
Whether the samples are clinical samples or if they prepared in some way-
Concentration- should both challenge the cutoff or the medical decision making point(s) of the assay.
And span the expected range
Sometimes difficult- clinical samples
If prepared-
- Both the matrix and what is being spiked into it- act the same as clinical
Samples may need to be prospectively collected if…..
Also to be considered– don’t want to endanger participants by having them handle bio-hazardous materials
Last important consideration for samples is whether- properly characterized?

12. Consumer Study Protocol Considerations
Comprehensive
Enough participants and samples
Correct Statistical Tools
Masking and randomizing- avoid bias
Written prior to study

13. Human Factors (HF)
A discipline concerned with how people interact with the device
Why are HF so important?
Users- varying abilities and behaviors
Impact on safety and effectiveness
Another important area we consider is HF.
The little things users do or don’t do.
Why
Because people have varying abilities, both physical an mentally
And we each have behavior patterns:
Everyone has- like not reading the directions to their VCR
Particular to an individual- being color blind
And those tendencies that we all have can impact S&E

14. Human Factors “Errors” occur when users:
Fail to perform necessary actions
Add actions
Choose the wrong action
Perform actions poorly
These “errors” or events occur when……….
Performing poorly – not seeing well, or having poor technique

15. Human Factors Who is responsible for “user” errors?
“The instructions clearly state that results are to be read from right to left.”
“How was I to know they’d read it left to right?”
Who’s responsible?
Sometimes we find that sponsors don’t accept the responsibility as is illustrated in this example.
But device manufacturers need to anticipate all “reasonable” errors.
………………………NEXT

16. All user errors cannot be prevented
but some can be avoided
ACME
Toenail Cutter
Not all errors can be prevented…
We’ve seen-
-diabetics have cut their test strips in half to save money. No one- anticipated, but….
also
Kit has 2 btls of reagent with two different color caps.

17. Risk Mitigation & Safeguards What do we expect regarding errors?
Effective procedures to address
HF and other potential problems:
Hazard Analysis: Identify potential errors and estimate their risk
Mitigate Risks: Eliminate or reduce risks to acceptable level
Verify effectiveness of safeguards
So what does CDRH expect from sponsors when it comes to potential errors of a device?
Not only HFs, but others –
-battery loosing its chanrgep
-mechanical or electrical failure
We look for a systematic approach to these issues:
1. By conducting a thorough Haz Analysis sponsors can identify, and estimate the risk associated with that potential problem
2. Then to mitigate that risk by some mechanism to either eliminate or reduce risk- acceptable level
3. And after they have mitigated the risk, verify that the mechanism is effective.

18. Risk Mitigation & Safeguards Examples
Built-in features to detect:
Short sample
Wrong sample
Reagent integrity
Internal QC- often limited scope
Lock-out mechanisms?
Labeling- techniques, but when is it adequate?
Goal- alert before results are compromised
Some examples of what a sponsor might do to mitigate a risk are to build features into their device that detect:
Short sample
Wrong sample
Reagent integrity
Hardware failures
Environmental and storage conditions
Some have temperature strips that stay with the device, and show if it was shipped or stored improperly
May have an internal QC- typically monitor only one aspect of the device.
Example- immunochromatographic single-use tests –only volume
Lock out mechanisms to turn off the device or make it unusable if some parameter is not working properly or if QC wasn’t run, or was out of range
Labeling is often used as a way to mitigate the risks of a device.
Various techniques, bolding, highlighting, or blocking that are known to bring information to a users attention
Often difficult to know whether labeling will be an adequate mitigation until it goes on the market.
Before- short sample detection system –not work until almost half.

19. Risk Mitigation & Safeguards Quality Control
Is External QC
Available and practical?
Adequate?
Needed?
Will consumers do it ?!!
External QC, although beneficial and useful in professional settings, not always a good fit.
Not always adequate - provide assurance that the result is correct.
And even if it did monitor the entire analytical process, NOT correct sample collected or handled properly
False sense of security.
How critical is it to have an external check?
Great technological advances with devices, and…
Many argue that with the advent of single use, disposable devices, external QC is not the answer
Can’t verify performance of the device the user is actually using to test themselves.
And even if you built the best external QC mouse-trap in town, if doesn’t mean that consumers would do it.

20. Stress Studies Demonstrating Effective Mitigation
Need to verify effectiveness of safeguards
What are stress studies?
Challenges to the system in order to characterize performance under various conditions of use
After a sponsor describes their mechanism for addressing a potential problem, how do we know its effective?
CDRH asks sponsors to provide evidence that it is effective.
Often done with stress studies…..
What are…………………
For instance, if the directions say to wait 10 minutes, what happens when the user reads it early because they’re late for an appointment?

21. Stress Studies How are they performed?
Artificially simulate potential errors
Concentrate on areas prone to errors or failures
Technique
Follow directions
Deviations in environmental conditions
Observe results
Simulate deviations from expected perfect condition:
-battery loosing its charge
-environmental conditions. (got delayed at the mall and kit was in the hot trunk for 3 hours)
Technique- shaking something instead of inverting
subtle variations- not think about or realize
Examples……….

22. Stress Study Examples Instructions Type of Study
Procedure- store kit between °F
Store kit as directed. Store the kit with intermittent freezing,
50 and 90 °F then run samples.
Procedure- add 3 drops of sample.
Perform the test by adding 2, 3, 4, and 5 drops of sample.
Compare results to expected
If small deviations and big affect- may not suitable
Compare the results observed when deviations from instructions are simulated to
Results from when it was performed properly.
How significant is the difference?
Really looking at how much deviation from the instructions had to occur before you had…..
Have a clinically significant difference in the result?

23. Second Review Element Is the device adequately labeled?
Are the step-by-step instructions simple, concise, and easy to follow?
7th grade reading level?
Do consumers know who should use the device and when?
Clear instructions for interpretation and follow-up?
Limitations?
Go to a doctor?
What if it didn’t work?
Repeat test?
Are steps or reading results counter-intuitive
Immunochromatographic tests- positive is the absence of a test line, line= negative
Concise- unnecessarily detailed and long- not read
Look for there to be as few words as possible to get the message across. ……
Limitations-
Tests are not perfect”
Effects from meds or combinations
Cannot identify all possible interferences
- if inconsistent results- don’t assume correct
Great example of people not knowing the limitations of a device- hand held glucose meters.
Did you know it was dangerous to use a meter when you’re really sick?
If you are a critical care nurse or a diabetic, you might want to look at the limitations in the package insert………..

24. Labeling How do you know its adequate?
Verified-Consumer Study
Value is Limited
“Trial effect”
Will they read it?
Questionnaire
Completed by consumers after testing
Must be challenging:
Who, what, where, when, how & why?
How do you know the directions are good enough?
Look to consumer study- but recognizing it has limits:
Studies typically evaluate a relatively small number of subjects.
Often we observe a “trial effect”…..when users are extra careful because they know they are either being watched or their results will be scrutinized
AND- WON’T READ
Ask challenging , not open ended!
NOT:
- have trouble performing?
- understand your test result?
YES:
what would you do if..??? Got a particular test result, or
how affect interpretation IF LAST TIME?

25. Third Review Element Risk vs. Benefit Consideration
Lay users are unique:
Unregulated environment
May lack medical and technical training:
evaluate other medical information
collect and transport samples
read and follow directions
May or may not carry out follow-up actions
Highly variable
This brings us to the third and final element of our review.
As we consider R and B, we must also keep in mind that lay users are unique individuals……
Evaluate other… like results from other tests or from a physical exam.
Read and follow- language or comprehension barriers
One thing for certain- there is a broad spectrum of lay users in terms of their knowledge and skill level.
From those that are reading and functioning at a low level to…
-internet junkies reading everything or those….
-have jobs that require attention to details or needing fine motor skills.
A good test- one that was appropriate for use by all lay users.

26. Risk vs. Benefit
Risk
1. What is the impact of a false-positive or false-negative result on the user and public health e.g., no follow-up, adverse medical conditions, unnecessary treatment?
2. What are the medical risks to the user and public health if there is a delay in obtaining a professional examination and treatment, or if the device gives an equivocal result?
two questions perhaps capture
Consider MEDICAL risks and benefits to both individuals and to the public.

27. Risk vs. Benefit Benefit
1. What is the clinical benefit to the patient and public health in terms of screening, diagnosing, or monitoring a particular disease or condition, or risk factor?
2. What are the medical benefits to the patient and public health in having the test available for home-use as opposed to having the test performed only by health care professionals?

28. CONCLUSION The Final Decision
Is the device accurate and reliable in the hands of the lay user?
Does the Labeling adequately convey all information necessary to use the device safely and effectively?
Do benefits outweigh risks?
If yes, generally found suitable for home-use
In the end, after reviewing all of the evidence, CDRH returns to the same three questions to make their decision:
Whether its accurate and reliable enough
labeling is good enough, and
there are more benefits than risks????
If the answers to these three questions are yes, the device will generally be found suitable for home-use.