1. A Review of Methods used to Quantify Effect Sizes in Clinical Trials
Joanne Rothwell
University of Sheffield
PSI Conference, London, May 2017

2. PSI Conference, London, May 2017
Outline
Aims of Research
Methods
Results
Limitations
Further Work
PSI Conference, London, May 2017

3. PSI Conference, London, May 2017
Aims of Research
To investigate:
which methods are commonly used and reported for eliciting the target effect size
how “accurate” the target effect size is compared to the observed effect size
whether particular clinical areas are consistently over- or under-achieving the target effect size
examples of well-justified sample size calculations and target effect size elicitation. To be completed.
PSI Conference, London, May 2017

4. PSI Conference, London, May 2017
Aims of Research
Standard sample size formula for superiority trials
Fixed
Most sensitive variable
As Steven mentioned
Estimated
PSI Conference, London, May 2017

5. PSI Conference, London, May 2017
Aims of Research
Standard sample size formula for superiority trials
Fixed
If d is halved, the sample size quadruples
Estimated
PSI Conference, London, May 2017

6. PSI Conference, London, May 2017
Methods
Why is this research useful?
It would be interesting to know which methods are most commonly used to elicit the target difference and which are most reported. This would be useful for the DELTA2 guidance.
It would be useful to know how many studies are not observing a difference close to the target difference which the study was originally powered on.
PSI Conference, London, May 2017

7. PSI Conference, London, May 2017
Methods
How was this research completed?
Review the Health Technology Assessment (HTA) reports for randomised controlled trials
Extract data from the reports over the period
HTA used because they are the main public funder in the UK
PSI Conference, London, May 2017

8. Inclusion/Exclusion Criteria for Reports
Extracted variables included:
Target and observed effect size, target and achieved sample size, elicitation method (if reported), clinical area, outcome measure, target power.
Exclusion based on initial title and abstract reading:
systematic reviews (most common X%), literature reviews, feasibility and pilot studies, observational studies, cost-effectiveness and decision analysis.
Once extraction commenced, other exclusions were required: factorial trials (no clear target difference or elicitation appeared), non-inferiority studies (no target difference, based on NI margin), equivalence studies (again, based on equivalence margin), vaccination trials (Hard to extract all relevant information from these).
PSI Conference, London, May 2017

9. Report Selection PSI Conference, London, May 2017 994 reports
(107 RCT reports since 5 reports document two RCTs in one report.)
684 reports
310 reports excluded:
Published before 2006
75 reports excluded:
9 Factorial trials
5 Non-inferiority trials
10 Equivalence trials
20 Cluster Trials
19 reports not RCTs
1 Cross-over trial
11 Too difficult to extract or not enough information provided
177 reports
507 reports excluded:
Not RCTs
PSI Conference, London, May 2017

10. Total Number of Reports
General Results
Volume
Year
Total Number of Reports
Included RCTs 20
2016 95 19
2015
102 18
2014 71 12 17
2013 61 11 16
2012 49 8 15
2011 45 6 14
2010 60 13
2009 62 10
2008 36 2
2007 53 3
2006 50
Total
684
107
PSI Conference, London, May 2017

11. Elicitation Categories
7 categories based on DELTA publication (Cook et al. 2014) [1]
Anchor method
Distribution method
Health Economic method
Opinion-seeking method
Pilot-study method
Review of evidence base method
Standardised effect size method
Jonathan will define these more formally later. I'm sure many of you have used one or a combination of these methods to elicit a target effect size.
PSI Conference, London, May 2017

12. Study Characteristics
Most common
Clinical Area
Mental Health (16.7%)
Setting
Hospital (51%)
Elicitation Methods
Review of Evidence Base (46%)
PSI Conference, London, May 2017

13. PSI Conference, London, May 2017
General Results
PSI Conference, London, May 2017

14. PSI Conference, London, May 2017
General Results
PSI Conference, London, May 2017

15. PSI Conference, London, May 2017
General Results
PSI Conference, London, May 2017

16. PSI Conference, London, May 2017
General Results
Shows that most common method across all the clinical areas is using previous research
These elicitation categories are ones which contain slightly more detail than the condensed DELTA categories. This shows that across all clinical areas previous research is most commonly used to eliciti the target difference.
PSI Conference, London, May 2017

17. PSI Conference, London, May 2017
General Results
Using DELTA categories, most common method is consistently Review of Evidence
Using the DELTA categories, it is clear to see that the Review of Evidence method (using previous research) comes out on top as the most common method, particularly in Mental health.
PSI Conference, London, May 2017

18. Estimated/Target Standardised Effect Size
Smallest Target Effect sizes
Largest Target Effect sizes
PSI Conference, London, May 2017

19. Estimated/Target Standardised Effect Size
Smallest Target Effect sizes
Seems very large
Largest Target Effect sizes
PSI Conference, London, May 2017

20. Estimated/Target Standardised Effect Size
Smallest Target Effect sizes
Large variation
Whilst this could be due to different interventions, it is still interesting to see.
Largest Target Effect sizes
PSI Conference, London, May 2017

21. Estimated vs Observed Standardised Effect Size
General relationship - Estimated SES are higher than Observed
It would be expected for the protocol to have a higher expected/target effect size than observed, as half of all publicly fundedtrials are not statistically significant.
PSI Conference, London, May 2017

22. Estimated vs Observed Standardised Effect Size
General relationship - Estimated SES are higher than Observed.
As predicted
PSI Conference, London, May 2017

23. Estimated vs Observed Standardised Effect Size
Very high observed effect size - Renal/Urology study. p<0.001
PSI Conference, London, May 2017

24. PSI Conference, London, May 2017
Limitations
Only one reviewer, 177 reports to read through and 102 to extract information from.
No QA available
Some reports difficult to extract from due to unclear sample size justifications
In order to compare effect sizes for clinical categories, categories may need to be combined. This needs consultation with a clinical expert.
Only one reviewer.
PSI Conference, London, May 2017

25. PSI Conference, London, May 2017
Summary
Mental Health is most commonly published clinical area for RCTs.
Most common method of elicitation is using previous research or a combination of various methods including a review of the evidence. (49%)
Target effect sizes are typically larger than observed effect sizes (this is as expected)
Only one reviewer.
PSI Conference, London, May 2017

26. PSI Conference, London, May 2017
References
JA Cook, J Hislop, TE Adewuyi, K Harrild, DG Altman, CR Ramsay, C Fraser, B Buckley, P Fayers, I Harvey, AH Briggs, JDNorrie, D Fergusson, I Ford, LD Vale. Assessing methods to specify the target difference for a randomised controlled trial: DELTA (Difference ELicitation in TriAls). Health Technology Assessment 18(28), 2014
Only one reviewer.
PSI Conference, London, May 2017

27. Thank you for your attention.? ? ?
Any Questions? ?
Only one reviewer.
PSI Conference, London, May 2017