1. Agreeing outcomes that matter to patients – what are the challenges?
Heather Bagley & Bridget Young (University of Liverpool)
COMET - People and Patient Participation, Involvement & Engagement (PoPPIE) Working Group
(Acknowledge co-author Bridget Young) Heather Bagley – PPI co-ordinator with the Core Outcome Measures in Effectiveness Trials (COMET) Initiative. Also public contributor in research for about 9 years, commenting as a parent about research study design etc. As a parent I attended a meeting about outcomes for research back in 2010 where I learnt about the problems with outcomes in research.

2. What are the problems with outcomes in research?
Health care research is untidy
Selective reporting of outcomes
Outcomes used in research may not be relevant to patients
I learnt that:
At the moment, different studies looking at the same treatments for the same condition often measure different outcomes. When such studies are completed, we cannot compare or combine their results in systematic reviews and meta-analyses because they have used different outcomes. And the scale of the untidiness is striking - Over 25,000 of the outcomes in cancer trials have only been used once or twice (Hirsch et al JAMA Intern Med 2013, 173:
Alongside the problem of inconsistency in outcomes is the problem of the selective reporting of outcomes.
The third problem is that currently outcomes may not be relevant to patients. In deciding what outcomes are most important, people who know what it’s like to live with the effects of conditions and treatments should have a say. The people who usually know this best are patients and their families or carers. We need to be confident that the outcomes that we use are important outcomes for patients and relevant to them.

3. What’s the solution?
If all studies in a particular health condition used the same outcomes, they could all be compared and combined.
This would reduce waste by making best use of all the research and reduce selective reporting bias.
Patient input into agreeing outcomes of importance is vital.
When a set of main outcomes has been agreed for a health condition, it’s called a ‘core outcome set’.

4. Core outcome set
Core Outcome Set
Additional outcome
An agreed standardised set of outcomes that should be measured and reported, as a minimum, in all clinical trials in specific areas of health or health care
A core outcome set is an agreed standardised set of a minimum outcomes that should be measured and reported, as a minimum, in all clinical trials in specific areas of health or health care
When a core outcome set has been agreed the hope is that researchers will use it in all studies for a particular condition.
We fully expect researchers to measure other outcomes as well but the core set is the minimum for all.

5. How are core outcome sets agreed upon?
Consensus processes eg. Delphi
Interviews
Core outcomes have to be relevant not just to health professionals, but also to patients and carers.
People working on core outcome sets need to make sure that this expertise is used to agree on the core outcomes – So this expertise is from patients, carers, professionals (and some groups also involve other stakeholders e.g. Regulators, industry reps).
To seek agreement on core outcomes researchers often use ‘consensus methods’ such as the delphi process. Indeed over 90% of ongoing COS are using the Delphi process
For round one need to mention that the long list of outcomes typically derived from SRs of outcomes reported in previous clinical trials, which until recently, will have had very little patient input
Interviews with patients may take place, particularly in the pre-Delphi stage. FOR EXAMPLE This can help to:
identify what outcomes are important to stakeholders;
determine the scope of outcomes;
identify appropriate CONCEPTS /language for use in the Delphi survey
Participants sent long list of outcomes to score in terms of the importance of the outcomes.
Responses analysed and the persons scores from the previous round and the scores of other stakeholder groups are presented back to the patient for rescoring.
Then there is usually a face to face consensus meeting where the outcomes are discussed
The COS is then identified for the condition.

6. Core Outcome Sets developed and ongoing
319 Published COS (24% included patient participants)
149 Ongoing COS (86% included patient participants)
Sadly, patients and carers have not always been involved in this type of work, but, as you can see, now they are being included, recognising the importance of their first-hand experience of living with a condition.
There are examples of how involving patients identified an outcome that was important to them as a group but which might have been overlooked if practitioners had come to the decision on their own.
The rheumatoid arthritis core outcome set study was run with health professionals (no patients). 7 core outcomes were identified.
Patients were then involved in reviewing this same core outcome set in They identified that fatigue was really important to them.
Gargon, E., B. Gurung, et al. (2014). "Choosing important health outcomes for comparative effectiveness research: a systematic review." PLoS ONE 9(6): e99111.
Gorst, S., E. Gargon, et al. (2015). “Choosing important health outcomes for comparative effectiveness research: an updated review and user survey.” Submitted.

7. COMET Initiative and the COMET PoPPIE Working Group and how are they helping?
COMET stands for the ‘Core Outcome Measures in Effectiveness Trials’ Initiative.
COMET is an international Initiative and provides a database of core outcome sets that have already been developed or are still under development. They also provide advise and resources for COS developers and raise awareness about COS
COMET PoPPIE Working Group leads the public participation, involvement and engagement work of the COMET Initiative,
Examples of the work of PoPPIE members are that we host workshops to help researchers explore and troubleshoot the issue of engaging patients in deciding what outcomes should be measured in clinical trials. We also develop resources to help researchers who are engaging with patients in their COS work.

8. Challenges of engaging patients in deciding what outcomes should be measured in clinical trials?
In my earlier description of methods used to develop COS you can see that by undertaking quite a complex process to agree a COS, that there may be significant challenges in ensuring the input of patients.
So this is not for example, the same as asking one or two patients or a small patient panel their opinions about choice of the outcomes for a particular study, it is about undertaking quite a challenging process of consensus building, which may be quite technical, unfamiliar and even dry for patients to get involved in and may result in them having to ‘let go of’ outcomes that a patient personally may believe to be important but others (including other patients and carers) see as less important. (all outcomes are important). Another challenge is that Spend 20 minutes filling in the Delphi thinking everything’s very important (Gill Gyte).
These are the kinds of challenges that are the focus of our work in the COMET PoPPIE Working Group.
Key role of COMET - help researchers think about how to engage with patients in deciding what outcomes should be measured in clinical trials.

9. COMET meetings workshop - Involving patients in core outcome set development: identifying the challenges and potential solutions
4 Workshops:
Manchester
Rome
Calgary
Amsterdam
Over 120 delegates participated
Facilitators included patients with experience of involvement in outcomes related work
Talk today about issues raised in 4 interactive workshops in COMET meetings in Manchester, Italy, Canada and Amsterdam with about 120 international delegates, including researchers, clinicians and patients.
5 main questions explored in these workshops – notes were made - now summarise these discussions but first a note about terminology.

10. Patient input Participation (participants)
Involvement (research partners)
Engagement
Patient participation is where patients and the public take part in a COS development study, for example alongside clinicians and other stakeholders in a consensus survey, and give data on their opinions regarding what outcomes are important. We refer to the people in this role as ‘patient participants’ or ‘participants’.
Patients and the members of the public may also be involved in designing a COS development study, overseeing it once it is running and helping with its dissemination, a process referred to in the UK as patient and public involvement or PPI. We refer to the people who take these roles as ‘patient research partners’ or ‘partners’, although others may use terms such as patient ‘advocates’, ‘representatives’, ‘contributors’, ‘surrogates’ or ‘community stakeholders’ etc.
In countries such as Canada the term engagement is used rather than involvement. In the UK engagement is more often used to describe informing the public about research and science.

11. Question 1 - What principles and practicalities are important to consider in accessing and sampling patients?
Concern about inclusivity and fairness
Proportion of patients relative to other stakeholders
Ethical approval
Just going to provide an overview of some key topics discussed in the workshops – reference at the end if want to read our commentary article
Workshop participants expressed strong agreement on the importance of the principles of inclusivity and fairness. – much discussion about making COS development activities accessible to patients and sensitive to the needs of particular groups
So there was discussion about routes to access – community outreach / maximum variation sampling.
? How many patient participants? / what proportion of patients relative to other stakeholders little guidance on numbers / compositions of stakeholder groups - indications that these factors can influence what outcomes are rated as important
As COS development aims to produce generalizable knowledge need to seek ethical approval

12. Question 2: What principles and practicalities are important in selecting a method for patient input in COS development?
Differing views on the Delphi process / some favoured qualitative methods
Different methods for different patient groups
Some delegates favoured qualitative methods as they capture the complexity of the patient perspective, use language patients use and as it helps researchers to understand why patients feel some outcomes are more important than others.
Delphis described as intimidating by some Long list of outcomes in a Delphi to score can be off putting
Others recognised Delphi surveys widen patient participation, promoting transparency and offering anonymity & helping to minimise influence of power differential between different stakeholders
Delegates recognised different methods needed for different patient groups /dependent upon sensitivity health area.
Importance of involving patients as partners in designing the method was seen as crucial.

13. Question 3: What are the best ways of explaining the concept of a ‘core outcome set’ to patients and what types of questions can be used to elicit their perspectives?
Concept of COS can be difficult to grasp
Which outcomes are important? Patients just want a cure
Haven’t researchers sorted this out years ago?
Concept of COS can be difficult to grasp- certainly this was my experience when I first heard about them as a parent.
Which outcomes are important? Patients just want a cure (need to focus on more achievable outcome)
Researchers can also be faced with patients wondering ‘Haven’t you researchers sorted this out years ago’?
Methods dictate information patients need:
Qualitative interviews – patients set agenda –participate meaningfully without needing detailed understanding of what outcome is / why COS needed.
E-delphis / consensus meetings more detailed understanding needed. Cos developers need to ensure provide understandable information but don’t unduly influence patient responses.
Delegates emphasised patient research partners help teams ensure clear explanations provided to patients.
Focus on the goal – how COS will benefit patients and use simple analogies eg comparing apples and pears – See the COMET plain language summaries about COS and about Delphi surveys
By emphasising the goal – only what is ‘core’ (to avoid problems ‘letting go’ of other outcomes)
Importance of excellent facilitation skills in consensus meetings / ggod support for patients
Possible questions to use to ask about outcomes include:‘ what is the most limiting aspect of your condition’, ‘what would you hope for from a new treatment’, ‘what sort of things would you think about in making a decision about what treatment to take’ or ‘what sorts of changes in your symptoms/condition/life would tell you that a treatment was actually helping you’.

14. Question 4: How can we maintain the input of patients over time?
Vital to keep people engaged in processes like Delphi surveys over time
Importance of building rapport / showing appreciation
Delphi surveys require input on two or more occasions – so vital to keep engaged. Discussion about use of patient involvement to ensure the research process is well designed for patient participation
Important to build rapport with patients, show appreciation for contribution, reimburse costs and ensure feedback provided (plain language)
Incentives – possibly doing COS quickly, minimal times between rounds / offer payments / vouchers / build sense curiosity and excitement about the COS & promote feeling ownership

15. Question 5: Should patients be brought together with other stakeholders to achieve consensus? If so, how should this be done?
Patients
Other stakeholders
Differing opinions / concern about powerful voices
Reconciling divergences
Differing opinions – some said meet together to discuss / others keep patients separate. Concerns about powerful voices dominating patients (but this can also happen between patients / patients and carers)
Importance of strong support and facilitation (expert in facilitation rather than expert in particular condition).
Issues of reconciling divergences between patients / other stakeholders – all stakeholders need to be open to challenge, prepared to explain their reasons and reconsider opinions, but people may feel uncomfortable.

16. Another topic which featured in a number of conversations at the COMET VI meetings in Amsterdam was about the use of COS once developed. The tweet by a patient in the COMET meeting emphasises this concern.

17. Resources around public input into COS development
Hopefully that provides a brief overview about some of the challenges of patient input into COS studies. For further reading about these challenges see our commentary paper.
The PoPPIE Working Group are also developing resources to support patient input into COS development (for example plain language summaries explaining COS) and these can be found on the public tab of the COMET website

18. In Summary Explaining COS to patients
Technical / potentially challenging nature of COS development
So there are numerous challenges to gaining patient input into COS development. Some of theses are challenges common to other areas such as finding patients to participate and retaining patients in the COS development process.
More specific challenges in this COS process include: a) explaining what COS are all about b) the rather dry/technical nature of COS development
The potential benefits of PPI into the design of COS studies were recognised in the workshops. We also need to think about fresh / creative ways to tackle some of the challenges outlined.