Representation of published core outcome sets for research in regulatory guidance: protocol

Susanna Dodd; Rebecca Fish; Sarah Gorst; Deborah Hall; Pamela Jacobsen; Jamie Kirkham; Barry Main; Karen Matvienko-Sikar; Ian J. Saldanha; Dominic Trépel; Paula R. Williamson

doi:10.12688/hrbopenres.13139.3

Home Browse Representation of published core outcome sets for research in regulatory...

ALL Metrics

Views

Downloads

Get PDF

Get XML

Export

▬

✚

Study Protocol

Revised

Representation of published core outcome sets for research in regulatory guidance: protocol

[version 3; peer review: 2 approved]

Susanna Dodd ¹, Rebecca Fish^2,3, Sarah Gorst¹, [...] Deborah Hall^4,5, Pamela Jacobsen⁶, Jamie Kirkham⁷, Barry Main⁸, Karen Matvienko-Sikar⁹, Ian J. Saldanha¹⁰, Dominic Trépel^11-13, Paula R. Williamson¹

Susanna Dodd ¹, Rebecca Fish^2,3, [...] Sarah Gorst¹, Deborah Hall^4,5, Pamela Jacobsen⁶, Jamie Kirkham⁷, Barry Main⁸, Karen Matvienko-Sikar⁹, Ian J. Saldanha¹⁰, Dominic Trépel^11-13, Paula R. Williamson¹

PUBLISHED 02 Aug 2021

Author details Author details

¹ MRC/NIHR Trials Methodology Research Partnership, Department of Health Data Science, University of Liverpool (a member of Liverpool Health Partners), Liverpool, UK
² Division of Cancer Sciences, University of Manchester, Manchester, UK
³ Colorectal and Peritoneal Oncology Centre, The Christie NHS Foundation Trust, Manchester, UK
⁴ Hearing Sciences, Division of Clinical Neuroscience, School of Medicine, University of Nottingham, Nottingham, UK
⁵ School of Social Sciences, Heriot-Watt University Malaysia, Putrajaya, Malaysia
⁶ Department of Psychology, University of Bath, Bath, UK
⁷ Centre for Biostatistics, Manchester Academic Health Science Centre, University of Manchester, Manchester, UK
⁸ University of Bristol Medical School, Centre for Surgical Research, Bristol, UK
⁹ School of Public Health, University College Cork, Cork, Ireland
¹⁰ Center for Evidence Synthesis in Health; Department of Health Services, Policy, and Practice, Brown University School of Public Health, Providence, Rhode Island, USA
¹¹ Global Brain Health Institute, Trinity College Dublin, Dublin, Ireland
¹² School of Medicine, Trinity College Dublin, Dublin, Ireland
¹³ University California, San Francisco, California, USA

Susanna Dodd
Roles: Conceptualization, Investigation, Methodology, Project Administration, Writing – Original Draft Preparation

Rebecca Fish
Roles: Methodology, Writing – Review & Editing

Sarah Gorst
Roles: Methodology, Writing – Review & Editing

Deborah Hall
Roles: Methodology, Writing – Review & Editing

Pamela Jacobsen
Roles: Methodology, Writing – Review & Editing

Jamie Kirkham
Roles: Methodology, Writing – Review & Editing

Barry Main
Roles: Methodology, Writing – Review & Editing

Karen Matvienko-Sikar
Roles: Methodology, Writing – Review & Editing

Ian J. Saldanha
Roles: Methodology, Writing – Review & Editing

Dominic Trépel
Roles: Methodology, Writing – Review & Editing

Paula R. Williamson
Roles: Conceptualization, Methodology, Writing – Review & Editing

OPEN PEER REVIEW

REVIEWER STATUS

Abstract

Background: The COMET Initiative promotes the development and use of ‘core outcome sets’ (COS), agreed standardised sets of outcomes that should be measured and reported in all studies in a particular clinical condition. COS are determined by consensus amongst key stakeholders, including health professionals, policymakers and patients, ensuring that the priorities and expertise of these representatives inform the choice of the most important outcomes to measure for a given condition. There is increased recognition of the need to integrate COS across the healthcare system and with existing regulatory apparatus, to ensure that outcomes being recorded are those of key relevance to important stakeholders. The aim of this study is to assess the degree of concordance between outcomes recommended in COS for research and in guidance provided by two key regulators: US Food and Drug Administration (FDA) and the European Medicines Agency (EMA).
Methods: COS for research published during 2015-2019 with patient involvement and covering drug or device interventions will be compared against relevant regulatory guidelines, matched by condition. Guidance documents matching in scope (relating to intervention and population) to a COS for research will be scrutinised to identify all suggested outcomes for comparison against the core outcomes in the corresponding COS.
Discussion: This study will identify variation between outcomes suggested in FDA and EMA regulatory guidance relative to outcomes included in published COS for research, thus demonstrating the degree of representation of COS in regulatory guidance and vice versa. We will share the study findings (in particular, highlighting any lack of concordance between COS and regulatory guidance overall or for particular disease areas) and will invite feedback from FDA and EMA; we will seek to highlight where findings support the recommendations towards using well-developed COS or will make recommendations to COS developers on outcomes of importance to these key regulators.

Keywords

core outcome sets, regulatory guidance

Corresponding author: Susanna Dodd

Competing interests: No competing interests were disclosed.

Grant information: The author(s) declared that no grants were involved in supporting this work.

Copyright: © 2021 Dodd S et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

How to cite: Dodd S, Fish R, Gorst S et al. Representation of published core outcome sets for research in regulatory guidance: protocol [version 3; peer review: 2 approved]. HRB Open Res 2021, 4:45 (https://doi.org/10.12688/hrbopenres.13139.3) First published: 07 May 2021, 4:45 (https://doi.org/10.12688/hrbopenres.13139.1) Latest published: 02 Aug 2021, 4:45 (https://doi.org/10.12688/hrbopenres.13139.3)

Revised Amendments from Version 2

We have made minor changes to the Introduction, addressing the comments from reviewer 2.

See the authors' detailed response to the review by Donna A. Messner
See the authors' detailed response to the review by Sophie Werkö and Marie Österberg

Introduction

Measuring patient health outcomes helps to inform healthcare decisions that are made by patients, healthcare professionals and funders. The Core Outcome Measures in Effectiveness Trials (COMET) Initiative^1,2 brings together people and groups interested in the development and application of agreed standardised sets of outcomes, known as ‘‘core outcome sets’’ (COS). One of the successes of COMET has been the development of a publicly available searchable database of completed and ongoing COS development projects³. COS may be developed for research or clinical practice, and are determined by consensus amongst health professionals, researchers, policymakers and patients or their representatives, thus ensuring the priorities and expertise of these key stakeholders determine the most important outcomes to measure for a given condition. COS are increasingly being recommended for use by trial funders and healthcare organisations⁴. The Core Outcome Set-STAndards for Development (COS-STAD)⁵ minimum standards was published in 2017, providing benchmarks against which to assess the quality of COS. COS-STAD covers 11 key features of COS development relating to three aspects of the COS development process: scope (health condition, population and intervention covered by the COS), stakeholder involvement (including patients, healthcare professionals and researchers) and consensus process (relating to the initial outcomes lists, scoring and consensus decisions, and unambiguous wording of outcomes).

Healthcare regulators play an important role in quality improvement, and frameworks adopted by certain organisations rely on evidence on outcomes to inform decision-making^6–8. Specifically, as an example, to support improvement in healthcare services in the UK, bodies such as the Healthcare Quality Improvement Partnership (HQIP) or UK National Institute for Health and Care Excellence (NICE), are recognising the relevance of considering COS for consistent measurements to inform their guidance^9,10. In 2018, NICE guidance on methods to determine relevant guideline outcomes was updated to indicate that COS should be used, if suitable based on quality and validity⁹. The HQIP tool describing key features of national clinical audits and registries states that the rationale for quality improvement objectives should take into account relevant evidence from the COMET database¹⁰. Similarly, the Swedish Agency for Health Technology Assessment and Assessment of Social Services (SBU)¹¹ endorse the use of well-developed COS to inform choice of outcomes in trials and systematic reviews.

A number of research funding agencies (particularly those commissioning the use of pragmatic randomised control trial to inform policy and regulation) are increasingly recommending that applicants should consider using a COS if one exists⁴. For example, the international SPIRIT reporting guidelines endorse consulting the COMET database to identify relevant COS^12,13, and in the UK, as an example, National Institute for Health Research Health Technology Assessment (NIHR HTA) programme refers applicants to the COMET database, suggesting that they include established core outcomes “unless there is good reason to do otherwise”¹⁴. The authors believe that uniformity in recommendations from NIHR and other public funders regarding use of COS would promote greater consistency in outcome collection globally. This benefit would have additional impact if the consistency in such recommendations extended to those for commercial sponsors. The NIHR is a unique health funding agency as Technology Assessment Review teams are funded to provide NICE with independent research to inform their guidance committees¹⁵. Regulators such as the US Food and Drug Administration (FDA) and European Medicines Agency (EMA) have a powerful role as arbiters of evidence for commercialisation of new products. Furthermore, whilst the relationship between funding research and regulating health varies internationally, both the FDA and EMA are influential in commissioning of research to help inform their decisions. It is important to assess the degree of concordance between patient outcomes suggested in FDA and EMA guidance and core outcomes included within COS for research, matched by condition.

The US FDA publishes official Guidance Documents and other regulatory guidance¹⁶, covering topics such as biologics, drugs, medical devices and food, as well as general guidance on study design and outcomes, such as their guidance on the conduct of randomised trials during the COVID-19 pandemic¹⁷ or on Patient-Reported Outcome Measures.¹⁸. These guidance documents describe the FDA’s current opinion on regulatory issues but are not legally binding (unlike FDA regulations, which are the details of how US Congress laws should be implemented)¹⁹. Similarly, the EMA publishes scientific guidelines prepared in consultation with regulatory authorities in the European Union Member States to inform marketing authorisation applications for human medicines²⁰, with full justification required for any deviations from these guidelines. This study will compare the outcomes suggested in these guidance documents against core outcomes included within COS, matched by condition, in order to progress this field by furthering our understanding of the similarities and the differences between COS and guidelines²¹.

Methods

Search strategy

The COMET database contains 108 COS for research published between 2015 and 2019 which involved patients in the consensus process. Selection of only those COS published in the last five years which involved patients will increase the number of COS-STAD (Core Outcome Set-STAndards for Development⁵) standards met. (Note that this study includes COS identified as part of the annual COMET systematic reviews up to and including the systematic review conducted in 2020, which only included studies published up to the end of 2019.) The scope of the COS meeting these criteria will be assessed to ensure that they cover drugs or devices, and if not, they will be excluded from the cohort. For each COS for research, we will search the FDA¹⁶ and EMA²⁰ websites to identify guidance covering the relevant disease/condition, using the key clinical terms as search terms. If necessary, we will refine these searches using the Google site-specific search facility e.g. searching for “diabetes site: fda.gov” or “diabetes site: ema.europa.eu” when searching for guidelines relating to diabetes on the FDA or EMA websites respectively. We will engage with COS developers if clinical input is required for guidance on appropriate search terms (e.g. to determine synonymous clinical terms to those used to describe the disease/condition under investigation in the COS) or if there is any ambiguity regarding whether identified guidance documents match the scope of the COS (in terms of the disease/condition or interventions). We will initially identify regulatory guidance/COS pairs where the scope is an exact match but will also consider situations where one of the pair may be more general than the other, based on an assessment of the descriptions of the population (i.e. clinical condition/disease) and intervention in the COS publications and regulatory guideline documents, using a previously-developed framework²² (see Figure 1). Pairs which focused on different interventions or different populations were not considered to be a match (i.e. only matches corresponding to types a-c. e-g, i-k in Figure 1 were eligible for inclusion). Each reviewer independently applied this matching algorithm to each pair of FDA/EMA guideline and corresponding potentially relevant COS. Discrepancies were resolved through discussion.

Figure 1. Scope matching algorithm determined according to the descriptions of the population and intervention within the FDA/EMA guideline versus the corresponding COS.

Eligibility

COS eligibility: COS for research (including those intended for both research and practice) were included if published between 2015 and 2019, involved patients in the consensus process and related to drug or device interventions.
Regulatory guidance eligibility: EMA/FDA guidance were considered eligible for comparison against relevant COS for research if their scope (in terms of clinical condition/disease and intervention) matched at least generally with that of the corresponding COS for research (i.e. matches of type a-c. e-g, i-k in Figure 1).

Data extraction

Data on the year of publication, disease name, specific condition and outcomes included in the eligible published COS for research will be exported from the COMET database and COS database into an Excel spreadsheet. We will record whether the COS developers consulted FDA and/or EMA guidelines as part of the COS development process, as detailed in the COS publication. We will also record from the COMET database whether participants from low/middle-income countries (LMIC) were involved in COS development. Once FDA and EMA guidance documents are identified which match in scope to a COS for research, the guidance documents will be scrutinised in order to identify all suggested outcomes (i.e. those outcomes which the guidelines state should/could/may/might be considered) relating to the specific COS population and intervention. Any additional caveats included in the guidelines about each of the recommended outcomes (e.g. relating to the age category or severity) will be recorded. If specific measurement tools (e.g. quality of life questionnaires) are recommended, the reviewer will search for and extract the individual items within these measurement tools, in order to assess whether these individual items correspond to any outcomes recommended by the corresponding COS/guidelines. Verbatim guidance document text regarding the suggested outcome measures will be recorded in tabular form for each COS. The matching between the scope of the COS and regulatory guidance will be classified as exact or general (e.g. COS is narrower/broader) in relation to both the population with the condition and the interventions (as per Figure 1), with input from clinical members of the research team and/or the COS developers, if necessary. Data extraction will be carried out by all researchers for the initial three COS/guidance pairs to ensure consistency of approach; subsequent data extraction will be carried out independently by two researchers. Disagreements will be resolved by discussion with SD/PW if necessary. Mapping between core outcomes and outcomes suggested in guidelines will be checked by the lead author (SD).

Analysis

The mapping of the verbatim extracted text from the EMA/FDA guidance to each of the core outcomes will be coded as specific (i.e. direct correspondence between the wording of the core outcome references in the guidance compared to the wording in COS) or general (i.e. only general alignment between the wording of text in the guidance relative to the wording in the COS), and this mapping will be summarised using a table as demonstrated for the type 2 diabetes SCORE-IT COS²³ in Appendix 1. Again, we will contact COS developers if clinical input is required regarding general or specific correspondence between core outcomes and those suggested in the guidance. We will use a tick to demonstrate specific correspondence between the wording of the core outcome references in the guidelines compared to the wording in COS, whereas a tick in brackets will be used to indicate general alignment (with further detail provided in a footnote) between the wording of suggested outcomes in the guidelines relative to the wording in the COS. For each COS, we will record the number (and percentage) of COS outcomes which were covered in the guidance (separately for FDA and EMA) in general or specific terms (separately) and either general/specific terms. The distribution of these percentages will be summarised across guidance documents as a whole, split by FDA and EMA, using descriptive statistics and graphical presentation, overall and split by disease category. We will also present results according to the breakdown of matching between scope of intervention and population between the COS and guidelines, as shown in the matrix in Appendix 2. Note that only results for highlighted cells a-c, e-g, i-k will be presented (i.e. those corresponding to at least a general match in both intervention and population between the COS and guidelines).

In addition, by way of symmetry we will present the results above which instead compare how the core outcomes relate to those suggested in EMA and FDA guidelines, in order to identify the agreement of COS with outcomes suggested in corresponding guidelines; i.e. we will present two additional sets of tables/results, the first with outcomes suggested in the EMA guidelines, and the second with outcomes suggested in the FDA guidelines, as the index list of outcomes. We will explore the impact of COS characteristics (for example, the number of core outcomes or the involvement of LMIC participants in COS development) on their concordance with corresponding EMA/FDA guidelines.

Dissemination

The findings of this study will be disseminated through publication in an open access peer-reviewed journal and presentation at both national and international conferences. Contact will be made with FDA and EMA colleagues and feedback on our findings requested.

Discussion

This study will identify any misalignment between outcomes suggested by EMA and FDA regulatory guidance relative to those included in published COS for research, thus demonstrating the degree of representation of core outcomes, which have been agreed by consensus by key stakeholders, within regulatory guidance, and vice versa. A lack of concordance between COS and regulatory guidance may highlight the opportunity for such guidelines to be better informed by COS and vice versa, and we will use the evidence obtained from this study to engage the relevant regulatory bodies in discussions accordingly. We will endeavour to discuss, and ultimately produce guidance about, how researchers should determine key outcomes in the case of lack of concordance between COS and regulatory guidelines.

Details of working group

ICMJE guidance will be followed with regards to publication policy.

Appendix 1. Tabulated results for SCORE-IT COS (T2D case study²⁴).

SCORE-IT COS	Guidance		SCORE-IT core outcome not explicitly mentioned but covered by the following general terms
SCORE-IT COS	EMA	FDA
Overall survival
Death from a diabetes related cause such as heart disease		(✓)^a	^aCardiovascular disease/safety profile
Heart failure	(✓)^a^b	(✓)^a^c	^aCardiovascular disease/safety profile ^bCoronary complications ^cDiabetes-related complications
Gangrene or amputation of the leg, foot or toe	(✓)^d	(✓)^c	^dPeripheral vascular diseases ^cDiabetes-related complications
Hyperglycaemic emergencies¹		(✓)^c	^cDiabetes-related complications
Hyperglycaemia		(✓)^c	^cDiabetes-related complications
Hypoglycaemia	✓⁴	(✓)^c	^cDiabetes-related complications
Cerebrovascular disease	✓	(✓)^c	^cDiabetes-related complications
Hospital admissions due to diabetes
Side effects of treatment	✓	✓
Global quality of life	✓
Nonfatal myocardial infarction	(✓)^a^b	(✓)^a^c	^aCardiovascular disease/safety profile ^bCoronary complications ^cDiabetes-related complications
Visual deterioration or blindness	(✓)^g	(✓)^c	^gRetinopathy ^cDiabetes-related complications
Glycaemic control	✓⁴	✓
Neuropathy²	✓	(✓)^c	^cDiabetes-related complications
Kidney function	✓	(✓)^c	^cDiabetes-related complications
Activities of daily living³	✓
Body weight	✓⁴

¹ Including diabetic ketoacidosis and hyperosmolar hyperglycaemic state ² Damage to the nerves caused by high glucose. This can lead to tingling and pain or numbness in the feet or legs. It can also affect bowel control; stomach emptying and sexual function. ³ Including those related to personal care; household tasks or community-based tasks. ⁴ Included as core efficacy or safety outcome

Faculty Opinions recommended

References

1. http://www.comet-initiative.org/ Accessed 8 March 2021.
2. Gargon E, Williamson PR, Altman DG, et al.: The COMET Initiative database: progress and activities from 2011 to 2013. Trials. 2014; 15(8): 279. PubMed Abstract | Publisher Full Text | Free Full Text
3. http://www.comet-initiative.org/Studies Accessed 8 March 2021.
4. http://www.comet-initiative.org/COSEndorsement Accessed 8 March 2021.
5. Kirkham JJ, Davis K, Altman DG, et al.: Core Outcome Set-STAndards for Development: The COS-STAD recommendations. PLoS Med. 2017; 11(11): e1002447. PubMed Abstract | Publisher Full Text | Free Full Text
6. https://www.nice.org.uk/process/pmg4/chapter/determining-the-evidence-for-review-and-consideration Accessed 8 March 2021.
7. https://www.nice.org.uk/process/pmg6/chapter/developing-review-questions-and-planning-the-systematic-review Accessed 8 March 2021.
8. https://www.ispor.org/docs/default-source/councils/hta-council/health-technology-assessment-good-practices-recommendation-appendix.pdf?sfvrsn=350e6d96_4 Accessed 8 March 2021.
9. https://www.nice.org.uk/process/pmg20/chapter/developing-review-questions-and-planning-the-evidence-review Accessed 8 March 2021.
10. Healthcare Quality Improvement Partnership UPCARE tool. https://www.rcpsych.ac.uk/improving-care/ccqi/national-clinical-audits/national-audit-of-dementia/nad-reports-and-resources?searchTerms=UPCARE Accessed 8 March 2021.
11. Swedish Agency for Health Technology Assessment and Assessment of Social Services. https://www.sbu.se/en/collaboration/cos/ Accessed 2 June 2021.
12. https://www.spirit-statement.org/#1 Accessed 8 March 2021.
13. Chan AW, Tetzlaff JM, Gøtzsche PC, et al.: SPIRIT 2013 explanation and elaboration: guidance for protocols of clinical trials. BMJ. 2013; 346: e7586. PubMed Abstract | Publisher Full Text | Free Full Text
14. https://www.nihr.ac.uk/documents/hta-stage-1-guidance-notes/11743#What_is_the_research_question_/_aims_and_objectives Accessed 8 March 2021.
15. https://www.nihr.ac.uk/explore-nihr/funding-programmes/evidence-synthesis.htm Accessed 8 March 2021.
16. https://www.fda.gov/regulatory-information/search-fda-guidance-documents Accessed 8 March 2021.
17. FDA Guidance: Conduct of Clinical Trials of Medical Products during COVID-19 Public Health Emergency. 2020; https://www.fda.gov/media/136238/download Accessed 8 March 2021.
18. Guidance for Industry: Patient-Reported Outcome Measures: Use in Medical Product Development to Support Labeling Claims. https://www.fda.gov/media/77832/download
19. https://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfcfr/cfrsearch.cfm Accessed 8 March 2021.
20. https://www.ema.europa.eu/en/human-regulatory/research-development/scientific-guidelines Accessed 8 March 2021.
21. Ellwood KC, Trumbo PR, Kavanaugh CJ: How the US Food and Drug Administration evaluates the scientific evidence for health claims. Nutr Rev. 2010; 68(2): 114–21. PubMed Abstract | Publisher Full Text
22. Saldanha IJ, Dodd S, Gorst SL, et al.: More than half of systematic reviews have relevant core outcome sets. J Clin Epidemiol.: 2021; 136: 168–179. PubMed Abstract | Publisher Full Text
23. Harman NL, Wilding JPH, Curry D, et al.: Selecting Core Outcomes for Randomised Effectiveness trials In Type 2 diabetes (SCORE-IT): a patient and healthcare professional consensus on a core outcome set for type 2 diabetes. BMJ Open Diabetes Res Care. 2019; 7(1): e000700. PubMed Abstract | Publisher Full Text | Free Full Text
24. Dodd S, Harman N, Taske N, et al.: Core outcome sets through the healthcare ecosystem: the case of type 2 diabetes mellitus. Trials. 2020; 21(1): 570. PubMed Abstract | Publisher Full Text | Free Full Text

Comments on this article Comments (0)

Version 3

VERSION 3 PUBLISHED 07 May 2021

Author details Author details

Susanna Dodd
Roles: Conceptualization, Investigation, Methodology, Project Administration, Writing – Original Draft Preparation

Rebecca Fish
Roles: Methodology, Writing – Review & Editing

Sarah Gorst
Roles: Methodology, Writing – Review & Editing

Deborah Hall
Roles: Methodology, Writing – Review & Editing

Pamela Jacobsen
Roles: Methodology, Writing – Review & Editing

Jamie Kirkham
Roles: Methodology, Writing – Review & Editing

Barry Main
Roles: Methodology, Writing – Review & Editing

Karen Matvienko-Sikar
Roles: Methodology, Writing – Review & Editing

Ian J. Saldanha
Roles: Methodology, Writing – Review & Editing

Dominic Trépel
Roles: Methodology, Writing – Review & Editing

Paula R. Williamson
Roles: Conceptualization, Methodology, Writing – Review & Editing

Competing interests

No competing interests were disclosed.

Grant information

The author(s) declared that no grants were involved in supporting this work.

Article Versions (3)

version 3

Revised

Published: 02 Aug 2021, 4:45

https://doi.org/10.12688/hrbopenres.13139.3

version 2

Revised

Published: 09 Jun 2021, 4:45

https://doi.org/10.12688/hrbopenres.13139.2

version 1

Published: 07 May 2021, 4:45

https://doi.org/10.12688/hrbopenres.13139.1

© 2021 Dodd S et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Download

Export To

metrics

VIEWS

836

downloads

Citations

SEE MORE DETAILS

CITE

how to cite this article

Dodd S, Fish R, Gorst S et al. Representation of published core outcome sets for research in regulatory guidance: protocol [version 3; peer review: 2 approved]. HRB Open Res 2021, 4:45 (https://doi.org/10.12688/hrbopenres.13139.3)

NOTE: If applicable, it is important to ensure the information in square brackets after the title is included in all citations of this article.

track

receive updates on this article

Track an article to receive email alerts on any updates to this article.

Open Peer Review

Version 2

VERSION 2

PUBLISHED 09 Jun 2021

Revised

Views

Reviewer Report 22 Jul 2021

Donna A. Messner, Center for Medical Technology Policy, Baltimore, MD, USA

Approved

https://doi.org/10.21956/hrbopenres.14512.r29789

The authors have identified a question that is relevant for researchers, COS developers, and the broader health care community: to what extent do FDA and EMA guidance recommendations align with the results of published core outcome sets? The authors set out a well-conceived protocol for exploring this question in a way that makes relevant comparisons between COS and guidance and reports it coherently. I think this is a project that absolutely needs to be done.

In the third paragraph the authors note that uniformity in recommendations from public funders like NIHR would promote greater consistency in outcomes, and “this benefit would have additional impact if the consistency in such recommendations extended to those from commercial sponsors.” Do you mean "for" commercial sponsors? The statement doesn't make sense to me otherwise.

The third paragraph notes that FDA and EMA are influential in commissioning research to help inform their decisions. Yes, but in this context it seems much less important to me than their powerful role as arbiters of evidence for commercialization of new products. This is a primary function of their guidance.

Another thought on impact: FDA and EMA guidance are often not well aligned (to the great frustration of drug developers). So another possible impact of this work might be to facilitate alignment of international regulatory guidance (at least for outcomes) with COS as a common touchstone.

Also, one technical mistake in the introduction. The fourth paragraph of the intro states that FDA guidance is not legally binding, which is true, but then adds “unlike FDA regulations, which are federal laws.” Regulations are not laws. Laws can only be made by the U.S. Congress. Congress authorizes FDA to write regulations to work out the details of how the laws should be implemented. So regulations are legally binding, but they are not laws. (And there is a missing closed parenthesis at the end of the statement about federal laws.)

Is the rationale for, and objectives of, the study clearly described?

Yes
Is the study design appropriate for the research question?

Yes
Are sufficient details of the methods provided to allow replication by others?

Yes
Are the datasets clearly presented in a useable and accessible format?

Not applicable

Competing Interests: No competing interests were disclosed.

Reviewer Expertise: Expertise in core outcome set development, procedures for medical product regulation and approval in the U.S., health care coverage and reimbursement policy, stakeholder engagement methods, and social/policy research design.

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

CITE

Report a concern

Author Response 02 Aug 2021

Susanna Dodd, MRC/NIHR Trials Methodology Research Partnership, Department of Health Data Science, University of Liverpool (a member of Liverpool Health Partners), Liverpool, UK

02 Aug 2021

Author Response
In the third paragraph the authors note that uniformity in recommendations from public funders like NIHR would promote greater consistency in outcomes, and “this benefit would have additional impact if ... Continue reading
In the third paragraph the authors note that uniformity in recommendations from public funders like NIHR would promote greater consistency in outcomes, and “this benefit would have additional impact if the consistency in such recommendations extended to those from commercial sponsors.” Do you mean "for" commercial sponsors? The statement doesn't make sense to me otherwise.

Thank you for pointing out this error – we have changed the word from “from” to “for”.

The third paragraph notes that FDA and EMA are influential in commissioning research to help inform their decisions. Yes, but in this context it seems much less important to me than their powerful role as arbiters of evidence for commercialization of new products. This is a primary function of their guidance.

Thank you for this comment - we have added this point in the introduction.

Another thought on impact: FDA and EMA guidance are often not well aligned (to the great frustration of drug developers). So another possible impact of this work might be to facilitate alignment of international regulatory guidance (at least for outcomes) with COS as a common touchstone.

This is an interesting discussion point which we will be covered in the discussion section of the main publication.

Also, one technical mistake in the introduction. The fourth paragraph of the intro states that FDA guidance is not legally binding, which is true, but then adds “unlike FDA regulations, which are federal laws.” Regulations are not laws. Laws can only be made by the U.S. Congress. Congress authorizes FDA to write regulations to work out the details of how the laws should be implemented. So regulations are legally binding, but they are not laws. (And there is a missing closed parenthesis at the end of the statement about federal laws.)

Thank you for pointing out this technical error – we have changed the text “which are federal laws” to “which are the details of how US Congress laws should be implemented”.
In the third paragraph the authors note that uniformity in recommendations from public funders like NIHR would promote greater consistency in outcomes, and “this benefit would have additional impact if the consistency in such recommendations extended to those from commercial sponsors.” Do you mean "for" commercial sponsors? The statement doesn't make sense to me otherwise.

Thank you for pointing out this error – we have changed the word from “from” to “for”.

The third paragraph notes that FDA and EMA are influential in commissioning research to help inform their decisions. Yes, but in this context it seems much less important to me than their powerful role as arbiters of evidence for commercialization of new products. This is a primary function of their guidance.

Thank you for this comment - we have added this point in the introduction.

Another thought on impact: FDA and EMA guidance are often not well aligned (to the great frustration of drug developers). So another possible impact of this work might be to facilitate alignment of international regulatory guidance (at least for outcomes) with COS as a common touchstone.

This is an interesting discussion point which we will be covered in the discussion section of the main publication.

Also, one technical mistake in the introduction. The fourth paragraph of the intro states that FDA guidance is not legally binding, which is true, but then adds “unlike FDA regulations, which are federal laws.” Regulations are not laws. Laws can only be made by the U.S. Congress. Congress authorizes FDA to write regulations to work out the details of how the laws should be implemented. So regulations are legally binding, but they are not laws. (And there is a missing closed parenthesis at the end of the statement about federal laws.)

Thank you for pointing out this technical error – we have changed the text “which are federal laws” to “which are the details of how US Congress laws should be implemented”.
Competing Interests: None Close
Report a concern
Respond or Comment

COMMENTS ON THIS REPORT

Author Response 02 Aug 2021

Susanna Dodd, MRC/NIHR Trials Methodology Research Partnership, Department of Health Data Science, University of Liverpool (a member of Liverpool Health Partners), Liverpool, UK

02 Aug 2021

Author Response
In the third paragraph the authors note that uniformity in recommendations from public funders like NIHR would promote greater consistency in outcomes, and “this benefit would have additional impact if ... Continue reading
In the third paragraph the authors note that uniformity in recommendations from public funders like NIHR would promote greater consistency in outcomes, and “this benefit would have additional impact if the consistency in such recommendations extended to those from commercial sponsors.” Do you mean "for" commercial sponsors? The statement doesn't make sense to me otherwise.

Thank you for pointing out this error – we have changed the word from “from” to “for”.

The third paragraph notes that FDA and EMA are influential in commissioning research to help inform their decisions. Yes, but in this context it seems much less important to me than their powerful role as arbiters of evidence for commercialization of new products. This is a primary function of their guidance.

Thank you for this comment - we have added this point in the introduction.

Another thought on impact: FDA and EMA guidance are often not well aligned (to the great frustration of drug developers). So another possible impact of this work might be to facilitate alignment of international regulatory guidance (at least for outcomes) with COS as a common touchstone.

This is an interesting discussion point which we will be covered in the discussion section of the main publication.

Also, one technical mistake in the introduction. The fourth paragraph of the intro states that FDA guidance is not legally binding, which is true, but then adds “unlike FDA regulations, which are federal laws.” Regulations are not laws. Laws can only be made by the U.S. Congress. Congress authorizes FDA to write regulations to work out the details of how the laws should be implemented. So regulations are legally binding, but they are not laws. (And there is a missing closed parenthesis at the end of the statement about federal laws.)

Thank you for pointing out this technical error – we have changed the text “which are federal laws” to “which are the details of how US Congress laws should be implemented”.
In the third paragraph the authors note that uniformity in recommendations from public funders like NIHR would promote greater consistency in outcomes, and “this benefit would have additional impact if the consistency in such recommendations extended to those from commercial sponsors.” Do you mean "for" commercial sponsors? The statement doesn't make sense to me otherwise.

Thank you for pointing out this error – we have changed the word from “from” to “for”.

The third paragraph notes that FDA and EMA are influential in commissioning research to help inform their decisions. Yes, but in this context it seems much less important to me than their powerful role as arbiters of evidence for commercialization of new products. This is a primary function of their guidance.

Thank you for this comment - we have added this point in the introduction.

Another thought on impact: FDA and EMA guidance are often not well aligned (to the great frustration of drug developers). So another possible impact of this work might be to facilitate alignment of international regulatory guidance (at least for outcomes) with COS as a common touchstone.

This is an interesting discussion point which we will be covered in the discussion section of the main publication.

Also, one technical mistake in the introduction. The fourth paragraph of the intro states that FDA guidance is not legally binding, which is true, but then adds “unlike FDA regulations, which are federal laws.” Regulations are not laws. Laws can only be made by the U.S. Congress. Congress authorizes FDA to write regulations to work out the details of how the laws should be implemented. So regulations are legally binding, but they are not laws. (And there is a missing closed parenthesis at the end of the statement about federal laws.)

Thank you for pointing out this technical error – we have changed the text “which are federal laws” to “which are the details of how US Congress laws should be implemented”.
Competing Interests: None Close
Report a concern

Views

Reviewer Report 24 Jun 2021

Sophie Werkö, Swedish Council on Health Technology Assessment and Assessment of Social Services (SBU), Stockholm, Sweden

Marie Österberg, Swedish Agency for Health Technology Assessment and Assessment of Social Services (SBU), Stockholm, Sweden

Approved

https://doi.org/10.21956/hrbopenres.14512.r29639

CITE

Report a concern

Author Response 02 Aug 2021

Susanna Dodd, MRC/NIHR Trials Methodology Research Partnership, Department of Health Data Science, University of Liverpool (a member of Liverpool Health Partners), Liverpool, UK

02 Aug 2021

Author Response

Thank you for your comment. We agree with your point and will ensure that we include a discussion in the study publication about the complex issue of COS quality.
Competing Interests: No competing interests were disclosed.
Thank you for your comment. We agree with your point and will ensure that we include a discussion in the study publication about the complex issue of COS quality.
Thank you for your comment. We agree with your point and will ensure that we include a discussion in the study publication about the complex issue of COS quality.
Competing Interests: No competing interests were disclosed. Close
Report a concern
Respond or Comment

COMMENTS ON THIS REPORT

Author Response 02 Aug 2021

Susanna Dodd, MRC/NIHR Trials Methodology Research Partnership, Department of Health Data Science, University of Liverpool (a member of Liverpool Health Partners), Liverpool, UK

02 Aug 2021

Author Response

Thank you for your comment. We agree with your point and will ensure that we include a discussion in the study publication about the complex issue of COS quality.
Competing Interests: No competing interests were disclosed.
Thank you for your comment. We agree with your point and will ensure that we include a discussion in the study publication about the complex issue of COS quality.
Thank you for your comment. We agree with your point and will ensure that we include a discussion in the study publication about the complex issue of COS quality.
Competing Interests: No competing interests were disclosed. Close
Report a concern

Version 1

VERSION 1

PUBLISHED 07 May 2021

Views

Reviewer Report 24 May 2021

Sophie Werkö, Swedish Council on Health Technology Assessment and Assessment of Social Services (SBU), Stockholm, Sweden

Marie Österberg, Swedish Agency for Health Technology Assessment and Assessment of Social Services (SBU), Stockholm, Sweden

Approved with Reservations

https://doi.org/10.21956/hrbopenres.14252.r29391

Thank you for letting us review this interesting study. We think the research question is well chosen and will give interesting insights and results of the use and applicability of COS. We only have some minor comments which we hope ... Continue reading

In the last sentence of the Discussion part in the abstract, the authors state that they will seek feedback from FDA and EMA “if there is a lack of concordance between COS and regulatory guidance overall or for particular disease areas.” We think feedback from the regulatory authorities would be interesting regardless of what the results are.
As the COMET database in our understanding includes studies from different countries, we suggest that an example from another country than the UK as well, is included in the introduction to give a broader perspective. As it reads now, there seems to be only the UK or “globally”.
Regarding methods, it is stated that the studies should have been published between 2015-2019. Why is 2020 not included (as we are in 2021 now). Please explain why this is: because of COVID-19, nothing was published or any other reason.
We currently think there is a lack of discussion on the potential limitations of the study. For example, limitations regarding relevance and context. It would be interesting to see whether a COS -set produced in a high-income country compared to a low-income country differ, and also, if one or the other has influenced EMA or FDA more. We understand that there could be difficulties in drawing conclusions out of this type of information, however it is important to discuss the potential importance of this implication which in the long run may mean that the production of COS will have impact on research outcomes of thus more importance to high-income countries. (Patients or other stakeholders may look upon the prioritization of outcomes differently depending on contextual factors).
There is also a lack of discussion on the quality of the included COS. In the section on methods, we think the authors should add a paragraph on how they will assess quality of the included COS. This should be discussed regarding both the selection of participants, how the actual prioritization process is done as well as numbers of the outcomes included in the core outcome set. (Core outcome sets that include large number of outcomes will likely have more of a concordance with regulatory guidance than those with fewer outcomes. It is our opinion that a core set by definition should include less than 10 outcomes, but when it doesn’t this needs to be addressed.
It would be interesting if the authors note whether the included published core outcome sets had searched for FDA or EMA documentation regarding important outcomes before they started their study.
Finally, we assume the authors will discuss how researchers should relate to core outcome sets that do not include all the requested outcomes requested by regulators. That will be an interesting discussion to read. Looking forward to see this study published!

Is the rationale for, and objectives of, the study clearly described?

Yes
Is the study design appropriate for the research question?

Yes
Are sufficient details of the methods provided to allow replication by others?

Yes
Are the datasets clearly presented in a useable and accessible format?

Yes

Competing Interests: No competing interests were disclosed.

Reviewer Expertise: Health Technology Assessment (HTA)

We confirm that we have read this submission and believe that we have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however we have significant reservations, as outlined above.

CITE

Report a concern

Author Response 14 Jul 2021

Susanna Dodd, MRC/NIHR Trials Methodology Research Partnership, Department of Health Data Science, University of Liverpool (a member of Liverpool Health Partners), Liverpool, UK

14 Jul 2021

Author Response
Thank you for your comments. Our responses are given below:
- In the last sentence of the Discussion part in the abstract, the authors state that they will seek
... Continue reading
Thank you for your comments. Our responses are given below:

In the last sentence of the Discussion part in the abstract, the authors state that they will seek feedback from FDA and EMA “if there is a lack of concordance between COS and regulatory guidance overall or for particular disease areas.” We think feedback from the regulatory authorities would be interesting regardless of what the results are.

Thank you for highlighting this point. We agree with your comment and will endeavour to share the results of this study to the regulatory authorities, regardless of the findings.

As the COMET database in our understanding includes studies from different countries, we suggest that an example from another country than the UK as well, is included in the introduction to give a broader perspective. As it reads now, there seems to be only the UK or “globally”.

Thank you for this suggestion. We have added the example of COS endorsement for use in the development of trials and systematic reviews by the Swedish Agency for Health Technology Assessment and Assessment of Social Services (SBU).

Regarding methods, it is stated that the studies should have been published between 2015-2019. Why is 2020 not included (as we are in 2021 now). Please explain why this is: because of COVID-19, nothing was published or any other reason.

This project was initiated in late 2020, and COS identified as part of the annual COMET systematic reviews were included up to and including the systematic review conducted in 2020, which included studies published up to the end of 2019. This explanation has been added to the manuscript.

We currently think there is a lack of discussion on the potential limitations of the study. For example, limitations regarding relevance and context. It would be interesting to see whether a COS -set produced in a high-income country compared to a low-income country differ, and also, if one or the other has influenced EMA or FDA more. We understand that there could be difficulties in drawing conclusions out of this type of information, however it is important to discuss the potential importance of this implication which in the long run may mean that the production of COS will have impact on research outcomes of thus more importance to high-income countries. (Patients or other stakeholders may look upon the prioritization of outcomes differently depending on contextual factors).

Thank you for this suggestion. The vast majority of COS for research are led from a high-income country (HIC); a recent systematic review found only four COS for research that were led from low/middle-income countries (LMIC). As such it is unlikely that it will be possible to make meaningful comparisons between COS from a HIC/LMIC to determine their respective impact on regulatory guidelines. However, the COMET database contains information on whether LMIC participants were involved in COS development, and therefore we could assess whether the involvement of LMIC participants in COS development impacts on the concordance between COS and regulatory guidelines. This has been added to the data extraction and analysis sections of the manuscript.

There is also a lack of discussion on the quality of the included COS. In the section on methods, we think the authors should add a paragraph on how they will assess quality of the included COS. This should be discussed regarding both the selection of participants, how the actual prioritization process is done as well as numbers of the outcomes included in the core outcome set. (Core outcome sets that include large number of outcomes will likely have more of a concordance with regulatory guidance than those with fewer outcomes. It is our opinion that a core set by definition should include less than 10 outcomes, but when it doesn’t this needs to be addressed.

It is relatively difficult to define COS ‘quality’; the COS-STAD standards were introduced in 2017, but we do not think it would be fair or helpful to apply COS-STAD retrospectively to studies that were published before 2017. We decided to include only those COS for research which were published in recent years (between 2015 and 2019) and which involved patients in the consensus process, in order to increase the likelihood that high standard COS are included. We have added to the analysis section that we will explore the impact of the number of core outcomes on the level of agreement between COS and corresponding EMA/FDA guidelines.

It would be interesting if the authors note whether the included published core outcome sets had searched for FDA or EMA documentation regarding important outcomes before they started their study.

Thank you for this comment. This information is currently being extracted from each COS publication, so we have added this explicitly to the methods section.

Finally, we assume the authors will discuss how researchers should relate to core outcome sets that do not include all the requested outcomes requested by regulators. That will be an interesting discussion to read. Looking forward to see this study published!

Thank you for this suggestion. This is an interesting point which we had not mentioned in the protocol, so we have now added it to the discussion.
Thank you for your comments. Our responses are given below:

In the last sentence of the Discussion part in the abstract, the authors state that they will seek feedback from FDA and EMA “if there is a lack of concordance between COS and regulatory guidance overall or for particular disease areas.” We think feedback from the regulatory authorities would be interesting regardless of what the results are.

Thank you for highlighting this point. We agree with your comment and will endeavour to share the results of this study to the regulatory authorities, regardless of the findings.

As the COMET database in our understanding includes studies from different countries, we suggest that an example from another country than the UK as well, is included in the introduction to give a broader perspective. As it reads now, there seems to be only the UK or “globally”.

Thank you for this suggestion. We have added the example of COS endorsement for use in the development of trials and systematic reviews by the Swedish Agency for Health Technology Assessment and Assessment of Social Services (SBU).

Regarding methods, it is stated that the studies should have been published between 2015-2019. Why is 2020 not included (as we are in 2021 now). Please explain why this is: because of COVID-19, nothing was published or any other reason.

This project was initiated in late 2020, and COS identified as part of the annual COMET systematic reviews were included up to and including the systematic review conducted in 2020, which included studies published up to the end of 2019. This explanation has been added to the manuscript.

We currently think there is a lack of discussion on the potential limitations of the study. For example, limitations regarding relevance and context. It would be interesting to see whether a COS -set produced in a high-income country compared to a low-income country differ, and also, if one or the other has influenced EMA or FDA more. We understand that there could be difficulties in drawing conclusions out of this type of information, however it is important to discuss the potential importance of this implication which in the long run may mean that the production of COS will have impact on research outcomes of thus more importance to high-income countries. (Patients or other stakeholders may look upon the prioritization of outcomes differently depending on contextual factors).

Thank you for this suggestion. The vast majority of COS for research are led from a high-income country (HIC); a recent systematic review found only four COS for research that were led from low/middle-income countries (LMIC). As such it is unlikely that it will be possible to make meaningful comparisons between COS from a HIC/LMIC to determine their respective impact on regulatory guidelines. However, the COMET database contains information on whether LMIC participants were involved in COS development, and therefore we could assess whether the involvement of LMIC participants in COS development impacts on the concordance between COS and regulatory guidelines. This has been added to the data extraction and analysis sections of the manuscript.

There is also a lack of discussion on the quality of the included COS. In the section on methods, we think the authors should add a paragraph on how they will assess quality of the included COS. This should be discussed regarding both the selection of participants, how the actual prioritization process is done as well as numbers of the outcomes included in the core outcome set. (Core outcome sets that include large number of outcomes will likely have more of a concordance with regulatory guidance than those with fewer outcomes. It is our opinion that a core set by definition should include less than 10 outcomes, but when it doesn’t this needs to be addressed.

It is relatively difficult to define COS ‘quality’; the COS-STAD standards were introduced in 2017, but we do not think it would be fair or helpful to apply COS-STAD retrospectively to studies that were published before 2017. We decided to include only those COS for research which were published in recent years (between 2015 and 2019) and which involved patients in the consensus process, in order to increase the likelihood that high standard COS are included. We have added to the analysis section that we will explore the impact of the number of core outcomes on the level of agreement between COS and corresponding EMA/FDA guidelines.

It would be interesting if the authors note whether the included published core outcome sets had searched for FDA or EMA documentation regarding important outcomes before they started their study.

Thank you for this comment. This information is currently being extracted from each COS publication, so we have added this explicitly to the methods section.

Finally, we assume the authors will discuss how researchers should relate to core outcome sets that do not include all the requested outcomes requested by regulators. That will be an interesting discussion to read. Looking forward to see this study published!

Thank you for this suggestion. This is an interesting point which we had not mentioned in the protocol, so we have now added it to the discussion.
Competing Interests: I have no competing interests. Close
Report a concern
Respond or Comment

COMMENTS ON THIS REPORT

Author Response 14 Jul 2021

Susanna Dodd, MRC/NIHR Trials Methodology Research Partnership, Department of Health Data Science, University of Liverpool (a member of Liverpool Health Partners), Liverpool, UK

14 Jul 2021

Author Response
Thank you for your comments. Our responses are given below:
- In the last sentence of the Discussion part in the abstract, the authors state that they will seek
... Continue reading
Thank you for your comments. Our responses are given below:

In the last sentence of the Discussion part in the abstract, the authors state that they will seek feedback from FDA and EMA “if there is a lack of concordance between COS and regulatory guidance overall or for particular disease areas.” We think feedback from the regulatory authorities would be interesting regardless of what the results are.

Thank you for highlighting this point. We agree with your comment and will endeavour to share the results of this study to the regulatory authorities, regardless of the findings.

As the COMET database in our understanding includes studies from different countries, we suggest that an example from another country than the UK as well, is included in the introduction to give a broader perspective. As it reads now, there seems to be only the UK or “globally”.

Thank you for this suggestion. We have added the example of COS endorsement for use in the development of trials and systematic reviews by the Swedish Agency for Health Technology Assessment and Assessment of Social Services (SBU).

Regarding methods, it is stated that the studies should have been published between 2015-2019. Why is 2020 not included (as we are in 2021 now). Please explain why this is: because of COVID-19, nothing was published or any other reason.

This project was initiated in late 2020, and COS identified as part of the annual COMET systematic reviews were included up to and including the systematic review conducted in 2020, which included studies published up to the end of 2019. This explanation has been added to the manuscript.

We currently think there is a lack of discussion on the potential limitations of the study. For example, limitations regarding relevance and context. It would be interesting to see whether a COS -set produced in a high-income country compared to a low-income country differ, and also, if one or the other has influenced EMA or FDA more. We understand that there could be difficulties in drawing conclusions out of this type of information, however it is important to discuss the potential importance of this implication which in the long run may mean that the production of COS will have impact on research outcomes of thus more importance to high-income countries. (Patients or other stakeholders may look upon the prioritization of outcomes differently depending on contextual factors).

Thank you for this suggestion. The vast majority of COS for research are led from a high-income country (HIC); a recent systematic review found only four COS for research that were led from low/middle-income countries (LMIC). As such it is unlikely that it will be possible to make meaningful comparisons between COS from a HIC/LMIC to determine their respective impact on regulatory guidelines. However, the COMET database contains information on whether LMIC participants were involved in COS development, and therefore we could assess whether the involvement of LMIC participants in COS development impacts on the concordance between COS and regulatory guidelines. This has been added to the data extraction and analysis sections of the manuscript.

There is also a lack of discussion on the quality of the included COS. In the section on methods, we think the authors should add a paragraph on how they will assess quality of the included COS. This should be discussed regarding both the selection of participants, how the actual prioritization process is done as well as numbers of the outcomes included in the core outcome set. (Core outcome sets that include large number of outcomes will likely have more of a concordance with regulatory guidance than those with fewer outcomes. It is our opinion that a core set by definition should include less than 10 outcomes, but when it doesn’t this needs to be addressed.

It is relatively difficult to define COS ‘quality’; the COS-STAD standards were introduced in 2017, but we do not think it would be fair or helpful to apply COS-STAD retrospectively to studies that were published before 2017. We decided to include only those COS for research which were published in recent years (between 2015 and 2019) and which involved patients in the consensus process, in order to increase the likelihood that high standard COS are included. We have added to the analysis section that we will explore the impact of the number of core outcomes on the level of agreement between COS and corresponding EMA/FDA guidelines.

It would be interesting if the authors note whether the included published core outcome sets had searched for FDA or EMA documentation regarding important outcomes before they started their study.

Thank you for this comment. This information is currently being extracted from each COS publication, so we have added this explicitly to the methods section.

Finally, we assume the authors will discuss how researchers should relate to core outcome sets that do not include all the requested outcomes requested by regulators. That will be an interesting discussion to read. Looking forward to see this study published!

Thank you for this suggestion. This is an interesting point which we had not mentioned in the protocol, so we have now added it to the discussion.
Thank you for your comments. Our responses are given below:

In the last sentence of the Discussion part in the abstract, the authors state that they will seek feedback from FDA and EMA “if there is a lack of concordance between COS and regulatory guidance overall or for particular disease areas.” We think feedback from the regulatory authorities would be interesting regardless of what the results are.

Thank you for highlighting this point. We agree with your comment and will endeavour to share the results of this study to the regulatory authorities, regardless of the findings.

As the COMET database in our understanding includes studies from different countries, we suggest that an example from another country than the UK as well, is included in the introduction to give a broader perspective. As it reads now, there seems to be only the UK or “globally”.

Thank you for this suggestion. We have added the example of COS endorsement for use in the development of trials and systematic reviews by the Swedish Agency for Health Technology Assessment and Assessment of Social Services (SBU).

Regarding methods, it is stated that the studies should have been published between 2015-2019. Why is 2020 not included (as we are in 2021 now). Please explain why this is: because of COVID-19, nothing was published or any other reason.

This project was initiated in late 2020, and COS identified as part of the annual COMET systematic reviews were included up to and including the systematic review conducted in 2020, which included studies published up to the end of 2019. This explanation has been added to the manuscript.

We currently think there is a lack of discussion on the potential limitations of the study. For example, limitations regarding relevance and context. It would be interesting to see whether a COS -set produced in a high-income country compared to a low-income country differ, and also, if one or the other has influenced EMA or FDA more. We understand that there could be difficulties in drawing conclusions out of this type of information, however it is important to discuss the potential importance of this implication which in the long run may mean that the production of COS will have impact on research outcomes of thus more importance to high-income countries. (Patients or other stakeholders may look upon the prioritization of outcomes differently depending on contextual factors).

Thank you for this suggestion. The vast majority of COS for research are led from a high-income country (HIC); a recent systematic review found only four COS for research that were led from low/middle-income countries (LMIC). As such it is unlikely that it will be possible to make meaningful comparisons between COS from a HIC/LMIC to determine their respective impact on regulatory guidelines. However, the COMET database contains information on whether LMIC participants were involved in COS development, and therefore we could assess whether the involvement of LMIC participants in COS development impacts on the concordance between COS and regulatory guidelines. This has been added to the data extraction and analysis sections of the manuscript.

There is also a lack of discussion on the quality of the included COS. In the section on methods, we think the authors should add a paragraph on how they will assess quality of the included COS. This should be discussed regarding both the selection of participants, how the actual prioritization process is done as well as numbers of the outcomes included in the core outcome set. (Core outcome sets that include large number of outcomes will likely have more of a concordance with regulatory guidance than those with fewer outcomes. It is our opinion that a core set by definition should include less than 10 outcomes, but when it doesn’t this needs to be addressed.

It is relatively difficult to define COS ‘quality’; the COS-STAD standards were introduced in 2017, but we do not think it would be fair or helpful to apply COS-STAD retrospectively to studies that were published before 2017. We decided to include only those COS for research which were published in recent years (between 2015 and 2019) and which involved patients in the consensus process, in order to increase the likelihood that high standard COS are included. We have added to the analysis section that we will explore the impact of the number of core outcomes on the level of agreement between COS and corresponding EMA/FDA guidelines.

It would be interesting if the authors note whether the included published core outcome sets had searched for FDA or EMA documentation regarding important outcomes before they started their study.

Thank you for this comment. This information is currently being extracted from each COS publication, so we have added this explicitly to the methods section.

Finally, we assume the authors will discuss how researchers should relate to core outcome sets that do not include all the requested outcomes requested by regulators. That will be an interesting discussion to read. Looking forward to see this study published!

Thank you for this suggestion. This is an interesting point which we had not mentioned in the protocol, so we have now added it to the discussion.
Competing Interests: I have no competing interests. Close
Report a concern

Comments on this article Comments (0)

Version 3

VERSION 3 PUBLISHED 07 May 2021

Open Peer Review

Reviewer Status

Reviewer Reports

	Invited Reviewers
	1	2
Version 3 (revision) 02 Aug 21
Version 2 (revision) 09 Jun 21	read	read
Version 1 07 May 21	read

Sophie Werkö, Swedish Council on Health Technology Assessment and Assessment of Social Services (SBU), Stockholm, Sweden

Marie Österberg, Swedish Agency for Health Technology Assessment and Assessment of Social Services (SBU), Stockholm, Sweden
Donna A. Messner, Center for Medical Technology Policy, Baltimore, USA

Comments on this article

All Comments(0)

Add a comment

Back to all reports

Reviewer Report

33 Views

22 Jul 2021 | for Version 2

Donna A. Messner, Center for Medical Technology Policy, Baltimore, MD, USA

33 Views Cite this report Responses(1)

Approved

Is the rationale for, and objectives of, the study clearly described?

Yes
Is the study design appropriate for the research question?

Yes
Are sufficient details of the methods provided to allow replication by others?

Yes
Are the datasets clearly presented in a useable and accessible format?

Not applicable

Competing Interests

No competing interests were disclosed.

Reviewer Expertise

Expertise in core outcome set development, procedures for medical product regulation and approval in the U.S., health care coverage and reimbursement policy, stakeholder engagement methods, and social/policy research design.

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

Respond to this report

Responses (1)

Author Response

02 Aug 2021

Susanna Dodd, MRC/NIHR Trials Methodology Research Partnership, Department of Health Data Science, University of Liverpool (a member of Liverpool Health Partners), Liverpool, UK

In the third paragraph the authors note that uniformity in recommendations from public funders like NIHR would promote greater consistency in outcomes, and “this benefit would have additional impact if the consistency in such recommendations extended to those from commercial sponsors.” Do you mean "for" commercial sponsors? The statement doesn't make sense to me otherwise.

Thank you for pointing out this error – we have changed the word from “from” to “for”.

The third paragraph notes that FDA and EMA are influential in commissioning research to help inform their decisions. Yes, but in this context it seems much less important to me than their powerful role as arbiters of evidence for commercialization of new products. This is a primary function of their guidance.

Thank you for this comment - we have added this point in the introduction.

Another thought on impact: FDA and EMA guidance are often not well aligned (to the great frustration of drug developers). So another possible impact of this work might be to facilitate alignment of international regulatory guidance (at least for outcomes) with COS as a common touchstone.

This is an interesting discussion point which we will be covered in the discussion section of the main publication.

Also, one technical mistake in the introduction. The fourth paragraph of the intro states that FDA guidance is not legally binding, which is true, but then adds “unlike FDA regulations, which are federal laws.” Regulations are not laws. Laws can only be made by the U.S. Congress. Congress authorizes FDA to write regulations to work out the details of how the laws should be implemented. So regulations are legally binding, but they are not laws. (And there is a missing closed parenthesis at the end of the statement about federal laws.)

Thank you for pointing out this technical error – we have changed the text “which are federal laws” to “which are the details of how US Congress laws should be implemented”.

View more View less

Competing Interests

None

Back to all reports

Reviewer Report

36 Views

24 Jun 2021 | for Version 2

Sophie Werkö, Swedish Council on Health Technology Assessment and Assessment of Social Services (SBU), Stockholm, Sweden

Marie Österberg, Swedish Agency for Health Technology Assessment and Assessment of Social Services (SBU), Stockholm, Sweden

36 Views Cite this report Responses(1)

Approved

Thank you, we have now checked the document and have really only one remaining comment for potential improvement.

We still lack a discussion on the quality of the included COS. We understand if this may be out of scope for this study, but we still believe that it would be interesting to hold such a discussion as we already today are faced with only a little guidance on how to deal with obvious low quality COS. It would be a pity when conducting this otherwise great study, to not include these aspects, because they are already part of the current problems in addressing COS today. Please take this as advice only. We are happy to approve of the current version.

Competing Interests

No competing interests were disclosed.

We confirm that we have read this submission and believe that we have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

Respond to this report

Responses (1)

Back to all reports

Reviewer Report

43 Views

24 May 2021 | for Version 1

Sophie Werkö, Swedish Council on Health Technology Assessment and Assessment of Social Services (SBU), Stockholm, Sweden

Marie Österberg, Swedish Agency for Health Technology Assessment and Assessment of Social Services (SBU), Stockholm, Sweden

43 Views Cite this report Responses(1)

Approved With Reservations

In the last sentence of the Discussion part in the abstract, the authors state that they will seek feedback from FDA and EMA “if there is a lack of concordance between COS and regulatory guidance overall or for particular disease areas.” We think feedback from the regulatory authorities would be interesting regardless of what the results are.
As the COMET database in our understanding includes studies from different countries, we suggest that an example from another country than the UK as well, is included in the introduction to give a broader perspective. As it reads now, there seems to be only the UK or “globally”.
Regarding methods, it is stated that the studies should have been published between 2015-2019. Why is 2020 not included (as we are in 2021 now). Please explain why this is: because of COVID-19, nothing was published or any other reason.
We currently think there is a lack of discussion on the potential limitations of the study. For example, limitations regarding relevance and context. It would be interesting to see whether a COS -set produced in a high-income country compared to a low-income country differ, and also, if one or the other has influenced EMA or FDA more. We understand that there could be difficulties in drawing conclusions out of this type of information, however it is important to discuss the potential importance of this implication which in the long run may mean that the production of COS will have impact on research outcomes of thus more importance to high-income countries. (Patients or other stakeholders may look upon the prioritization of outcomes differently depending on contextual factors).
There is also a lack of discussion on the quality of the included COS. In the section on methods, we think the authors should add a paragraph on how they will assess quality of the included COS. This should be discussed regarding both the selection of participants, how the actual prioritization process is done as well as numbers of the outcomes included in the core outcome set. (Core outcome sets that include large number of outcomes will likely have more of a concordance with regulatory guidance than those with fewer outcomes. It is our opinion that a core set by definition should include less than 10 outcomes, but when it doesn’t this needs to be addressed.
It would be interesting if the authors note whether the included published core outcome sets had searched for FDA or EMA documentation regarding important outcomes before they started their study.
Finally, we assume the authors will discuss how researchers should relate to core outcome sets that do not include all the requested outcomes requested by regulators. That will be an interesting discussion to read. Looking forward to see this study published!

Is the rationale for, and objectives of, the study clearly described?

Yes
Is the study design appropriate for the research question?

Yes
Are sufficient details of the methods provided to allow replication by others?

Yes
Are the datasets clearly presented in a useable and accessible format?

Yes

Competing Interests

No competing interests were disclosed.

Reviewer Expertise

Health Technology Assessment (HTA)

Respond to this report

Responses (1)

Author Response

14 Jul 2021

Susanna Dodd, MRC/NIHR Trials Methodology Research Partnership, Department of Health Data Science, University of Liverpool (a member of Liverpool Health Partners), Liverpool, UK

Thank you for your comments. Our responses are given below:

In the last sentence of the Discussion part in the abstract, the authors state that they will seek feedback from FDA and EMA “if there is a lack of concordance between COS and regulatory guidance overall or for particular disease areas.” We think feedback from the regulatory authorities would be interesting regardless of what the results are.

Thank you for highlighting this point. We agree with your comment and will endeavour to share the results of this study to the regulatory authorities, regardless of the findings.

As the COMET database in our understanding includes studies from different countries, we suggest that an example from another country than the UK as well, is included in the introduction to give a broader perspective. As it reads now, there seems to be only the UK or “globally”.

Thank you for this suggestion. We have added the example of COS endorsement for use in the development of trials and systematic reviews by the Swedish Agency for Health Technology Assessment and Assessment of Social Services (SBU).

Regarding methods, it is stated that the studies should have been published between 2015-2019. Why is 2020 not included (as we are in 2021 now). Please explain why this is: because of COVID-19, nothing was published or any other reason.

This project was initiated in late 2020, and COS identified as part of the annual COMET systematic reviews were included up to and including the systematic review conducted in 2020, which included studies published up to the end of 2019. This explanation has been added to the manuscript.

We currently think there is a lack of discussion on the potential limitations of the study. For example, limitations regarding relevance and context. It would be interesting to see whether a COS -set produced in a high-income country compared to a low-income country differ, and also, if one or the other has influenced EMA or FDA more. We understand that there could be difficulties in drawing conclusions out of this type of information, however it is important to discuss the potential importance of this implication which in the long run may mean that the production of COS will have impact on research outcomes of thus more importance to high-income countries. (Patients or other stakeholders may look upon the prioritization of outcomes differently depending on contextual factors).

Thank you for this suggestion. The vast majority of COS for research are led from a high-income country (HIC); a recent systematic review found only four COS for research that were led from low/middle-income countries (LMIC). As such it is unlikely that it will be possible to make meaningful comparisons between COS from a HIC/LMIC to determine their respective impact on regulatory guidelines. However, the COMET database contains information on whether LMIC participants were involved in COS development, and therefore we could assess whether the involvement of LMIC participants in COS development impacts on the concordance between COS and regulatory guidelines. This has been added to the data extraction and analysis sections of the manuscript.

There is also a lack of discussion on the quality of the included COS. In the section on methods, we think the authors should add a paragraph on how they will assess quality of the included COS. This should be discussed regarding both the selection of participants, how the actual prioritization process is done as well as numbers of the outcomes included in the core outcome set. (Core outcome sets that include large number of outcomes will likely have more of a concordance with regulatory guidance than those with fewer outcomes. It is our opinion that a core set by definition should include less than 10 outcomes, but when it doesn’t this needs to be addressed.

It is relatively difficult to define COS ‘quality’; the COS-STAD standards were introduced in 2017, but we do not think it would be fair or helpful to apply COS-STAD retrospectively to studies that were published before 2017. We decided to include only those COS for research which were published in recent years (between 2015 and 2019) and which involved patients in the consensus process, in order to increase the likelihood that high standard COS are included. We have added to the analysis section that we will explore the impact of the number of core outcomes on the level of agreement between COS and corresponding EMA/FDA guidelines.

It would be interesting if the authors note whether the included published core outcome sets had searched for FDA or EMA documentation regarding important outcomes before they started their study.

Thank you for this comment. This information is currently being extracted from each COS publication, so we have added this explicitly to the methods section.

Finally, we assume the authors will discuss how researchers should relate to core outcome sets that do not include all the requested outcomes requested by regulators. That will be an interesting discussion to read. Looking forward to see this study published!

Thank you for this suggestion. This is an interesting point which we had not mentioned in the protocol, so we have now added it to the discussion.

View more View less

Competing Interests

I have no competing interests.

Alongside their report, reviewers assign a status to the article:

Approved - the paper is scientifically sound in its current form and only minor, if any, improvements are suggested

Approved with reservations - A number of small changes, sometimes more significant revisions are required to address specific details and improve the papers academic merit.

Not approved - fundamental flaws in the paper seriously undermine the findings and conclusions

[1] 1. http://www.comet-initiative.org/ Accessed 8 March 2021.

[2] 2. Gargon E, Williamson PR, Altman DG, et al.: The COMET Initiative database: progress and activities from 2011 to 2013. Trials. 2014; 15(8): 279. PubMed Abstract | Publisher Full Text | Free Full Text

[3] 3. http://www.comet-initiative.org/Studies Accessed 8 March 2021.

[4] 4. http://www.comet-initiative.org/COSEndorsement Accessed 8 March 2021.

[5] 5. Kirkham JJ, Davis K, Altman DG, et al.: Core Outcome Set-STAndards for Development: The COS-STAD recommendations. PLoS Med. 2017; 11(11): e1002447. PubMed Abstract | Publisher Full Text | Free Full Text

[6] 6. https://www.nice.org.uk/process/pmg4/chapter/determining-the-evidence-for-review-and-consideration Accessed 8 March 2021.

[7] 7. https://www.nice.org.uk/process/pmg6/chapter/developing-review-questions-and-planning-the-systematic-review Accessed 8 March 2021.

[8] 8. https://www.ispor.org/docs/default-source/councils/hta-council/health-technology-assessment-good-practices-recommendation-appendix.pdf?sfvrsn=350e6d96_4 Accessed 8 March 2021.

[9] 9. https://www.nice.org.uk/process/pmg20/chapter/developing-review-questions-and-planning-the-evidence-review Accessed 8 March 2021.

[10] 10. Healthcare Quality Improvement Partnership UPCARE tool. https://www.rcpsych.ac.uk/improving-care/ccqi/national-clinical-audits/national-audit-of-dementia/nad-reports-and-resources?searchTerms=UPCARE Accessed 8 March 2021.

[11] 11. Swedish Agency for Health Technology Assessment and Assessment of Social Services. https://www.sbu.se/en/collaboration/cos/ Accessed 2 June 2021.

[12] 12. https://www.spirit-statement.org/#1 Accessed 8 March 2021.

[13] 13. Chan AW, Tetzlaff JM, Gøtzsche PC, et al.: SPIRIT 2013 explanation and elaboration: guidance for protocols of clinical trials. BMJ. 2013; 346: e7586. PubMed Abstract | Publisher Full Text | Free Full Text

[14] 14. https://www.nihr.ac.uk/documents/hta-stage-1-guidance-notes/11743#What_is_the_research_question_/_aims_and_objectives Accessed 8 March 2021.

[15] 15. https://www.nihr.ac.uk/explore-nihr/funding-programmes/evidence-synthesis.htm Accessed 8 March 2021.

[16] 16. https://www.fda.gov/regulatory-information/search-fda-guidance-documents Accessed 8 March 2021.

[17] 17. FDA Guidance: Conduct of Clinical Trials of Medical Products during COVID-19 Public Health Emergency. 2020; https://www.fda.gov/media/136238/download Accessed 8 March 2021.

[18] 18. Guidance for Industry: Patient-Reported Outcome Measures: Use in Medical Product Development to Support Labeling Claims. https://www.fda.gov/media/77832/download

[19] 19. https://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfcfr/cfrsearch.cfm Accessed 8 March 2021.

[20] 20. https://www.ema.europa.eu/en/human-regulatory/research-development/scientific-guidelines Accessed 8 March 2021.

[21] 21. Ellwood KC, Trumbo PR, Kavanaugh CJ: How the US Food and Drug Administration evaluates the scientific evidence for health claims. Nutr Rev. 2010; 68(2): 114–21. PubMed Abstract | Publisher Full Text

[22] 22. Saldanha IJ, Dodd S, Gorst SL, et al.: More than half of systematic reviews have relevant core outcome sets. J Clin Epidemiol.: 2021; 136: 168–179. PubMed Abstract | Publisher Full Text

[23] 23. Harman NL, Wilding JPH, Curry D, et al.: Selecting Core Outcomes for Randomised Effectiveness trials In Type 2 diabetes (SCORE-IT): a patient and healthcare professional consensus on a core outcome set for type 2 diabetes. BMJ Open Diabetes Res Care. 2019; 7(1): e000700. PubMed Abstract | Publisher Full Text | Free Full Text

[24] 24. Dodd S, Harman N, Taske N, et al.: Core outcome sets through the healthcare ecosystem: the case of type 2 diabetes mellitus. Trials. 2020; 21(1): 570. PubMed Abstract | Publisher Full Text | Free Full Text

Representation of published core outcome sets for research in regulatory guidance: protocol

Abstract

Keywords

Revised Amendments from Version 2

Introduction

Methods

Search strategy

Figure 1. Scope matching algorithm determined according to the descriptions of the population and intervention within the FDA/EMA guideline versus the corresponding COS.

Eligibility

Data extraction

Analysis

Dissemination

Discussion

Details of working group

Appendix 1. Tabulated results for SCORE-IT COS (T2D case study²⁴).

References

Comments on this article Comments (0)

Open Peer Review

Comments on this article Comments (0)

Open Peer Review

Reviewer Status

Reviewer Reports

Comments on this article

Are you a HRB-funded researcher?

Thank you!

Representation of published core outcome sets for research in regulatory guidance: protocol

Abstract

Keywords

Revised Amendments from Version 2

Introduction

Methods

Search strategy

Figure 1. Scope matching algorithm determined according to the descriptions of the population and intervention within the FDA/EMA guideline versus the corresponding COS.

Eligibility

Data extraction

Analysis

Dissemination

Discussion

Details of working group

Appendix 1. Tabulated results for SCORE-IT COS (T2D case study24).

References

Comments on this article Comments (0)

Open Peer Review

Comments on this article Comments (0)

Open Peer Review

Reviewer Status

Reviewer Reports

Comments on this article

Competing Interests Policy

Stay Updated

Are you a HRB-funded researcher?

Thank you!

Appendix 1. Tabulated results for SCORE-IT COS (T2D case study²⁴).