Uptake of core outcome sets by clinical trialists publishing in major medical journals: Protocol

Karen Matvienko-Sikar; Kerry Avery; Jane Blazeby; Karen Hughes; Pamela Jacobsen; Jamie Kirkham; Jan Kottner; Katie Mellor; Ian J Saldanha; Valerie Smith; Caroline B. Terwee; Paula R. Williamson; MRC-NIHR TMRP Outcomes Working Group Core Outcome Set Theme

doi:10.12688/hrbopenres.13109.2

Home Browse Uptake of core outcome sets by clinical trialists publishing in major...

ALL Metrics

Views

Downloads

Get PDF

Get XML

Export

▬

✚

Study Protocol

Revised

Uptake of core outcome sets by clinical trialists publishing in major medical journals: Protocol

[version 2; peer review: 2 approved]

Karen Matvienko-Sikar ¹, Kerry Avery², Jane Blazeby², [...] Karen Hughes³, Pamela Jacobsen⁴, Jamie Kirkham⁵, Jan Kottner⁶, Katie Mellor⁷, Ian J Saldanha⁸, Valerie Smith⁹, Caroline B. Terwee¹⁰, Paula R. Williamson³, MRC-NIHR TMRP Outcomes Working Group Core Outcome Set Theme

Karen Matvienko-Sikar ¹, Kerry Avery², [...] Jane Blazeby², Karen Hughes³, Pamela Jacobsen⁴, Jamie Kirkham⁵, Jan Kottner⁶, Katie Mellor⁷, Ian J Saldanha⁸, Valerie Smith⁹, Caroline B. Terwee¹⁰, Paula R. Williamson³, MRC-NIHR TMRP Outcomes Working Group Core Outcome Set Theme

PUBLISHED 25 Jan 2021

Author details Author details

¹ School of Public Health, University College Cork, Cork, Ireland
² NIHR Biomedical Research Centre, Bristol Medical School: Population Health Sciences, University of Bristol, Bristol, UK
³ MRC/NIHR Trials Methodology Research Partnership, Department of Biostatistics, University of Liverpool, Liverpool, UK
⁴ Department of Psychology, University of Bath, Bath, UK
⁵ Centre for Biostatistics, Manchester Academic Health Science Centre,, University of Manchester, Manchester, UK
⁶ Department of Dermatology and Allergy, Clinical Research Centre for Hair and Skin Science, Berlin, Germany
⁷ Centre for Statistics in Medicine, University of Oxford, Oxford, UK
⁸ Center for Evidence Synthesis in Health; Department of Health Services, Policy, and Practice, Brown University School of Public Health, Providence, Rhode Island, USA
⁹ School of Nursing and Midwifery, Trinity College Dublin, Dublin, Ireland
¹⁰ Amsterdam UMC, Department of Epidemiology and Biostatistics, Amsterdam Public Health Research Institute, Vrije Universiteit Amsterdam, Amsterdam, The Netherlands

Karen Matvienko-Sikar
Roles: Conceptualization, Methodology, Writing – Original Draft Preparation, Writing – Review & Editing

Kerry Avery
Roles: Methodology, Writing – Review & Editing

Jane Blazeby
Roles: Methodology, Writing – Review & Editing

Karen Hughes
Roles: Methodology, Writing – Review & Editing

Pamela Jacobsen
Roles: Methodology, Writing – Review & Editing

Jamie Kirkham
Roles: Methodology, Writing – Review & Editing

Jan Kottner
Roles: Methodology, Writing – Review & Editing

Katie Mellor
Roles: Methodology, Writing – Review & Editing

Ian J Saldanha
Roles: Methodology, Writing – Review & Editing

Valerie Smith
Roles: Methodology, Writing – Review & Editing

Caroline B. Terwee
Roles: Methodology, Writing – Review & Editing

Paula R. Williamson
Roles: Conceptualization, Methodology, Writing – Original Draft Preparation, Writing – Review & Editing

OPEN PEER REVIEW

REVIEWER STATUS

Abstract

Background: Outcome heterogeneity, selective reporting, and choosing outcomes that do not reflect needs and priorities of stakeholders, limit the examination of health intervention effects, particularly in late phase trials. Core outcome sets (COS) are a proposed solution to these issues. A COS is an agreed-upon, standardised set of outcomes that should be measured and reported as a minimum in all trials in a specific area of health or healthcare. COS are intended to increase standardisation of outcome measurement and reporting to better enable comparisons between, and synthesis of findings of trials in a particular health area.
Methods: This study will examine late phase trials, published between October 2019 and March 2020 (inclusive), in the following five medical journals: New England Journal of Medicine, Journal of the American Medical Association, Lancet, BMJ, and Annals of Internal Medicine. Trials will be examined to determine if they refer to a COS, and whether they use a COS. Trialists for each identified trial will subsequently be contacted to complete an online survey examining trialists’ awareness of, and decisions to search for and use a COS.
Discussion: This study will provide important information on uptake of COS by later phase trialists in major medical journals, and the views of these trialists on COS use in trials. These findings will inform approaches to increasing awareness and uptake of COS in future health trials.

Keywords

Core outcome sets, trials, uptake

Corresponding author: Karen Matvienko-Sikar

Competing interests: No competing interests were disclosed.

Grant information: Health Research Board Ireland [HRB-ARPP-A-011]

KMS is supported by a Health Research Board Applying Research into Policy and Practice Fellowship [HRB-ARPP-A-011]. KA and JB are supported by the NIHR Biomedical Research Centre at University Hospitals Bristol and Weston NHS Foundation Trust and the University of Bristol. The views expressed are those of the author(s) and not necessarily those of the NIHR or the Department of Health and Social Care.
The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Copyright: © 2021 Matvienko-Sikar K et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

How to cite: Matvienko-Sikar K, Avery K, Blazeby J et al. Uptake of core outcome sets by clinical trialists publishing in major medical journals: Protocol [version 2; peer review: 2 approved]. HRB Open Res 2021, 3:53 (https://doi.org/10.12688/hrbopenres.13109.2) First published: 10 Aug 2020, 3:53 (https://doi.org/10.12688/hrbopenres.13109.1) Latest published: 25 Jan 2021, 3:53 (https://doi.org/10.12688/hrbopenres.13109.2)

Revised Amendments from Version 1

Information on how the same size estimate was derived (“based on an initial scoping review”) has been added.

The manuscript has been amended to read pharmacologic/drug trials rather than pharmacologic trials.

Details of how we will extract data from protocols has been clarified (Information on the date of registration of the trial and/or publication of the trial protocol will be identified, with the earliest date retained where there is more than one registration/protocol) and (Data will be extracted from the trial protocol or trial registry on whether a COS was mentioned).

Details about checking the COMET database have clarified (Checking the COMET database will be done independently by paired reviewers, with each pair identifying COS for a subset of trials. At least one member of each pair will have prior experience of identifying COS for specific populations and interventions. Agreement on identified COS will be reached by discussion between reviewer pairs and will be verified by the first and senior authors).

Details of ethical approval are included (Ethical approval for this survey has been obtained from the University College Cork Social Research Ethics Committee (2020-137).

Additional trial designs eligible for inclusion (adaptive, and stepped-wedge designs) are included.

A clarification of study exclusions now added (Systematic reviews and other study designs will also be excluded).

Information on survey processes included (To maximise survey response rate, survey invitation emails to trialists will be personalised and will be sent from the COMET Initiative and the Medical Research Council (MRC) Trials Methodology Research Partnership. A reminder email will also be sent to all trialists after one week).

More information about question types now included (The survey includes closed ended questions ... One open-ended question at the end of the survey asks participants if there is anything additional they wish to share).

See the authors' detailed response to the review by Christine Bond
See the authors' detailed response to the review by Anne Spinewine

Introduction

Core outcome sets (COS) are standardised sets of agreed upon outcomes that should be measured and reported in all trials of a particular health area¹. COS represent the minimum outcomes that should be measured and reported to facilitate standardisation and to improve the examination of intervention effectiveness^1,2. Heterogeneity in outcomes across trials has been noted as a significant problem in terms of evidence syntheses because not all outcomes can be compared across trials³. Selective outcome reporting, or outcome reporting bias, which relates to inclusion of a subset of originally measured outcomes in the final publication is also problematic as it reduces research validity of trials and contributes to outcome heterogeneity⁴. Outcome heterogeneity and selective reporting limit transparent examinations of intervention effects and contribute to research waste⁵. In addition, outcomes included in trials do not always reflect those outcomes that are of importance to patients and other key stakeholders, such as healthcare professionals and policy makers^6,7. COS represent an approach to minimising these problems by providing a minimum outcome set, that has been agreed by consensus by key stakeholders¹.

Development and use of COS is supported by the COMET Initiative, and resources including a COS handbook¹ and development and reporting guidelines^8–10 are available. The development and use of COS in trials are increasing over time; the most recent update to a systematic review of COS development reported that over 300 COS studies have been published up to 2019, and at least 200 are currently being developed¹¹. Reviews of COS uptake indicate varying, though typically low, rates of COS use in trials¹². Use of COS to inform outcome choice in systematic reviews has been reported as just 7% in a recent 2020 review of 100 Cochrane reviews (Williamson et al. under review). Similarly, a recent examination of primary research applications to the National Institute for Health Research Health Technology Assessment (NIHR HTA) reported that 19% of applicants referenced a COS in relation to outcome choice¹². In this study, applicants reported that patient and public opinion, outcomes used in other studies, and recommendations from funders and/or professional bodies influenced outcome choice in funding applications¹². Though research has been conducted on uptake of individual COS, and COS in specific health areas¹³, data on the use of COS in a general unselected cohort of published trials is lacking. Examinations of trialists’ views and perceived barriers and facilitators to using COS in trials are similarly lacking. This information is of importance to inform strategies to increase awareness and implementation of COS.

The aims of this study are to examine: (1) current practices of later phase trials published in top medical journals, in relation to the use of COS in choosing trial outcomes; and (2) views of trial authors on the use of COS in relation to choosing trial outcomes.

Methods

Search strategy

We will examine late phase trials published in the following journals: New England Journal of Medicine, Journal of the American Medical Association (JAMA), Lancet, BMJ, and Annals of Internal Medicine. Each journal website will be searched across a 6-month period, from October 2019 to March 2020. Based on initial scoping of the target journals it is estimated that 115 trials, of various phases, have been published in these journals during this timeframe and so this period has been chosen to ensure identification of a sufficient, yet pragmatically manageable number of recent trials for review by the review team. In addition, this time frame ensures a sample of pre-COVID-19 trials (COVID-19 trials are being examined in a separate project in collaboration with https://covid-evidence.org).

Inclusion/exclusion criteria

Late phase trials will be eligible for inclusion. For this study, late phase trials are defined as studies examining effectiveness of an intervention (pharmacological or otherwise), typically in relation to standard care or another comparator. In pharmacologic trials, these are typically referred to as phase III or phase IV clinical trials, though we are cognisant that this classification is not typically used in non-drug trials. In this study, late phase trials can include various trial designs (e.g. parallel, crossover, factorial, adaptive, and stepped-wedge designs) and with any level of randomisation (e.g. individual and cluster levels). There are no restrictions based on sample size, topic/health area, or intervention type. Trials will not be included if they are: feasibility trials aiming to examine whether some aspect of the trial or intervention can be done¹⁴; pilot and exploratory trials preparing the conduct of the future trial, or part of the future trial, on a smaller scale¹⁴; follow-up studies; or secondary analyses of late phase trials, or phase 1 and phase 2 studies. Systematic reviews and other study designs will also be excluded.

Screening

Two reviewers (KMS & VS) will independently screen all identified trial publications to determine whether they are late phase trials meeting eligibility criteria for inclusion in the review. Discrepancies between reviewers will be resolved by consensus discussion or by recourse to a third reviewer (PW).

Data extraction and analysis

A standardised data extraction form will be used for all articles, with data extracted by one reviewer and verified by a second reviewer. Extracted data will include: author, date, title, funding information including location of funder, study aims, disease or health category (using the COMET categories), disease name, target population, type of intervention used. Data will be extracted from the trial protocol or trial registry on whether a COS was mentioned. Data will also be extracted on whether a COS was mentioned and the reason for which it was mentioned (e.g. mentioned because it was used in trial, or mentioned to support a discussion point); whether a COS was used and if so, whether the full COS was used or whether only some COS outcomes were used. Details of the COS used will be extracted, including the individual outcomes used in trial that do not use the full COS. Whether the primary outcome of the trial was a COS, and if so which one, will also be extracted. For trials not reporting use of a COS, the trial authors’ rationale and justification for the choice of outcomes used will be extracted from the published trial if reported.

In addition, for trials not reporting COS use, we will examine whether a COS existed that could have been used at the time of trial commencement to determine trial outcomes. This will be done by first searching for a published protocol or trial registry entry (e.g. in ClinicalTrials.gov, ISRCTN registry) to identify an indication of when the trial started. As trials may begin prior to protocol publication/registration, this will be taken into account by extracting information on the start of trial recruitment from either the registry entry or the trial publication. The COMET database will then be searched by disease and health categories to identify whether a potentially relevant COS could have been used for each trial based on the timing of trial commencement and the timing of COS publication. Where a published protocol or trial registry entry cannot be identified to establish when the trial was being designed, the COMET database will be searched for a COS of relevant scope that had been published by 2017, such that it could potentially have informed choice of outcomes for the trial. We will check this assumption with the trialists (see below). A COS will be considered to be of relevant scope if it was developed for the same population (or a broader subset within which the trial population sits) and/or for the same intervention type (or a broader subset within which the trial intervention sits). Checking the COMET database will be done by one reviewer with prior experience of identifying COS for specific populations and interventions, and will be verified by a second reviewer.

Survey of trialists

A survey will be sent to all corresponding authors of identified trials. When senior/corresponding authors cannot be contacted via emails, another author from the author list (i.e. first or last author) will be approached. The survey will examine trialists’ awareness of, and decisions to search for and use, a COS. An email will be sent to all trialists, including a link to the online survey, hosted on Google Forms® (see extended data¹⁵). One of four versions of the survey will be sent as follows: 1) where trial publications mentioned a COS and the full COS was used; 2) where trial publications mentioned a COS and some but not all COS outcomes were used; 3) where trial publications do no mention a COS and we identified a potentially relevant COS that could have been used; and 4) where trial publications do no mention a COS and we did not identify a potentially relevant COS that could have been used. The surveys will ask about trialists’ identification and use of a COS, or not; experiences and issues with COS use where a COS was used; and reasons for choice of outcomes where a COS was not used.

Analysis

Data collected from review of identified eligible trials and the survey of trialists will be analysed and presented descriptively. The main outcomes of this study will be the numbers and percentage of trials using a COS and the numbers and percentage of trials that could have used a COS. Secondary outcomes are trialists’ awareness of, and decisions to search for and use, a COS. Open-ended survey questions will be analysed used content analysis. Findings will be presented narratively and in tabular format.

Ethical considerations and consent

Ethical approval is not necessary for examination of the published trials but is required for, and will be obtained prior to commencement of, the trialist survey. All participants will receive an electronic information leaflet and, following reading this, will provide electronic consent prior to completing the online survey. While it is not anticipated that the survey will cause any distress, the researchers contact details will be provided at the end of the survey should participants wish to discuss any issues raised or be provided with further support contact details.

Dissemination

The findings of this study will be disseminated through the publication of peer-reviewed manuscripts. Additionally, findings will be presented at both national and international conferences.

Study status

This study has not yet commenced.

Discussion

Use of COS in trials is important to improve standardisation of outcomes, reduce bias and research waste, and improve examination and understanding of the effects of interventions in particular health areas. This study will provide information on the proportion of trialists in major medical journals who currently are, or are not, using COS. These findings will provide important insight into current uptake of COS in trials published in major medical journals. In addition, the study will provide information on trialists’ views and reasons for using, or not using, COS in trials. This is essential to better understand barriers and facilitators to COS uptake in medical trials.

Data availability

Underlying data

No data are associated with this article

Extended data

Open Science Framework: Uptake of core outcome sets by clinical trialists publishing in major medical journals. https://doi.org/10.17605/OSF.IO/H4EKV¹⁵

- COS TMRP Survey.pdf (Four versions of survey to be used in study)
- Uptake of COS by clinical trialists publishing in major medical journals.pdf (full study protocol document)

Data are available under the terms of the Creative Commons Attribution 4.0 International license (CC-BY 4.0).

Faculty Opinions recommended

References

1. Williamson PR, Altman DG, Bagley H, et al.: The COMET Handbook: version 1.0. Trials. 2017; 18(Suppl 3): 280. PubMed Abstract | Publisher Full Text | Free Full Text
2. Williamson P, Altman D, Blazeby J, et al.: Driving up the Quality and Relevance of Research Through the Use of Agreed Core Outcomes. J Health Serv Res Policy. 2012; 17(1): 1–2. PubMed Abstract | Publisher Full Text
3. Clarke M, Williamson PR: Core outcome sets and systematic reviews. Syst Rev. 2016; 5: 11. PubMed Abstract | Publisher Full Text | Free Full Text
4. Kirkham JJ, Dwan KM, Altman DG, et al.: The impact of outcome reporting bias in randomised controlled trials on a cohort of systematic reviews. BMJ. 2010; 340: c365. PubMed Abstract | Publisher Full Text
5. Glasziou P, Altman DG, Bossuyt P, et al.: Reducing waste from incomplete or unusable reports of biomedical research. Lancet. 2014; 383(9913): 267–276. PubMed Abstract | Publisher Full Text
6. Biggane AM, Brading L, Ravaud P, et al.: Survey indicated that core outcome set development is increasingly including patients, being conducted internationally and using Delphi surveys. Trials. 2018; 19(1): 113. PubMed Abstract | Publisher Full Text | Free Full Text
7. Chalmers I, Bracken MB, Djulbegovic B, et al.: How to increase value and reduce waste when research priorities are set. Lancet. 2014; 383(9912): 156–165. PubMed Abstract | Publisher Full Text
8. Kirkham JJ, Gorst S, Altman DG, et al.: Core Outcome Set-STAndardised Protocol Items: the COS-STAP Statement. Trials. 2019; 20(1): 116. PubMed Abstract | Publisher Full Text | Free Full Text
9. Kirkham JJ, Gorst S, Altman DG, et al.: Core Outcome Set–STAndards for Reporting: The COS-STAR Statement. PLoS Med. 2016; 13(10): e1002148. PubMed Abstract | Publisher Full Text | Free Full Text
10. Kirkham JJ, Davis K, Altman DG, et al.: Core Outcome Set-STAndards for Development: The COS-STAD recommendations. PLoS Med. 2017; 14(11): e1002447. PubMed Abstract | Publisher Full Text | Free Full Text
11. Gargon E, Gorst SL, Williamson PR: Choosing important health outcomes for comparative effectiveness research: 5th annual update to a systematic review of core outcome sets for research. PLoS One. 2019; 14(12): e0225980. PubMed Abstract | Publisher Full Text | Free Full Text
12. Hughes KL, Kirkham JJ, Clarke M, et al.: Assessing the impact of a research funder’s recommendation to consider core outcome sets. PLoS One. 2019; 14(9): e0222418. PubMed Abstract | Publisher Full Text | Free Full Text
13. Kirkham JJ, Clarke M, Williamson PR: A methodological approach for assessing the uptake of core outcome sets using ClinicalTrials.gov: findings from a review of randomised controlled trials of rheumatoid arthritis. BMJ. 2017; 357: j2262. PubMed Abstract | Publisher Full Text | Free Full Text
14. Eldridge SM, Lancaster GA, Campbell MJ, et al.: Defining Feasibility and Pilot Studies in Preparation for Randomised Controlled Trials: Development of a Conceptual Framework. PLoS One. 2016; 11(3): e0150205. PubMed Abstract | Publisher Full Text | Free Full Text
15. Matvienko-Sikar K: Uptake of core outcome sets by clinical trialists publishing in major medical journals. 2020. http://www.doi.org/10.17605/OSF.IO/H4EKV

Comments on this article Comments (0)

Version 2

VERSION 2 PUBLISHED 10 Aug 2020

Author details Author details

Karen Matvienko-Sikar
Roles: Conceptualization, Methodology, Writing – Original Draft Preparation, Writing – Review & Editing

Kerry Avery
Roles: Methodology, Writing – Review & Editing

Jane Blazeby
Roles: Methodology, Writing – Review & Editing

Karen Hughes
Roles: Methodology, Writing – Review & Editing

Pamela Jacobsen
Roles: Methodology, Writing – Review & Editing

Jamie Kirkham
Roles: Methodology, Writing – Review & Editing

Jan Kottner
Roles: Methodology, Writing – Review & Editing

Katie Mellor
Roles: Methodology, Writing – Review & Editing

Ian J Saldanha
Roles: Methodology, Writing – Review & Editing

Valerie Smith
Roles: Methodology, Writing – Review & Editing

Caroline B. Terwee
Roles: Methodology, Writing – Review & Editing

Paula R. Williamson
Roles: Conceptualization, Methodology, Writing – Original Draft Preparation, Writing – Review & Editing

Competing interests

No competing interests were disclosed.

Grant information

Health Research Board Ireland [HRB-ARPP-A-011]

KMS is supported by a Health Research Board Applying Research into Policy and Practice Fellowship [HRB-ARPP-A-011]. KA and JB are supported by the NIHR Biomedical Research Centre at University Hospitals Bristol and Weston NHS Foundation Trust and the University of Bristol. The views expressed are those of the author(s) and not necessarily those of the NIHR or the Department of Health and Social Care.
The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Article Versions (2)

version 2

Revised

Published: 25 Jan 2021, 3:53

https://doi.org/10.12688/hrbopenres.13109.2

version 1

Published: 10 Aug 2020, 3:53

https://doi.org/10.12688/hrbopenres.13109.1

© 2021 Matvienko-Sikar K et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Download

Export To

metrics

VIEWS

700

downloads

Citations

SEE MORE DETAILS

CITE

how to cite this article

Matvienko-Sikar K, Avery K, Blazeby J et al. Uptake of core outcome sets by clinical trialists publishing in major medical journals: Protocol [version 2; peer review: 2 approved]. HRB Open Res 2021, 3:53 (https://doi.org/10.12688/hrbopenres.13109.2)

NOTE: If applicable, it is important to ensure the information in square brackets after the title is included in all citations of this article.

track

receive updates on this article

Track an article to receive email alerts on any updates to this article.

Open Peer Review

Version 2

VERSION 2

PUBLISHED 25 Jan 2021

Revised

Views

Reviewer Report 26 Feb 2021

Christine Bond, Institute of Applied Health Sciences, School of Medicine, Medical Sciences & Nutrition, University of Aberdeen, Aberdeen, UK

Approved

https://doi.org/10.21956/hrbopenres.14343.r28748

CITE

Report a concern

Respond or Comment

Version 1

VERSION 1

PUBLISHED 10 Aug 2020

Views

Reviewer Report 21 Oct 2020

Anne Spinewine, Louvain Drug Research Institute, Clinical Pharmacy Research Group, Université catholique de Louvain, Brussels, Belgium; Pharmacy Department, CHU UCL Namur, Namur, Belgium

Approved

https://doi.org/10.21956/hrbopenres.14215.r28034

This is an interesting and well written protocol for a study evaluating the uptake of COS and exploring reasons for (not) using COS.
The rationale is clearly justified, the methodological approach is sound and clearly written, the discussion is ... Continue reading

The choice of the journals selected could be better described.
Inclusion/exclusion criteria: would you also include trials with slightly different designs such as: stepped-wedge trials; adaptive trials?
I suppose that SRs of trials would be excluded – although it would also be interesting to look if the use of COS is also addressed. Exclusion could be specifically mentioned.
Data extraction: if the use of a COS is not mentioned in the main paper, would it be safe to check the detailed protocol (in some areas, it might have been previously published in another journal) if no reference is made to an existing COS.
The search of existing COS is made in the COMET database. Is there any data to confirm that this is sufficient, i.e. that searching in parallel another database (eg PubMed) for published COS is not needed?
Questionnaire: how will you maximize response rate? Has the questionnaire been pilot-tested? Will there be close and open-ended questions?
The trialists’ views is essential to better understand the barriers and facilitators. Could there be any value of looking at some of the data extracted in the first part of the study, and identify from there factors associated with the (non)use of a COS? E.g. domain, type of funding, patient and public involvement…?

Is the rationale for, and objectives of, the study clearly described?

Yes
Is the study design appropriate for the research question?

Yes
Are sufficient details of the methods provided to allow replication by others?

Yes
Are the datasets clearly presented in a useable and accessible format?

Not applicable

Competing Interests: No competing interests were disclosed.

Reviewer Expertise: Use of medicines in older people. I have previous experience in developing a COS.

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

CITE

Report a concern

Author Response 25 Jan 2021

Karen Matvienko-Sikar, School of Public Health, University College Cork, Cork, Ireland

25 Jan 2021

Author Response

Reviewer comment 1. This is an interesting and well written protocol for a study evaluating the uptake of COS and exploring reasons for (not) using COS.
The rationale is clearly ... Continue reading Reviewer comment 1. This is an interesting and well written protocol for a study evaluating the uptake of COS and exploring reasons for (not) using COS.
The rationale is clearly justified, the methodological approach is sound and clearly written, the discussion is adequate.
A few points for consideration are:
The choice of the journals selected could be better described.
Author response 1. Thank you for your comments and useful feedback, which has now been incorporated.

Reviewer comment 2. Inclusion/exclusion criteria: would you also include trials with slightly different designs such as: stepped-wedge trials; adaptive trials?

Author response 2. Yes, trials of any design that meet the study definition of a late phase trial are eligible for inclusion; the additional examples you provided have now been included to the example list also. (New text in bold).
late phase trials can include various trial designs (e.g. parallel, crossover, factorial, adaptive, and stepped-wedge designs)

Reviewer comment 3. I suppose that SRs of trials would be excluded – although it would also be interesting to look if the use of COS is also addressed. Exclusion could be specifically mentioned.

Author response 3. There is on-going work examining uptake of COS in systematic reviews, and it is correct that systematic reviews are not eligible for inclusion in this review. This has been clarified in the methods section as follows:
Systematic reviews and other study designs will also be excluded.

Reviewer comment 4. Data extraction: if the use of a COS is not mentioned in the main paper, would it be safe to check the detailed protocol (in some areas, it might have been previously published in another journal) if no reference is made to an existing COS.

Author response 4. Data will now also be extracted from the trial protocol and/or registry on whether a COS was mentioned. This has ben added to the methods section as follows: Data will be extracted from the trial protocol or trial registry on whether a COS was mentioned.

Reviewer comment 5. The search of existing COS is made in the COMET database. Is there any data to confirm that this is sufficient, i.e. that searching in parallel another database (eg PubMed) for published COS is not needed?

Author response 5. The COMET database is a repository of all completed, on-going, and planned COS. As such, we are confident that it is sufficient to search the COMET database only.

Reviewer comment 6. Questionnaire: how will you maximize response rate? Has the questionnaire been pilot-tested? Will there be close and open-ended questions?

Author response 6. To maximise survey response rates, survey invitation emails to trialists will be personalised and will be sent from the COMET Initiative and the Medical Research Council (MRC) Trials Methodology Research Partnership, as these are recognised trial methodology initiatives and so may enhance trust in, and motivation to the complete, the survey. A reminder email will also be sent to all trialists after one week. This information has been included in the protocol as follows:
To maximise survey response rate, survey invitation emails to trialists will be personalised and will be sent from the COMET Initiative and the Medical Research Council (MRC) Trials Methodology Research Partnership. A reminder email will also be sent to all trialists after one week.

The survey contains closed ended questions, with the exception of a final open-ended question asking trialists if there is anything else they wish to share. This has been clarified in the protocol :
The survey includes closed ended questions asking about trialists’ awareness of, and decisions to search for and use, a COS. One open-ended question at the end of the survey asks participants if there is anything additional they wish to share.

Reviewer comment 7. The trialists’ views is essential to better understand the barriers and facilitators. Could there be any value of looking at some of the data extracted in the first part of the study, and identify from there factors associated with the (non)use of a COS? E.g. domain, type of funding, patient and public involvement…?

Author response 7. Thank you for this useful suggestion. For the purposes of this review, we have based the survey questions on preliminary data and information emerging from existing and on-going research regarding COS uptake. The findings of the current survey, incorporating both the review data and the survey, will provide a useful guide moving forward to examine and target barriers and facilitators to COS use.
Reviewer comment 1. This is an interesting and well written protocol for a study evaluating the uptake of COS and exploring reasons for (not) using COS.
The rationale is clearly justified, the methodological approach is sound and clearly written, the discussion is adequate.
A few points for consideration are:
The choice of the journals selected could be better described.
Author response 1. Thank you for your comments and useful feedback, which has now been incorporated.

Reviewer comment 2. Inclusion/exclusion criteria: would you also include trials with slightly different designs such as: stepped-wedge trials; adaptive trials?

Author response 2. Yes, trials of any design that meet the study definition of a late phase trial are eligible for inclusion; the additional examples you provided have now been included to the example list also. (New text in bold).
late phase trials can include various trial designs (e.g. parallel, crossover, factorial, adaptive, and stepped-wedge designs)

Reviewer comment 3. I suppose that SRs of trials would be excluded – although it would also be interesting to look if the use of COS is also addressed. Exclusion could be specifically mentioned.

Author response 3. There is on-going work examining uptake of COS in systematic reviews, and it is correct that systematic reviews are not eligible for inclusion in this review. This has been clarified in the methods section as follows:
Systematic reviews and other study designs will also be excluded.

Reviewer comment 4. Data extraction: if the use of a COS is not mentioned in the main paper, would it be safe to check the detailed protocol (in some areas, it might have been previously published in another journal) if no reference is made to an existing COS.

Author response 4. Data will now also be extracted from the trial protocol and/or registry on whether a COS was mentioned. This has ben added to the methods section as follows: Data will be extracted from the trial protocol or trial registry on whether a COS was mentioned.

Reviewer comment 5. The search of existing COS is made in the COMET database. Is there any data to confirm that this is sufficient, i.e. that searching in parallel another database (eg PubMed) for published COS is not needed?

Author response 5. The COMET database is a repository of all completed, on-going, and planned COS. As such, we are confident that it is sufficient to search the COMET database only.

Reviewer comment 6. Questionnaire: how will you maximize response rate? Has the questionnaire been pilot-tested? Will there be close and open-ended questions?

Author response 6. To maximise survey response rates, survey invitation emails to trialists will be personalised and will be sent from the COMET Initiative and the Medical Research Council (MRC) Trials Methodology Research Partnership, as these are recognised trial methodology initiatives and so may enhance trust in, and motivation to the complete, the survey. A reminder email will also be sent to all trialists after one week. This information has been included in the protocol as follows:
To maximise survey response rate, survey invitation emails to trialists will be personalised and will be sent from the COMET Initiative and the Medical Research Council (MRC) Trials Methodology Research Partnership. A reminder email will also be sent to all trialists after one week.

The survey contains closed ended questions, with the exception of a final open-ended question asking trialists if there is anything else they wish to share. This has been clarified in the protocol :
The survey includes closed ended questions asking about trialists’ awareness of, and decisions to search for and use, a COS. One open-ended question at the end of the survey asks participants if there is anything additional they wish to share.

Reviewer comment 7. The trialists’ views is essential to better understand the barriers and facilitators. Could there be any value of looking at some of the data extracted in the first part of the study, and identify from there factors associated with the (non)use of a COS? E.g. domain, type of funding, patient and public involvement…?

Author response 7. Thank you for this useful suggestion. For the purposes of this review, we have based the survey questions on preliminary data and information emerging from existing and on-going research regarding COS uptake. The findings of the current survey, incorporating both the review data and the survey, will provide a useful guide moving forward to examine and target barriers and facilitators to COS use.
Competing Interests: No competing interests were disclosed. Close
Report a concern
Respond or Comment

COMMENTS ON THIS REPORT

Author Response 25 Jan 2021

Karen Matvienko-Sikar, School of Public Health, University College Cork, Cork, Ireland

25 Jan 2021

Author Response

Reviewer comment 1. This is an interesting and well written protocol for a study evaluating the uptake of COS and exploring reasons for (not) using COS.
The rationale is clearly ... Continue reading Reviewer comment 1. This is an interesting and well written protocol for a study evaluating the uptake of COS and exploring reasons for (not) using COS.
The rationale is clearly justified, the methodological approach is sound and clearly written, the discussion is adequate.
A few points for consideration are:
The choice of the journals selected could be better described.
Author response 1. Thank you for your comments and useful feedback, which has now been incorporated.

Reviewer comment 2. Inclusion/exclusion criteria: would you also include trials with slightly different designs such as: stepped-wedge trials; adaptive trials?

Author response 2. Yes, trials of any design that meet the study definition of a late phase trial are eligible for inclusion; the additional examples you provided have now been included to the example list also. (New text in bold).
late phase trials can include various trial designs (e.g. parallel, crossover, factorial, adaptive, and stepped-wedge designs)

Reviewer comment 3. I suppose that SRs of trials would be excluded – although it would also be interesting to look if the use of COS is also addressed. Exclusion could be specifically mentioned.

Author response 3. There is on-going work examining uptake of COS in systematic reviews, and it is correct that systematic reviews are not eligible for inclusion in this review. This has been clarified in the methods section as follows:
Systematic reviews and other study designs will also be excluded.

Reviewer comment 4. Data extraction: if the use of a COS is not mentioned in the main paper, would it be safe to check the detailed protocol (in some areas, it might have been previously published in another journal) if no reference is made to an existing COS.

Author response 4. Data will now also be extracted from the trial protocol and/or registry on whether a COS was mentioned. This has ben added to the methods section as follows: Data will be extracted from the trial protocol or trial registry on whether a COS was mentioned.

Reviewer comment 5. The search of existing COS is made in the COMET database. Is there any data to confirm that this is sufficient, i.e. that searching in parallel another database (eg PubMed) for published COS is not needed?

Author response 5. The COMET database is a repository of all completed, on-going, and planned COS. As such, we are confident that it is sufficient to search the COMET database only.

Reviewer comment 6. Questionnaire: how will you maximize response rate? Has the questionnaire been pilot-tested? Will there be close and open-ended questions?

Author response 6. To maximise survey response rates, survey invitation emails to trialists will be personalised and will be sent from the COMET Initiative and the Medical Research Council (MRC) Trials Methodology Research Partnership, as these are recognised trial methodology initiatives and so may enhance trust in, and motivation to the complete, the survey. A reminder email will also be sent to all trialists after one week. This information has been included in the protocol as follows:
To maximise survey response rate, survey invitation emails to trialists will be personalised and will be sent from the COMET Initiative and the Medical Research Council (MRC) Trials Methodology Research Partnership. A reminder email will also be sent to all trialists after one week.

The survey contains closed ended questions, with the exception of a final open-ended question asking trialists if there is anything else they wish to share. This has been clarified in the protocol :
The survey includes closed ended questions asking about trialists’ awareness of, and decisions to search for and use, a COS. One open-ended question at the end of the survey asks participants if there is anything additional they wish to share.

Reviewer comment 7. The trialists’ views is essential to better understand the barriers and facilitators. Could there be any value of looking at some of the data extracted in the first part of the study, and identify from there factors associated with the (non)use of a COS? E.g. domain, type of funding, patient and public involvement…?

Author response 7. Thank you for this useful suggestion. For the purposes of this review, we have based the survey questions on preliminary data and information emerging from existing and on-going research regarding COS uptake. The findings of the current survey, incorporating both the review data and the survey, will provide a useful guide moving forward to examine and target barriers and facilitators to COS use.
Reviewer comment 1. This is an interesting and well written protocol for a study evaluating the uptake of COS and exploring reasons for (not) using COS.
The rationale is clearly justified, the methodological approach is sound and clearly written, the discussion is adequate.
A few points for consideration are:
The choice of the journals selected could be better described.
Author response 1. Thank you for your comments and useful feedback, which has now been incorporated.

Reviewer comment 2. Inclusion/exclusion criteria: would you also include trials with slightly different designs such as: stepped-wedge trials; adaptive trials?

Author response 2. Yes, trials of any design that meet the study definition of a late phase trial are eligible for inclusion; the additional examples you provided have now been included to the example list also. (New text in bold).
late phase trials can include various trial designs (e.g. parallel, crossover, factorial, adaptive, and stepped-wedge designs)

Reviewer comment 3. I suppose that SRs of trials would be excluded – although it would also be interesting to look if the use of COS is also addressed. Exclusion could be specifically mentioned.

Author response 3. There is on-going work examining uptake of COS in systematic reviews, and it is correct that systematic reviews are not eligible for inclusion in this review. This has been clarified in the methods section as follows:
Systematic reviews and other study designs will also be excluded.

Reviewer comment 4. Data extraction: if the use of a COS is not mentioned in the main paper, would it be safe to check the detailed protocol (in some areas, it might have been previously published in another journal) if no reference is made to an existing COS.

Author response 4. Data will now also be extracted from the trial protocol and/or registry on whether a COS was mentioned. This has ben added to the methods section as follows: Data will be extracted from the trial protocol or trial registry on whether a COS was mentioned.

Reviewer comment 5. The search of existing COS is made in the COMET database. Is there any data to confirm that this is sufficient, i.e. that searching in parallel another database (eg PubMed) for published COS is not needed?

Author response 5. The COMET database is a repository of all completed, on-going, and planned COS. As such, we are confident that it is sufficient to search the COMET database only.

Reviewer comment 6. Questionnaire: how will you maximize response rate? Has the questionnaire been pilot-tested? Will there be close and open-ended questions?

Author response 6. To maximise survey response rates, survey invitation emails to trialists will be personalised and will be sent from the COMET Initiative and the Medical Research Council (MRC) Trials Methodology Research Partnership, as these are recognised trial methodology initiatives and so may enhance trust in, and motivation to the complete, the survey. A reminder email will also be sent to all trialists after one week. This information has been included in the protocol as follows:
To maximise survey response rate, survey invitation emails to trialists will be personalised and will be sent from the COMET Initiative and the Medical Research Council (MRC) Trials Methodology Research Partnership. A reminder email will also be sent to all trialists after one week.

The survey contains closed ended questions, with the exception of a final open-ended question asking trialists if there is anything else they wish to share. This has been clarified in the protocol :
The survey includes closed ended questions asking about trialists’ awareness of, and decisions to search for and use, a COS. One open-ended question at the end of the survey asks participants if there is anything additional they wish to share.

Reviewer comment 7. The trialists’ views is essential to better understand the barriers and facilitators. Could there be any value of looking at some of the data extracted in the first part of the study, and identify from there factors associated with the (non)use of a COS? E.g. domain, type of funding, patient and public involvement…?

Author response 7. Thank you for this useful suggestion. For the purposes of this review, we have based the survey questions on preliminary data and information emerging from existing and on-going research regarding COS uptake. The findings of the current survey, incorporating both the review data and the survey, will provide a useful guide moving forward to examine and target barriers and facilitators to COS use.
Competing Interests: No competing interests were disclosed. Close
Report a concern

Views

Reviewer Report 30 Sep 2020

Christine Bond, Institute of Applied Health Sciences, School of Medicine, Medical Sciences & Nutrition, University of Aberdeen, Aberdeen, UK

Approved with Reservations

https://doi.org/10.21956/hrbopenres.14215.r27876

This is an interesting well written protocol for a study to explore whether researchers are using core outcome sets and if not why not. The method involves identifying and critically reviewing recently published papers in major journals followed by a survey of authors. The paper explains why this is an important topic and the value of researchers using core outcomes, especially for evidence synthesis. I have just a few comments for the authors to consider.

My main concern is about greater clarity of what this review will add to the existing studies which have shown poor uptake of COS in Cochrane, previous trials and recently funded HTA applications (do these HTA application need more explanation for international audiences?). This review is looking at major impact journals but why is that important or different. Also I agree the selected journals are subjectively important and influential but is there an objective justification for their selection?

Are the authors suggesting Journal Editors should use their influence to encourage use of COS, in the same way as reporting guidelines are required?

In many ways the triallists’ views are the most interesting aspect of the paper as a way of understanding what needs to be done in the future to promote use of COS. Building on that, could behavioural theory be used to inform the questions asked and allow identification of appropriate behaviour change interventions?

Minor points

Is there a justification for the sample size of estimated 115 papers? This relates both to the generalisability of the findings and the value of the survey, especially for any sub sample analyses.
Is ‘pharmacologic trials’ a normal label? Often referred to more as investigational products or drug trials.
Under items to be extracted some e.g. disease name will not necessarily always be relevant.
For trials not reporting COS or part of COS is there a field for specifying the outcome that was actually used as well as any justification, for that decision?
First column page 4 has a longish paragraph on identifying if a relevant COS existed at the time of the trial. There is reference to time of designing the trial as being the base line. I am not sure how easy the date of the initial design would be to identify but one option might be to look at a relevant date of any grant application if the study had been externally funded.
In the same paragraph maybe add initials of the reviewer with prior experience and the second reviewer as presume they are members of the authorship team.
Will any inferential statistics be conducted on the survey results?
Who will be approached for an ethical opinion – presume it will be a University review board?
Who is funding this work?

Is the rationale for, and objectives of, the study clearly described?

Partly
Is the study design appropriate for the research question?

Yes
Are sufficient details of the methods provided to allow replication by others?

Yes
Are the datasets clearly presented in a useable and accessible format?

Not applicable

Competing Interests: No competing interests were disclosed.

Reviewer Expertise: Mixed methods health services researcher who has been part of a team developing a COS.

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.

CITE

Report a concern

Author Response 25 Jan 2021

Karen Matvienko-Sikar, School of Public Health, University College Cork, Cork, Ireland

25 Jan 2021

Author Response

Reviewer comment 1. This is an interesting well written protocol for a study to explore whether researchers are using core outcome sets and if not why not. The method involves ... Continue reading Reviewer comment 1. This is an interesting well written protocol for a study to explore whether researchers are using core outcome sets and if not why not. The method involves identifying and critically reviewing recently published papers in major journals followed by a survey of authors. The paper explains why this is an important topic and the value of researchers using core outcomes, especially for evidence synthesis. I have just a few comments for the authors to consider.

My main concern is about greater clarity of what this review will add to the existing studies which have shown poor uptake of COS in Cochrane, previous trials and recently funded HTA applications (do these HTA application need more explanation for international audiences?). This review is looking at major impact journals but why is that important or different. Also I agree the selected journals are subjectively important and influential but is there an objective justification for their selection?

Author response 1.
Thank you for your comments, the protocol has now been amended taking these into account.
The rationale for examining the use of core outcome sets in major impact journals is that these journals are expected to publish trials of the highest quality, of clinical importance and relevance, and are likely to impact on policy and practice. This is different to Cochrane, which includes every trial, and HTA applications that may be more standardised. As such, this review will provide a different perspective on core outcome set use in trials, in terms of identifying uptake among those trials that are expected to be among those of the highest quality.

Reviewer comment 2. Are the authors suggesting Journal Editors should use their influence to encourage use of COS, in the same way as reporting guidelines are required?

Author response 2. The authors feel that Journal Editors are in a position to encourage and promote use of COS in trials. However, at protocol stage of this project we are not suggesting that Journal Editors use their influence to encourage COS because recommendations about how best to encourage COS use in trials will be derived from the data provided by the trialists in our survey.

Reviewer comment 3. In many ways the triallists’ views are the most interesting aspect of the paper as a way of understanding what needs to be done in the future to promote use of COS. Building on that, could behavioural theory be used to inform the questions asked and allow identification of appropriate behaviour change interventions?

Author response 3. We agree that the trialists survey responses will provide useful insights into how best to promote use of core outcome sets. Approaches such as the behaviour change wheel could be used to determine which behavioural determinants to target moving forward to enhance COS use. This approach was used to inform an earlier qualitative study topic guide examining COS use, which has informed the survey used in the current study.

Reviewer comment 4
Minor points
Is there a justification for the sample size of estimated 115 papers? This relates both to the generalisability of the findings and the value of the survey, especially for any sub sample analyses.

Author response 4. The sample size estimate was derived from initial scoping of the target journals, and is based on a pragmatic decision regarding the number of trials that are manageable to examine within the study timeframe but that are still representative of trials published by these journals. Sub sample analyses are not planned. The following has been added to the methods for clarity (new text in bold):
Based on initial scoping of the target journals, it is estimated that 115 trials, of various phases, have been published during in these journals during this timeframe and so this period has been chosen to ensure identification of a sufficient, yet pragmatically manageable number of recent trials for review by the review team.

Reviewer comment 5. Is ‘pharmacologic trials’ a normal label? Often referred to more as investigational products or drug trials.

Author response 5. The manuscript has been amended to state pharmacologic/drug trials

Reviewer comment 6. Under items to be extracted some e.g. disease name will not necessarily always be relevant.

Author response 6. As the focus of the review is on clinical trials published in major medical journals, it is important that details including ‘disease name’ are extracted. However if during data extraction, where disease name is not applicable for a specific trial the reviewer will specify ‘not applicable’.

Reviewer comment 7. For trials not reporting COS or part of COS is there a field for specifying the outcome that was actually used as well as any justification, for that decision?

Author Response 7. The following is included in the ‘Data extraction and analysis’ section of the protocol to specify data extracted in instances where COS are not used: For trials not reporting use of a COS, or reporting use of only some COS outcomes, the trial authors’ rationale and justification for the choice of outcomes used will be extracted from the published trial if reported.

Reviewer comment 8. First column page 4 has a longish paragraph on identifying if a relevant COS existed at the time of the trial. There is reference to time of designing the trial as being the base line. I am not sure how easy the date of the initial design would be to identify but one option might be to look at a relevant date of any grant application if the study had been externally funded.

Author Response 8. Reference to identifying when the trial started is now removed and replaced with: Information on the date of registration of the trial and/or publication of the trial protocol will be identified, with the earliest date retained where there is more than one registration/protocol.

Reviewer comment 9. In the same paragraph maybe add initials of the reviewer with prior experience and the second reviewer as presume they are members of the authorship team.

Author Response 9. We have now amended the process for COS identification, such that it will be conducted by pairs of reviewers who will work independently to identify COS in the COMET database and reach agreement by discussion. This has been updated below, though initials have not been included due to the on-going nature of this process.
Checking the COMET database will be done independently by paired reviewers, with each pair identifying COS for a subset of trials. At least one member of each pair will have prior experience of identifying COS for specific populations and interventions. Agreement on identified COS will be reached by discussion between reviewer pairs and will be verified by the first and senior authors.

Reviewer comment 10. Will any inferential statistics be conducted on the survey results?

Author Response 10. No the studies will be descriptively analysed to examine the numbers and percentage of trials using a COS, the numbers and percentage of trials that could have used a COS, and trialists’ awareness of, and decisions to search for and use, a COS.

Reviewer comment 11. Who will be approached for an ethical opinion – presume it will be a University review board?

Author response 11. Ethical approval has now been obtained from the University College Cork Social Research Ethics Committee. This information has now been added to the manuscript as follows: Ethical approval for this survey has been obtained from the University College Cork Social Research Ethics Committee (2020-137).

Reviewer comment 12. Who is funding this work?

Author response 12. This work is not being directly funded. Karen Matvienko-Sikar is supported by a Health Research Board Applying Research into Policy and Practice Fellowship [HRB-ARPP-A-011]. Kerry Avery and Jane Blazeby are supported by the NIHR Biomedical Research Centre at University Hospitals Bristol and Weston NHS Foundation Trust and the University of Bristol. Funding support for Paula Williamson has now been included (Paula Williamson is supported by the Medical Research Council (MRC) Trials Methodology Research Partnership (grant reference MR/S014357/1). These funders are not directly funding this project however.
Reviewer comment 1. This is an interesting well written protocol for a study to explore whether researchers are using core outcome sets and if not why not. The method involves identifying and critically reviewing recently published papers in major journals followed by a survey of authors. The paper explains why this is an important topic and the value of researchers using core outcomes, especially for evidence synthesis. I have just a few comments for the authors to consider.

My main concern is about greater clarity of what this review will add to the existing studies which have shown poor uptake of COS in Cochrane, previous trials and recently funded HTA applications (do these HTA application need more explanation for international audiences?). This review is looking at major impact journals but why is that important or different. Also I agree the selected journals are subjectively important and influential but is there an objective justification for their selection?

Author response 1.
Thank you for your comments, the protocol has now been amended taking these into account.
The rationale for examining the use of core outcome sets in major impact journals is that these journals are expected to publish trials of the highest quality, of clinical importance and relevance, and are likely to impact on policy and practice. This is different to Cochrane, which includes every trial, and HTA applications that may be more standardised. As such, this review will provide a different perspective on core outcome set use in trials, in terms of identifying uptake among those trials that are expected to be among those of the highest quality.

Reviewer comment 2. Are the authors suggesting Journal Editors should use their influence to encourage use of COS, in the same way as reporting guidelines are required?

Author response 2. The authors feel that Journal Editors are in a position to encourage and promote use of COS in trials. However, at protocol stage of this project we are not suggesting that Journal Editors use their influence to encourage COS because recommendations about how best to encourage COS use in trials will be derived from the data provided by the trialists in our survey.

Reviewer comment 3. In many ways the triallists’ views are the most interesting aspect of the paper as a way of understanding what needs to be done in the future to promote use of COS. Building on that, could behavioural theory be used to inform the questions asked and allow identification of appropriate behaviour change interventions?

Author response 3. We agree that the trialists survey responses will provide useful insights into how best to promote use of core outcome sets. Approaches such as the behaviour change wheel could be used to determine which behavioural determinants to target moving forward to enhance COS use. This approach was used to inform an earlier qualitative study topic guide examining COS use, which has informed the survey used in the current study.

Reviewer comment 4
Minor points
Is there a justification for the sample size of estimated 115 papers? This relates both to the generalisability of the findings and the value of the survey, especially for any sub sample analyses.

Author response 4. The sample size estimate was derived from initial scoping of the target journals, and is based on a pragmatic decision regarding the number of trials that are manageable to examine within the study timeframe but that are still representative of trials published by these journals. Sub sample analyses are not planned. The following has been added to the methods for clarity (new text in bold):
Based on initial scoping of the target journals, it is estimated that 115 trials, of various phases, have been published during in these journals during this timeframe and so this period has been chosen to ensure identification of a sufficient, yet pragmatically manageable number of recent trials for review by the review team.

Reviewer comment 5. Is ‘pharmacologic trials’ a normal label? Often referred to more as investigational products or drug trials.

Author response 5. The manuscript has been amended to state pharmacologic/drug trials

Reviewer comment 6. Under items to be extracted some e.g. disease name will not necessarily always be relevant.

Author response 6. As the focus of the review is on clinical trials published in major medical journals, it is important that details including ‘disease name’ are extracted. However if during data extraction, where disease name is not applicable for a specific trial the reviewer will specify ‘not applicable’.

Reviewer comment 7. For trials not reporting COS or part of COS is there a field for specifying the outcome that was actually used as well as any justification, for that decision?

Author Response 7. The following is included in the ‘Data extraction and analysis’ section of the protocol to specify data extracted in instances where COS are not used: For trials not reporting use of a COS, or reporting use of only some COS outcomes, the trial authors’ rationale and justification for the choice of outcomes used will be extracted from the published trial if reported.

Reviewer comment 8. First column page 4 has a longish paragraph on identifying if a relevant COS existed at the time of the trial. There is reference to time of designing the trial as being the base line. I am not sure how easy the date of the initial design would be to identify but one option might be to look at a relevant date of any grant application if the study had been externally funded.

Author Response 8. Reference to identifying when the trial started is now removed and replaced with: Information on the date of registration of the trial and/or publication of the trial protocol will be identified, with the earliest date retained where there is more than one registration/protocol.

Reviewer comment 9. In the same paragraph maybe add initials of the reviewer with prior experience and the second reviewer as presume they are members of the authorship team.

Author Response 9. We have now amended the process for COS identification, such that it will be conducted by pairs of reviewers who will work independently to identify COS in the COMET database and reach agreement by discussion. This has been updated below, though initials have not been included due to the on-going nature of this process.
Checking the COMET database will be done independently by paired reviewers, with each pair identifying COS for a subset of trials. At least one member of each pair will have prior experience of identifying COS for specific populations and interventions. Agreement on identified COS will be reached by discussion between reviewer pairs and will be verified by the first and senior authors.

Reviewer comment 10. Will any inferential statistics be conducted on the survey results?

Author Response 10. No the studies will be descriptively analysed to examine the numbers and percentage of trials using a COS, the numbers and percentage of trials that could have used a COS, and trialists’ awareness of, and decisions to search for and use, a COS.

Reviewer comment 11. Who will be approached for an ethical opinion – presume it will be a University review board?

Author response 11. Ethical approval has now been obtained from the University College Cork Social Research Ethics Committee. This information has now been added to the manuscript as follows: Ethical approval for this survey has been obtained from the University College Cork Social Research Ethics Committee (2020-137).

Reviewer comment 12. Who is funding this work?

Author response 12. This work is not being directly funded. Karen Matvienko-Sikar is supported by a Health Research Board Applying Research into Policy and Practice Fellowship [HRB-ARPP-A-011]. Kerry Avery and Jane Blazeby are supported by the NIHR Biomedical Research Centre at University Hospitals Bristol and Weston NHS Foundation Trust and the University of Bristol. Funding support for Paula Williamson has now been included (Paula Williamson is supported by the Medical Research Council (MRC) Trials Methodology Research Partnership (grant reference MR/S014357/1). These funders are not directly funding this project however.
Competing Interests: No competing interests were disclosed. Close
Report a concern
Respond or Comment

COMMENTS ON THIS REPORT

Author Response 25 Jan 2021

Karen Matvienko-Sikar, School of Public Health, University College Cork, Cork, Ireland

25 Jan 2021

Author Response

Reviewer comment 1. This is an interesting well written protocol for a study to explore whether researchers are using core outcome sets and if not why not. The method involves ... Continue reading Reviewer comment 1. This is an interesting well written protocol for a study to explore whether researchers are using core outcome sets and if not why not. The method involves identifying and critically reviewing recently published papers in major journals followed by a survey of authors. The paper explains why this is an important topic and the value of researchers using core outcomes, especially for evidence synthesis. I have just a few comments for the authors to consider.

My main concern is about greater clarity of what this review will add to the existing studies which have shown poor uptake of COS in Cochrane, previous trials and recently funded HTA applications (do these HTA application need more explanation for international audiences?). This review is looking at major impact journals but why is that important or different. Also I agree the selected journals are subjectively important and influential but is there an objective justification for their selection?

Author response 1.
Thank you for your comments, the protocol has now been amended taking these into account.
The rationale for examining the use of core outcome sets in major impact journals is that these journals are expected to publish trials of the highest quality, of clinical importance and relevance, and are likely to impact on policy and practice. This is different to Cochrane, which includes every trial, and HTA applications that may be more standardised. As such, this review will provide a different perspective on core outcome set use in trials, in terms of identifying uptake among those trials that are expected to be among those of the highest quality.

Reviewer comment 2. Are the authors suggesting Journal Editors should use their influence to encourage use of COS, in the same way as reporting guidelines are required?

Author response 2. The authors feel that Journal Editors are in a position to encourage and promote use of COS in trials. However, at protocol stage of this project we are not suggesting that Journal Editors use their influence to encourage COS because recommendations about how best to encourage COS use in trials will be derived from the data provided by the trialists in our survey.

Reviewer comment 3. In many ways the triallists’ views are the most interesting aspect of the paper as a way of understanding what needs to be done in the future to promote use of COS. Building on that, could behavioural theory be used to inform the questions asked and allow identification of appropriate behaviour change interventions?

Author response 3. We agree that the trialists survey responses will provide useful insights into how best to promote use of core outcome sets. Approaches such as the behaviour change wheel could be used to determine which behavioural determinants to target moving forward to enhance COS use. This approach was used to inform an earlier qualitative study topic guide examining COS use, which has informed the survey used in the current study.

Reviewer comment 4
Minor points
Is there a justification for the sample size of estimated 115 papers? This relates both to the generalisability of the findings and the value of the survey, especially for any sub sample analyses.

Author response 4. The sample size estimate was derived from initial scoping of the target journals, and is based on a pragmatic decision regarding the number of trials that are manageable to examine within the study timeframe but that are still representative of trials published by these journals. Sub sample analyses are not planned. The following has been added to the methods for clarity (new text in bold):
Based on initial scoping of the target journals, it is estimated that 115 trials, of various phases, have been published during in these journals during this timeframe and so this period has been chosen to ensure identification of a sufficient, yet pragmatically manageable number of recent trials for review by the review team.

Reviewer comment 5. Is ‘pharmacologic trials’ a normal label? Often referred to more as investigational products or drug trials.

Author response 5. The manuscript has been amended to state pharmacologic/drug trials

Reviewer comment 6. Under items to be extracted some e.g. disease name will not necessarily always be relevant.

Author response 6. As the focus of the review is on clinical trials published in major medical journals, it is important that details including ‘disease name’ are extracted. However if during data extraction, where disease name is not applicable for a specific trial the reviewer will specify ‘not applicable’.

Reviewer comment 7. For trials not reporting COS or part of COS is there a field for specifying the outcome that was actually used as well as any justification, for that decision?

Author Response 7. The following is included in the ‘Data extraction and analysis’ section of the protocol to specify data extracted in instances where COS are not used: For trials not reporting use of a COS, or reporting use of only some COS outcomes, the trial authors’ rationale and justification for the choice of outcomes used will be extracted from the published trial if reported.

Reviewer comment 8. First column page 4 has a longish paragraph on identifying if a relevant COS existed at the time of the trial. There is reference to time of designing the trial as being the base line. I am not sure how easy the date of the initial design would be to identify but one option might be to look at a relevant date of any grant application if the study had been externally funded.

Author Response 8. Reference to identifying when the trial started is now removed and replaced with: Information on the date of registration of the trial and/or publication of the trial protocol will be identified, with the earliest date retained where there is more than one registration/protocol.

Reviewer comment 9. In the same paragraph maybe add initials of the reviewer with prior experience and the second reviewer as presume they are members of the authorship team.

Author Response 9. We have now amended the process for COS identification, such that it will be conducted by pairs of reviewers who will work independently to identify COS in the COMET database and reach agreement by discussion. This has been updated below, though initials have not been included due to the on-going nature of this process.
Checking the COMET database will be done independently by paired reviewers, with each pair identifying COS for a subset of trials. At least one member of each pair will have prior experience of identifying COS for specific populations and interventions. Agreement on identified COS will be reached by discussion between reviewer pairs and will be verified by the first and senior authors.

Reviewer comment 10. Will any inferential statistics be conducted on the survey results?

Author Response 10. No the studies will be descriptively analysed to examine the numbers and percentage of trials using a COS, the numbers and percentage of trials that could have used a COS, and trialists’ awareness of, and decisions to search for and use, a COS.

Reviewer comment 11. Who will be approached for an ethical opinion – presume it will be a University review board?

Author response 11. Ethical approval has now been obtained from the University College Cork Social Research Ethics Committee. This information has now been added to the manuscript as follows: Ethical approval for this survey has been obtained from the University College Cork Social Research Ethics Committee (2020-137).

Reviewer comment 12. Who is funding this work?

Author response 12. This work is not being directly funded. Karen Matvienko-Sikar is supported by a Health Research Board Applying Research into Policy and Practice Fellowship [HRB-ARPP-A-011]. Kerry Avery and Jane Blazeby are supported by the NIHR Biomedical Research Centre at University Hospitals Bristol and Weston NHS Foundation Trust and the University of Bristol. Funding support for Paula Williamson has now been included (Paula Williamson is supported by the Medical Research Council (MRC) Trials Methodology Research Partnership (grant reference MR/S014357/1). These funders are not directly funding this project however.
Reviewer comment 1. This is an interesting well written protocol for a study to explore whether researchers are using core outcome sets and if not why not. The method involves identifying and critically reviewing recently published papers in major journals followed by a survey of authors. The paper explains why this is an important topic and the value of researchers using core outcomes, especially for evidence synthesis. I have just a few comments for the authors to consider.

My main concern is about greater clarity of what this review will add to the existing studies which have shown poor uptake of COS in Cochrane, previous trials and recently funded HTA applications (do these HTA application need more explanation for international audiences?). This review is looking at major impact journals but why is that important or different. Also I agree the selected journals are subjectively important and influential but is there an objective justification for their selection?

Author response 1.
Thank you for your comments, the protocol has now been amended taking these into account.
The rationale for examining the use of core outcome sets in major impact journals is that these journals are expected to publish trials of the highest quality, of clinical importance and relevance, and are likely to impact on policy and practice. This is different to Cochrane, which includes every trial, and HTA applications that may be more standardised. As such, this review will provide a different perspective on core outcome set use in trials, in terms of identifying uptake among those trials that are expected to be among those of the highest quality.

Reviewer comment 2. Are the authors suggesting Journal Editors should use their influence to encourage use of COS, in the same way as reporting guidelines are required?

Author response 2. The authors feel that Journal Editors are in a position to encourage and promote use of COS in trials. However, at protocol stage of this project we are not suggesting that Journal Editors use their influence to encourage COS because recommendations about how best to encourage COS use in trials will be derived from the data provided by the trialists in our survey.

Reviewer comment 3. In many ways the triallists’ views are the most interesting aspect of the paper as a way of understanding what needs to be done in the future to promote use of COS. Building on that, could behavioural theory be used to inform the questions asked and allow identification of appropriate behaviour change interventions?

Author response 3. We agree that the trialists survey responses will provide useful insights into how best to promote use of core outcome sets. Approaches such as the behaviour change wheel could be used to determine which behavioural determinants to target moving forward to enhance COS use. This approach was used to inform an earlier qualitative study topic guide examining COS use, which has informed the survey used in the current study.

Reviewer comment 4
Minor points
Is there a justification for the sample size of estimated 115 papers? This relates both to the generalisability of the findings and the value of the survey, especially for any sub sample analyses.

Author response 4. The sample size estimate was derived from initial scoping of the target journals, and is based on a pragmatic decision regarding the number of trials that are manageable to examine within the study timeframe but that are still representative of trials published by these journals. Sub sample analyses are not planned. The following has been added to the methods for clarity (new text in bold):
Based on initial scoping of the target journals, it is estimated that 115 trials, of various phases, have been published during in these journals during this timeframe and so this period has been chosen to ensure identification of a sufficient, yet pragmatically manageable number of recent trials for review by the review team.

Reviewer comment 5. Is ‘pharmacologic trials’ a normal label? Often referred to more as investigational products or drug trials.

Author response 5. The manuscript has been amended to state pharmacologic/drug trials

Reviewer comment 6. Under items to be extracted some e.g. disease name will not necessarily always be relevant.

Author response 6. As the focus of the review is on clinical trials published in major medical journals, it is important that details including ‘disease name’ are extracted. However if during data extraction, where disease name is not applicable for a specific trial the reviewer will specify ‘not applicable’.

Reviewer comment 7. For trials not reporting COS or part of COS is there a field for specifying the outcome that was actually used as well as any justification, for that decision?

Author Response 7. The following is included in the ‘Data extraction and analysis’ section of the protocol to specify data extracted in instances where COS are not used: For trials not reporting use of a COS, or reporting use of only some COS outcomes, the trial authors’ rationale and justification for the choice of outcomes used will be extracted from the published trial if reported.

Reviewer comment 8. First column page 4 has a longish paragraph on identifying if a relevant COS existed at the time of the trial. There is reference to time of designing the trial as being the base line. I am not sure how easy the date of the initial design would be to identify but one option might be to look at a relevant date of any grant application if the study had been externally funded.

Author Response 8. Reference to identifying when the trial started is now removed and replaced with: Information on the date of registration of the trial and/or publication of the trial protocol will be identified, with the earliest date retained where there is more than one registration/protocol.

Reviewer comment 9. In the same paragraph maybe add initials of the reviewer with prior experience and the second reviewer as presume they are members of the authorship team.

Author Response 9. We have now amended the process for COS identification, such that it will be conducted by pairs of reviewers who will work independently to identify COS in the COMET database and reach agreement by discussion. This has been updated below, though initials have not been included due to the on-going nature of this process.
Checking the COMET database will be done independently by paired reviewers, with each pair identifying COS for a subset of trials. At least one member of each pair will have prior experience of identifying COS for specific populations and interventions. Agreement on identified COS will be reached by discussion between reviewer pairs and will be verified by the first and senior authors.

Reviewer comment 10. Will any inferential statistics be conducted on the survey results?

Author Response 10. No the studies will be descriptively analysed to examine the numbers and percentage of trials using a COS, the numbers and percentage of trials that could have used a COS, and trialists’ awareness of, and decisions to search for and use, a COS.

Reviewer comment 11. Who will be approached for an ethical opinion – presume it will be a University review board?

Author response 11. Ethical approval has now been obtained from the University College Cork Social Research Ethics Committee. This information has now been added to the manuscript as follows: Ethical approval for this survey has been obtained from the University College Cork Social Research Ethics Committee (2020-137).

Reviewer comment 12. Who is funding this work?

Author response 12. This work is not being directly funded. Karen Matvienko-Sikar is supported by a Health Research Board Applying Research into Policy and Practice Fellowship [HRB-ARPP-A-011]. Kerry Avery and Jane Blazeby are supported by the NIHR Biomedical Research Centre at University Hospitals Bristol and Weston NHS Foundation Trust and the University of Bristol. Funding support for Paula Williamson has now been included (Paula Williamson is supported by the Medical Research Council (MRC) Trials Methodology Research Partnership (grant reference MR/S014357/1). These funders are not directly funding this project however.
Competing Interests: No competing interests were disclosed. Close
Report a concern

Comments on this article Comments (0)

Version 2

VERSION 2 PUBLISHED 10 Aug 2020

Open Peer Review

Reviewer Status

Reviewer Reports

	Invited Reviewers
	1	2
Version 2 (revision) 25 Jan 21	read
Version 1 10 Aug 20	read	read

Christine Bond, University of Aberdeen, Aberdeen, UK
Anne Spinewine, Université catholique de Louvain, Brussels, Belgium; CHU UCL Namur, Namur, Belgium

Comments on this article

All Comments(0)

Add a comment

Back to all reports

Reviewer Report

10 Views

26 Feb 2021 | for Version 2

Christine Bond, Institute of Applied Health Sciences, School of Medicine, Medical Sciences & Nutrition, University of Aberdeen, Aberdeen, UK

10 Views Cite this report Responses(0)

Approved

The authors have responded thoroughly and satisfactorily to all my comments. However their response does not always say if they also amended the actual paper. It looks as though my first comment on what this adds to the Cochrane review has not been explicitly added although it is implied. Similarly I don’t think anything has been added to take account of my second and third comments. Other than that all my comments seem to have been addressed by changes to the paper.

Competing Interests

Since my first review I am on a Steering committee for another project involving at least one of these authors.

Reviewer Expertise

Mixed methods health services researcher who has been part of a team developing a COS.

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

Respond to this report

Responses (0)

Back to all reports

Reviewer Report

15 Views

21 Oct 2020 | for Version 1

Anne Spinewine, Louvain Drug Research Institute, Clinical Pharmacy Research Group, Université catholique de Louvain, Brussels, Belgium; Pharmacy Department, CHU UCL Namur, Namur, Belgium

15 Views Cite this report Responses(1)

Approved

The choice of the journals selected could be better described.
Inclusion/exclusion criteria: would you also include trials with slightly different designs such as: stepped-wedge trials; adaptive trials?
I suppose that SRs of trials would be excluded – although it would also be interesting to look if the use of COS is also addressed. Exclusion could be specifically mentioned.
Data extraction: if the use of a COS is not mentioned in the main paper, would it be safe to check the detailed protocol (in some areas, it might have been previously published in another journal) if no reference is made to an existing COS.
The search of existing COS is made in the COMET database. Is there any data to confirm that this is sufficient, i.e. that searching in parallel another database (eg PubMed) for published COS is not needed?
Questionnaire: how will you maximize response rate? Has the questionnaire been pilot-tested? Will there be close and open-ended questions?
The trialists’ views is essential to better understand the barriers and facilitators. Could there be any value of looking at some of the data extracted in the first part of the study, and identify from there factors associated with the (non)use of a COS? E.g. domain, type of funding, patient and public involvement…?

Is the rationale for, and objectives of, the study clearly described?

Yes
Is the study design appropriate for the research question?

Yes
Are sufficient details of the methods provided to allow replication by others?

Yes
Are the datasets clearly presented in a useable and accessible format?

Not applicable

Competing Interests

No competing interests were disclosed.

Reviewer Expertise

Use of medicines in older people. I have previous experience in developing a COS.

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

Respond to this report

Responses (1)

Author Response

25 Jan 2021

Karen Matvienko-Sikar, School of Public Health, University College Cork, Cork, Ireland

Reviewer comment 1. This is an interesting and well written protocol for a study evaluating the uptake of COS and exploring reasons for (not) using COS.
The rationale is clearly justified, the methodological approach is sound and clearly written, the discussion is adequate.
A few points for consideration are:
The choice of the journals selected could be better described.
Author response 1. Thank you for your comments and useful feedback, which has now been incorporated.

Reviewer comment 2. Inclusion/exclusion criteria: would you also include trials with slightly different designs such as: stepped-wedge trials; adaptive trials?

Author response 2. Yes, trials of any design that meet the study definition of a late phase trial are eligible for inclusion; the additional examples you provided have now been included to the example list also. (New text in bold).
late phase trials can include various trial designs (e.g. parallel, crossover, factorial, adaptive, and stepped-wedge designs)

Reviewer comment 3. I suppose that SRs of trials would be excluded – although it would also be interesting to look if the use of COS is also addressed. Exclusion could be specifically mentioned.

Author response 3. There is on-going work examining uptake of COS in systematic reviews, and it is correct that systematic reviews are not eligible for inclusion in this review. This has been clarified in the methods section as follows:
Systematic reviews and other study designs will also be excluded.

Reviewer comment 4. Data extraction: if the use of a COS is not mentioned in the main paper, would it be safe to check the detailed protocol (in some areas, it might have been previously published in another journal) if no reference is made to an existing COS.

Author response 4. Data will now also be extracted from the trial protocol and/or registry on whether a COS was mentioned. This has ben added to the methods section as follows: Data will be extracted from the trial protocol or trial registry on whether a COS was mentioned.

Reviewer comment 5. The search of existing COS is made in the COMET database. Is there any data to confirm that this is sufficient, i.e. that searching in parallel another database (eg PubMed) for published COS is not needed?

Author response 5. The COMET database is a repository of all completed, on-going, and planned COS. As such, we are confident that it is sufficient to search the COMET database only.

Reviewer comment 6. Questionnaire: how will you maximize response rate? Has the questionnaire been pilot-tested? Will there be close and open-ended questions?

Author response 6. To maximise survey response rates, survey invitation emails to trialists will be personalised and will be sent from the COMET Initiative and the Medical Research Council (MRC) Trials Methodology Research Partnership, as these are recognised trial methodology initiatives and so may enhance trust in, and motivation to the complete, the survey. A reminder email will also be sent to all trialists after one week. This information has been included in the protocol as follows:
To maximise survey response rate, survey invitation emails to trialists will be personalised and will be sent from the COMET Initiative and the Medical Research Council (MRC) Trials Methodology Research Partnership. A reminder email will also be sent to all trialists after one week.

The survey contains closed ended questions, with the exception of a final open-ended question asking trialists if there is anything else they wish to share. This has been clarified in the protocol :
The survey includes closed ended questions asking about trialists’ awareness of, and decisions to search for and use, a COS. One open-ended question at the end of the survey asks participants if there is anything additional they wish to share.

Reviewer comment 7. The trialists’ views is essential to better understand the barriers and facilitators. Could there be any value of looking at some of the data extracted in the first part of the study, and identify from there factors associated with the (non)use of a COS? E.g. domain, type of funding, patient and public involvement…?

Author response 7. Thank you for this useful suggestion. For the purposes of this review, we have based the survey questions on preliminary data and information emerging from existing and on-going research regarding COS uptake. The findings of the current survey, incorporating both the review data and the survey, will provide a useful guide moving forward to examine and target barriers and facilitators to COS use.

View more View less

Competing Interests

No competing interests were disclosed.

Back to all reports

Reviewer Report

33 Views

30 Sep 2020 | for Version 1

Christine Bond, Institute of Applied Health Sciences, School of Medicine, Medical Sciences & Nutrition, University of Aberdeen, Aberdeen, UK

33 Views Cite this report Responses(1)

Approved With Reservations

Is there a justification for the sample size of estimated 115 papers? This relates both to the generalisability of the findings and the value of the survey, especially for any sub sample analyses.
Is ‘pharmacologic trials’ a normal label? Often referred to more as investigational products or drug trials.
Under items to be extracted some e.g. disease name will not necessarily always be relevant.
For trials not reporting COS or part of COS is there a field for specifying the outcome that was actually used as well as any justification, for that decision?
First column page 4 has a longish paragraph on identifying if a relevant COS existed at the time of the trial. There is reference to time of designing the trial as being the base line. I am not sure how easy the date of the initial design would be to identify but one option might be to look at a relevant date of any grant application if the study had been externally funded.
In the same paragraph maybe add initials of the reviewer with prior experience and the second reviewer as presume they are members of the authorship team.
Will any inferential statistics be conducted on the survey results?
Who will be approached for an ethical opinion – presume it will be a University review board?
Who is funding this work?

Is the rationale for, and objectives of, the study clearly described?

Partly
Is the study design appropriate for the research question?

Yes
Are sufficient details of the methods provided to allow replication by others?

Yes
Are the datasets clearly presented in a useable and accessible format?

Not applicable

Competing Interests

No competing interests were disclosed.

Reviewer Expertise

Mixed methods health services researcher who has been part of a team developing a COS.

Respond to this report

Responses (1)

Author Response

25 Jan 2021

Karen Matvienko-Sikar, School of Public Health, University College Cork, Cork, Ireland

Reviewer comment 1. This is an interesting well written protocol for a study to explore whether researchers are using core outcome sets and if not why not. The method involves identifying and critically reviewing recently published papers in major journals followed by a survey of authors. The paper explains why this is an important topic and the value of researchers using core outcomes, especially for evidence synthesis. I have just a few comments for the authors to consider.

My main concern is about greater clarity of what this review will add to the existing studies which have shown poor uptake of COS in Cochrane, previous trials and recently funded HTA applications (do these HTA application need more explanation for international audiences?). This review is looking at major impact journals but why is that important or different. Also I agree the selected journals are subjectively important and influential but is there an objective justification for their selection?

Author response 1.
Thank you for your comments, the protocol has now been amended taking these into account.
The rationale for examining the use of core outcome sets in major impact journals is that these journals are expected to publish trials of the highest quality, of clinical importance and relevance, and are likely to impact on policy and practice. This is different to Cochrane, which includes every trial, and HTA applications that may be more standardised. As such, this review will provide a different perspective on core outcome set use in trials, in terms of identifying uptake among those trials that are expected to be among those of the highest quality.

Reviewer comment 2. Are the authors suggesting Journal Editors should use their influence to encourage use of COS, in the same way as reporting guidelines are required?

Author response 2. The authors feel that Journal Editors are in a position to encourage and promote use of COS in trials. However, at protocol stage of this project we are not suggesting that Journal Editors use their influence to encourage COS because recommendations about how best to encourage COS use in trials will be derived from the data provided by the trialists in our survey.

Reviewer comment 3. In many ways the triallists’ views are the most interesting aspect of the paper as a way of understanding what needs to be done in the future to promote use of COS. Building on that, could behavioural theory be used to inform the questions asked and allow identification of appropriate behaviour change interventions?

Author response 3. We agree that the trialists survey responses will provide useful insights into how best to promote use of core outcome sets. Approaches such as the behaviour change wheel could be used to determine which behavioural determinants to target moving forward to enhance COS use. This approach was used to inform an earlier qualitative study topic guide examining COS use, which has informed the survey used in the current study.

Reviewer comment 4
Minor points
Is there a justification for the sample size of estimated 115 papers? This relates both to the generalisability of the findings and the value of the survey, especially for any sub sample analyses.

Author response 4. The sample size estimate was derived from initial scoping of the target journals, and is based on a pragmatic decision regarding the number of trials that are manageable to examine within the study timeframe but that are still representative of trials published by these journals. Sub sample analyses are not planned. The following has been added to the methods for clarity (new text in bold):
Based on initial scoping of the target journals, it is estimated that 115 trials, of various phases, have been published during in these journals during this timeframe and so this period has been chosen to ensure identification of a sufficient, yet pragmatically manageable number of recent trials for review by the review team.

Reviewer comment 5. Is ‘pharmacologic trials’ a normal label? Often referred to more as investigational products or drug trials.

Author response 5. The manuscript has been amended to state pharmacologic/drug trials

Reviewer comment 6. Under items to be extracted some e.g. disease name will not necessarily always be relevant.

Author response 6. As the focus of the review is on clinical trials published in major medical journals, it is important that details including ‘disease name’ are extracted. However if during data extraction, where disease name is not applicable for a specific trial the reviewer will specify ‘not applicable’.

Reviewer comment 7. For trials not reporting COS or part of COS is there a field for specifying the outcome that was actually used as well as any justification, for that decision?

Author Response 7. The following is included in the ‘Data extraction and analysis’ section of the protocol to specify data extracted in instances where COS are not used: For trials not reporting use of a COS, or reporting use of only some COS outcomes, the trial authors’ rationale and justification for the choice of outcomes used will be extracted from the published trial if reported.

Reviewer comment 8. First column page 4 has a longish paragraph on identifying if a relevant COS existed at the time of the trial. There is reference to time of designing the trial as being the base line. I am not sure how easy the date of the initial design would be to identify but one option might be to look at a relevant date of any grant application if the study had been externally funded.

Author Response 8. Reference to identifying when the trial started is now removed and replaced with: Information on the date of registration of the trial and/or publication of the trial protocol will be identified, with the earliest date retained where there is more than one registration/protocol.

Reviewer comment 9. In the same paragraph maybe add initials of the reviewer with prior experience and the second reviewer as presume they are members of the authorship team.

Author Response 9. We have now amended the process for COS identification, such that it will be conducted by pairs of reviewers who will work independently to identify COS in the COMET database and reach agreement by discussion. This has been updated below, though initials have not been included due to the on-going nature of this process.
Checking the COMET database will be done independently by paired reviewers, with each pair identifying COS for a subset of trials. At least one member of each pair will have prior experience of identifying COS for specific populations and interventions. Agreement on identified COS will be reached by discussion between reviewer pairs and will be verified by the first and senior authors.

Reviewer comment 10. Will any inferential statistics be conducted on the survey results?

Author Response 10. No the studies will be descriptively analysed to examine the numbers and percentage of trials using a COS, the numbers and percentage of trials that could have used a COS, and trialists’ awareness of, and decisions to search for and use, a COS.

Reviewer comment 11. Who will be approached for an ethical opinion – presume it will be a University review board?

Author response 11. Ethical approval has now been obtained from the University College Cork Social Research Ethics Committee. This information has now been added to the manuscript as follows: Ethical approval for this survey has been obtained from the University College Cork Social Research Ethics Committee (2020-137).

Reviewer comment 12. Who is funding this work?

Author response 12. This work is not being directly funded. Karen Matvienko-Sikar is supported by a Health Research Board Applying Research into Policy and Practice Fellowship [HRB-ARPP-A-011]. Kerry Avery and Jane Blazeby are supported by the NIHR Biomedical Research Centre at University Hospitals Bristol and Weston NHS Foundation Trust and the University of Bristol. Funding support for Paula Williamson has now been included (Paula Williamson is supported by the Medical Research Council (MRC) Trials Methodology Research Partnership (grant reference MR/S014357/1). These funders are not directly funding this project however.

View more View less

Competing Interests

No competing interests were disclosed.

Alongside their report, reviewers assign a status to the article:

Approved - the paper is scientifically sound in its current form and only minor, if any, improvements are suggested

Approved with reservations - A number of small changes, sometimes more significant revisions are required to address specific details and improve the papers academic merit.

Not approved - fundamental flaws in the paper seriously undermine the findings and conclusions

[1] 1. Williamson PR, Altman DG, Bagley H, et al.: The COMET Handbook: version 1.0. Trials. 2017; 18(Suppl 3): 280. PubMed Abstract | Publisher Full Text | Free Full Text

[2] 2. Williamson P, Altman D, Blazeby J, et al.: Driving up the Quality and Relevance of Research Through the Use of Agreed Core Outcomes. J Health Serv Res Policy. 2012; 17(1): 1–2. PubMed Abstract | Publisher Full Text

[3] 3. Clarke M, Williamson PR: Core outcome sets and systematic reviews. Syst Rev. 2016; 5: 11. PubMed Abstract | Publisher Full Text | Free Full Text

[4] 4. Kirkham JJ, Dwan KM, Altman DG, et al.: The impact of outcome reporting bias in randomised controlled trials on a cohort of systematic reviews. BMJ. 2010; 340: c365. PubMed Abstract | Publisher Full Text

[5] 5. Glasziou P, Altman DG, Bossuyt P, et al.: Reducing waste from incomplete or unusable reports of biomedical research. Lancet. 2014; 383(9913): 267–276. PubMed Abstract | Publisher Full Text

[6] 6. Biggane AM, Brading L, Ravaud P, et al.: Survey indicated that core outcome set development is increasingly including patients, being conducted internationally and using Delphi surveys. Trials. 2018; 19(1): 113. PubMed Abstract | Publisher Full Text | Free Full Text

[7] 7. Chalmers I, Bracken MB, Djulbegovic B, et al.: How to increase value and reduce waste when research priorities are set. Lancet. 2014; 383(9912): 156–165. PubMed Abstract | Publisher Full Text

[8] 8. Kirkham JJ, Gorst S, Altman DG, et al.: Core Outcome Set-STAndardised Protocol Items: the COS-STAP Statement. Trials. 2019; 20(1): 116. PubMed Abstract | Publisher Full Text | Free Full Text

[9] 9. Kirkham JJ, Gorst S, Altman DG, et al.: Core Outcome Set–STAndards for Reporting: The COS-STAR Statement. PLoS Med. 2016; 13(10): e1002148. PubMed Abstract | Publisher Full Text | Free Full Text

[10] 10. Kirkham JJ, Davis K, Altman DG, et al.: Core Outcome Set-STAndards for Development: The COS-STAD recommendations. PLoS Med. 2017; 14(11): e1002447. PubMed Abstract | Publisher Full Text | Free Full Text

[11] 11. Gargon E, Gorst SL, Williamson PR: Choosing important health outcomes for comparative effectiveness research: 5th annual update to a systematic review of core outcome sets for research. PLoS One. 2019; 14(12): e0225980. PubMed Abstract | Publisher Full Text | Free Full Text

[12] 12. Hughes KL, Kirkham JJ, Clarke M, et al.: Assessing the impact of a research funder’s recommendation to consider core outcome sets. PLoS One. 2019; 14(9): e0222418. PubMed Abstract | Publisher Full Text | Free Full Text

[13] 13. Kirkham JJ, Clarke M, Williamson PR: A methodological approach for assessing the uptake of core outcome sets using ClinicalTrials.gov: findings from a review of randomised controlled trials of rheumatoid arthritis. BMJ. 2017; 357: j2262. PubMed Abstract | Publisher Full Text | Free Full Text

[14] 14. Eldridge SM, Lancaster GA, Campbell MJ, et al.: Defining Feasibility and Pilot Studies in Preparation for Randomised Controlled Trials: Development of a Conceptual Framework. PLoS One. 2016; 11(3): e0150205. PubMed Abstract | Publisher Full Text | Free Full Text

[15] 15. Matvienko-Sikar K: Uptake of core outcome sets by clinical trialists publishing in major medical journals. 2020. http://www.doi.org/10.17605/OSF.IO/H4EKV

Uptake of core outcome sets by clinical trialists publishing in major medical journals: Protocol

Abstract

Keywords

Revised Amendments from Version 1

Introduction

Methods

Search strategy

Inclusion/exclusion criteria

Screening

Data extraction and analysis

Survey of trialists

Analysis

Ethical considerations and consent

Dissemination

Study status

Discussion

Data availability

Underlying data

Extended data

References

Comments on this article Comments (0)

Open Peer Review

Comments on this article Comments (0)

Open Peer Review

Reviewer Status

Reviewer Reports

Comments on this article

Are you a HRB-funded researcher?

Thank you!

Uptake of core outcome sets by clinical trialists publishing in major medical journals: Protocol

Abstract

Keywords

Revised Amendments from Version 1

Introduction

Methods

Search strategy

Inclusion/exclusion criteria

Screening

Data extraction and analysis

Survey of trialists

Analysis

Ethical considerations and consent

Dissemination

Study status

Discussion

Data availability

Underlying data

Extended data

References

Comments on this article Comments (0)

Open Peer Review

Comments on this article Comments (0)

Open Peer Review

Reviewer Status

Reviewer Reports

Comments on this article

Competing Interests Policy

Stay Updated

Are you a HRB-funded researcher?

Thank you!