Investigating how the GRADE Evidence to Decision (EtD) framework is used in Clinical Guidelines: a scoping review protocol

Ruairí Murray; Melissa Sharp; Adriana Razidan; Ben Hibbitts; Máirín Ryan; Kamal Mahtani; Rosarie Lynch; Susan Smith; Michelle O'Neill; Holger Schünemann; Pablo Alonso-Coello; Zachary Munn; Barbara Clyne

doi:10.12688/hrbopenres.13757.1

Home Browse Investigating how the GRADE Evidence to Decision (EtD) framework is...

ALL Metrics

-

Views

69

Downloads

Get PDF

Get XML

Export

▬

✚

Study Protocol

Investigating how the GRADE Evidence to Decision (EtD) framework is used in Clinical Guidelines: a scoping review protocol

[version 1; peer review: 2 approved]

Ruairí Murray ¹, Melissa Sharp², Adriana Razidan³, [...] Ben Hibbitts³, Máirín Ryan^1,4, Kamal Mahtani⁵, Rosarie Lynch⁶, Susan Smith⁷, Michelle O'Neill¹, Holger Schünemann⁸, Pablo Alonso-Coello⁹, Zachary Munn¹⁰, Barbara Clyne²

Ruairí Murray ¹, Melissa Sharp², [...] Adriana Razidan³, Ben Hibbitts³, Máirín Ryan^1,4, Kamal Mahtani⁵, Rosarie Lynch⁶, Susan Smith⁷, Michelle O'Neill¹, Holger Schünemann⁸, Pablo Alonso-Coello⁹, Zachary Munn¹⁰, Barbara Clyne²

PUBLISHED 13 Sep 2023

Author details Author details

¹ Health Technology Assessment, Health Information and Quality Authority, Dublin, D07 E98Y, Ireland
² Department of Public Health & Epidemiology, School of Population Health, RCSI University of Medicine and Health Sciences, Dublin, Ireland
³ School of Pharmacy and Biomolecular Sciences (PBS), RCSI University of Medicine and Health Sciences, Dublin, Ireland
⁴ Department of Pharmacology & Therapeutics, Trinity College Dublin, Trinity Health Sciences, Dublin, D02 PN40, Ireland
⁵ Nuffield Department of Primary Care Health Sciences, University of Oxford, Oxford, England, UK
⁶ Department of health, Clinical Effectiveness and Antimicrobial Resistance Unit, National Patient Safety Office, Dublin, Ireland
⁷ Department of Public Health and Primary Care, School of Medicine, Trinity College Dublin, Dublin, D02 PN40, Ireland
⁸ Department of Health Research Methods, Evidence, and Impact McMaster University, Ontario, Canada
⁹ CIBER of Epidemiology and Public Health, Iberoamerican Cochrane Center-Servicio de Epidemiología Clínica y Salud Pública, Biomedical Research Institute, Barcelona, Spain
¹⁰ Faculty of Health and Medical Sciences, JBI Adelaide GRADE Centre, University of Adelaide, Adelaide, Australia

Ruairí Murray
Roles: Data Curation, Formal Analysis, Investigation, Methodology, Project Administration, Validation, Writing – Original Draft Preparation, Writing – Review & Editing

Melissa Sharp
Roles: Conceptualization, Data Curation, Formal Analysis, Investigation, Methodology, Project Administration, Supervision, Validation, Visualization, Writing – Original Draft Preparation, Writing – Review & Editing

Adriana Razidan
Roles: Data Curation, Formal Analysis, Investigation

Ben Hibbitts
Roles: Data Curation, Formal Analysis, Investigation

Máirín Ryan
Roles: Conceptualization, Methodology

Kamal Mahtani
Roles: Conceptualization

Rosarie Lynch
Roles: Conceptualization, Methodology

Susan Smith
Roles: Methodology

Michelle O'Neill
Roles: Conceptualization, Methodology

Holger Schünemann
Roles: Conceptualization, Methodology

Pablo Alonso-Coello
Roles: Conceptualization, Methodology

Zachary Munn
Roles: Conceptualization, Methodology

Barbara Clyne
Roles: Conceptualization, Data Curation, Formal Analysis, Funding Acquisition, Investigation, Methodology, Project Administration, Resources, Supervision, Validation, Visualization, Writing – Original Draft Preparation, Writing – Review & Editing

OPEN PEER REVIEW

REVIEWER STATUS

Abstract

Introduction: The Grading of Recommendations, Assessment, Development and Evaluation (GRADE) evidence to decision (EtD) framework provides a structured and transparent approach for clinical guideline developers to use when formulating recommendations. Understanding how stakeholders use the EtD framework will inform how best to provide future training and support. This scoping review objective is to identify the key characteristics of how the GRADE EtD framework is used and identify studies on perception of use by those involved in developing clinical guidelines.
Methods: JBI methodology for scoping reviews will be followed. This scoping review will consider both peer review published literature and grey literature. This will include empirical studies on the use of EtDs (including both quantitative, qualitative, and mixed methods primary research articles) and discussion papers/ commentaries on the experience of using the EtD. It will also include a random sample of publicly available populated EtDs identified from databases and repositories of GRADE guidelines. The search strategy will aim to locate both published and unpublished documents. First, we will conduct an exploratory search of MEDLINE and Embase (Elsevier), supplemented with citation analysis of included articles. Populated EtDs will be identified through searches of databases and repositories of GRADE guidelines. Two researchers will independently screen, select, and extract identified documents. Data will be presented in tables and summarized descriptively.
Conclusion: This scoping review will identify the key characteristics of how the GRADE EtD framework is currently being used in clinical guidelines. Review findings can be used to inform future guidance and requirements for using GRADE EtD, as well as training on how to consider the criteria in developing recommendations. Results will be disseminated through publications in peer – reviewed journals and conference presentations. We will present our findings to relevant stakeholders via the networks of the co-author team at a one-day workshop.

Keywords

GRADE, guidelines, Decision-making, methodology, scoping review

Corresponding authors: Ruairí Murray, Barbara Clyne

Competing interests: BC, MON, MR, RM, SMS are members of HRB-CICER, who independently review evidence and provide scientific support for the development, by guideline development groups, of the National Clinical Guidelines in Ireland. RL is Head of Clinical Effectiveness and Antimicrobial Resistance, Department of Health, Ireland. HS is co-chair of the GRADE Working Group and PAC, ZM are key contributors. AR, BH, MKS, KRM have no competing interests.

Grant information: Health Research Board [EIA-2019-09; Emerging Investigator Awards (EIA) 2017 to BC, MKS, and RM].
The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Copyright: © 2023 Murray R et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

How to cite: Murray R, Sharp M, Razidan A et al. Investigating how the GRADE Evidence to Decision (EtD) framework is used in Clinical Guidelines: a scoping review protocol [version 1; peer review: 2 approved]. HRB Open Res 2023, 6:50 (https://doi.org/10.12688/hrbopenres.13757.1) First published: 13 Sep 2023, 6:50 (https://doi.org/10.12688/hrbopenres.13757.1) Latest published: 13 Sep 2023, 6:50 (https://doi.org/10.12688/hrbopenres.13757.1)

Introduction

Clinical guidelines are considered a key foundation for quality improvement in health care¹. They are systematically developed statements to assist practitioner and patient decisions about appropriate health care for specific clinical circumstances¹. The development of guidelines is intended to organize and provide the best available evidence to support clinical decision making in order to improve quality of care, patient outcomes, and cost-effectiveness.

Ideally, within a guideline, each recommendation will be presented with a rating of both its strength and the certainty of the underlying evidence. The guideline development groups (GDGs) charged with developing these recommendations generally comprise a mix of key clinical and non-clinical stakeholders. These diverse groups bring a plurality of experiences, perspectives, and backgrounds to the discussions informing evidence-based recommendations. However, this complexity can produce different interpretations of the same evidence by different stakeholders, as patients and physicians may process, interpret, and respond to various types of uncertainty inherent in clinical decisions in different ways^2–4. Therefore, a systematic approach that helps groups consider all the relevant factors can facilitate a more structured and explicit process when developing recommendations⁵.

The Grading of Recommendations, Assessment, Development and Evaluation (GRADE) approach, used internationally, presents a methodologically rigorous and transparent system for making judgments about the certainty of evidence and strength of recommendations⁶. The GRADE Evidence to Decision (EtD) framework (Table 1) was developed as a systematic and transparent way for guideline developers and decision makers to structure the process of moving from evidence to decisions, using pre-specified criteria⁵. These nine criteria (as outlined in Table 1) aim to ensure that important factors that determine a decision are considered in the decision-making process and a number of tailorable GRADE-EtD framework templates and guidance documents have been developed^5,7,8. More than 100 organisations globally have adopted the principles of the GRADE system⁵, many of which include detail on using GRADE EtD in their methodological guidance documents⁹. Although GRADE has been widely endorsed, its use may be inconsistent with GRADE guidance and there is variation across organisations in terms of how the EtD should be adopted and applied^9,10. The level of supporting evidence provided in published EtDs for core criteria such as resources required and cost effectiveness¹¹, and equity¹² has also been shown to vary.

Table 1. Criteria for an EtD framework.

Criterion	Question prompts
Problem priority	• Is the problem a priority?
Benefits and harms	• How substantial are the desirable anticipated effects? • How substantial are the undesirable anticipated effects?
Certainty of the evidence	• What is the overall certainty of the evidence of effects?
Outcome importance	• Is there important uncertainty about or variability in how much people value the main outcomes?
Balance	• Do the desirable effects outweigh the undesirable effects?
Resource use	• How large are the resource requirements? • What is the certainty of the evidence of resource requirements?
Equity	• What would be the impact on health equity?
Acceptability	• Is the intervention/option acceptable to key stakeholders?
Feasibility	• Is the intervention feasible to implement?

The use of an EtD framework is associated with guidelines of better quality, and more credible and transparent recommendations¹³. It has been reported to help in structuring a complex process through relatively simple steps in an explicit and transparent way¹⁴, managing discussions, keeping guideline panels on track, and dealing with disagreements¹⁵. However, to take full advantage of the EtD framework, it is necessary to be familiar with the GRADE approach and the framework itself. Therefore, for new users, prior training may be helpful for successful use¹⁴. While numerous guidance documents have been developed^5,7,8, emerging evidence has highlighted that there is variation in terms of how EtDs are being adopted and applied by organisations^9,10. The first study assessing the use of GRADE's EtD framework during real-time guideline development using panel discussions found that the proportion of time dedicated to certain framework criteria (as outlined in Table 1) is greater than others (e.g. research evidence 53% versus equity 2%)¹⁶. Further studies have indicated that the level of supporting evidence provided in published EtDs for criteria such as resources required and cost effectiveness¹¹, and equity¹² has also been shown to vary. Factors contributing to this disparity were unclear.

Given the widespread use and endorsement, of the GRADE EtD approach and its potential to increase the quality of guidelines¹³, understanding how they are currently utilised and perceived may identify areas of variation and the contributing factors, as well as informing how best to provide training and support - an important facilitator of engagement, particularly for patients¹⁷. Therefore, this scoping review aims to identify the key characteristics of how the GRADE EtD is used and identify studies on perception of use by clinical guideline developers.

Review question

How do guideline development groups apply, populate, and perceive the GRADE EtD when developing recommendations in clinical guidelines?

Specific questions:

How is the GRADE EtD being used in GDG meetings in terms of the approaches to completing the EtD, guiding panel discussions and reaching consensus?
Are there studies on how the GRADE EtD is perceived by various stakeholders involved in GDGs?
How are the sections of the GRADE EtDs populated in terms of the research evidence and additional considerations supplied/cited to support developing recommendations?

Methods

The proposed scoping review will be conducted in accordance with the JBI methodology for scoping reviews¹⁸ and reported in line with the Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) extension for scoping reviews¹⁹. This scoping review aligns to the indication for conducting a scoping review in that it aims to identify key characteristics or factors related to a concept, including those related to methodological research²⁰. The protocol has been registered with Open Science Framework²¹.

Eligibility criteria

Participants. The focus of this review is on how GDGs apply the GRADE EtD, therefore participants in included studies will be guideline panel members, including but not limited to healthcare professionals, managers, methodologists, and patient representatives.

Concept. The concept of interest is how GDGs apply and perceive the GRADE EtD. This includes the approach to using the framework within panel meetings, how the framework is populated and with what sources, how it is presented and communicated, and the perspectives of those using the framework.

Context. This review will be limited to the inclusion of EtDs used in clinical guidelines. Guidelines can target any clinical area and can be at a local, national, regional, or global level. We will include guideline EtDs (provided in English) and evaluations. For guidelines that have been updated, only the most recent version will be included.

Types of documents. This scoping review will consider both peer reviewed published literature and grey literature. This will include empirical studies on the use of EtDs (including quantitative, qualitative, and mixed methods primary research articles) and discussion papers/ commentaries on the experience of using the EtD. Manuals and guidance documents on how to use the EtD, or articles describing its development or structure will not be included. Articles providing detailed descriptions of guidelines and the methodology to develop them will also be excluded. There will be no exclusions based on language or publication status (i.e., published, in press, in progress, or pre-print) for peer reviewed literature. For documents in languages other than English, Google Translate will be used, supplemented by interpretation by a native speaker where possible. Given the complexity of guideline EtDs and the potential for misinterpreting context, only EtDs produced in English will be included.

It will also include a random sample of populated EtDs (n=200) that have been made publicly available identified from databases and repositories of GRADE guidelines listed below. Given the number of published guidelines and EtDs published, it would not be feasible to include them all, therefore we have chosen an approach to restricting the numbers, as previous analyses have also done¹³. These guidelines can be created by any group but do need to have used a version of the GRADE EtD.

Search strategy

The search strategy, developed with an expert health librarian (PM), will aim to locate both published and unpublished documents. An initial exploratory search of Medline was undertaken to identify key articles on the topic. The terminology used within the articles was analysed and used to develop a full search strategy (Table 2) for both Medline and Embase (Elsevier), conducted April 2023. The search will be supplemented by a citation analysis of the seminal EtD papers.

Table 2. Database search: MEDLINE (Ovid).

Term
1. (Guideline$ adj1 development).mp.
2. Group decision making.mp
3. (guideline adj2 panel).mp.
4. guideline group$.mp.
5. 1 or 2 or 3 or 4
6. Evidence to decision framework.mp.
7. ((GRADE adj1 EtD) or (GRADE adj2 iEdT)).mp.
8. (EtD framework or iEtD ajd2 framework$).mp.
9. strength of recommendation.mp
10. ((grade or grading) adj2 (evidence or recommendation*)).ab,ti.
11. 6 or 7 or 8 or 9 or 10
12. 5 AND 11
13. limit 15 to yr="2015 -Current"

The reference list of all included documents will be screened for additional studies using citationchaser²² supplemented by hand-searching reference lists where necessary.

The following databases, repositories, and websites will be filtered and screened for guideline EtDs, following the approach adopted in a scoping review of public health guidelines²³:

the GIN international guideline library and registry of guidelines in development https://guidelines.ebmportal.com/
BIGG international database of GRADE guidelines https://sites.bvsalud.org/bigg/en/biblio/
Epistemonikos GRADE guidelines repository https://www.epistemonikos.org/en/groups/grade_guideline
GRADEpro Database of GRADE EtD’s and Guidelines https://gradepro.org/guidelines/
MAGICapp (https://app.magicapp.org/#/guidelines

Given the recency of the use of GRADE EtD (published in 2016), we will limit the search to an eight year span (2015–2023). The guideline authors will not be contacted; only information provided in the guidelines will be used. It is proposed to include a random sample of 200 (using a random number generator) of identified published EtDs, due to the feasibility of a very large volume being identified for inclusion.

Selection and screening

Following the search, all identified citations will be collated and imported into Endnote X8. After deduplication, citations will be uploaded into Covidence. The title and abstracts will be screened independently by pairs of reviewers (so that each record is screened by at least two people) for assessment against the inclusion criteria for the review. The full text of selected citations will be assessed in detail against the inclusion criteria by two or more independent reviewers (so that each record is screened by at least two people) in Covidence. Reasons for exclusion of full texts will be recorded and reported. Any disagreements between the reviewers at each stage of the screening process will be resolved through discussion, or with an additional reviewer/s.

Data extraction

Using a standardized extraction form specifically designed for this review, we will conduct a pilot exercise using 10% of the sample (selected based on Covidence display, most relevant) for the data extractors to calibrate and test the review form. Each document will be extracted by pairs of reviewers working independently (so that each document is extracted by at least two people)

For included empirical studies, commentaries, and discussion papers (Table 3), data will be extracted on:

study characteristics (e.g., author, publication year, context, geographic setting)
study design (e.g., quantitative or qualitative) and approach to evaluation (e.g. survey, focus groups, interviews)
study participants (disciplines, amount)
description of application of EtD
evaluation outcome data

Table 3. Data extraction template for empirical studies.

Study characteristics				Study design and participants			Description of application of EtD		Evaluation outcome
Author/ Year	Country	Care Setting	Clinical Area	Study Design	Evaluation methods	Study Participants	EtD Distributed before meeting	EtD application	Evaluation outcome
Free text	Free text	Options: • Primary • Secondary • Tertiary • All • Other • Not applicable	Free text	Options: • Quantitative • Qualitative • Mixed Methods	Options: • Interviews • Focus groups • Survey • Observations • Mixed • Other	Total N: Brief description, e.g., Nurses, Patient Reps, etc.	Options: • Yes • No • Not applicable	Options: • Structured discussion • Unstructured discussion • Not applicable	Free text

For included EtDs (Table 4), data will be extracted on:

the characteristics of the original guideline (e.g., authors, topic, setting)
the structure and application of the EtD
the type of evidence source/information used within each of the EtD criteria. Specifically, we will be interested in:
- Was a systematic review conducted?
- Why types of evidence (RCTs, NRSI, qualitative studies, formal economic evaluations, health utility studies, panel input, expert consensus etc) are used and where are they cited.

Table 4. Data extraction template for EtDs

Guideline characteristics				EtD use description			EtD criteria example (extraction will occur for each criteria)
Title	Organisation	Year of publication	Clinical Area	Location of EtD	Application of EtD	No of criteria included	Problem Priority included	Supporting evidence supplied	Type supporting evidence supplied	GRADE certainty supplied
Free text	Free text	• Free text	Free text	Options: • Appendix • Glossary • Main body • Other	Options: • Full • Partial	Total N:	Options: • Yes • No	Options: • Yes • No	Options: • Peer-Reviewed research • Expert opinion • Public opinion, • Mix of research & opinions • Guideline group opinion	Options: • Yes • No If yes, which GRADE system

Data analysis and presentation

All included articles will be described in detail in tables and figures and summarised using descriptive statistics. Published EtDs will be analysed using summative content analysis^24,25, aimed at identifying and quantifying which criteria are presented and the type of evidence source/information used.

Study status

At time of publication of this protocol, database searches have been carried out and title and abstract screening is currently underway.

Dissemination

We intend to disseminate the results through publication in a peer-reviewed journal and conference presentations. We will present our findings to professionals engaged in the development of clinical guidelines through the networks of the co-author team.

Discussion

This scoping review aims to identify the key characteristics of how the GRADE EtD is used and identify studies on perception of use by clinical guideline developers. Given its widespread use and endorsement, it is important to evaluate the user experience of the GRADE-EtD framework. Understanding how GDG stakeholders perceive and use the EtD framework will inform how best to provide training and support - an important facilitator of engagement, particularly patient engagement.

Data availability

No data are associated with this article.

Acknowledgements

The authors would like to thank Information Specialist Paul Murphy (RCSI University of Medicine and Health Sciences) for assistance with the development of the search strategy.

Faculty Opinions recommended

References

1. Eccles MP, Grimshaw JM, Shekelle P, et al.: Developing clinical practice guidelines: target audiences, identifying topics for guidelines, guideline group composition and functioning and conflicts of interest. Implement Sci. 2012; 7: 60. PubMed Abstract | Publisher Full Text | Free Full Text
2. Politi MC, Han PKJ, Col NF: Communicating the uncertainty of harms and benefits of medical interventions. Med Decis Making. 2007; 27(5): 681–95. PubMed Abstract | Publisher Full Text
3. Atkins L, Smith JA, Kelly MP, et al.: The process of developing evidence-based guidance in medicine and public health: a qualitative study of views from the inside. Implement Sci. 2013; 8(1): 101. PubMed Abstract | Publisher Full Text | Free Full Text
4. Wieringa S, Engebretsen E, Heggen K, et al.: Clinical guidelines and the pursuit of reducing epistemic uncertainty. An ethnographic study of guideline development panels in three countries. Soc Sci Med. 2021; 272: 113702. PubMed Abstract | Publisher Full Text
5. Alonso-Coello P, Schünemann HJ, Moberg J, et al.: GRADE Evidence to Decision (EtD) frameworks: a systematic and transparent approach to making well informed healthcare choices. 1: Introduction. BMJ. 2016; 353: i2016. PubMed Abstract | Publisher Full Text
6. Guyatt GH, Oxman AD, Vist GE, et al.: GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ. 2008; 336(7650): 924–6. PubMed Abstract | Publisher Full Text | Free Full Text
7. Schünemann HJ, Mustafa R, Brozek J, et al.: GRADE Guidelines: 16. GRADE evidence to decision frameworks for tests in clinical practice and public health. J Clin Epidemiol. 2016; 76: 89–98. PubMed Abstract | Publisher Full Text
8. Moberg J, Oxman AD, Rosenbaum S, et al.: The GRADE Evidence to Decision (EtD) framework for health system and public health decisions. Health Res Policy Syst. 2018; 16(1): 45. PubMed Abstract | Publisher Full Text | Free Full Text
9. Meneses-Echavez JF, Bidonde J, Yepes-Nuñez JJ, et al.: Evidence to decision frameworks enabled structured and explicit development of healthcare recommendations. J Clin Epidemiol. 2022; 150: 51–62. PubMed Abstract | Publisher Full Text
10. Schünemann HJ, Brennan S, Akl EA, et al.: The development methods of official GRADE articles and requirements for claiming the use of GRADE – a statement by the GRADE Guidance Group. J Clin Epidemiol. 2023; 159: 79–84. PubMed Abstract | Publisher Full Text
11. Riva JJ, Bhatt M, Brunarski DJ, et al.: Guidelines that use the GRADE approach often fail to provide complete economic information for recommendations: A systematic survey. J Clin Epidemiol. 2021; 136: 203–215. PubMed Abstract | Publisher Full Text
12. Dewidar O, Tsang P, León-García M, et al.: Over half of the WHO guidelines published from 2014 to 2019 explicitly considered health equity issues: a cross-sectional survey. J Clin Epidemiol. 2020; 127: 125–133. PubMed Abstract | Publisher Full Text
13. Meneses-Echavez JF, Bidonde J, Montesinos-Guevara C, et al.: Using Evidence to Decision Frameworks Led to Guidelines of Better Quality and More Credible and Transparent Recommendations. 2023. Publisher Full Text
14. Neumann I, Brignardello-Petersen R, Wiercioch W, et al.: The GRADE evidence-to-decision framework: a report of its testing and application in 15 international guideline panels. Implement Sci. 2016; 11(1): 93. PubMed Abstract | Publisher Full Text | Free Full Text
15. Rosenbaum SE, Moberg J, Glenton C, et al.: Developing Evidence to Decision Frameworks and an Interactive Evidence to Decision Tool for Making and Using Decisions and Recommendations in Health Care. Glob Chall. 2018; 2(9): 1700081. PubMed Abstract | Publisher Full Text | Free Full Text
16. Li SA, Alexander PE, Reljic T, et al.: Evidence to Decision framework provides a structured "roadmap" for making GRADE guidelines recommendations. J Clin Epidemiol. 2018; 104: 103–112. PubMed Abstract | Publisher Full Text | Free Full Text
17. G-I-N PUBLIC Toolkit: Patient and Public Involvement in Guidelines. Accessed Aug 2018. Reference Source
18. Peters MDJ, Godfrey C, McInerney P, et al.: Best practice guidance and reporting items for the development of scoping review protocols. JBI Evid Synth. 2022; 20(4): 953–968. PubMed Abstract | Publisher Full Text
19. Tricco AC, Lillie E, Zarin W, et al.: PRISMA Extension for Scoping Reviews (PRISMA-ScR): Checklist and Explanation. Ann Intern Med. 2018; 169(7): 467–473. PubMed Abstract | Publisher Full Text
20. Munn Z, Pollock D, Khalil H, et al.: What are scoping reviews? Providing a formal definition of scoping reviews as a type of evidence synthesis. JBI Evid Synth. 2022; 20(4): 950–952. PubMed Abstract | Publisher Full Text
21. Clyne B, Sharp MK, Murray R: Investigating how the GRADE Evidence to Decision (EtD) framework is used in Clinical Guidelines: a scoping review. 2023. Publisher Full Text
22. Haddaway N, Grainger M, Gray C: citationchaser: An R package and Shiny app for forward and backward citations chasing in academic searching. 2021. Publisher Full Text
23. Kantorová L, Friessová T, Slezáková S, et al.: Addressing Challenges When Applying GRADE to Public Health Guidelines: A Scoping Review Protocol and Pilot Analysis. Int J Environ Res Public Health. 2022; 19(2): 992. PubMed Abstract | Publisher Full Text | Free Full Text
24. Hsieh HF, Shannon SE: Three approaches to qualitative content analysis. Qual Health Res. 2005; 15(9): 1277–88. PubMed Abstract | Publisher Full Text
25. Kleinheksel AJ, Rockich-Winston N, Tawfik H, et al.: Demystifying Content Analysis. Am J Pharm Educ. 2020; 84(1): 7113. PubMed Abstract | Publisher Full Text | Free Full Text

Comments on this article Comments (0)

Version 1

VERSION 1 PUBLISHED 13 Sep 2023

Author details Author details

¹ Health Technology Assessment, Health Information and Quality Authority, Dublin, D07 E98Y, Ireland
² Department of Public Health & Epidemiology, School of Population Health, RCSI University of Medicine and Health Sciences, Dublin, Ireland
³ School of Pharmacy and Biomolecular Sciences (PBS), RCSI University of Medicine and Health Sciences, Dublin, Ireland
⁴ Department of Pharmacology & Therapeutics, Trinity College Dublin, Trinity Health Sciences, Dublin, D02 PN40, Ireland
⁵ Nuffield Department of Primary Care Health Sciences, University of Oxford, Oxford, England, UK
⁶ Department of health, Clinical Effectiveness and Antimicrobial Resistance Unit, National Patient Safety Office, Dublin, Ireland
⁷ Department of Public Health and Primary Care, School of Medicine, Trinity College Dublin, Dublin, D02 PN40, Ireland
⁸ Department of Health Research Methods, Evidence, and Impact McMaster University, Ontario, Canada
⁹ CIBER of Epidemiology and Public Health, Iberoamerican Cochrane Center-Servicio de Epidemiología Clínica y Salud Pública, Biomedical Research Institute, Barcelona, Spain
¹⁰ Faculty of Health and Medical Sciences, JBI Adelaide GRADE Centre, University of Adelaide, Adelaide, Australia

Ruairí Murray
Roles: Data Curation, Formal Analysis, Investigation, Methodology, Project Administration, Validation, Writing – Original Draft Preparation, Writing – Review & Editing

Melissa Sharp
Roles: Conceptualization, Data Curation, Formal Analysis, Investigation, Methodology, Project Administration, Supervision, Validation, Visualization, Writing – Original Draft Preparation, Writing – Review & Editing

Adriana Razidan
Roles: Data Curation, Formal Analysis, Investigation

Ben Hibbitts
Roles: Data Curation, Formal Analysis, Investigation

Máirín Ryan
Roles: Conceptualization, Methodology

Kamal Mahtani
Roles: Conceptualization

Rosarie Lynch
Roles: Conceptualization, Methodology

Susan Smith
Roles: Methodology

Michelle O'Neill
Roles: Conceptualization, Methodology

Holger Schünemann
Roles: Conceptualization, Methodology

Pablo Alonso-Coello
Roles: Conceptualization, Methodology

Zachary Munn
Roles: Conceptualization, Methodology

Barbara Clyne
Roles: Conceptualization, Data Curation, Formal Analysis, Funding Acquisition, Investigation, Methodology, Project Administration, Resources, Supervision, Validation, Visualization, Writing – Original Draft Preparation, Writing – Review & Editing

Competing interests

BC, MON, MR, RM, SMS are members of HRB-CICER, who independently review evidence and provide scientific support for the development, by guideline development groups, of the National Clinical Guidelines in Ireland. RL is Head of Clinical Effectiveness and Antimicrobial Resistance, Department of Health, Ireland. HS is co-chair of the GRADE Working Group and PAC, ZM are key contributors. AR, BH, MKS, KRM have no competing interests.

Grant information

Health Research Board [EIA-2019-09; Emerging Investigator Awards (EIA) 2017 to BC, MKS, and RM].
The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Article Versions (1)

version 1

Published: 13 Sep 2023, 6:50

https://doi.org/10.12688/hrbopenres.13757.1

Copyright

© 2023 Murray R et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Download

Export To

metrics

VIEWS

735

downloads

69

Citations

0

SEE MORE DETAILS

CITE

how to cite this article

Murray R, Sharp M, Razidan A et al. Investigating how the GRADE Evidence to Decision (EtD) framework is used in Clinical Guidelines: a scoping review protocol [version 1; peer review: 2 approved]. HRB Open Res 2023, 6:50 (https://doi.org/10.12688/hrbopenres.13757.1)

NOTE: If applicable, it is important to ensure the information in square brackets after the title is included in all citations of this article.

track

receive updates on this article

Track an article to receive email alerts on any updates to this article.

Open Peer Review

Version 1

VERSION 1

PUBLISHED 13 Sep 2023

Views

10

Reviewer Report 11 May 2024

Jarrod Dusin, The University of Kansas Medical Center, Kansas City, USA

Approved

https://doi.org/10.21956/hrbopenres.15049.r38906

The study aims to understand the application and perception of the GRADE EtD framework by those involved in developing clinical guidelines. Guidelines are important for informing healthcare decisions and improving patient outcomes. The GRADE EtD framework provides a systematic and ... Continue reading

The study aims to understand the application and perception of the GRADE EtD framework by those involved in developing clinical guidelines. Guidelines are important for informing healthcare decisions and improving patient outcomes. The GRADE EtD framework provides a systematic and transparent approach to guideline recommendations. Understanding how guideline groups utilize the GRADE EtD framework could optimize its effectiveness and provide necessary training and support. This paper will provide a much-needed understanding of how the EtD process is used.

Major Points Feedback:

Methods: The methods section is detailed and outlines a clear process for conducting the review.

Specify the criteria for inclusion and exclusion of studies in the review.
Discuss how disagreements between the two researchers during screening, selection, and extraction will be resolved.
Expand on Data Analysis:
The data analysis section could benefit from a more detailed explanation of how the descriptive statistics and content analysis will be conducted. This could include information on the specific statistical tests that will be used and how the results will be interpreted.

Minor Points Feedback:

Please mention any potential limitations or challenges you anticipate in conducting this review.

Is the rationale for, and objectives of, the study clearly described?

Yes
Is the study design appropriate for the research question?

Yes
Are sufficient details of the methods provided to allow replication by others?

Yes
Are the datasets clearly presented in a useable and accessible format?

Yes

Competing Interests: No competing interests were disclosed.

Reviewer Expertise: My work involves the development of guidelines and clinical pathways. I have used both GRADE and EtD in my work.

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

CITE

Report a concern

Respond or Comment

Views

14

Reviewer Report 18 Jan 2024

Chris Carmona, University of Sheffield, Sheffield, UK; National Institute for Health and Care Excellence, Manchester, UK

Approved

https://doi.org/10.21956/hrbopenres.15049.r37720

This well designed protocol outlines a scoping review to explore how the GRADE EtD framework is used by guideline developers. It will do this in two ways, it will summarise the current published and grey literature about experiences of using ... Continue reading

This well designed protocol outlines a scoping review to explore how the GRADE EtD framework is used by guideline developers. It will do this in two ways, it will summarise the current published and grey literature about experiences of using the EtD framework and will also sample 200 EtDs that have been used by guideline committees. The literature will be summarised using descriptive statistics and in depth evidence tables and figures. The EtD content will be analysed using a form of content analysis.

The proposal is well written and clear and will ensure that a transparent process is followed. I note that some large guideline producing agencies (for example NICE in the UK) do not use EtD frameworks for their guideline recommendations and this research will not explore why it isnt used by some agencies, however I suspect that is a separate research project.

Is the rationale for, and objectives of, the study clearly described?

Yes
Is the study design appropriate for the research question?

Yes
Are sufficient details of the methods provided to allow replication by others?

Yes
Are the datasets clearly presented in a useable and accessible format?

Not applicable

Competing Interests: I am an employee of NICE (UK) which produces health guidelines. NICE is an active user of GRADE and member of GRADE working groups, however NICE does not currently adopt the GRADE EtD framework in its guideline production process.

Reviewer Expertise: Health guideline methods, systematic reviewing, qualitative evidence synthesis

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

CITE

Report a concern

Respond or Comment

Comments on this article Comments (0)

Version 1

VERSION 1 PUBLISHED 13 Sep 2023

Open Peer Review

Reviewer Status

Reviewer Reports

	Invited Reviewers
	1	2
Version 1 13 Sep 23	read	read

Chris Carmona, University of Sheffield, Sheffield, UK; National Institute for Health and Care Excellence, Manchester, UK
Jarrod Dusin, The University of Kansas Medical Center, Kansas City, USA

Comments on this article

All Comments(0)

Add a comment

Sign up for content alerts

Back to all reports

Reviewer Report

10 Views

11 May 2024 | for Version 1

Jarrod Dusin, The University of Kansas Medical Center, Kansas City, USA

10 Views Cite this report Responses(0)

Approved

The study aims to understand the application and perception of the GRADE EtD framework by those involved in developing clinical guidelines. Guidelines are important for informing healthcare decisions and improving patient outcomes. The GRADE EtD framework provides a systematic and transparent approach to guideline recommendations. Understanding how guideline groups utilize the GRADE EtD framework could optimize its effectiveness and provide necessary training and support. This paper will provide a much-needed understanding of how the EtD process is used.

Major Points Feedback:

Methods: The methods section is detailed and outlines a clear process for conducting the review.

Specify the criteria for inclusion and exclusion of studies in the review.
Discuss how disagreements between the two researchers during screening, selection, and extraction will be resolved.
Expand on Data Analysis:
The data analysis section could benefit from a more detailed explanation of how the descriptive statistics and content analysis will be conducted. This could include information on the specific statistical tests that will be used and how the results will be interpreted.

Minor Points Feedback:

Please mention any potential limitations or challenges you anticipate in conducting this review.

Is the rationale for, and objectives of, the study clearly described?

Yes
Is the study design appropriate for the research question?

Yes
Are sufficient details of the methods provided to allow replication by others?

Yes
Are the datasets clearly presented in a useable and accessible format?

Yes

Competing Interests

No competing interests were disclosed.

Reviewer Expertise

My work involves the development of guidelines and clinical pathways. I have used both GRADE and EtD in my work.

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

Respond to this report

Responses (0)

Back to all reports

Reviewer Report

14 Views

18 Jan 2024 | for Version 1

Chris Carmona, University of Sheffield, Sheffield, UK; National Institute for Health and Care Excellence, Manchester, UK

14 Views Cite this report Responses(0)

Approved

This well designed protocol outlines a scoping review to explore how the GRADE EtD framework is used by guideline developers. It will do this in two ways, it will summarise the current published and grey literature about experiences of using the EtD framework and will also sample 200 EtDs that have been used by guideline committees. The literature will be summarised using descriptive statistics and in depth evidence tables and figures. The EtD content will be analysed using a form of content analysis.

The proposal is well written and clear and will ensure that a transparent process is followed. I note that some large guideline producing agencies (for example NICE in the UK) do not use EtD frameworks for their guideline recommendations and this research will not explore why it isnt used by some agencies, however I suspect that is a separate research project.

Is the rationale for, and objectives of, the study clearly described?

Yes
Is the study design appropriate for the research question?

Yes
Are sufficient details of the methods provided to allow replication by others?

Yes
Are the datasets clearly presented in a useable and accessible format?

Not applicable

Competing Interests

I am an employee of NICE (UK) which produces health guidelines. NICE is an active user of GRADE and member of GRADE working groups, however NICE does not currently adopt the GRADE EtD framework in its guideline production process.

Reviewer Expertise

Health guideline methods, systematic reviewing, qualitative evidence synthesis

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

Respond to this report

Responses (0)

[1] 1. Eccles MP, Grimshaw JM, Shekelle P, et al.: Developing clinical practice guidelines: target audiences, identifying topics for guidelines, guideline group composition and functioning and conflicts of interest. Implement Sci. 2012; 7: 60. PubMed Abstract | Publisher Full Text | Free Full Text

[2] 2. Politi MC, Han PKJ, Col NF: Communicating the uncertainty of harms and benefits of medical interventions. Med Decis Making. 2007; 27(5): 681–95. PubMed Abstract | Publisher Full Text

[3] 3. Atkins L, Smith JA, Kelly MP, et al.: The process of developing evidence-based guidance in medicine and public health: a qualitative study of views from the inside. Implement Sci. 2013; 8(1): 101. PubMed Abstract | Publisher Full Text | Free Full Text

[4] 4. Wieringa S, Engebretsen E, Heggen K, et al.: Clinical guidelines and the pursuit of reducing epistemic uncertainty. An ethnographic study of guideline development panels in three countries. Soc Sci Med. 2021; 272: 113702. PubMed Abstract | Publisher Full Text

[5] 5. Alonso-Coello P, Schünemann HJ, Moberg J, et al.: GRADE Evidence to Decision (EtD) frameworks: a systematic and transparent approach to making well informed healthcare choices. 1: Introduction. BMJ. 2016; 353: i2016. PubMed Abstract | Publisher Full Text

[6] 6. Guyatt GH, Oxman AD, Vist GE, et al.: GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ. 2008; 336(7650): 924–6. PubMed Abstract | Publisher Full Text | Free Full Text

[7] 7. Schünemann HJ, Mustafa R, Brozek J, et al.: GRADE Guidelines: 16. GRADE evidence to decision frameworks for tests in clinical practice and public health. J Clin Epidemiol. 2016; 76: 89–98. PubMed Abstract | Publisher Full Text

[8] 8. Moberg J, Oxman AD, Rosenbaum S, et al.: The GRADE Evidence to Decision (EtD) framework for health system and public health decisions. Health Res Policy Syst. 2018; 16(1): 45. PubMed Abstract | Publisher Full Text | Free Full Text

[9] 9. Meneses-Echavez JF, Bidonde J, Yepes-Nuñez JJ, et al.: Evidence to decision frameworks enabled structured and explicit development of healthcare recommendations. J Clin Epidemiol. 2022; 150: 51–62. PubMed Abstract | Publisher Full Text

[10] 10. Schünemann HJ, Brennan S, Akl EA, et al.: The development methods of official GRADE articles and requirements for claiming the use of GRADE – a statement by the GRADE Guidance Group. J Clin Epidemiol. 2023; 159: 79–84. PubMed Abstract | Publisher Full Text

[11] 11. Riva JJ, Bhatt M, Brunarski DJ, et al.: Guidelines that use the GRADE approach often fail to provide complete economic information for recommendations: A systematic survey. J Clin Epidemiol. 2021; 136: 203–215. PubMed Abstract | Publisher Full Text

[12] 12. Dewidar O, Tsang P, León-García M, et al.: Over half of the WHO guidelines published from 2014 to 2019 explicitly considered health equity issues: a cross-sectional survey. J Clin Epidemiol. 2020; 127: 125–133. PubMed Abstract | Publisher Full Text

[13] 13. Meneses-Echavez JF, Bidonde J, Montesinos-Guevara C, et al.: Using Evidence to Decision Frameworks Led to Guidelines of Better Quality and More Credible and Transparent Recommendations. 2023. Publisher Full Text

[14] 14. Neumann I, Brignardello-Petersen R, Wiercioch W, et al.: The GRADE evidence-to-decision framework: a report of its testing and application in 15 international guideline panels. Implement Sci. 2016; 11(1): 93. PubMed Abstract | Publisher Full Text | Free Full Text

[15] 15. Rosenbaum SE, Moberg J, Glenton C, et al.: Developing Evidence to Decision Frameworks and an Interactive Evidence to Decision Tool for Making and Using Decisions and Recommendations in Health Care. Glob Chall. 2018; 2(9): 1700081. PubMed Abstract | Publisher Full Text | Free Full Text

[16] 16. Li SA, Alexander PE, Reljic T, et al.: Evidence to Decision framework provides a structured "roadmap" for making GRADE guidelines recommendations. J Clin Epidemiol. 2018; 104: 103–112. PubMed Abstract | Publisher Full Text | Free Full Text

[17] 17. G-I-N PUBLIC Toolkit: Patient and Public Involvement in Guidelines. Accessed Aug 2018. Reference Source

[18] 18. Peters MDJ, Godfrey C, McInerney P, et al.: Best practice guidance and reporting items for the development of scoping review protocols. JBI Evid Synth. 2022; 20(4): 953–968. PubMed Abstract | Publisher Full Text

[19] 19. Tricco AC, Lillie E, Zarin W, et al.: PRISMA Extension for Scoping Reviews (PRISMA-ScR): Checklist and Explanation. Ann Intern Med. 2018; 169(7): 467–473. PubMed Abstract | Publisher Full Text

[20] 20. Munn Z, Pollock D, Khalil H, et al.: What are scoping reviews? Providing a formal definition of scoping reviews as a type of evidence synthesis. JBI Evid Synth. 2022; 20(4): 950–952. PubMed Abstract | Publisher Full Text

[21] 21. Clyne B, Sharp MK, Murray R: Investigating how the GRADE Evidence to Decision (EtD) framework is used in Clinical Guidelines: a scoping review. 2023. Publisher Full Text

[22] 22. Haddaway N, Grainger M, Gray C: citationchaser: An R package and Shiny app for forward and backward citations chasing in academic searching. 2021. Publisher Full Text

[23] 23. Kantorová L, Friessová T, Slezáková S, et al.: Addressing Challenges When Applying GRADE to Public Health Guidelines: A Scoping Review Protocol and Pilot Analysis. Int J Environ Res Public Health. 2022; 19(2): 992. PubMed Abstract | Publisher Full Text | Free Full Text

[24] 24. Hsieh HF, Shannon SE: Three approaches to qualitative content analysis. Qual Health Res. 2005; 15(9): 1277–88. PubMed Abstract | Publisher Full Text

[25] 25. Kleinheksel AJ, Rockich-Winston N, Tawfik H, et al.: Demystifying Content Analysis. Am J Pharm Educ. 2020; 84(1): 7113. PubMed Abstract | Publisher Full Text | Free Full Text

Investigating how the GRADE Evidence to Decision (EtD) framework is used in Clinical Guidelines: a scoping review protocol

Abstract

Keywords

Introduction

Table 1. Criteria for an EtD framework.

Review question

Methods

Eligibility criteria

Search strategy

Table 2. Database search: MEDLINE (Ovid).

Selection and screening

Data extraction

Table 3. Data extraction template for empirical studies.

Table 4. Data extraction template for EtDs

Data analysis and presentation

Study status

Dissemination

Discussion

Data availability

Acknowledgements

References

Comments on this article Comments (0)

Open Peer Review

Comments on this article Comments (0)

Open Peer Review

Reviewer Status

Reviewer Reports

Comments on this article

Are you a HRB-funded researcher?

Thank you!

Investigating how the GRADE Evidence to Decision (EtD) framework is used in Clinical Guidelines: a scoping review protocol

Abstract

Keywords

Introduction

Table 1. Criteria for an EtD framework.

Review question

Methods

Eligibility criteria

Search strategy

Table 2. Database search: MEDLINE (Ovid).

Selection and screening

Data extraction

Table 3. Data extraction template for empirical studies.

Table 4. Data extraction template for EtDs

Data analysis and presentation

Study status

Dissemination

Discussion

Data availability

Acknowledgements

References

Comments on this article Comments (0)

Open Peer Review

Comments on this article Comments (0)

Open Peer Review

Reviewer Status

Reviewer Reports

Comments on this article

Competing Interests Policy

Stay Updated

Are you a HRB-funded researcher?

Thank you!