Investigating how the GRADE Evidence to Decision (EtD) framework is used in Clinical Guidelines: a scoping review protocol

Introduction: The Grading of Recommendations, Assessment, Development and Evaluation (GRADE) evidence to decision (EtD) framework provides a structured and transparent approach for clinical guideline developers to use when formulating recommendations. Understanding how stakeholders use the EtD framework will inform how best to provide future training and support. This scoping review objective is to identify the key characteristics of how the GRADE EtD framework is used and identify studies on perception of use by those involved in developing clinical guidelines. Methods: JBI methodology for scoping reviews will be followed. This scoping review will consider both peer review published literature and grey literature. This will include empirical studies on the use of EtDs (including both quantitative, qualitative, and mixed methods primary research articles) and discussion papers/ commentaries on the experience of using the EtD. It will also include a random sample of publicly available populated EtDs identified from databases and repositories of GRADE guidelines. The search strategy will aim to locate both published and unpublished documents. First, we will conduct an exploratory search of MEDLINE and Embase (Elsevier), supplemented with citation analysis of included articles. Populated EtDs will be identified through searches of databases and repositories of GRADE guidelines. Two researchers will independently screen, select, and extract identified documents. Data will be presented in tables and summarized descriptively. Conclusion: This scoping review will identify the key characteristics of how the GRADE EtD framework is currently being used in clinical guidelines. Review findings can be used to inform future guidance and requirements for using GRADE EtD, as well as training on how to consider the criteria in developing recommendations. Results will be disseminated through publications in peer – reviewed journals and conference presentations. We will present our findings to relevant stakeholders via the networks of the co-author team at a one-day workshop.


Introduction
Clinical guidelines are considered a key foundation for quality improvement in health care 1 .They are systematically developed statements to assist practitioner and patient decisions about appropriate health care for specific clinical circumstances 1 .The development of guidelines is intended to organize and provide the best available evidence to support clinical decision making in order to improve quality of care, patient outcomes, and cost-effectiveness.
Ideally, within a guideline, each recommendation will be presented with a rating of both its strength and the certainty of the underlying evidence.The guideline development groups (GDGs) charged with developing these recommendations generally comprise a mix of key clinical and non-clinical stakeholders.These diverse groups bring a plurality of experiences, perspectives, and backgrounds to the discussions informing evidence-based recommendations.However, this complexity can produce different interpretations of the same evidence by different stakeholders, as patients and physicians may process, interpret, and respond to various types of uncertainty inherent in clinical decisions in different ways [2][3][4] .Therefore, a systematic approach that helps groups consider all the relevant factors can facilitate a more structured and explicit process when developing recommendations 5 .
The Grading of Recommendations, Assessment, Development and Evaluation (GRADE) approach, used internationally, presents a methodologically rigorous and transparent system for making judgments about the certainty of evidence and strength of recommendations 6 .The GRADE Evidence to Decision (EtD) framework (Table 1) was developed as a systematic and transparent way for guideline developers and decision makers to structure the process of moving from evidence to decisions, using pre-specified criteria 5 .These nine criteria (as outlined in Table 1) aim to ensure that important factors that determine a decision are considered in the decisionmaking process and a number of tailorable GRADE-EtD framework templates and guidance documents have been developed 5,7,8 .More than 100 organisations globally have adopted the principles of the GRADE system 5 , many of which include detail on using GRADE EtD in their methodological guidance documents 9 .Although GRADE has been widely endorsed, its use may be inconsistent with GRADE guidance and there is variation across organisations in terms of how the EtD should be adopted and applied 9,10 .The level of supporting evidence provided in published EtDs for core criteria such as resources required and cost effectiveness 11 , and equity 12 has also been shown to vary.
The use of an EtD framework is associated with guidelines of better quality, and more credible and transparent recommendations 13 .It has been reported to help in structuring a complex process through relatively simple steps in an explicit and transparent way 14 , managing discussions, keeping guideline panels on track, and dealing with disagreements 15 .However, to take full advantage of the EtD framework, it is necessary to be familiar with the GRADE approach and the framework itself.Therefore, for new users, prior training may be helpful for successful use 14 .While numerous guidance documents have been developed 5,7,8 , emerging evidence has highlighted that there is variation in terms of how EtDs are being adopted and applied by organisations 9,10 .The first study assessing the use of GRADE's EtD framework during real-time guideline development using panel discussions found that the proportion of time dedicated to certain framework criteria (as outlined in Table 1) is greater than others (e.g. research evidence 53% versus equity 2%) 16 .Further studies have indicated that the level of supporting evidence provided in published EtDs for criteria such as resources required and cost effectiveness 11 , and equity 12 has also been shown to vary.Factors contributing to this disparity were unclear.Given the widespread use and endorsement, of the GRADE EtD approach and its potential to increase the quality of guidelines 13 , understanding how they are currently utilised and perceived may identify areas of variation and the contributing factors, as well as informing how best to provide training and support -an important facilitator of engagement, particularly for patients 17 .Therefore, this scoping review aims to identify the key characteristics of how the GRADE EtD is used and identify studies on perception of use by clinical guideline developers.

Review question
How do guideline development groups apply, populate, and perceive the GRADE EtD when developing recommendations in clinical guidelines?Specific questions: • How is the GRADE EtD being used in GDG meetings in terms of the approaches to completing the EtD, guiding panel discussions and reaching consensus?
• Are there studies on how the GRADE EtD is perceived by various stakeholders involved in GDGs?
• How are the sections of the GRADE EtDs populated in terms of the research evidence and additional considerations supplied/cited to support developing recommendations?

Methods
The proposed scoping review will be conducted in accordance with the JBI methodology for scoping reviews 18 and reported in line with the Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) extension for scoping reviews 19 .This scoping review aligns to the indication for conducting a scoping review in that it aims to identify key characteristics or factors related to a concept, including those related to methodological research 20 .The protocol has been registered with Open Science Framework 21 .

Eligibility criteria
Participants.The focus of this review is on how GDGs apply the GRADE EtD, therefore participants in included studies will be guideline panel members, including but not limited to healthcare professionals, managers, methodologists, and patient representatives.

Concept.
The concept of interest is how GDGs apply and perceive the GRADE EtD.This includes the approach to using the framework within panel meetings, how the framework is populated and with what sources, how it is presented and communicated, and the perspectives of those using the framework.
Context.This review will be limited to the inclusion of EtDs used in clinical guidelines.Guidelines can target any clinical area and can be at a local, national, regional, or global level.We will include guideline EtDs (provided in English) and evaluations.For guidelines that have been updated, only the most recent version will be included.

Types of documents.
This scoping review will consider both peer reviewed published literature and grey literature.This will include empirical studies on the use of EtDs (including quantitative, qualitative, and mixed methods primary research articles) and discussion papers/ commentaries on the experience of using the EtD.Manuals and guidance documents on how to use the EtD, or articles describing its development or structure will not be included.Articles providing detailed descriptions of guidelines and the methodology to develop them will also be excluded.There will be no exclusions based on language or publication status (i.e., published, in press, in progress, or pre-print) for peer reviewed literature.For documents in languages other than English, Google Translate will be used, supplemented by interpretation by a native speaker where possible.Given the complexity of guideline EtDs and the potential for misinterpreting context, only EtDs produced in English will be included.
It will also include a random sample of populated EtDs (n=200) that have been made publicly available identified from databases and repositories of GRADE guidelines listed below.
Given the number of published guidelines and EtDs published, it would not be feasible to include them all, therefore we have chosen an approach to restricting the numbers, as previous analyses have also done 13 .These guidelines can be created by any group but do need to have used a version of the GRADE EtD.

Search strategy
The search strategy, developed with an expert health librarian (PM), will aim to locate both published and unpublished documents.An initial exploratory search of Medline was undertaken to identify key articles on the topic.The terminology used within the articles was analysed and used to develop a full search strategy (Table 2) for both Medline and Embase (Elsevier), conducted April 2023.The search will be supplemented by a citation analysis of the seminal EtD papers.
The reference list of all included documents will be screened for additional studies using citationchaser 22 supplemented by hand-searching reference lists where necessary.
The following databases, repositories, and websites will be filtered and screened for guideline EtDs, following the approach adopted in a scoping review of public health guidelines 23 : • the GIN international guideline library and registry of guidelines in development https://guidelines.ebmportal.com/ • BIGG international database of GRADE guidelines https:// sites.bvsalud.org/bigg/en/biblio/Given the recency of the use of GRADE EtD (published in 2016), we will limit the search to an eight year span (2015-2023).The guideline authors will not be contacted; only information provided in the guidelines will be used.It is proposed to include a random sample of 200 (using a random number generator) of identified published EtDs, due to the feasibility of a very large volume being identified for inclusion.

Selection and screening
Following the search, all identified citations will be collated and imported into Endnote X8.After deduplication, citations will be uploaded into Covidence.The title and abstracts will be screened independently by pairs of reviewers (so that each record is screened by at least two people) for assessment against the inclusion criteria for the review.The full text of selected citations will be assessed in detail against the inclusion criteria by two or more independent reviewers (so that each record is screened by at least two people) in Covidence.
Reasons for exclusion of full texts will be recorded and reported.Any disagreements between the reviewers at each stage of the screening process will be resolved through discussion, or with an additional reviewer/s.

Data extraction
Using a standardized extraction form specifically designed for this review, we will conduct a pilot exercise using 10% of the sample (selected based on Covidence display, most relevant) for the data extractors to calibrate and test the review form.Each document will be extracted by pairs of reviewers working independently (so that each document is extracted by at least two people) For included empirical studies, commentaries, and discussion papers (Table 3), data will be extracted on: • study characteristics (e.g., author, publication year, context, geographic setting) • study design (e.g., quantitative or qualitative) and approach to evaluation (e.g.survey, focus groups, interviews) • study participants (disciplines, amount) • description of application of EtD

• evaluation outcome data
For included EtDs (Table 4), data will be extracted on: • the characteristics of the original guideline (e.g., authors, topic, setting) • the structure and application of the EtD • the type of evidence source/information used within each of the EtD criteria.Specifically, we will be interested in:

EtD application
Free text Free text Options: • Primary

• All
• Other

• Not applicable
Free text Options: • Quantitative • Qualitative

• Not applicable
Options: • Appendix

• Glossary
• Main body

• No
Options: • Peer-Reviewed research • Mix of research & opinions

• Guideline group opinion
Options: o Was a systematic review conducted?
o Why types of evidence (RCTs, NRSI, qualitative studies, formal economic evaluations, health utility studies, panel input, expert consensus etc) are used and where are they cited.

Data analysis and presentation
All included articles will be described in detail in tables and figures and summarised using descriptive statistics.Published EtDs will be analysed using summative content analysis 24,25 , aimed at identifying and quantifying which criteria are presented and the type of evidence source/information used.

Study status
At time of publication of this protocol, database searches have been carried out and title and abstract screening is currently underway.

Dissemination
We intend to disseminate the results through publication in a peer-reviewed journal and conference presentations.We will present our findings to professionals engaged in the development of clinical guidelines through the networks of the co-author team.

Discussion
This scoping review aims to identify the key characteristics of how the GRADE EtD is used and identify studies on perception of use by clinical guideline developers.Given its widespread use and endorsement, it is important to evaluate the user experience of the GRADE-EtD framework.Understanding how GDG stakeholders perceive and use the EtD framework will inform how best to provide training and support -an important facilitator of engagement, particularly patient engagement.

Open Peer Review
Current Peer Review Status:

Jarrod Dusin
The University of Kansas Medical Center, Kansas City, USA The study aims to understand the application and perception of the GRADE EtD framework by those involved in developing clinical guidelines.Guidelines are important for informing healthcare decisions and improving patient outcomes.The GRADE EtD framework provides a systematic and transparent approach to guideline recommendations.Understanding how guideline groups utilize the GRADE EtD framework could optimize its effectiveness and provide necessary training and support.This paper will provide a much-needed understanding of how the EtD process is used.
Major Points Feedback: Methods: The methods section is detailed and outlines a clear process for conducting the review.

○
Specify the criteria for inclusion and exclusion of studies in the review.

○
Discuss how disagreements between the two researchers during screening, selection, and extraction will be resolved.

Expand on Data Analysis:
○ The data analysis section could benefit from a more detailed explanation of how the descriptive statistics and content analysis will be conducted.This could include information on the specific statistical tests that will be used and how the results will be interpreted.This well designed protocol outlines a scoping review to explore how the GRADE EtD framework is used by guideline developers.It will do this in two ways, it will summarise the current published and grey literature about experiences of using the EtD framework and will also sample 200 EtDs that have been used by guideline committees.The literature will be summarised using descriptive statistics and in depth evidence tables and figures.The EtD content will be analysed using a form of content analysis.
The proposal is well written and clear and will ensure that a transparent process is followed.I note that some large guideline producing agencies (for example NICE in the UK) do not use EtD frameworks for their guideline recommendations and this research will not explore why it isnt used by some agencies, however I suspect that is a separate research project.
Is the rationale for, and objectives of, the study clearly described?Yes

Are sufficient details of the methods provided to allow replication by others? Yes
Are the datasets clearly presented in a useable and accessible format?

Not applicable
Competing Interests: I am an employee of NICE (UK) which produces health guidelines.NICE is an active user of GRADE and member of GRADE working groups, however NICE does not currently adopt the GRADE EtD framework in its guideline production process.
Reviewer Expertise: Health guideline methods, systematic reviewing, qualitative evidence synthesis I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

Table 4 . Data extraction template for EtDs. Guideline characteristics EtD use description EtD criteria example (extraction will occur for each criteria) Title Organisation Year of publication
This is an open access peer review report distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

the rationale for, and objectives of, the study clearly described? Yes Is the study design appropriate for the research question? Yes Are sufficient details of the methods provided to allow replication by others? Yes Are the datasets clearly presented in a useable and accessible format? Yes Competing Interests:
No competing interests were disclosed. ○Is

have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.
Reviewer Report 18 January 2024 https://doi.org/10.21956/hrbopenres.15049.r37720© 2024 Carmona C.This is an open access peer review report distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.Chris Carmona 1 University of Sheffield, Sheffield, UK 2 National Institute for Health and Care Excellence, Manchester, UK