Evidence synthesis summary formats for clinical guideline development group members: a mixed-methods systematic review protocol

Melissa K. Sharp; Barrie Tyner; Dayang Anis Binti Awang Baki; Cormac Farrell; Declan Devane; Kamal R. Mahtani; Susan M. Smith; Michelle O'Neill; Máirín Ryan; Barbara Clyne

doi:10.12688/hrbopenres.13325.2

Home Browse Evidence synthesis summary formats for clinical guideline development...

ALL Metrics

Views

Downloads

Get PDF

Get XML

Export

▬

✚

Study Protocol

Revised

Evidence synthesis summary formats for clinical guideline development group members: a mixed-methods systematic review protocol

[version 2; peer review: 2 approved]

Melissa K. Sharp ¹, Barrie Tyner², Dayang Anis Binti Awang Baki³, [...] Cormac Farrell², Declan Devane^4,5, Kamal R. Mahtani⁶, Susan M. Smith¹, Michelle O'Neill², Máirín Ryan^2,7, Barbara Clyne^1,2

Melissa K. Sharp ¹, Barrie Tyner², [...] Dayang Anis Binti Awang Baki³, Cormac Farrell², Declan Devane^4,5, Kamal R. Mahtani⁶, Susan M. Smith¹, Michelle O'Neill², Máirín Ryan^2,7, Barbara Clyne^1,2

PUBLISHED 10 May 2022

Author details Author details

¹ Health Research Board Centre for Primary Care Research, Department of General Practice,, Royal College of Surgeons in Ireland, Dublin, Ireland
² Health Information and Quality Authority, Dublin, Ireland
³ School of Medicine, Royal College of Surgeons in Ireland, Dublin, Ireland
⁴ School of Nursing and Midwifery, NUI Galway, Galway, Ireland
⁵ Evidence Synthesis Ireland and Cochrane Ireland, Galway, Ireland
⁶ Nuffield Department of Primary Care Health Sciences, University of Oxford, Oxford, UK
⁷ Department of Pharmacology & Therapeutics, Trinity College Dublin, Dublin, Ireland

Melissa K. Sharp
Roles: Conceptualization, Data Curation, Formal Analysis, Investigation, Methodology, Project Administration, Supervision, Writing – Original Draft Preparation, Writing – Review & Editing

Barrie Tyner
Roles: Methodology, Writing – Review & Editing

Dayang Anis Binti Awang Baki
Roles: Writing – Review & Editing

Cormac Farrell
Roles: Writing – Review & Editing

Declan Devane
Roles: Methodology, Writing – Review & Editing

Kamal R. Mahtani
Roles: Methodology, Writing – Review & Editing

Susan M. Smith
Roles: Methodology, Writing – Review & Editing

Michelle O'Neill
Roles: Conceptualization, Funding Acquisition, Methodology, Writing – Review & Editing

Máirín Ryan
Roles: Conceptualization, Funding Acquisition, Methodology, Writing – Review & Editing

Barbara Clyne
Roles: Conceptualization, Funding Acquisition, Methodology, Project Administration, Supervision, Writing – Original Draft Preparation, Writing – Review & Editing

OPEN PEER REVIEW

REVIEWER STATUS

Abstract

Introduction: Evidence syntheses, often in the form of systematic reviews, are essential for clinical guideline development and informing changes to health policies. However, clinical guideline development groups (CGDG) are multidisciplinary, and participants such as policymakers, healthcare professionals and patient representatives can face obstacles when trying to understand and use evidence synthesis findings. Summary formats to communicate the results of evidence syntheses have become increasingly common, but it is currently unclear which format is most effective for different stakeholders. This mixed-methods systematic review (MMSR) evaluates the effectiveness and acceptability of different evidence synthesis summary formats for CGDG members.
Methods: This protocol follows guidance from the Joanna Briggs Institute on MMSRs and is reported according to the Preferred Reporting Items for Systematic Reviews (PRISMA)-P guideline. A comprehensive search of six databases will be performed with no language restrictions. Primary outcomes are those relating to the effectiveness and preferences for and attitudes towards the different summary formats. We will include qualitative research and randomised controlled trials. Two reviewers will perform title, abstract, and full-text screening. Independent double-extraction of study characteristics and critical appraisal items will be undertaken using a standardised form. We will use a convergent segregated approach to analyse quantitative and qualitative data separately; results will then be integrated.
Discussion: The results of this systematic review will provide an overview of the effectiveness and acceptability of different summary formats for evidence synthesis findings. These findings can be helpful for those in or communicating to guideline development groups. The results can also inform the development and pilot-testing of summary formats for evidence summaries.

Keywords

presentation of findings, evidence summaries, summary of findings table, communication, mixed-methods systematic review

Corresponding author: Melissa K. Sharp

Competing interests: No competing interests were disclosed.

Grant information: Health Research Board Ireland [EIA-2019-09].
The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Copyright: © 2022 Sharp MK et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

How to cite: Sharp MK, Tyner B, Awang Baki DAB et al. Evidence synthesis summary formats for clinical guideline development group members: a mixed-methods systematic review protocol [version 2; peer review: 2 approved]. HRB Open Res 2022, 4:76 (https://doi.org/10.12688/hrbopenres.13325.2) First published: 15 Jul 2021, 4:76 (https://doi.org/10.12688/hrbopenres.13325.1) Latest published: 10 May 2022, 4:76 (https://doi.org/10.12688/hrbopenres.13325.2)

Revised Amendments from Version 1

We thank the reviewers for their helpful and insightful comments. We have responded to each individual item from each reviewer and have included quoted changes where applicable. In summary, we have edited
1) the introduction section to more clearly relate to the main objectives of the mixed-methods systematic review, and
2) added clarifying information to the methods section. Specifically we
a) clarified information regarding the inclusion criteria (PICO);
b) added additional information about the quantitative systematic review including what study designs and summary formats were eligible and how we defined ‘health literacy’ as an exclusion outcome;
c) clarified information about the data extraction form and data collection;
d) edited the bias and quality assessments section for clarity;
e) clarified the exploration of heterogeneity/subgroup analyses;
f) elaborated and simplified the mixed methods synthesis section; and
g) added a few additional references to support our edits.

See the authors' detailed response to the review by Karin Hannes
See the authors' detailed response to the review by Ivan Buljan

Introduction

Clinical guidelines support decision making to improve patient outcomes and quality of care in a cost-effective manner¹. The development of a clinical guideline involves a rigorous synthesis of the best available evidence on a specific clinical topic. It may involve formal consensus methods with a range of multidisciplinary stakeholders^2–5. Guideline development groups comprise a range of decision makers, often including healthcare professionals, methodologists, health policymakers, clinicians, and patient representatives – all of whom have varying levels of expertise in evidence synthesis methods. This complicates the consensus process as stakeholders may prioritise and understand the findings of evidence syntheses, such as systematic reviews, differently⁶.

While the methods and recognition of the importance of systematic reviews have advanced in recent decades^7,8, there are still barriers to their creation and use^9,10. A meta-analysis of nearly 200 systematic reviews registered on the international Prospective Register of Systematic Reviews (PROSPERO) registry found that the average systematic review, from registration to publication date, takes 67.3 weeks, involves an average of five authors, and requires the full-text screening of 63 papers (range: 0–4385)⁹. The number of academic papers and systematic reviews being published in recent decades has rapidly increased^11–13, further accelerating during the recent COVID-19 (coronavirus disease) pandemic¹⁴. The expanding evidence base and acceptance of trade-offs in validity in time-sensitive matters¹⁵, has resulted in the growing popularity of other evidence synthesis methods, such as rapid reviews¹⁶. This increase in different types of evidence synthesis methods further complicates matters for guideline development groups, who may interpret different types of systematic reviews in different ways based on how familiar they might be with particular approaches.

For those using different types of evidence synthesis to inform clinical guideline development and health policy, the amount of included studies, length, and technical nature of evidence syntheses can make it difficult to find answers about the effectiveness of healthcare interventions^10,17. Previous work has highlighted that decision makers more easily understand evidence summaries than complete systematic reviews^18,19. These summaries can come in a variety of different formats such as policy briefs, one-page reports, abstracts, summary of findings tables, plain language summaries, visual abstracts or infographics, podcasts, and more. While formatting may vary, decision-makers have expressed several key preferences, such as succinct summaries highlighting contextual factors like local applicability and costs^17,20.

Succinctness should be inherent in an evidence summary, but how this distilled information is formatted and presented affects the interpretation and use of systematic reviews²¹. It is currently unclear which evidence summary format is most helpful for decision making for different guideline development group stakeholders. For example, Cochrane recommends a ‘summary of findings’ table⁷ but testing with users familiar with the Cochrane library and evidence-based practices raised concerns around comprehension and presentation of results and the balance between precision and simplicity²². Others have tested the presentation of information using different formats such as an abstract, plain-language summary, podcast or podcast transcription with no clear answer regarding which format was most suited to which stakeholder and resulted in the best understanding²³. Similarly, infographics, plain-language summaries, and traditional abstracts were found to be equally effective in transmitting knowledge to healthcare providers; however, there were differences in measures of acceptability (i.e., user-friendliness and reading experience)²⁴.

To better support clinical guideline development groups and decision-makers, it is important to identify which format works best for which stakeholder. Previous reviews have focused on identifying barriers and facilitators to use, or have been solely based on summary of findings tables^10,25. As impacts on decision-making and preferences for formats may be evaluated through different study designs, a comprehensive synthesis of the evidence is needed beyond a typical single method systematic review. Mixed methods systematic reviews (MMSR) can more easily identify discrepancies within available evidence, pinpoint how quantitative or qualitative research has focused on particular interest areas, and offer a deeper understanding of findings²⁶. A MMSR is especially useful for this project as it brings together findings of effectiveness and experience so findings are more useful for decision makers²⁷. Guideline developers need to consider diverse considerations in their work such as feasibility, priority, cost effectiveness, equity, acceptability, and patient values and preferences^28,29. Similarly, a MMSR allows us to consider and integrate data from a variety of different questions and synthesize information in a single project.

Objectives

The aim of this mixed methods systematic review is to evaluate the effectiveness of, preferences for, and attitudes towards, different communication formats of evidence summary findings amongst guideline development group members, including healthcare providers, policy makers and patient representatives. To achieve this, the proposed MMSR will answer the following questions:

1. How and to what degree do different summary formats (digital, visual, audio) of presenting evidence synthesis findings impact the end user’s understanding of the review findings?
2. What are the end users’ preferences for and attitudes towards these formats?

Protocol

The proposed systematic review will be conducted in accordance with the Joanna Briggs Institute (JBI) Manual for Evidence Synthesis which details the methodology for mixed methods systematic reviews (MMSR)²⁶.

Eligibility criteria

As this is a MMSR, we will include quantitative (i.e., randomised controlled trials), qualitative, and mixed methods studies evaluating the effectiveness and/or preferences for and attitudes towards evidence summary formats. We will exclude conference abstracts, case reports, case series, editorials, and letters. Further details regarding eligibility criteria are given within the review-relevant sections below.

We are interested in studies involving stakeholders such as policy makers, healthcare providers, and health systems managers, as well as other GDG members such as clinicians, patient representatives, and methodologists such as systematic review authors. We will exclude studies where the sole participants are students, the general population (those not involved in the clinical guideline development process), and journalists as communication to these populations is more complex given a wide variety of confounding factors. We will also exclude studies related to clinical decision-making for individual patients.

We have followed the Population, Intervention, Comparison, Outcome (PICO) format for the quantitative review (Table 1) and the Sample, Phenomenon of Interest, Design, Evaluation, Research type (SPiDER) format for the qualitative review (Table 2) and will present unique aspects of each methodological approach within the relevant sections below.

Table 1. PICO for the quantitative review of effectiveness.

Population	Members of guideline development groups (e.g., policy makers, decision makers, healthcare professionals, methodologists, patient representatives)
Intervention	A summary format which communicates evidence synthesis findings
Comparator	Alternative summary formats
Outcomes	Quantitative estimates of effectiveness and acceptability

Table 2. SPiDER for the qualitative evidence synthesis.

Sample	Members of guideline development groups (e.g., policy makers, decision makers, clinicians, methodologists, patient representatives)
Phenomenon of interest	How summary formats impact decision-making and understanding of evidence synthesis findings
Design	Focus groups, interviews, questionnaires, open-ended survey responses
Evaluation outcomes	Views, attitudes, opinions, experiences, perceptions, beliefs, feelings, understanding
Research type	Qualitative studies and mixed-methods studies with primary qualitative data collection

Quantitative systematic review. Due to the complexity of stakeholders, evidence synthesis types, and summary formats, there is a high potential that confounding factors will be extensive. Relatedly, randomised controlled trials (RCTs) are the most appropriate design to evaluate the effectiveness of the interventions in question. Thus, we chose to restrict to RCTs (e.g., parallel, crossover, cluster, stepped-wedge, etc.) only in order to focus on the performance and impact of summary formats in optimal settings. We will include studies where the intervention is any summary mode (e.g., visual, audio, text-based, etc.) which communicates the findings from an evidence synthesis study (e.g., systematic review, qualitative evidence synthesis, rapid review, etc.) to policy-makers and decision makers, including guideline development groups (GDGs). We anticipate that included summary formats may encompass visual abstracts, Summary of Findings tables, one-page summaries, podcasts, Graphical Overview of Evidence Reviews (GofER) diagrams, and others. We will not exclude a summary format if it is one that we did not explicitly list in our search strategy (Table 3). Studies in which the summaries are one component of a multi-component intervention will be excluded, as will decision aids for direct patient care.

Table 3. Ovid MEDLINE search strategy.

	Ovid MEDLINE(R) and Epub Ahead of Print, In-Process, In-Data-Review & Other Non-Indexed Citations, Daily and Versions(R) 1946 to April 13, 2021	Search results
1	exp Administrative Personnel/	41017
2	((health OR healthcare OR hospital) ADJ2 (administrator OR analyst* OR decisionmak* OR decision-mak* OR manager* OR official* OR policymak* OR policy-mak* OR policy OR policies OR provider)).tw.	73550
3	exp Decision Making/ OR Exp Policy Making/ OR exp Health Policy/	332820
4	((decision* OR policy OR policies) ADJ2 (analys* OR analyz* OR maker* OR making OR develop*)).tw.	213763
5	(analyst* OR clinician OR decision-mak* OR decisionmak* OR doctor OR guideline development group* OR advisory group OR knowledge user* OR knowledge-user* OR policy-mak* OR policymak* OR stakeholder* OR stake-holder* OR stake holder* OR end user* OR end-user*).tw.	359621
6	1 OR 2 OR 3 OR 4 OR 5	731506
7	exp Evidence-Based Practice/ OR exp "Review Literature as Topic"/ OR meta-analysis as topic/ OR exp Technology Assessment, Biomedical/	128395
8	(knowledge ADJ2 synthes*).tw.	1051
9	(meta) ADJ2 (analysis OR regression OR review OR overview OR synthes)	251572
10	meta-analy* OR meta-regression OR meta-review* OR meta-synthes* OR megasynthes*	231916
11	(evidence) ADJ2 (synthes* OR summar*)	20882
12	(quantitative OR qualitative OR systematic OR rapid OR scoping OR realist OR Cochrane OR evidence) ADJ2 (review* OR overview*)	270363
13	HTA OR health technology assessment	6744
14	7 OR 8 OR 9 OR 10 OR 11 OR 12 OR 13	530362
15	exp Data Visualization/ OR exp Health communication/ OR exp Implementation science/	3579
16	summary of findings OR summary-of-findings OR table* OR tabular	156130
17	plain-language summar* OR plain language summar*	1758
18	infographic* OR podcast* OR visual abstract* OR fact box* OR summary format OR blogshot OR blog shot OR podcast OR video OR GRADE evidence profile OR policy brief OR league table* OR bulletin OR infogram or 1-page summary OR SUPPORT summary OR brief* or summar* OR graphic* OR audio	1011424
19	(communicat* OR presentat) ADJ2 (finding)	2500
20	15 OR 16 OR 17 OR 18 OR 19	1156243
21	perceive OR understand OR understanding OR acceptability OR effectiveness OR efficacy OR satisfaction OR usability	2703824
22	usefulness OR credibility OR clarity OR comprehensive OR appeal OR appropriateness OR preference$	693195
23	21 OR 22	3247866
24	6 AND 14 AND 20 AND 23	3830

For studies examining the effectiveness of evidence summary formats, we will include any comparison to an alternative active comparator. Studies where the comparison is no intervention (e.g., the plain full-text of a manuscript) will be excluded. We do not anticipate finding evidence syntheses with no form of summary or abstract as international organisations, journals, and reporting guidelines would consider a summary to be a mandatory component of any report or peer review manuscript produced.

Our primary outcomes of interest are:

1. Effectiveness
- a. User understanding and knowledge, and/or beliefs in key findings of evidence synthesis (e.g., changes in knowledge scores about the topic included in the summary)
- b. Self-reported impact on decision‐making
- c. Intervention metrics (e.g., the time needed to read the summary, expressed language accessibility issues or scale scores)

2. Acceptability
- a. Preferences and attitudes (e.g. Likert scales reporting user satisfaction, perceptions, readability).

We will not be including outcomes related to health literacy, numeracy, nor risk communication in patient-centred care. We are aligning our definition of ‘health literacy’ with a recent systematic review on its meaning, which is complex in nature and composed of ‘(1) knowledge of health, healthcare and health systems; (2) processing and using information in various formats in relation to health and healthcare; and (3) ability to maintain health through self-management and working in partnerships with health providers.’ As impacts on one’s individual health or clinical care is not the main focus of this review, we are focusing only on one aspect (2) -- impacts on one’s understanding of knowledge which is constrained to a specific topic that an evidence summary is covering³⁰.

Qualitative evidence synthesis. Primary studies investigating the understanding and acceptability of evidence summary formats will include qualitative studies (e.g, interviews or focus groups). Mixed-methods studies with primary qualitative data collection will be included if they meet the inclusion criteria for a randomised controlled trial and where it is possible to extract the findings derived from the qualitative research. We prioritized the inclusion of qualitative data from primary studies over free text from questionnaire surveys as we hypothesized primary data would be richer and thicker and thus more informative.

Our primary outcomes of interest relate to participant’s views and experiences with summary formats. This includes their perceptions of the impact of summary formats on their understanding, knowledge, and decision making, and participant’s beliefs, attitudes, and feelings towards usability and readability.

Information sources and search strategy

The following databases will be searched from inception to May 2021: Ovid MEDLINE, EMBASE, APA PsycINFO, CINAHL (Cumulative Index to Nursing and Allied Health Literature), Web of Science, and Cochrane Library. The search strategy for Ovid MEDLINE includes a combination of keywords and medical subject headings (MeSH) terms for GDG members, evidence syntheses, and formats for the communication of findings (see Table 3). As we are looking for primary research on the impacts or effects of interventions and attitudes towards them, we do not anticipate that this literature will be found in grey literature sources such as government or agency websites. Additionally, it is anticipated that controlled trials will have short time points of assessment (and follow-up) thus we do not believe that searching registries will benefit our study. This search strategy has been informed by the strategies of similar reviews in the same topic area^10,25. Aligned with the Peer Review of Electronic Search Straggles (PRESS) Statement³¹, we engaged a medical librarian after the MEDLINE search was drafted but before it was translated to the other databases. As we are including a range of study designs, we did not apply study design specific filters. Although we have used a PICO and SPiDER approach for the quantitative and qualitative reviews we used the PICO format to inform the search strategy as previous researchers found that the SPiDER approach for search strategies may be too restrictive and specific³². Language and date restrictions will not be applied.

Backwards citation identification on all eligible studies will be performed using the citationchaser Shiny application built in R version 1.4³³. This application performs backwards citation screening (reviewing reference lists) and internally de-duplicates results. Each step of the search is summarised for transparency and references are given as a downloadable RIS file.

Data management and selection process

All citations will be downloaded and stored in Zotero reference manager version 5.0. For ease, rather than using Zotero for screening, title and abstract screening will be managed using Covidence. Two reviewers will independently screen titles and abstracts for inclusion criteria. Disagreements for inclusion will be resolved through discussions. If it is still unclear if the paper should be included, both authors will review the full version of the paper and discuss it again. If there is still disagreement, a third review author will be consulted. The screening process will be documented in the final manuscript using the Preferred Reporting Items for Systematic Reviews (PRISMA) flow diagram³⁴ and a supplemental file detailing the reason for exclusion for each individual study will be made publicly available.

Data collection

Two review authors will independently extract data from each of the included studies using a standardised data-extraction form. If there are disagreements or discrepancies, the two authors will discuss and consult with a third review author if needed. Where possible, qualitative outcomes such as themes and categories will be extracted into the standardized form. In parallel, articles containing qualitative methods will be also imported in NVivo12 for line-by-line coding for information related to outcomes. This separate but parallel data extraction is important for our analytical approach of the qualitative data which is discussed in greater detail in the Qualitative Analysis section. The following information will be extracted using the pilot-tested standardized data-extraction form

Bibliometric data (first author, title, journal, year of publication, language)
Study characteristics (setting, participants demographics, country, study design, intervention, comparators, theoretical framework, analytical approach)
Intervention characteristics will be collected following the structure of the Template for Intervention Description and Replication (TiDieR) checklist³⁵ to provide detailed information on the why, what, who, how, where, and when of the intervention described.
Primary and secondary outcomes (quantitative estimates of effectiveness and acceptability; qualitative expressions of views, attitudes, opinions, experiences, perceptions, beliefs, feelings, and understanding)
Data from the domains listed within the JBI critical appraisal tools for qualitative and quantitative studies
Funding sources

If information is missing from the study report, we will contact authors to inquire about these gaps. We will provide narrative syntheses in lieu of imputing missing data.

Bias and quality assessments

The JBI critical appraisal checklists will be used to assess the individual randomized controlled trials and qualitative studies. Two review authors will independently complete the critical appraisal checklist for each included study. Differences will be resolved through discussion and consultation with a third review author if necessary. These checklists will provide useful contextual information about the included studies such as information about performance bias. Checklist items cover things like intervention assessors and their reflexivity which are important factors to consider as participant attitudes towards summary formats may be influenced by external factors such as who created the summary (e.g., their own vs. an external organisation).

An assessment of the overall certainty of evidence using the GRADE or ConQual approach is not recommended for a JBI MMSR26. This is due to the complexities in the analysis wherein the data from separate quantitative and qualitative evidence is transformed and integrated.

If quantitative data allows for a meta-analysis, a forest plot will be generated using R. If we find a low number of studies, large treatment effects, few events per trial, or all trials are of similar sizes, we will use the Harboard test for publication bias³⁶ as it reduces the false positive rate. Egger’s test³⁷ for funnel plot asymmetry will be used to investigate small study effects and publication bias.

Quantitative analysis

A narrative synthesis will be performed, however, if appropriate, quantitative data from randomised control trials will be synthesised using meta-analysis. Heterogeneity will first be explored by assessing pertinent study characteristics that may vary across the included studies (i.e., participant group, or summary format type). If sufficient data is available, subgroup analyses (e.g., participant groups such as medical professionals versus policy makers or intervention type such as visual abstracts versus plain abstracts) will be conducted. Furthermore, statistical heterogeneity will be explored according to statistical guidance on heterogeneity⁷, an estimated I²of 50–90% represents substantial heterogeneity. We will weigh this against an χ² test for heterogeneity (<.10). If our results indicate 50% or greater and a low χ² statistic, this indicates that the heterogeneity may not be due to chance, thus we will not pool results into a meta-analysis. If data can be pooled, effect sizes and accompanying 95% confidence intervals will be reported as either relative risks (for dichotomous and dichotomised ordinal data) or standardized mean differences (for continuous data). Where data is available, we will compare and contrast reported findings on preference and whether or not preference is aligned with improvement of outcomes of impact such as knowledge.

Qualitative analysis

Where possible, qualitative findings will be pooled together using the meta-aggregation approach, which allows a reviewer to present findings of included studies as originally intended by the original authors³⁸. This approach organises and categorises findings based on similarity in meaning and avoids re-interpretation. Therefore, it does not violate paradigms and approaches used by the original study authors. This approach also enables meaningful generalizable recommendations for practitioners and policy makers³⁹. If we are unable to pool findings together (i.e., create and present categories), likely due to an insufficient number of studies identified, a narrative summary will be presented.

Mixed methods synthesis

Following JBI guidance for MMSR, we will use a convergent segregated approach that conducts separate quantitative and qualitative syntheses separately but at the same time and then integrates the findings of each^26,40. The segregated design integrates evidence through a method called configuration which essentially arranges complementary evidence into a single line of reasoning²⁶. After the separate quantitative and qualitative analyses are conducted, they will be organized into a coherent whole, as they cannot be directly combined nor can one refute the other^26,41. Converging or complementary data assumes that, while the streams of evidence may ask different research questions, they are related to different aspects or dimensions of the same phenomenon of interest²⁶. Data will be triangulated during the interpretation stage, comparing quantitative and qualitative findings side-by-side to identify areas where there is convergence, inconsistency, or contradiction in the data. We do not aim to transform the qualitative data into quantitative, nor vice versa.

There are several methods for integrating qualitative and quantitative evidence syntheses^26,42 in a convergent segregated MMSR. We will use a thematic synthesis method for integration which groups together similar codes, develops descriptive categories (or themes) to create an overall summary of findings^43,44. Initial coding will be performed independently by two authors who will meet and discuss similarities and differences in coding to start grouping them into descriptive categories. A drafted summary of findings will be created by one author, reviewed by both, and discussions will be held until a final version is agreed upon. Two authors will discuss the descriptive categories and, as a group, will draft the final analytical categories with accompanying detailed descriptions.

If we have a sufficient number of included studies for meta-analysis (minimum of three), we will report information according to participant subgroups (e.g., clinicians versus policy makers), and outcomes (e.g., understanding, acceptability, etc.).

Registration and amendments

As the focus of this review is not evaluating health-related interventions nor outcomes, we will not register the protocol on PROSPERO. However, we will preregister the study on Open Science Framework. If an amendment to this protocol is necessary, the date of each amendment will be given alongside the rationale and description of the change(s). This information will be detailed in an appendix accompanying the final systematic review publication. Changes will not be incorporated into the protocol.

Dissemination of information

Findings will be disseminated as peer-reviewed publications. Data generated from the work proposed within this protocol will be made available on the aforementioned OSF project page.

Discussion

This review will summarise the evidence on the effectiveness and acceptability of different evidence synthesis summary formats. By including a variety of evidence summary types and stakeholder participants, results can help tease apart the real-world complexity of guideline development groups and provide an overview of what summary formats work for which stakeholders in what circumstances. It is expected that review findings can support decision-making by policy-makers and GDGs, by establishing the best summary formats for presenting evidence synthesis findings.

Data availability

No data are associated with this article.

Reporting guidelines

OSF: PRISMA-P checklist for ‘Evidence synthesis summary formats for clinical guideline development group members: a mixed-methods systematic review protocol’. https://doi.org/10.17605/OSF.IO/SK4NX⁴⁵

Data are available under the terms of the Creative Commons Attribution 4.0 International license (CC-BY 4.0).

Acknowledgements

We would like to thank Paul Murphy for his help with developing the search strategy and translating it to other databases.

Faculty Opinions recommended

References

1. Eccles MP, Grimshaw JM, Shekelle P, et al.: Developing clinical practice guidelines: target audiences, identifying topics for guidelines, guideline group composition and functioning and conflicts of interest. Implement Sci. 2012; 7(1): 60. PubMed Abstract | Publisher Full Text | Free Full Text
2. Health Information and Quality Authority & National Clinical Effectiveness Committee: HIQA NCEC National Quality Assurance Criteria NCG v2 - April 2015. 2015; [cited 2020 Jul 8]. Reference Source
3. Qaseem A, Forland F, Macbeth F, et al.: Guidelines International Network: Toward International Standards for Clinical Practice Guidelines. Ann Intern Med. 2012; 156(7): 525–31. PubMed Abstract | Publisher Full Text
4. Schünemann HJ, Wiercioch W, Etxeandia I, et al.: Guidelines 2.0: systematic development of a comprehensive checklist for a successful guideline enterprise. CMAJ. 2014; 186(3): E123–42. PubMed Abstract | Publisher Full Text | Free Full Text
5. Fretheim A, Schünemann HJ, Oxman AD: Improving the use of research evidence in guideline development: 3. Group composition and consultation process. Health Res Policy Syst. 2006; 4(1): 15. PubMed Abstract | Publisher Full Text | Free Full Text
6. Wieringa S, Engebretsen E, Heggen K, et al.: Clinical guidelines and the pursuit of reducing epistemic uncertainty. An ethnographic study of guideline development panels in three countries. Soc Sci Med. 2021; 272: 113702. PubMed Abstract | Publisher Full Text
7. Higgins JPT, Thomas J, Chandler J, et al.: Cochrane Handbook for Systematic Reviews of Interventions. Cochrane. 2020; 6.1: [cited 2021 Feb 23]. Reference Source
8. Chandler J, Hopewell S: Cochrane methods -- twenty years experience in developing systematic review methods. Syst Rev. 2013; 2(1): 76. PubMed Abstract | Publisher Full Text | Free Full Text
9. Borah R, Brown AW, Capers PL, et al.: Analysis of the time and workers needed to conduct systematic reviews of medical interventions using data from the PROSPERO registry. BMJ Open. 2017; 7(2): e012545. PubMed Abstract | Publisher Full Text | Free Full Text
10. Tricco AC, Cardoso R, Thomas SM, et al.: Barriers and facilitators to uptake of systematic reviews by policy makers and health care managers: a scoping review. Implement Sci. 2016; 11(1): 4. PubMed Abstract | Publisher Full Text | Free Full Text
11. Fontelo P, Liu F: A review of recent publication trends from top publishing countries. Syst Rev. 2018; 7(1): 147. PubMed Abstract | Publisher Full Text | Free Full Text
12. Bastian H, Glasziou P, Chalmers I: Seventy-Five Trials and Eleven Systematic Reviews a Day: How Will We Ever Keep Up? PLoS Med. 2010; 7(9): e1000326. PubMed Abstract | Publisher Full Text | Free Full Text
13. Page MJ, Shamseer L, Altman DG, et al.: Epidemiology and Reporting Characteristics of Systematic Reviews of Biomedical Research: A Cross-Sectional Study. PLoS Med. 2016; 13(5): e1002028. PubMed Abstract | Publisher Full Text | Free Full Text
14. Tangcharoensathien V, Calleja N, Nguyen T, et al.: Framework for Managing the COVID-19 Infodemic: Methods and Results of an Online, Crowdsourced WHO Technical Consultation. J Med Internet Res. 2020; 22(6): e19659. PubMed Abstract | Publisher Full Text | Free Full Text
15. Wagner G, Nussbaumer-Streit B, Greimel J, et al.: Trading certainty for speed - how much uncertainty are decisionmakers and guideline developers willing to accept when using rapid reviews: an international survey. BMC Med Res Methodol. 2017; 17(1): 121. PubMed Abstract | Publisher Full Text | Free Full Text
16. Tricco AC, Garritty CM, Boulos L, et al.: Rapid review methods more challenging during COVID-19: Commentary with a focus on 8 knowledge synthesis steps. J Clin Epidemiol. 2020; 126: 177–183. PubMed Abstract | Publisher Full Text | Free Full Text
17. Oliver K, Innvar S, Lorenc T, et al.: A systematic review of barriers to and facilitators of the use of evidence by policymakers. BMC Health Serv Res. 2014; 14(1): 2. PubMed Abstract | Publisher Full Text | Free Full Text
18. Perrier L, Mrklas K, Lavis JN, et al.: Interventions encouraging the use of systematic reviews by health policymakers and managers: A systematic review. Implement Sci. 2011; 6(1): 43. PubMed Abstract | Publisher Full Text | Free Full Text
19. Petkovic J, Welch V, Jacob MH, et al.: The effectiveness of evidence summaries on health policymakers and health system managers use of evidence from systematic reviews: a systematic review. Implement Sci. 2016; 11(1): 162. PubMed Abstract | Publisher Full Text | Free Full Text
20. Wallace J, Byrne C, Clarke M: Making evidence more wanted: a systematic review of facilitators to enhance the uptake of evidence from systematic reviews and meta-analyses. Int J Evid Based Healthc. 2012; 10(4): 338–46. PubMed Abstract | Publisher Full Text
21. Marquez C, Johnson AM, Jassemi S, et al.: Enhancing the uptake of systematic reviews of effects: what is the best format for health care managers and policy-makers? A mixed-methods study. Implement Sci. 2018; 13(1): 84. PubMed Abstract | Publisher Full Text | Free Full Text
22. Rosenbaum SE, Glenton C, Nylund HK, et al.: User testing and stakeholder feedback contributed to the development of understandable and useful Summary of Findings tables for Cochrane reviews. J Clin Epidemiol. 2010; 63(6): 607–19. PubMed Abstract | Publisher Full Text
23. Maguire LK, Clarke M: How much do you need: a randomised experiment of whether readers can understand the key messages from summaries of Cochrane Reviews without reading the full review. J R Soc Med. 2014; 107(11): 444–9. PubMed Abstract | Publisher Full Text | Free Full Text
24. Buljan I, Malički M, Wager E, et al.: No difference in knowledge obtained from infographic or plain language summary of a Cochrane systematic review: three randomized controlled trials. J Clin Epidemiol. 2018; 97: 86–94. PubMed Abstract | Publisher Full Text
25. Conway A, Clarke MJ, Treweek S, et al.: Summary of findings tables for communicating key findings of systematic reviews. Cochrane Database Syst Rev. 2017; (2). Publisher Full Text
26. Lizarondo L, Stern C, Carrier J, et al.: Chapter 8: Mixed Methods Systematic Reviews. In: Aromataris E, Munn Z, editors. JBI Manual for Evidence Synthesis. Joanna Briggs Institute; 2020; [cited 2021 Feb 23]. Publisher Full Text
27. Bressan V, Bagnasco A, Aleo G, et al.: Mixed-methods research in nursing - a critical review. J Clin Nurs. 2017; 26(19–20): 2878–90. PubMed Abstract | Publisher Full Text
28. Alonso-Coello P, Schünemann HJ, Moberg J, et al.: GRADE Evidence to Decision (EtD) frameworks: a systematic and transparent approach to making well informed healthcare choices. 1: Introduction. BMJ. 2016; 353: i2016. PubMed Abstract | Publisher Full Text
29. Zhang Y, Alonso-Coello P, Guyatt GH, et al.: GRADE Guidelines: 19. Assessing the certainty of evidence in the importance of outcomes or values and preferences-Risk of bias and indirectness. J Clin Epidemiol. 2019; 111: 94–104. PubMed Abstract | Publisher Full Text
30. Liu C, Wang D, Liu C, et al.: What is the meaning of health literacy? A systematic review and qualitative synthesis. Fam Med Community Health. 2020; 8(2): e000351. PubMed Abstract | Publisher Full Text | Free Full Text
31. McGowan J, Sampson M, Salzwedel DM, et al.: PRESS Peer Review of Electronic Search Strategies: 2015 Guideline Statement. J Clin Epidemiol. 2016; 75: 40–6. PubMed Abstract | Publisher Full Text
32. Methley AM, Campbell S, Chew-Graham C, et al.: PICO, PICOS and SPIDER: a comparison study of specificity and sensitivity in three search tools for qualitative systematic reviews. BMC Health Serv Res. 2014; 14: 579. PubMed Abstract | Publisher Full Text | Free Full Text
33. Haddaway NR: citationchaser: an R package for forward and backward citations chasing in academic searching. Zenodo. 2021; [cited 2021 Feb 15]. Publisher Full Text
34. Page MJ, McKenzie JE, Bossuyt PM, et al.: The PRISMA 2020 statement: An updated guideline for reporting systematic reviews. J Clin Epidemiol. 2021; 134: 178–189. PubMed Abstract | Publisher Full Text
35. Hoffmann TC, Glasziou PP, Boutron I, et al.: Better reporting of interventions: template for intervention description and replication (TIDieR) checklist and guide. BMJ. 2014; 348: g1687. PubMed Abstract | Publisher Full Text
36. Harbord RM, Egger M, Sterne JAC: A modified test for small-study effects in meta-analyses of controlled trials with binary endpoints. Stat Med. 2006; 25(20): 3443–57. PubMed Abstract | Publisher Full Text
37. Egger M, Smith GD, Schneider M, et al.: Bias in meta-analysis detected by a simple, graphical test. BMJ. 1997; 315(7109): 629–34. PubMed Abstract | Publisher Full Text | Free Full Text
38. Lockwood C, Munn Z, Porritt K: Qualitative research synthesis: methodological guidance for systematic reviewers utilizing meta-aggregation. Int J Evid Based Healthc. 2015; 13(3): 179–87. PubMed Abstract | Publisher Full Text
39. Hannes K, Lockwood C: Pragmatism as the philosophical foundation for the Joanna Briggs meta-aggregative approach to qualitative evidence synthesis. J Adv Nurs. 2011; 67(7): 1632–42. PubMed Abstract | Publisher Full Text
40. Hong QN, Pluye P, Bujold M, et al.: Convergent and sequential synthesis designs: implications for conducting and reporting systematic reviews of qualitative and quantitative evidence. Syst Rev. 2017; 6(1): 61. PubMed Abstract | Publisher Full Text | Free Full Text
41. Sandelowski M, Voils CI, Barroso J: Defining and Designing Mixed Research Synthesis Studies. Res Sch. 2006; 13(1): 29. PubMed Abstract | Free Full Text
42. 8.2 Concepts and considerations for mixed methods systematic reviews. JBI Manual for Evidence Synthesis. 2019. Reference Source
43. Thomas J, Harden A: Methods for the thematic synthesis of qualitative research in systematic reviews. BMC Med Res Methodol. 2008; 8: 45. PubMed Abstract | Publisher Full Text | Free Full Text
44. Braun V, Clarke V: Using thematic analysis in psychology. Qual Res Psychol. 2006; 3(2): 77–101. Publisher Full Text
45. Clyne B, Sharp MK: Evidence synthesis and translation of findings for national clinical guideline development: addressing the needs and preferences of guideline development groups. 2021. http://www.doi.org/10.17605/OSF.IO/SK4NX

Comments on this article Comments (1)

Version 2

VERSION 2 PUBLISHED 10 May 2022

Revised

Comment

Version 1

VERSION 1 PUBLISHED 15 Jul 2021

Discussion is closed on this version, please comment on the latest version above.

Author Response 05 May 2022

Melissa Sharp, Health Research Board Centre for Primary Care Research, Department of General Practice,, Royal College of Surgeons in Ireland, Dublin, Ireland

05 May 2022

Author Response

We thank the reviewers for their helpful and insightful comments. We have responded to each individual item from each reviewer and have included quoted changes where applicable. In summary, we ... Continue reading We thank the reviewers for their helpful and insightful comments. We have responded to each individual item from each reviewer and have included quoted changes where applicable. In summary, we have edited
1) the introduction section to more clearly relate to the main objectives of the mixed-methods systematic review, and
2) added clarifying information to the methods section. Specifically we
a) clarified information regarding the inclusion criteria (PICO);
b) added additional information about the quantitative systematic review including what study designs and summary formats were eligible and how we defined ‘health literacy’ as an exclusion outcome;
c) clarified information about the data extraction form and data collection;
d) edited the bias and quality assessments section for clarity;
e) clarified the exploration of heterogeneity/subgroup analyses;
f) elaborated and simplified the mixed methods synthesis section; and
g) added a few additional references to support our edits.
We thank the reviewers for their helpful and insightful comments. We have responded to each individual item from each reviewer and have included quoted changes where applicable. In summary, we have edited
1) the introduction section to more clearly relate to the main objectives of the mixed-methods systematic review, and
2) added clarifying information to the methods section. Specifically we
a) clarified information regarding the inclusion criteria (PICO);
b) added additional information about the quantitative systematic review including what study designs and summary formats were eligible and how we defined ‘health literacy’ as an exclusion outcome;
c) clarified information about the data extraction form and data collection;
d) edited the bias and quality assessments section for clarity;
e) clarified the exploration of heterogeneity/subgroup analyses;
f) elaborated and simplified the mixed methods synthesis section; and
g) added a few additional references to support our edits.
Competing Interests: No competing interests were disclosed. Close
Report a concern
Discussion is closed on this version, please comment on the latest version above.

Author details Author details

Barrie Tyner
Roles: Methodology, Writing – Review & Editing

Dayang Anis Binti Awang Baki
Roles: Writing – Review & Editing

Cormac Farrell
Roles: Writing – Review & Editing

Declan Devane
Roles: Methodology, Writing – Review & Editing

Kamal R. Mahtani
Roles: Methodology, Writing – Review & Editing

Susan M. Smith
Roles: Methodology, Writing – Review & Editing

Michelle O'Neill
Roles: Conceptualization, Funding Acquisition, Methodology, Writing – Review & Editing

Máirín Ryan
Roles: Conceptualization, Funding Acquisition, Methodology, Writing – Review & Editing

Barbara Clyne
Roles: Conceptualization, Funding Acquisition, Methodology, Project Administration, Supervision, Writing – Original Draft Preparation, Writing – Review & Editing

Competing interests

No competing interests were disclosed.

Grant information

Health Research Board Ireland [EIA-2019-09].
The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Article Versions (2)

version 2

Revised

Published: 10 May 2022, 4:76

https://doi.org/10.12688/hrbopenres.13325.2

version 1

Published: 15 Jul 2021, 4:76

https://doi.org/10.12688/hrbopenres.13325.1

© 2022 Sharp MK et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Download

Export To

metrics

VIEWS

1405

downloads

Citations

SEE MORE DETAILS

CITE

how to cite this article

Sharp MK, Tyner B, Awang Baki DAB et al. Evidence synthesis summary formats for clinical guideline development group members: a mixed-methods systematic review protocol [version 2; peer review: 2 approved]. HRB Open Res 2022, 4:76 (https://doi.org/10.12688/hrbopenres.13325.2)

NOTE: If applicable, it is important to ensure the information in square brackets after the title is included in all citations of this article.

track

receive updates on this article

Track an article to receive email alerts on any updates to this article.

Open Peer Review

Version 2

VERSION 2

PUBLISHED 10 May 2022

Revised

Views

Reviewer Report 31 Aug 2022

Karin Hannes, Research group SoMeTHin'K (Social, Methodological and Theoretical Innovation Kreative), CESO, Faculty of Social Sciences, KU Leuven, Leuven, Belgium

Approved

https://doi.org/10.21956/hrbopenres.14775.r32053

I am happy with ... Continue reading

CITE

Report a concern

Respond or Comment

Version 1

VERSION 1

PUBLISHED 15 Jul 2021

Views

Reviewer Report 15 Feb 2022

Ivan Buljan, Department of Research in Biomedicine and Health, University of Split School of Medicine, Split, Croatia

Approved

https://doi.org/10.21956/hrbopenres.14507.r29873

Thank you for the opportunity to review this interesting protocol.

Overall, the research plan is sound and described in detail. I have only a few minor suggestions to propose to make the protocol more precise.

In Methods sections, under Eligibility criteria, I propose that you list all research designs that will be included in the analysis. Currently, only RCT-s are included, but it is not stated will all types of RCT-s will be included (parallel, crossover, etc). On top of that, what about non-randomized trials and control before and after studies? Will those studies be included, too?

The sentence: "We will not be including outcomes related to health literacy, numeracy, nor risk communication in patient-centred care." is a bit problematic. Based on my understanding, health literacy is the examination of knowledge and/or understanding of health information, which will be the case of most research studies you include in the review. Maybe you wanted to state that you will not include studies that assessed health literacy with specific tests of health literacy?

Would you consider including Open Science Framework registry as one of the databases in which you would perform a literature search?

Minor points:

"If we have a sufficient number of included studies, subgroup analyses will be performed to investigate differences based on participant groups (e.g., clinicians versus policy makers) and outcomes (e.g., understanding, acceptability, etc.)" What would be considered a sufficient number of studies?

I hope those comments will help you in the process.

Is the rationale for, and objectives of, the study clearly described?

Yes
Is the study design appropriate for the research question?

Yes
Are sufficient details of the methods provided to allow replication by others?

Partly
Are the datasets clearly presented in a useable and accessible format?

Not applicable

Competing Interests: No competing interests were disclosed.

Reviewer Expertise: Public health; Health communication

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

CITE

Report a concern

Author Response 05 May 2022

Melissa Sharp, Health Research Board Centre for Primary Care Research, Department of General Practice,, Royal College of Surgeons in Ireland, Dublin, Ireland

05 May 2022

Author Response

Thank you for the opportunity to review this interesting protocol.
Overall, the research plan is sound and described in detail. I have only a few minor suggestions to propose to ... Continue reading Thank you for the opportunity to review this interesting protocol.
Overall, the research plan is sound and described in detail. I have only a few minor suggestions to propose to make the protocol more precise.
Author’s response: Thank you for the time spent reviewing our manuscript and for your valuable feedback.

In Methods sections, under Eligibility criteria, I propose that you list all research designs that will be included in the analysis. Currently, only RCT-s are included, but it is not stated will all types of RCT-s will be included (parallel, crossover, etc). On top of that, what about non-randomized trials and control before and after studies? Will those studies be included, too?
Author’s response: Thank you for this question. We have edited the text to explicitly given examples of the types of RCTs that will be included ‘(e.g., parallel, crossover, cluster, stepped-wedge, etc.)’ The other reviewer had a related question regarding inclusion of different study designs. We have chosen to restrict study designs for this review to randomized controlled trials and qualitative studies as we believe these study designs are best able to either 1) control for confounding/interacting factors that could influence one’s attitude towards a format, such as who is disseminating the message (RCTs) or 2) better able to fully explore multiple factors influencing one’s perception (qualitative studies). We chose not to include non-randomised designs or observational studies due to the complexity of factors that influence one’s attitude towards a format. These study designs also relate directly to the objectives of the review (i.e., effectiveness and preferences for/attitudes towards formats).

The sentence: "We will not be including outcomes related to health literacy, numeracy, nor risk communication in patient-centred care." is a bit problematic. Based on my understanding, health literacy is the examination of knowledge and/or understanding of health information, which will be the case of most research studies you include in the review. Maybe you wanted to state that you will not include studies that assessed health literacy with specific tests of health literacy?
Author’s response: Thank you for this comment. We believe that health literacy outcomes are quite distinct from knowledge/understanding outcomes. We recognize that the definition of health literacy has evolved over time and can often mean different things to different people so we have added additional information to the end of the outcomes section to clarify the focus of our project and the outcomes we are including. We have added text to the manuscript to clarify ‘We are aligning our definition of ‘health literacy’ with a recent systematic review on its meaning, which is complex in nature and composed of ‘(1) knowledge of health, healthcare and health systems; (2) processing and using information in various formats in relation to health and healthcare; and (3) ability to maintain health through self-management and working in partnerships with health providers.’ As impacts on one’s individual health or clinical care is not the main focus of this review, we are focusing only on one aspect (2) -- impacts on one’s understanding of knowledge which is constrained to a specific topic that an evidence summary is covering. [1]

Would you consider including Open Science Framework registry as one of the databases in which you would perform a literature search?
Author’s response: Thank you for this suggestion. We did not search Open Science Framework as this registry is more often suited for observational studies and qualitative study protocols are still rare in the published literature. The search functionality of OSF (using Lucene) is also quite limited. For example using ‘evidence synthesis summary formats’ would result in over 23,000 registrations to screen. We did not search any registries for this study and will add this as a limitation in the discussion section of the final manuscript detailing the MMSR findings.

Minor points:
"If we have a sufficient number of included studies, subgroup analyses will be performed to investigate differences based on participant groups (e.g., clinicians versus policy makers) and outcomes (e.g., understanding, acceptability, etc.)" What would be considered a sufficient number of studies?
Author’s response: Thank you for this question. To our knowledge there is no numeric threshold for subgroup analyses for a MMSR. We edited the text to say ‘If we have a sufficient number of included studies for meta-analysis (minimum of 3), we will report information according to participant subgroups (e.g., clinicians versus policy makers), and outcomes (e.g., understanding, acceptability, etc.).’ as the analysis will be descriptive anyway.
Thank you for the opportunity to review this interesting protocol.
Overall, the research plan is sound and described in detail. I have only a few minor suggestions to propose to make the protocol more precise.
Author’s response: Thank you for the time spent reviewing our manuscript and for your valuable feedback.

In Methods sections, under Eligibility criteria, I propose that you list all research designs that will be included in the analysis. Currently, only RCT-s are included, but it is not stated will all types of RCT-s will be included (parallel, crossover, etc). On top of that, what about non-randomized trials and control before and after studies? Will those studies be included, too?
Author’s response: Thank you for this question. We have edited the text to explicitly given examples of the types of RCTs that will be included ‘(e.g., parallel, crossover, cluster, stepped-wedge, etc.)’ The other reviewer had a related question regarding inclusion of different study designs. We have chosen to restrict study designs for this review to randomized controlled trials and qualitative studies as we believe these study designs are best able to either 1) control for confounding/interacting factors that could influence one’s attitude towards a format, such as who is disseminating the message (RCTs) or 2) better able to fully explore multiple factors influencing one’s perception (qualitative studies). We chose not to include non-randomised designs or observational studies due to the complexity of factors that influence one’s attitude towards a format. These study designs also relate directly to the objectives of the review (i.e., effectiveness and preferences for/attitudes towards formats).

The sentence: "We will not be including outcomes related to health literacy, numeracy, nor risk communication in patient-centred care." is a bit problematic. Based on my understanding, health literacy is the examination of knowledge and/or understanding of health information, which will be the case of most research studies you include in the review. Maybe you wanted to state that you will not include studies that assessed health literacy with specific tests of health literacy?
Author’s response: Thank you for this comment. We believe that health literacy outcomes are quite distinct from knowledge/understanding outcomes. We recognize that the definition of health literacy has evolved over time and can often mean different things to different people so we have added additional information to the end of the outcomes section to clarify the focus of our project and the outcomes we are including. We have added text to the manuscript to clarify ‘We are aligning our definition of ‘health literacy’ with a recent systematic review on its meaning, which is complex in nature and composed of ‘(1) knowledge of health, healthcare and health systems; (2) processing and using information in various formats in relation to health and healthcare; and (3) ability to maintain health through self-management and working in partnerships with health providers.’ As impacts on one’s individual health or clinical care is not the main focus of this review, we are focusing only on one aspect (2) -- impacts on one’s understanding of knowledge which is constrained to a specific topic that an evidence summary is covering. [1]

Would you consider including Open Science Framework registry as one of the databases in which you would perform a literature search?
Author’s response: Thank you for this suggestion. We did not search Open Science Framework as this registry is more often suited for observational studies and qualitative study protocols are still rare in the published literature. The search functionality of OSF (using Lucene) is also quite limited. For example using ‘evidence synthesis summary formats’ would result in over 23,000 registrations to screen. We did not search any registries for this study and will add this as a limitation in the discussion section of the final manuscript detailing the MMSR findings.

Minor points:
"If we have a sufficient number of included studies, subgroup analyses will be performed to investigate differences based on participant groups (e.g., clinicians versus policy makers) and outcomes (e.g., understanding, acceptability, etc.)" What would be considered a sufficient number of studies?
Author’s response: Thank you for this question. To our knowledge there is no numeric threshold for subgroup analyses for a MMSR. We edited the text to say ‘If we have a sufficient number of included studies for meta-analysis (minimum of 3), we will report information according to participant subgroups (e.g., clinicians versus policy makers), and outcomes (e.g., understanding, acceptability, etc.).’ as the analysis will be descriptive anyway.
Competing Interests: No competing interests were disclosed. Close
Report a concern
Respond or Comment

COMMENTS ON THIS REPORT

Author Response 05 May 2022

Melissa Sharp, Health Research Board Centre for Primary Care Research, Department of General Practice,, Royal College of Surgeons in Ireland, Dublin, Ireland

05 May 2022

Author Response

Thank you for the opportunity to review this interesting protocol.
Overall, the research plan is sound and described in detail. I have only a few minor suggestions to propose to ... Continue reading Thank you for the opportunity to review this interesting protocol.
Overall, the research plan is sound and described in detail. I have only a few minor suggestions to propose to make the protocol more precise.
Author’s response: Thank you for the time spent reviewing our manuscript and for your valuable feedback.

In Methods sections, under Eligibility criteria, I propose that you list all research designs that will be included in the analysis. Currently, only RCT-s are included, but it is not stated will all types of RCT-s will be included (parallel, crossover, etc). On top of that, what about non-randomized trials and control before and after studies? Will those studies be included, too?
Author’s response: Thank you for this question. We have edited the text to explicitly given examples of the types of RCTs that will be included ‘(e.g., parallel, crossover, cluster, stepped-wedge, etc.)’ The other reviewer had a related question regarding inclusion of different study designs. We have chosen to restrict study designs for this review to randomized controlled trials and qualitative studies as we believe these study designs are best able to either 1) control for confounding/interacting factors that could influence one’s attitude towards a format, such as who is disseminating the message (RCTs) or 2) better able to fully explore multiple factors influencing one’s perception (qualitative studies). We chose not to include non-randomised designs or observational studies due to the complexity of factors that influence one’s attitude towards a format. These study designs also relate directly to the objectives of the review (i.e., effectiveness and preferences for/attitudes towards formats).

The sentence: "We will not be including outcomes related to health literacy, numeracy, nor risk communication in patient-centred care." is a bit problematic. Based on my understanding, health literacy is the examination of knowledge and/or understanding of health information, which will be the case of most research studies you include in the review. Maybe you wanted to state that you will not include studies that assessed health literacy with specific tests of health literacy?
Author’s response: Thank you for this comment. We believe that health literacy outcomes are quite distinct from knowledge/understanding outcomes. We recognize that the definition of health literacy has evolved over time and can often mean different things to different people so we have added additional information to the end of the outcomes section to clarify the focus of our project and the outcomes we are including. We have added text to the manuscript to clarify ‘We are aligning our definition of ‘health literacy’ with a recent systematic review on its meaning, which is complex in nature and composed of ‘(1) knowledge of health, healthcare and health systems; (2) processing and using information in various formats in relation to health and healthcare; and (3) ability to maintain health through self-management and working in partnerships with health providers.’ As impacts on one’s individual health or clinical care is not the main focus of this review, we are focusing only on one aspect (2) -- impacts on one’s understanding of knowledge which is constrained to a specific topic that an evidence summary is covering. [1]

Would you consider including Open Science Framework registry as one of the databases in which you would perform a literature search?
Author’s response: Thank you for this suggestion. We did not search Open Science Framework as this registry is more often suited for observational studies and qualitative study protocols are still rare in the published literature. The search functionality of OSF (using Lucene) is also quite limited. For example using ‘evidence synthesis summary formats’ would result in over 23,000 registrations to screen. We did not search any registries for this study and will add this as a limitation in the discussion section of the final manuscript detailing the MMSR findings.

Minor points:
"If we have a sufficient number of included studies, subgroup analyses will be performed to investigate differences based on participant groups (e.g., clinicians versus policy makers) and outcomes (e.g., understanding, acceptability, etc.)" What would be considered a sufficient number of studies?
Author’s response: Thank you for this question. To our knowledge there is no numeric threshold for subgroup analyses for a MMSR. We edited the text to say ‘If we have a sufficient number of included studies for meta-analysis (minimum of 3), we will report information according to participant subgroups (e.g., clinicians versus policy makers), and outcomes (e.g., understanding, acceptability, etc.).’ as the analysis will be descriptive anyway.
Thank you for the opportunity to review this interesting protocol.
Overall, the research plan is sound and described in detail. I have only a few minor suggestions to propose to make the protocol more precise.
Author’s response: Thank you for the time spent reviewing our manuscript and for your valuable feedback.

In Methods sections, under Eligibility criteria, I propose that you list all research designs that will be included in the analysis. Currently, only RCT-s are included, but it is not stated will all types of RCT-s will be included (parallel, crossover, etc). On top of that, what about non-randomized trials and control before and after studies? Will those studies be included, too?
Author’s response: Thank you for this question. We have edited the text to explicitly given examples of the types of RCTs that will be included ‘(e.g., parallel, crossover, cluster, stepped-wedge, etc.)’ The other reviewer had a related question regarding inclusion of different study designs. We have chosen to restrict study designs for this review to randomized controlled trials and qualitative studies as we believe these study designs are best able to either 1) control for confounding/interacting factors that could influence one’s attitude towards a format, such as who is disseminating the message (RCTs) or 2) better able to fully explore multiple factors influencing one’s perception (qualitative studies). We chose not to include non-randomised designs or observational studies due to the complexity of factors that influence one’s attitude towards a format. These study designs also relate directly to the objectives of the review (i.e., effectiveness and preferences for/attitudes towards formats).

The sentence: "We will not be including outcomes related to health literacy, numeracy, nor risk communication in patient-centred care." is a bit problematic. Based on my understanding, health literacy is the examination of knowledge and/or understanding of health information, which will be the case of most research studies you include in the review. Maybe you wanted to state that you will not include studies that assessed health literacy with specific tests of health literacy?
Author’s response: Thank you for this comment. We believe that health literacy outcomes are quite distinct from knowledge/understanding outcomes. We recognize that the definition of health literacy has evolved over time and can often mean different things to different people so we have added additional information to the end of the outcomes section to clarify the focus of our project and the outcomes we are including. We have added text to the manuscript to clarify ‘We are aligning our definition of ‘health literacy’ with a recent systematic review on its meaning, which is complex in nature and composed of ‘(1) knowledge of health, healthcare and health systems; (2) processing and using information in various formats in relation to health and healthcare; and (3) ability to maintain health through self-management and working in partnerships with health providers.’ As impacts on one’s individual health or clinical care is not the main focus of this review, we are focusing only on one aspect (2) -- impacts on one’s understanding of knowledge which is constrained to a specific topic that an evidence summary is covering. [1]

Would you consider including Open Science Framework registry as one of the databases in which you would perform a literature search?
Author’s response: Thank you for this suggestion. We did not search Open Science Framework as this registry is more often suited for observational studies and qualitative study protocols are still rare in the published literature. The search functionality of OSF (using Lucene) is also quite limited. For example using ‘evidence synthesis summary formats’ would result in over 23,000 registrations to screen. We did not search any registries for this study and will add this as a limitation in the discussion section of the final manuscript detailing the MMSR findings.

Minor points:
"If we have a sufficient number of included studies, subgroup analyses will be performed to investigate differences based on participant groups (e.g., clinicians versus policy makers) and outcomes (e.g., understanding, acceptability, etc.)" What would be considered a sufficient number of studies?
Author’s response: Thank you for this question. To our knowledge there is no numeric threshold for subgroup analyses for a MMSR. We edited the text to say ‘If we have a sufficient number of included studies for meta-analysis (minimum of 3), we will report information according to participant subgroups (e.g., clinicians versus policy makers), and outcomes (e.g., understanding, acceptability, etc.).’ as the analysis will be descriptive anyway.
Competing Interests: No competing interests were disclosed. Close
Report a concern

Views

Reviewer Report 20 Sep 2021

Karin Hannes, Research group SoMeTHin'K (Social, Methodological and Theoretical Innovation Kreative), CESO, Faculty of Social Sciences, KU Leuven, Leuven, Belgium

Approved with Reservations

https://doi.org/10.21956/hrbopenres.14507.r29872

This is a review underpinned by clear and interesting questions, that warrant a mixed method strategy. There is room for improvement though on the level of motivating choices for concepts and finetuning methods.

The biggest issue I have is on the relation between content and form in the choice for the mediums to be compared. The authors tend to compare forms (?), but attitude towards form is not fully independent to the way messages are framed, nor by whom they are disseminated. How do the authors intend to deal with this in their comparison? (e.g. people might prefer a form such as video but it might fail to bring the message across, as participants are distracted by the form).

I was not fully sure why things like surveys measuring attitudes and opinions of people were not taken along in this review. (I am aware of the fact that guidance on the inclusion of such designs in reviews is scarce though-consider it a free offer to work towards a more comprehensive review type). Particularly, because in the RCTs authors tend to rely on self-reported measures, with LIKERT scales. Here, a comparison could sit with time, rather than comparator. Does people's attitude change when exposed to different formats over time?

Sometimes the arguments are not fully clear. In the introduction, attention is paid to problems complicating the conduct of systematic reviews. This is not the core focus of the project. The project is linked to the translation phase of turning review findings into evidence summaries. This is a matter of content, form and channels rather than the SR pre-track issues. It would therefore be more appropriate to highlight the diversity of actors involved in producing guidelines in relation to how communication and dissemination channels have changed because of that. I am a big fan of the authors discussing things like podcasts and multimedia accounts as examples. Naturally, this is linked to a society moving into multimodality and the limitations of numbers and narratives in reaching out to or working with different publics.

The PICO would perhaps work better if clinicians is changed into professionals. It would make it relevant to a broader target group. Also, the comparator could also be no summary at all, for example, reading findings from reviews straight into the guideline procedure. Why are these explicitly excluded from the review?

Do the authors only include qualitative studies where an intervention is evaluated by participants in terms of meaningfulness or applicability? Or do they also include qualitative studies where attitudes, viewpoints and options for form/channel/content are discussed from a more theoretical point of view, or in pilot cases? It is not clear where the selection sits on the qualitative side, but the data collection part suggests the first option.

Perhaps the authors may want to bring a rationale for why they want to limit the formats to textual and audio-visual productions, as there are many more forms available through which evidence is and can be communicated (such as installations, theatre, etc.) Perhaps these forms haven't been studied extensively in the context of guidelines, but in a protocol phase we are not sure what to find yet :-).

I would find it more practical if the authors would discuss their position towards critical appraisal and how to structure this instead of putting it in the data collection part as an extraction category (it is not clear what will actually be extracted for what purpose). Also, critical appraisal checklist for primary studies have a different purpose than GRADE and CONQUAL that work on a review level (which is clearly not the focus here when talking about the pool of studies included). I think the authors need to spell the two out in more detail, as they are used in different phases of the review process.

Would quality of a study not rather fit a sensitivity analysis rather than a subgroup analysis? What purpose would it have as a subgroup in the review? It has disappeared in the mixed method part as a subgroup analysis, so wouldn't mention it as a subgroup in other parts of the text.

What do the authors mean with the phrase "If textual pooling is not available, a narrative summary will be presented." Is a narrative summary not already a textual pooling? Unclear.

While configuration might have different meanings, it is generally not commonly used as a metaphor for a separate type of synthesis. Configuration would make the different parts unrecognizable in the whole. I don't think meta-aggregation would achieve this level of integration. Perhaps integration is better than configuration here.

Also, thematic synthesis is not the same as meta-aggregation, and line by line coding is certainly not a part of meta-aggregation. I believe the authors need to make an informed choice between qualitative evidence synthesis approaches and then follow the guidance through.

Is the rationale for, and objectives of, the study clearly described?

Yes
Is the study design appropriate for the research question?

Yes
Are sufficient details of the methods provided to allow replication by others?

No
Are the datasets clearly presented in a useable and accessible format?

Not applicable

Competing Interests: No competing interests were disclosed.

Reviewer Expertise: Methods

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.

CITE

Report a concern

Author Response 05 May 2022

Melissa Sharp, Health Research Board Centre for Primary Care Research, Department of General Practice,, Royal College of Surgeons in Ireland, Dublin, Ireland

05 May 2022

Author Response
This is a review underpinned by clear and interesting questions, that warrant a mixed method strategy. There is room for improvement though on the level of motivating choices for concepts ... Continue reading
This is a review underpinned by clear and interesting questions, that warrant a mixed method strategy. There is room for improvement though on the level of motivating choices for concepts and finetuning methods.
Author’s response: Thank you for your review and thoughtful suggestions for our manuscript.

The biggest issue I have is on the relation between content and form in the choice for the mediums to be compared. The authors tend to compare forms (?), but attitude towards form is not fully independent to the way messages are framed, nor by whom they are disseminated. How do the authors intend to deal with this in their comparison? (e.g. people might prefer a form such as video but it might fail to ring the message across, as participants are distracted by the form).
Author’s response: Thank you for this comment. This is an interesting point, the studies we include in our review may well demonstrate a disconnect between what formats participants prefer, and what formats may actually produce improvement in outcomes such as knowledge. If included studies present this data we will be able to explore this more detail in our analysis but we suspect from the studies we have seen that this level of detail will not be available. Based on the importance of this suggestion, we will include this as a point of consideration in the discussion in the main manuscript in the protocol, we have made the following amendments to clarify our position on this:

We have added additional information to the ‘bias and quality assessments’ section in the methods regarding this point: ‘These checklists will provide useful contextual information about the included studies such as information about performance bias. Checklists items cover things like intervention assessors and their reflexivity which are important factors to consider as participant attitudes towards summary formats may be influenced by external factors such as who created the summary (e.g., their own vs. an external organisation).’

We agree that the issue is complex and there are many confounders and interacting factors that could influence one’s attitude towards a format. Relatedly, we have chosen to restrict study designs for this review to randomized controlled trials and qualitative studies as we believe these study designs are best able to control for factors such as who is disseminating the message (RCTs) and are better able to fully explore multiple factors influencing one’s perception (qualitative studies). For example, these attitudes can be more fully explored with probes during semi-structured interviews. We have included the use of assessment tools for bias to further investigate issues such as the potential influence of the study assessors (e.g., those who designed or created the format) and their recipients (i.e, the readers/participants). We have also kept our outcomes broad so as to not take a one-dimensional view towards preference for one format over another.

We can describe differences between preferences for formats and quantitative outcomes such as knowledge in the analysis where the data is available in included studies. We have clarified this in the analysis section of the methods as follows: “Where data is available, we will compare and contrast reported findings on preference and whether or not preference is aligned with improvement of outcomes of impact such as knowledge.”

I was not fully sure why things like surveys measuring attitudes and opinions of people were not taken along in this review. (I am aware of the fact that guidance on the inclusion of such designs in reviews is scarce though-consider it a free offer to work towards a more comprehensive review type). Particularly, because in the RCTs authors tend to rely on self-reported measures, with LIKERT scales. Here, a comparison could sit with time, rather than comparator. Does people's attitude change when exposed to different formats over time?
Author’s response: Thank you for this comment. The aim of this mixed-methods systematic review was to evaluate the effectiveness and acceptability of different evidence synthesis summary formats for CGDG members. For effectiveness we chose to restrict to RCTs as the most appropriate design to evaluate the effectiveness of the interventions in question. Therefore, observational studies like surveys were not included for the effectiveness question as they would not be appropriate to answer the question posed, as described in the methods section. We did not explicitly exclude RCT designs that had interventions with ‘repeated exposures,’ multiple timepoints for outcome assessment or longitudinal follow ups so if identified such studies will be included. However, these design seem unlikely in this context. While policy makers, clinicians, and methodologists could potentially be involved in multiple guideline development groups, this would likely be over quite an extended period of time and would likely have objectives related to the decision-making process and attitude change over time. RCTs exploring this seem unlikely and repeated surveys would not provide effectiveness data.

To explore opinions and attitudes, we included qualitative studies and mixed-methods studies with primary qualitative data collection. We choose to focus on primary qualitative data as this data in this context. In keeping with Cochrane guidance, we do not consider a questionnaire survey to be a qualitative study (https://training.cochrane.org/handbook/current/chapter-21). We prioritized inclusion of qualitative data from primary studies over free text from questionnaire surveys as we hypothesized primary data would be richer and thicker and thus more informative. We have clarified this in the methods section as follows: “We prioritized the inclusion of qualitative data from primary studies over free text from questionnaire surveys as we hypothesized primary data would be richer and thicker and thus more informative.”

Sometimes the arguments are not fully clear. In the introduction, attention is paid to problems complicating the conduct of systematic reviews. This is not the core focus of the project. The project is linked to the translation phase of turning review findings into evidence summaries. This is a matter of content, form and channels rather than the SR pre-track issues. It would therefore be more appropriate to highlight the diversity of actors involved in producing guidelines in relation to how communication and dissemination channels have changed because of that. I am a big fan of the authors discussing things like podcasts and multimedia accounts as examples. Naturally, this is linked to a society moving into multimodality and the limitations of numbers and narratives in reaching out to or working with different publics.
Author’s response: Thank you for this feedback. We have edited the introduction to try to truncate the paragraph on systematic reviews and to try to relate more to the core focus of the project. The process of conducting a systematic review is related to the time and resources the team will have to create the summary formats themselves so we have largely kept the information about the ‘pre-track’ problems. Furthermore, we are focusing on multiple types of evidence syntheses and the landscape has recently evolved in this regard. This also directly relates to the search strategy used and inclusion criteria.

The PICO would perhaps work better if clinicians is changed into professionals. It would make it relevant to a broader target group. Also, the comparator could also be no summary at all, for example, reading findings from reviews straight into the guideline procedure. Why are these explicitly excluded from the review?
Author’s response: Thank you for this question. We have changed clinicians to healthcare professionals to be more inclusive. We have excluded no summary at all as a comparator because we do not think it is appropriate to compare the reading experience of a summary format versus none. We have added text to the ‘Quantitative systematic review’ section: Studies where the comparison is no intervention (e.g., the full-text of an evidence synthesis manuscript) will be excluded. Producing a full text of an evidence synthesis without some form of summary is not ‘usual care’. Internationally, organisations, journals and reporting guidelines would consider a summary to be a mandatory component of any report/peer review manuscript produced (e.g., Cochrane, PRISMA guidance, etc.). Therefore no form of summary or abstract ‘is not an appropriate comparator

Do the authors only include qualitative studies where an intervention is evaluated by participants in terms of meaningfulness or applicability? Or do they also include qualitative studies where attitudes, viewpoints and options for form/channel/content are discussed from a more theoretical point of view, or in pilot cases? It is not clear where the selection sits on the qualitative side, but the data collection part suggests the first option.
Author’s response: Thank you for this question. We have chosen the SPiDER approach for the qualitative evidence synthesis as we are not restricting qualitative studies to those that evaluate an intervention in terms of meaningfulness of applicability. Our evaluation outcomes include the ‘Views, attitudes, opinions, experiences, perceptions, beliefs, feelings, understanding’ of participants. The information in the data collection section regarding the qualitative data is just to provide a summary overview of the study’s main findings (outcomes) to help with descriptive analyses. We edited the text to emphasize that NVivo12 coding will be conducted ‘in parallel’ (rather than ‘however, included’ as previously worded).

Perhaps the authors may want to bring a rationale for why they want to limit the formats to textual and audio-visual productions, as there are many more forms available through which evidence is and can be communicated (such as installations, theatre, etc.) Perhaps these forms haven't been studied extensively in the context of guidelines, but in a protocol phase we are not sure what to find yet :-).
Author’s response: Thank you for this suggestion. We did not list any exclusion of formats explicitly, nor did we limit the formats to textual or audio-visual productions. In the list provided, we used e.g. ‘e.g., visual, audio, text based’ (for example) to list potential summary modes we may find rather than ‘i.e.’ (that is) to explicitly list summary modes we are restricting our search to. Our search strategy included specific formats that we thought we might find but we would not exclude other types of communications that weren’t explicitly listed. We have added text to the ‘quantitative systematic review’ section to clarify this ‘We will not exclude a summary format if it is one that we did not explicitly list in our search strategy (Table 3).’

I would find it more practical if the authors would discuss their position towards critical appraisal and how to structure this instead of putting it in the data collection part as an extraction category (it is not clear what will actually be extracted for what purpose). Also, critical appraisal checklist for primary studies have a different purpose than GRADE and CONQUAL that work on a review level (which is clearly not the focus here when talking about the pool of studies included). I think the authors need to spell the two out in more detail, as they are used in different phases of the review process.
Author’s response: Thank you for this suggestion. In reporting this protocol, we have followed the JBI manual and as such the ‘bias and quality assessments’ subheading should be it’s own unique section and does not fall under the data collection section. The JBI critical appraisal tools are mentioned in the data collection section as they are a part of the standard data-extraction form that will be used by the reviewers. These should be two separate sections; apologies if there was a formatting error. We have edited the text within the ‘data collection’ and ‘bias and quality assessments’ sections to try to clarify what will be extracted for what purpose and to clarify the GRADE/ConQual concern. The text changed in the data collection now reads: ‘The following data information will be extracted using the pilot-tested standardized data-extraction form’ and ‘Data from the domains listed within the JBI critical appraisal tools for qualitative and quantitative studies.’
We agree that the critical appraisal checklist for primary studies have a different purpose than GRADE and CONQUAL. We have reworded the information in the ‘bias and quality assessments’ section to clarify that the JBI checklists are for individual studies AND there is no checklist recommended for the final assessment in the entirety of evidence (similar to the GRADE and CONQUAL purposes) for a mixed methods systematic review.

Would quality of a study not rather fit a sensitivity analysis rather than a subgroup analysis? What purpose would it have as a subgroup in the review? It has disappeared in the mixed method part as a subgroup analysis, so wouldn't mention it as a subgroup in other parts of the text.
Author’s response: Thank you for this question. We recognize that the text in the quantitative analysis section was unclear. We have edited the text, deleting the for example list and restricting it only to the subgroups of interest (participant group or summary format type). We note subgroup analyses for participant groups or intervention types as these involve comparisons across the different groups. There are no planned sensitivity analyses for study quality as there will likely be far too much heterogeneity in outcomes (and the studies in general) to be able to compare the different methods.

What do the authors mean with the phrase "If textual pooling is not available, a narrative summary will be presented." Is a narrative summary not already a textual pooling? Unclear.
Author’s response: Thank you for this question. Apologies for this issue. We have edited the text accordingly. ‘If we are unable to pool findings together (i.e., create and present categories), likely due to an insufficient number of studies identified, a narrative summary will be presented.’

While configuration might have different meanings, it is generally not commonly used as a metaphor for a separate type of synthesis. Configuration would make the different parts unrecognizable in the whole. I don't think meta-aggregation would achieve this level of integration. Perhaps integration is better than configuration here.
Author’s response: Thank you for this suggestion. We have reworded the mixed methods synthesis section to make our approach more clear and have added a references to the specific sections of the JBI manual that we are referencing regarding configuration and the thematic analysis (chapter 8 and section 8.2 for methodology; not section 2 which is for SR of qualitative evidence which details the meta-aggregation approach). We are following JBI’s guidance for MMSRs, and have chosen to use a convergent segregated approach. We have added additional text to clarify that ‘There are several methods for integrating qualitative and quantitative evidence syntheses26, [2] in a convergent segregated MMSR. We will use a thematic synthesis method for integration which groups together similar codes, develops descriptive categories (or themes) to create an overall summary of findings. ^{41, 42’}Meta-aggregation is solely used for the qualitative analysis, not for the mixed methods synthesis which includes the integration.

Also, thematic synthesis is not the same as meta-aggregation, and line by line coding is certainly not a part of meta-aggregation. I believe the authors need to make an informed choice between qualitative evidence synthesis approaches and then follow the guidance through.
Author’s response: Thank you for this comment. Apologies for the confusion but we are not using a meta-aggregation approach for the mixed methods synthesis – only for the qualitative analysis which is done prior to the integration of the two separate streams of evidence. We have edited the mixed methods synthesis section to further clarify our methodological approach which is in accordance with chapter 8 of the JBI manual.
This is a review underpinned by clear and interesting questions, that warrant a mixed method strategy. There is room for improvement though on the level of motivating choices for concepts and finetuning methods.
Author’s response: Thank you for your review and thoughtful suggestions for our manuscript.

The biggest issue I have is on the relation between content and form in the choice for the mediums to be compared. The authors tend to compare forms (?), but attitude towards form is not fully independent to the way messages are framed, nor by whom they are disseminated. How do the authors intend to deal with this in their comparison? (e.g. people might prefer a form such as video but it might fail to ring the message across, as participants are distracted by the form).
Author’s response: Thank you for this comment. This is an interesting point, the studies we include in our review may well demonstrate a disconnect between what formats participants prefer, and what formats may actually produce improvement in outcomes such as knowledge. If included studies present this data we will be able to explore this more detail in our analysis but we suspect from the studies we have seen that this level of detail will not be available. Based on the importance of this suggestion, we will include this as a point of consideration in the discussion in the main manuscript in the protocol, we have made the following amendments to clarify our position on this:

We have added additional information to the ‘bias and quality assessments’ section in the methods regarding this point: ‘These checklists will provide useful contextual information about the included studies such as information about performance bias. Checklists items cover things like intervention assessors and their reflexivity which are important factors to consider as participant attitudes towards summary formats may be influenced by external factors such as who created the summary (e.g., their own vs. an external organisation).’

We agree that the issue is complex and there are many confounders and interacting factors that could influence one’s attitude towards a format. Relatedly, we have chosen to restrict study designs for this review to randomized controlled trials and qualitative studies as we believe these study designs are best able to control for factors such as who is disseminating the message (RCTs) and are better able to fully explore multiple factors influencing one’s perception (qualitative studies). For example, these attitudes can be more fully explored with probes during semi-structured interviews. We have included the use of assessment tools for bias to further investigate issues such as the potential influence of the study assessors (e.g., those who designed or created the format) and their recipients (i.e, the readers/participants). We have also kept our outcomes broad so as to not take a one-dimensional view towards preference for one format over another.

We can describe differences between preferences for formats and quantitative outcomes such as knowledge in the analysis where the data is available in included studies. We have clarified this in the analysis section of the methods as follows: “Where data is available, we will compare and contrast reported findings on preference and whether or not preference is aligned with improvement of outcomes of impact such as knowledge.”

I was not fully sure why things like surveys measuring attitudes and opinions of people were not taken along in this review. (I am aware of the fact that guidance on the inclusion of such designs in reviews is scarce though-consider it a free offer to work towards a more comprehensive review type). Particularly, because in the RCTs authors tend to rely on self-reported measures, with LIKERT scales. Here, a comparison could sit with time, rather than comparator. Does people's attitude change when exposed to different formats over time?
Author’s response: Thank you for this comment. The aim of this mixed-methods systematic review was to evaluate the effectiveness and acceptability of different evidence synthesis summary formats for CGDG members. For effectiveness we chose to restrict to RCTs as the most appropriate design to evaluate the effectiveness of the interventions in question. Therefore, observational studies like surveys were not included for the effectiveness question as they would not be appropriate to answer the question posed, as described in the methods section. We did not explicitly exclude RCT designs that had interventions with ‘repeated exposures,’ multiple timepoints for outcome assessment or longitudinal follow ups so if identified such studies will be included. However, these design seem unlikely in this context. While policy makers, clinicians, and methodologists could potentially be involved in multiple guideline development groups, this would likely be over quite an extended period of time and would likely have objectives related to the decision-making process and attitude change over time. RCTs exploring this seem unlikely and repeated surveys would not provide effectiveness data.

To explore opinions and attitudes, we included qualitative studies and mixed-methods studies with primary qualitative data collection. We choose to focus on primary qualitative data as this data in this context. In keeping with Cochrane guidance, we do not consider a questionnaire survey to be a qualitative study (https://training.cochrane.org/handbook/current/chapter-21). We prioritized inclusion of qualitative data from primary studies over free text from questionnaire surveys as we hypothesized primary data would be richer and thicker and thus more informative. We have clarified this in the methods section as follows: “We prioritized the inclusion of qualitative data from primary studies over free text from questionnaire surveys as we hypothesized primary data would be richer and thicker and thus more informative.”

Sometimes the arguments are not fully clear. In the introduction, attention is paid to problems complicating the conduct of systematic reviews. This is not the core focus of the project. The project is linked to the translation phase of turning review findings into evidence summaries. This is a matter of content, form and channels rather than the SR pre-track issues. It would therefore be more appropriate to highlight the diversity of actors involved in producing guidelines in relation to how communication and dissemination channels have changed because of that. I am a big fan of the authors discussing things like podcasts and multimedia accounts as examples. Naturally, this is linked to a society moving into multimodality and the limitations of numbers and narratives in reaching out to or working with different publics.
Author’s response: Thank you for this feedback. We have edited the introduction to try to truncate the paragraph on systematic reviews and to try to relate more to the core focus of the project. The process of conducting a systematic review is related to the time and resources the team will have to create the summary formats themselves so we have largely kept the information about the ‘pre-track’ problems. Furthermore, we are focusing on multiple types of evidence syntheses and the landscape has recently evolved in this regard. This also directly relates to the search strategy used and inclusion criteria.

The PICO would perhaps work better if clinicians is changed into professionals. It would make it relevant to a broader target group. Also, the comparator could also be no summary at all, for example, reading findings from reviews straight into the guideline procedure. Why are these explicitly excluded from the review?
Author’s response: Thank you for this question. We have changed clinicians to healthcare professionals to be more inclusive. We have excluded no summary at all as a comparator because we do not think it is appropriate to compare the reading experience of a summary format versus none. We have added text to the ‘Quantitative systematic review’ section: Studies where the comparison is no intervention (e.g., the full-text of an evidence synthesis manuscript) will be excluded. Producing a full text of an evidence synthesis without some form of summary is not ‘usual care’. Internationally, organisations, journals and reporting guidelines would consider a summary to be a mandatory component of any report/peer review manuscript produced (e.g., Cochrane, PRISMA guidance, etc.). Therefore no form of summary or abstract ‘is not an appropriate comparator

Do the authors only include qualitative studies where an intervention is evaluated by participants in terms of meaningfulness or applicability? Or do they also include qualitative studies where attitudes, viewpoints and options for form/channel/content are discussed from a more theoretical point of view, or in pilot cases? It is not clear where the selection sits on the qualitative side, but the data collection part suggests the first option.
Author’s response: Thank you for this question. We have chosen the SPiDER approach for the qualitative evidence synthesis as we are not restricting qualitative studies to those that evaluate an intervention in terms of meaningfulness of applicability. Our evaluation outcomes include the ‘Views, attitudes, opinions, experiences, perceptions, beliefs, feelings, understanding’ of participants. The information in the data collection section regarding the qualitative data is just to provide a summary overview of the study’s main findings (outcomes) to help with descriptive analyses. We edited the text to emphasize that NVivo12 coding will be conducted ‘in parallel’ (rather than ‘however, included’ as previously worded).

Perhaps the authors may want to bring a rationale for why they want to limit the formats to textual and audio-visual productions, as there are many more forms available through which evidence is and can be communicated (such as installations, theatre, etc.) Perhaps these forms haven't been studied extensively in the context of guidelines, but in a protocol phase we are not sure what to find yet :-).
Author’s response: Thank you for this suggestion. We did not list any exclusion of formats explicitly, nor did we limit the formats to textual or audio-visual productions. In the list provided, we used e.g. ‘e.g., visual, audio, text based’ (for example) to list potential summary modes we may find rather than ‘i.e.’ (that is) to explicitly list summary modes we are restricting our search to. Our search strategy included specific formats that we thought we might find but we would not exclude other types of communications that weren’t explicitly listed. We have added text to the ‘quantitative systematic review’ section to clarify this ‘We will not exclude a summary format if it is one that we did not explicitly list in our search strategy (Table 3).’

I would find it more practical if the authors would discuss their position towards critical appraisal and how to structure this instead of putting it in the data collection part as an extraction category (it is not clear what will actually be extracted for what purpose). Also, critical appraisal checklist for primary studies have a different purpose than GRADE and CONQUAL that work on a review level (which is clearly not the focus here when talking about the pool of studies included). I think the authors need to spell the two out in more detail, as they are used in different phases of the review process.
Author’s response: Thank you for this suggestion. In reporting this protocol, we have followed the JBI manual and as such the ‘bias and quality assessments’ subheading should be it’s own unique section and does not fall under the data collection section. The JBI critical appraisal tools are mentioned in the data collection section as they are a part of the standard data-extraction form that will be used by the reviewers. These should be two separate sections; apologies if there was a formatting error. We have edited the text within the ‘data collection’ and ‘bias and quality assessments’ sections to try to clarify what will be extracted for what purpose and to clarify the GRADE/ConQual concern. The text changed in the data collection now reads: ‘The following data information will be extracted using the pilot-tested standardized data-extraction form’ and ‘Data from the domains listed within the JBI critical appraisal tools for qualitative and quantitative studies.’
We agree that the critical appraisal checklist for primary studies have a different purpose than GRADE and CONQUAL. We have reworded the information in the ‘bias and quality assessments’ section to clarify that the JBI checklists are for individual studies AND there is no checklist recommended for the final assessment in the entirety of evidence (similar to the GRADE and CONQUAL purposes) for a mixed methods systematic review.

Would quality of a study not rather fit a sensitivity analysis rather than a subgroup analysis? What purpose would it have as a subgroup in the review? It has disappeared in the mixed method part as a subgroup analysis, so wouldn't mention it as a subgroup in other parts of the text.
Author’s response: Thank you for this question. We recognize that the text in the quantitative analysis section was unclear. We have edited the text, deleting the for example list and restricting it only to the subgroups of interest (participant group or summary format type). We note subgroup analyses for participant groups or intervention types as these involve comparisons across the different groups. There are no planned sensitivity analyses for study quality as there will likely be far too much heterogeneity in outcomes (and the studies in general) to be able to compare the different methods.

What do the authors mean with the phrase "If textual pooling is not available, a narrative summary will be presented." Is a narrative summary not already a textual pooling? Unclear.
Author’s response: Thank you for this question. Apologies for this issue. We have edited the text accordingly. ‘If we are unable to pool findings together (i.e., create and present categories), likely due to an insufficient number of studies identified, a narrative summary will be presented.’

While configuration might have different meanings, it is generally not commonly used as a metaphor for a separate type of synthesis. Configuration would make the different parts unrecognizable in the whole. I don't think meta-aggregation would achieve this level of integration. Perhaps integration is better than configuration here.
Author’s response: Thank you for this suggestion. We have reworded the mixed methods synthesis section to make our approach more clear and have added a references to the specific sections of the JBI manual that we are referencing regarding configuration and the thematic analysis (chapter 8 and section 8.2 for methodology; not section 2 which is for SR of qualitative evidence which details the meta-aggregation approach). We are following JBI’s guidance for MMSRs, and have chosen to use a convergent segregated approach. We have added additional text to clarify that ‘There are several methods for integrating qualitative and quantitative evidence syntheses26, [2] in a convergent segregated MMSR. We will use a thematic synthesis method for integration which groups together similar codes, develops descriptive categories (or themes) to create an overall summary of findings. ^{41, 42’}Meta-aggregation is solely used for the qualitative analysis, not for the mixed methods synthesis which includes the integration.

Also, thematic synthesis is not the same as meta-aggregation, and line by line coding is certainly not a part of meta-aggregation. I believe the authors need to make an informed choice between qualitative evidence synthesis approaches and then follow the guidance through.
Author’s response: Thank you for this comment. Apologies for the confusion but we are not using a meta-aggregation approach for the mixed methods synthesis – only for the qualitative analysis which is done prior to the integration of the two separate streams of evidence. We have edited the mixed methods synthesis section to further clarify our methodological approach which is in accordance with chapter 8 of the JBI manual.
Competing Interests: No competing interests were disclosed. Close
Report a concern
Respond or Comment

COMMENTS ON THIS REPORT

Author Response 05 May 2022

Melissa Sharp, Health Research Board Centre for Primary Care Research, Department of General Practice,, Royal College of Surgeons in Ireland, Dublin, Ireland

05 May 2022

Author Response
This is a review underpinned by clear and interesting questions, that warrant a mixed method strategy. There is room for improvement though on the level of motivating choices for concepts ... Continue reading
This is a review underpinned by clear and interesting questions, that warrant a mixed method strategy. There is room for improvement though on the level of motivating choices for concepts and finetuning methods.
Author’s response: Thank you for your review and thoughtful suggestions for our manuscript.

The biggest issue I have is on the relation between content and form in the choice for the mediums to be compared. The authors tend to compare forms (?), but attitude towards form is not fully independent to the way messages are framed, nor by whom they are disseminated. How do the authors intend to deal with this in their comparison? (e.g. people might prefer a form such as video but it might fail to ring the message across, as participants are distracted by the form).
Author’s response: Thank you for this comment. This is an interesting point, the studies we include in our review may well demonstrate a disconnect between what formats participants prefer, and what formats may actually produce improvement in outcomes such as knowledge. If included studies present this data we will be able to explore this more detail in our analysis but we suspect from the studies we have seen that this level of detail will not be available. Based on the importance of this suggestion, we will include this as a point of consideration in the discussion in the main manuscript in the protocol, we have made the following amendments to clarify our position on this:

We have added additional information to the ‘bias and quality assessments’ section in the methods regarding this point: ‘These checklists will provide useful contextual information about the included studies such as information about performance bias. Checklists items cover things like intervention assessors and their reflexivity which are important factors to consider as participant attitudes towards summary formats may be influenced by external factors such as who created the summary (e.g., their own vs. an external organisation).’

We agree that the issue is complex and there are many confounders and interacting factors that could influence one’s attitude towards a format. Relatedly, we have chosen to restrict study designs for this review to randomized controlled trials and qualitative studies as we believe these study designs are best able to control for factors such as who is disseminating the message (RCTs) and are better able to fully explore multiple factors influencing one’s perception (qualitative studies). For example, these attitudes can be more fully explored with probes during semi-structured interviews. We have included the use of assessment tools for bias to further investigate issues such as the potential influence of the study assessors (e.g., those who designed or created the format) and their recipients (i.e, the readers/participants). We have also kept our outcomes broad so as to not take a one-dimensional view towards preference for one format over another.

We can describe differences between preferences for formats and quantitative outcomes such as knowledge in the analysis where the data is available in included studies. We have clarified this in the analysis section of the methods as follows: “Where data is available, we will compare and contrast reported findings on preference and whether or not preference is aligned with improvement of outcomes of impact such as knowledge.”

I was not fully sure why things like surveys measuring attitudes and opinions of people were not taken along in this review. (I am aware of the fact that guidance on the inclusion of such designs in reviews is scarce though-consider it a free offer to work towards a more comprehensive review type). Particularly, because in the RCTs authors tend to rely on self-reported measures, with LIKERT scales. Here, a comparison could sit with time, rather than comparator. Does people's attitude change when exposed to different formats over time?
Author’s response: Thank you for this comment. The aim of this mixed-methods systematic review was to evaluate the effectiveness and acceptability of different evidence synthesis summary formats for CGDG members. For effectiveness we chose to restrict to RCTs as the most appropriate design to evaluate the effectiveness of the interventions in question. Therefore, observational studies like surveys were not included for the effectiveness question as they would not be appropriate to answer the question posed, as described in the methods section. We did not explicitly exclude RCT designs that had interventions with ‘repeated exposures,’ multiple timepoints for outcome assessment or longitudinal follow ups so if identified such studies will be included. However, these design seem unlikely in this context. While policy makers, clinicians, and methodologists could potentially be involved in multiple guideline development groups, this would likely be over quite an extended period of time and would likely have objectives related to the decision-making process and attitude change over time. RCTs exploring this seem unlikely and repeated surveys would not provide effectiveness data.

To explore opinions and attitudes, we included qualitative studies and mixed-methods studies with primary qualitative data collection. We choose to focus on primary qualitative data as this data in this context. In keeping with Cochrane guidance, we do not consider a questionnaire survey to be a qualitative study (https://training.cochrane.org/handbook/current/chapter-21). We prioritized inclusion of qualitative data from primary studies over free text from questionnaire surveys as we hypothesized primary data would be richer and thicker and thus more informative. We have clarified this in the methods section as follows: “We prioritized the inclusion of qualitative data from primary studies over free text from questionnaire surveys as we hypothesized primary data would be richer and thicker and thus more informative.”

Sometimes the arguments are not fully clear. In the introduction, attention is paid to problems complicating the conduct of systematic reviews. This is not the core focus of the project. The project is linked to the translation phase of turning review findings into evidence summaries. This is a matter of content, form and channels rather than the SR pre-track issues. It would therefore be more appropriate to highlight the diversity of actors involved in producing guidelines in relation to how communication and dissemination channels have changed because of that. I am a big fan of the authors discussing things like podcasts and multimedia accounts as examples. Naturally, this is linked to a society moving into multimodality and the limitations of numbers and narratives in reaching out to or working with different publics.
Author’s response: Thank you for this feedback. We have edited the introduction to try to truncate the paragraph on systematic reviews and to try to relate more to the core focus of the project. The process of conducting a systematic review is related to the time and resources the team will have to create the summary formats themselves so we have largely kept the information about the ‘pre-track’ problems. Furthermore, we are focusing on multiple types of evidence syntheses and the landscape has recently evolved in this regard. This also directly relates to the search strategy used and inclusion criteria.

The PICO would perhaps work better if clinicians is changed into professionals. It would make it relevant to a broader target group. Also, the comparator could also be no summary at all, for example, reading findings from reviews straight into the guideline procedure. Why are these explicitly excluded from the review?
Author’s response: Thank you for this question. We have changed clinicians to healthcare professionals to be more inclusive. We have excluded no summary at all as a comparator because we do not think it is appropriate to compare the reading experience of a summary format versus none. We have added text to the ‘Quantitative systematic review’ section: Studies where the comparison is no intervention (e.g., the full-text of an evidence synthesis manuscript) will be excluded. Producing a full text of an evidence synthesis without some form of summary is not ‘usual care’. Internationally, organisations, journals and reporting guidelines would consider a summary to be a mandatory component of any report/peer review manuscript produced (e.g., Cochrane, PRISMA guidance, etc.). Therefore no form of summary or abstract ‘is not an appropriate comparator

Do the authors only include qualitative studies where an intervention is evaluated by participants in terms of meaningfulness or applicability? Or do they also include qualitative studies where attitudes, viewpoints and options for form/channel/content are discussed from a more theoretical point of view, or in pilot cases? It is not clear where the selection sits on the qualitative side, but the data collection part suggests the first option.
Author’s response: Thank you for this question. We have chosen the SPiDER approach for the qualitative evidence synthesis as we are not restricting qualitative studies to those that evaluate an intervention in terms of meaningfulness of applicability. Our evaluation outcomes include the ‘Views, attitudes, opinions, experiences, perceptions, beliefs, feelings, understanding’ of participants. The information in the data collection section regarding the qualitative data is just to provide a summary overview of the study’s main findings (outcomes) to help with descriptive analyses. We edited the text to emphasize that NVivo12 coding will be conducted ‘in parallel’ (rather than ‘however, included’ as previously worded).

Perhaps the authors may want to bring a rationale for why they want to limit the formats to textual and audio-visual productions, as there are many more forms available through which evidence is and can be communicated (such as installations, theatre, etc.) Perhaps these forms haven't been studied extensively in the context of guidelines, but in a protocol phase we are not sure what to find yet :-).
Author’s response: Thank you for this suggestion. We did not list any exclusion of formats explicitly, nor did we limit the formats to textual or audio-visual productions. In the list provided, we used e.g. ‘e.g., visual, audio, text based’ (for example) to list potential summary modes we may find rather than ‘i.e.’ (that is) to explicitly list summary modes we are restricting our search to. Our search strategy included specific formats that we thought we might find but we would not exclude other types of communications that weren’t explicitly listed. We have added text to the ‘quantitative systematic review’ section to clarify this ‘We will not exclude a summary format if it is one that we did not explicitly list in our search strategy (Table 3).’

I would find it more practical if the authors would discuss their position towards critical appraisal and how to structure this instead of putting it in the data collection part as an extraction category (it is not clear what will actually be extracted for what purpose). Also, critical appraisal checklist for primary studies have a different purpose than GRADE and CONQUAL that work on a review level (which is clearly not the focus here when talking about the pool of studies included). I think the authors need to spell the two out in more detail, as they are used in different phases of the review process.
Author’s response: Thank you for this suggestion. In reporting this protocol, we have followed the JBI manual and as such the ‘bias and quality assessments’ subheading should be it’s own unique section and does not fall under the data collection section. The JBI critical appraisal tools are mentioned in the data collection section as they are a part of the standard data-extraction form that will be used by the reviewers. These should be two separate sections; apologies if there was a formatting error. We have edited the text within the ‘data collection’ and ‘bias and quality assessments’ sections to try to clarify what will be extracted for what purpose and to clarify the GRADE/ConQual concern. The text changed in the data collection now reads: ‘The following data information will be extracted using the pilot-tested standardized data-extraction form’ and ‘Data from the domains listed within the JBI critical appraisal tools for qualitative and quantitative studies.’
We agree that the critical appraisal checklist for primary studies have a different purpose than GRADE and CONQUAL. We have reworded the information in the ‘bias and quality assessments’ section to clarify that the JBI checklists are for individual studies AND there is no checklist recommended for the final assessment in the entirety of evidence (similar to the GRADE and CONQUAL purposes) for a mixed methods systematic review.

Would quality of a study not rather fit a sensitivity analysis rather than a subgroup analysis? What purpose would it have as a subgroup in the review? It has disappeared in the mixed method part as a subgroup analysis, so wouldn't mention it as a subgroup in other parts of the text.
Author’s response: Thank you for this question. We recognize that the text in the quantitative analysis section was unclear. We have edited the text, deleting the for example list and restricting it only to the subgroups of interest (participant group or summary format type). We note subgroup analyses for participant groups or intervention types as these involve comparisons across the different groups. There are no planned sensitivity analyses for study quality as there will likely be far too much heterogeneity in outcomes (and the studies in general) to be able to compare the different methods.

What do the authors mean with the phrase "If textual pooling is not available, a narrative summary will be presented." Is a narrative summary not already a textual pooling? Unclear.
Author’s response: Thank you for this question. Apologies for this issue. We have edited the text accordingly. ‘If we are unable to pool findings together (i.e., create and present categories), likely due to an insufficient number of studies identified, a narrative summary will be presented.’

While configuration might have different meanings, it is generally not commonly used as a metaphor for a separate type of synthesis. Configuration would make the different parts unrecognizable in the whole. I don't think meta-aggregation would achieve this level of integration. Perhaps integration is better than configuration here.
Author’s response: Thank you for this suggestion. We have reworded the mixed methods synthesis section to make our approach more clear and have added a references to the specific sections of the JBI manual that we are referencing regarding configuration and the thematic analysis (chapter 8 and section 8.2 for methodology; not section 2 which is for SR of qualitative evidence which details the meta-aggregation approach). We are following JBI’s guidance for MMSRs, and have chosen to use a convergent segregated approach. We have added additional text to clarify that ‘There are several methods for integrating qualitative and quantitative evidence syntheses26, [2] in a convergent segregated MMSR. We will use a thematic synthesis method for integration which groups together similar codes, develops descriptive categories (or themes) to create an overall summary of findings. ^{41, 42’}Meta-aggregation is solely used for the qualitative analysis, not for the mixed methods synthesis which includes the integration.

Also, thematic synthesis is not the same as meta-aggregation, and line by line coding is certainly not a part of meta-aggregation. I believe the authors need to make an informed choice between qualitative evidence synthesis approaches and then follow the guidance through.
Author’s response: Thank you for this comment. Apologies for the confusion but we are not using a meta-aggregation approach for the mixed methods synthesis – only for the qualitative analysis which is done prior to the integration of the two separate streams of evidence. We have edited the mixed methods synthesis section to further clarify our methodological approach which is in accordance with chapter 8 of the JBI manual.
This is a review underpinned by clear and interesting questions, that warrant a mixed method strategy. There is room for improvement though on the level of motivating choices for concepts and finetuning methods.
Author’s response: Thank you for your review and thoughtful suggestions for our manuscript.

The biggest issue I have is on the relation between content and form in the choice for the mediums to be compared. The authors tend to compare forms (?), but attitude towards form is not fully independent to the way messages are framed, nor by whom they are disseminated. How do the authors intend to deal with this in their comparison? (e.g. people might prefer a form such as video but it might fail to ring the message across, as participants are distracted by the form).
Author’s response: Thank you for this comment. This is an interesting point, the studies we include in our review may well demonstrate a disconnect between what formats participants prefer, and what formats may actually produce improvement in outcomes such as knowledge. If included studies present this data we will be able to explore this more detail in our analysis but we suspect from the studies we have seen that this level of detail will not be available. Based on the importance of this suggestion, we will include this as a point of consideration in the discussion in the main manuscript in the protocol, we have made the following amendments to clarify our position on this:

We have added additional information to the ‘bias and quality assessments’ section in the methods regarding this point: ‘These checklists will provide useful contextual information about the included studies such as information about performance bias. Checklists items cover things like intervention assessors and their reflexivity which are important factors to consider as participant attitudes towards summary formats may be influenced by external factors such as who created the summary (e.g., their own vs. an external organisation).’

We agree that the issue is complex and there are many confounders and interacting factors that could influence one’s attitude towards a format. Relatedly, we have chosen to restrict study designs for this review to randomized controlled trials and qualitative studies as we believe these study designs are best able to control for factors such as who is disseminating the message (RCTs) and are better able to fully explore multiple factors influencing one’s perception (qualitative studies). For example, these attitudes can be more fully explored with probes during semi-structured interviews. We have included the use of assessment tools for bias to further investigate issues such as the potential influence of the study assessors (e.g., those who designed or created the format) and their recipients (i.e, the readers/participants). We have also kept our outcomes broad so as to not take a one-dimensional view towards preference for one format over another.

We can describe differences between preferences for formats and quantitative outcomes such as knowledge in the analysis where the data is available in included studies. We have clarified this in the analysis section of the methods as follows: “Where data is available, we will compare and contrast reported findings on preference and whether or not preference is aligned with improvement of outcomes of impact such as knowledge.”

I was not fully sure why things like surveys measuring attitudes and opinions of people were not taken along in this review. (I am aware of the fact that guidance on the inclusion of such designs in reviews is scarce though-consider it a free offer to work towards a more comprehensive review type). Particularly, because in the RCTs authors tend to rely on self-reported measures, with LIKERT scales. Here, a comparison could sit with time, rather than comparator. Does people's attitude change when exposed to different formats over time?
Author’s response: Thank you for this comment. The aim of this mixed-methods systematic review was to evaluate the effectiveness and acceptability of different evidence synthesis summary formats for CGDG members. For effectiveness we chose to restrict to RCTs as the most appropriate design to evaluate the effectiveness of the interventions in question. Therefore, observational studies like surveys were not included for the effectiveness question as they would not be appropriate to answer the question posed, as described in the methods section. We did not explicitly exclude RCT designs that had interventions with ‘repeated exposures,’ multiple timepoints for outcome assessment or longitudinal follow ups so if identified such studies will be included. However, these design seem unlikely in this context. While policy makers, clinicians, and methodologists could potentially be involved in multiple guideline development groups, this would likely be over quite an extended period of time and would likely have objectives related to the decision-making process and attitude change over time. RCTs exploring this seem unlikely and repeated surveys would not provide effectiveness data.

To explore opinions and attitudes, we included qualitative studies and mixed-methods studies with primary qualitative data collection. We choose to focus on primary qualitative data as this data in this context. In keeping with Cochrane guidance, we do not consider a questionnaire survey to be a qualitative study (https://training.cochrane.org/handbook/current/chapter-21). We prioritized inclusion of qualitative data from primary studies over free text from questionnaire surveys as we hypothesized primary data would be richer and thicker and thus more informative. We have clarified this in the methods section as follows: “We prioritized the inclusion of qualitative data from primary studies over free text from questionnaire surveys as we hypothesized primary data would be richer and thicker and thus more informative.”

Sometimes the arguments are not fully clear. In the introduction, attention is paid to problems complicating the conduct of systematic reviews. This is not the core focus of the project. The project is linked to the translation phase of turning review findings into evidence summaries. This is a matter of content, form and channels rather than the SR pre-track issues. It would therefore be more appropriate to highlight the diversity of actors involved in producing guidelines in relation to how communication and dissemination channels have changed because of that. I am a big fan of the authors discussing things like podcasts and multimedia accounts as examples. Naturally, this is linked to a society moving into multimodality and the limitations of numbers and narratives in reaching out to or working with different publics.
Author’s response: Thank you for this feedback. We have edited the introduction to try to truncate the paragraph on systematic reviews and to try to relate more to the core focus of the project. The process of conducting a systematic review is related to the time and resources the team will have to create the summary formats themselves so we have largely kept the information about the ‘pre-track’ problems. Furthermore, we are focusing on multiple types of evidence syntheses and the landscape has recently evolved in this regard. This also directly relates to the search strategy used and inclusion criteria.

The PICO would perhaps work better if clinicians is changed into professionals. It would make it relevant to a broader target group. Also, the comparator could also be no summary at all, for example, reading findings from reviews straight into the guideline procedure. Why are these explicitly excluded from the review?
Author’s response: Thank you for this question. We have changed clinicians to healthcare professionals to be more inclusive. We have excluded no summary at all as a comparator because we do not think it is appropriate to compare the reading experience of a summary format versus none. We have added text to the ‘Quantitative systematic review’ section: Studies where the comparison is no intervention (e.g., the full-text of an evidence synthesis manuscript) will be excluded. Producing a full text of an evidence synthesis without some form of summary is not ‘usual care’. Internationally, organisations, journals and reporting guidelines would consider a summary to be a mandatory component of any report/peer review manuscript produced (e.g., Cochrane, PRISMA guidance, etc.). Therefore no form of summary or abstract ‘is not an appropriate comparator

Do the authors only include qualitative studies where an intervention is evaluated by participants in terms of meaningfulness or applicability? Or do they also include qualitative studies where attitudes, viewpoints and options for form/channel/content are discussed from a more theoretical point of view, or in pilot cases? It is not clear where the selection sits on the qualitative side, but the data collection part suggests the first option.
Author’s response: Thank you for this question. We have chosen the SPiDER approach for the qualitative evidence synthesis as we are not restricting qualitative studies to those that evaluate an intervention in terms of meaningfulness of applicability. Our evaluation outcomes include the ‘Views, attitudes, opinions, experiences, perceptions, beliefs, feelings, understanding’ of participants. The information in the data collection section regarding the qualitative data is just to provide a summary overview of the study’s main findings (outcomes) to help with descriptive analyses. We edited the text to emphasize that NVivo12 coding will be conducted ‘in parallel’ (rather than ‘however, included’ as previously worded).

Perhaps the authors may want to bring a rationale for why they want to limit the formats to textual and audio-visual productions, as there are many more forms available through which evidence is and can be communicated (such as installations, theatre, etc.) Perhaps these forms haven't been studied extensively in the context of guidelines, but in a protocol phase we are not sure what to find yet :-).
Author’s response: Thank you for this suggestion. We did not list any exclusion of formats explicitly, nor did we limit the formats to textual or audio-visual productions. In the list provided, we used e.g. ‘e.g., visual, audio, text based’ (for example) to list potential summary modes we may find rather than ‘i.e.’ (that is) to explicitly list summary modes we are restricting our search to. Our search strategy included specific formats that we thought we might find but we would not exclude other types of communications that weren’t explicitly listed. We have added text to the ‘quantitative systematic review’ section to clarify this ‘We will not exclude a summary format if it is one that we did not explicitly list in our search strategy (Table 3).’

I would find it more practical if the authors would discuss their position towards critical appraisal and how to structure this instead of putting it in the data collection part as an extraction category (it is not clear what will actually be extracted for what purpose). Also, critical appraisal checklist for primary studies have a different purpose than GRADE and CONQUAL that work on a review level (which is clearly not the focus here when talking about the pool of studies included). I think the authors need to spell the two out in more detail, as they are used in different phases of the review process.
Author’s response: Thank you for this suggestion. In reporting this protocol, we have followed the JBI manual and as such the ‘bias and quality assessments’ subheading should be it’s own unique section and does not fall under the data collection section. The JBI critical appraisal tools are mentioned in the data collection section as they are a part of the standard data-extraction form that will be used by the reviewers. These should be two separate sections; apologies if there was a formatting error. We have edited the text within the ‘data collection’ and ‘bias and quality assessments’ sections to try to clarify what will be extracted for what purpose and to clarify the GRADE/ConQual concern. The text changed in the data collection now reads: ‘The following data information will be extracted using the pilot-tested standardized data-extraction form’ and ‘Data from the domains listed within the JBI critical appraisal tools for qualitative and quantitative studies.’
We agree that the critical appraisal checklist for primary studies have a different purpose than GRADE and CONQUAL. We have reworded the information in the ‘bias and quality assessments’ section to clarify that the JBI checklists are for individual studies AND there is no checklist recommended for the final assessment in the entirety of evidence (similar to the GRADE and CONQUAL purposes) for a mixed methods systematic review.

Would quality of a study not rather fit a sensitivity analysis rather than a subgroup analysis? What purpose would it have as a subgroup in the review? It has disappeared in the mixed method part as a subgroup analysis, so wouldn't mention it as a subgroup in other parts of the text.
Author’s response: Thank you for this question. We recognize that the text in the quantitative analysis section was unclear. We have edited the text, deleting the for example list and restricting it only to the subgroups of interest (participant group or summary format type). We note subgroup analyses for participant groups or intervention types as these involve comparisons across the different groups. There are no planned sensitivity analyses for study quality as there will likely be far too much heterogeneity in outcomes (and the studies in general) to be able to compare the different methods.

What do the authors mean with the phrase "If textual pooling is not available, a narrative summary will be presented." Is a narrative summary not already a textual pooling? Unclear.
Author’s response: Thank you for this question. Apologies for this issue. We have edited the text accordingly. ‘If we are unable to pool findings together (i.e., create and present categories), likely due to an insufficient number of studies identified, a narrative summary will be presented.’

While configuration might have different meanings, it is generally not commonly used as a metaphor for a separate type of synthesis. Configuration would make the different parts unrecognizable in the whole. I don't think meta-aggregation would achieve this level of integration. Perhaps integration is better than configuration here.
Author’s response: Thank you for this suggestion. We have reworded the mixed methods synthesis section to make our approach more clear and have added a references to the specific sections of the JBI manual that we are referencing regarding configuration and the thematic analysis (chapter 8 and section 8.2 for methodology; not section 2 which is for SR of qualitative evidence which details the meta-aggregation approach). We are following JBI’s guidance for MMSRs, and have chosen to use a convergent segregated approach. We have added additional text to clarify that ‘There are several methods for integrating qualitative and quantitative evidence syntheses26, [2] in a convergent segregated MMSR. We will use a thematic synthesis method for integration which groups together similar codes, develops descriptive categories (or themes) to create an overall summary of findings. ^{41, 42’}Meta-aggregation is solely used for the qualitative analysis, not for the mixed methods synthesis which includes the integration.

Also, thematic synthesis is not the same as meta-aggregation, and line by line coding is certainly not a part of meta-aggregation. I believe the authors need to make an informed choice between qualitative evidence synthesis approaches and then follow the guidance through.
Author’s response: Thank you for this comment. Apologies for the confusion but we are not using a meta-aggregation approach for the mixed methods synthesis – only for the qualitative analysis which is done prior to the integration of the two separate streams of evidence. We have edited the mixed methods synthesis section to further clarify our methodological approach which is in accordance with chapter 8 of the JBI manual.
Competing Interests: No competing interests were disclosed. Close
Report a concern

Comments on this article Comments (1)

Version 2

VERSION 2 PUBLISHED 10 May 2022

Revised

Comment

Version 1

VERSION 1 PUBLISHED 15 Jul 2021

Discussion is closed on this version, please comment on the latest version above.

Author Response 05 May 2022

Melissa Sharp, Health Research Board Centre for Primary Care Research, Department of General Practice,, Royal College of Surgeons in Ireland, Dublin, Ireland

05 May 2022

Author Response

We thank the reviewers for their helpful and insightful comments. We have responded to each individual item from each reviewer and have included quoted changes where applicable. In summary, we ... Continue reading We thank the reviewers for their helpful and insightful comments. We have responded to each individual item from each reviewer and have included quoted changes where applicable. In summary, we have edited
1) the introduction section to more clearly relate to the main objectives of the mixed-methods systematic review, and
2) added clarifying information to the methods section. Specifically we
a) clarified information regarding the inclusion criteria (PICO);
b) added additional information about the quantitative systematic review including what study designs and summary formats were eligible and how we defined ‘health literacy’ as an exclusion outcome;
c) clarified information about the data extraction form and data collection;
d) edited the bias and quality assessments section for clarity;
e) clarified the exploration of heterogeneity/subgroup analyses;
f) elaborated and simplified the mixed methods synthesis section; and
g) added a few additional references to support our edits.
We thank the reviewers for their helpful and insightful comments. We have responded to each individual item from each reviewer and have included quoted changes where applicable. In summary, we have edited
1) the introduction section to more clearly relate to the main objectives of the mixed-methods systematic review, and
2) added clarifying information to the methods section. Specifically we
a) clarified information regarding the inclusion criteria (PICO);
b) added additional information about the quantitative systematic review including what study designs and summary formats were eligible and how we defined ‘health literacy’ as an exclusion outcome;
c) clarified information about the data extraction form and data collection;
d) edited the bias and quality assessments section for clarity;
e) clarified the exploration of heterogeneity/subgroup analyses;
f) elaborated and simplified the mixed methods synthesis section; and
g) added a few additional references to support our edits.
Competing Interests: No competing interests were disclosed. Close
Report a concern
Discussion is closed on this version, please comment on the latest version above.

Open Peer Review

Reviewer Status

Reviewer Reports

	Invited Reviewers
	1	2
Version 2 (revision) 10 May 22	read
Version 1 15 Jul 21	read	read

Karin Hannes, KU Leuven, Leuven, Belgium
Ivan Buljan, University of Split School of Medicine, Split, Croatia

Comments on this article

All Comments(1)

Add a comment

Back to all reports

Reviewer Report

25 Views

31 Aug 2022 | for Version 2

Karin Hannes, Research group SoMeTHin'K (Social, Methodological and Theoretical Innovation Kreative), CESO, Faculty of Social Sciences, KU Leuven, Leuven, Belgium

25 Views Cite this report Responses(0)

Approved

I am happy with the responses of the authors.

Competing Interests

No competing interests were disclosed.

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

Respond to this report

Responses (0)

Back to all reports

Reviewer Report

43 Views

15 Feb 2022 | for Version 1

Ivan Buljan, Department of Research in Biomedicine and Health, University of Split School of Medicine, Split, Croatia

43 Views Cite this report Responses(1)

Approved

Thank you for the opportunity to review this interesting protocol.

Overall, the research plan is sound and described in detail. I have only a few minor suggestions to propose to make the protocol more precise.

In Methods sections, under Eligibility criteria, I propose that you list all research designs that will be included in the analysis. Currently, only RCT-s are included, but it is not stated will all types of RCT-s will be included (parallel, crossover, etc). On top of that, what about non-randomized trials and control before and after studies? Will those studies be included, too?

The sentence: "We will not be including outcomes related to health literacy, numeracy, nor risk communication in patient-centred care." is a bit problematic. Based on my understanding, health literacy is the examination of knowledge and/or understanding of health information, which will be the case of most research studies you include in the review. Maybe you wanted to state that you will not include studies that assessed health literacy with specific tests of health literacy?

Would you consider including Open Science Framework registry as one of the databases in which you would perform a literature search?

Minor points:

"If we have a sufficient number of included studies, subgroup analyses will be performed to investigate differences based on participant groups (e.g., clinicians versus policy makers) and outcomes (e.g., understanding, acceptability, etc.)" What would be considered a sufficient number of studies?

I hope those comments will help you in the process.

Is the rationale for, and objectives of, the study clearly described?

Yes
Is the study design appropriate for the research question?

Yes
Are sufficient details of the methods provided to allow replication by others?

Partly
Are the datasets clearly presented in a useable and accessible format?

Not applicable

Competing Interests

No competing interests were disclosed.

Reviewer Expertise

Public health; Health communication

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

Respond to this report

Responses (1)

Author Response

05 May 2022

Melissa Sharp, Health Research Board Centre for Primary Care Research, Department of General Practice,, Royal College of Surgeons in Ireland, Dublin, Ireland

Thank you for the opportunity to review this interesting protocol.
Overall, the research plan is sound and described in detail. I have only a few minor suggestions to propose to make the protocol more precise.
Author’s response: Thank you for the time spent reviewing our manuscript and for your valuable feedback.

In Methods sections, under Eligibility criteria, I propose that you list all research designs that will be included in the analysis. Currently, only RCT-s are included, but it is not stated will all types of RCT-s will be included (parallel, crossover, etc). On top of that, what about non-randomized trials and control before and after studies? Will those studies be included, too?
Author’s response: Thank you for this question. We have edited the text to explicitly given examples of the types of RCTs that will be included ‘(e.g., parallel, crossover, cluster, stepped-wedge, etc.)’ The other reviewer had a related question regarding inclusion of different study designs. We have chosen to restrict study designs for this review to randomized controlled trials and qualitative studies as we believe these study designs are best able to either 1) control for confounding/interacting factors that could influence one’s attitude towards a format, such as who is disseminating the message (RCTs) or 2) better able to fully explore multiple factors influencing one’s perception (qualitative studies). We chose not to include non-randomised designs or observational studies due to the complexity of factors that influence one’s attitude towards a format. These study designs also relate directly to the objectives of the review (i.e., effectiveness and preferences for/attitudes towards formats).

The sentence: "We will not be including outcomes related to health literacy, numeracy, nor risk communication in patient-centred care." is a bit problematic. Based on my understanding, health literacy is the examination of knowledge and/or understanding of health information, which will be the case of most research studies you include in the review. Maybe you wanted to state that you will not include studies that assessed health literacy with specific tests of health literacy?
Author’s response: Thank you for this comment. We believe that health literacy outcomes are quite distinct from knowledge/understanding outcomes. We recognize that the definition of health literacy has evolved over time and can often mean different things to different people so we have added additional information to the end of the outcomes section to clarify the focus of our project and the outcomes we are including. We have added text to the manuscript to clarify ‘We are aligning our definition of ‘health literacy’ with a recent systematic review on its meaning, which is complex in nature and composed of ‘(1) knowledge of health, healthcare and health systems; (2) processing and using information in various formats in relation to health and healthcare; and (3) ability to maintain health through self-management and working in partnerships with health providers.’ As impacts on one’s individual health or clinical care is not the main focus of this review, we are focusing only on one aspect (2) -- impacts on one’s understanding of knowledge which is constrained to a specific topic that an evidence summary is covering. [1]

Would you consider including Open Science Framework registry as one of the databases in which you would perform a literature search?
Author’s response: Thank you for this suggestion. We did not search Open Science Framework as this registry is more often suited for observational studies and qualitative study protocols are still rare in the published literature. The search functionality of OSF (using Lucene) is also quite limited. For example using ‘evidence synthesis summary formats’ would result in over 23,000 registrations to screen. We did not search any registries for this study and will add this as a limitation in the discussion section of the final manuscript detailing the MMSR findings.

Minor points:
"If we have a sufficient number of included studies, subgroup analyses will be performed to investigate differences based on participant groups (e.g., clinicians versus policy makers) and outcomes (e.g., understanding, acceptability, etc.)" What would be considered a sufficient number of studies?
Author’s response: Thank you for this question. To our knowledge there is no numeric threshold for subgroup analyses for a MMSR. We edited the text to say ‘If we have a sufficient number of included studies for meta-analysis (minimum of 3), we will report information according to participant subgroups (e.g., clinicians versus policy makers), and outcomes (e.g., understanding, acceptability, etc.).’ as the analysis will be descriptive anyway.

View more View less

Competing Interests

No competing interests were disclosed.

Back to all reports

Reviewer Report

62 Views

20 Sep 2021 | for Version 1

Karin Hannes, Research group SoMeTHin'K (Social, Methodological and Theoretical Innovation Kreative), CESO, Faculty of Social Sciences, KU Leuven, Leuven, Belgium

62 Views Cite this report Responses(1)

Approved With Reservations

Is the rationale for, and objectives of, the study clearly described?

Yes
Is the study design appropriate for the research question?

Yes
Are sufficient details of the methods provided to allow replication by others?

No
Are the datasets clearly presented in a useable and accessible format?

Not applicable

Competing Interests

No competing interests were disclosed.

Reviewer Expertise

Methods

Respond to this report

Responses (1)

Author Response

05 May 2022

Melissa Sharp, Health Research Board Centre for Primary Care Research, Department of General Practice,, Royal College of Surgeons in Ireland, Dublin, Ireland

This is a review underpinned by clear and interesting questions, that warrant a mixed method strategy. There is room for improvement though on the level of motivating choices for concepts and finetuning methods.
Author’s response: Thank you for your review and thoughtful suggestions for our manuscript.

The biggest issue I have is on the relation between content and form in the choice for the mediums to be compared. The authors tend to compare forms (?), but attitude towards form is not fully independent to the way messages are framed, nor by whom they are disseminated. How do the authors intend to deal with this in their comparison? (e.g. people might prefer a form such as video but it might fail to ring the message across, as participants are distracted by the form).
Author’s response: Thank you for this comment. This is an interesting point, the studies we include in our review may well demonstrate a disconnect between what formats participants prefer, and what formats may actually produce improvement in outcomes such as knowledge. If included studies present this data we will be able to explore this more detail in our analysis but we suspect from the studies we have seen that this level of detail will not be available. Based on the importance of this suggestion, we will include this as a point of consideration in the discussion in the main manuscript in the protocol, we have made the following amendments to clarify our position on this:

We have added additional information to the ‘bias and quality assessments’ section in the methods regarding this point: ‘These checklists will provide useful contextual information about the included studies such as information about performance bias. Checklists items cover things like intervention assessors and their reflexivity which are important factors to consider as participant attitudes towards summary formats may be influenced by external factors such as who created the summary (e.g., their own vs. an external organisation).’
We agree that the issue is complex and there are many confounders and interacting factors that could influence one’s attitude towards a format. Relatedly, we have chosen to restrict study designs for this review to randomized controlled trials and qualitative studies as we believe these study designs are best able to control for factors such as who is disseminating the message (RCTs) and are better able to fully explore multiple factors influencing one’s perception (qualitative studies). For example, these attitudes can be more fully explored with probes during semi-structured interviews. We have included the use of assessment tools for bias to further investigate issues such as the potential influence of the study assessors (e.g., those who designed or created the format) and their recipients (i.e, the readers/participants). We have also kept our outcomes broad so as to not take a one-dimensional view towards preference for one format over another.
We can describe differences between preferences for formats and quantitative outcomes such as knowledge in the analysis where the data is available in included studies. We have clarified this in the analysis section of the methods as follows: “Where data is available, we will compare and contrast reported findings on preference and whether or not preference is aligned with improvement of outcomes of impact such as knowledge.”

I was not fully sure why things like surveys measuring attitudes and opinions of people were not taken along in this review. (I am aware of the fact that guidance on the inclusion of such designs in reviews is scarce though-consider it a free offer to work towards a more comprehensive review type). Particularly, because in the RCTs authors tend to rely on self-reported measures, with LIKERT scales. Here, a comparison could sit with time, rather than comparator. Does people's attitude change when exposed to different formats over time?
Author’s response: Thank you for this comment. The aim of this mixed-methods systematic review was to evaluate the effectiveness and acceptability of different evidence synthesis summary formats for CGDG members. For effectiveness we chose to restrict to RCTs as the most appropriate design to evaluate the effectiveness of the interventions in question. Therefore, observational studies like surveys were not included for the effectiveness question as they would not be appropriate to answer the question posed, as described in the methods section. We did not explicitly exclude RCT designs that had interventions with ‘repeated exposures,’ multiple timepoints for outcome assessment or longitudinal follow ups so if identified such studies will be included. However, these design seem unlikely in this context. While policy makers, clinicians, and methodologists could potentially be involved in multiple guideline development groups, this would likely be over quite an extended period of time and would likely have objectives related to the decision-making process and attitude change over time. RCTs exploring this seem unlikely and repeated surveys would not provide effectiveness data.

To explore opinions and attitudes, we included qualitative studies and mixed-methods studies with primary qualitative data collection. We choose to focus on primary qualitative data as this data in this context. In keeping with Cochrane guidance, we do not consider a questionnaire survey to be a qualitative study (https://training.cochrane.org/handbook/current/chapter-21). We prioritized inclusion of qualitative data from primary studies over free text from questionnaire surveys as we hypothesized primary data would be richer and thicker and thus more informative. We have clarified this in the methods section as follows: “We prioritized the inclusion of qualitative data from primary studies over free text from questionnaire surveys as we hypothesized primary data would be richer and thicker and thus more informative.”

Sometimes the arguments are not fully clear. In the introduction, attention is paid to problems complicating the conduct of systematic reviews. This is not the core focus of the project. The project is linked to the translation phase of turning review findings into evidence summaries. This is a matter of content, form and channels rather than the SR pre-track issues. It would therefore be more appropriate to highlight the diversity of actors involved in producing guidelines in relation to how communication and dissemination channels have changed because of that. I am a big fan of the authors discussing things like podcasts and multimedia accounts as examples. Naturally, this is linked to a society moving into multimodality and the limitations of numbers and narratives in reaching out to or working with different publics.
Author’s response: Thank you for this feedback. We have edited the introduction to try to truncate the paragraph on systematic reviews and to try to relate more to the core focus of the project. The process of conducting a systematic review is related to the time and resources the team will have to create the summary formats themselves so we have largely kept the information about the ‘pre-track’ problems. Furthermore, we are focusing on multiple types of evidence syntheses and the landscape has recently evolved in this regard. This also directly relates to the search strategy used and inclusion criteria.

The PICO would perhaps work better if clinicians is changed into professionals. It would make it relevant to a broader target group. Also, the comparator could also be no summary at all, for example, reading findings from reviews straight into the guideline procedure. Why are these explicitly excluded from the review?
Author’s response: Thank you for this question. We have changed clinicians to healthcare professionals to be more inclusive. We have excluded no summary at all as a comparator because we do not think it is appropriate to compare the reading experience of a summary format versus none. We have added text to the ‘Quantitative systematic review’ section: Studies where the comparison is no intervention (e.g., the full-text of an evidence synthesis manuscript) will be excluded. Producing a full text of an evidence synthesis without some form of summary is not ‘usual care’. Internationally, organisations, journals and reporting guidelines would consider a summary to be a mandatory component of any report/peer review manuscript produced (e.g., Cochrane, PRISMA guidance, etc.). Therefore no form of summary or abstract ‘is not an appropriate comparator

Do the authors only include qualitative studies where an intervention is evaluated by participants in terms of meaningfulness or applicability? Or do they also include qualitative studies where attitudes, viewpoints and options for form/channel/content are discussed from a more theoretical point of view, or in pilot cases? It is not clear where the selection sits on the qualitative side, but the data collection part suggests the first option.
Author’s response: Thank you for this question. We have chosen the SPiDER approach for the qualitative evidence synthesis as we are not restricting qualitative studies to those that evaluate an intervention in terms of meaningfulness of applicability. Our evaluation outcomes include the ‘Views, attitudes, opinions, experiences, perceptions, beliefs, feelings, understanding’ of participants. The information in the data collection section regarding the qualitative data is just to provide a summary overview of the study’s main findings (outcomes) to help with descriptive analyses. We edited the text to emphasize that NVivo12 coding will be conducted ‘in parallel’ (rather than ‘however, included’ as previously worded).

Perhaps the authors may want to bring a rationale for why they want to limit the formats to textual and audio-visual productions, as there are many more forms available through which evidence is and can be communicated (such as installations, theatre, etc.) Perhaps these forms haven't been studied extensively in the context of guidelines, but in a protocol phase we are not sure what to find yet :-).
Author’s response: Thank you for this suggestion. We did not list any exclusion of formats explicitly, nor did we limit the formats to textual or audio-visual productions. In the list provided, we used e.g. ‘e.g., visual, audio, text based’ (for example) to list potential summary modes we may find rather than ‘i.e.’ (that is) to explicitly list summary modes we are restricting our search to. Our search strategy included specific formats that we thought we might find but we would not exclude other types of communications that weren’t explicitly listed. We have added text to the ‘quantitative systematic review’ section to clarify this ‘We will not exclude a summary format if it is one that we did not explicitly list in our search strategy (Table 3).’

I would find it more practical if the authors would discuss their position towards critical appraisal and how to structure this instead of putting it in the data collection part as an extraction category (it is not clear what will actually be extracted for what purpose). Also, critical appraisal checklist for primary studies have a different purpose than GRADE and CONQUAL that work on a review level (which is clearly not the focus here when talking about the pool of studies included). I think the authors need to spell the two out in more detail, as they are used in different phases of the review process.
Author’s response: Thank you for this suggestion. In reporting this protocol, we have followed the JBI manual and as such the ‘bias and quality assessments’ subheading should be it’s own unique section and does not fall under the data collection section. The JBI critical appraisal tools are mentioned in the data collection section as they are a part of the standard data-extraction form that will be used by the reviewers. These should be two separate sections; apologies if there was a formatting error. We have edited the text within the ‘data collection’ and ‘bias and quality assessments’ sections to try to clarify what will be extracted for what purpose and to clarify the GRADE/ConQual concern. The text changed in the data collection now reads: ‘The following data information will be extracted using the pilot-tested standardized data-extraction form’ and ‘Data from the domains listed within the JBI critical appraisal tools for qualitative and quantitative studies.’
We agree that the critical appraisal checklist for primary studies have a different purpose than GRADE and CONQUAL. We have reworded the information in the ‘bias and quality assessments’ section to clarify that the JBI checklists are for individual studies AND there is no checklist recommended for the final assessment in the entirety of evidence (similar to the GRADE and CONQUAL purposes) for a mixed methods systematic review.

Would quality of a study not rather fit a sensitivity analysis rather than a subgroup analysis? What purpose would it have as a subgroup in the review? It has disappeared in the mixed method part as a subgroup analysis, so wouldn't mention it as a subgroup in other parts of the text.
Author’s response: Thank you for this question. We recognize that the text in the quantitative analysis section was unclear. We have edited the text, deleting the for example list and restricting it only to the subgroups of interest (participant group or summary format type). We note subgroup analyses for participant groups or intervention types as these involve comparisons across the different groups. There are no planned sensitivity analyses for study quality as there will likely be far too much heterogeneity in outcomes (and the studies in general) to be able to compare the different methods.

What do the authors mean with the phrase "If textual pooling is not available, a narrative summary will be presented." Is a narrative summary not already a textual pooling? Unclear.
Author’s response: Thank you for this question. Apologies for this issue. We have edited the text accordingly. ‘If we are unable to pool findings together (i.e., create and present categories), likely due to an insufficient number of studies identified, a narrative summary will be presented.’

While configuration might have different meanings, it is generally not commonly used as a metaphor for a separate type of synthesis. Configuration would make the different parts unrecognizable in the whole. I don't think meta-aggregation would achieve this level of integration. Perhaps integration is better than configuration here.
Author’s response: Thank you for this suggestion. We have reworded the mixed methods synthesis section to make our approach more clear and have added a references to the specific sections of the JBI manual that we are referencing regarding configuration and the thematic analysis (chapter 8 and section 8.2 for methodology; not section 2 which is for SR of qualitative evidence which details the meta-aggregation approach). We are following JBI’s guidance for MMSRs, and have chosen to use a convergent segregated approach. We have added additional text to clarify that ‘There are several methods for integrating qualitative and quantitative evidence syntheses26, [2] in a convergent segregated MMSR. We will use a thematic synthesis method for integration which groups together similar codes, develops descriptive categories (or themes) to create an overall summary of findings. ^{41, 42’}Meta-aggregation is solely used for the qualitative analysis, not for the mixed methods synthesis which includes the integration.

Also, thematic synthesis is not the same as meta-aggregation, and line by line coding is certainly not a part of meta-aggregation. I believe the authors need to make an informed choice between qualitative evidence synthesis approaches and then follow the guidance through.
Author’s response: Thank you for this comment. Apologies for the confusion but we are not using a meta-aggregation approach for the mixed methods synthesis – only for the qualitative analysis which is done prior to the integration of the two separate streams of evidence. We have edited the mixed methods synthesis section to further clarify our methodological approach which is in accordance with chapter 8 of the JBI manual.

View more View less

Competing Interests

No competing interests were disclosed.

Alongside their report, reviewers assign a status to the article:

Approved - the paper is scientifically sound in its current form and only minor, if any, improvements are suggested

Approved with reservations - A number of small changes, sometimes more significant revisions are required to address specific details and improve the papers academic merit.

Not approved - fundamental flaws in the paper seriously undermine the findings and conclusions

[1] 1. Eccles MP, Grimshaw JM, Shekelle P, et al.: Developing clinical practice guidelines: target audiences, identifying topics for guidelines, guideline group composition and functioning and conflicts of interest. Implement Sci. 2012; 7(1): 60. PubMed Abstract | Publisher Full Text | Free Full Text

[2] 2. Health Information and Quality Authority & National Clinical Effectiveness Committee: HIQA NCEC National Quality Assurance Criteria NCG v2 - April 2015. 2015; [cited 2020 Jul 8]. Reference Source

[3] 3. Qaseem A, Forland F, Macbeth F, et al.: Guidelines International Network: Toward International Standards for Clinical Practice Guidelines. Ann Intern Med. 2012; 156(7): 525–31. PubMed Abstract | Publisher Full Text

[4] 4. Schünemann HJ, Wiercioch W, Etxeandia I, et al.: Guidelines 2.0: systematic development of a comprehensive checklist for a successful guideline enterprise. CMAJ. 2014; 186(3): E123–42. PubMed Abstract | Publisher Full Text | Free Full Text

[5] 5. Fretheim A, Schünemann HJ, Oxman AD: Improving the use of research evidence in guideline development: 3. Group composition and consultation process. Health Res Policy Syst. 2006; 4(1): 15. PubMed Abstract | Publisher Full Text | Free Full Text

[6] 6. Wieringa S, Engebretsen E, Heggen K, et al.: Clinical guidelines and the pursuit of reducing epistemic uncertainty. An ethnographic study of guideline development panels in three countries. Soc Sci Med. 2021; 272: 113702. PubMed Abstract | Publisher Full Text

[7] 7. Higgins JPT, Thomas J, Chandler J, et al.: Cochrane Handbook for Systematic Reviews of Interventions. Cochrane. 2020; 6.1: [cited 2021 Feb 23]. Reference Source

[8] 8. Chandler J, Hopewell S: Cochrane methods -- twenty years experience in developing systematic review methods. Syst Rev. 2013; 2(1): 76. PubMed Abstract | Publisher Full Text | Free Full Text

[9] 9. Borah R, Brown AW, Capers PL, et al.: Analysis of the time and workers needed to conduct systematic reviews of medical interventions using data from the PROSPERO registry. BMJ Open. 2017; 7(2): e012545. PubMed Abstract | Publisher Full Text | Free Full Text

[10] 10. Tricco AC, Cardoso R, Thomas SM, et al.: Barriers and facilitators to uptake of systematic reviews by policy makers and health care managers: a scoping review. Implement Sci. 2016; 11(1): 4. PubMed Abstract | Publisher Full Text | Free Full Text

[11] 11. Fontelo P, Liu F: A review of recent publication trends from top publishing countries. Syst Rev. 2018; 7(1): 147. PubMed Abstract | Publisher Full Text | Free Full Text

[12] 12. Bastian H, Glasziou P, Chalmers I: Seventy-Five Trials and Eleven Systematic Reviews a Day: How Will We Ever Keep Up? PLoS Med. 2010; 7(9): e1000326. PubMed Abstract | Publisher Full Text | Free Full Text

[13] 13. Page MJ, Shamseer L, Altman DG, et al.: Epidemiology and Reporting Characteristics of Systematic Reviews of Biomedical Research: A Cross-Sectional Study. PLoS Med. 2016; 13(5): e1002028. PubMed Abstract | Publisher Full Text | Free Full Text

[14] 14. Tangcharoensathien V, Calleja N, Nguyen T, et al.: Framework for Managing the COVID-19 Infodemic: Methods and Results of an Online, Crowdsourced WHO Technical Consultation. J Med Internet Res. 2020; 22(6): e19659. PubMed Abstract | Publisher Full Text | Free Full Text

[15] 15. Wagner G, Nussbaumer-Streit B, Greimel J, et al.: Trading certainty for speed - how much uncertainty are decisionmakers and guideline developers willing to accept when using rapid reviews: an international survey. BMC Med Res Methodol. 2017; 17(1): 121. PubMed Abstract | Publisher Full Text | Free Full Text

[16] 16. Tricco AC, Garritty CM, Boulos L, et al.: Rapid review methods more challenging during COVID-19: Commentary with a focus on 8 knowledge synthesis steps. J Clin Epidemiol. 2020; 126: 177–183. PubMed Abstract | Publisher Full Text | Free Full Text

[17] 17. Oliver K, Innvar S, Lorenc T, et al.: A systematic review of barriers to and facilitators of the use of evidence by policymakers. BMC Health Serv Res. 2014; 14(1): 2. PubMed Abstract | Publisher Full Text | Free Full Text

[18] 18. Perrier L, Mrklas K, Lavis JN, et al.: Interventions encouraging the use of systematic reviews by health policymakers and managers: A systematic review. Implement Sci. 2011; 6(1): 43. PubMed Abstract | Publisher Full Text | Free Full Text

[19] 19. Petkovic J, Welch V, Jacob MH, et al.: The effectiveness of evidence summaries on health policymakers and health system managers use of evidence from systematic reviews: a systematic review. Implement Sci. 2016; 11(1): 162. PubMed Abstract | Publisher Full Text | Free Full Text

[20] 20. Wallace J, Byrne C, Clarke M: Making evidence more wanted: a systematic review of facilitators to enhance the uptake of evidence from systematic reviews and meta-analyses. Int J Evid Based Healthc. 2012; 10(4): 338–46. PubMed Abstract | Publisher Full Text

[21] 21. Marquez C, Johnson AM, Jassemi S, et al.: Enhancing the uptake of systematic reviews of effects: what is the best format for health care managers and policy-makers? A mixed-methods study. Implement Sci. 2018; 13(1): 84. PubMed Abstract | Publisher Full Text | Free Full Text

[22] 22. Rosenbaum SE, Glenton C, Nylund HK, et al.: User testing and stakeholder feedback contributed to the development of understandable and useful Summary of Findings tables for Cochrane reviews. J Clin Epidemiol. 2010; 63(6): 607–19. PubMed Abstract | Publisher Full Text

[23] 23. Maguire LK, Clarke M: How much do you need: a randomised experiment of whether readers can understand the key messages from summaries of Cochrane Reviews without reading the full review. J R Soc Med. 2014; 107(11): 444–9. PubMed Abstract | Publisher Full Text | Free Full Text

[24] 24. Buljan I, Malički M, Wager E, et al.: No difference in knowledge obtained from infographic or plain language summary of a Cochrane systematic review: three randomized controlled trials. J Clin Epidemiol. 2018; 97: 86–94. PubMed Abstract | Publisher Full Text

[25] 25. Conway A, Clarke MJ, Treweek S, et al.: Summary of findings tables for communicating key findings of systematic reviews. Cochrane Database Syst Rev. 2017; (2). Publisher Full Text

[26] 26. Lizarondo L, Stern C, Carrier J, et al.: Chapter 8: Mixed Methods Systematic Reviews. In: Aromataris E, Munn Z, editors. JBI Manual for Evidence Synthesis. Joanna Briggs Institute; 2020; [cited 2021 Feb 23]. Publisher Full Text

[27] 27. Bressan V, Bagnasco A, Aleo G, et al.: Mixed-methods research in nursing - a critical review. J Clin Nurs. 2017; 26(19–20): 2878–90. PubMed Abstract | Publisher Full Text

[28] 28. Alonso-Coello P, Schünemann HJ, Moberg J, et al.: GRADE Evidence to Decision (EtD) frameworks: a systematic and transparent approach to making well informed healthcare choices. 1: Introduction. BMJ. 2016; 353: i2016. PubMed Abstract | Publisher Full Text

[29] 29. Zhang Y, Alonso-Coello P, Guyatt GH, et al.: GRADE Guidelines: 19. Assessing the certainty of evidence in the importance of outcomes or values and preferences-Risk of bias and indirectness. J Clin Epidemiol. 2019; 111: 94–104. PubMed Abstract | Publisher Full Text

[30] 30. Liu C, Wang D, Liu C, et al.: What is the meaning of health literacy? A systematic review and qualitative synthesis. Fam Med Community Health. 2020; 8(2): e000351. PubMed Abstract | Publisher Full Text | Free Full Text

[31] 31. McGowan J, Sampson M, Salzwedel DM, et al.: PRESS Peer Review of Electronic Search Strategies: 2015 Guideline Statement. J Clin Epidemiol. 2016; 75: 40–6. PubMed Abstract | Publisher Full Text

[32] 32. Methley AM, Campbell S, Chew-Graham C, et al.: PICO, PICOS and SPIDER: a comparison study of specificity and sensitivity in three search tools for qualitative systematic reviews. BMC Health Serv Res. 2014; 14: 579. PubMed Abstract | Publisher Full Text | Free Full Text

[33] 33. Haddaway NR: citationchaser: an R package for forward and backward citations chasing in academic searching. Zenodo. 2021; [cited 2021 Feb 15]. Publisher Full Text

[34] 34. Page MJ, McKenzie JE, Bossuyt PM, et al.: The PRISMA 2020 statement: An updated guideline for reporting systematic reviews. J Clin Epidemiol. 2021; 134: 178–189. PubMed Abstract | Publisher Full Text

[35] 35. Hoffmann TC, Glasziou PP, Boutron I, et al.: Better reporting of interventions: template for intervention description and replication (TIDieR) checklist and guide. BMJ. 2014; 348: g1687. PubMed Abstract | Publisher Full Text

[36] 36. Harbord RM, Egger M, Sterne JAC: A modified test for small-study effects in meta-analyses of controlled trials with binary endpoints. Stat Med. 2006; 25(20): 3443–57. PubMed Abstract | Publisher Full Text

[37] 37. Egger M, Smith GD, Schneider M, et al.: Bias in meta-analysis detected by a simple, graphical test. BMJ. 1997; 315(7109): 629–34. PubMed Abstract | Publisher Full Text | Free Full Text

[38] 38. Lockwood C, Munn Z, Porritt K: Qualitative research synthesis: methodological guidance for systematic reviewers utilizing meta-aggregation. Int J Evid Based Healthc. 2015; 13(3): 179–87. PubMed Abstract | Publisher Full Text

[39] 39. Hannes K, Lockwood C: Pragmatism as the philosophical foundation for the Joanna Briggs meta-aggregative approach to qualitative evidence synthesis. J Adv Nurs. 2011; 67(7): 1632–42. PubMed Abstract | Publisher Full Text

[40] 40. Hong QN, Pluye P, Bujold M, et al.: Convergent and sequential synthesis designs: implications for conducting and reporting systematic reviews of qualitative and quantitative evidence. Syst Rev. 2017; 6(1): 61. PubMed Abstract | Publisher Full Text | Free Full Text

[41] 41. Sandelowski M, Voils CI, Barroso J: Defining and Designing Mixed Research Synthesis Studies. Res Sch. 2006; 13(1): 29. PubMed Abstract | Free Full Text

[42] 42. 8.2 Concepts and considerations for mixed methods systematic reviews. JBI Manual for Evidence Synthesis. 2019. Reference Source

[43] 43. Thomas J, Harden A: Methods for the thematic synthesis of qualitative research in systematic reviews. BMC Med Res Methodol. 2008; 8: 45. PubMed Abstract | Publisher Full Text | Free Full Text

[44] 44. Braun V, Clarke V: Using thematic analysis in psychology. Qual Res Psychol. 2006; 3(2): 77–101. Publisher Full Text

[45] 45. Clyne B, Sharp MK: Evidence synthesis and translation of findings for national clinical guideline development: addressing the needs and preferences of guideline development groups. 2021. http://www.doi.org/10.17605/OSF.IO/SK4NX

Evidence synthesis summary formats for clinical guideline development group members: a mixed-methods systematic review protocol

Abstract

Keywords

Revised Amendments from Version 1

Introduction

Objectives

Protocol

Eligibility criteria

Table 1. PICO for the quantitative review of effectiveness.

Table 2. SPiDER for the qualitative evidence synthesis.

Table 3. Ovid MEDLINE search strategy.

Information sources and search strategy

Data management and selection process

Data collection

Bias and quality assessments

Quantitative analysis

Qualitative analysis

Mixed methods synthesis

Registration and amendments

Dissemination of information

Discussion

Data availability

Reporting guidelines

Acknowledgements

References

Comments on this article Comments (1)

Open Peer Review

Comments on this article Comments (1)

Open Peer Review

Reviewer Status

Reviewer Reports

Comments on this article

Are you a HRB-funded researcher?

Thank you!

Evidence synthesis summary formats for clinical guideline development group members: a mixed-methods systematic review protocol

Abstract

Keywords

Revised Amendments from Version 1

Introduction

Objectives

Protocol

Eligibility criteria

Table 1. PICO for the quantitative review of effectiveness.

Table 2. SPiDER for the qualitative evidence synthesis.

Table 3. Ovid MEDLINE search strategy.

Information sources and search strategy

Data management and selection process

Data collection

Bias and quality assessments

Quantitative analysis

Qualitative analysis

Mixed methods synthesis

Registration and amendments

Dissemination of information

Discussion

Data availability

Reporting guidelines

Acknowledgements

References

Comments on this article Comments (1)

Open Peer Review

Comments on this article Comments (1)

Open Peer Review

Reviewer Status

Reviewer Reports

Comments on this article

Competing Interests Policy

Stay Updated

Are you a HRB-funded researcher?

Thank you!