Exploring Discrepancies between Protocols and Published Scoping Reviews in Implementation Science: Protocol for a Methodological Study

Introduction

Availability of evidence synthesis protocols has become commonplace, to increase transparency regarding review methodology and potential sources of bias in this process (Higgins et al., 2023). Preparation of protocols is beneficial in ensuring careful planning and documentation of the intended method, enabling a more consistent approach within the review team, and reducing arbitrary decision making later in the process (Moher et al., 2015; The PLoS Medicine Editors, 2011). Making review protocols publicly available also enables readers to identify changes made to the planned approaches, allowing them to consider whether such deviations may have biased the results obtained and the interpretations drawn from these.

Multiple studies demonstrate that discrepancies exist between systematic reviews and their protocols (Kirkham et al., 2010; Koensgen et al., 2019; Pandis et al., 2015; Parsons et al., 2019; Silagy et al., 2002; Tricco et al., 2016). Various terms have been used in previous studies, such as ‘difference’ (Koensgen et al., 2019), or ‘change’ (Silagy et al., 2002) but ‘discrepancy’ appears to be the most commonly used term across studies. This is the case both in studies of systematic reviews (Kirkham et al., 2010; Pandis et al., 2015; Parsons et al., 2019; Tricco et al., 2016), and in studies of reporting discrepancies between clinical trials and their trial registry records (e.g. He et al., 2025; Hudson et al., 2020; Matvienko-Sikar et al., 2024; Rivero-de-Aguilar et al., 2024). While explicit definitions of this term are generally lacking in previous studies, discrepancies have been previously defined as any incongruity between the content of a manuscript and its associated protocol (TARG Meta-Research Group and Collaborators, 2022, p2). This could, for example, involve the addition, removal, or modification of methodological details. Such classifications of discrepancies as additions, removals or modifications have been used in previous studies investigating protocol-review discrepancies in outcome reporting specifically (Chan et al., 2004; Kirkham et al., 2010; Pandis et al., 2015).

While some changes to the review plan may be required for logistical reasons, unexpected challenges or complexity, changes could also be influenced by reporting bias or publication bias which may undermine the reviews’ credibility. When undisclosed, this increased risk of bias is disguised from the reader, hindering their ability to judge the validity of the study (TARG Meta-Research Group and Collaborators, 2022). Several studies have demonstrated such discrepancies within systematic reviews, despite being widely considered to be a gold standard of evidence synthesis. For example, Tricco et al. (2016) report that one-third of systematic reviews contained a discrepancy in their primary outcome, while Koensgen et al. (2019) report that almost all systematic reviews they investigated had at least one methodological difference between their protocol and review. Furthermore, Silagy et al. (2002) claim that even when changes may be beneficial to a study, these changes and the rationale for them should still be clearly documented in the report of the final review.

No evidence appears to exist regarding discrepancies in scoping reviews. Scoping reviews are defined as “a type of evidence synthesis that aims to systematically identify and map the breadth of evidence available on a particular topic, field, concept, or issue, often irrespective of source (i.e., primary research, reviews, non-empirical evidence) within or across particular contexts. Scoping reviews can clarify key concepts/definitions in the literature and identify key characteristics or factors related to a concept, including those related to methodological research” (Munn et al., 2022, p950). Therefore, while such reviews should still be conducted systematically, they tend to be more exploratory and more inclusive of different types of evidence, compared to approaches such as systematic reviews. However, it is possible to set narrower boundaries for scoping reviews, including through the specificity of the predefined eligibility criteria such as focusing on particular subgroups, study designs, or time periods (Arksey & O'Malley, 2005; Levac et al., 2010).

Although scoping reviews continue to be a popular form of evidence synthesis, they are often criticised for their poor conduct (Khalil et al., 2021; Xue et al., 2024). Tricco et al. (2016) found that key steps recommended by JBI were commonly not used or reported, such as screening studies in duplicate, or use of a study flow diagram. Additionally, they report that protocols were apparent for only 13% of the scoping reviews they examined. Khalil et al. (2021) and Xue et al. (2024) also report that availability of such protocols appears low. This is perhaps unsurprising, since the use of protocols in scoping reviews has only been advocated for since JBI released its conduct guidance on scoping reviews in 2020 (Peters et al., 2020). The fact that PROSPERO does not publish scoping review protocols may also discourage some reviewers from sharing their protocols, even if they are accustomed to doing so for systematic reviews, although other repositories such as the Open Science Framework (OSF) remain available to facilitate this. As scoping reviews are more flexible and iterative than systematic reviews (Mak & Thomes, 2022; Peters et al., 2020; Peters et al., 2021; Peters et al., 2022), deviations from planned approaches may be more common in scoping reviews than in other review types. For example, adjustments to their questions, inclusion/exclusion criteria and search can be made during the review process (Levac et al., 2010). Furthermore, the complexity of the concepts and information being considered (e.g. Thomas et al., 2020), as well as the large body of literature and wide range of study designs typically included in scoping reviews (Pawliuk et al., 2021) may lead to unforeseen challenges in time and resources, necessitating methodological changes. Authors may struggle to decide when changes are required, how to navigate these, and how to report deviations transparently. Reviewers and editors may also experience challenges in evaluating reviews that include such deviations, particularly as it has been suggested that many reviewers and editors lack the necessary expertise to evaluate scoping review methodology in general (Khalil et al., 2021). There is therefore a need for clearer guidance on navigating and reporting such methodological changes during the scoping review process, to aid not only authors but also those evaluating such reviews.

Scoping reviews are commonly used in implementation science as they allow researchers to map the available literature, define concepts and identify research gaps. Their popularity within this discipline may be due to the approach’s versatility. Implementation science is applicable across many different settings, populations and topics, and is a highly multidisciplinary field which utilises concepts, theories and methods drawn from other disciplines. Scoping reviews’ suitability for exploring broad topics across large volumes of literature therefore enables them to accommodate the wide and diverse scope of literature that may be relevant to implementation. Furthermore, the holistic overview provided by scoping reviews may allow to better capture the complexity of the topics which implementation science investigates. As implementation science is intended to be a practical and applied discipline, any biases in the evidence base may be detrimental to the real-world settings in which the insights gained from scoping reviews are used. It is therefore important to understand to what extent methodological discrepancies appear to occur within scoping reviews in implementation science in particular, and to promote transparent reporting of such discrepancies.

Methods

Discussion

To the best of our knowledge, this is the first study examining discrepancies in the conduct or reporting of scoping reviews in any discipline. It is expected that the findings of this study will reveal the extent to which the published scoping review studies adhere to the approach planned in their publicly available protocols, whether any discrepancies are acknowledged in the review and if so, the reasons reported for any changes made and justifications for alternative approaches chosen. This may help to demonstrate the need for more careful planning and conduct of scoping reviews, and support calls for scoping reviews to have an associated protocol as an important standard of rigour (e.g. Khalil et al., 2022; Peters et al., 2022).

The interpretations that can be drawn from the study’s results will be limited by the study’s observational nature and its reliance on the clarity and detail provided in the sampled publications. Due to the dearth of literature on this topic, however, this study serves as an important first step in exploring the extent of, and solutions to, this issue.

By understanding the extent and nature of discrepancies and the reasons for these, this study can inform future guidance for researchers conducting scoping reviews, particularly when writing protocols, enabling them to better anticipate and mitigate common challenges. The findings could also inform the development of recommendations for reporting protocol-review discrepancies. Such guidance could in turn help to inform reviewers and aid their decision-making when handling scoping reviews or other forms of evidence synthesis containing changes to the methodological approach between the protocol and final publication stages. Additionally, the insights gained from the findings could be used to inform future scoping review training. Future research, informed by the current study’s findings, could also contribute towards each of these longer-term goals.