Understanding the Mechanisms that Operate within CHIME: A Realist Review Protocol

Realist review phases

Systematic and scoping reviews have a step-by-step systemic approach to the review of the literature^26–28. This is important for both review types so that they can enhance the reproducibility and transparency of how papers were searched from within the wider literature base^29,30. Realist reviews differ from systematic and scoping reviews as they do not conform to a prescribed method to review the literature¹⁸. Instead, because of the realist’s focus on theory creation and explanation, the review process will be supported by a set of guiding principles rather than a prescribed structure^21,31. As such, for the purposes of this realist review protocol, an essential step in creating the protocol was identifying what methodology to follow. This protocol followed the format proposed by Masterson et al.³² and McCormack et al.³³. The rationale for this is twofold: 1) both studies adhered more strictly to the RAMESES-I guidance and 2) both studies were situated within either the co-design space³² or within an aspect of mental health service provision³³. McCormack and colleagues³³ in particular note that a realist review comprises of five generalised steps that were originally devised by Pawson and colleagues³⁴ in their seminal work. These steps are as follows:

These steps are similar to those utilised in generic systematic reviews, but they are notably different in terms of theoretical depth required to complete each step³⁵. Figure 2 illustrates the five major steps of the realist review process adapted from McCormack et al.³³. Additionally, a sixth step will be added to examine the ethics and dissemination of results, given that the present paper represents a protocol for a proposed realist review. Each step is described in detail below.

Figure 2. Figure depicts the five phases of Pawson and colleagues process of conducting realist reviews as adapted from the work of McCormack and colleagues.

Step one: setting up the review advisory panel and constructing the initial programme theories. Realist reviews are, by their very nature, are cyclical and iterative in their approach³⁶. This means that the refinement of the initial programme theories constructed here occurs throughout the review itself³⁷. In realist reviews, the first step is to identify a research focus and construct and reach an agreement regarding initial programme theories. Initial programme theories are both a theory of change and a theory of implementation and as such are iterative and subject to change. As noted previously, for this realist review, our focus will be to explore how and why the mechanisms within the CHIME [Connectedness, Hope, Identity, Meaning and purpose and Empowerment] framework work in individuals who are in recovery from mental health challenges. Given that realist reviews are explicitly theory orientated along with an emphasis on “how” and “why” within the focus statement above, this review approach is suitable for exploring the mechanisms that bring a person from a place of unwellness to the characteristics of CHIME as noted by Leamy et al.,^8,25,38.

The next subphase in this first step is to break down this focus into a series of “If...then...” statements³⁹. If... then statements are simply explanatory accounts or statements that allow one to begin the process of programme theorising⁴⁰. As noted in Table One, each element of CHIME received its own if... then... statement. This is because each element of CHIME works independently and together to support an individual’s personal recovery journey.

Once the if... then... statements are created, they are presented to the team members [ÉNS, JPB, and TB] for review and refinement. Once these if... then... statements are finalised, the next sub-step is to create the initial programme theories⁴¹. Initial programme theories are useful as they help reviewers understand how, for whom, why, and under what circumstances complex interventions such as CHIME work to achieve personal recovery in individuals with mental health challenges⁴². Essentially, an initial programme theory comes from the if... then... statements and consists of three essential parts: context, mechanism, and outcome (CMO)⁴³. In a realist review CMOs, the context refers to the outside parameters of the formal programme architecture. In other words, it relates to the environment in which the mechanism occurs⁴⁴. This can be in the backdrop of a historical event, or perhaps social and cultural norms⁴⁵. However, it is a contextual interaction with the mechanism that produces the outcome. The mechanism is the underpinning generative force that, when interacting with the context, produces the outcome. It is subdivided into resources and how people respond to them. It is often hidden and extremely sensitive to variations in context⁴⁶. Finally, the outcome is the result of the mechanism’s interaction with the context. These outcomes are often behavioural and can be either intentional or unintentional⁴⁵. In short, these components come together to be known as a CMO configuration or an initial programme theory. The premise behind a CMO configuration is that the context and mechanism lead to an outcome. This process involved multiple readings of the if...then... statements in order to clearly assign elements to the context, mechanism, and outcome. Once MJN was satisfied that these elements were aligned correctly, the CMO configurations were circulated to the supervisory team for initial analysis and refinement, resulting in an initial set of plausible programme theories, as presented in Table Two.

Once the initial CMO configurations are constructed, the next subphase is to present these CMO configurations to a group of experts for review. This is necessary in order to ensure that the theories we test the literature with are relevant to the situational context of modern mental health services, which can be determined by consulting experts in this area. From MJNs’ previous work experience, the expert panel was brought together and consisted of six individuals: three academics: Dr. Éidín Ní Shé, Dr. John Paul Byrne, Dr. Tina Bedenik, the Head of MHER, Michael Ryan, the Chair of the National Implementation and Monitoring Committee [NIMC], and former Advancing Recovery in Ireland [ARI] co-lead: Catherine Brogan and Involvement Centre Co-Ordinator and former Peer Educator: David Dwyer. In this way, the expert panel represents a diverse group of academics and local Irish experts who have a key role in implementing CHIME into practice within an Irish context. Éidín Ní Shé is a health systems academic who has a particular proficiency in realist review methodology, having published a number of realist reviews in areas of health systems management. John Paul Byrne is an academic sociologist whose interest spans the social determinents shaping work in health and psychosocial work-life conditions. Tina Bedenik is a senior postdoctural fellow with a particular interent in the ethical applications of AI in clinical decision making. Michael Ryan is a service user by background and has utilised his lived experience to support the creation and implementation of the recovery movement in Ireland. Catherine Brogan is a service provider, originally a psychiatric nurse by trade. She has utilised her practice wisdom to support, along with Michael Ryan, the creation of ARI and has for several years been a leader in creating environments that are recovery orientated within mental health. Finally, David Dwyer is also a service user by background and has utilised his lived experience to support recovery through recovery education and through the management of recovery on the ground through peer-led involvement centres. These individuals were sent an invitation letter [Appendix A] containing the above CMO configurations, along with the meeting date and associated details. Due to the proximity of all parties to one another in the country, the expert panel convened online via MS Teams and lasted approximately an hour and a half. At this meeting, the entire PhD project was presented to the expert panel, with specific attention given to work package one: the realist review. As part of this meeting, each CMO configuration was presented individually, with adequate time given for discussion of all aspects of each CMO configuration under examination. As a result, CMO configurations were added and others were refined with the support of the expert panel. After the meeting, the original CMOs were amended, and a sixth CMO was added based on the discussion. Additionally, these were added to the minutes, which were prepared and circulated among the expert panel members [Appendix B], who were then given the appropriate amount of time to respond and make further amendments to the edited CMO configurations. The finalised and amended CMO configurations are presented in Table Three below.

Step two: searching for evidence. According to Pawson et al.³⁴, the next step of the realist review process will be to test these CMO configurations to see if the literature agrees and aligns with these hypothesised initial programme theories. In order to do this, the first step is to create a search strategy⁴⁷. Search strategy development should be an iterative process that results in a strategy to search the literature that is both sensitive and specific to the topic under investigation⁴⁸. Achieving this can be challenging and often requires several hours to properly construct⁴⁹. As such, obtaining the advice and expertise provided by a librarian is seen as crucial to the process^50,51. In keeping with best practice, KW, an information specialist librarian from the Royal College of Surgeons in Ireland, was consulted in the search strategy development. As a result of the expertise of KWs, Table Four, Table Five and Table Six were constructed to contain the search terms that were devised to support the research team in gathering citations for this realist review from three databases: CINAHL, PubMed, and psychINFO. Grey literature will be gathered through a process of grey literature searching proposed by Godin et al.⁵².

The next step was to construct a pre-determined inclusion and exclusion criteria [Table Seven]. A pre-determined inclusion/exclusion criteria is important, as it will be used to support the shortlisting of articles for this review. For this reason, each inclusion/exclusion criterion was selected for a reason. The rationale for this is now discussed below:

In the identification of appropriate resources for inclusion in this realist review, qualitative and mixed methods papers will be included, whereas quantitative papers will not. The rationale for this is that papers with qualitative data allow for theoretical depth to occur, which is not possible with other quantitative methodologies and positivist positionings. Literature reviews of any kind, including meta-synthesis, will be excluded because of the reviewers’ positionality towards the original raw data used in the original study¹. In other words, these review papers will be excluded as the reviewer is removed at least twice from the original raw data that was collected by the original study authors. As such, if reviews were included, it is possible that such reviews could misinterpret the original findings and make conclusions that are misleading due to the misinterpretation of the data presented by the original studies. Equally, the studies that report on the raw data included in such reviews may also be subject to bias or may not have been of sufficient quality, but yet included in such reviews due to, for example, a lack of papers identified for inclusion. As such, the findings of the review may not be accurate because of the inclusion of poorly conducted studies, which may impede the intended impact of the realist review. Grey literature will also be searched for relevant papers through a the process described by Godin and colleagues⁵². Godin and colleagues approach involves a four step process: 1) searching grey literature databases, 2) customised Google search engines, 3) targeted websites and 4) consultation with contact experts⁵². The articles to be collected will be from 2011 to the present [2025]. Articles published before 2011 will not be included as they would have been published before the literature review that created CHIME was published. Additionally, pre-2011 papers would have been captured by the original systematic review and narrative synthesis that constructed CHIME. Articles not published in English were excluded due to the inability of the review team in translating articles from other languages to English. Articles will be included only if they are relevant to mental health and illness. Articles discussing a dual diagnosis of any kind will be excluded because of the difficulties in distinguishing changes relating and not relating to mental health and illness alone. Finally, papers were also excluded if they spoke of an aspect of CHIME on their own and included if they spoke of all concepts of CHIME amalgamated together. Articles that examined other areas of mental health recovery were excluded from the review.

Once the inclusion/exclusion criteria are defined, the next step is to search the literature to determine whether these initial programme theories are credible or not. The planned search will use an exhaustive screening process that describes a systematic process whereby all citations that arise from applying the search strategy to databases will be included as part of round one screening. As part of the creation of this realist review protocol, the search strings were applied to the relevant databases, and all citations that arose were then placed into a citation software package Covidence to help manage the remainder of the screening process, which will be discussed in further detail as part of step three of the realist review process devised by Pawson et al.³⁴. Round one of the exhaustive screening process will be undertaken by both MJN and Information Specialist Librarian KW.

Step three: select and appraise evidence. After round one of exhaustive searching concludes, a number of citations will have been found and added to the citation software package Covidence for the review team to review and narrow down further. Round two screening then occurs. Round two screening involves firstly eliminating any duplicates that may have been gathered as part of round one screening and then applying pre-determined inclusion/exclusion criteria [Table Four] to the abstracts of the remainder of the citations gathered over the course of round one screening. Round two screening will be undertaken by MJN and ÉNS, who will screen 10% of the papers at this phase to ensure rigor and adherence to the pre-determined inclusion/exclusion criteria⁵³. Finally, round three screening will involve the application of a pre-determined inclusion/exclusion criteria [Table Four] to the full text of citations included in this study. Again, this round was conducted by MJN with ÉNS, reviewing 10% of papers at this stage of searching also. If there are any disagreements regarding the inclusion or exclusion of a paper, then a third reviewer [JPB] will review the citation and make the ultimate decision regarding the paper’s inclusion or exclusion. Any citations left after the three rounds of screening were included in this realist review. As such, the process described above is similar to that of an original systematic review with a particular focus on thoroughness and transparency⁵⁴.

This realist review will appraise included studies for richness, relevance, and rigor in line with the realist methodology of Pawson et al.³⁴. According to Pawson et al.,³⁴ relevance relates to whether the paper addresses the theory under investigation. In addition, rigor relates to how the research supports the conclusions drawn by the author of the paper being examined³⁴. Finally, richness refers to whether a paper has sufficient depth to meaningfully contribute to theory-building²¹. Unlike systematic reviews that appraise for quality purposes^55–57, in realist reviews, one appraises relevance as the development of theory²¹. However, there is no agreed upon method for assessing relevance in realist reviews⁵⁸. As such, in this review, when appraising included studies, the following questions will be posed to the literature:

Step four: extract the data. Data will be gathered only from primary sources because of the issue of positionality when it comes to secondary sources, as noted by Norton¹. When it comes to primary sources, each citation will be examined thoroughly to identify the key aspects of each study, including the CMO configuration. Data collected from the citations that match these areas will be extracted by MJN into a data extraction form that will be developed by MJN prior to the commencement of this stage of Pawson and colleagues’ realist methodology. This data extraction document will be validated through consultation with the supervisory members of the review team (ÉNS, JPB, and TB). Once approved, this data extraction tool will test 10% of included studies using the below key aspects from each citation:

Once 10% of studies have their data extracted, ÉNS will review the data extraction tool and the data collected from extraction. If she is satisfied by the tool’s performance on 10% of included paper, the primary reviewer [MJN] will utilize this tool for the remainder of the papers to be extracted. The process of data extraction is iterative, meaning that the data extraction tool will be constantly refined over the course of the realist review process¹⁸. It is also important to note that each paper may not contain a context, a mechanism or an outcome and as such, this can only be identified if the particular IPT is in itself identified. In other words, this protocol acknowledges that one cannot infer that something functions as a context or mechanism without first knowling the outcome it is relating to. In addition, rigor will be captured using the Crowe Critical Appraisal Tool⁵⁹. It was decided to utilise this tool as it has been utilised in another realist review within the health sciences space⁶⁰.

Step five: analyse and synthesise data. The primary purpose of data synthesis is to determine what works, for whom, how, and under what circumstances, all in an attempt to refine the initial programme theories³⁵. In this way, the word synthesis in this context differs from the same word for systematic reviews, as in this process, the purpose of synthesis is to make progress in the explanation of initial programme theories into refined intial programme theories⁶¹. In this realist review, two mechanisms are used to refine the initial programme theory. Firstly, summative content analysis will be conducted and will support the process of retroductive reasoning in order to develop the refined initial programme theories. Summative content analysis was devised by Hsieh and Shannon⁶², and it involves an iterative process of identifying keywords based on the stated headings in stage four of Pawson and colleagues’³⁴ realist methodology to create a set of refined intial programme theories. Once these refined intial programme theories are defined, the second process is a validation exercise where our subject experts appointed during stage one of the realist methodology will reconvene to approve the newly developed refined intial programme theories.

Step six: ethics and dissemination. In line with similar realist reviews, ethical approval will not be required, as although we have an expert panel consisting of academics [ÉNS, JPB, TB] and expert panel members [MR, CB, DD], they have all been included in the authorship of this paper. In addition, no ethical approval is required, as no primary data collection will take place for the purpose of this review. They will also be included in the authorship of the realist review itself. Reporting bias will be contained through a process of reflexivity where the primary reviewer [MJN] will engage in journalling to support the process of retroductive reasoning whilst also acknowledging any reporting bias that may impede the fair development of the refined intial programme theories. The complete review will be presented as a second article and submitted for publication in a peer-reviewed journal. In addition, the results of this proposed realist review may also be presented through other means, including conferences and recovery workshops delivered by recovery college personnel. This will be made possible through the Recovery and Engagement Programme Manager in charge of the recovery education workstream of MHER.