Application of the intervention Complexity Assessment Tool for Systematic Reviews within a Cochrane review: an illustrative case study

Assessment of intervention complexity using the iCAT_SR

Assessment of intervention complexity was performed independently by two reviewers (CC, AR) using the iCAT_SR (Table 1). Both reviewers are experienced health services researchers who led on the most recent update of the Cochrane review and, therefore, had in-depth knowledge of the included studies. In order to mitigate against a lack of detailed reporting in published study reports, study authors were emailed to request further information (e.g. intervention protocols).

For each intervention, key information was extracted using a purposefully developed data extraction form (Extended data File 1¹³). In completing the assessments, the iCAT_SR guidance document⁷ was used as the coding manual. For each complexity dimension, the assessment level and criteria detailed in the guidance document were applied. The coders identified relevant information from the description of the intervention under each assessment dimension, assigned a complexity rating and provided support for their assessments. Notes were taken during the coding process on any issues with applying the iCAT_SR based on the assessment levels/criteria and any refinements that were needed for the guidance document.

To aid graphical presentation of results, assessment levels across each dimension were categorised as ‘high’, ‘intermediate’, ‘low’ or ‘unclear’ using the definitions in the published tool (Table 1). Cohen’s Kappa coefficient was used to assess inter-rater reliability between the two reviewers (≤0.2 = poor agreement, 0.21–0.40 = fair agreement, 0.51–0.6 = moderate agreement, 0.61–0.8 = substantial agreement, 0.81–1.00 = good agreement)¹⁴. Any disagreements were resolved by discussion with another member of the research team (SL) who led on the development of iCAT_SR⁶.

Interventions

Overall, 19 studies examined pharmaceutical care-based interventions across various settings. Pharmaceutical care reflects a systematic approach to the provision of care that ensures patients receive the correct medication, at appropriate doses, for appropriate indications. It typically involves medication reviews by pharmacists in collaboration with physicians, patients and carers³⁷. One study²⁹ evaluated a single component intervention in the form of computerised decision support that was provided to general practitioners (GPs) in their own practices. Further details about the interventions are summarised in Extended data File 2¹³.

iCAT_SR complexity assessments

Overviews of the intervention complexity assessments across the six core dimensions for the 20 studies are displayed in Table 3. A brief outline of the assessments under each of the iCAT_SR dimensions is provided in the subsections below. A detailed breakdown of individual iCAT_SR assessments for each study (including justifications for assigned ratings) is provided in Extended data File 3¹³. The inter-rater reliability was substantial (Cohen's Kappa = 0.75)¹⁴.

Efforts were made to apply the iCAT_SR optional dimensions. However, a lack of detailed information in the study reports made it difficult to apply the iCAT_SR’s optional dimensions consistently across the included studies. For example, none of the included studies reported on the interaction between intervention components or the nature of the causal pathway between intervention components and outcomes.

Organisational levels/categories

Half of the interventions targeted a single category of individuals (n=10, low level complexity rating), comprising healthcare professionals or patients. In the other studies, interventions were categorised as multi-level (n=7, high level complexity rating) when the interventions typically targeted both patients and healthcare professionals, or multi-category (n=3, intermediate level complexity rating), where multiple groups of healthcare professionals (i.e. GPs, pharmacists, nurses) within the same setting were targeted.

Target behaviours/actions and active components

All of the interventions were deemed to be multi-target (high level complexity rating) in that they involved multiple target behaviours/actions. In all studies, this included appropriate prescribing for older patients receiving polypharmacy and the component behaviours/actions (e.g. reviewing prescriptions, implementing prescribing changes).

With the exception of one study, all of the interventions involved more than one component. Tamblyn et al. evaluated an intervention comprising computerised decision support which was categorised as a single component intervention (low level complexity rating)²⁹. The interventions involving more than one component were further subcategorised according to whether there was a defined order to the delivery of intervention components. There were 17 interventions delivered as a bundle (as opposed to an intervention package) as there was an order/sequence to the delivery of the interventions components (high level complexity rating). This was evident in a number of the hospital-based studies that involved the application of prescribing criteria to identify potentially inappropriate medications, which were then communicated to relevant members of a patient’s medical team and prescribing changes subsequently implemented^19,22.

In two studies^16,32, there was no apparent order to the delivery of intervention components and the interventions were therefore categorised as packages as opposed to intervention bundles (intermediate level complexity rating). For example, the multi-level intervention evaluated by Bucci et al.¹⁶ targeted both patients and prescribers. Directive guidance was provided to patients with the intention of improving medication adherence, while prescribers were targeted by the pharmacist-led intervention to improve the appropriateness of medication prescribing. However, the order in which the intervention components were delivered was not explicitly stated and it was not clear if the patient-targeted component came, or needed to come, before or after the prescriber-targeted component.

Level of tailoring

Eight interventions were categorised as highly tailored/flexible (high level complexity rating). These interventions typically involved the application of implicit tools (e.g. Medication Appropriateness Index³⁵) in assessing the appropriateness of patients’ medications. Given the judgement-based nature of these types of tools, a higher degree of tailoring/flexibility would have been permitted when being applied at the individual patient level.

Nine interventions were categorised as moderately tailored/flexible (intermediate level complexity rating). These interventions typically involved the application of explicit tools in assessing the appropriateness of patients’ medications (e.g. STOPP/START³⁶, Beers’ criteria³⁸). The rationale for rating explicit tools as moderately flexible was that while they consist of criteria relating to potentially inappropriate prescribing in older people, it is at clinicians’ discretion as to the whether the criteria are applicable to individual patients. Therefore, although these tools consist of pre-specified criteria, there is a degree of flexibility in terms of their application at the individual patient level.

The three interventions categorised as inflexible (low level complexity rating) involved education and training for healthcare professionals^20,23,26. In contrast to the studies involving prescribing tools, the intention with these interventions would be to deliver a defined and clearly specified learning activity which would have arguably less flexibility once it had been developed.

Skill level required by those delivering and receiving interventions

For all 20 studies, assessments of the level of skill for those delivering and receiving interventions were rated as intermediate (intermediate level complexity rating). In both instances, the target behaviours/actions related to appropriate prescribing for older people. This was considered to be within the scope of normal practice of those involved in delivering and/or receiving the interventions (e.g. physicians working in geriatric hospital wards reviewing older patients’ medications), such that no specialisation was deemed to have been required.

Experience in applying iCAT_SR

Despite detailed available guidance on the tool’s application⁷, this work was not without challenges. For example, time needed to be allocated to upskilling the review team on the iCAT_SR and then applying it to the interventions. Hence, this initial coding exercise focused on a subset of studies identified following the initial round of searches for the most recent update of this Cochrane review¹². Therefore, this work is not intended as a definitive assessment of intervention complexity in this field of research but as proof of concept of the iCAT_SR’s application.

The inter-rater reliability was substantial. Most of the observed variation between the individual rater assessments was primarily attributable to differences in the interpretation of the two skill-related dimensions (i.e. skill level required by those delivering or receiving the intervention) between the coders and whether they represented intermediate or high level skills. In assessing these dimensions, it is important to consider the baseline level of skill that would be expected of the individual(s) delivering or receiving the intervention and to note whether the study reports on the skill level required or possessed. For instance, if a study reports that individuals delivering an intervention were highly skilled and qualified/experienced, this does not necessarily mean that this level of skill/experience was required to deliver the intervention. Assessments based on the reported level of skill/experience as opposed to the required skill level could result in the complexity level assessment for this dimension being overestimated.

For the purpose of this coding exercise, appropriate prescribing for older people was deemed to be within the scope of practice of those involved in delivering and/or receiving the interventions (e.g. physicians working in geriatric hospital wards reviewing older patients’ medications), such that no specialisation was considered necessary. Consequently, all skills-related assessments were rated as intermediate. Assessments of skill level requirements should ideally be based on details of prior training. However, this may not always be explicitly outlined in study reports, in which case a judgement is required based on expected baseline skill level. In such cases, we recommend specifying a priori how decisions will be made between each of the assessments levels.

An additional challenge in applying the tool’s coding manual related to the assessment of the target behaviour (i.e. prescribing of appropriate polypharmacy) across included studies. The more precisely a behaviour is defined, the greater the specificity of the barriers and facilitators identified (e.g. reducing overprescribing of benzodiazepines, a commonly identified class of potentially inappropriate medications in older people); however, cases exist where it is not possible to isolate and target one behaviour for change, particularly where multiple interdependent behaviours exist (e.g. ensuring prescribing of appropriate polypharmacy for older people)^40,41. Unless explicitly outlined at the outset for any given intervention, compiling an exhaustive list of these interdependent behaviours (also referred to as sub-behaviours) is practically impossible⁴¹. This was the case with regard to the prescribing of appropriate polypharmacy – although these interventions typically involved medication reviews and implementation of prescribing changes, this represented an over-simplification of the key behaviours/actions. As such, the intricacies of the nursing home-based interventions that involved multi-disciplinary case conferences was not adequately captured^18,34. Consequently, the target behaviour across all included studies was categorised as multi-target. However, this detracted from the potential of this core complexity dimension to discriminate between interventions. Reporting on the behaviours and actions taken as part of interventions more explicitly in future research may help in discriminating more clearly between interventions. In the interim, future research involving application of iCAT_SR may look to prioritise key expected behaviours or actions across interventions⁴⁰.

Finally, a lack of detailed reporting and the absence of an explicit theoretical underpinning across the interventions precluded consistent application of the iCAT_SR’s optional dimensions. These are both well recognised issues with the existing literature^42–47. Consequently, assessments were largely based on the reported interventions without detailed consideration of the context/settings in which they were delivered, the interaction between intervention components and the nature of the causal pathway between intervention components and outcomes. Addressing these widely recognised issues through application of relevant reporting guidelines and checklists^48–51, as well as operationalisation of appropriate theory in future research, would help to ensure more consistent application of the tool’s optional dimensions. This could enhance the tool’s capability of discriminating between interventions and offering plausible explanations for sources of heterogeneity between studies. Reporting on interactions between intervention components and the nature of the causal pathways underlying intervention effects would enhance our understanding of not just what interventions work, but also in understanding what happens when they are implemented⁵². This could ultimately enable systematic reviews of interventions to extend beyond assessing whether interventions are effective or not, to interrogating the role of different intervention components, and exploring how, why and for whom the intervention works, and under what circumstances⁵³.

Interpreting iCAT_SR assessments

The iCAT_SR assessments for individual studies were consistent across three dimensions which related to behaviour and skills. This was to some extent expected as the interventions focussed on improving appropriate prescribing and were delivered by clinicians involved in the care of older patients. For the remaining dimensions, there was no consistent pattern for intervention complexity assessments across included studies even when categorised according to setting. We had considered the application of a scoring system to the iCAT_SR assessments as part of this work. However, we were unable to establish a firm evidence base on which to base one.

Refinements to iCAT_SR

The experience gained from applying the iCAT_SR has identified two potential refinements to the tool and the associated guidance document⁶. Firstly, the assessment dimensions in the original tool have been re-ordered to enable a more logical sequence of conceptualisation and application (i.e. starting with ‘Organisational levels/categories’ and progressing to ‘Behaviour or actions’, ‘Active components’, etc.). Secondly, additional examples have been incorporated to supplement the existing ones which were largely directed towards health system and public health interventions⁷. These may assist with the tool’s application to systematic reviews in other clinical areas. A revised version will be published on the Cochrane Collaboration’s website at a later date.

Relationship to other work in the field

A recent systematic review of interventions to promote active transport to school in children has also applied the iCAT_SR⁵⁴. The authors assigned arbitrary scores to each assessment dimension to calculate a global complexity score for each study and then assessed if there was a correlation between intervention complexity and effectiveness. Considerable variation was reported in global complexity scores across included studies and no correlation was detected between intervention complexity and effectiveness. However, the review authors noted the need for more robust methods of evaluating the relationship between intervention complexity and effectiveness.

A previous overview of systematic reviews seeking to compare the effectiveness of multifaceted interventions and uni-faceted interventions in changing healthcare professionals' behaviour found no compelling evidence that the former were more effective⁵⁵. The current study allowed limited comparison of uni-faceted interventions and multifaceted interventions. The computerised decision support-based intervention evaluated by Tamblyn et al.²⁹ that was previously described in the review as uni-faceted⁴ showed comparable assessments across a number of complexity dimensions to other interventions that were categorised as multi-faceted. This simple dichotomy of interventions according to the presence of one component or more than one component may overlook critical dimensions of complexity that impact on effectiveness. This lends support to the view that it is difficult and probably not useful to create a simple definition of a complex intervention^6,56. Therefore, it may be more appropriate to define complexity in terms of intervention characteristics as opposed to the number of intervention components alone. For example, Guise et al.⁵⁷ have defined complex interventions based on key characteristics that extend beyond the intervention and components and encompass a range of other dimensions including pathway complexity, population complexity, implementation complexity and contextual complexity. It is important to recognise that complexity is not merely a characteristic of the interventions themselves but also a feature of the systems (i.e. context, setting) in which they are delivered⁵⁸.

Implications for use in future reviews

The application of the iCAT_SR was a useful exercise, as it allowed dimensions of complexity to be assessed systematically across included studies. However, the value of including iCAT_SR assessments in future updates of the review is currently unclear, in relation to the additional work involved. Further work is needed to determine if application of the iCAT_SR can help to interpret whether varying levels of complexity impact on intervention effectiveness. Systematic reviews involving larger numbers of studies with data that can be pooled in meta-analyses may allow more detailed analysis of relationships between iCAT_SR assessments and intervention effect sizes using appropriate statistical techniques such as meta-regression. It is important that other systematic reviews focusing on other intervention areas apply the iCAT_SR and report on both the findings and experiences of using the tool. This will help in continuing to refine the tool and establishing methods for determining whether varying levels of complexity impact on intervention effectiveness.

Underlying data

All data underlying the results are available as part of the article and no additional source data are required.

Extended data

Open Science Framework: Application of the intervention Complexity Assessment Tool for Systematic Reviews (iCAT_SR) within a Cochrane review. https://doi.org/10.17605/OSF.IO/N3KW2¹³.

This project contains the following extended data:

Data are available under the terms of the Creative Commons Zero "No rights reserved" data waiver (CC0 1.0 Public domain dedication).