The impact of a run-in period on treatment effects in cardiovascular prevention randomised control trials: A protocol for a comprehensive review and meta-analysis

Introduction

A pre-randomisation run-in period is used widely in randomised controlled trials evaluating cardiovascular preventative therapies, with intended advantages including exclusion of non-adherent subjects, placebo responders, non-responders, or those who experience early side effects or could not tolerate the intervention^1–5. Run-in is reported to be most advantageous in long-term trials in which loss of adherence may adversely affect the ability to obtain a clear result, or in which cumbersome follow-up regimens leads to poor adherence⁶. However, there is also concern that use of a run-in period reduces external validity, as the trial population does not include participants intolerant of the study intervention, and those adherent with medications may be less representative of a general population^1,7.

In this study, we will evaluate whether use of run-in is associated with differences in treatments effects (efficacy and safety) and adherence with study medications, employing a type of nested case-control review of randomised controlled trials (RCTs) of cardiovascular therapies of proven effectiveness. We will perform a meta-analysis of the ratio of relative risks between matched run-in and non-run-in trials. This will answer the questions of whether use of a run-in period truly influences treatment effects in large scale clinical trials, and whether it truly improves long-term adherence with preventative therapies.

Methods

Matching protocol

Population, Intervention, Control, Outcome (PICO) summaries will be extracted for each trial. Matching data entry forms will be autogenerated using custom software developed in R (V3.5.3 “Great Truth”). Each matching data entry form will present the matcher with trial name, population description (sex, mean age, inclusion criteria, primary or secondary prevention), intervention description (drug name and dose), control description and outcome description (list of cardiovascular outcomes). Matching of run-in trials to non-run-in trials will be performed using a PICO-based matching system, with individual scores given to each match for Population, Intervention, Control, and Outcome (individual components of the PICO acronym). We are adapting a matching scheme used in a previous high-quality systematic review²¹. A score will be given to each potential match based on the following pre-specified criteria (Extended data, Supplementary Appendix 2²⁰). For each component of PICO, a score of 0 is defined as not a match, 1 an acceptable match, 2 a close match, and 3 an exact match.

We will complete the matching in two steps to differentiate essential matching requirements from desirable matching criteria. In step 1 we will match similar patient populations, and essential criterion. For interventions, the initial matching will be based on the mechanism of action of the study drug in question. Comparisons within the same drug class is an essential criterion. For example, all beta-blocker drug trials will be matched together, and all statin drug trials will be matched together. Within each drug class, autogenerated matching data entry forms will be generated, and population matching will be done between each run-in trial, and the corresponding potential non-run-in matches. This population match will be completed by 2 independent assessors with disagreements resolved by consensus with a third reviewer.

In step 2 of the matching process run-in trials with a population matching score of 1 or greater will then be matched based on the intervention, control and outcomes. This will give each match a score ranging from minimum 4 (a score of “acceptable match” in each domain of PICO) to maximum of 12 (a score of “exact match” in each domain of PICO). An algorithm with the different stages of the matching process in given in Figure 1.

Figure 1. Algorithm with levels of matching considerations.

Statistical analysis

We will provide a summary of the key differences between the run-in and matched non-run-in studies including journal of publication, impact factor, study design, number of participants randomised, intervention/control sample size, number of participants included in intention-to-treat analysis, population description (including primary or secondary prevention), and outcome measures (number of non-fatal myocardial infarctions, non-fatal strokes, adverse events resulting in permanent discontinuation of drug, all-cause mortality, and cardiovascular composites).

Each run-in trial will be either matched to a single non-run-in trial or matched to a meta-analysis of several non-run-in trials when there is more than one non-run-in trial which shares a similar population. For each run-in and non-run-in match, we will calculate the ratio of relative risk and 95% confidence interval by subtracting the log(non-run-in trial relative risk) from the log(run-in trial relative risk). We will then meta-analyse the ratio of relative risks between run-in and matched non-run-in trials to obtain a summary ratio of relative risks. A number less than 1 represents an exaggerated treatment effect of run-in i.e. the relative risk is lower for run-in trial compared to matched non-run-in trial(s). We will test for heterogeneity using the I² statistic.

We will perform sensitivity analyses using a hierarchical approach with three levels. The first level (primary analysis) will allow a non-run-in trial to match only once to a run-in trial. Each non-run-in trial and it’s highest corresponding run-in PICO score match will be selected. If a non-run-in trial has several equal PICO matches, then the non-run-in trial will be matched with the run-in trial with the largest sample size. This level will optimise precision matching. The second level will allow a non-run-in trial to match multiple run-in trials. Each run-in trial will then be matched to a meta-analysis of several non-run-in trials, across the range of PICO scores. For the third level, we will use a bootstrapping approach. The bootstrapping approach will iterate 1000 times and select one random match from all possible run-in and non-run-in matches. In this analysis, we will meta-analyse the 1000 results to obtain a bootstrapping summary estimate of the ratio of relative risks.

Assessment of the quality of the studies: risk of bias

We will use the Cochrane Risk of Bias Tool²² to assess methodological quality of eligible trials including random sequence generation, allocation concealment, blinding of participants and healthcare personnel, blinded outcome assessment, completeness of outcome data, evidence of selective reporting and other biases. Risk of bias assessments will be performed independently by two reviewers, and disagreements resolved by a third reviewer. We will create a risk of bias summary table using Review Manager²³. We will compare the proportion of run-in trials versus non-run-in trials which are low risk of bias.

Discussion

This meta-analysis intends to systematically examine the effects of using a run-in period to address a gap in research methodology literature. We expect to provide the following results: first, we will determine whether use of run-in is associated with different treatment estimates, compared to non-run-in trials, second, it will provide an insight into the proportion of randomised controlled trials in cardiovascular prevention that include a run-in design, second, we will report how run-in affects adherence with treatment and loss to follow-up (test primary purpose of run-in) and third, determine whether event rates (mortality, cardiovascular events, safety events) are different in run-in trials compared to non-run-in trials²⁴.

Findings from our systematic review may have implications for use of the run-in in future clinical trial design of cardiovascular preventative therapies. This information may also influence risk of bias assessments, funders and policy makers about the utility of a run-in trial design^25,26.

Data availability