The impact of a run-in period on treatment effects in cardiovascular prevention randomised control trials: A protocol for a comprehensive review and meta-analysis [version 1; peer review: 1 approved]

Background: A run-in period is often employed in randomised controlled trials to increase adherence to the intervention and reduce participant loss to follow-up in the trial population. However, it is uncertain whether use of a run-in period affects the magnitude of treatment effect. Methods: We will conduct a sensitive search for systematic reviews of cardiovascular preventative trials and a complete meta-analysis of treatment effects comparing cardiovascular prevention trials using a run-in period (“run-in trials”) with matched cardiovascular prevention trials that did not use a run-in period (“non-run-in trials”). We describe a comprehensive matching process which will match run-in trials with non-run-in trials by patient populations, interventions, and outcomes. For each pair of run-in trial and matched non-run-in trial(s), we will estimate the ratio of relative risks and 95% confidence interval. We will evaluate differences in treatment effect between run-in and non-runin trials and our and our priamry outcome will be the ratio of relative risks for matched run-in and non-run-in trials for their reported cardiovascular composite outcome. Our secondary outcomes are comparisons of mortality, loss to follow up, frequency of adverse events and methodological quality of trials. Conclusions: This study will answer a key question about what influence a run-in period has on the magnitude of treatment effects in randomised controlled trials for cardiovascular prevention therapies.


Introduction
A pre-randomisation run-in period is used widely in randomised controlled trials evaluating cardiovascular preventative therapies, with intended advantages including exclusion of non-adherent subjects, placebo responders, non-responders, or those who experience early side effects or could not tolerate the intervention [1][2][3][4][5] . Run-in is reported to be most advantageous in long-term trials in which loss of adherence may adversely affect the ability to obtain a clear result, or in which cumbersome follow-up regimens leads to poor adherence 6 . However, there is also concern that use of a run-in period reduces external validity, as the trial population does not include participants intolerant of the study intervention, and those adherent with medications may be less representative of a general population 1,7 .
In this study, we will evaluate whether use of run-in is associated with differences in treatments effects (efficacy and safety) and adherence with study medications, employing a type of nested case-control review of randomised controlled trials (RCTs) of cardiovascular therapies of proven effectiveness. We will perform a meta-analysis of the ratio of relative risks between matched run-in and non-run-in trials. This will answer the questions of whether use of a run-in period truly influences treatment effects in large scale clinical trials, and whether it truly improves long-term adherence with preventative therapies.

Data sources
We have identified high quality previous systematic reviews of cardiovascular preventative therapies and will extract data these reviews assessing the treatment effect of anti-hypertensive 8-12 , lipid lowering 13-16 , and glucose lowering drugs 17,18 in primary and secondary prevention trials. These groups of interventions were selected because of their established effectiveness in reducing cardiovascular events, and therefore, suited to determining whether run-in enhances treatment effect. This strategy of sourcing trials for inclusion from a range of published systematic reviews will allow us to reduce research waste 19 . To allow a consistent comparison of treatment effects trials in the individual systematic reviews will be eligible for inclusion in the matched meta-analysis of treatment effects if an active agent is compared with a placebo control, or if an active agent in addition to standard therapy is compared with standard therapy.

Data extraction
We will repeat primary data extraction for all papers to confirm accuracy. Each of the extraction variables will be extracted from the selected papers by pairs of researchers (researcher one and researcher two) (Extended data, Supplementary Appendix 1 20 ). All data points will be initially extracted by a designated researcher (researcher one in the pair). Extracted data will be then double-checked by a second independent researcher (researcher two in the pair). Any discrepancies between researcher one and researcher two will be noted during consolidation and then resolved by consensus with the senior author (CJ). The first author (RM) will complete a final double check of all data prior to completion of statistical analysis.

Matching protocol
Population, Intervention, Control, Outcome (PICO) summaries will be extracted for each trial. Matching data entry forms will be autogenerated using custom software developed in R (V3.5.3 "Great Truth"). Each matching data entry form will present the matcher with trial name, population description (sex, mean age, inclusion criteria, primary or secondary prevention), intervention description (drug name and dose), control description and outcome description (list of cardiovascular outcomes). Matching of run-in trials to non-run-in trials will be performed using a PICO-based matching system, with individual scores given to each match for Population, Intervention, Control, and Outcome (individual components of the PICO acronym). We are adapting a matching scheme used in a previous high-quality systematic review 21 . A score will be given to each potential match based on the following pre-specified criteria (Extended data, Supplementary Appendix 2 20 ). For each component of PICO, a score of 0 is defined as not a match, 1 an acceptable match, 2 a close match, and 3 an exact match.
We will complete the matching in two steps to differentiate essential matching requirements from desirable matching criteria. In step 1 we will match similar patient populations, and essential criterion. For interventions, the initial matching will be based on the mechanism of action of the study drug in question. Comparisons within the same drug class is an essential criterion. For example, all beta-blocker drug trials will be matched together, and all statin drug trials will be matched together. Within each drug class, autogenerated matching data entry forms will be generated, and population matching will be done between each run-in trial, and the corresponding potential non-run-in matches. This population match will be completed by 2 independent assessors with disagreements resolved by consensus with a third reviewer.
In step 2 of the matching process run-in trials with a population matching score of 1 or greater will then be matched based on the intervention, control and outcomes. This will give each match a score ranging from minimum 4 (a score of "acceptable match" in each domain of PICO) to maximum of 12 (a score of "exact match" in each domain of PICO). An algorithm with the different stages of the matching process in given in Figure 1.

Statistical analysis
We will provide a summary of the key differences between the run-in and matched non-run-in studies including journal of publication, impact factor, study design, number of participants randomised, intervention/control sample size, number of participants included in intention-to-treat analysis, population description (including primary or secondary prevention), and outcome measures (number of non-fatal myocardial infarctions, non-fatal strokes, adverse events resulting in permanent discontinuation of drug, all-cause mortality, and cardiovascular composites).
Each run-in trial will be either matched to a single non-run-in trial or matched to a meta-analysis of several non-run-in trials when there is more than one non-run-in trial which shares a similar population. For each run-in and non-runin match, we will calculate the ratio of relative risk and 95% confidence interval by subtracting the log(non-run-in trial relative risk) from the log(run-in trial relative risk). We will then meta-analyse the ratio of relative risks between run-in and matched non-run-in trials to obtain a summary ratio of relative risks. A number less than 1 represents an exaggerated treatment effect of run-in i.e. the relative risk is lower for run-in trial compared to matched non-run-in trial(s). We will test for heterogeneity using the I² statistic.
We will perform sensitivity analyses using a hierarchical approach with three levels. The first level (primary analysis) will allow a non-run-in trial to match only once to a run-in trial. Each non-run-in trial and it's highest corresponding runin PICO score match will be selected. If a non-run-in trial has several equal PICO matches, then the non-run-in trial will be matched with the run-in trial with the largest sample size. This level will optimise precision matching. The second level will allow a non-run-in trial to match multiple run-in trials. Each run-in trial will then be matched to a meta-analysis of several non-run-in trials, across the range of PICO scores. For the third level, we will use a bootstrapping approach. The bootstrapping approach will iterate 1000 times and select one random match from all possible run-in and non-run-in matches. In this analysis, we will meta-analyse the 1000 results to obtain a bootstrapping summary estimate of the ratio of relative risks.

Assessment of the quality of the studies: risk of bias
We will use the Cochrane Risk of Bias Tool 22 to assess methodological quality of eligible trials including random sequence generation, allocation concealment, blinding of participants and healthcare personnel, blinded outcome assessment, completeness of outcome data, evidence of selective reporting and other biases. Risk of bias assessments will be performed independently by two reviewers, and disagreements resolved by a third reviewer. We will create a risk of bias summary table using Review Manager 23 . We will compare the proportion of run-in trials versus non-run-in trials which are low risk of bias.

Discussion
This meta-analysis intends to systematically examine the effects of using a run-in period to address a gap in research methodology literature. We expect to provide the following results: first, we will determine whether use of run-in is associated with different treatment estimates, compared to non-run-in trials, second, it will provide an insight into the proportion of randomised controlled trials in cardiovascular prevention that include a run-in design, second, we will report how run-in affects adherence with treatment and loss to follow-up (test primary purpose of run-in) and third, determine whether event rates (mortality, cardiovascular events, safety events) are different in run-in trials compared to non-run-in trials 24 .
Findings from our systematic review may have implications for use of the run-in in future clinical trial design of cardiovascular preventative therapies. This information may also influence risk of bias assessments, funders and policy makers about the utility of a run-in trial design 25,26 .

Data availability
Underlying data No underlying data are associated with this article.  Murphy and colleagues provide the protocol for a proposed metanalysis quantifying the impact of a pre-randomization run-in period of the reported effect-sizes in preventative cardiovascular trials. Trials that utilize a run-in will be matched with those that do not in terms of their population, intervention type and outcome. PICO summaries will be independently extracted by 2 reviewers, and differences resolved by consensus.

Open Peer Review
The proposed analysis addresses an important question regarding the impact of a run-in period on the internal validity of the trial, as distinct to the more obvious impact on external validity. Some insight as to the mechanism by which the run-in may impact internal validity, -in particular whether it is related to an unmasking effect arising from the pre-randomization exposure to the agent of interest during the run-in period-may be obtained by stratifying the analysis by masking status of the trial pairs, as only the former are at risk for an unmasking effect and resulting biases.
The report is succinct, well written and sufficiently detailed. The potential role of residual confounding especially in the matching criteria will need to be carefully explored in the interpretation of the results of the analysis, which I look forward to seeing in due course.

Are sufficient details of the methods provided to allow replication by others? Yes
Are the datasets clearly presented in a useable and accessible format?

Not applicable
Competing Interests: No competing interests were disclosed.
I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.