A review of public health economic modelling in the National Institute for Health and Care Excellence (NICE)

When is modelling required?

The approaches used for economic evaluation in the CPH, as elsewhere in NICE, compared the costs of interventions under consideration with their expected benefits, making explicit how effectively they meet the directorate’s objectives. While there are differing approaches to quantifying the benefits and costs depending on the economic evaluation technique used, the CPH defined their aim broadly as “the promotion of good health and the prevention of ill health”, taking an “inclusive” perspective⁸. Over the long term, this quantification should reduce the potential for inconsistent prioritisation and the opportunity costs associated with this, and lead to the “conscientious, explicit and judicious use of current best evidence in making decisions about the care of individual patients”¹⁴. Evidence of the effectiveness of certain interventions is not always available and it is not always possible in practice to capture the full range of non-health population impacts into the model, and potentially hence into the broader decision process.

The available evidence in public health tends to be from a broader range of settings and hence is sometimes shallower than in pharmacoeconomic studies, at least in terms of the hierarchy of evidence. These hierarchies follow a standardised ranking from randomised control trials to expert evidence, and are based upon the increasing probability of bias being introduced¹⁵. But they are not necessarily appropriate for use in public health settings³. The length of the causal chains for these interventions requires assumptions and judgements to be used, which must be tested through modelling and deliberation as the “principle of the accumulation of results of trials does not sit well with model and theory based sciences”³.

Similarly, expert judgement, deliberation and experience remain vital to decision-making within NICE. There is wide variation in this – the initial evidence review carried out for a topic may find overwhelming evidence of effectiveness or cost-effectiveness, or in other cases effectively no such evidence. Economic modelling is used in the area in between these extremes to better inform decision-makers, where further insight is required to interpret the evidence available. Where it is already clear in advance that the intervention would be cost saving or where it is obvious the costs are relatively small compared to the expected health gains, modelling is not necessary⁹.

How is modelling used?

Interpreting and deciding which evidence is most relevant to the model requires a pragmatic approach, utilising simplifying assumptions. As with any scientific setting, the model can therefore offer only an approximation. The model makes explicit the logical implications of the data available, making it easier for decision-makers to draw rational conclusions given the uncertainty associated with each potential course of action. In many cases ballpark figures will provide a strong indication of whether or not the intervention is likely to prove cost-effective. Many interventions in public health are extremely so; about 15% of those investigated were shown to be cost saving and 89% were below the usual NICE Incremental cost-effectiveness ratio (ICER) threshold of £30,000¹⁶. Probabilistic and sensitivity analyses can be used to further indicate the robustness of a model’s conclusions.

These assumptions and decisions are an inevitable part of the process and “economic modelling requires judgements to be made by both modellers and decision-makers”⁶. There are several stages to this process. NICE usually tenders the evidence review and modelling processes to experts in academia, but CPH staff would liaise throughout to ensure that the methods used were in line with NICE’s requirements. The model must be requisite – parsimonious but good enough to do the job¹⁷ – requiring appropriate balancing between the aims, costs and effects of the interventions under review. Modellers, in collaboration with topic appraisal committees, must also make decisions as to where insufficient data are available to investigate certain interventions further. The study into Workplace interventions to promote smoking cessation (PH5)¹⁸, for example, was initially also intended to investigate mass media interventions, but the previously completed literature review found no relevant evidence of effect; it was therefore not pursued further in the modelling stage. As well as the use of the reference case, the 2012 CPH methods manual⁹ stated that modellers must ensure that:

In some cases, de novo models are required, though on many occasions it is possible to base them upon previous work. Because over time the guidance on specific topics is revised, previous models may be recycled, reducing the work required and easing comparisons between conclusions. There is also a limited number of themes explored in practice, and several tobacco related topics, for example, used a similar simulation-based model indicating the probability of acquiring a number of smoking related conditions (or death) as simulated individuals aged (such as PH5¹⁸, PH10¹⁹, PH14²⁰). Similarly guidance on physical activity reused updated versions of a model incorporating the risk of coronary heart disease, stroke and type 2 diabetes mellitus (e.g. PH44²¹, PH54²²). This allowed not just for comparisons between the effectiveness of interventions versus ‘doing nothing’, but also between topics. Other models are based upon or incorporate pre-existing work previously completed externally to NICE. Economic modelling may also solely focus on a specific part of the guidance or investigate a sub-population of those to whom the broader guidance will apply.

Model perspectives and economic evaluation techniques

The CPH’s reference case, the set of standard approaches to be used in economic modelling and broader decision-making, was intended to be flexible where necessary on a case-by-case basis. This standard approach recommended a public sector perspective, to take account of the costs and benefits of each intervention⁹. The 2014 methods manual¹³ formally allowed for discretion on the perspective used across all NICE directorates for the first time, varying according to the nature of the problem. Previously a public sector perspective was largely confined to public health settings. An NHS and Personal Social Services (PSS) specific perspective is used where costs and benefits are largely related to health alone. This has also previously been used in many cases in public health and is the standard for drug and technology appraisals in NICE. Where these criteria largely fall upon local authorities, a local government perspective can similarly be used. However, in practice which perspective is used can be something of a moot point. In many cases models claiming to use a public sector perspective used only healthcare costs (e.g. PH32²³); in others, no perspective has been explicitly stated. Other perspectives are also possible, such as the employer benefit for workplace interventions (PH5¹⁸) or a societal perspective for example in cases of domestic violence (PH50²⁴). These were codified in the reference case for the first time in the 2012 guidance⁹, but had nonetheless previously been used. The most common economic evaluation techniques used in the CPH were Cost-effectiveness analysis (CEA), Cost-utility analysis (CUA), Cost-benefit analysis (CBA) and Cost-consequence analysis (CCA); for definitions of these techniques see Drummond et al.²⁵.

Economic modelling is informed by a prior evidence review, specifying the range of relevant interventions that have been used in the area in the past, the nature and strength of studies investigating them, and their effectiveness, costs and applicability to a UK context. Where no previous study has directly measured the outcomes under investigation, economic modelling can be used to compute estimates of these outcomes, using parameters populated based upon results derived from the literature discovered as part the evidence review. In cases where it is not appropriate or possible to further model results, generic discussion pieces may still be written to discuss how an economic model would be used if such a thing were possible, and to indicate the magnitude, 'wheres' and 'hows' of the underlying uncertainty that make more formal modelling impossible.

Where possible, the costs of an intervention could be assessed based on the information available in the literature described in the evidence review. This typically included the costs of any device or pharmaceutical required, staff time, monitoring and maintenance costs, treating adverse events, rent and so on. Public health also throws up more unusual consequences; the costs found in the review also included, for example: decreased tax revenues (PH41²⁶); the impact of having to pay for extra years of health care (PH23²⁷ - though these were more than outweighed by the decrease in costs treating the illnesses associated with smoking); and issues relating to injuries occurring as a side effect of the intervention (PH44²¹). The impact upon individuals volunteering to give up their time may be highlighted (PH9²⁸), though this cannot be fully incorporated into CUA as costs to carers and non-patients have in practice been considered outside the scope of NICE’s reference case.

Final decisions rest with the relevant committee for the topic, titled “Public Health Advisory Committees” (PHACs). Prior to the 2012 public health methods manual⁹, there were two different types of committees in their place. The Public Health Intervention Advisory Committee (PHIAC) was a standing multi-disciplinary panel which could look at clear, well defined public health topics, such as “Prevention of sexually transmitted infections and under 18 conceptions”²⁹ and “School-based interventions on alcohol”³⁰. Programme Development Groups (PDGs) were assembled on a topic-by-topic basis, to create guidance in broader, more complicated or less clearly defined areas, such as “Behaviour change”³¹ or “Community engagement”²⁸.

Equity in economic modelling

Issues relating to equity are also not currently formally considered as part of the economic modelling stage, though of course they remain a fundamental part of the broader guidance development process. In a similar manner to the impact of volunteerism, this does not mean that such issues are irrelevant to the modelling stage. The Tuberculosis: hard-to-reach groups (PH37³²) guidance explained how out-of-pocket expenses – again relevant to private individuals rather than to the public sector perspective, which was used in this model – are likely to have a disproportionate effect on homeless populations. As such, they may be relevant not solely to ethical issues around access for these groups, but also in the practical terms of the likely real-life effectiveness of the intervention (assuming that homeless populations will have significantly lower uptake than might otherwise be expected as a result). The committee could then take this into account during the later stages of formalising recommendations. Equally, though the modelling itself may not deal with equity head on, at times the topic itself may obviously relate to improving health outcomes for specific vulnerable groups. And during several topics, public sector (and societal) perspectives may bring into consideration areas outside of healthcare where outcomes appear to be heavily influenced by socioeconomic and other factors.

Interventions may also increase inequality in the short term if higher social status groups are likely to benefit from first mover advantage. Having worked on developing the guidance on Walking and Cycling (PH41²⁶), there was a concern that such an issue may arise (at least temporarily), though it was hoped that in the longer term there would be increased uptake amongst all groups. A similar concern was expressed in published guidance on Physical activity and the environment (PH8³³). It is tempting to draw a comparison with the Kuznets curve³⁴, which implied that as a nation’s economy develops inequality will widen for a time before ultimately reducing once a certain overall level is attained; a similar phenomenon relating to health inequalities may exist in certain public health settings.

At times decisions in the CPH could even hinge on seemingly inconsequential differences in interpretation of the scope at present and such subtleties may prove difficult to represent in models. The scopes may themselves change, via gradual, subtle reinterpretation in light of the facts as they emerge. The facts used are not always self-evident and require judgement and an understanding of the causal chains of effect³⁵. With a new government, wholesale changes in scope are also possible. PHACs always needed to interpret some issues in an ad hoc manner as they arose – using their experience and judgement as part of this – though recommendations had to be formulated carefully.

An example framework from start to finish for developing the economic model, taken from PH41²⁶, is shown in Figure 1, displaying the variety of stages required from defining to whom the modelling stage should apply through to quantifying the cost-effectiveness in monetary units.

Figure 1. Example prospective modelling framework, as used in Public Health (PH)41²⁵.

Comparison to previous studies

Weatherly et al.⁴ (also described in an associated report by Drummond et al.⁵¹) reviewed 154 economic evaluations of public health interventions worldwide from 2000 to 2005 and found that 32% used a health service perspective and 31% a self-described societal perspective (though this was felt to be an overestimate) and 48% of which related solely to health. 24% had no stated perspective, 3% used multiple perspectives and the remainder related to local healthcare provider, government or patient. In contrast to the CPH guidance, the Weatherly paper found CCA was used in a relatively high 37% of studies, while 27% used CUA (whether based upon QALYs or the related Disability adjusted life years). A further 36% used CEA (excluding CUA), such as units of weight lost, alongside their cost information. CUA and CEA are recorded separately. 4 reports (3%) claimed to use CBA but these were upon further investigation re-categorised as CCA or CUA. Though it is worth reiterating that evaluations described in the Weatherly paper were not confined to UK settings (61% were from the US, 15% UK, 6% Canada and 4% others) – and are not directly comparable to NICE economic modelling – issues relating to costs to the voluntary sector and to private citizens were felt by the authors to merit further attention. Equity considerations, however, were rarely described in the literature and never addressed formally (and the authors argued that this implies that QALYs were simply summed directly for the studies using CUA). It is argued that these should be better highlighted, and opportunity costs of implementing more equitable interventions made transparent.

McDaid and Needles’ report⁵² featured 1700 studies from the mid-1960s to mid-2000s. 49% of studies were based in the US, 13% in the UK, 5% Canada, 4% Australia and 4% from the Netherlands. Intervention settings, rather than perspectives, are reported. 22% took place in workplace settings (overwhelmingly in the US). 8% took place in schools or colleges of higher education. Others (though no specific proportions are given) were funded by the state, social health insurance or by individuals; understandably, these tended to have less direct impact on productivity than the interventions that employers had chosen to provide. 57% of studies used CEA, 21% CCA, 13% used CUA, and the remainder used CBA (5%), econometric techniques (3%) or cost-minimisation approaches (1%). The authors emphasise the critical importance of context in understanding the influences of uptake and successful implementation of techniques, and argue that novel policy level approaches to funding are needed to ensure that non-health impacts of interventions are given adequate consideration. These will likely require further government investment given the diffuse nature of the benefits that accrue from interventions of this nature.

Though not strictly public health, a literature review describing the economic evaluation techniques used in social care was published in 2002⁵³. Mental health and public health were the two most common topics that used such techniques, totalling about two thirds of all reports. Many related to multiple issues at the same time, and this complexity presents further similarities with the attempts at applying such evaluations in public health. In total, 131 reports are reported, taking place over 5 years. The perspective used was not reported (though elsewhere in the report, the importance of a societal approach is emphasised). 65% of studies used CCA, 18% CEA, 5% CBA and 6% each for cost-minimisation analysis and cost-saving analysis. 72% of studies had taken place in the US, 15% UK and 13% other.

Nonetheless, a Cochrane review on public health evidence in practice⁵⁴ found that the findings of studies to inform local public health decision-making are rarely published (and that they rarely meet the standards required for subsequent systematic reviews anyway). This may mean that the reviews listed in this section do not reflect the full range of research undertaken applying economic evaluation techniques in public health.

Potential implications

Economics is ultimately about the studying of decisions, the incentives behind them and their consequences⁵⁵; health economics relate to these choices in the context of the resources available in health generally. Public health economics can be defined as the “study of the economic role of government in public health, particularly, but not exclusively, in supplying public goods and addressing externalities”⁵⁶. The broader effects of such interventions, taking into account a range of other factors, mean that such decision-making is much more complex than some other settings in health. This paper has already highlighted the range of factors considered in NICE’s public health economic modelling. Complicated decisions merit that models of some sort are used to structure the information available to better inform decisions makers, who may otherwise make unnecessary mistakes⁵⁷.

Ultimately, nearly all economic models used in the CPH translated the findings from relevant papers into gains in utility in order to facilitate a CUA approach for NICE guidance. However, such translation runs the risk of oversimplification⁵. While it is accepted that potential costs and benefits should be identified and highlighted in the limitations section or elsewhere, even where they cannot be measured - Sheill argued that those issues that are not measured, may ultimately be cast to one side if they cannot be incorporated into calculations⁵⁸. This could in principle be remedied somewhat if these issues were listed explicitly for later deliberations, but it is well established that human intuition is systematically subject to predictable biases for complex problems⁵⁹ of this nature. The use of a model can counter this, though in such a scenario all factors that are considered relevant would need to be measured, or else incorporated in an innovative way⁵⁸. There may therefore be a case for using these CCAs and CEAs directly in future in certain circumstances, or potentially extending these to multiple criteria decision analysis (MCDA) approaches. These techniques potentially hold a number of advantages for prioritisation in policy making settings⁶⁰, as well as well-documented possible stumbling blocks⁶¹.

The use of different methodologies may not lead to as many changes as one might think, as there is a large amount of overlap and ambiguity between approaches at times. CBA and net financial benefit appear to be for all intents and purposes equivalent approaches. Both apply monetary valuations to costs and benefits arising from implementing the approach in order to arrive at estimations of the net benefit arising from it. The main difference appears to be that net financial benefit was used as the term of choice when an employer perspective was used. As such, it has been maintained in the tables above as initially described, but it is worth bearing this in mind.

CBA also has links to CEA (and ultimately CUA). Guidance on Contraceptive services with a focus on young people up to the age of 25 (PH51⁶²) used a CBA approach which was built up upon the cost per negative outcome avoided (i.e. CEA) on a range of criteria, such as costs of maternity care, miscarriage, ectopic pregnancy and stillbirth, which had all previously been estimated in prior NICE research. The model estimated probabilities of each event occurring with and without the intervention. These could be combined formally (including the costs arising to future governments of benefit pay outs) to calculate the cost savings attributed to the intervention. The approach was found to be dominant. It would therefore similarly be dominant for CUA, regardless of any conceivable threshold used (assuming the same costs were taken into account in both cases). A similar approach was used for Domestic violence and abuse: how health services, social care and the organisations they work with can respond effectively (PH50²⁴), in which all conceivable costs and benefits were calculated on a societal level and the interventions modelled were found to be dominant. While it was described as a CUA, in such a scenario there is no clear line between CBA and CUA.

Such an approach nonetheless poses challenges. CUA has usually been used as part of an NHS and PSS perspective, and extending it to a public sector or societal perspective makes it unclear how to draw a threshold when non-health expenditure becomes involved. This can (rightly or wrongly) be ignored in the particular cases of PH50²⁴ and PH51⁶², where the interventions were found to be dominant. But if this was not found to be the case, it is not clear if the normally used threshold would be applicable. Would it be acceptable, for example, to approve an intervention with an ICER of £35,000, but where there are broader implications outside of healthcare? Such questions will need further revisiting in future if CBA does become more common in NICE’s modelling. These issues have been discussed elsewhere^4,63,64. While no consensus has as yet emerged, the unambiguous cost-effectiveness of most of NICE’s public health interventions^16,65 make the issue somewhat moot for now.

There is also the risk that in giving modellers free rein over which consequences of the intervention to include when using a societal perspective, they may (unconsciously or otherwise) cherry-pick effects which will help their case in approving the guidance. Even if similar risks are possible using a healthcare perspective, there will be clearer borders and experience in terms of what is and is not relevant. In a societal approach, aspects relating to criminal justice, education, or even tourism or the arts could feasibly be included or excluded at the whim of modellers, and if the PHAC members fail to query the position then these factors will influence all future decision-making. Even if the 2014 methods guidance allows other directorates to use such perspectives, if they are not widely adopted outside of public health it may make like-for-like comparisons of the cost-effectiveness of interventions more difficult between directorates.

On a more positive note, the ambiguity between CBA/CUA/CEA (and potentially CCA) does offer potential advantages. Where there is evidence that an approach is effective but no evidence of its magnitude, in the past this has generally been omitted from modelling entirely. CCA could be used to measure the direction of such issues and checked for dominance or what-if analyses to investigate whether the required effectiveness of interventions was plausible. PH51⁶² offers what appears to be an equivalent way of structuring the problem. It implicitly assumed that the intervention’s outcomes were better than the alternative (e.g. no ectopic pregnancy is better than an ectopic pregnancy), making intuitive sense, even though mortality estimates for these were zero and utility measurements for each were not available. Such an approach could be further extended where committees are willing to explain the logic of their assumptions, allowing formal quantitative approaches to be used in settings where they are not currently available.

The 2012 methods manual⁹ stated that CCA had previously been used ‘implicitly’ for trade-offs between cost-effectiveness and equity (and other concerns) at other stages of the guideline development process, but could now to be adopted more formally. Even within the economic modelling stage, a type of pseudo-CCA was used in “Alcohol-use disorders – preventing harmful drinking” (PH24⁴¹). The first time it was used more formally was in PH44⁴¹. In the past, equity considerations have been considered out of scope according to the reference case. While other areas of the reference case have been considered flexible on a case-by-case basis, the change in emphasis underway may permit equity trade-offs to be considered more explicitly as part of CCA modelling and that “the sets of consequences have been implicitly weighted should be recorded as openly, transparently and as accurately as possible” and “various tools are available to support this part of the process”⁹. Again, for a variety of reasons, MCDA techniques may be well placed to carry this out.

These equity concerns – which are in theory fundamentally important to public health decision-making – could also be given some formal weighting. Where equity factors are seen to influence the results of the guidance this will likely lead to some controversy – but it is worth bearing in mind that such considerations are already carried out without the help of a formal model and that opening such decisions up to some level of scrutiny seems entirely appropriate. If the equity concerns are seen to be too influential and costly in terms of societal health, or vice versa, then the weighting on equity for future appraisals could be adjusted. At present the decisions of PHACs cannot be held to the same level of accountability in such circumstances.

On the other hand, even if a workable definition of equity was specified⁶⁶, then assessing what weight is appropriate for equity is not immediately clear⁶⁷. It is plausible that this might not be necessary however; for example, MCDA could be used as part of a what-if style analysis. If, for example, an intervention would only be worth investing in if equity concerns made up 90% of the total decision, then this might provide a clear justification to decision-makers for refusing to invest. Over time, a de facto threshold level may emerge through precedent and prior experience, though this step may not be necessary to improve decision-making processes.

Limitations and challenges

Similar to other studies that are reviews of previously published studies, our review suffers some limitations. These include limitations of sample size, availability of data, type of data extraction and self-reporting. It was difficult to extract the data about economic modelling as there was no clear template for reporting PH studies and thus, all the documents had to be searched to extract the information about modelling. Whilst it is possible that some data could potentially be missed, we are confident in our review process extracted all relevant information. This review only extracted information at an overview level (i.e. type of economic evaluation methods) and involved qualitative synthesis. It would have been useful to extract detailed information (e.g. the results, whether the intervention was deemed cost-effective, other benefits considered beyond QALYs) and analyse the underlying relationships using quantitative techniques. The data extracted is based on the CPH authors’ interpretation of the economic modelling, however, given the established nature of this field we are confident that our interpretation of the CPH reports are as the authors intended. Regarding the number of studies included, we have a reasonable sample size as our review included all the topics in CPH from inception to the end of 2014. It should be noted that the new methods manual that came into effect in 2015 (including even how public health topics are coded) and thus our review only included studies prior to the new methods manual. Future research on the differences before and after the introduction of the new methods manual would be useful.

Underlying data

All data underlying the results are available as part of the article and no additional source data are required.

Extended data

Figshare: Pub health economic modelling NICE appendices.docx. https://doi.org/10.6084/m9.figshare.12156063.v1³⁹

- Pub health economic modelling NICE appendices.docx (Word document containing the further tables)

Data are available under the terms of the Creative Commons Attribution 4.0 International license (CC-BY 4.0).