Development of an objective assessment tool for appraising reporting quality of economic evaluations with the Consolidated Health Economic Evaluations Reporting Standards 2022: The CHEERS framework

Scope of the quality assessment tool

The suggested framework (Table 1³⁶) is intended to assess reporting quality of economic evaluations as defined by Drummond¹⁸. The quality is defined by grade of fulfilling criteria as proposed in the CHEERS 2022 checklist^1–16 and its complementary publication with further explanations and elaborations¹⁷. The framework incorporates direct citations from both the CHEERS 2022 checklist and the explanation and elaboration paper. The intention is for the framework to serve as an extension to both. We have adopted a checklist format and proposed a scoring method that shows the percentage of overall criteria fulfilment and allows visualisation via a traffic-light system.

General application guidance of the framework

Several general rules to apply the framework were agreed upon to ensure a feasible and comprehensive assessment of reporting quality of EEs. First, as described above, it should only be applied together with the original CHEERS 2022 publications^1–17. Second, decisions should be supported by highlighting or referring to the specific text passage in case a third reviewer needs to comprehend the decision in a consensus. Third, all reporting given to fulfill the criteria must be in the publication or appendix. Due to resource constraints during systematic reviews, it cannot be expected to additionally read all references the publication refers to. Fourth, reporting of information should happen in the same section the item is referring to, e.g. the information on the perspective should be given in the Methods chapter. To increase transparency, decisions on the application of the tool should be stated

Suggested scoring method and traffic light visualisation

The scoring method was developed by Hoang et al.³⁹ and a coding example was developed in Stata (Version 18, StataCorp, 2023). The scoring result is expressed via a traffic light system for all 28 main items of the CHEERS 2022^1–16 depending on the level of comprehensive reporting of all associated subitems. Accordingly, items can be considered completely fulfilled (green light), somewhat fulfilled (orange), completely lacking (red), or “not applicable” (black).

First, each subitem is graded based on the assessment made by the researchers in the framework. As shown above in the framework, there are several different assessment options including “Yes/No”, “Comprehensive/Minimal/No” as well as “Not applicable”. “Yes/No” was translated into a “1/0” grading and “Comprehensive/Minimal/No” into a “1/0,5/0” grading. Subitems deemed as “Not applicable” were labeled “99”. The total score of an item was calculated by summarising all scores of its subitems and dividing it by the number of subitems.

Scoring items which contained “Not applicable” as an assessment option was slightly different. The summary score of all subitems was calculated excluding the subitems items assessed “Not applicable” and in turn the summary score was also divided by the amount of subitems which were not assessed “Not applicable”.

When visualising the results, items resulting in a score of “1” were depicted in green, items resulting in a score below “1” and above “0” were depicted in orange, items resulting in a score of “0” were depicted in red. Items resulting in a score of “99” were depicted as black.

A framework summary score for each article was calculated by summing all subitem scores unless they were deemed “not applicable” and dividing the sum by the number of included subitems again excluding items assessed “not applicable”:

In order to calculate the level of reporting of one item in all included EEs of the systematic review the following equation was applied:

A script written in Stata and R can be found in Supplements 2 and 3³⁶. This is easy to apply if the quality assessment was conducted in Covidence. Otherwise, the Excel template (Supplement 4³⁶) is more suitable to use.

Summary of results

The iterative piloting process led to a framework with 67 subitems with the intention to guide researchers while applying the CHEERS 2022^1–16 as a checklist to assess reporting quality of EEs included in systematic literature reviews. Subitems continue to have a guidance which in its words is close to or exactly like the wording of the ISPOR explanation and elaboration publication regarding CHEERS 2022¹⁷. Thus, this framework does not provide a de novo checklist but a method to utilise the CHEERS 2022 statement^1–16. In addition to the classic binary assessment of fulfilling an item this framework introduced the variation of “minimal” and “comprehensive” reporting. Furthermore, the suggestion of a method to calculate the percentage of fulfillment of all quality requirements defined in the checklist gives researchers the opportunity to quantitatively compare the reporting quality of each item within an article and across all included articles. A traffic-light design was chosen as an adequate and easy-to-grasp visualization of the scoring. The application of the framework so far includes a review on pre-diagnosis interventions for people with dementia³⁹ and on geriatric assessment with a focus on frailty⁴⁰. Therefore, transferability to other topics than the original piloting review has been demonstrated.

Mending shortcomings of prior quality assessment tools

Reoccurring reported shortcomings in prior quality assessment tools for EEs have been lack of comprehensiveness, difficulty in use and inherent subjectivity in the assessment process^25,26.

The largest strength of this study is that it is an extension of the existing CHEERS 2022^1–16. CHEERS 2022 is a guidance on reporting developed by multiple experts in the field of health economics and health outcomes research, and specifically EEs. Thus, this framework is based on extensive and well reflected suggested standards for all types of economic EEs. By producing a complementary framework to the most recent published guidance, comprehensiveness was assured.

Several items of the CHEERS 2022 statement are not fulfilled by one singular fact, such as choice of discount rate, but leave room for the researchers to decide when the item has been reported complete enough. Subjectivity in the assessment of each item was minimised by first translating the guidance to “OR” or “AND” options, and in a second step splitting items which had more than one condition, marked by the Boolean operator “AND”. Providing clear decision rules leaves little to no room for interpretation of assessments and the fulfillment of multiple criteria per item can be distinguished and analysed. The format of a checklist with clear guidance is intended to make the process more applicable for any level of experience.

Addressing the shortcomings mentioned by Watt and Li²⁵, particularly lack of comprehensiveness, difficulty in use and inherent subjectivity in the assessment process, leaves this suggested framework to be less concise than prior checklists such as BMJs²¹, CHEC²² or Phillips²³. Nonetheless, a comprehensive, subjective and easy to apply grading of quality in systematic reviews is essential. Furthermore, the preference of the research community to apply CHEERS checklists has been demonstrated, as CHEERS 2013 was the most applied checklist for quality appraisal of EEs only a few years after its publication²⁵. This framework mends preexisting shortcomings of other checklists and satisfies the expectation of the research community.

Scoring quality

Few published quality appraisal checklists for EEs include a scoring method²⁶. The QHES has been identified as the only published checklist with a score that is weighted by importance of the assessed items for the cost-effectiveness measure⁴¹. Though, limitations on the assessment of methodological quality with QHES have been discussed²⁶. The CHEQUE also developed a scoring method weighted by item relevance which lets researchers estimate a relative score and rank studies of a systematic review²⁷. An application study of CHEQUE criticized the method due to too much flexibility in options as well as no clear guideline on how to interpret results and suggested an alternative scoring²⁸. 57,7% of systematic literature reviews of EEs conducted between 2000 and 2018 have calculated scores, partially without a scoring method present in the original publication of the applied quality assessment tools. Additionally, the presentation and structuring of scoring results varied, preventing a possible comparison across reviews. Arbitrary thresholds for high quality ranged between 63% and 94%²⁵. This framework presents a possibility to visualise item scoring across included articles via a traffic light system and calculates overall fulfillment of the CHEERS checklist on a subitem level. No weighting was applied as to our knowledge only singular studies are present on to which extent each subitem or item affects the cost-effectiveness measures⁴². Nevertheless, the framework contains rich data on reporting shortcomings for each item via its subitem assessments. An additional analysis of fulfillment of the subitems can be helpful to identify common gaps or trends in reporting.

Limitations

Some limitations remain to be mentioned. First, the application of the framework per article is time-intensive. It is, therefore, important to emphasize documenting citations or text passages when assessing the subitems to ensure a timely consensus decision. To further consolidate the framework while not diminishing its comprehensiveness a Delphi panel of experts could establish key reporting items. Second, the framework is only as good as its template. Despite CHEERS 2022 being the considered gold standard when conducting EEs, some questions remain unanswered. Item 13, for example, refers to the valuation of outcomes but focuses mainly on preference-based outcomes whereas this would not refer to cost-effectiveness or cost-benefit analyses. More specifications or variations in examples would further improve the guidance for researchers conducting or assessing EEs. Third, the scoring method does not allow researchers to grade the quality of the appraised studies leaving the assessment to the interpretation of the researchers, readers or decision-makers. Research is warranted to assess the influence of items or subitems of the cost-effectiveness measures. Fourth, indicators such as inter-rater reliability, retest-reliability or comparisons of results across multiple checklists have not been analysed. This would further strengthen its standing compared to preexisting checklists. Fifth, additional publications on use of AI tools as well as the analysis of value of information have been published during the development of this framework but have not been included.

Recommendations for future research

As mentioned, this proposed checklist has not been assessed regarding its own validity or reliability which should be completed in further research. This shortcoming has also been reported regarding prior checklists^25,26. Additionally, to enable the development of grading of quality of EEs studies are necessary which assess the impact of various items or subitems on the outcomes of the studies. Furthermore, a Delphi panel could identify key reporting items to make the framework more concise.