Validation of the rates of adverse event incidence in administrative healthcare data through patient chart review: A scoping review protocol

Introduction

Patient safety is a key issue for health systems and a growing global public health challenge. The human cost of patient safety incidents is high with the occurrence of adverse events causing injury, disability or death and resulting in suffering for patients and their family members. The financial and economic burdens of patient safety incidents are also of concern with further healthcare expenditure required for increased length of stays in hospital and additional tests, treatments and healthcare to be provided to patients (WHO, 2021b). The Harvard Medical Practice Study (Brennan et al., 1991) and reports published by the Institute of Medicine (2000) identified high rates of adverse events in hospital admissions and marked the beginning of the patient safety movement. This led to rates of adverse events becoming an important quality indicator across health systems. Adverse events have since been reported and measured however, challenges associated with the accuracy of their recording in administrative data has been widely acknowledged (Kim et al., 2022; Rodrigo-Rincon et al., 2015). Evidently, patient safety is a key healthcare quality concern and with adverse events as main contributors to patient harm, their monitoring and measurement is a major priority.

Routinely collected healthcare data, are defined as data that are regularly collected about a patient’s engagement with health services for purposes other than research (Welk & Kwong, 2017). These data, frequently called administrative data, capture the patient’s characteristics, their diagnoses, their treatment and any procedures that took place during their hospital stay, essentially providing a coded summary of the patient and their encounter with the healthcare system (Healthcare Pricing Office, 2020).

As these data are routinely collected following a patient’s discharge from hospital, they are also commonly referred to as routinely collected hospital discharge data. Throughout a patient’s stay in hospital, certain key events of their stay are documented and displayed in a discharge summary within their clinical chart. Once a patient has been discharged from hospital, their chart along with their discharge summary is translated into alphanumerical data by professional coding specialists (Tang et al., 2017). In addition to a patient’s discharge summary, the clinical coders may also code information in other documents such as nursing notes, consultation reports, progress notes, operative reports, pre- and post-operative reports and pathology reports (Healthcare Pricing Office, 2020). Nursing notes have previously been identified as important sources of data for clinical coders (Alonso et al., 2020; Doktorchik et al., 2020; Lucyk et al., 2017) and are particularly insightful in relation to identifying adverse events such as pressure injuries, which are a well-recognised quality indicator (Weller et al., 2022). The types of data captured by the alphanumerical codes include patient diagnoses, procedures, hospital services used and any complications that arose during a patient’s stay.

Various classification systems such as the International Classification of Diseases (ICD) are used to code this data. The ICD translates diagnoses of diseases and other health-related issues from words into alphanumerical data, allowing for an efficient method of storing, retrieving, and analysing data (WHO, 2016). Using coding systems such as ICD allows for a consistent and standard way of recording, reporting and monitoring diseases. ICD is the classification system that is used by all members states of WHO, which therefore allows for data to be shared, understood and compared across hospitals, regions and countries (WHO, 2023a). Various derived classifications or extensions of the fundamental ICD classifications have been developed and are used to monitor diagnoses in specific areas or settings (WHO, 2023b). Clinical coders follow clearly defined ICD coding guidelines to assign alphanumeric codes to each diagnosis or event recorded in a patient’s chart (O’ Malley et al., 2005). As the most widely used classification of diseases, ICD has had 10 iterations since 1900 (O’ Malley et al., 2005). The 10^th revision of the ICD (ICD-10), which contains more than 155,000 codes that can be assigned to medical diagnoses and additional information presented within patient charts (DiSantostefano, 2009), has been widely used since its development about 30 years ago (WHO, 2021a). An 11^th revision of the ICD has recently been developed. This version, ICD-11, was adopted by World Health Assembly in 2019 and came into effect in 2022 (WHO, 2023c). Various other coding systems such as the Current Procedural Terminology (CPT), which was developed by the American Medical Association, are used to capture and assign codes to aspects of healthcare data (Dotson, 2013) however, ICD is widely recognised globally as all WHO member states are committed to using this classification system.

Although it is not its intended primary purpose, routinely collected hospital discharge data, hereafter referred to as administrative data, are often used to inform research and decisions relating to health service planning. These datasets are used at national and international levels to inform policy and epidemiological and clinical research. They have also been utilised for carrying out audits, developing financial strategies and determining the distribution of health resources (Burns et al., 2012). Their use in the evaluation of the quality of health care delivery has also been well-recognised (Clarke et al., 2019), in addition to their use in informing biomedical research (Hemkens et al., 2016). Given that these datasets are often used for purposes other than those for which they were initially collected, their use in research and evaluation has been labelled as a secondary use (Jorm, 2015). Administrative data can contain potentially rich and valuable information and their potential use for research purposes has been increasingly recognised in recent years. The use of these databases is advantageous for research as they allow research questions that require large sample sizes to be answered (Harron et al., 2017). Their potential to inform research to enhance the efficiency of health systems and improve population health is well-documented and has led to their increased use in research (Jorm, 2015; Mitchell & Braithwaite, 2021; Moorthie et al., 2022). Patient safety researchers in particular have been increasingly recognising the potential of using administrative data for research purposes and for the monitoring and reporting of patient safety events (Raleigh et al., 2008).

Given that these data are used to inform research that may help to improve patients’ health and well-being and inform decisions that will impact on the quality of care that patients receive, it is vital that these administrative datasets are of high-quality and that precise and accurate diagnostic codes are used to generate them (Gologorsky et al., 2014; Ibrahim et al., 2021). Accurate measurement of adverse event incidence is central to improving patient safety and healthcare quality as it allows priority areas to be identified, improvement strategies to be designed and implemented interventions to be evaluated (Classen et al., 2011). Accurate data on adverse events may contribute to the development, implementation and evaluation of more targeted and evidence-based patient safety interventions that can prevent adverse events from occurring and improve the quality of care provided to patients (Dillner et al., 2023). Despite this, and although these datasets can provide rich and detailed information, they are subject to limitations. Previous studies indicate that administrative datasets are not always accurate and therefore, may contribute to inaccuracies in research and lead to poorly informed decisions (McGuckin et al., 2022; Nicholls et al., 2017). In particular, the inaccuracy of rates of adverse events in administrative data has been found in previous studies (Kim et al., 2022). Coding of primary diagnoses and procedures in administrative data is reported to be generally complete and accurate however, coding of secondary diagnoses is less complete and therefore, adverse events rates are often underestimated (Raleigh et al., 2008).

Various challenges associated with using administrative data for research purposes have been reported. Such challenges include not all data being abstracted from the patient chart, coded and represented in the database, incomplete data and missing measures, difficulty determining the number of out-patient visits due to imprecise coding and difficulty capturing all relevant tests due to various diagnostic codes being used for the same test (McGuckin et al., 2022). In relation to adverse events coded within administrative data, the inconsistency in the definitions and terminology used in practice have previously been identified by Naessens et al. (2009) as another challenge to the accurate reporting of these patient safety issues. Given that these administrative datasets are generated for purposes other than research, their accuracy and appropriateness for detecting adverse events is varied and their poor representation in some coding schemes has also previously been reported (Bates et al., 2003). The limited accuracy and variability of rates of hospital acquired infections in administrative data has also been identified as a challenge associated with using these datasets for research or benchmarking purposes. This has also resulted in calls for validation of these datasets and improvement of the algorithms used to generate them (van Mourik et al., 2015). Whilst the inaccuracy of administrative data has been widely reported, with a recent study conducted by Walther et al. (2021) on the incidence of postpartum haemorrhage indicating the insufficiency of such data to accurately record data on the prevalence of this adverse event, the potential of these datasets to provide accurate information is also being recognised. A study conducted by Ackroyd-Stolarz et al. (2014) found a relatively high degree of accuracy in such datasets for identifying adverse events in older patients. This offers evidence that supports and demonstrates the potential value of these datasets to inform research and improve patient safety.

Given the lack of consistency in the accuracy of these datasets for detecting and recording adverse events, there have been calls for validation studies to explore the reliability of these datasets and investigate further their potential to inform research and health policy (Ehrenstein et al., 2016; Jorm, 2015; Nicholls et al., 2017). Again, the potential of these datasets to inform research and policy in relation to patient safety and the human and economic cost of adverse events is evident, however specific calls for validating rates of adverse events within these datasets have been made (Raleigh et al., 2008; van Mourik et al., 2015). Many researchers have since carried out validation studies on various administrative datasets and therefore, it seems necessary and appropriate to identify and chart the evidence and results of these studies within this scoping review.

Validating administrative datasets is vital in determining the reliability and credibility of research based on this data and a common way of validating such data is through chart reviewing by manually comparing the coded data to the data recorded in a patient’s healthcare record (Nissen et al., 2019). Patient charts have been referred to as the ‘gold standard’ reference point in many validation studies (Ehrenstein et al., 2016), and therefore this scoping review aims to present an overview of how this method of validating administrative data has been used by previous researchers. There are various methodological approaches that can be used to conduct reviews of patient charts, for example, the Harvard Medical Practice Study (HMPS) and the Global Trigger Tool (GTT). The HMPS, conducted by Brennan et al. (1991), resulted in the development of a method for measuring incidence rates of adverse events through chart reviews and has since been used to conduct validation studies and verify rates of such patient safety incidents. This two-stage approach involves a nurse review of patient notes for triggers that suggest the occurrence of an adverse event, followed by a physician review of triggered charts for the recording of any adverse events using set definitions. The GTT was developed by the Institute for Healthcare Improvement in response to the demand for a less labour-intensive approach to reviewing patient charts for assessing patient safety and identifying adverse events. Stage one of the GTT process involves a screening phase for a more comprehensive list of triggers than the HMPS, however the duration of this stage is limited to 20 minutes. In the next stage, a physician reviews the specific trigger or potential incidence of an adverse event, rather than the entire chart, and uses broader definitions to identify the occurrence of an adverse event (Rafter et al., 2015). The methodological approaches to chart reviewing, such as those aforementioned, will also be explored and evidence of their use in validating rates of adverse events in administrative datasets will be presented in this review.

Methods

This scoping review will be conducted in accordance with the Joanna Briggs Institute (JBI) methodology for scoping review (Peters et al., 2020). The JBI methodological framework will be used to guide this scoping review and ensure that the review is carried out appropriately.

Conclusions

This protocol provides the structure for the conduction of a review to identify and chart the evidence of validation studies of rates of adverse events in administrative data. In line with the JBI guidelines (Peters et al., 2020), any deviations of the scoping review from the protocol will be acknowledged and discussed in the scoping review. It is widely acknowledged that administrative data have the potential to enhance patient safety and be used as a quality indicator however, given that the rates of adverse events in these datasets have been frequently reported as inaccurate, there is a strong rationale to conduct this scoping review. This review will aim to identify research gaps, chart the evidence of and highlight any flaws within administrative datasets in order to improve extraction and coding practices and enable researchers and policy makers to use these data to their full potential.