Validation of the rates of adverse event incidence in administrative healthcare data through patient chart review: A scoping review protocol

Introduction

Patient safety is a key issue for health systems and a growing global public health challenge. The human cost of patient safety incidents is high with adverse events causing injury, disability or death and resulting in suffering for patients and family members. The financial and economic burdens of patient safety incidents are also of concern with further healthcare expenditure required for increased length of stays in hospital and additional tests, treatments and healthcare to be provided to patients (WHO, 2021b). The Harvard Medical Practice Study (Brennan et al., 1991) and the Institute of Medicine (2000) identified high rates of adverse events in hospital admissions and marked the beginning of the patient safety movement. This led to rates of adverse events becoming an important quality indicator across health systems. Adverse events have since been reported and measured however, challenges associated with the accuracy of their recording in administrative data has been widely acknowledged (Kim et al., 2022; Rodrigo-Rincon et al., 2015). Evidently, patient safety is a key healthcare quality concern and with adverse events as main contributors to patient harm, their monitoring and measurement is a major priority.

Routinely collected healthcare data are regularly collected for purposes other than research (Welk & Kwong, 2017). These data, frequently called administrative data, capture the patient’s characteristics, their diagnoses, treatments and procedures and, essentially provide a coded summary of the patient and their encounter with the healthcare system (Healthcare Pricing Office, 2020).

The key events of a patient’s stay in hospital are documented in a discharge summary within their clinical chart or in their electronic health record (EHR). Once a patient has been discharged from hospital, their chart is translated into alphanumerical data by professional coding specialists (Tang et al., 2017). In addition to a patient’s discharge summary, clinical coders may also code information in documents such as nursing notes, consultation reports, progress notes, operative reports, pre- and post-operative reports and pathology reports (Healthcare Pricing Office, 2020). Nursing notes have previously been identified as important sources of data for clinical coders (Alonso et al., 2020; Doktorchik et al., 2020; Lucyk et al., 2017) and are particularly insightful in relation to identifying adverse events such as pressure injuries, which are a well-recognised quality indicator (Weller et al., 2022). The data captured by the alphanumerical codes include patient diagnoses, procedures, hospital services used and any complications that arose during a patient’s stay.

Various classification systems such as the International Classification of Diseases (ICD) are used to code this data. The ICD translates diagnoses and other health-related issues from words into alphanumerical data, allowing for an efficient method of storing, retrieving, and analysing data (WHO, 2016). Coding systems such as ICD allow for a consistent and standard way of recording, reporting and monitoring diseases. ICD is the is used by all members states of WHO, therefore data can be shared, understood and compared across hospitals, regions and countries (WHO, 2023a). Various derived classifications or extensions of the fundamental ICD classifications have been developed and are used to monitor diagnoses in specific areas or settings (WHO, 2023b). Clinical coders follow clearly defined ICD coding guidelines to assign alphanumeric codes to each diagnosis or event recorded in a patient’s chart (O’Malley et al., 2005). As the most widely used classification of diseases, ICD has had 10 iterations since 1900 (O’Malley et al., 2005). The 10^th revision of the ICD (ICD-10), which contains more than 155,000 codes (DiSantostefano, 2009), has been widely used since its development about 30 years ago (WHO, 2021a). An 11^th revision of the ICD (ICD-11 was adopted by World Health Assembly in 2019 and came into effect in 2022 (WHO, 2023c). Various other coding systems such as the Current Procedural Terminology (CPT), which was developed by the American Medical Association, are used to capture and assign codes to aspects of healthcare data (Dotson, 2013) however, ICD is recognised globally with all WHO member states committed to using this classification system.

Administrative data are often used at national and international levels in health service planning and to inform policy and epidemiological, clinical and biomedical research (Hemkens et al., 2016). They have also been utilised for carrying out audits, developing financial strategies and determining the distribution of health resources (Burns et al., 2012). Their use in the evaluation of the quality of health care delivery has also been well-recognised (Clarke et al., 2019). Given that these datasets are often used for purposes other than those for which they were initially collected, their use in research and evaluation has been labelled as a secondary use (Jorm, 2015). Administrative data can contain rich and valuable information and their use in research has been increasingly recognised. These databases are advantageous for research as they allow research questions that require large sample sizes to be answered (Harron et al., 2017). Their potential to inform research to enhance the efficiency of health systems and improve population health is well-documented and has led to their increased use in research (Jorm, 2015; Mitchell & Braithwaite, 2021; Moorthie et al., 2022). Patient safety researchers in particular have been increasingly recognising the potential of using administrative data for research purposes and for the monitoring and reporting of patient safety events (Raleigh et al., 2008). Given the association between lower registered nurse staffing levels, increased mortality rates and other adverse outcomes (Griffiths et al., 2016), validating administrative datasets and ensuring their accuracy may also be beneficial in encouraging the potential for these datasets to be linked to nurse staffing data to demonstrate the impact of nurse staffing on patient outcomes and highlight areas for improvement.

Given that these data are used to inform research that may help to improve patients’ health and well-being and inform decisions that will impact on the quality of care that patients receive, it is vital that these administrative datasets are of high-quality and that precise and accurate diagnostic codes are used to generate them (Gologorsky et al., 2014; Ibrahim et al., 2021). Accurate measurement of adverse events is central to improving patient safety and healthcare quality by allowing priority areas to be identified, improvement strategies to be designed and implemented interventions to be evaluated (Classen et al., 2011). Accurate data on adverse events may contribute to the development, implementation and evaluation of more targeted and evidence-based patient safety interventions that can prevent adverse events and improve quality of care (Dillner et al., 2023). Although these datasets can provide rich and detailed information, they are subject to limitations. Previous studies indicate that administrative datasets are not always accurate and therefore, may contribute to inaccuracies in research and lead to poorly informed decisions (McGuckin et al., 2022; Nicholls et al., 2017). In particular, the inaccuracy of rates of adverse events in administrative data has been previously identified (Kim et al., 2022). Coding of primary diagnoses and procedures in administrative data is generally accurate however, coding of secondary diagnoses is less complete and therefore, adverse events rates are often underestimated (Raleigh et al., 2008).

Various challenges associated with using administrative data in research have been reported. Such challenges include not all data being abstracted from the patient chart, coded and represented in the database, incomplete data, difficulty determining the number of out-patient visits due to imprecise coding and difficulty capturing all relevant tests due to various diagnostic codes being used for the same test (McGuckin et al., 2022). The inconsistency in the definitions and terminology used in practice have previously been identified by Naessens et al. (2009) as another challenge to the accurate reporting of patient safety issues. Given that these administrative datasets are generated for purposes other than research, their accuracy and appropriateness for detecting adverse events is varied and their poor representation in some coding schemes has also previously been reported (Bates et al., 2003). The limited accuracy and variability of rates of hospital acquired infections in administrative data has also been identified as a challenge associated with using these datasets for research or benchmarking purposes. This has also resulted in calls for validation of these datasets and improvement of the algorithms used to generate them (van Mourik et al., 2015). Although the inaccuracy of administrative data for recording rates of adverse events has been previously reported by researchers such as Walther et al. (2021), who identified poor recording of postpartum haemorrhage within administrative data, the potential of these datasets to provide accurate information is also being recognised. Ackroyd-Stolarz et al. (2014) found a relatively high degree of accuracy in such datasets for identifying adverse events in older patients, therefore demonstrating the potential value of these datasets to inform research and improve patient safety.

Given the lack of consistency in the accuracy of these datasets for recording adverse events, there have been calls for validation studies to explore their reliability and investigate further their potential to inform research and health policy (Ehrenstein et al., 2016; Jorm, 2015; Nicholls et al., 2017). Again, the potential of these datasets to inform research and policy in relation to patient safety and the human and economic cost of adverse events is evident, however specific calls for validating rates of adverse events within these datasets have been made (Raleigh et al., 2008; van Mourik et al., 2015). Many researchers have since carried out validation studies on various administrative datasets therefore, it seems necessary to chart the evidence and results of these studies within this scoping review.

Validating administrative datasets is vital in determining the reliability and credibility of research based on this data and a common way of validating such data by manually comparing the coded data to the data recorded in a patient’s healthcare record (Nissen et al., 2019). Patient charts have been referred to as the ‘gold standard’ reference point in many validation studies (Ehrenstein et al., 2016), therefore this scoping review aims to present an overview of how this method of validating administrative data has been used by previous researchers. Various methodological approaches such as the Harvard Medical Practice Study (HMPS) and the Global Trigger Tool (GTT) can be used to conduct chart reviews. The HMPS method is a two-stage approach involving a nurse review for the triggering of charts, followed by a physician review of triggered charts to confirm rates of patient safety incidents using set definitions (Brennan et al., 1991) The GTT was developed by the Institute for Healthcare Improvement in response to demands for a less labour-intensive approach to chart reviewing. This two-stage approach involves a 20-minute screening phase for a comprehensive list of triggers, followed a physician review of the triggers using broader definitions to confirm the occurrence of an adverse event (Rafter et al., 2015). The HMPS method is a two-stage approach involving a nurse review for the triggering of charts, followed by a physician review of triggered charts to confirm rates of patient safety incidents using set definitions (Brennan et al., 1991) The GTT was developed by the Institute for Healthcare Improvement in response to demands for a less labour-intensive approach to chart reviewing. This two-stage approach involves a 20-minute screening phase for a comprehensive list of triggers, followed a physician review of the triggers using broader definitions to confirm the occurrence of an adverse event (Rafter et al., 2015).

Methods

This scoping review will be conducted in accordance with the Joanna Briggs Institute (JBI) methodology for scoping review (Peters et al., 2020). The JBI methodological framework will be used to guide this scoping review and ensure that the review is carried out appropriately.

Conclusions

This protocol provides the structure for the conduction of a review to identify and chart the evidence of validation studies of rates of adverse events in administrative data. In line with the JBI guidelines (Peters et al., 2020), any deviations of the scoping review from the protocol will be acknowledged and discussed in the scoping review. It is widely acknowledged that administrative data have the potential to enhance patient safety and be used as a quality indicator however, given that the rates of adverse events in these datasets have been frequently reported as inaccurate, there is a strong rationale to conduct this scoping review. This review will aim to present an overview of the chart review approaches and tools that have been used to validate rates of AEs in administrative datasets and ultimately highlighting the need to develop a systematic approach to measuring rates of AEs in administrative data. This will allow the accuracy of these datasets to be improved, which will enable researchers and policy makers to use them to their full potential to improve patient safety and healthcare quality.