Development of performance indicators for systems of urgent and emergency care in the Republic of Ireland. Update of a systematic review and consensus development exercise

Systematic review

This study comprised of an update of a previously performed systematic review which had covered the period up to 2007⁵ and a formal consensus development exercise. The systematic review update was conducted in August 2014 by one person (RD). Articles cited in PubMed over the period 2008 to 2014 were systematically searched by combining variations of the text terms ‘emergency’ and ‘indicator’ using the AND operator. Our search for novel indicators was supplemented by a review of the reference lists of articles selected for review. We also contacted experts and organisations working on the assessment of urgent and emergency care performance to identify relevant grey literature. These included the Society for Academic Emergency Medicine (USA), the Centre for Medicare and Medical Services (USA), the Emergency Department Benchmarking Alliance (USA), the Canadian Association of Emergency Physicians, the European Society for Emergency Medicine, the Royal College of Surgeons England, the Pre-Hospital Emergency Care Council (Ireland) and relevant HSE Clinical Programme Directorates.

Articles were selected for review by two persons (RD and JB) on the basis that they might contain definitions of system-level indicators of emergency and urgent care performance. Articles were excluded after review of the full text version if the indicators that they contained were already listed by the previous systematic review or if they focused on individual components of the urgent and emergency care system such as emergency department waiting times or ambulance response times. Non-English articles were also excluded. Included articles were read and discussed by RD and JB who came to a consensus about whether they contained at least one indicator that measured system-level performance in urgent and emergency care. A conservative approach was taken and indicators were excluded only if they clearly related to a specific service and did not capture elements of system function. This approach was warranted on the basis that the subsequent consensus development exercise was to be the definitive judgement on the extent to which indicators were useful signals of system performance.

The systematic review has been reported according to PRISMA (Supplementary File 1).

Consensus development exercise

The consensus development exercise comprised an online survey and a face-to-face nominal group meeting. A broad range of Irish-based experts were recruited to the consensus development group. Experts were recruited by contacting professional representative bodies, policy making organisations, regulatory bodies and patient advocacy groups. The following clinical disciplines were recruited to the group: emergency nursing, acute medicine, minor injuries/urgent care nursing, anaesthesia/intensive care, emergency medicine, psychiatry, public health, paediatrics, pre-hospital care, general practice, pharmacy and geriatric medicine. The HSE quality improvement directorate, the Irish Department of Health, the Irish healthcare regulator (Health Information and Quality Authority) and two patient advocacy groups were also represented. Once individuals were highlighted as potential members, they were approached through email and phone calls to join the group. Interested parties were then sent a formal invitation letter to join the group. In total the group was composed of 17 national experts on urgent and emergency care in Ireland.

All novel indicators identified in the updated systematic review were combined with those identified in the original systematic review and grouped under the following headings in an online survey: outcome based indicators, process based indicators and structural indicators (see Supplementary File 2). The definitions of urgent and emergency conditions were adopted from those used in previous consensus development work⁴ performed by the University of Sheffield for the English NHS (see Supplementary File 3). The survey was designed and distributed to the consensus development group using the online tool, Survey Monkey. All members of the group were sent a link to the online survey and asked to complete it. Each member was asked to rate their agreement with the statement ‘this measure is likely to be a good indicator of the performance of the emergency and urgent care system’, on a Likert scale anchored by 1 (‘disagree strongly’) and 9 (‘agree strongly’). There was also space for members to add any comments. Participants were asked not to limit their views about the potential usefulness of an indicator by perceived difficulties in collecting or processing the data required to calculate them. 14 consensus group members completed the survey.

A face to face meeting was held in October 2014 and all members of the consensus group who had completed the online survey were asked to attend. Thirteen of the 14 invited members attended the meeting. One member had planned to attend but was not able to do so because of a separate work commitment which arose close to the meeting. Each participant was provided with the original questionnaire which now included a record of their individual responses to the online survey and the group’s median score and interquartile ranges. The meeting was conducted using a nominal group technique format. Once each participant had been given the opportunity to provide their opinion about an indicator, that indicator was ranked again by the members of the group. This procedure was followed for each individual indicator until all indicators had been discussed. Following the meeting, the performance indicators were ranked by their median agreement score. Those with a median greater than 7 were classified as potentially good performance indicators. A second online survey was then created using the online tool, Survey Monkey. Those indicators which had scored a median greater than 7 were included and all participants were asked to rank these indicators in order of preference. Higher ranks indicate greater preference and are represented by lower numbers. This exercise was sent to the 13 members of the group who had attended the consensus development meeting and there was a 100% completion rate.

Study limitations

Our study has some limitations. The systematic review only covers the period up to 2014 and may, therefore, miss some more recent literature. Since 2014 at least two further consensus development studies have been published in this field. These studies were not focused on the whole system of urgent and emergency care: one covered only prehospital care⁸ and one focused only on care in the ED⁹. Nevertheless, it is notable that the study of prehospital performance found that “direct transport of ST-elevation myocardial infarction patients to a primary percutaneous intervention (PCI)-capable facility for ECG to PCI time <90 min” was the highest ranked indicator. This is consistent with our finding that ‘time from call to care’ for indicator conditions is rated as the best performance indicator for a whole system of urgent and emergency care. This provides some support to the validity of our consensus development exercise.

Data extraction in the systematic review was conducted independently by two reviewers who then came to consensus. However, no formal assessment of inter-rater reliability was conducted.

No attempt was made to achieve unanimity so it is possible that some of the indicators may be controversial to certain stakeholder groups. We also requested that panel members did not consider the feasibility of collecting data required to calculate an indicator. This may mean that the chosen performance indicators are not immediately measurable; however, we are hopeful that progress in data collection may allow these performance indicators to be measured in the future.

Implications

The results of this paper have been used to conduct three studies on urgent and emergency care in Ireland, each using one of the highly ranked indicators described above. The first study measured “case fatality rates for serious, emergency conditions for which a well-performing EUCS could improve chances of survival” to assess whether national and regional population outcomes in Ireland over the period 2002–2014 were associated with the reconfiguration or regional urgent and emergency care systems. No distinct pattern of change was found among regions which underwent substantial reconfiguration compared to those that did not¹⁰. The second study measured the impact of population and health system factors on county-level variation in “conditions whose exacerbations could be managed out of hospital or in ED’s without admission to an inpatient bed” in Ireland over the period 2014–2016¹¹. This found that potentially avoidable emergency admissions are primarily driven by socioeconomic conditions, hospital admission policies and private health insurance coverage and were not associated with primary care resources. This is an important finding for policy in this area as it suggests that reductions in emergency admissions for ambulatory conditions will not necessarily be achieved by simply increasing the quantity of relevant primary care resources. The study also found that the distinction between ‘potentially avoidable’ and all other emergency admissions may not be as useful as previously believed when attempting to identify the causes of regional variation in emergency admission rates. The third study used “patient reported experience of whole episodes of emergency and urgent care” to compare user experiences of eight regional urgent and emergency care systems in Ireland¹². It found no consistent relationship between patient experience and the type of urgent and emergency care model in different regions, and concluded that composite questionnaire data may not offer a useful metric for exploring the impact of system-level service change.

Our research programme, including the present study, have produced some consistent learning points which may be useful for researchers in this field. First, it is not easy to distinguish between indicators of the performance of individual services and the system as a whole because of the inter-dependency of different services. For example, the indicator “call to ambulance time on scene” at first glance seems to cover only one service, but during the discussions of the consensus group it was felt to reflect a number of system-level issues such as the geographical configuration of ambulance stations and Emergency Department locations, and capacity pressures on the ambulance service that are caused by system-level decisions (e.g. need to spend more ambulance resources on inter-hospital transfers).

Second, it is difficult to operationalise research where populations, interventions and outcomes are difficult to define. For example, in all three of the studies described above the populations studied were compared at geographical levels (e.g. county, region) but these are inevitably an imperfect level of analysis because geographical units do not map perfectly to health system interventions, especially at boundaries where populations are exposed to both old and new models of care. This problem did not fatally undermine the internal validity of our case-fatality study¹⁰ because all geographical units were subject to some ‘boundary leakage’ of populations, and the proportion of the overall study populations affected was low. In general, however, researchers should be sensitive to instances where this leakage is likely to pose a substantial challenge, for example when boundaries cut across urban areas and a high proportion of the population has easy access to more than one model of care.

A third, related issue, is the overall complexity of system evaluation. It is debatable whether traditional study designs are a useful way to compare whole systems of care because of the difficulty in making strong inferences about causal relationships within complex systems. Although there are growing calls for evaluations of complex organisational interventions at the level of whole health systems, progress has been limited because of the difficulty in dealing with complexity¹³. In two of our studies^10,12 we found no evidence of difference in outcomes between populations treated by different models of care. It is likely that true differences do exist for patient sub-groups on specific care pathways in particular contexts but the signal of these differences is hidden by system-level analyses for whole populations. Our studies provide useful information about aggregate outcomes and trends over time but more focused studies are required to provide evidence to policy makers about the impact of system changes such as hospital reconfiguration. These studies should focus on processes, such as compliance with new hospital referral guidelines, as well as outcomes¹³. It is also possible that the impact of interventions in complex systems may not be apparent in the timeframes we used, and that longer-term studies are required¹³.

Fourth, each indicator is targeted at a heterogeneous group of patients and is a composite of different outcomes. This can hide valuable information about patient sub-groups and specific outcomes. The case-fatality study, for example, combines outcomes for 16 different conditions¹⁰. The patient experience study combined multiple questions into composite scale scores¹². Both of these found no signal of outcome differences between models of urgent and emergency care which may indicate that a finer grain of analysis is required when using performance indicators for urgent and emergency care systems.

Fifth, routinely collected secondary data can rarely be used to calculate the indicators presented in this paper and original data collection is often required. Routine data could only be used for two of the 17 indicators (case-fatality ratios for serious emergency conditions and admission rates for ambulatory conditions). The study of patient experience was original because no such data was available at the time it was conducted. Ireland has since introduced a national patient experience survey which is conducted annually¹⁴. However, this still focuses on individual emergency departments and would not, therefore, be suitable for an analysis of the impact of whole system changes on patient experience. Time-based indicators are almost completely absent from the health information systems in public hospitals in Ireland.

Finally, one of our studies found that the indicator ‘emergency admission rates for conditions whose exacerbations could be managed out of hospital or in ED’s without admission to an inpatient bed’ was no more sensitive to variations in care settings than total emergency admission rates. This underlines the importance of a critical approach to performance indicators, which may not be fit for purpose despite seeming to have obvious face validity.