Generating actionable insights from free-text care experience survey data using qualitative and computational text analysis: A study protocol

The National Care Experience Programme

The Committee on the Future of Healthcare’s Sláintecare report, which outlines a cross-party vision for Ireland’s health and social care services and policy, highlights that patients’ needs must ‘come first in driving safety, quality and the coordination of care’¹⁴. The Irish National Standards for Safer Better Healthcare aim to ‘create a basis for improving the quality and safety of healthcare services by identifying strengths and highlighting areas for improvement’¹⁵. The National Care Experience Programme (NCEP) seeks to improve the quality of health and social care services in Ireland by asking people about their experiences of care and acting on their feedback. The NCEP collects a large volume of qualitative data, received in response to open-ended care experience survey questions. These questions allow service-users to describe the care they received in their own words, including aspects that are not captured by other survey questions. The comments provide a rich source of information on patients’ experiences with healthcare. Previous analyses of these data, however, have tended to focus on specific, limited areas or topics, such as women’s experiences of initiating infant feeding or older people’s experiences in the emergency department^16,17. The volume of comments makes it difficult to extract all key themes and areas of experience that could be used to inform quality improvement plans and changes in practice and policy on a larger scale.

To date, 93,135 comments have been received from participants in the National Inpatient Experience Survey and National Maternity Experience Survey. As the NCEP expands to include additional surveys, the amount of qualitative data collected will increase substantially. This study will provide important, in-depth insights into areas of good experience and areas requiring improvement across acute hospital and maternity services, based on secondary analysis of detailed first-hand accounts from patients and women using these services.

Aims and objectives

This study aims to employ a systematic, computational approach to the categorisation and analysis of large volumes of qualitative data from Irish national care experience surveys in order to extract key themes and areas of experience, which will in turn inform national efforts to improve health and social care delivery, policy and practice. A dashboard will be developed to make the findings available and useable to those with the power and responsibility to act on them. A second aim of this study is to inform the development of a tool that will facilitate more efficient and standardised processing and analysis of care experience data received by healthcare services and other organisations.

The objectives of the study are to:

Study design and data sources

This study will involve secondary analysis of data from the National Inpatient Experience Survey and National Maternity Experience Survey. The study commenced on 1 April 2022, and will run for 24 months. National Inpatient Experience Survey and National Maternity Experience Survey data were requested for this research purpose from the National Care Experience Programme by completing a data access request form.

National Inpatient Experience Survey. The National Inpatient Experience Survey (NIES) is a national, repeat cross-sectional survey of inpatient experiences in all 40 public acute hospitals in Ireland (Table 1). The NIES was conducted annually from 2017 to 2019, cancelled in 2020 as a result of the COVID-19 pandemic, and resumed in 2021. All patients aged 16 years and above, who spend at least 24 hours in one of the participating hospitals and who are discharged during the survey month are eligible to participate, with response rates ranging from 42–51%. The survey asks patients about their experiences in hospital from admission to discharge. Overall experience of care is measured on a scale from 0 (very poor) to 10 (very good). Three open-ended questions are included, which ask patients if there was anything particularly good about their hospital care, if there was anything that could be improved, and if they have any other comments or suggestions. In 2021, the last question was amended to ask if patients had any other comments about how the COVID-19 pandemic affected their care. Since 2017, 50,196 patients have participated in the survey, with a total of 86,239 comments received in response to the open-ended questions.

National Maternity Experience Survey. The National Maternity Experience Survey is a national survey of maternity care experiences, which was conducted for the first time in 2020 (Table 1). The survey asks women about their maternity care experience from antenatal care to care during labour and birth and care at home after the birth. Overall experience of care is measured on a scale from 0 (very poor) to 10 (very good). Three open-ended questions ask women if there was anything particularly good about their maternity care, if there was anything that could be improved, and if there were any other important parts of their maternity care experience that were not covered by the questions in the survey. Women aged 16 years and above, who gave birth in one of Ireland’s 19 maternity hospitals or units, or who had a home birth in October or November 2019, were eligible to participate. In 2020, 3,205 women participated, representing a response rate of 50%. A total of 6,896 comments were received in response to the three open-ended questions.

Data analysis

Development of an analytical framework. In-depth analysis of qualitative data requires an overarching theoretical or conceptual foundation. This is often operationalised as an analytical framework, comprised of a set of codes or categories that are used to organise and manage unstructured, free-text data¹⁸. To explore issues related to both positive and negative experiences of patients in acute hospital and maternity care contexts, the analysis will be grounded in the Service (Dominant) Logic theoretical framework¹⁹. In this framework, the notion of patient or maternity care as a service involves a process that facilitates the realisation of value (e.g. safe delivery) from the patient’s or mother’s perspective²⁰. Patient experiences are conceptualised as the outcome of the interactions of the patient with systems, administrative and clinical processes, and with the different types of staff providing care²⁰. The use of Ordenes et al.’s Activities, Resources, and Context (ARC) framework enables a focus on information about the different processes or activities and resources (human, material and non-material) characterising the patient interaction, as well as the context of that interaction (i.e. the personal situation of the patient or woman and the circumstances of their care)²⁰.

For this study, activities refer to specific patient touchpoints; resources comprise the care staff, administrative and medical processes, equipment and facilities utilised when providing care to patients and women; while contexts cover the different types of situational and personal circumstances of patients and women when receiving care. The NCEP stages of care (Table 1) will be used as top-level categories for activities, while the Systematized Nomenclature of Medicine (SNOMED) healthcare terminology will be adopted in defining the taxonomy for resources. The taxonomy for contexts will be developed inductively from the qualitative data. The resulting Activity-Resource-Context Framework is shown in Figure 1.

Figure 1. The Activity-Resource-Context Framework.

Exploratory data analysis. The initial exploratory phase of this study will entail automated analysis of the qualitative data using computational text analytics and predictive machine learning techniques, including structural topic models and predictive modelling. The exploratory analyses will help to refine the analytical framework by uncovering additional important concepts related to activities, resources and contexts contained within the comments but not captured in the analytical framework, and will inform the more detailed subsequent analysis of annotated data.

Structural topic models

Structural topic models of the qualitative data will be developed to uncover important themes and relationships between these themes and demographic (age, sex, ethnicity) and organisational (e.g. hospital size) factors. Topic modelling is a statistical approach used to extract the hidden semantics that occur in a collection of documents or other textual data. The Latent Dirichlet Allocation (LDA) technique is one of the most notable approaches for unsupervised topic modelling. LDA defines themes in the textual data as distributions over a vocabulary of words that represent semantically interpretable topics. The meaning of those topics is based on the emergent quality of the relationship between words²¹. Structural topic modelling (STM) is an extension of the LDA framework that has previously been adopted in qualitative patient experience studies^22–24. STM extends LDA, which only analyses information “about the content of the text”, by also analysing information about the unstructured data itself²⁵. Consequently, STM facilitates the exploration of relationships between themes in the qualitative data and demographic and organisational factors in a systematic manner and on a larger scale than would be possible using traditional qualitative analysis techniques.

Predictive modelling with Random Forest

Predictive modelling, using an ensemble of machine learning algorithms such as Random Forest and Support Vector Machines, will then be used to determine which of the extracted themes or topics most strongly influence overall care experience ratings. The Random Forest algorithm combines multiple decision trees and relies on each random sample to build each decision tree, reducing the risk of overfitting. As a classifier, Random Forest performs an implicit feature selection, using a subset of "strongest variables" for the classification only to reach the best performance on high dimensional data²⁶. The outcome of this implicit feature selection of the Random Forest algorithm can be determined and visualised as a concept of variable importance (Gini Importance)²⁷. The concept of variable importance here is synonymous with topic importance regarding the overall patient experience rating. The predictive model will determine the overall care experience rating (ranging from poor to very good) that is most probable for a specific distribution of topics across a comment. The model will also identify the relative importance of the latent topics or issues characterising the data.

Pilot and full dataset annotation

Annotation of the datasets using the analytical framework will be completed in two stages. The pilot phase will test and refine the analytical framework to reduce ambiguity or inconsistency in the developed ARC taxonomies, using a random sample of comments (~10%) from the two datasets. Two experienced qualitative researchers will create high-quality annotations, which will serve as training data for automatic annotation of the full datasets by a machine or deep learning model in the next stage. Given the anticipated use of the annotated dataset, we will aim for strong interrater reliability (>0.8). Computer-based qualitative research software (MaxQda, as well as an in-house developed application) will be used for the pilot annotation task.

The full annotation of the dataset will involve the development of a machine or deep learning model to automatically identify and label ARC entities in the comments, due to the significant amount of time and resources required to reliably manually annotate over 90,000 free-text feedback comments. A variety of features from the manually coded datasets from the pilot phase will be extracted, including Part-of-Speech (PoS) tags, trigger terms identified by annotators for ARC terms in the text, and multidimensional vector representations of the ARC terms using combinations of embeddings. We frame the process of automatic ARC annotation of comments as Named Entity Recognition (NER) and (ARC) Relation Extraction problems^28,29. A Long Short Term Memory (LSTM) based deep-learning model will be adopted for ARC NER and relation extraction. Specifically, we will explore the use of bidirectional LSTM, LSTM-Conditional Random Field and the n-Gram Convolutional Neural Nets (GRAM-CNN) models^30,31.

ARC pattern mining. Following the complete annotation of qualitative data, important or frequently occurring ARC patterns or events associated with both positive and negative experiences will be identified. An Association Rule and Sequence Mining, such as the Apriori algorithm, will be used to extract ARC patterns in the annotated datasets. The Apriori algorithm is a popular data mining algorithm for mining frequent item sets and associated association rules from a dataset. The extracted patterns will encapsulate information on the contextual factors, specific service touchpoints and associated resources, providing important input for how to prioritise improvements in care experiences.

Emotion analysis and suggestion extraction. In order to identify and prioritise the key areas requiring improvements from patients’ and women’s perspectives, the affect and emotion expressed by patients and women who have given birth will be quantified. To estimate the emotion, we propose to use a linguistic analytics-based state anxiety detection algorithm in addition to traditional sentiment analysis. Our anxiety detection algorithm was trained on a dataset of 600 randomly selected tweets from 10,386 users³², and scored by 604 zero-acquaintance human raters based on a six-item short-form of the Spielberger State-Trait Anxiety Inventory (STAI)³³. On average, each tweet was rated five times on a scale from 1 (“not at all”; low anxiety) to 4 (“very much”; high anxiety) by each rater. The training procedure involved the use of a 6-fold cross-validation resampling plan and was validated using a set of 3.33 million tweets. Using this algorithm has allowed us to account for the interaction between personality and anxiety in correspondence to the COVID-19 crisis³⁴, examine the influence of leaders over followers in organisations over time, and study public sentiment in response to organisational crisis communications on social media^35,36.

To analyse the suggestions provided by patients as part of their qualitative feedback, we intend to use a deep learning model to automatically identify and classify comments where a suggestion is made. The automated approach is necessary given the volume of qualitative data, and will involve multi-label text classification, where each comment could be assigned one or more labels (topics of suggestion). We propose an end-to-end neural network architecture for multi-label classification of suggestion comments, and will use an existing training dataset for our machine learning task. Linking the extracted suggestions to the identified priority areas for improvement will facilitate the development of initiatives to improve inpatient and maternity care experiences. Figure 2 shows a summary of the study design and methodological approaches.

Figure 2. Research design and methodological approach.

Dashboard development

To provide knowledge users and other stakeholders with access to the results of our analyses, we will develop an analytics dashboard. The dashboard will enable the visualisation and exploration of findings with support to drill down to specific hospitals, hospital groups and practices within hospitals or specific themes such as safety, hygiene or ambulatory services. Development of the dashboard will involve iteratively developing scenarios for the use of the dashboard by different user groups, followed by elaboration of these scenarios into concrete information and decision needs for different groups. Finally, the information and decision needs will be translated into wireframe designs of the dashboard, and implemented using a robust analytics framework.

Patient and public involvement

Patients and carers will be involved throughout the conduct of the study. PPI representatives were invited to review and suggest amendments on the proposed research design, aims, and objectives. A patient, carer and public involvement (PPI) advisory group consisting of c. 5–10 patient representatives, family members, carers and the public will be involved in this study on an ongoing basis, including at key milestones, important strategic and methodological decisions. The PPI advisory group will attend project steering group meetings. Additional lay PPI representatives will be invited to participate through existing contacts and channels, as the National Care Experience Programme regularly consults with and involves lay patients, family members, carers and the public in the development of new surveys.

A number of formal engagement activities are planned in order to ensure continued and meaningful engagement. A collective intelligence session will allow PPI representatives to review the draft analytical framework and identify missing themes, ensuring participants can directly shape the analysis and insights to be produced from the secondary analysis.

Workshops will facilitate patient, carer and public input in the preliminary analysis and findings. PPI members and health service staff will engage in a co-design session of the analytics dashboard for exploration of findings. A final engagement will involve the presentation of the final results from the project, validation of the developed analytics dashboard through hands-on sessions for participants and obtaining input on the development of dissemination materials for the project. PPI members will help to inform and frame how findings from the surveys should be communicated in order to engage patients and carers. In organising these engagement workshops, we will carefully consider possible barriers to participation and address them to ensure engagement goals are realised.

Ethics and dissemination

Ethical considerations. This study involves secondary analysis of anonymised qualitative data from the National Inpatient Experience Survey and National Maternity Experience Survey. Ethical approval for the National Inpatient Experience Survey was granted by the Royal College of Physicians in Ireland (RCPI) Research Ethics Committee in March 2018, with approval updated on an annual basis thereafter. Ethical approval for the National Maternity Experience Survey was granted by the Royal College of Physicians in Ireland (RCPI) as an addendum to the ethical approval for the National Inpatient Experience Survey on 28 January 2019. While all qualitative data collected as part of these surveys are anonymised prior to the reporting of results by the National Care Experience Programme, there is a possibility that some potentially identifiable information (such as the names of hospital wards) may be present in the data. As part of the data management and governance tasks, the study team will re-screen the data to identify any mentions of names of people, specific ward etc. and remove them prior to further analysis. All PPI members will give informed, written consent prior to their involvement.

Dissemination. Findings from this study will be disseminated through the publication of a national report, a webinar, press release, video animation, social media campaign and infographics for the general public, as well as publications in peer-reviewed journals and presentations at academic fora. Knowledge translation activities will include regular meetings of all project partners and development of quality improvement plans. This study will build upon the well-established partnership between the Irish Health Information and Quality Authority (HIQA), the Health Service Executive (HSE) and the Department of Health to ensure that senior HIQA, HSE and Department of Health stakeholders are accountable for responding to the findings. The involvement of knowledge users, stakeholder workshops and the development of eLearning tools will ensure purpose-built resources are in place to aid understanding of project findings, assist in the use of the newly developed dashboard and provide practical guidance on using the findings to improve areas where deficiencies in care experiences are highlighted.