Altmetric coverage of health research in Ireland 2017-2023: a protocol for a cross-sectional analysis

Dataset

Data for this study will be gathered from Altmetric institutional access. Altmetric (altmetric.com) tracks 4,000 global news outlets, X (formerly Twitter), YouTube, Reddit, Stack Overflow (Q&A), a curated list of public Facebook pages, blogs, public policy documents, IFI CLAIMS patents, Wikipedia, Mendeley, and Publons. It uses a unique identifier (e.g., DOI, PubMedID, arXiv ID, ISBN, etc.) to track online attention given to a specific research output. (https://www.altmetric.com/about-us/our-data/how-does-it-work/) Research outputs included are largely journal articles although book chapters and books will also be included. Of note, some items within Altmetric are classified as ‘news’ but can be considered articles as they are perspectives, commentaries, overviews, or hot topics.

We will use Altmetric Explorer to search for all research outputs published between 1 January 2017 and 31 December 2023 from Irish organisations that have Research Organisation Registry (ROR) IDs. ROR is a global registry of open persistent identifiers for research organisations which helps link researchers and their outputs to institutions across sectors (e.g., education, government, healthcare, non-profit, etc.) (https://ror.org/about/). Altmetric uses the predecessor system, the Global Research Identifier Database (GRID) (https://www.grid.ac/) which maps to ROR. As of 9 April 2024, there were 663 research organisations with Ireland listed as their country of address. We searched both active and inactive IDs in case an organisation had outputs during the time period but then decided to inactivate their indexing GRID (e.g., they published in 2017–2019 but then deactivated their ID). The lead author (MKS) and two medical students will use the Altmetric Explorer interface to search for and download research outputs published within our date frame from each individual organisation. Datasets will be tracked using a tracking log in Excel to record the downloader (e.g., MKS), total number of research outputs, number of outputs mentioned, filename, and download date. If an organisation has a least 1 output, we will download their data as a csv file using a standard naming notation (ID_YYYY-MM-DD). These csv files will be stored in one folder, spot checked for completeness (MKS), then combined into one dataset which will include research output data from all organisations producing output from 2017 – 2023. Of note, the datasets have the same 46 variables, will be downloaded in UTF-8 to account for non-English characters, and dates will be checked prior to stacking. This tracking log and our RMarkdown code detailing combining of datasets, cleaning, pre-processing, and more can be available on our Open Science Framework accompanying our results manuscript³². While the Altmetric API (https://www.altmetric.com/solutions/altmetric-api/) is also available for information retrieval, we did not find it suitable for pulling data based on an organisation’s GRID ID.

Each dataset (e.g., csv file) contains 46 standard columns or variables, one of which is the field of research. Each research output in the dataset is classified to at least one field of research (FoR) using the 2020 Australian and New Zealand Standard Research Classification (ANZSRC) system. The ANZSRC is a hierarchical system which contains divisions (broad subject areas or research disciplines) which are further detailed into subsets: groups and fields. ANZSCR was developed for use in the measurement and analysis of research and experimental development (R&D) statistics in Australia and New Zealand³³. In instances where there is a lack of information and it cannot be classified at this level, the code is assigned based on a journal-level classification.

For the purposes of our project we are primarily interested in the following Divisions of biomedical research listed in Table 1: Biological Sciences (31), Biomedical and Clinical Sciences (32), Chemical Sciences (34), Health Sciences (42), and Psychology (52). Excluded areas include: Agricultural, Veterinary and Food Sciences (30), Built Environment and Design (33), Commerce, Management, Tourism and Services (35), Creative Arts and Writing (36), Earth Sciences (37), Education (39), Engineering (40), Environmental Sciences (41), History, Heritage and Archaeology (43), Human Society (44), Indigenous Studies (45), Information and Computing Sciences (46), Language, Communication and Culture (46), Law and Legal Studies (48), Mathematical Sciences (49), Philosophy and Religious Studies (50), and Physical Sciences (51). As research outputs can be classified to several areas, as long as at least one of our included Divisions is in the field, the output will be included. For example, if a study is about the health impacts of environmental pollution and contamination, it could be classified under Biomedical and Clinical Sciences (32) and Environmental Sciences (41), thus it would still be retained.

All individual organisation datasets will be stacked and combined, deduplicated, and filtered to only contain research outputs pertaining to at least one field of health research as defined by the ANZSRC (Figure 1). Deduplication will be based on an output’s DOI and impact metrics should be identical across outputs with the same DOI. Of note, author affiliations with a research output do not depend on placement (e.g., first, corresponding). We will create a new variable to maintain links in the likely case where a research output is associated with multiple organisations. For example, if research output X was published by both author 1 at organisation A and author 2 at organisation B, the research output should contain the same DOI which is the information that is being used to track all the attention, therefore, that information should not change and deduplication does not pose a threat to data loss.

Figure 1. Flow diagram of dataset creation.

Analysis

We will use R version 4.3.2 (https://shiny.rstudio.com/) to analyse data for this project. As data will be gathered using an institutional license for Altmetric Explorer, we will use R Markdown (https://rmarkdown.rstudio.com/) to create an html document to maintain privacy of the proprietary data but promote reproducible research practices. The .rmd file will also be available on GitHub (https://github.com/sharpmel) and the project will be registered on the Open Science Framework. (https://osf.io/kfct6/)

Data will be cleaned using R and descriptive analyses will be performed for general bibliometric information such as: the type of research output (i.e., article, book, chapter); open access status and type; 20 most frequent journals, funders, and organisations in our dataset; the prevalence of sectors (e.g., education, healthcare); and the five subject areas of health research and their subdivisions (yearly). To investigate trends over time, counts of frequencies will be plotted on a quarterly and yearly basis for the overall number of research outputs and yearly for subject and subdivision areas, sectors, and publishers.

For Altmetric analyses, we will report counts, and averages and medians per medium (e.g., X, Facebook, policy documents) for both the deduplicated dataset as a whole and for the deduplicated dataset with the zero-count AAS removed as there may be large amounts of zero counts in our dataset, potentially skewing measures of central tendency. We will also report the number of outputs with a score of 20 or above as Altmetric has indicated that this is a general score which can be considered as doing better than most of its ‘colleagues’³⁴. All medium data will be plotted on a quarterly and yearly basis from 1 January 2017 through 31 December 2023. We will also segment data by subject areas and divisions. As the World Health Organisation (WHO) declared Covid-19 a pandemic on 11 March 2020³⁵ and no longer a public health emergency of international concern on 5 May 2023³⁶, these cut points will be used for general discussions on pre-, during, and post-pandemic.

Assuming an adequately sized dataset, we will also investigate whether Altmetric Attention Scores correlate to traditional article-level citation metrics. Crossref lags significantly behind other major sources in terms of comprehensiveness³⁷, therefore we will use OpenAlex to obtain citation data. OpenAlex^38,39 is an open-source catalogue which includes metadata for 209 million outputs like journal articles and books and its average reference numbers are comparable to Web of Science and Scopus⁴⁰. We will use zero-inflated negative binomial regression to account for the large number of AAS scores of zero. As articles in 2017 have had more time to accrue citations than those published in 2023, we will restrict our analyses and split our dataset by year. Previous research has indicated that a 3-year citation window is relatively stable so we will perform these analyses on the data from 2017–2020 only^41,42.

Field and clinical area. Although the FoR classification system will give some insights as to content areas which are being covered, it does not provide more granular detail. OpenAlex provides a more thorough and in-depth view of academic research as it contains 4,500 topics and builds upon and fills the gap from the discontinued Microsoft Academic Graph (MAG)^43,44. We will gather topic information by using the OpenAlexAPI to pull this information matched by the DOIs in our final dataset. OpenAlex uses a hierarchical system that organises topics into levels ranging from broad to more specific, from domain to field, subfield, and finally topic system. We will focus on lower level topics and will report the 20 most frequent topics per year.

Limitations. Our main limitation is the dataset itself and the quantitative focus of measuring impact. Firstly, Altmetric does not track certain platforms such as LinkedIn, TikTok, and Instagram, thus its generalisability is diminished. It has also been reported to have issues tracking publications with multiple versions (i.e., a pre- and post-print) and the replication of the data can sometimes be difficult due to constantly changing access agreements with data providers⁴⁵.

The lack of a links between preprints (manuscripts uploaded to databases without peer review) and postprints is a larger issue (peer reviewed journal articles) within academic publishing⁴⁶ as each item is assigned a unique DOI and there are challenges indexing preprints alongside their peer-reviewed publication. Of note, in the Altmetric dataset, preprints are included and tracked, they just have their own individual altmetrics separate from the final publication. Preprints played an unprecedented role in disseminating Covid-19 research⁴⁷, so we will include them in our dataset, account for this in statistical analyses, and explicitly disclose this preprint prevalence in our dataset, and frame our findings with this in mind.

Furthermore, although higher scores can be expected from newer papers as time since publication has shown to be associated with Altmetric scores¹¹, even running searches one month apart resulted in changing numbers as old articles can be brought up for discussion at any time. Altmetric data has also been noted to be prone to manipulation and artificial inflation⁴⁸ and some sources are particularly unstable, with certain items ‘vanishing’¹³. We will try to address this by pulling the data in a discrete period of time and we have included a time buffer (i.e., the end of 2023). However, we do recognise that certain mediums may still be ‘incomplete’ as they have different trajectories of attention growth – for example, Twitter (X) attention starts and ends quickly, Mendeley readers accumulate quickly but continue to grow over the years, and policy attention is the slowest form of impact to accumulate⁴⁹.

The quantitative focus of the metrics in the dataset can also lose the context or tone of the coverage and does not account for ‘dose’, i.e., where a mention may be an extensive discussion or extremely brief. A high AAS does not indicate a high-quality piece of research⁵⁰ and the AAS is a measure of attention, not quality. However, recent work comparing altmetrics to norm-referenced peer review scores from the UK Research Excellence Framework 2021 found that Altmetric correlated more strongly with research quality than previously thought although there is large variability with the strength of correlations amongst mediums and between fields (e.g., stronger in health and physical sciences than in the arts and humanities)¹⁴.

Providing a broad overview of the online attention given to health research in Ireland in recent years is our primary objective. Our datasets are not meant for social listening purposes and audience metrics may be limited. However, our project’s results may provide a basis to build upon for future studies investigating how the media is actually covering the work. Altmetric may also be a flawed metric of impact on the public as previous research has shown that most tweets came from within academia with other academics interacting with them⁵¹. Notably, the data in our project likely will contain more health research produced from the academic sector as they primarily communicate via articles, books, and chapters, however, we have included pharmaceutical agencies, governmental health bodies, and hospitals (where they have an ROR) which may provide a broader overview of the online coverage of health research in Ireland. A recent bibliometric analysis of HRB supported publications⁵² found the academic sector well-represented although our project is much broader in its scope.