A cross-sectional analysis of Altmetric coverage of health research from Irish research organisations (2017-2023)

Study design, setting, and eligibility

We previously published this project’s protocol which provides more detail about the Altmetric dataset and planned analyses²⁴. We used Altmetric to search for all research outputs (i.e., articles, book, and chapters) published between 1 January 2017 and 31 December 2023 that were associated with an Irish research organisation. This period was chosen to give insight into pre-, during and ‘post’- pandemic trends and to allow for a time-buffer for retrieving data (due to instability concerns). Associations were determined from an author’s affiliation listed as being attached to an Irish research organisation. Of note, order of authorship (e.g., first or senior) was not extracted, thus this is Irish-engaged research not necessarily Irish-led. Data on attention scores from all Irish health research outputs were obtained using an institutional license for Altmetric Explorer, thus we are unable to share the datasets; however, the .rmd files and non-proprietary datasets are available on the Open Science Framework²⁵.

We used R version 4.3.2 (https://posit.co/download/rstudio-desktop/) to analyse data for this project. We are reporting this cross-sectional study according to the STROBE reporting guideline (Appendix 1)²⁶. As data is publicly available, no ethical approval nor consent was needed for this study.

Datasets

Altmetric currently uses the Global Research Identifier Database (GRID) ID²⁷ to identify organisations and maps to the current Research Organisation Registry (ROR) ID system²⁸. ROR is a global registry which helps link researchers and their outputs to the research organisations where they work. As of 13 May 2024, there were 663 Irish research organisations listed in the ROR database. These organisations spanned multiple sectors including: Archive, Company, Education, Facility, Government, Healthcare, Nonprofit, or Other. Three authors (BY, SA, MKS) accessed Altmetric Explorer to manually download comma-separated values (csv) files from June 6 – June 21 2024. Of the 663 listed organisations, 251 had no research outputs during our inclusion dates and 31 organisations were not or no longer indexed in Altmetric – leaving us with an initial dataset including research outputs associated with 381 Irish research organisations.

As we were only interested in ‘health research’, we filtered our dataset to only include biomedical research outputs. To identify these outputs we used the 2020 Australian and New Zealand Standard Research Classification (ANZSRC) system’s Field of Research divisions relating to biomedical research²⁹. Altmetric uses the ANZSRC system to classify outputs. The following five divisions were used: Biological Sciences (31), Biomedical and Clinical Sciences (32), Chemical Sciences (34), Health Sciences (42), and Psychology (52). Excluded fields are listed in our protocol²⁴. ANZSRC contains divisions (broad subject areas or research disciplines) which are further delineated into groups and fields. Each research output is classified to ≥1 division — where an output cannot be classified at this level, the division assignment is based on a journal-level classification. We included research outputs if they belonged to ≥1 of the five divisions listed above.

After filtering and deduplicating our dataset, we retrieved additional citation and topic area information from the OpenAlex API^30–34 which matched items based on their DOI. OpenAlex is an open-source catalogue which has average reference numbers comparable to Web of Science and Scopus³². OpenAlex contains over 4,500 topics which are organised in a hierarchy containing domain, field, and subfield. These correspond to the categories used by Scopus’s ASJC system, except that they are applied at the level of ‘works’, rather than at the journal level.

Analysis

As established in our protocol²⁴, we performed descriptive analyses (counts and frequencies) for the following bibliometric information: type of research output (i.e., journal article, book, chapter), open access status and type³⁵, sector prominence, five subject divisions and their subdivisions, and 20 most frequent journals, funders, and organisations.

To investigate trends, we plotted the overall number of outputs, subjects, subdivision areas, sectors, and publishers on a yearly basis. When attempting to plot data on a daily and quarterly basis, we discovered indexing errors where 1 January of every year was overrepresented in the dataset. After contacting Altmetric support, we were notified that when a publication does not include a complete publication date, the Altmetric system automatically fills in the missing information with ‘01’, creating January 1st if both the month and day are not specified. We also tried to address date issues by pulling publication dates from OpenAlex API but there were more incorrectly indexed items (e.g., showing years prior to 2017). For this reason, we deviated from our protocol and chose not to conduct any analyses as daily or quarterly. We also did not use the yearly OpenAlex prespecified topics and concepts due to resource constraints although the data is available on the Open Science Framework²⁵.

For Altmetric analyses, we calculated the overall and yearly averages and medians for the AAS and per medium (e.g., X, Facebook, policy documents).

We also performed several subset analyses. Firstly, a subset analysis of the outputs with a score of ≥20 as Altmetric has indicated that this score can generally be considered as receiving more attention than most of its ‘colleagues.’ Secondly, we removed outputs with an AAS score of 0 to see how measures of central tendency could be affected. Next, we calculated the average and median AAS and per medium for a subset of items only on pre-print servers. These subgroup analyses were to highlight differences between ‘under-performers’, ‘over-performers’, and pre-prints, which rose in prominence during the pandemic.

Finally, as outlined in the published protocol²⁴, we split our dataset by year and performed citation analyses on a subset of the data (2017–2020) as previous research has indicated that a 3-year citation window is relatively stable (e.g., the increasing trend generally stagnates)^36,37. To investigate whether there was an association between AAS and traditional article-level citation metrics (as measured by both Altmetric Dimensions and OpenAlex) and open access status, we used zero-inflated negative binomial regression models to account for the large number of AAS scores of zero in our dataset. The zero-inflated model contains two parts: 1) a binary model to model the excess zeros and 2) a count model for the count process. Ignoring the zero-inflation component for these metrics could potentially lead to incorrect conclusions. A low probability of excess zeros indicates that almost all zeros in the AAS are adequately explained by the count process, and not due to a separate zero-generating mechanism.

Descriptive statistics

Of the listed 663 Irish research organisations, 381 (Appendix 2) had 135,855 research outputs within our time period with 100,016 being unique. After filtering and deduplicating research outputs to only include those which belonged to at least one of our included areas of health research²⁴, there were 58,056 unique-health related research outputs associated with 303 research organisations (Figure 1).

Figure 1. Search flow diagram.

There was an increase in the number of outputs from 2017 to 2020 with a steep incline from 2019 to 2020 (23%), and an increasing trend continuing into 2021 (8.7%). After 2021, the number of outputs declined but it has not returned to pre-pandemic (2019) levels. (Figure 2) An overwhelming majority of research outputs were articles (96.6%), with much less being chapters (3.34%) or books (<1%). A majority of outputs were available as open access (73.2%) with general increases over time, although rates in 2021 (78%) and 2022 (79%) were higher than 2023 (76%) (Figure 3). Closed access has decreased over time although it still represents roughly 1 in 5 outputs in the dataset. Open access (OA) can be classified by Altmetric³⁵ in several ways: 1) hybrid is freely available under an open licence in a paid-access journal, 2) green is freely available in an OA repository, 3) gold is published in a fully open access journal, 4) bronze is freely available on publisher’s website, but without an open licence. Hybrid and gold access have increased over time while green open access has remained relatively stable and bronze has decreased over time.

Figure 2. Research outputs over time (yearly).

Figure 3. Open Access (OA) status and type over time (yearly).

The sectors most represented were education (67.2%) and healthcare (23.8%) with the others being far less prominent: facilities (4.6%), government (1.4%), other (1.1%) and less than 1% for companies, non-profits, and archive organisations. Accordingly, the organisations with the most research outputs in our dataset were predominately education (i.e., universities) and healthcare organisations. (Table 1) There was a lot of cross-sectoral collaboration, with every sector collaborating at least once. The highest number of collaborations took place between education-healthcare, education-facility, facility-healthcare, and education-government organisations. (Appendix 3, Tables 1–2, Figure 1)

Of the five included ANZSRC Fields of Research (Biological Sciences, Biomedical and Clinical Sciences, Chemical Sciences, Health Sciences, and Psychology), Biomedical and Clinical Sciences was the most prominent (n = 33,060, 57%), however this category does have more (16) subdivisions than the others (6–10). Generally, there was not much fluctuation over the years. The five most frequent divisions were: clinical sciences (20.7%), health services and systems (9.7%), oncology and carcinogenesis (5.1%), public health (4.3%), and biochemistry and cell biology (3.5%) (Table 2). Of the clinical sciences outputs, 44% were from the generic clinical sciences subdivision, with oncology and carcinogenesis (10.8%), cardiovascular medicine and haematology (6.1%), reproductive medicine (5.7%), immunology (4.7%), and pharmacology and pharmaceutical sciences (4.7%) rounding out the top 5 specific clinical areas. (Appendix 3, Figures 2–4, Table 3–4)

The 20 most frequent publishers in the dataset align with the largest publishers globally (e.g., Elsevier, Springer, Wiley) which did not show much yearly variation (remaining in the top 20), however MDPI and Frontiers became much more prominent over time. (Appendix 3, Figures 5–6) When investigating the most prevalent journals in the dataset, it became clear that Altmetric makes no distinction nor linkage between a pre-print and a post-print as both are classified as articles with separate DOIs. Pre-print servers (i.e., platforms that only host pre-prints and not peer-reviewed post-prints) and hybrid publishing platforms (i.e., platforms that post a pre-print, conduct open peer review, and publish a final indexed version such as HRB Open Research and F1000) were all within the 20 most frequently reported journals in the dataset. (Appendix 3, Figure 7) Upon investigating known preprint servers (i.e., bioRxiv, ChemRxiv, medRxiv, and Research Square) and hybrid open-access publishing platforms (i.e., Gates Open Research, F1000 Open Research, HRB Open Research, and Wellcome Open Research), from our entire dataset (n = 58,056), there were 2,299 preprints from any of the major preprint servers and 528 (411 unique) items on the hybrid platforms. This represents less than 5% of our dataset. Of note, each output has a unique AAS (i.e., a pre-print and a post-print discussing the same research project could have a different AAS), so we did not exclude them from analyses. Prominent Irish and European funding agencies were well represented in the top 20 funders amongst 1,959 unique funders in our dataset (Table 3).

Altmetric attention

Of the 58,056 unique-health related research outputs in our dataset, 10,917 (18.8%) had an AAS of 0, meaning that no sources covered them. (Table 4, Table 5) There were also no Mendeley readers for 1,754 of these outputs indicating that they received no academic nor public attention from the Altmetric-tracked sources. 9,054 outputs (15.6%) had an AAS of 20 or above (considered as doing better than most of its contemporaries)³⁸. Outputs which were on pre-print servers had the lowest AAS, average mentions, and citations.

The average AAS for the entire dataset was 21.16 with Twitter (X) being the largest medium (n = 1,276,135) represented by far with an average of 21.98 mentions per output. (Appendix 3, Figure 10) The average mentions for the other outputs were much smaller in comparison: 1.38 for news sources (n = 79,964), .31 for Facebook (17,879), .18 for blogs (n = 10,245), .11 Wikipedia (n = 6,278), .07 for policy documents (n = 4,078), and .04 for patents (n = 2,423) (Appendix 3, Figure 9). The average AAS was highest for 2020 (24.7) (Appendix 3, Figure 11) and for outputs in the Biological Sciences Field of Research (28.37). Chemical Sciences had the lowest average AAS as well as average mentions for most mediums. There were some yearly trends for certain mediums with Facebook averages decreasing from 2019 onwards and Twitter peaking in 2020 then decreasing back towards pre-pandemic averages. Higher patent and policy averages were seen in 2017–2019 than in recent years. Other slower-to-accumulate metrics such as citations and Mendeley readers showed yearly variations as well with higher averages for the oldest years in the dataset. Table 5 data obtained from Altmetrics (Dimensions) and the Open Alex API varied slightly but generally aligned. There were many skewed distributions so additional metrics (i.e., medians and IQRs are available in Appendix 4).

We found positive statistically significant associations between the Number of Dimensions citations, OpenAlex citations, and Mendeley readers with the AAS for the dataset (2017 – 2020) and yearly subsets. (Appendix 3, Table 5) As the number of citations increased, the average AAS tended to increase, although the overall effect size was small (a 1–1.5% increase per citation). There were no overall trends for the zero-inflation models. Open access was also found to have a strong positive association with AAS. There was some yearly variation with 2020 showing the largest effect size (1.12), meaning open access articles in 2020 had a larger relative increase in AAS compared to previous years.

Limitations

The main limitations of our study concern the content and quality of our datasets which should be considered when interpreting the data. For Altmetric data, we found some stability and quality issues which should be considered for future analyses of Altmetric data. Stability issues have been documented previously by work investigating research from 2012 to 2021 which found vanishing mentions, 23.7% of publications with AAS fluctuations over a year, and nearly 30% of papers showing high volatility for Twitter (X)⁵². We downloaded Altmetric data in one time period to reduce instability but unfortunately due to license and access issues, we were unable to use the API to obtain the Altmetric data by ROR. Due to loss of licensing access, we could not update the data to include 2024 in our dataset. However, we included a range of years and a time buffer as previous research has indicated that Altmetric mentions accrue for a longer period than previously believed⁵³. Our code can help inform future analyses investigating how stable the AAS and mediums are over time.

Due to indexing issues with publication dates, we were also unable to run analyses with more granular detail (quarterly and daily). When a publication does not include a complete publication date, the Altmetric system automatically fills in the missing information with “01”, creating 1 January when both the month and day are not specified. We attempted to pull in the publication date data from the OpenAlex API to address this but found further issues with years incorrectly outside of our range. In fact, OpenAlex notes that filtering by publication date is ‘not a reliable way to retrieve recently updated and created works, due to the way publishers assign publication dates’⁵⁴.

We also found that pre-print servers (e.g., medRxiv) were well-represented in the top ‘journals’ in our dataset and it has previously been reported that Altmetric has issues tracking publications with multiple versions (i.e., a pre- and post-print)⁵⁵. We decided to keep pre-prints in our dataset and treat them as unique outputs as each has its own AAS and previous research has indicated that while COVID-19-related articles were more likely to have a preprint, the presence of a preprint does not seem to affect AAS⁵⁶.

In addition to debates around stability and quality, there are also many critiques of the AAS and its reproducibility issues and how it weights different sources of attention^57,58. For example, Christin et al.⁵⁹ encourage researchers to observe different social media logics in healthcare, such as informing or raising awareness for well-being, creating public debate or curating wider partnership-based engagement. The AAS is weighted to favour traditional posts over social media but the prominence of Twitter (X) in our dataset raises concerns about potential overweighting and focus on academic impact as research has indicated that tweets mostly originated from within academia which engaged other academics⁶⁰. Lastly, Altmetric does not track other mediums or platforms which are popular in Ireland such as TikTok and Instagram¹⁸.