Funders’ responsibility to ensure value in research: a self-audit by the Health Research Board Ireland

Introduction

Funders of health research typically receive money from members of the public directly in the case of charitable organisations or via taxation in the case of state agencies. Funders have an obligation to make best use of this income, to maximise its societal benefits. Research waste occurs when the use of money and/or effort is not optimal. Ways in which research waste can occur have been examined by researchers over the years (Chalmers & Glasziou, 2009), leading to the publication in 2014 of a series in the Lancet, put together by Chalmers and Glasziou, (Chalmers et al., 2014), (Ioannidis et al., 2014), (Salman et al., 2014), (Chan et al., 2014) and (Glasziou et al., 2014) that systematically analysed every step of the funding lifecycle in five major stages. These include 1) funding research that is relevant to knowledge users, 2) ensuring appropriate design, methods and analysis, 3) efficient research regulation and management, 4) information about the research being fully accessible, and 5) unbiased and useable research reports. It spelled out recommendations for funders, researchers, research performing institutions, publishers, policy makers, regulators and research ethics committees.

Individual funders across the world have grappled with how best to optimise the use of their funding for a long time. This is a complex and multi-faceted endeavour where best practice shifts over time. In 2017 a group of funders from different countries, including the Health Research Board (HRB), came together under the banner of the ‘Ensuring Value in Research Funders’ Forum’ to work together on approaches to optimising the use of their funding (Chinnery et al., 2018). This is a place where health research funders of all sizes and with different remits can share ideas, learn from each other, collaborate and create impact. In 2019 the HRB undertook a self-assessment of its own practices and policies in a systematic way.

The HRB is the main funder of health research in Ireland, and a statutory agency under the Department of Health. The HRB funds research across a wide spectrum from patient-oriented and clinical research to population health sciences and health services research. Funding is provided for project and programme grants, career development, infrastructures and networks.

The HRB has an annual funding envelope of €45 million and manages 350 active awards with a total value of €240 million. Its funding Directorate includes approximately 27 staff. The HRB sees itself as a learning organisation with a tradition of leading in the development of best and next practice in specific policy areas. The HRB has three other functions: the Evidence Centre develops evidence synthesis products for the Department of Health; the National Health Information Systems provide information for health service planning and makes its data available for research; and the nascent regulatory function of the HRB encompasses the Health Research Consent Declaration Committee and a National Research Ethics Committee.

Methods

For comparability the methods used for the HRB self-audit mirrored those reported previously by the Patient-Centered Outcomes Research Institute (PCORI, USA) (Whitlock et al., 2019) and are informed by those used by National Institutes of Health Research (NIHR, UK) for a similar exercise (M. Westmore, personal communication). Both PCORI and NIHR are members of the Ensuring Value in Research Funders’ Forum.

In total, four HRB members of staff (all authors) were chosen to participate in this self-audit based on their role in the organisation: Head of Pre-Award, Head of Post-Award and Evaluation, Programme Manager Policy and EU Funding, and Director of Research Strategy and Funding, respectively. All roles include a remit for policy. Jointly they are familiar with the spectrum of HRB funding policies, most of which have been driven personally by these individuals, and the national and international discussions informing these policies. The first author suggested the concept of a self-audit, the composition of the study team, methodology and respective roles at a team meeting. Approval for the project was given by the Director of Research Strategy and Funding. Work was completed as part of the overall work of the study team members for the HRB, without any additional compensation.

The group examined HRB’s existing policies, initiatives and practices against 17 recommendations for funding agencies from the Lancet series. Many of the recommendations contain a number of sub-recommendations to capture multiple dimensions, leading to a total of 35 areas to assess (Table 1).

To ensure that diverse perspectives were captured, initially each author collated relevant HRB materials, policies, or practices of which they were aware in their area of expertise. This material was collated by the first author, shared with the other authors and then added to collectively. The scope of the material to be included under each recommendation was agreed jointly and included for example:

On this basis, each author then independently categorised fidelity to the 17 recommendations as: 1) “area of strength” – HRB’s practices reasonably address all sub-recommendations; 2) “area of partial strength” – HRB’s practices reasonably or partially address all sub-recommendations; 3) “area of growth” – HRB’s practices do not address all sub-recommendations, either reasonably or partially; or 4) not applicable. These rating were used by PCORI in their self-audit. Discrepancies were resolved through discussion and final ratings reflect consensus.

The kick-off meeting took place in the HRB offices in June 2019, with in an depth discussion of the methodology and approach. Two more meetings took place in July to finalise the collation of materials feeding into the self-audit. The individual assessments of HRB performance against the recommendations were completed in October 2019 with good overlap between authors. For ten recommendations, all four authors individually came to the same rating. For the remaining seven recommendations, three authors individually came to the same rating, and one rated it with an adjacent rating (e.g. 2 instead of 1 or 1 instead of 2). Any differences in rating were resolved at a meeting in October 2019. The wording for publication of the HRB related processes or initiatives was finalised and agreed in February 2020.

Results

The self-assessment reflects the state of HRB policies and practices in October 2019. It adopted a whole-of-organisation approach beyond the HRB funding remit.

Table 1 sets out the results across the five key stages of research: (1) relevance of questions; (2) appropriate design, conduct and analysis; (3) efficient regulation and management; (4) full reporting and accessible data; and (5) complete, unbiased and useable reports. Of the 17 recommendations, five were found to be areas of strength (3, 6, 9, 13 and 16) and six were found to be areas of partial strength (2, 10, 11, 12, 15 and 17). These 11 recommendations encompass 22 sub-recommendations. Four recommendations encompassing 11 subrecommendations were found to be areas for growth (4, 5, 7 and 14). Two were found not to apply to the HRB (1 and 8) due to its remit.

Discussion

By nature, any self-audit has the potential for bias. We aimed to avoid bias by using an external framework that had been published previously, including contributors with different perspectives, and concluding multiple rounds of discussion and feedback on each recommendation and its sub-recommendations.

This self-assessment positively highlighted areas where significant effort has been made over many years. The areas that scored well have been a core part of the HRB work programme for some time: quality of the scientific peer review process in all its aspects, in-house and extramural support for evidence synthesis and methodological support for researchers.

For example, in early 2018 the HRB launched its own open publishing platform, HRB Open Research, that facilitates immediate publication followed by open peer review and combined with an open data policy. This plays an important role in strengthening the assessment around the reporting-related recommendations.

The role of the HRB in the implementation of Irish legislation accompanying general data protection regulation (GDPR) in the context of health research is also reflected positively. Since the self-assessment, the HRB has hosted and managed a national research ethics committee for coronavirus disease 2019 (COVID-19)-related research with expedited turnaround times for decisions, and national research ethics committees for clinical trials of regulated medicinal products and of medical devices are now in place.

As the main funder of health research in a small country, the HRB typically issues funding calls that focus on the type of outcome expected, not on the subject area. We therefore emphasise collaboration between researchers and knowledge users in a variety of funding schemes to ensure the relevance of research findings.

The HRB has been very active in the international and national development of research integrity, FAIR (Wilkinson et al., 2016) data and open research, including the development of coordinated policy and frameworks. Training in research integrity is mandatory for recipients of funding and their teams, and HRB contributes to a national subscription to an online training platform for research integrity. In Ireland the HRB has played a leading role in the development of PPI capacity with innovative and new system-wide approaches.

Progress has been made in all these areas, but there are still opportunities to further improve the implementation of policies across the spectrum and enhance institutional and researcher capacity to ‘do the right thing’. The authors view the undertaking of self-audits as part of continuous improvement efforts. For example:

Whilst work is ongoing in some of the four areas for growth, it is currently not clear how to address some sub-recommendations. Some areas are challenging for many funders including the HRB. The registration and real time reporting of ongoing funded research (particularly outside of clinical trials), which is captured in recommendations 4, 5 and 14, poses difficulties, with few suitable repositories. In their self-audit PCORI note similar challenges in this space (Whitlock et al., 2019). This is an area that requires more consideration in the future and would benefit from infrastructural solutions beyond the remit of the HRB. Information on HRB awards has since become searchable in Dimensions, a bibliometric and data analytics tool.

The framework used here included recommendations for all players within the research ecosystem. It is relevant to research funding organisations but not specifically tailored towards them. Based on the experience of their respective self-audits, HRB and PCORI are currently contributing to the development of a new tailored tool for the self-assessment of research funding organisations by the Ensuring Value in Research Funders’ Forum. This will align with the Ensuring Value in Research principles, provide guidance on methodology and areas to consider, and better focus the work associated with a self-audit.

Data availability

All data underlying the results are available as part of the article and no additional source data are required.