Funders’ responsibility to ensure value in research: a self-audit by the Health Research Board Ireland

As a public funder of health research, the Health Research Board (HRB) Ireland has an obligation to manage its funds well and to maximise the value of the research that it funds. Ways in which research funding can be wasted have been examined by researchers over the years, and a seminal series on research waste was published in the Lancet in 2014. The series systematically analysed every step of the funding lifecycle in five major stages and made recommendations to various actors including research funders. Prompted by its participation in the Ensuring Value in Research Funders’ Forum, between June and October 2019 the HRB undertook a self-audit against the 17 recommendations identified in the Lancet series. Key HRB staff collated relevant policies and practices regarding each recommendation and sub-recommendation and assessed the HRB’s performance under each heading. The self-assessment reflects the state of HRB policies and practices in October 2019. Of the 17 recommendations, five were found to be areas of strength and six were found to be areas of partial strength. Areas of strength reflect work over many years such as support for evidence synthesis, strong processes around award selection, driving research integrity and open data including an HRB-funded open publishing platform. Four recommendations were found to be areas for growth. These mostly revolve around real time reporting of study protocols and of ongoing funded research outside of clinical trials. Work is progressing to address some of these areas. Two were found not to apply to the HRB.


Introduction
Funders of health research typically receive money from members of the public directly in the case of charitable organisations or via taxation in the case of state agencies. Funders have an obligation to make best use of this income, to maximise its societal benefits. Research waste occurs when the use of money and/or effort is not optimal. Ways in which research waste can occur have been examined by researchers over the years (Chalmers & Glasziou, 2009 ) that systematically analysed every step of the funding lifecycle in five major stages. These include 1) funding research that is relevant to knowledge users, 2) ensuring appropriate design, methods and analysis, 3) efficient research regulation and management, 4) information about the research being fully accessible, and 5) unbiased and useable research reports. It spelled out recommendations for funders, researchers, research performing institutions, publishers, policy makers, regulators and research ethics committees.
Individual funders across the world have grappled with how best to optimise the use of their funding for a long time. This is a complex and multi-faceted endeavour where best practice shifts over time. In 2017 a group of funders from different countries, including the Health Research Board (HRB), came together under the banner of the 'Ensuring Value in Research Funders' Forum' to work together on approaches to optimising the use of their funding (Chinnery et al., 2018). This is a place where health research funders of all sizes and with different remits can share ideas, learn from each other, collaborate and create impact. In 2019 the HRB undertook a self-assessment of its own practices and policies in a systematic way.
The HRB is the main funder of health research in Ireland, and a statutory agency under the Department of Health. The HRB funds research across a wide spectrum from patientoriented and clinical research to population health sciences and health services research. Funding is provided for project and programme grants, career development, infrastructures and networks.
The HRB has an annual funding envelope of €45 million and manages 350 active awards with a total value of €240 million. Its funding Directorate includes approximately 27 staff. The HRB sees itself as a learning organisation with a tradition of leading in the development of best and next practice in specific policy areas. The HRB has three other functions: the Evidence Centre develops evidence synthesis products for the Department of Health; the National Health Information Systems provide information for health service planning and makes its data available for research; and the nascent regulatory function of the HRB encompasses the Health Research Consent Declaration Committee and a National Research Ethics Committee.

Methods
For comparability the methods used for the HRB self-audit In total, four HRB members of staff (all authors) were chosen to participate in this self-audit based on their role in the organisation: Head of Pre-Award, Head of Post-Award and Evaluation, Programme Manager Policy and EU Funding, and Director of Research Strategy and Funding, respectively. All roles include a remit for policy. Jointly they are familiar with the spectrum of HRB funding policies, most of which have been driven personally by these individuals, and the national and international discussions informing these policies. The first author suggested the concept of a self-audit, the composition of the study team, methodology and respective roles at a team meeting. Approval for the project was given by the Director of Research Strategy and Funding. Work was completed as part of the overall work of the study team members for the HRB, without any additional compensation.
The group examined HRB's existing policies, initiatives and practices against 17 recommendations for funding agencies from the Lancet series. Many of the recommendations contain a number of sub-recommendations to capture multiple dimensions, leading to a total of 35 areas to assess (Table 1).
To ensure that diverse perspectives were captured, initially each author collated relevant HRB materials, policies, or practices of which they were aware in their area of expertise. This material was collated by the first author, shared with the other authors and then added to collectively. The scope of the material to be included under each recommendation was agreed jointly and included for example: • HRB's track record in supporting specific areas such as evidence synthesis, clinical trials or public and patient involvement;

Amendments from Version 1
We have addressed the constructive feedback of the reviewers and made amendments in line with most of their suggestions.
-Slightly restructured information on result in abstract for better readability.
- Added some specific examples and links in result table. -Under methods, added detail about the scope of material used and the assessment of individual authors prior to discussion and resolving of discrepancies.
-Slightly restructured description of results for better readability, in line with the abstract.
-Provided a number of further updates on relevant improvements since the self-audit in the discussion.
Any further responses from the reviewers can be found at the end of the article   HRB has a suite of policies on Open Access, data management etc and has worked nationally and internationally to determine best way of making data FAIR • Practices such as those identified in HRB Call Guidance Notes for each call including the requirement to present systematically gathered evidence, and reference to good practices such as use of core outcome sets, registration of protocols etc, selection of only international reviewers to avoid conflicts of interest, unconscious bias briefings and measures, external observers invited to selection panel meetings.
On this basis, each author then independently categorised fidelity to the 17 recommendations as: 1) "area of strength" -HRB's practices reasonably address all sub-recommendations; 2) "area of partial strength" -HRB's practices reasonably or partially address all sub-recommendations; 3) "area of growth" -HRB's practices do not address all sub-recommendations, either reasonably or partially; or 4) not applicable. These rating were used by PCORI in their self-audit. Discrepancies were resolved through discussion and final ratings reflect consensus.
The kick-off meeting took place in the HRB offices in June 2019, with in an depth discussion of the methodology and approach. Two more meetings took place in July to finalise the collation of materials feeding into the self-audit. The individual assessments of HRB performance against the recommendations were completed in October 2019 with good overlap between authors. For ten recommendations, all four authors individually came to the same rating. For the remaining seven recommendations, three authors individually came to the same rating, and one rated it with an adjacent rating (e.g. 2 instead of 1 or 1 instead of 2). Any differences in rating were resolved at a meeting in October 2019. The wording for publication of the HRB related processes or initiatives was finalised and agreed in February 2020.

Results
The self-assessment reflects the state of HRB policies and practices in October 2019. It adopted a whole-of-organisation approach beyond the HRB funding remit. (2) appropriate design, conduct and analysis; (3) efficient regulation and management; (4) full reporting and accessible data; and (5) complete, unbiased and useable reports. Of the 17 recommendations, five were found to be areas of strength (3, 6, 9, 13 and 16) and six were found to be areas of partial strength (2, 10, 11, 12, 15 and 17). These 11 recommendations encompass 22 sub-recommendations. Four recommendations encompassing 11 subrecommendations were found to be areas for growth (4, 5, 7 and 14). Two were found not to apply to the HRB (1 and 8) due to its remit.

Discussion
By nature, any self-audit has the potential for bias. We aimed to avoid bias by using an external framework that had been published previously, including contributors with different perspectives, and concluding multiple rounds of discussion and feedback on each recommendation and its sub-recommendations.
This self-assessment positively highlighted areas where significant effort has been made over many years. The areas that scored well have been a core part of the HRB work programme for some time: quality of the scientific peer review process in all its aspects, in-house and extramural support for evidence synthesis and methodological support for researchers. The role of the HRB in the implementation of Irish legislation accompanying general data protection regulation (GDPR) in the context of health research is also reflected positively. Since the self-assessment, the HRB has hosted and managed a national research ethics committee for coronavirus disease 2019 (COVID-19)-related research with expedited turnaround times for decisions, and national research ethics committees for clinical trials of regulated medicinal products and of medical devices are now in place.
As the main funder of health research in a small country, the HRB typically issues funding calls that focus on the type of outcome expected, not on the subject area. We therefore emphasise collaboration between researchers and knowledge users in a variety of funding schemes to ensure the relevance of research findings.
The HRB has been very active in the international and national development of research integrity, FAIR (Wilkinson et al., 2016) data and open research, including the development of coordinated policy and frameworks. Training in research integrity is mandatory for recipients of funding and their teams, and HRB contributes to a national subscription to an online training platform for research integrity. In Ireland the HRB has played a leading role in the development of PPI capacity with innovative and new system-wide approaches.
Progress has been made in all these areas, but there are still opportunities to further improve the implementation of policies across the spectrum and enhance institutional and researcher capacity to 'do the right thing'. The authors view the undertaking of self-audits as part of continuous improvement efforts.
For example: • The culture change required for the meaningful inclusion of members of the public across the research endeavour has only started. The HRB introduced public reviews for some funding schemes in 2017 and is providing infrastructure support for institutions to enhance their capacity for PPI, but more researchers and PPI contributors need to gain more direct experience.
• Having a route via HRB Open Research to publish protocols and outcomes quickly and without publication bias is important, but alone does not guarantee that they are published. In a recent call for COVID-19 research the publication of protocols was mandatory, which was fully implemented. We are reflecting on this experience and starting to roll out such a requirement across other schemes. There is a noticeable increase in study protocol publications on HRB Open Research across other funded projects.
• A current area of focus is around data, encompassing the review and publication of data management plans, the further broadening of FAIR data capacity through the training of data stewards in institutions, and a proof-of-concept initiative facilitating the safe linkage of datasets and secondary use of data. This is an evolving area internationally and capability and capacity are currently limited in Ireland. HRB has partnered with the Central Statistics Office Ireland to provide researcher access to COVID-19 data facilitated by a National Research Data Governance Board (hosted by the HRB).
• The HRB is now a member of the Irish ORCID consortium and capture of researcher ORCID numbers and linkage to ORCID profiles is now enabled in our electronic grant management system.
Whilst work is ongoing in some of the four areas for growth, it is currently not clear how to address some subrecommendations. Some areas are challenging for many funders including the HRB. The registration and real time reporting of ongoing funded research (particularly outside of clinical trials), which is captured in recommendations 4, 5 and 14, poses difficulties, with few suitable repositories. In their selfaudit PCORI note similar challenges in this space (Whitlock  et al., 2019). This is an area that requires more consideration in the future and would benefit from infrastructural solutions beyond the remit of the HRB. Information on HRB awards has since become searchable in Dimensions, a bibliometric and data analytics tool.
The framework used here included recommendations for all players within the research ecosystem. It is relevant to research funding organisations but not specifically tailored towards them. Based on the experience of their respective self-audits, HRB and PCORI are currently contributing to the development of a new tailored tool for the self-assessment of research funding organisations by the Ensuring Value in Research Funders' Forum. This will align with the Ensuring Value in Research principles, provide guidance on methodology and areas to consider, and better focus the work associated with a self-audit.

Data availability
All data underlying the results are available as part of the article and no additional source data are required.
In order to be able to reproduce the data, the readers need to know more about, "...initially each author collated relevant HRB materials, policies, or practices of which they were aware in their area of expertise." -Either how they identified/searched them or a list of these documents (if not all of them is possible, at least some examples of them).
In Table 1: It would be useful that in the self-assessment that you would differentiate between transparency around having criteria and process in place on how decisions are made (which a lot of funders do) versus transparency around how those procdess are implemented e.g. minutes of meeting or external observants in the room (which less often done by funders). Table 1, row 3 item (a): For this, "Requirement of systematically gathered evidence ((1) systematic identification of previous work, 2) critical appraisal, 3) synthesis of the evidence and 4) interpretation of findings) in growing number of schemes", is it possible to give a more accurate number than growing number of schemes or at least an estimate of how many schemes or applications that applies to. It will be also useful to highlight whether it is for researcher-driven grants or commission based one too or both?  Table 1: It would be useful that in the self-assessment that you would differentiate between transparency around having criteria and process in place on how decisions are made (which a lot of funders do) versus transparency around how those process are implemented e.g. minutes of meeting or external observants in the room (which less often done by funders). Reference added. Table 1, row 3 item (a): For this, "Requirement of systematically gathered evidence ((1) systematic identification of previous work, 2) critical appraisal, 3) synthesis of the evidence and 4) interpretation of findings) in growing number of schemes", is it possible to give a more accurate number than growing number of schemes or at least an estimate of how many schemes or applications that applies to. It will be also useful to highlight whether it is for researcher-driven grants or commission based one too or both?
We have added text: "in currently three investigator-led schemes with a total of approx. 120 applications." The HRB has few commissioned grants.

Table 1, row 3 item (b): "Applications for clinical trials requests search of relevant registries" -Is that part of a systematic review or separate?
We have added text: "also". This is the the same context as the systematic review and ensures that the these registries are searched to avoid any ambiguity.

Methods:
In Recommendation 1, please provide information on how you define basic research (e.g. Health Research Classification System -Research Activities).
What does 'good overlap between authors' mean? Please elaborate on differences in rating to provide a clear sense of the extent of agreement between authors.

Other comments:
In Recommendation 4a, data on awarded HRB research grants should also be submitted to a grant data portal that consolidates grant data from various funding sources. This would greatly facilitate broader research funding analyses. We admit, however, that there is currently a scarcity of available public data repositories.
It would be helpful to provide a timeframe for the next iteration of HRB's self-audit to monitor progress on areas of partial strength and areas for growth.