Exploring factors that influence the practice of Open Science by early career health researchers: a mixed methods study

Ethical approval and study protocol

Ethical approval for the study was obtained from NUI Galway Research Ethics Committee [Ref no 19-Mar-31]. Written informed consent was obtained from all study participants prior to data collection. The study protocol is accessible at https://doi.org/10.17605/OSF.IO/PKREN (Zecevic et al., 2020). As the qualitative method had priority and no reporting criteria for mixed methods research exists, this study is reported in accordance with COREQ criteria (Extended data: Appendix 1 (Zecevic et al., 2020)).

Study sample and setting

Study participants were a convenience sample recruited from a two-day introductory training workshop on open science, which was held in NUI Galway (Republic of Ireland) in April 2019 for early career researchers funded by the Irish Health Research Board. Participants self-defined themselves as ECRs when registering for the event, with no restrictions placed on eligibility. There is no unanimously accepted definition of ECR, however previous research has defined an ECR as ‘one who is currently within their first five years of academic or other research-related employment allowing uninterrupted, stable research development following completion of their postgraduate research training’ (Bazeley, 2003).

Study design

We used a convergent mixed methods design to elicit a comprehensive and holistic understanding of the research question (Creswell et al., 2011). A convergent mixed methods design typically requires qualitative and quantitative data from different sources to be collected, analysed separately and brought together and integrated during the interpretation phase (Creswell et al., 2011). Participants provided quantitative data via questionnaires and were subsequently followed up with individual semi-structured qualitative interviews. In our study, participants had self-selected themselves into the two-day workshop, with another level of self-selection to participate in this study, therefore interviewees may not have been truly representative of the general population of ECRs. As such, the qualitative data provided comprehensive insights of their experiences and perceptions of open science, with the quantitative data used to further describe the characteristics, beliefs and experiences of this sample and to add further context to the qualitative findings. Although we also originally intended (as outlined in the study protocol) to integrate quantitative and qualitative data regarding the factors influencing open science practices using triangulation methods (Farmer et al., 2006), this was deemed largely unnecessary and uninformative upon completion of analysis for each dataset, as due to their semi-structured nature and conversational flow during interviews, not all interviewees spoke about specific survey items.

Quantitative data collection

Participants completed a study questionnaire before and after the workshop. The content and structure of the questionnaire were informed by the 2017 European open science survey (O'Carroll et al., 2017). The pre-workshop questionnaire collected data on participant demographics, such as gender, age, discipline etc. Both pre and post-workshop questionnaires (Extended data: Appendix 2 (Zecevic et al., 2020)) included closed and open-ended questions exploring the knowledge and awareness of open science components and initiatives among early career health researchers, as well as their perceptions of the barriers and facilitators influencing their practice of open science activities. As interviews were carried out after the workshop, only data regarding knowledge, awareness and influencing factors from the post-workshop questionnaire were used in this study as this was perceived to be more relevant for characterising the sample. Pre-post questionnaire data was reported separately as part of a workshop evaluation for the funders (Toomey, 2019).

Qualitative data collection

Individual semi-structured interviews were carried out either in person or by telephone to participant preference. Interview duration ranged from 13–34 minutes, with an average of 21 minutes in duration. All the participants were interviewed within three weeks after the workshop and interviews were facilitated by one or two members of the research team (KZ, CH). Interviewers may have been known to participants from attendance at the workshop, but were introduced during interviews as independent researchers with no bias or hidden interests in the topic. A topic guide was developed by an experienced qualitative researcher (CH) with input from KZ and ET, and used to structure the interviews (Extended data: Appendix 3 (Zecevic et al., 2020)). The topic guide specifically explored participants’ understanding and experience with open science and their perceptions of barriers and enablers to practicing open science, with specific probes to enable deeper exploration of the topic in question. Interviews were audio-recorded and transcribed verbatim. Member checking of transcripts was not conducted due to time constraints and also due to evidence suggesting the benefits of member checking for verifying accuracy of transcripts may be relatively small (Hagens et al., 2009).

Data analysis

Basic descriptive statistical analyses including percentage distributions and median calculations were used to describe the closed-ended questionnaire data. Data analysis was conducted using Microsoft Office Excel. Open-ended questions were analysed by coding answers using themes identified during the qualitative analysis, whilst remaining open to the potential for iterative generation of new themes.

Interviews were analysed using thematic analysis (Braun & Clarke, 2006), which was facilitated within NVivo 12 qualitative management software. Thematic analysis is an inductive approach to analysis, thus moving beyond description into interpretation and “telling a story” of the data (Clarke & Braun, 2018), p106). First, transcripts were read several times by one member of the research team (KZ). The researcher then coded statements and citations from the interviews into nodes. Each node was named according to the content of the citations it encapsulated. If a citation did not fit in any of the existing nodes, the researcher created a new node and named it appropriately. The coding was an iterative process, and the researcher went back and forth between the transcripts and the formed nodes, sometimes re-naming the node to give a better idea of its content.

Once this initial round of coding was complete, the NVivo file was sent to a second researcher (CH) to review the nodes. Initial themes were then generated and refined by the two researchers through iterative cycles of discussion and review. The final themes, subthemes and their descriptions were then reviewed by a third researcher (ET) alongside the transcripts. The role of this third researcher was as a ‘critical friend’, by offering critical feedback on interpretations of the data and encouraged reflexivity by providing a “theoretical sounding board” (Smith & McGannon, 2018) p113).

Rigour

A number of strategies were employed to ensure the study was carried out in a rigorous and transparent way. Firstly, as outlined above, the research team agreed the coding and theme development from the qualitative phase. This exercise is known as peer de-briefing and is used to ensure the data is represented fairly in the developed themes to minimise researcher bias (Houghton et al., 2013). Secondly, we created a codebook within QSR NVivo to demonstrate the dependability of our findings (Extended data: Appendix 4 (Zecevic et al., 2020)). Using the coding query function, we were able to illustrate the density of coded references from each participant across all sub themes, emphasising that our findings were grounded in the data (Extended data: Appendix 5 (Zecevic et al., 2020)). This strategy also enhances transparency without the privacy concerns of publishing raw transcripts (Tsai et al., 2016).

Participant characteristics

14 out of the 40 workshop participants agreed to participate in the mixed methods study. Ten were female. Four participants had obtained a PhD in the previous 1–2 years, one was 6 years post-PhD, seven were undertaking a PhD at the time of participating in the study and two of the participants did not have a PhD. Participant demographics are further described in Table 1.

Briefly, survey data identified that participants reported better knowledge of open science components like open access and open peer review than of components such as open data, open source, open notebooks, open education and citizen science. 77% of participants reported concerns over personal data protection and confidentiality, and the lack of institutional guidelines as barriers to data sharing, while incentivising and recognition of open science activities for career progression were identified as important facilitators. Over half of survey respondents believed there to be a lack of financial support and infrastructure within their institution for open science practices. Specifically, the lack of supports such as funding, training, time and institutional buy-in were identified by 92% of survey respondents as one of the most substantial barriers to their ability to practice open science. Further details of post-workshop survey findings are available in Extended data: Appendix 6 (Zecevic et al., 2020).

Thematic analysis

We identified two main themes from the qualitative data: ‘Valuing Open Science’ and ‘Creating a Culture for Open Science’ with a number of subthemes also identified (summarised in Table 2). A ‘wordcloud’ created using QSR NVIVO queries to illustrate most commonly used words when participants talked about open science is provided in Extended data: Appendix 7 (Zecevic et al., 2020).

Valuing Open Science

This theme explores how ECRs define and perceive open science and its importance.

Participants perceived open science as a broad umbrella term, encapsulating 'openness' across the entire research cycle, i.e. from before a study starts (for example, using pre-registration and open notebooks) until after it finishes (for example, with open access publishing and data sharing). In general, this openness equated to a broad sense of transparency, availability, fairness, and replicability and reproducibility. Most interviewees had only recently learned that open science was more than just open access publishing at the end of the research cycle, an insight many attributed directly to the workshop.

Overall interviewees felt they did not have much experience with practicing specific open science activities, but the experiences they had were predominantly positive. Most interviewees had pre-registered a study or published a study protocol and some of them were planning to do so at the next opportunity.

Interviewees generally agreed open science has many benefits, with some less obvious than others. They were concerned that there is a lack of awareness of the benefits of open science. In general, participants felt that practicing open science behaviours led to better research. Ultimately, open science was perceived as the better, more ethical and impactful, way of doing research than standard or traditional approaches.

For example, participants felt that practices like protocol publication would enable others to see what researchers planned to do, in a timely and accessible way. This provided opportunities for peer-reviewers to spot potential errors at an early stage of the research cycle. Pre-registering a study or publishing a protocol also helped in the planning of the research and thinking more thoroughly about the adopted study procedures.

Open science also provided opportunities for collaboration between researchers, through sharing data and other research materials. Another advantage of open science identified by participants was the reduction of research waste and duplication. According to some interviewees, because of the transparency open science brings, it makes research more reproducible and gives the possibility to build-up from existing research.

Further, open science allows publishing results that otherwise might not get published (such as studies reporting data that do not support their hypotheses).

Creating a Culture for Open Science

Cultural and academic pressures

Overwhelmingly, the current academic culture and its according pressures were discussed by interviewees as one of the main factors influencing open science practices. Despite being at an early stage in their academic careers, interviewees already felt under pressure in the academic world. They experienced time pressure, and an expectation to be continuously publishing. Because of this, many activities, including those required for open science, proved difficult to incorporate into their already busy schedules:

Open access journals, interviewees reported, tended not have high impact factors which could be off-putting when trying to achieve career progression. Furthermore, some interviewees believed there was a perception that publishing in an open science journal or platform was easier and therefore would not be valued as highly for career progression as publishing in a traditional high impact journal:

Some interviewees felt that pressures on ECRs were unique due to their unique position in the research world. Most were currently undertaking a PhD or working as a post-doctoral researcher for a supervisor or principal investigator. Interviewees felt that ECRs typically do not manage their own grants and have less autonomy in decision-making regarding activities such as practicing open science. Some felt that if an ECR decided to engage in open science, then they may not get support from the wider research team:

This was a problem especially if the opinions of their supervisor or more senior colleagues differed from their own. Interviewees reported that despite the wish to carry out open science activities, they had to prioritise other activities in order to advance their career.

Interviewees identified a resistance to change in the current culture, some particularly among their more senior colleagues. Interviewees reported that more senior colleagues may not see the value in changing an established system that works, that they are used to and where they are also able to delegate certain tasks they may not see as important. However, at the same time interviewees felt the academic pressures and many challenges to doing open science were similar regardless of being at an early or more advanced career stage:

The concept of increased accountability was discussed by many as a key factor influencing the practice of open science by ECRs, both as a barrier and a facilitator. For example, open science is inherently transparent, which is valued, but this increased transparency also created several concerns for interviewees. The fact that others can see their work led to concerns about feeling exposed, and somehow vulnerable and open to criticism. They also had concerns about potential mistakes being identified:

However, on the other hand, many also felt that open science practices such as open peer review would also make reviewers accountable and encourage them to be more thorough and diplomatic in their feedback to authors.

In addition to the desire to protect themselves, interviewees also expressed a strong need to protect others, for example, research participants. A breach of confidentiality was considered of great importance, particularly when dealing with qualitative data and clinical data. The protection of research participants was an important factor influencing how interviewees felt about the practice of sharing data. They felt that openly sharing data was a key concern especially when treating rare conditions or narrow, very specific groups of people. Interviewees had legal, ethical and personal concerns around sharing data and making data accessible to anyone. They felt that there was still room for improvement and further consideration regarding how to achieve a balance between data sharing and confidentiality:

The issue of nefarious practices came up briefly in some interviews, e.g. a fear of others misusing available data, or research ideas taken from them (i.e. being ‘scooped’). For example, one participant spoke of a fellow ECR colleague who felt their research plan had been stolen from them due to it being available publicly, and that it was important to have a way of safeguarding against that:

Striving to be open

Participants identified many challenges to practicing open science. However, interviewees spoke also about what would help them to engage in open science more often, more willingly and more confidently. They felt that a change in university research culture was needed, starting at ECR stage or even earlier, at an under-graduate level. They felt that a research culture with open science at its core was needed so that open science activities would be encouraged and rewarded. This would enable research to be done in a collaborative and cooperative way, with wider audiences and researchers, and in turn lead to more impactful and applicable research:

Overall, however, a key requirement discussed was a shift in how university metrics are considered. Interviewees felt that a key incentive to practice open science would be the valuing or requirement of open science activities for career promotion and hiring criteria. Specific suggestions included recognising open science activities in a dedicated section of curriculum vitae, as a way to demonstrate alternative forms of impact:

One specific barrier discussed by participants was the lack of funding specifically for open science activities. They identified some solutions for the funding issue, such as internal funding from universities to reward the quality of pre-registered studies:

Other aspects that would help support and develop an open science culture were discussed, including publishing regulations that would enable tracking and reporting of potential deviations from a pre-registered report. The creation of more policies around open science in general were also suggested, such as with a particular focus perhaps on sharing data. Another suggestion was to have an individual in academic settings responsible for open science; like an “open science officer” who could raise awareness, provide support and organise training events:

Overall, the need for broader structure and support was very present among the interviewed ECRs. Support for open science needs to come from academic institutions but also from funding agencies, and the wider research community. In addition, there is a responsibility for researchers themselves to start having conversations about the importance of open science. Learning from colleagues could encourage ECRS to practice it also:

Social media and technology, more broadly, seemed to play a very important role in facilitating open science. Interviewees spoke about the value of specific publishing platforms in connection to open science (e. g. OSF platform, F1000 and HRB Open Research), as well as about blogs or tweets they came across. They also identified a benefit of technology for getting support for practicing open science:

Training and education was considered by interviewees as critical in promoting open science:

Participants spoke about helpful events, related to open science to raise knowledge and awareness around it, but underlined the importance of events focused on training. They had a lot of issues and questions around “how to do open science”, where can one pre-register a study, where can one share their data and how, where and how can one access materials of other researchers. Some interviewees felt that this information should be included in undergraduate and postgraduate curricula. They also felt that events should target as a wide an audience as possible:

Implications for Practice, Policy and Research

It is clear that appropriate support systems are needed to help ECRs to become the new generation of researchers and develop a culture that embraces open science. For example, our participants emphasised the importance of the availability of training events, education and resources within research active institutions to improve open science awareness, knowledge and skills for researchers. However, both formal and informal undergraduate and postgraduate research training and supervision in these institutions should also endeavour to include a holistic focus on the bigger picture beyond knowledge and skills, e.g., by focusing on the benefits and impact of transparent and better quality health research on patients and society, in essence to intrinsically motivate ECRs to practice open science behaviours. Moreover, both ‘top down’ and ‘bottom up’ approaches are needed from all stakeholders including research funders, university and institutional management and individual researchers at all levels (e.g. principal investigators, supervisors, ECRs) to facilitate the development of a research culture of transparency and openness. Specifically, as advocated for by Moher and colleagues, research active institutions and governing bodies should revisit the policies and metrics by which research and its outputs are measured, and accordingly how researchers are hired, promoted and evaluated and to explicitly seek engagement in open science activities (Moher et al., 2020). To support this, institutions could consider the appointment of specific open science officers or champions, or establishing open research committees that consider beyond just open data or open access publishing, but the entire research cycle, as well as building an open research culture.

For ECRs, our study highlights some substantial challenges to practicing open science, but with several opportunities also identified for ‘bottom up’ solutions and strategies to overcome and address these. First, ECRs can ‘get active’ by seeking out others interested in practicing open science and building an open science community through grassroots initiatives like ReproducibiliTea journal clubs, or Open Science Cafés, and build themselves a platform to be heard. Second, ECRs can ‘get vocal’ by using these platforms to lobby and engage with institutional management like ethics committees around data sharing, libraries regarding open access and open publishing policies, and university promotion committees. Third, ECRs can ‘get inquisitive’ by conducting research on research that would facilitate and promote open research practices, such as addressing sensitivities of data sharing and identify evidence-based ways of balancing ethics with data transparency. Further research is also needed to explore, develop, implement and evaluate the impact of different initiatives and interventions, such as the Framework for Open and Reproducible Research Training (FORRT), Facilitate Open Science Training for European Research (FOSTER) (Pontika et al., 2015) and Principles and Practices of Open Research: Teaching, Research, Impact, and Learning (PaPOR TRaIL) (Egan et al., 2020), at different levels (e.g. undergraduate, postgraduate, postdoctoral) to help overcome the obstacles to open science faced by ECRs working in health research and across other disciplines and improve open science practices.

Strengths and weaknesses

This study has some limitations. It is important to note that our interviewees were recruited from participants of a two-day open science training workshop based in Ireland, and also volunteered to participate in interviews. This implies that our study sample is a sub-sample of the wider ECR target population who had a pre-existing interest in open science and may with, and who may also have been pre-equipped with awareness and deeper understanding of open science challenges and opportunities. Taking part in the workshop inevitably also influenced their knowledge about open science, which should be taken into consideration when interpreting the study findings. However, this also means that participants were in a position to provide substantial depth and richness of insight into an under-explored area of research. As such, these findings will provide useful comparison data for future similar studies or replications amongst other samples of ECRs. The rigour with which we conducted data collection and analysis also enhances the transparency and transferability of our findings.

Ironically, as a team of ECRs, challenges we encountered with making data from this study available mirror that of the study findings. Given the size and unique nature of our sample, de-identification of qualitative study transcripts was not deemed possible. However, as outlined in our methods section, we used NVIVO coding queries as advocated by Tsai et al. (2016) to provide a clear audit trail and facilitate transparency (Houghton et al., 2013). In addition, although de-identification of the quantitative data was deemed more easily achievable, the nature of our ethics approval meant that explicit consent for access to data beyond the research team had not been obtained. On reflection, this was deemed mostly due to a lack of forward planning for data sharing, with priority given to data confidentiality and participant protection. As such, our experience highlights another example of the need for better training and awareness for researchers in collaboration with other bodies such as ethics committees and data protection offices to facilitate appropriate data sharing.

Underlying data

Given the size and unique nature of our sample, de-identification of qualitative study transcripts was not deemed possible. As outlined in our methods section, we used NVIVO coding queries as advocated by Tsai et al. (2016) to provide a clear audit trail and permit the running of coding and matrix queries to facilitate transparency (Houghton et al., 2013) (see Extended data). The data cannot be shared via an alternative route of closed access, since ethics approval did not provide explicit consent for access to the data beyond the research team. For the quantitative data, the information sheet given to participants clearly stated that their data would not be shared by anyone other than the research team. Aggregated results are provided in Extended data.

Extended data

Open Science Framework: Open science study 2019; factors for practicing OS by ECRs ('Exploring factors that influence the practice of Open Science by early career health researchers: a mixed methods study'), https://doi.org/10.17605/OSF.IO/PKREN (Zecevic et al., 2020).

This project contains the following extended data:

Reporting guidelines

Open Science Framework: COREQ checklist (Appendix 1) for ‘Exploring factors that influence the practice of Open Science by early career health researchers: a mixed methods study’, https://doi.org/10.17605/OSF.IO/PKREN (Zecevic et al., 2020).

Data are available under the terms of the Creative Commons Zero "No rights reserved" data waiver (CC0 1.0 Public domain dedication).