In total, 59 individuals participated in the consultation (Netherlands n = 25, Spain n = 17, and Croatia n = 17), with many participating in multiple rounds. Figure 1 illustrates the location and number of participants per round. As mentioned above, whilst it was hoped that the same participants would join all rounds, there were some drop outs and, subsequently, new recruits were purposively selected (16 to round 2, 3 to round 3). Table 2 shows the participant characteristics per consultation round and across the consultation as a whole (for individual country breakdown, see supplementary Table S3). Participation was greatest from the Netherlands (43%), followed by Spain and Croatia (both 29%). The age category most frequently represented was those aged 30–39 (39%), and gender representation was fairly even (45% female). Almost all stakeholders had multiple roles and represented more than one stakeholder group. On aggregate, however, the largest group represented was researchers, followed by members of RE or RI committees, journal editors or assistant editors, research managers, policy makers, funders, and representatives from industry (Table 2). Researchers from different career stages were represented: 16% of participants were early career researchers (PhDs), 34% were mid-career (Post docs, Assistant and Associate Professors) and 14% were senior researchers (Professors and Department Heads).
The final themes represent the different levels at which RI support was experienced by participants in the three countries:
RI governance and institutional implementation
RI roles and structures
RI education and supervision
Infrastructure, technology and tools supporting daily practice
These themes are described below, with selected illustrative quotes in the text and additional illustrative quotes for each theme from each country provided in supplementary Table S2.
RI Governance and Institutional Implementation
The Netherlands, Spain and Croatia have national guidance and structures governing RI and are similarly subject to broader European codes and regulations, such as the Revised European Code of Conduct for RI (ESF-ALLEA, 2017) and the mandatory General Data Protection Regulation (GDPR) (https://gdpr-info.eu/). European level guidance was described as providing the impetus for positive changes in national guidance, particularly in Croatia and Spain, even if implementation within existing regulatory systems could require a degree of adaptation to specific contexts. There were however differences between countries in how participants discussed national RI codes. Diverse Dutch stakeholders, for instance, referred to the Netherlands Code of Conduct for RI (2004, updated 2018)(VSNU, 2018) with some anticipating the update of the code and the implications for their institution codes and practices.
[W]hat we do with the policy advisers of the academic medical centers is that we get the concept code now, and we see, well, what's different from the old one. And how does it affect the policies and codes we have in our own medical academic center, so it could be that the research code we have is going to be changed because of the new code.
P4, Female, Policy-maker, the Netherlands
In Croatia, participants discussed the Ethical Code of the Board of Ethics in Science and Higher Education (2006, updated 2015) (Agency for Science and Higher Education, 2015), however it was described as being adequately formulated but not sufficiently adhered to in research practice, often due to external social pressures.
[W]e have a code of conduct which was established in 2006, basically, so in Croatia it isn't a problem to have everything set up on the place, it's a problem with how this is working in reality. So that is the problem. When you have corruption, when you have politics which is going inside RI, inside the science system, and know that this is a problem in Croatia.
P34, Male, Researcher and RE or RI Committee Member, Croatia
Spanish participants discussed European guidance and national laws; however, the Spanish National Statement on Scientific Integrity (COSCE-CRUE-CSIC, 2015) was not mentioned at all. Participants from all countries described some researchers as lacking awareness of institutional RI codes however, and expressed frustrations at insufficient efforts to translate guidance into practice; indicating that, in the absence of commitment to implementation, institutional level RI codes and policies can actually create dissatisfaction. Participants described a need for sustained commitment from the highest ranks of the university, action plans for the implementation of guidance in research practice, and designated persons responsible for overseeing that implementation (e.g., research managers, research funding officers, RI officers).
You need the directors of the centers, or the manager or whatever, to actually be interested in this and say and “I'm going to, every year when I do the annual talk to my whole center, I'm going to touch on the issue of RI, and gender” [which was discussed as another cross-cutting issue], just to show that it is important.
P18, Female, Research Manager, Spain
Some participants called for more far-reaching regulation of the research process. A few Croatian participants even called for EU regulation enforcing mandatory research audits. Others however argued that, ideally, science should be self-regulating, and rejected the additional bureaucratic burden that comes with greater oversight.
RI Roles and Structures
Participants frequently discussed the roles and structures put in place in institutions to support RI. Important sub-themes were: (1) People with designated RI support roles and (2) Committees Supporting RI.
People with Designated RI Support Roles
In the Netherlands, many participants described well-defined structures and roles, including RE committees (for ethical aspects), scientific committees (for scientific quality), clinical research offices (for safety and quality of trials), RI committees (for misconduct allegations), RI counsellors (for confidential advice), quality commission (for random audits) and other experts (e.g. data protection officers, library staff, and RI teachers). Although not all roles and structures were available in all participants’ institutions, and the availability of support could differ depending on discipline and if research was required, by law, to pass through an accredited RE committee (i.e. interventions on animals or humans/personal data) or not. Despite quite clearly delineated roles and responsibilities, problems could however arise if advice was not consistent between different sources. For example, one researcher discussed her frustrations at the lack of consistency in the advice on data management given by the RE committee, the data protection officer, and the clinical research office.
We had a trial, and it was finished, so then of course, you let all the institutions know that you are finished, then you get information back what you should do now. So, the privacy officer, he said 'well, you should immediately destroy the connection between your persons and data. You should immediately destroy the connection.' And that surprised us. So, then we called our monitor from the Clinical Research Bureau, and they said, 'oh, we don't have an opinion, just do what the privacy officer says.' And then we thought, 'well, what would the medical ethical [RE] committee have to say?' And they said, 'no, you should save it for five years.' So then, yeah, well, what do you do?
P2, Female, Researcher, the Netherlands
The accessibility and availability of other experts for RI support was less elaborate in the other countries. In Spain, participants described research funding officers and research methodology advisors of the Research Unit and the Quality Unit as important sources of support. Croatian participants, in contrast, described particular difficulties finding confidential and independent RI advice during a project and often turned to more senior colleagues for advice. Croatian participants’ also experienced difficulties getting RI advice from a RE committee during a project, which is described further below.
Committees Supporting RI
Perhaps counterintuitively for some, RE committees were often discussed in relation to RI support; particularly in Spain and Croatia where they are responsible for promoting RI and dealing with research misconduct cases and also in the Netherlands for certain types of research.
In the Netherlands, participants from law, computer science, engineering and social sciences described the development of non-accredited ethics advisory committees at departmental, faculty or institutional level to provide advice on research not requiring ethical review by law, since no interventions on human subjects are involved. The advice of these committees has no legal status; their aim is to improve the quality of research and to strengthen departmental, faculty or institutional responsibility. In these committees, there was little separation in the organisation of RE and RI support, either on the level of the responsible committee or the people involved in providing advice. For example, a member of the faculty of law described the combined integrity and ethics roles of their advisory committee because they “were unable to split it up” (P43). Challenges for non-accredited committees included a lack of legal status of their advice, and the small volume of cases which created difficulties for committee members to develop expertise in providing advice.
In Spain, RE committees are responsible for the promotion of RI and in dealing with research misconduct cases. Furthermore, participants described accredited RE committees increasingly following-up on research after authorization. One RE committee member described auditing a project two years after study approval on research outcomes and financial issues.
P54: Assessing what was happening after our approval, we have learnt that the investigators, the researchers, forgot all the things that they are going to do […] it's one of the most important things for us that they have planned it well, but they are [also] doing well and they are publishing well.
I: So, you actually follow them through the whole process?
P54: We have begun two months ago only.
I: Okay, and what is your experience of that?
P54: Our experience is that like a wall, big high wall, they are, they have to learn because they have to explain that this public money has been well invested.
P54, Female, RE or RI Committee Member, Spain
In this example, the participant refers to research for which post authorization follow-up is not obligatory but “less than an order, but more than a suggestion” (P54) because the same committee would appraise protocols from the researchers in the future and because the committee tends to apply the same procedures for studies that are legally required to submit to a post authorisation audit (namely those involving pharmaceuticals or observation studies in healthcare settings) to the majority of research protocols approved.
In Croatia, RI is not differentiated from RE (indeed RI is difficult to translate in Croatian), and RI is considered a part of RE. RE structures, therefore, also serve RI functions, and RE committee members described being asked to give opinions on allegations of misconduct. However, due to a lack of authority and, sometimes, expertise, RE committee members expressed reluctance to do so. RE committee members also described having little power in such cases: their advice may or may not be followed by a faculty Dean.
P28: For instance, there was a case where some results were published in a bad journal and then sent to a good journal and it was claimed that the number of data was much much bigger, but the curves are completely the same. You cannot have the same average and the same statistical error; it was 10 times or 100 times bigger number of data. This was 100% proof that this was wrong. Nevertheless, other members of the committee did not want to prosecute or to make any strong decision, and the complaint came from the colleagues.
P28, Male, Researcher and RE or RI Committee Member, Croatia
Croatian researchers were deterred from seeking RI advice from a RE committee, however, because the committee must start a formal process if a person’s name is mentioned in relation to questionable research practices. This requirement was unique to Croatia, however participants from all countries described difficulties in seeking advice from RE committees after approvals were granted.
In Croatia and Spain, there are no formal RI committees. As mentioned above, RE committees deal with RI complaints. In contrast, in the Netherlands, RI committees deal solely with misconduct complaints. In all countries, the advice offered by committees in relation to RI breaches carries little legal authority. Some participants described concerns about the credibility of RI-related expertise of some members nominated to institutional or national committees and a need for education and support of their members.
Actually, it's my personal opinion about how we're dealing with this RI is quite critical to how we're dealing with cases of research misconduct and one of the reasons is that you let these peers deal with these cases but they have no actual skills to do that, they're just professors who voluntarily, or sort of less voluntarily, applied to do it and who do it on top of their job without any training, without any guidance, without any support actually.
P10, Female, RE or RI Committee Member, and Policy Maker, The Netherlands
Croatian participants described concerns about external pressures on RI investigations, and referred frequently to the Croatian parliament’s attempts to limit the remit of the National Committee for Ethics—which deals with alleged cases of misconduct—because it ruled unfavourably in the case of a former science minister accused of plagiarizing a part of his doctoral thesis (See P34 quote under the first theme). These pressures were not, in contrast, reported by participants from Spain or the Netherlands, or in the multi-country focus groups where the Croatian situation was discussed in some depth.
RI Education and Supervision
Participants from the three countries frequently emphasized the need for education to create RI awareness, adherence to RI in practice, and internalisation of RI ideals. Dutch participants however described more experiences with formal RI education, with some describing RI training as an obligatory part of PhD education and even, according to one participant, embedded in their university’s competency model.
At my university we made research integrity a learning line in our competence model, so every course has to think about what issues of research integrity do we want to address to our PhD students.
P10, Female, RE or RI Committee Member and Policy-maker, the Netherlands
In Spain and Croatia, some pioneer initiatives were described, which had received increasing interest from other institutions, sometimes in response to the demands of funders. These, however, were often voluntary and included in only specific courses. In all countries, RI education was described as being predominately targeted at PhD students, with the training of more senior researchers considered important but difficult to enforce. Due to senior researchers’ perceived aversion to RI training, some participants suggested making the training mandatory or integrating it into other continuous education courses. A few also suggested that RI education should be offered to all members of an institution’s staff, including project managers, IT support, and clinicians conducting research in academic hospitals, to promote an organisation-wide culture change.
In regard to the content of courses, Croatian participants expressed a greater need for training materials, more frequently referred to RI training as narrowly concerning ‘falsification, fabrication and plagiarism’, and criticised Croatian approaches compared to what they perceived as more ‘positive’ approaches in other countries.
What is typical in Croatia, is that we find a lot of regulations, and lot of consequences, so 'if you will plagiarize, then you will, you know, something will fall from the sky and kill you.' But actually, nothing is happening, and there are no actually educational materials, there are not guidelines, anything, but just some kind of rules and threats, that's all what we find. And we compared it to international university, and you can see almost immediately how they are trying really to educate, not to blame someone.
P36, Female, Researcher, RE or RI Committee Member, Croatia
In contrast, Dutch and Spanish participants who delivered RI training described using interactive approaches based on real cases of questionable research conduct that resonate with researchers’ day-to-day dilemmas.
Participants often, however, emphasized that education alone would not improve research practices, but needed to be implemented in conjunction with other measures. The emphasis on which other measures were most complementary depended very much on each participant’s personal preference; ranging from calls for multiple levels of support, to a focus on specific interventions (e.g. protocols close to research practice). One participant linked this back to institutions taking responsibility for RI rather than shifting that responsibility onto researchers.
We tend to put the blame on the individual and take it a long way from the responsibilities of universities. And I think when you approach them as, when you teach research integrity, ideally you want the right resources to prevent this at the university/institute level, that could prevent data manipulation, that could have to do with data storage.
P35, Male, Researcher, Croatia
A few participants stressed the need for more reflexive research practice, however, found it difficult to describe how that might be achieved as it would require a change in research culture rather than training.
An informal aspect of RI education that was frequently discussed was learning about research from more senior colleagues. Indeed, participants from all countries often described senior researchers as the most important influence on their own research practice. Although some Dutch participants described departmental initiatives to promote good role modelling and develop coaching skills, participants from all countries also frequently questioned the knowledge and behaviour of senior researchers and, as mentioned above, very few participants described obligatory RI training for more senior researchers.
In these sessions, senior, but also the juniors, are stimulated to share like, for instance, if you published an article, and then in hindsight you realized 'oh my god, I did one of the analyses in the wrong way, what should I do?' So, we really want to stimulate that the juniors are not keeping it to themselves, but share them, and then we can cope with it all together.
P2, Female, Researcher, the Netherlands
Infrastructure, Technology and Tools Supporting Daily Practice
Participants’ discussion of infrastructure, technology and tools focused on their potential to simplify and guide researchers through complex processes and/or to prevent questionable research practices through greater transparency. Because the focus groups took place around the 2018 reform of EU data protection laws, participants from all countries frequently discussed data management and often described a lack of adequate infrastructure, tools, and practical guidance on data storage and transfer as hindering compliance with the new law.
[w]e're struggling more with the infrastructure part. I don't know if it's in your organisation for a researcher to find his way to get advice on data management or advice on storage or computers or whatever, well that's really a challenge for us at this moment. And that's not only on data infrastructure but that's a big part of it. So that's maybe the support we're missing.
P15, Female, Policy Maker and Other, The Netherlands
Across the three countries, participants commented on diverse levels of awareness within teams and between disciplines. The diversity of data handled and difficulties understanding regulations and good practices related to specific types of data was a key challenge. Indeed, differences between disciplines were more noteworthy than differences between the three countries, a distinction which was also commented on directly by one participant working on a European project:
We were very aware about going through an ethics committee, about writing protocols, about data management. Comparing with other stakeholders who were maybe from countries that we, from a prejudice let's say, we could think they were more strict, from Northern Europe. And they were less because they were from another field […] and they were not so used to working with participants on things like that.
P50, Female, Researcher, Spain
Although participants overwhelmingly focused on data management when discussing infrastructure, technology and tools to support research practice, some other supportive tools were described. For instance, Open Science platforms for making data and code available, software to check for plagiarism or to check analyses, and authorship agreements. Some concerns were raised that such tools might be inadequate to solve complex RI problems. Authorship agreements, for example, were considered difficult to negotiate or enforce considering the hierarchical structures within teams. Open Science platforms were also criticised because, although data and code are more readily available, they are still rarely checked.
So, I think the first step is everything should be available, but the second step should be we should also check. I mean, this is sort of the premise of science, right? We check things by other people and then we say it's okay, then it's published. That’s kind of the premise, and we're not doing that with data, with programmes, with things. We say they're available, we've read the paper.
P13, Male, Researcher, RE or RI Committee Member, The Netherlands