After completing two coding sessions, two key themes were identified. The first, academic practice, in my interpretation, reflects the fact that all participants’ beliefs and values regarding research in general and reflexivity in particular stem from their roots in the academic system. Often toxic (in part, due to perverse incentive structures), the academic environment has shaped the way that these researchers think as well as the way they practice science. This key theme has several subthemes, linking both to questions asked directly of participants, as well as to spontaneously discussed concepts. The second key theme, the crisis, to my interpretation, demonstrates that this group of academics are marked by the recent (and to some, current) crisis of confidence. Every participant spoke in language or described scenarios and thoughts that reflect how their behavior has been influenced by the crisis. Understandably, as some questions directly mentioned the participants’ influences on their research, and their reflections about themselves and their work, discussions surrounding reflexivity and objectivity and bias led to two more major themes. The first of these surrounded practical aspects of reflexivity, and the link between reflexive practice and research quality. This key theme has one subtheme. The final key theme mostly related to objectivity in the research process; its pros and cons and whether it is practically feasible.
Direct quotes from participants are included at times, to support the text. They are in italics and anonymous.Footnote 2
Theme: academic practice
The most dominant overarching theme, academic practice, links answers to questions 1, 3, and 6 above. This theme was complex and featured three subthemes. The transcripts revealed the influence of the practical aspects of academia and its environment on how participants view their results, their freedom and ability to be reflexive, and the possibilities available to include reflections in published work. The academic environment evidently contributes heavily to participants’ feelings about divergent results.
Subtheme: good science practices and the impact on career
Although no questions led sources to discuss the tension they feel between wanting to conduct good science and advance their careers, all sources raised the issue spontaneously. Often the limitations mentioned were in relation to publication records and career progress. Many participants link reflexivity with the costs associated generally with doing ‘good science’. For some, reflexivity introduces internal conflict because it carries the risk of influencing others’ perception of them as professionals, or shows that they are “biased researchers”. Many sources emphasized that despite difficulties and pitfalls, reflexivity had the potential for being good for their careers and for preventing bad research practice:
“But I think that by now we know, and I hope that something that will come of this debate is that people will become more reflexive and we need to evaluate that because it actually allows them to make better decisions and at the same time, makes them accountable for these decisions.”[18]
One participant indirectly mused on the point raised by Finlay regarding the productive use of subjectivity (that is, creativity is productive):
“Maybe it has to do with creativity, like when you stop asking new questions, instead on autopilot asking variations on the same question, that I think is a large danger. I think that can be avoided by reflexivity...questioning that process.”[9]
Five sources also explicitly reflected on how the research environment in terms of grant funding impacted upon their work. While two sources felt as though they had the “luxury” of not writing grants (because they did not feel the need to or because it was not required), three others felt that the grant process limited their freedom to conduct the best research of which they were capable. Interestingly, many sources equated reflexivity with being critical of oneself, thinking critically about your research questions and being skeptical throughout one’s research practice:
“And I think if you were less intrinsically skeptical about your own data, one would have more easily not gone back another time to go over all the steps, and just accepted all the results. ”[6]
Two thirds of participants explicitly mentioned that reflexivity is difficult to apply in research practice, due to the academic environment not being receptive or welcoming. Some participants mentioned that reflexivity and awareness might get in the way of career progress:
“It’s morally good, but maybe not for my career. Maybe I’m getting outcompeted or something, or I don’t get tenure and these others will…”[14]
Subtheme: divergent results
The sample had varied views about how they feel when they get results that diverge from their expectations. One striking finding (12 of 20 participants) revealed a strong tendency for researchers to automatically assume that they have made a mistake in their experiment or analyses when faced with divergent results, or to at least be skeptical of them. It appears that participants become reflexive when things in their research do not go as planned. This suggests that participants’ reflexivity is linked with self-criticism. One person linked the automatic assumption of error to training:
“The way you’re trained to deal with that is the very first thing is that you ask: Is this the thing we should have expected, was our initial logic flawed?”[10]
In contrast, most participants (16 of 20) prioritize the information value of a study over receiving predicted results. One participant describes receiving a series of disappointing results in her PhD project leading to sleepless nights and fears of her whole thesis unravelling. She was coached toward positive, constructive thinking by her then-supervisor (and now collaborator), and provides the same constructive advice to supervisees now, emphasizing the learning opportunity divergent results presents. The more senior (tenured) interviewees tended to be less emotionally involved in their findings. It appears as though these participants have the freedom (perhaps due to higher job security than more junior colleagues) to focus on fully understanding the phenomenon, warts and all, without fears about null results:
“I am interested in how something works, I am not so much interested in showing that it works in a certain way…so very often it’s more fun if it doesn’t work out because that means that there is something more to it than you originally thought. So, the thing that you’re studying is richer. And in that sense, I don’t really care.”[6]
Three respondents took a hyper-rational approach: if you have a strong theory, a sensible question and a well-designed experiment, then “any result is interesting”.[17].
Subtheme: role of mentorship and communication with colleagues
Research in most fields is a matter of teamwork, a sentiment echoed by many participants at different points in the interviews. A number of participants described the defining role supervisors, past and present, had in shaping their scientific practices and beliefs about their role as a researcher. They spoke fondly of old mentors, and described how they had taught them to think positively about unexpected results and problems encountered during the research process. It gives the impression that a positive relationship with a mentor or supervisor can be protective against the toxicity that can be found in academic culture.
Other informants spoke about how talking about their ideas with colleagues can help them against falling prey to their own biases and questionable practices. Participants also mentioned the role that discussions with others can have in helping them be more reflexive and challenge beliefs they hold:
“…one reason why I like working with other people, they provide another perspective on questions, or on data analysis, and that will already sort of like allow you to get some of your own bias out...”[6]
“I think as a researcher you're still exposing yourself to other talks, other beliefs so to say… other interesting people on conferences but also within the department, usually you can pitch your work and they all say what they think. That's one thing I really like is that community and the fact that you do get challenged.”[4]
Theme: crisis
As expected, based on the selection of the participants, evidence of the influence of the crisis ran through most participants’ narratives. Apparently, the crisis has impacted upon the respondents to quite an extent. This resulted in the crisis being a key theme itself. Much sentiment about the crisis was negative, or at least skeptical, especially about practical aspects of new practices (such as open data policies, transparency, and preregistration). The perception that new ‘open science’ practices have become moralized is apparent. One participant related a story about how he has recently started, in response to the crisis and subsequent push by crisis activists, to include honest reflections in his manuscripts. He fears negative repercussions, however, assuming he will be “punished” for it. Some participants expressed other fears about being transparent, feeling as though opening their research up would invite (undue) criticism, or attacks. These considerations seemed to relate to bullying that has occurred by self-appointed ‘police’ who have been active on social media in shaming those who have perpetrated (or are suspected to have perpetrated) QRP in their research.
Other sources were more positive about the crisis and incentives that have come from it. One participant said that although there is resistance toward the new methodology and ideas relating to the crisis, he is positive about them because it means that he can talk to his undergraduate students about problems in research with the silver lining that there is “improvement”. Many sources directly linked the crisis with changes in their research practice for the better. One credits it with having shaped him in terms of how he reports and reflects on himself as a researcher and his daily practice in that role.
It is evident from the participants’ accounts that reflexivity and transparency (‘popularized’ as a result of the crisis of confidence) are linked. Not only did participants tend toward using the words interchangeably, they tended toward conflating the concepts in their explanations. They also frequently associated the use of protocols like preregistration with transparent and reflexive practice. Most participants made references to the crisis in some way. Some directly by name, others by referring to QRP and preregistration/registered reporting.
Theme: reflexivity
This theme links closely to questions 5 and 6 and contains two subthemes. The sources expressed a range of opinions toward using reflexivity in practice. Reflexivity, according to the sources, is difficult, unwelcome and undervalued by peers and institutions, but nevertheless necessary for good and valid science.
Subtheme: Reflexivity and research quality
Sixteen participants emphasized an obvious link between reflexivity in research and higher quality science. For most of these participants, this link was made somewhat automatically, or at least without obvious hesitation. Two participants spontaneously made a further linkage between reflexivity and higher research quality on the one hand, and the crisis of confidence, suggesting that the crisis might be mitigated by more reflexivity. Several interviewees linked the practice of reflexivity with greater accountability, modesty and a better chance at avoiding slipping into questionable practices. Many participants were explicit about how reflexivity could be used as a means to decrease one’s biases by bringing them to awareness, and therefore reduce the negative impact of these biases on their research.
Positivity was tempered in many peoples’ accounts by limitations relating to the current academic environment. Several sources acknowledged that although reflexivity should be a key part of research, it might not be advisable for some people to include their reflections in publications because it is difficult:
“Everyone should know about it, but not everyone should write about it, because it’s hard. It should be main reading material for scientists, but I don’t know that a reflection in a paper is necessarily useful or, actually, even possible for some people.”[14]
And one described the ever-present issue of balancing between being part of your research and wanting to maintain a distance from your work:
“It's kind of like after a long time you've been doing this research for so long it's hard to know where you end and the truth begins right because your beliefs and the truth and your research is so interwoven. I find that a little bit scary because I like to think that I can kind of retain a certain element of logic and a certain element of being able to look at myself, but at the same time that gets harder and harder the more embedded in what you do.”[4]
Subtheme: sharing reflections as part of published work
Most sources were generally positive about the prospect of sharing their reflections and decision-making processes alongside their published work. Some felt it provided context to readers of their academic work:
“It’s important for other people to be able to evaluate the strength of evidence in light of how much they believe in it in the first place because that might have influenced the results.”[13]
Most participants mentioned the practical value of jotting down reflections and decisions during the research process. They reported that logging their reflections in some explicit form (such as a logbook), would save time for either themselves in the future, or for others conducting research in similar paradigms. A majority of interviewees noted that reflexivity in practice involves others in some ways. Three people recognised that difficulties with including reflections in written output may be more to do with their peers than with them - people are not open enough or reflexive enough themselves to appreciate such characteristics in others. Some participants saw value in logging their research as a way of maintaining accountability, for themselves, and between themselves and others in supervisory relationships.
“It just amazed me how much time we spent, and how little we knew about the other processes. We could have saved a few months of work if other people had written up the things that went wrong.”[4]
Several sources described an ideal/practice dichotomy in terms of reflexivity. They indicated that although reflexivity is a nice ideal, its practical application to research is limited. Reflexivity is difficult in practice because it is hard to be aware of what you’re unaware of. It is difficult because it takes time and care to cultivate a good sense of self-awareness, and it is confronting and uncomfortable at times.
Theme: objectivity is complex in practice
This theme also strongly reflected the questions asked (2, 3, 4, and 5). Opinions varied about how and where in the research process bias was most evident and damaging. Three sources suggested that bias influenced the process at every point along the way, while two others implied that bias enters the process during data interpretation. A common opinion, however, was that ideas held a priori about the research question would undoubtedly affect the research process. Another dominant opinion was that bias existing at any point had the potential to affect the whole process, as the research process is circular, and ultimately feeds back into itself:
“Ideally you get the evidence and you get your conclusion. Other times you get your conclusion and you come up with the logic of the evidence… It has to be kind of like that because the process of understanding is sort of circular.”[1]
It appears as though most participants have a complex relationship with objectivity in the research process: while it is probably good to strive for objectivity in the research process, it is difficult to achieve, if even possible. No consensus existed among the sources as to whether or not objectivity was important to improving the research process, or whether or not researchers are even capable of it (though most believed it was a bit of a pipe dream). It was the case that those sources that thought objectivity was key to good science also believed that objectivity was possible. It may be that because most participants were taught in empirical traditions where objectivity is prized, they feel a pull toward thinking of objectivity positively despite skepticism about whether it is possible. Most interviewees were fairly nuanced in their views: our research is always driven by our own values and beliefs, which is acceptable so long as we are transparent and reflexive about our role as researchers. One participant simply sees objectivity in a somewhat negative light:
“I think objectivity is overvalued in science. I think we should be more open about how subjective we are rather than striving for objectivity.”[5]