Experimenter as automaton; experimenter as human: exploring the position of the researcher in scientific research

Abstract

The crisis of confidence in the social sciences has many corollaries which impact our research practices. One of these is a push towards maximal and mechanical objectivity in quantitative research. This stance is reinforced by major journals and academic institutions that subtly yet certainly link objectivity with integrity and rigor. The converse implication of this may be an association between subjectivity and low quality. Subjectivity is one of qualitative methodology’s best assets, however. In qualitative methodology, that subjectivity is often given voice through reflexivity. It is used to better understand our own role within the research process, and is a means through which the researcher may oversee how they influence their research. Given that the actions of researchers have led to the poor reproducibility characterising the crisis of confidence, it is worthwhile to consider whether reflexivity can help improve the validity of research findings in quantitative psychology. In this report, we describe a combination approach of research: the data of a series of interviews helps us elucidate the link between reflexive practice and quality of research, through the eyes of practicing academics. Through our exploration of the position of the researcher in their research, we shed light on how the reflections of the researcher can impact the quality of their research findings, in the context of the current crisis of confidence. The validity of these findings is tempered, however, by limitations to the sample, and we advise caution on the part of our audience in their reading of our conclusions.

In my mind’s eye, I round the corner of Stanford University’s Cubberley Auditorium. Built circa 1938, the huge sandstone building is surrounded by neatly mown lawns and hedges. For the Metascience 2019 conference, small tables draped in garnet-coloured cloth populate the manicured grass, alongside the lush shrubbery. I am out of breath, due to running late for the opening plenary. By the time my path takes me into the auditorium itself, I am somewhat in awe of the opulence of the setting. Perhaps I am even mildly put off. “People from low-status backgrounds and first-generation academics may feel rather out of place here,” I think to myself.

I (SMF) come back to reality: I am sitting at my desk in my office. For the first time since returning from the conference more than a month ago, I realise that I am one of those people that felt out of place at Stanford. Although I wouldn’t say I came from a poor background, I was not brought up in a wealthy family. My father and an uncle are university lecturers, so I’m not a first-generation academic. Nevertheless, I felt out of place. As a doctoral student involved in a qualitative research project, I try to make a daily practice of self-reflection. To recognise that I missed such an important point is frustrating. I comfort myself that at least I have made the realisation in time to use it to contextualise my observations of the conference after having recorded the field notes.

The researcher’s background and position have great potential to affect all aspects of the research process, from what they choose to investigate to how they communicate the findings to the world (Malterud 2001). For that reason, reflexivity - continual, systematic and active introspection on the part of the researcher - is a crucial component of the academic’s toolbox. Reflexivity is usually linked to qualitative research approaches. However, interviews I conducted with a number of primarily quantitative academics reveal that in their eyes, reflexivity can lead to higher quality research, irrespective of the methods being applied. The sources help describe the practical implications of practicing reflexivity in scientific domains in general. I asked the participants to consider how one can incorporate self-reflection into the research product itself, and briefly highlight some concrete options. In this paper we (SMF and MD) analyze the interviewees’ answers and build on them to present our own arguments for the incorporation of reflexivity in quantitative research.

Mechanical objectivity: a crisis corollary

The most recent crisis of confidence in the social sciences has become something of a ubiquitous topic since it began gaining momentum in the early 2000s (Pashler and Wagenmakers 2012). The crisis was ushered in by a series of cases of fraud and the failed replications of several key findings in social psychology, arguably reaching ‘critical mass’ in 2011 (Wagenmakers 2012). Despite the focus on social psychology, the crisis has implications for all of the social sciences, and the debate has spread to include other disciplines (e.g., biomedicine and physics, Gorski 2016; Molteni 2017). Now, several years later, discussions of the topic and its resolution continue with little indication that the psychological research community’s interest is waning.

One notable by-product of the crisis is a heavy emphasis on the reproducibility of research findings (Yaffe 2015). Reproducibility is thought to provide an indication that results are objective and reliable, free of bias and not due to chance (Resnik and Shamoo 2017). Such a research climate implies that objectivity in research is synonymous with concepts of validity, reliability and quality (Kayes and McPherson 2010). While perhaps not going so far as to ban subjectivity (as in the beginning of the twentieth century; Dehue 2001, and see also Gigerenzer et al. 1990 in Flis and van Eck 2018), the field has generally interpreted reproducibility as requiring a maximally objective, tightly controlled approach. Certainly, the mechanical, procedural approaches of many psychological science ‘reformers’ are used as ways to drive out the demon of subjectivity in a bid to resolve the crisis of confidence (see e.g., Morawski 2020). This interpretation has led to the association of ‘good research’ with highly objective research, and the dominant attitude appears to be that subjective forms of research are associated with low quality.

Tools are being proposed to bolster objectivity in science, including preregistration and registered reporting. These protocols are quickly gaining popularity in a post-crisis academia. Preregistration refers to a process in which scientists articulate their plans for a study before they collect the data. In the case of registered reports, authors submit their methodology plan for peer review before data collection. These protocols are meant to decontaminate the research process by getting rid of human biases (such as hindsight and confirmation bias; Chambers 2013), and restrict researcher degrees of freedom, and are explicitly linked with increasing objectivity in the research process: “Scientists value objectivity and transparency in the practice of science, however, we are rewarded at nearly every step for making results both cleaner and more surprising. Preregistration helps keep our motivated reasoning in check when analyzing data and presenting our results.” (Mellor 2016).

The “established orientation toward objectivity”, as Gough and Madill call it (2012, p. 374) is strongly evident even at institutional levels in the academic sphere. For instance, the guidelines for submission to journals under the Nature publishing banner reflect the position with regard to declaring conflicts of interest for publications. Submitting authors are presented the following: “For the purposes of this policy, competing interests are defined as financial and non-financial interests that could directly undermine, or be perceived to undermine the objectivity, integrity and value of a publication, through a potential influence on the judgments and actions of authors with regard to objective data presentation, analysis and interpretation.” The Journal of Experimental Psychology: General (JEP: General) disclosure form for submitting authors reflects a similar position: “In psychology, as in other scientific disciplines, professional communications are presumed to be based on objective interpretations of evidence and unbiased interpretations of fact.”

A newer journal, Royal Society Open Science (RSOS) makes a similarly interesting warning to potential authors and peer-reviewers about ‘unconscious bias’. Naturally, financial conflicts of interest should be taken as serious threats to the quality of research. Consider, for example, the fraudulent research of Andrew Wakefield into the link between autism and childhood vaccinations which was funded by lawyers leading legal action against vaccination companies (Rao and Andrade 2011). Some non-financial conflicts are also problematic: for example, consider questionable research practices that are fueled by a personal belief in a phenomenon (John et al. 2012). It is, undoubtedly, also true that biased decisions made in the context of peer review can be damaging to a healthy and diverse literature body. However, it gives me pause to read that three of the most prestigious academic publishers in science are making blanket statements about subjectivity, in a sense, somewhat equating it with poor quality in the context of research results.

Of course, this stance is not unique to Nature and JEP: General- other organizations hold similar positions. Columbia University’s set of courses on research ethics state that “... objectivity is the sine qua non of scientific discovery.” The National Institutes of Health are also clear on the matter: “Objectivity of researchers is an essential value in scientific research and the basis for public trust [in the research results].” The common assumption is that all personal influence¹ in research is negative, and that the personal influence researchers have on their work is incompatible with its quality.

Tension between the ideal and practice of objectivity

There certainly is value in adding mechanisms to the research process that provide a way to clearly signpost distinctions between prediction from postdiction, and help steer researchers away from questionable research practices. However, the wholesale adoption of the kind of mechanical objectivity (Daston and Galison 1992) that dominates the natural sciences seems ill-advised. In her discussion of scientific objectivity Montuschi (2014) argues that, even if the natural sciences do indeed follow this ideal of objectivity in practice, which seems doubtful, it does not seem to fit the social sciences. She distinguishes three demands of objectivity: science must only concern itself with real facts, science must be value free, and it must only use methods that produce true results. Each of these demands, Montuschi argues, is problematic in the social sciences.

Firstly, many of the facts that the social sciences are concerned with have a different ontological status than those of the natural sciences. They are not natural facts, but constructions, products of the discourse and actions of people, including, importantly, social scientists. As Hacking (1995, 2006) put it, the phenomena of the social sciences are interactive kinds, characterized by a looping effect between classification and study and the people being so classified and studied. Secondly, as Weber (2017) argued in 1904, the social sciences are inherently value-laden, with regard to both its results and the means by which they are produced. What we classify, how we classify it, how we count the elements of that classification, are all value-laden choices. Thirdly, Montuschi continues, it is doubtful whether quantification, often taken as the only objective way to produce true results, by itself guarantees an objective representation of the phenomena that social science is concerned with. Again, classification depends on value-laden choices, and moreover quantitative methods such as the experiment or the survey necessarily limit the reality that can be studied. They cannot study reality ‘as it is’, but only as it fits their constraints.

All in all, it therefore seems unwise for the social sciences to strive for the same kind of objectivity as in the natural sciences, and a different attitude towards and way of dealing with subjectivity should be sought. We believe reflexivity, rather than an exclusive reliance on standards and protocols, should be central to the methodology of the social sciences, not as navel-gazing but as a means of making subjectivity transparent and creating shared, intersubjective epistemic attitudes and emotions. We return to argue these points in the discussion, in the light of our data.

Reflexivity

Research into human beings – their behavior, emotions and cognition – brings with it the issue of psychological reality and the observer being somewhat enmeshed. As historian of psychology Richards (2002) put it, the discipline psychology and its subject matter, psychology, are linked in a reflexive circuit, and the same is true of other human sciences. Though the link cannot be broken, awareness of the condition through reflexivity is possible. Reflexivity is the process by which the researcher continually and explicitly engages in self-awareness and analysis of personal influences on the research process (Finlay 2002a). Reflexivity on the part of the researcher allows them to question and adapt their interpretations, based on issues that arise during the study. It lends credibility and realism to the conclusions reported (Clancy 2013): through reflexivity, a researcher is able to produce a faithful account of the research (Hertz 1997). Pillow (2010) describes reflexivity as a way to “legitimize, validate, and question research practices and representations”, and to call into question our data and methods. It assists us in understanding the social world, and also provides insight into how that knowledge is constructed.

Although the unintentional incorporation of subjectivity into the research process can be seen as a barrier to good scientific practice, reflexivity has the potential to facilitate and even enhance research quality. Finlay (2002a) presents it as an ‘opportunity’ rather than a ‘problem’; that is, subjectivity can be used productively. It can function as a mode through which the researcher can become more aware of their influence on the research. It can provide a means to treat the influence accordingly, whether that means to isolate it, if the quality of the research is being threatened, or to work it into the research: reflexive practice can impart a richness to the data and their interpretation which is valuable in furnishing our understanding of human thought and behavior processes: the goal of scientific inquiry (Bourdieu and Wacquant 1992). For those researchers who consider their subjectivity a threat to the quality of their research, using qualitative methods involving reflexivity may be worthy of consideration. Almost as a remedy to the potential problem subjectivity can cause, Crang and Cook (2007) recommend for researchers to recognize their “partial and situated” subjectivity. They recommend to ‘tap into’ it as a resource; to use it for achieving a deeper understanding of the phenomena under study.

Those who engage in qualitative research typically consider the researcher as the linchpin of the entire research process, and create and maintain awareness of their presence in the research process. Reflexivity gives this presence a practical application, and it is woven into the fabric of the study and can be included in the written delivery of the conclusions to the scientific community. To practice reflexivity, the researcher attempts to keep one eye trained on himself, so to speak: he remains aware of his own feelings, thoughts and expectations as he engages in the process of research. In the eyes of some, when employed in this way, reflexive practice can ultimately serve to impart objectivity to the research. In practice, this requires researchers to critically assess their role as the measurement instrument (Denzin and Lincoln 2003), as well as the person conducting the study, and attempt to understand how their dual-function can impact their conclusions.

This is in stark contrast to the objective approach to empirical research, which casts the researcher as, ideally, nearly non-existent (Simon 2011); as the automaton that begins the experiment, records the responses and produces the output. Quantitative researchers attempt to circumvent their biases by way of methodological and statistical choices (Finlay 2002a; Hammersley and Atkinson 1995). They strive to sterilize the research setting. Regardless of the role given to the researcher, and the way in which that role plays out, it is plausible that the behavior and beliefs of the researcher will influence the research process at each stage, ultimately influencing the study’s conclusions (Finlay 2002b).

The interview

The interview is a method commonly chosen by qualitative researchers to develop a detailed and rich understanding of a given topic by drawing narratives out of informants. We have chosen this method for this purpose also, and, as such, one of the researcher’s roles in this study was as an interviewer. Although there is debate as to the merit of the interview as a communicative event (Briggs 1986), and it does have inherent issues that may threaten scientific validity, it can nevertheless be used as a vehicle through which one might understand the whole ‘picture’ of the phenomenon it seeks to explore. Bourdieu (1996) discusses the problems associated with the interview, exploring the ways in which factors such as symbolic violence (a show of power when a relationship is unbalanced, such as in the case of the researcher-participant relationship) may distort findings. On the other hand, for Bourdieu the interview is an opportunity to implement change in society. In the way a midwife assists birth, the interview brings participant voices to the fore, while the researcher’s self-awareness lends legitimacy to the research. We also consider the interview valuable from Ricoeur’s hermeneutics of faith perspective (1970): the interviewee is an expert in their own right– an expert on their own experiences and context, and the interview is a viewing window which looks in upon those experiences and that individual’s narrative. The interview can also be valuable to participants, enhancing and furnishing their appreciation of the issues at hand, and their situation within the discussion. An interview is always potentially an intervention in the process under study.

Aims and guiding questions

We have several aims in conducting this research. First, broadly, we seek to better understand how researchers think about reflexivity, particularly in the light of the crisis of confidence. We make the tacit assumption that many researchers reflect on themselves in their role as researcher, and aim to capture the nature of those reflections and better understand how the practice of reflexivity impacts research products. Practically speaking, we intend to prompt further reflection in participants, as well as in potential readers of our work. We hope to address these abstract aims through our use of the interview.

Concrete questions guide what we ask participants in the interviews. First, we want to probe others to consider a question best articulated by Patai (1994): Does all this self-reflexivity produce better research? We also want to know how participants might perceive objectivity in relation to their research practice, as well as to them as the researcher. To that end, we ask how participants perceive the concept of objectivity in terms of its impact on their own scientific practice. Is objectivity necessary for good research practice? Lastly, we aim to understand how participants feel about the idea of tying in reflections to their research output: should a discussion of reflexivity be explicitly included in the published research? These interviews were conducted in a period of time in which the scientific community is still reacting to and actively debating issues in connection with the crisis of confidence. We expect that the crisis will heavily influence the thoughts and practices of the participants, and that many will frame their answers with this context in place.

The link between quantitative research practice and reflexivity is rarely explicitly made in the literature, to the best of our knowledge. Ultimately, we contend that reflexivity is of practical use to all researchers, independent of whether their approach is qualitative or quantitative in nature. We seek to furnish this argument with the reflections of quantitative researchers.

Method

Sample characteristics

I (SMF) interviewed twenty practicing academics from the University of Groningen in 2017 and 2018. People who were currently active in academia, had conducted their own research at some point, and had an interest in metascience, research transparency and integrity were suitable candidates for recruitment. I chose such a target sample because I assumed people who had these interests would have heard about the crisis of confidence, and would have some opinions about its impact on their scientific practices and the way they viewed their role as the researcher. Most participants weren’t heavily or even directly involved in the open science community at the time they were interviewed, but they were interested in talking about good scientific practice (based on what I knew of their research interests, research activity and personal values) and would have been somewhat familiar with discussion surrounding the crisis of confidence. I made contact with some participants through existing personal relationships (e.g., through friendships established in Groningen, or recommendations from friends; sources 1, 3, 10, 16). Others I came into contact with were lecturers or other students in courses I attended during my education at the RuG, who had expressed views that aligned with the interests I mentioned above (metascience, research transparency and integrity; sources 6, 7, 8, 10, 13, 18, 19). I located the remaining participants through searches online on either Google (with search terms like “metascience” and “University of Groningen”), or the RuG staff database, where I searched for the term “metascience”.

Participants were recruited partly for convenience, but I also kept an eye for diversity during recruitment. As a result, the sample is varied in terms of the participants’ positions in academia as well as their disciplines, and contained male and female sources in equal measure. The following departments were represented in the sample: psychology (social, experimental, developmental, clinical, organizational and educational subdisciplines), sociology, psychometrics, statistics, youth studies, psychiatry, clinical psychology, theory and history, and economics and business. The sample was diverse in terms of the positions they occupied in the academic hierarchy: I interviewed full professors, associate and assistant professors, and postdoctoral and PhD researchers. The sample is primarily European: sources represented the Netherlands, Germany, Greece, Malta, but also the United States.

Interviews

Participants were asked to reserve an hour of their time for the interview and all but two interviews concluded within 60 min. The average interview lasted a total of 46 min. The interviews were conducted at locations mutually agreed upon by myself and the source, at the University of Groningen Behavioral and Social Sciences faculty building. Often this was a private setting (in the offices of the higher-ranking faculty members, or in reserved meeting rooms), but a minority of the interviews took place in public places (one in the university cafeteria, and one in a shared university study space).

The interviews were semi-structured, and included 6 pre-determined questions:

• 1. When you conduct your research, how do you feel when your findings do not support your hypotheses?

• 2. To what extent do you think your personal opinions about a subject sometimes affect your research?

• 3. Do you think your research affects you? If so, how?

• 4. Do you attempt to attenuate these effects? If so, how?

• 5. Do you believe that being more aware of the bias you naturally bring to your research might cause that research to be of higher quality?

• 6. Do you believe that there is a legitimate way in which you could tie those reflections into your publications?

As an ice-breaker exercise I asked each participant at the beginning of each interview to talk about their field and why they chose to do research in that discipline.

I recorded the audio for each interview. Transcripts of each interview were made from the audio files, and a qualitative thematic analysis was conducted on the resulting text. The process followed was based on Mayring’s (2000) steps for inductive category development, which include the inductive development of thematic categories from the textual material and the revision of those categories. I transcribed blocks of transcript text (sentences, or small paragraphs of two or three sentences) and assigned codes based on key questions from the guiding interview questions, or based on themes that seemed salient in the texts. Sometimes the same unit of text was assigned more than one code. At the point of thematic saturation, a thematic framework for analysis was established (Attride-Stirling 2001). I took notes at each interview regarding my observations, impressions and thoughts, and referred back to them when analyzing the data in order to ensure that the interpretations were based on all information available. I reviewed the transcripts and code set twice after initial coding to maximize internal consistency.

Limitations

While measures were taken to optimize the validity and reliability of the findings in this report, some limitations should be noted. Three of the four limitations raised in this section seriously impact the validity of the study. First, the study was based on a convenience sample, which introduces the methodological issues of selection bias and a narrow selection pool. While this was not considered problematic to the extent where it would invalidate the study (as statistical inference was not drawn from the data), we emphasize that it has undermined the generalizability of our results. Second, this study only contains data from 20 interviews. The majority of participants are Dutch, and 19 of the 20 are from European nations. This limits the external validity of the insights we share. Since the issues raised in this paper are potentially applicable to all scientific disciplines, it is necessary to collect data from a wider and more diverse sample and see whether the same emphasis on some topics is observed.

A third limitation concerns the rapport I (SMF) had with participants. As aforementioned, I have had contact with these participants to a variety of degrees. Some I had known for a few years before interviewing them, and some I had not spoken to in person before the interview at all. Naturally, these differences would have influenced the dynamics of each interview and has likely undermined the standardization I attempted to achieve. I say more in the interviews with people I knew in advance of the interview, in comparison with the interviews with people I had not spoken to before. Similarly, I say more in the interviews I conducted with people with whom I had a natural chemistry. As a result, I might have led those discussions more than I would have liked. I identified that this ‘affected’ 7 of the 20 interviews.

I was aware of this as it occurred, however, and did attempt to attenuate it (for instance, by suppressing the urge to ‘join in’ the discussion when participants were making points I agreed with). Moreover, the interviews are not quantitatively different: durations varied widely across the sample, and not seemingly in a manner linked to my familiarity with the participant. The average interview duration for familiar participants is almost equal to the average duration for unfamiliar participants (48 min versus 46 min, respectively). It is unclear whether a qualitative difference in the interviews exists, however. Ideally, for methodological soundness, another sampling technique such as snowballing (where subjects recruit participants from among their acquaintances) would have been used in the study, to have a similar degree of separation between each participant and SMF.

A final concern is that the interviews were all conducted in English, while only one participant was a native English speaker. It is important to flag this, given that textual analysis conclusions hinge on the word choices of participants. Nevertheless, it is unlikely to have strongly impacted the findings: for one, sentences (rather than words) are our smallest unit of textual analysis. Another consideration is that English is the universal scientific language, and each participant had a strong command of English.

Results

After completing two coding sessions, two key themes were identified. The first, academic practice, in my interpretation, reflects the fact that all participants’ beliefs and values regarding research in general and reflexivity in particular stem from their roots in the academic system. Often toxic (in part, due to perverse incentive structures), the academic environment has shaped the way that these researchers think as well as the way they practice science. This key theme has several subthemes, linking both to questions asked directly of participants, as well as to spontaneously discussed concepts. The second key theme, the crisis, to my interpretation, demonstrates that this group of academics are marked by the recent (and to some, current) crisis of confidence. Every participant spoke in language or described scenarios and thoughts that reflect how their behavior has been influenced by the crisis. Understandably, as some questions directly mentioned the participants’ influences on their research, and their reflections about themselves and their work, discussions surrounding reflexivity and objectivity and bias led to two more major themes. The first of these surrounded practical aspects of reflexivity, and the link between reflexive practice and research quality. This key theme has one subtheme. The final key theme mostly related to objectivity in the research process; its pros and cons and whether it is practically feasible.

Direct quotes from participants are included at times, to support the text. They are in italics and anonymous.²

Theme: academic practice

The most dominant overarching theme, academic practice, links answers to questions 1, 3, and 6 above. This theme was complex and featured three subthemes. The transcripts revealed the influence of the practical aspects of academia and its environment on how participants view their results, their freedom and ability to be reflexive, and the possibilities available to include reflections in published work. The academic environment evidently contributes heavily to participants’ feelings about divergent results.

Subtheme: good science practices and the impact on career

Although no questions led sources to discuss the tension they feel between wanting to conduct good science and advance their careers, all sources raised the issue spontaneously. Often the limitations mentioned were in relation to publication records and career progress. Many participants link reflexivity with the costs associated generally with doing ‘good science’. For some, reflexivity introduces internal conflict because it carries the risk of influencing others’ perception of them as professionals, or shows that they are “biased researchers”. Many sources emphasized that despite difficulties and pitfalls, reflexivity had the potential for being good for their careers and for preventing bad research practice:

“But I think that by now we know, and I hope that something that will come of this debate is that people will become more reflexive and we need to evaluate that because it actually allows them to make better decisions and at the same time, makes them accountable for these decisions.”[18]

One participant indirectly mused on the point raised by Finlay regarding the productive use of subjectivity (that is, creativity is productive):

“Maybe it has to do with creativity, like when you stop asking new questions, instead on autopilot asking variations on the same question, that I think is a large danger. I think that can be avoided by reflexivity...questioning that process.”[9]

Five sources also explicitly reflected on how the research environment in terms of grant funding impacted upon their work. While two sources felt as though they had the “luxury” of not writing grants (because they did not feel the need to or because it was not required), three others felt that the grant process limited their freedom to conduct the best research of which they were capable. Interestingly, many sources equated reflexivity with being critical of oneself, thinking critically about your research questions and being skeptical throughout one’s research practice:

“And I think if you were less intrinsically skeptical about your own data, one would have more easily not gone back another time to go over all the steps, and just accepted all the results. ”[6]

Two thirds of participants explicitly mentioned that reflexivity is difficult to apply in research practice, due to the academic environment not being receptive or welcoming. Some participants mentioned that reflexivity and awareness might get in the way of career progress:

“It’s morally good, but maybe not for my career. Maybe I’m getting outcompeted or something, or I don’t get tenure and these others will…”[14]

Subtheme: divergent results

The sample had varied views about how they feel when they get results that diverge from their expectations. One striking finding (12 of 20 participants) revealed a strong tendency for researchers to automatically assume that they have made a mistake in their experiment or analyses when faced with divergent results, or to at least be skeptical of them. It appears that participants become reflexive when things in their research do not go as planned. This suggests that participants’ reflexivity is linked with self-criticism. One person linked the automatic assumption of error to training:

“The way you’re trained to deal with that is the very first thing is that you ask: Is this the thing we should have expected, was our initial logic flawed?”[10]

In contrast, most participants (16 of 20) prioritize the information value of a study over receiving predicted results. One participant describes receiving a series of disappointing results in her PhD project leading to sleepless nights and fears of her whole thesis unravelling. She was coached toward positive, constructive thinking by her then-supervisor (and now collaborator), and provides the same constructive advice to supervisees now, emphasizing the learning opportunity divergent results presents. The more senior (tenured) interviewees tended to be less emotionally involved in their findings. It appears as though these participants have the freedom (perhaps due to higher job security than more junior colleagues) to focus on fully understanding the phenomenon, warts and all, without fears about null results:

“I am interested in how something works, I am not so much interested in showing that it works in a certain way…so very often it’s more fun if it doesn’t work out because that means that there is something more to it than you originally thought. So, the thing that you’re studying is richer. And in that sense, I don’t really care.”[6]

Three respondents took a hyper-rational approach: if you have a strong theory, a sensible question and a well-designed experiment, then “any result is interesting”.[17].

Subtheme: role of mentorship and communication with colleagues

Research in most fields is a matter of teamwork, a sentiment echoed by many participants at different points in the interviews. A number of participants described the defining role supervisors, past and present, had in shaping their scientific practices and beliefs about their role as a researcher. They spoke fondly of old mentors, and described how they had taught them to think positively about unexpected results and problems encountered during the research process. It gives the impression that a positive relationship with a mentor or supervisor can be protective against the toxicity that can be found in academic culture.

Other informants spoke about how talking about their ideas with colleagues can help them against falling prey to their own biases and questionable practices. Participants also mentioned the role that discussions with others can have in helping them be more reflexive and challenge beliefs they hold:

“…one reason why I like working with other people, they provide another perspective on questions, or on data analysis, and that will already sort of like allow you to get some of your own bias out...”[6]

“I think as a researcher you're still exposing yourself to other talks, other beliefs so to say… other interesting people on conferences but also within the department, usually you can pitch your work and they all say what they think. That's one thing I really like is that community and the fact that you do get challenged.”[4]

Theme: crisis

As expected, based on the selection of the participants, evidence of the influence of the crisis ran through most participants’ narratives. Apparently, the crisis has impacted upon the respondents to quite an extent. This resulted in the crisis being a key theme itself. Much sentiment about the crisis was negative, or at least skeptical, especially about practical aspects of new practices (such as open data policies, transparency, and preregistration). The perception that new ‘open science’ practices have become moralized is apparent. One participant related a story about how he has recently started, in response to the crisis and subsequent push by crisis activists, to include honest reflections in his manuscripts. He fears negative repercussions, however, assuming he will be “punished” for it. Some participants expressed other fears about being transparent, feeling as though opening their research up would invite (undue) criticism, or attacks. These considerations seemed to relate to bullying that has occurred by self-appointed ‘police’ who have been active on social media in shaming those who have perpetrated (or are suspected to have perpetrated) QRP in their research.

Other sources were more positive about the crisis and incentives that have come from it. One participant said that although there is resistance toward the new methodology and ideas relating to the crisis, he is positive about them because it means that he can talk to his undergraduate students about problems in research with the silver lining that there is “improvement”. Many sources directly linked the crisis with changes in their research practice for the better. One credits it with having shaped him in terms of how he reports and reflects on himself as a researcher and his daily practice in that role.

It is evident from the participants’ accounts that reflexivity and transparency (‘popularized’ as a result of the crisis of confidence) are linked. Not only did participants tend toward using the words interchangeably, they tended toward conflating the concepts in their explanations. They also frequently associated the use of protocols like preregistration with transparent and reflexive practice. Most participants made references to the crisis in some way. Some directly by name, others by referring to QRP and preregistration/registered reporting.

Theme: reflexivity

This theme links closely to questions 5 and 6 and contains two subthemes. The sources expressed a range of opinions toward using reflexivity in practice. Reflexivity, according to the sources, is difficult, unwelcome and undervalued by peers and institutions, but nevertheless necessary for good and valid science.

Subtheme: Reflexivity and research quality

Sixteen participants emphasized an obvious link between reflexivity in research and higher quality science. For most of these participants, this link was made somewhat automatically, or at least without obvious hesitation. Two participants spontaneously made a further linkage between reflexivity and higher research quality on the one hand, and the crisis of confidence, suggesting that the crisis might be mitigated by more reflexivity. Several interviewees linked the practice of reflexivity with greater accountability, modesty and a better chance at avoiding slipping into questionable practices. Many participants were explicit about how reflexivity could be used as a means to decrease one’s biases by bringing them to awareness, and therefore reduce the negative impact of these biases on their research.

Positivity was tempered in many peoples’ accounts by limitations relating to the current academic environment. Several sources acknowledged that although reflexivity should be a key part of research, it might not be advisable for some people to include their reflections in publications because it is difficult:

“Everyone should know about it, but not everyone should write about it, because it’s hard. It should be main reading material for scientists, but I don’t know that a reflection in a paper is necessarily useful or, actually, even possible for some people.”[14]

And one described the ever-present issue of balancing between being part of your research and wanting to maintain a distance from your work:

“It's kind of like after a long time you've been doing this research for so long it's hard to know where you end and the truth begins right because your beliefs and the truth and your research is so interwoven. I find that a little bit scary because I like to think that I can kind of retain a certain element of logic and a certain element of being able to look at myself, but at the same time that gets harder and harder the more embedded in what you do.”[4]

Subtheme: sharing reflections as part of published work

Most sources were generally positive about the prospect of sharing their reflections and decision-making processes alongside their published work. Some felt it provided context to readers of their academic work:

“It’s important for other people to be able to evaluate the strength of evidence in light of how much they believe in it in the first place because that might have influenced the results.”[13]

Most participants mentioned the practical value of jotting down reflections and decisions during the research process. They reported that logging their reflections in some explicit form (such as a logbook), would save time for either themselves in the future, or for others conducting research in similar paradigms. A majority of interviewees noted that reflexivity in practice involves others in some ways. Three people recognised that difficulties with including reflections in written output may be more to do with their peers than with them - people are not open enough or reflexive enough themselves to appreciate such characteristics in others. Some participants saw value in logging their research as a way of maintaining accountability, for themselves, and between themselves and others in supervisory relationships.

“It just amazed me how much time we spent, and how little we knew about the other processes. We could have saved a few months of work if other people had written up the things that went wrong.”[4]

Several sources described an ideal/practice dichotomy in terms of reflexivity. They indicated that although reflexivity is a nice ideal, its practical application to research is limited. Reflexivity is difficult in practice because it is hard to be aware of what you’re unaware of. It is difficult because it takes time and care to cultivate a good sense of self-awareness, and it is confronting and uncomfortable at times.

Theme: objectivity is complex in practice

This theme also strongly reflected the questions asked (2, 3, 4, and 5). Opinions varied about how and where in the research process bias was most evident and damaging. Three sources suggested that bias influenced the process at every point along the way, while two others implied that bias enters the process during data interpretation. A common opinion, however, was that ideas held a priori about the research question would undoubtedly affect the research process. Another dominant opinion was that bias existing at any point had the potential to affect the whole process, as the research process is circular, and ultimately feeds back into itself:

“Ideally you get the evidence and you get your conclusion. Other times you get your conclusion and you come up with the logic of the evidence… It has to be kind of like that because the process of understanding is sort of circular.”[1]

It appears as though most participants have a complex relationship with objectivity in the research process: while it is probably good to strive for objectivity in the research process, it is difficult to achieve, if even possible. No consensus existed among the sources as to whether or not objectivity was important to improving the research process, or whether or not researchers are even capable of it (though most believed it was a bit of a pipe dream). It was the case that those sources that thought objectivity was key to good science also believed that objectivity was possible. It may be that because most participants were taught in empirical traditions where objectivity is prized, they feel a pull toward thinking of objectivity positively despite skepticism about whether it is possible. Most interviewees were fairly nuanced in their views: our research is always driven by our own values and beliefs, which is acceptable so long as we are transparent and reflexive about our role as researchers. One participant simply sees objectivity in a somewhat negative light:

“I think objectivity is overvalued in science. I think we should be more open about how subjective we are rather than striving for objectivity.”[5]

Discussion

According to the participants of this study, reflexivity is hard, can be confronting and might not be received well by academic peers. Nevertheless, they generally consider it desirable and a sure tool to help one produce good research. There is no consensus about whether or not it should be included in written academic output. The same sources provide insights into how they see objectivity: generally, they consider themselves incapable of objectivity toward their findings, and at the same time they believe objectivity is necessary for good quality research. Every academic in the sample discusses issues and tells stories in the context of their work environment. They make remarks about incentive structures, grant-writing, academic output and the role of collegial support during difficult periods. Marks of the crisis of confidence are to be found in one way or another in the narratives of the participants. Some mention it directly by name, others talk about ‘QRPs’ using the acronym as though it is common academic vernacular, and discuss attempts at preregistration; a few hint at it, referring to ‘new methodology’ and the ‘open science’ people.

Some of these findings are somewhat at odds with one another. On one hand objectivity is considered necessary for good research. On the other hand, reflexivity, a tool for explicitly introducing subjectivity into research, is expected to improve research quality because it can be a means of circumventing harmful bias.

While these two stances are not mutually exclusive, they are not in harmony. This may generally reflect the dichotomy between what individuals appreciate in an ideal world and what is feasible in practice. Another possibility is that participants have been primed for self-reflection as a result of the crisis of confidence, while at the same time still clinging (consciously or otherwise) to ideas (such as objectivity in research being desirable and attainable) learned in early academic training. Participants seemed to see reflection as an error correction mechanism, or a form of self-criticism. That is, a tool with a somewhat negative function. However, reflexivity can also be productive, as Finlay argues: a way of enriching our understanding of the phenomena we study, and a source of insight. The latter is highlighted in my (SMF) description of arriving at the Metascience conference at the start of this paper. I used reflexivity to better understand my own observations, and interrogate my own biases. It is valuable, and, I would argue, vital, for me to understand myself as the researcher in my work as fully as is possible. Without a high level of self-awareness, I am not delivering a faithful description of the scientific communities I am studying.

This finding can also underline one of Bourdieu’s key arguments for using reflexivity in research practice: that reflexivity imparts objectivity to research via the awareness of subjectivity- an awareness achieved by engaging in subjectivity. Objectivity is important to research in the respect that it can allow the researcher to analyze and interpret the data with as little interference due to prior expectations or beliefs as possible, however objectivity has its limits. That is, as humans, our own interests, needs and beliefs at potentially every stage in the research process drive us, despite our best intentions. Reflexive practice allows the goal of objectivity to coexist alongside the inevitable subjectivity brought into the research process by the researcher. Practically speaking, this approach is feasible. As demonstrated in this project, it involves a sustained effort on the part of the researcher to be reflexive throughout the research process, culminating in a reflection included in the publication (or at least in a place accessible to would-be readers).

The highest quality research could result from this marriage of objective and subjective, and communicating the role of reflexivity in one’s publications could further bolster this advantage. Moreover, we argue that a shift toward qualitative research for the social sciences has the potential to establish a new culture of reflexivity in the discipline that can increase the quality of its findings, as well as complementing the growing culture of transparency.

Quantitative research: a new application for reflexivity

As discussed, reflexivity is a mainstay of qualitative methods, but it need not be restricted to usage in qualitative paradigms. Reflexivity as a research skill can be applied to any research where a human is involved in the research process, which has important implications for improving reproducibility, and decreasing the prevalence of QRP. Currently, it is not commonplace for researchers in empirical psychological science to make reflexivity an explicit part of their research practice, nor to include personal reflections explicitly in their reports. If they did so, and if their reflections were read by others as potential information sources, replicating authors of those reports could use the information to better guide their replication protocols. This might be especially useful in cases where an extension to the replication is planned.

Mitigating the negative impacts of the researcher on the research is a benefit of applying reflexivity to quantitative research. If the researcher is cognizant of his expectations regarding his findings from the beginning, and is aware of the pitfalls associated with such expectations, he can conduct his research with fidelity and report his outcomes with the appropriate nuances and caveats every research report should feature. While preregistration will often be beneficial, reflexive research practice can have a similar effect where the researcher is his own watchdog. If used in conjunction with one another, preregistration and reflexivity can work together to doubly inoculate the researcher against unwittingly influencing his scientific output, and instead remain mindful of and monitor that influence.

In the present crisis in psychology, the field as a whole has in fact engaged in reflexivity in this sense. The psychology of psychologists, with their biases and motivated reasoning and other forms of cognitive flaws, has been much-discussed (Flis 2019; Morawski 2020). Psychologists have shown themselves to be suspicious of their own minds and skeptical of their ability to be rational, unless disciplined and constrained by procedures such as pregistration. Reflexivity on an individual level can likewise guard a researcher against the unwanted influence of her bias and prejudice on her research. Many interviewees saw the value of reflexivity in those terms, as antidote to the pernicious influence of subjectivity. But subjectivity may be an asset in quantitative research, just as it is in qualitative research. For example, Kochan (2013) has pointed to the importance that many scientists assign to what Kochan calls ‘epistemic emotions’, such as aesthetic preferences for elegant theories or formulae, or McClintock’s famous ‘feeling for the organism’. Research is not a purely rational activity, but is always emotionally charged.

Building on the work of, among others, J.W. McAllister, Kochan argues that epistemic emotions support the reasoning process by acting as detectors of salience in empirical findings and theories. Epistemic emotions guide researchers’ perception and evaluation when they choose lines of enquiry, engage with their own findings and those of others, and develop models and theories. Doing science involves taste. Kochan’s main point is that scientists develop and shape their epistemic emotions in a social process, mutually sharing and calibrating their tastes and attuning them to their research context. As a result, epistemic emotions are not idiosyncratically subjective, but intersubjective. We propose that systematic reflexivity, in quantitative research as well as in qualitative work, can be made a part of this process, as it gives voice to a researcher’s feelings regarding their object of study, their own ideas and those of others. Sharing such reflections with other researchers assists in calibrating, cultivating, and shaping them, turning them from individual biases into intersubjective, robust and reliable “detectors of epistemic salience” (Kochan 2013, p. 361).

Reflexivity has been criticized as being “self-indulgent, narcissistic, and tiresome” (Pillow 2010). Patai’s condemnation of those using reflexive practice to more extreme lengths is scathing “...we are spending too much time wading in the morass of our own positionings...When is enough enough?” (1994, p. 69). If done badly, reflexivity becomes navel-gazing. Finlay has warned that reflexivity should be “neither an opportunity to wallow in subjectivity nor permission to engage in legitimized emoting” (Finlay 1998 in Finlay 2002b, p215). She explains that researchers must meet the challenge of reflexivity by using it as a means to direct other interpretations, and to gain insight, not as an end in and of itself. We agree that the point of reflexivity is not merely to turn inward, but to share these reflections and make them part of a communal sensitivity towards the phenomena one studies. The point is not to erase subjectivity by detecting and counteracting it, nor to lose oneself in it by endless reflection, but to make it productive by making one’s reflections part of the research community’s cultivation and calibration of epistemic emotions. This, we believe, is the lesson that qualitative approaches can teach quantitative research.

Conclusion

I (SMF) made one observation in nearly all the interviews I conducted: people are uncomfortable with subjectivity. Even those who describe embracing it seem to speak to some extent at odds with their demeanor and the other things they say. Perhaps the root of the problem lies in perspective, as Linstead (1994) argues. Take two key assumptions, first, that it is possible to have objective knowledge about a particular phenomenon, and second, that our role as researchers is only one of faithfully retrieving that knowledge. If these two are held, then it stands to reason that we must be as pure a vessel as possible, such that we can deliver truth without tainting it: we must eliminate ourselves of subjectivity and be objective. Although the first assumption may hold for some scientific fields, we would be surprised for it to hold in the behavioral sciences. We find it difficult to believe that the variability and malleability of behavior and even, to some extent, cognition, lends itself to producing ‘objective’ knowledge, other than in exceptional circumstances. Moreover, in psychology there is an inherent, intimate link between the researcher and the object of study. Reflexivity, in that sense, is a given, making research in psychology a highly social venture. As Crang and Cook put it, behavior and our understanding of it has not “been discovered in the third person by a detached researcher, but constructed out of an intersubjective research process always saturated with relations of power/knowledge” (2007, p. 8).

Even if the first assumption (that objective knowledge is possible) were to stand up to scrutiny, I do not accept the second assumption: that our single role as researchers is to retrieve information. The brief goes deeper than that, when working with human information. We must provide context, in some form, to transform the information into knowledge and understanding about ourselves, and we, as the researcher – indeed, as Bourdieu’s ‘midwife’ – are the only ones who can provide that interpretation. Geyer (2017, p.123) emphasizes the need for interpretation. Although in her work, Geyer writes about the enduring value of print journalism in the face of the information revolution, her words can be easily appropriated for use in the current context: “... the idea rampant in our ... world today – that ‘information’ has been transmogrified into a value and a creature of itself, without the need to have it interpreted by appropriate and trained people... Instead, information without context, without knowledge and, yes, without wisdom, is as empty as life without beauty, without history and without hope.”

Reflection

Carrying out this research project was a powerful and empowering experience. Of the research projects I (SMF) have been personally involved in, however, it has also been the most difficult thus far. Conducting qualitative research on a topic that is personal and important to many researchers, including myself, has required much self-reflection and honesty from me. I have had to become and remain highly aware of my own expectations and biases in the process, and have had to work continually to separate them from the data and those providing the data, in order to produce a maximally honest and clear account. For example, I expected the qualitative study themes to more closely follow the questions posed. I had to actively let go of the themes I had planned to explore, to allow room for those that I identified in the data.

As PhD supervisor my (MD) role was primarily that of the midwife’s midwife, to borrow Bourdieu’s metaphor: helping SMF interpret the interviews through our conversations and writing together. At the same time, I realize that in so doing I have to some extent steered those interpretations. Under the guise of teaching I have tried to impose my views on objectivity and subjectivity and on qualitative research. However, what my views exactly are only became clear to me in the process of working on this paper with SMF.