Reactivity and good data in qualitative data collection

Abstract

Reactivity in qualitative data collection occurs when a researcher generates data about a situation with reactivity, that is, a situation in which the ongoing research affects the research participants such that they, say, diverge from their routines when the researcher is present, or tell the researcher what they think she wants to hear. In qualitative research, there are two basic approaches to reactivity. The traditional position maintains that data should ideally be collected in situations without any reactivity. In other words, good data are reactivity free. By contrast, the more recent view holds that data from situations with reactivity are fine as long as the researcher is aware of the occurring reactivity so that she can take it into account when interpreting her data. In this fashion, good data are reactivity transparent. In this paper, I first spell out and defend the more recent approach to reactivity. I argue that qualitative data are reactivity transparent when conjoined with true reactivity assumptions and that, thus supplemented, data are informative about social life independently of its being studied. Next, I examine various issues raised by the requirement to put forth true reactivity assumptions. Lastly, I use my discussion of reactivity transparency as a basis for providing a framework for thinking about good qualitative data.

1 Introduction

Reactivity in data collection occurs when a researcher generates data about a situation with reactivity, that is, a situation in which the ongoing research affects the research participants such that they, say, diverge from their routines when the researcher is present, or tell the researcher what they think she wants to hear. In the following, I am concerned with reactivity in qualitative data collection, that is, data collection by way of methods like participant observation and qualitative interviewing.

In qualitative research, there is a long tradition for recognizing the (possible) occurrence of reactivity in data collection. In fact, this tradition can be traced all the way back to the early beginnings of using qualitative data collection methods. Already Bronislaw Malinowski, who is often credited with having provided the first account of participant observation as a scientific method, commented on the issue of reactivity (Malinowski, 1922). In his presentation of the method from 1922, he notes how the people he studied at the Trobriand Islands were affected by his presence until they got used to him (Malinowski, 1922:7–8). Since then, qualitative researchers have variously discussed how to handle reactivity. Among them, it is possible to discern two basic approaches to reactivity.

The traditional view has it that data should ideally describe or otherwise record situations without any reactivity. In short, good data are reactivity free data. Whenever data are contaminated by reactivity (i.e., they are about a situation with reactivity), they are of poor quality. In this case the data are only indicative of what people do and say, when research is being carried out: the data fail to be informative about social life independently of its being studied. Consequently, a researcher should try to minimize or prevent the occurrence of reactivity and/or attempt to reduce and ultimately eliminate occurring reactivity so that she ends up with reactivity free data.

Janet B. Johnson et al. are representative of this position when they state that reactivity is a threat to data validity, i.e., data quality (Johnson et al., 2008:254).¹ As a result, they observe, “many researchers prefer unobtrusive or nonreactive measures of political behavior because they think the data are more natural and real” (ibid.245). Likewise, Cynthia Lietz and Luis Zayas adopt this view and so they recommend that “[t]o manage research reactivity, researchers may try to make their data gathering efforts less conspicuous and intrusive without deception. For example, video-recorders may be concealed or employed more discretely with knowledge of the participants. Researchers conducting participant observation may choose at times to act less as a ‘participant’ and more as an ‘observer’ to minimize reactivity” (ibid.192).

In contrast to the traditional view, the more recent position holds that data may well describe and otherwise record situations with reactivity. What matters is that the researcher is aware of the reactivity, whenever it occurs, so that she can take it into account when interpreting her data. As long as this is the case, the data are informative about social life independently of its being studied. I summarize these considerations by saying that, in this view, good data are reactivity transparent. Proponents of this approach hold that a researcher should not, or should only occasionally, aim to prevent or reduce the occurrence of reactivity and/or minimize or eliminate occurring reactivity. The researcher’s only, or primary, concern should be to get clear on the reactivity in order to factor it into her analysis of the data.

Martyn Hammersley and Paul Atkinson express this view when they state that “[a]ssuming we understand how the presence of the researcher may have shaped the data, we can interpret the latter accordingly and it can provide important insights, allowing us to pursue the emerging analysis” (Hammersley & Atkinson, 2007:102). And, in a similar spirit, Joseph Maxwell observes that “the goal in a qualitative study is not to eliminate this influence [by the researcher], but to understand it and to use it productively” (Maxwell, 2013:125).

In this paper, I first spell out and defend the claim that good qualitative data are reactivity transparent. To this end, I argue that data should be supplemented by true reactivity assumptions, as I call them, about whether or not reactivity occurred and, if it did, about its nature and causes. I maintain that, in conjunction with true reactivity assumptions, qualitative data collected in situations both with and without reactivity are informative about social life independently of its being studied. Next, I elaborate on this account by discussing various issues raised by the requirement to put forth true reactivity assumptions. Lastly, I use my discussion of reactivity transparency as a basis for further reflection on what constitutes good qualitative data. The upshot is a framework for thinking about good qualitative data that I briefly relate to recent philosophical discussions of data quality in other research contexts.

The paper makes three contributions. First, it supplements social science discussions of reactivity. Typically, proponents of the recent position only briefly state that researchers should take reactivity into account when interpreting their data (see, e.g., Maxwell, 2013; Merriam, 1988; Murphy et al., 1998; Patton, 1980). I provide a far more comprehensive and systematic examination of this idea, most notably by laying out different ways in which data may be rendered reactivity transparent by supplementing them with true reactivity assumptions.

Second, the paper fills a lacuna in the philosophical debate on reactivity. Philosophers of science have lately begun to take more of an interest in reactivity in data collection.² This is reflected by the recent publication of several papers on reactivity in data collection in social science experiments (Jiménez-Buedo & Guala, 2016; Jiménez-Buedo, 2015, 2021; Teira, 2013; Feest, 2022) and in survey studies (Runhardt, 2021). So far, however, no philosophical analysis of reactivity in qualitative data collection has been offered.

Third, the paper contributes to current philosophical discussions of what constitutes good, i.e., high-quality, data. While this has traditionally been a neglected topic, there is now a growing interest in the issue (see, e.g., Canali, 2020; Feest, 2022; Illari, 2014; Leonelli, 2017; Stegenga, 2014). The paper weighs in on this trend by providing a framework for the analysis of what constitutes good qualitative data. So far philosophical discussions of data quality have only focused on the quality of other types of data.

Here is how I proceed. In section 2, I offer a brief introduction to qualitative data collection and, in section 3, I offer a characterization of reactivity. In section 4, I lay out and defend the claim that good qualitative data are reactivity transparent and then, in section 5, I take a closer look at the requirement to put forth true reactivity assumptions. In section 6, I provide a framework for the analysis of good qualitative data and, in section 7, I conclude.

2 Qualitative data collection

Qualitative data collection is exemplified by the use of methods like participant observation, qualitative interviewing, focus group interviewing, and the collection of documents. In order to focus the discussion, I concentrate on what are likely the two most frequently used qualitative methods, namely participant observation and qualitative interviewing (Bryman, 2012:493). Unless otherwise noticed, I have in mind the employment of these two methods, when I talk about qualitative data collection and qualitative research in the following.³

The method of participant observation requires a researcher to participate in the research participants’ ways of life in their own surroundings. She may do so to different degrees. For instance, she may participate in the weaker sense of simply hanging around or in the stronger sense of engaging actively in the research participants’ activities. While participating, the researcher should observe, i.e., note, what goes on. Typically, a researcher carries out participant observation over an extended period of time. For instance, earlier on in anthropology, this meant that the researcher should stay in the field for at least a year. Today, studies of a much shorter duration are also regarded as perfectly acceptable.

On the basis of her observations, the researcher generates field data in the form of field notes. These consist in detailed descriptions of the researcher’s observations of the research participants’ doings and sayings, their appearances, the physical surroundings, etc. in terms that come as close as possible to the research participants’ own terms and perspectives (DeWalt & DeWalt, 2011:165–6). Besides field notes, field data may consist in video recordings of the research participants in various situations. Field data in this format, though, are much less common.

Moving on to qualitatively interviewing, the method requires the researcher to pose questions to an interviewee who, in her replies, is permitted and encouraged to digress, to expand on her views, to exemplify her points, to introduce her own concerns, and the like. Qualitative interviewing is semi-structured or unstructured, or somewhere in between. In semi-structured interviewing, the researcher has a list of questions that she goes through. She may pose them in whatever order seems natural during the interview, just as she may add questions. In unstructured interviewing, the researcher has at most a list of topics that she wants to cover. She does not introduce these in any pre-fixed order and formulates her questions as she goes along. As a result, the interaction comes close to an ordinary conversation. A qualitative interview is often conducted in a setting familiar to the research participant who is interviewed at length on one or several occasions.

Interview data take the form of interview notes. These are transcriptions of audio recordings (or possibly video recordings), or descriptions of what was said during the interview as based on the researcher’s observations and scratch notes. In addition to detailing what was said, interview data describe other aspects of the interview situation such as the setting in which it took place and the research participant's nonverbal behavior during the interview. While it is most common to write out interviews, recordings of them are sometimes used directly, i.e. without being first transcribed, as basis for analysis. Interview data consist in these recordings too.

In the preceding outline of participant observation and qualitative interviewing, qualitative data have been identified with the researcher’s notes (field and interview notes) and recordings (audio and video recordings). This is in line with how qualitative data are often conceived of in qualitative research. While it seems most common to equate data with a researcher’s notes, recordings are sometimes regarded as data too (see, e.g., DeWalt & DeWalt, 2011:157ff; Miles et al., 2020:7). The conception of data as notes and recordings is also in accordance with an influential philosophical conception of scientific data, namely Woodward’s characterization. The gist of it is that scientific data are public records that are produced by scientific methods and that may serve as evidence for scientific findings (Woodward, 2011:166). For my purposes, there is no need to take a stance on whether this account is suited as a characterization of scientific data in general (as Woodward thinks).⁴ It suffices to register how the account fits the conception of qualitative data as notes and recordings: the latter are public records, that are produced by way of scientific methods, viz. qualitative methods of data collection, and that may be used as evidence for answers to qualitative research questions.

Though participant observation and qualitative interviewing may be employed on their own, they are often used together in a single study. Moreover, they are sometimes supplemented by other data collection methods both qualitative and quantitative. In the following, I focus on research as it involves the application of participant observation and qualitative interviewing either on their own or in tandem. Still, my defense of the claim that good qualitative data are reactivity transparent is meant to apply to all data gathered by way of qualitative methods. Accordingly, I shall, at various point, briefly note how my analysis also extends to data collected by way of other qualitative methods.⁵

3 Reactivity in qualitative data collection

When data are collected by way of qualitative methods, reactivity is a standing possibility. I take it that reactivity in qualitative data collection occurs when a researcher collects data in a situation with reactivity, that is, a situation in which the research participants’ doings, sayings, etc. are affected, in regularly unintended ways, by the ongoing qualitative research. In what follows, I unpack this characterization. Also, I briefly explain why it is preferable to two other common conceptions of reactivity.

A good place to start is by noting that it is typically a precondition for the successful application of participant observation and qualitative interviewing that the ongoing research affects the research participants in certain ways. The successful employment of participant observation usually requires that the research participants react by enabling the researcher to carry out participant observation, as when they make room for her to sit around the table, take her along, and the like. Similarly, to conduct a qualitative interview, the research participant must take time out to do the interview and respond to the researcher’s questions and prompts. These sorts of influences on the research participants are part and parcel of the successful application of the methods and, in this sense, they are regularly intended effects of their employment and do not exemplify reactivity. Likewise, when prospective research participants respond to the question of whether they are willing to take part in a study, this is a regularly intended effect of asking people to give their informed consent to participate in a study. Thus, like the previous examples, this is not an instance of reactivity according to my characterization.

Reactivity only occurs when the ongoing research affects the research participants in regularly unintended ways while data are being collected about them. In these situations, the impact of the research goes beyond the regularly intended effects of using participant observation and qualitative interviewing, of asking people for their informed consent, and the like. For instance, a researcher who stays in the background without taking part in the research participants’ interactions does not regularly intend to influence these. Hence, her having an impact on the interactions constitute reactivity. Similarly, if a researcher influences what the research participants say in response to her questions, this exemplifies reactivity as this effect is not regularly aimed for when conducting interviews. Or if the information research participants receive about a study make them change their behavior in ways that goes beyond merely enabling the researcher to carry out her research, this is likewise a regularly unintended effect of providing them with this information.

The conception of reactivity just outlined comes very close to an alternative view which states that reactivity occurs when the ongoing research has unintended effects on the research participants (see Paterson, 1994:301). I prefer to talk about regularly unintended effects, rather than just unintended effects, to make it possible to speak about reactivity in cases where a researcher actively explores her effects on the research participants’ behavior. That is, imagine that a female researcher is interested in how women are treated and that she decides to carry out fieldwork with the intention of noting, among other things, how her own gender impacts the research participants’ doings and sayings. Intuitively, if the researcher’s gender does indeed have an effect on the research participants’ behavior, this exemplifies reactivity despite the influence being intended. Moreover, it is useful to classify it as such since, in practice, the researcher should handle this intended effect in the same way as if it had been unintended (as will shortly become clear). By specifying reactivity as regularly unintended effects, the research participants being influenced by the researcher’s gender is a clearcut instance of reactivity because it is a regularly unintended effect of using participant observation and qualitative interviewing.

Basically, anything about the ongoing research, including a researcher’s doings and sayings, her equipment, her appearance, etc. may influence the research participants in regularly unintended ways. For instance, and as touched upon above, a researcher’s presentation of her research project, its purpose, and expected outcome may cause reactivity. The same goes for her research equipment such as her use of an audio- or video-recorder (see, e.g., Speer & Hutchby, 2003). Or, to mention one last example, the researcher’s ethnicity, race, age, way of dressing, marital status, and manners may cause reactivity among the research participants (see e.g., Berg, 1995:45).

The reactive effects that ongoing research may have on research participants’ doings and sayings are also countless. For instance, research participants may, as a regularly unintended effect of the ongoing research, speak very formally, be careful not to swear too much, be very polite towards each other, refrain from commenting on political issues, and depict various practices and institutions in a favorable light. Moreover, the research participants may be more or less affected, in regularly unintended ways, by the ongoing research. They may, say, slightly hold back in their criticism of the management or completely refrain from saying anything critical. Or they may try to be somewhat more politically correct or very much so.

In any case, the regularly unintended influence of the ongoing research goes via the research participants’ beliefs, desires, feelings, etc. relating to the research. In this connection, note that their beliefs about the ongoing research may not be true as also illustrated by Paterson. She relates how a research participant wrongly believed that the hidden agenda of Paterson’s study was to back up the view that all nursing schools should be university-based (Paterson, 1994:306). As the research participant was opposed to this, she described the non-university-based schools in a very favorable light to Paterson. As to desires that may cause reactivity, these are exemplified by the wish to please the researcher, to win the researcher over, and to behave appropriately in her presence.

Reactivity is sometimes associated with overt research only, that is, with research where the researcher has disclosed to the research participants that she carries out research (and has obtained their informed consent to her study). Typically, this view goes hand in hand with another common circumscription of reactivity which has it that reactivity occurs when the research participants’ awareness of being studied has an impact on their doings and sayings (see, e.g., Bryman, 2012:715). However, a researcher’s appearance, equipment, etc. may also affect the research participants in covert research (Davies, 2008:83). In this type of research, research participants do not realize that research is ongoing: the researcher participates and observes without saying anything about her research project, while conducting unstructured interviews that appear like normal conversations to the research participants. As a researcher should be equally attentive to, and similarly handle, the impact of her appearance, equipment, etc. in covert research, I think it is preferable to characterize reactivity such that reactivity may transpire in both overt and covert research (though I do not hereby mean to endorse covert research in general). In contrast to the alternative conception just mentioned, my understanding of reactivity meets this desideratum because it holds that reactivity occurs when a researcher collects data in a situation in which the ongoing research, be it overt or covert, has regularly unintended effects on the research participants’ doings and sayings.

4 Qualitative data and reactivity transparency

Qualitative researchers are interested in finding out about some aspect(s) of the social world. They seek answers to research questions like “how do people cope with severe disease?”, “why do some people decide to move to Spain after their retirement?” and “how did young Danes experience the lock down during the pandemic?”⁶

According to the traditional position on reactivity, it is only data collected in situations without any reactivity that are telling about social life independently of its being studied. Data from situations with reactivity are of poor quality as they are solely informative about social life as affected by ongoing research. Thus, good data are reactivity free data since data having this feature contribute to making them fit to serve as evidence base for answers to research questions. Accordingly, a researcher should try to minimize or prevent the occurrence of reactivity, and to minimize and ultimately eliminate occurring reactivity, so that she ends up with reactivity free data. In this spirit, proponents of the traditional position recommend the use of unobtrusive methods and, sometimes, of covert research as well (see Cassell & Wax, 1980:261, Emerson, 1981:365–366).

The traditional position used to be the dominant view and it is still advocated today (see, e.g., Johnson et al., 2008; Padgett, 2008; Lietz & Zayas, 2010). However, roughly around the 1980s, the recent position, as I have called it, began to gain prevalence at least in connection with qualitative research involving participant observation (Emerson, 1981:365, Murphy et al., 1998:109).⁷ And since then, the view has been widely propounded (see, e.g., Cassell & Wax, 1980; Hammersley & Atkinson, 2007; Merriam, 1988; Murphy et al., 1998; Patton, 1980).

The more recent position maintains that data collected about situations with reactivity may well be employed to answer research questions about the social world. As long as the researcher is clear on the reactivity while taking it into account when interpreting her data, the data are informative about social life independently of its being studied.⁸ I summarize these considerations by saying that good data are reactivity transparent: data being reactivity transparent contribute to making them suited to serve as evidence base for answers to research questions.

In the following, I spell out the notion of reactivity transparent data and argue that reactivity transparent data are indeed fit to serve as evidence for answers to research questions about the social world. In doing so, I significantly elaborate on the reflections offered by proponents of the more recent approach to reactivity. Typically, they confine themselves to stating that researchers “should take their impact upon the data obtained into account in the course of analysis” (Murphy et al., 1998:110) and then they illustrate this idea with an example or two. I go beyond these accounts most notably by offering a systematic and detailed examination of different ways in which data may be rendered reactivity transparent.

As I go along, I take a stance on two issues that divide proponents of the recent position. One concerns the possibility of reactivity free data. Some hold that there is always reactivity in a data collection situation (see, e.g., Maxwell, 2013:125, Murphy et al., 1998:85), whereas others maintain that situations without reactivity may occur (see, e.g., Hammersley & Atkinson, 2007:102; Merriam, 1988:96). I comment on this disagreement below. In the next section, I address the other issue. It concerns whether it is feasible and desirable to try to reduce or prevent the occurrence of reactivity, and/or to minimize or eliminate occurring reactivity. Some advocates of the recent position contend that a researcher should never attempt this (see, e.g., Maxwell, 2013:125; Cassell & Wax, 1980:261; Emerson, 1981 for a summary of this view). In contrast, others state that while a researcher’s primary concern should not be to decrease and get rid of reactivity, it may, occasionally, make sense to try to do so (see, e.g., Patton, 1980:190; Hammersley & Atkinson, 2007:102).

This clarified, reactivity transparent data may be specified as data supplemented by true reactivity assumptions about whether or not reactivity occurred and, if it did, about its nature and causes. More precisely, there are four ways in which data may be rendered reactivity transparent by conjoining them with true reactivity assumptions. I now consider, and exemplify, each way in turn. The examples I present are merely meant to illustrate the different ways in which data may be rendered reactivity transparent. Hence, I leave it open whether the reactivity assumptions they involve are true. For now, I also set to a side the question of how reactivity assumptions may be confirmed.

First, data may be rendered reactivity transparent by supplementing them with the true assumption that no reactivity occurred in the data collection situation. Accordingly, insofar as they are not about research participants’ doings and saying as affected, in regularly intended ways, by the ongoing research, a researcher may view her data as being straightforwardly informative about what the research participants would also have said and done if no research had taken place. By way of illustration, consider Malinowski’s comment on reactivity in his study of the Trobriand Islanders: “[i]t must be remembered that as the natives saw me constantly every day, they ceased to be interested or alarmed, or made self-conscious by my presence, and I ceased to be a disturbing element in the tribal life which I was to study, altering it by my very approach, as always happens with a new-comer to every […tribal] community” (Malinowski, 1922:7–8). What Malinowski states here is that once the Trobriand Islanders had gotten used to him, it was safe to assume that no reactivity occurred in the situations he observed. If this is granted, Malinowski’s data were reactivity transparent in conjunction with this assumption.

Note that the traditional approach to reactivity is in complete agreement with these reflections. In fact, the position may be explicated as holding that whenever data may be supplemented with the true assumption that no reactivity occurred in the data collection situation, the data are reactivity free. Additionally, the approach maintains that good data are limited to data accompanied by this assumption. For this reason, it denies that data conjoined with reactivity assumptions in the remaining three ways are also fit to serve as evidence for answers to research questions.

By claiming that data may be rendered reactivity transparent by making the true assumption that no reactivity occurred, I side with those proponents of the recent position who maintain that data collection about situations without any reactivity may well occur. According to my conception, reactivity transpires when the ongoing research has regularly unintended effects on research participants’ doings and sayings. Thus specified, it is reasonable to think that data collection about situations without any reactivity may sometimes occur both when conducting participant observation and qualitative interviewing.

Second, data may be rendered reactivity transparent by adding the true assumption that reactivity occurred in the data collection situation, but that certain aspects of the research participants' doings or sayings were not reactive. That is, a researcher may, in regularly unintended ways, affect only some of the research participants or only some (aspects) of their doings and sayings. Further, she may be able to lend support to an assumption along these lines. By its light, and insofar as the ongoing research had no regularly intended effects on the nonreactive doings and sayings, the researcher may interpret her data about them as being straightforwardly informative about what the research participants would also have said or done if no research had been ongoing.⁹ Jill Fisher’s study of the clinical trials industry involves an example of this way of rendering data reactivity transparent (Monahan & Fisher, 2010). On one occasion, she observed how a doctor explained an upcoming clinical trial to an elderly woman and her son so that the woman might decide, on an informed basis, whether or not to enroll in the trial (ibid.365). Fisher noted that whereas the doctor seemed affected by her presence during the meeting, the elderly woman and her son weren’t. In view of this assumption, Fisher’s data about the elderly woman and her son were indicative of how they would also have behaved in her absence. As also illustrated by this example, nonreactive (aspects of) behavior (the elderly woman and son’s behavior) may occur in response to reactive behavior (the doctor’s behavior). When this is the case, the researcher should, of course, take that into account when analyzing her data.

Third, data may be rendered reactivity transparent by conjoining them with the true assumption that certain aspects of the research participants’ doings or sayings were reactive and hereby diverged in specified ways from how they would have behaved in the absence of any ongoing research. Or differently put, a researcher may sometimes have good grounds for assuming that, due to the reactive influence of the ongoing research, research participants were, say, more (or less) careful in their doings and sayings, behaved more (or less) by the book than would be the case if no research had taken place. Equipped with a true assumption of this sort, the researcher may regard her data about the reactive aspects of their behavior as being roughly informative about what the research participants would have said and done if she had not been around: in the latter situation, the research participants would have been less (or more) careful in their doings and sayings, would have behaved less (or more) by the book, etc. than what the researcher observed.

Fisher’s research on the clinical trials industry may also serve to illustrate this way of handling reactivity (ibid.). As part of her study, she talked to the research staff, who enrolled participants into clinical trials. In this connection, she noted that they made remarks about different racial groups in terms of their suitability to participate in the trials (ibid.368). Moreover, she registered that, due to her presence, they made an effort to be more politically correct than usual. Accordingly, her data were roughly indicative of how they normally talked about potential trial participants, namely in less politically correct manners than she witnessed.

Fourth and finally, data may be rendered reactivity transparent by supplementing them with the true assumption that certain aspects of the research participants’ doings or sayings were reactive and that this was caused by certain characteristics of the research situation (including the researcher’s appearance, her doings and sayings, etc.) or by certain of their beliefs or desires relating to it. In view of a true assumption along these lines, the researcher may typically interpret her data about the reactive aspects as being informative about how the research participants would also have behaved in response to the characteristics, or beliefs or desires, if no research had been ongoing. An example may make the basic idea here clearer. In her study of young offenders in secure care units, Bengtsson was able to determine that the young offenders mainly viewed her as “an adult and a representative of ‘society’ and mainstream values and morals” (Bengtsson, 2014:739). Moreover, this perception on their part influenced how they behaved towards her and, in particular, she noted how it had an impact on what they told her in interviews about their plans for the future, their background, and the like. Accordingly, she interpreted her data as being telling about what the young offenders would have said to an adult representative of society and how they would have acted vis-à-vis such a representative also if no research had been ongoing.

The preceding considerations show that, whether data are collected in situations with or without reactivity, they are informative about social life independently of its being studied when conjoined with true reactivity assumptions. Hence, in line with the more recent approach to reactivity in qualitative research, I conclude that good data are reactivity transparent: data being reactivity transparent contribute to rendering the data suited to serve as evidence base for answers to research questions about the social world.

Importantly, this conclusion applies not only to data generated by way of participant observation and qualitative interviewing, but to all forms of qualitative data. For instance, a researcher may gather data in the form of recordings that research participants make of their conversations in her absence. Or a researcher may collect data in the form of documents. In these sorts of cases, the data should be reactivity transparent too: the data should be supplemented by true reactivity assumptions because these allow a researcher to determine how her data are informative about social life independently of its being studied.

5 Qualitative data and true reactivity assumptions

The insight that good qualitative data are reactivity transparent means that qualitative researchers should collect data that they supplement with true reactivity assumptions. Yet, the requirement to put forth true reactivity assumptions raises several questions including: how should reactivity assumptions be confirmed? At what point in the research process should reactivity assumptions be made and corroborated? And isn’t it too demanding to require that data be supplemented with true reactivity assumptions? I now examine these issues.

A researcher who carries out participant observation and qualitative interviewing should confirm her reactivity assumptions by appeal to data that she collects as part of her study.¹⁰ It is helpful to explicate this point in terms of the distinction between first-order and second-order evidence. In general terms, data play the role of first-order evidence when they serve as evidence for (or against) a researcher’s answer to her research question. In contrast, data function as second-order evidence when they serve as evidence for (or against) other pieces of data being suited to serve as first-order evidence (see Zahle, 2019, and for a related distinction Staley, 2004). In other words, second-order evidence is evidence indicating that other pieces of data are good in some respect. Applying these points to the present discussion, the data that lend support to a researcher’s reactivity assumptions function as second-order evidence: by confirming the assumptions, they show that other pieces of data, as supplemented by the assumptions, are suited to serve as first-order evidence from a reactivity perspective. This last qualification is important. As I discuss further below, reactivity transparency is just one feature that good qualitative data have.

Second-order evidence in support of a researcher’s reactivity assumptions may consist in data about the situation in which the reactivity (possibly) occurred. For instance, the researcher may have noted that the research participants stopped talking when she approached or that they made comments like “don’t talk like that in the presence of a woman” and she may take these incidences as indicative of reactivity in those situations. Or she may have asked the research participants whether they agreed with her reactivity assumptions in relation to a given situation (this would be a form of respondent validation).

Further, second-order evidence may be constituted by descriptions or recordings of situations other than the one in which the data, that the assumptions supplement, were collected. For example, data collected later in a study may lend support to the assumption that reactivity occurred in the early phases of a study. Likewise, research participants’ descriptions of situations in which the researcher was absent may confirm the researcher’s assumption that she reactively influenced the research participants’ doings and sayings in a certain way when she was around in similar situations. It goes without saying that even if a researcher does her best, there is no guarantee that she will be able to generate (sufficient) second-order evidence in support of her reactivity assumptions being true.

There are roughly two options as to when a researcher may start making her reactivity assumptions and providing second-order evidence in support of them: while data collection is ongoing or after its termination. The first option is clearly preferable. In fact, from very early on during data collection and onwards, the researcher should pay attention to the issue of reactivity.

Most notably, this puts the researcher in a better position to make sure that she collects the evidence needed to corroborate her reactivity assumptions. By comparison, if she does not search her data for second-order evidence until after she has left the field, she may realize that she lacks data to support various reactivity assumptions and at this point there is nothing to do about it.

Further, insofar as the researcher makes and corroborates her reactivity assumptions during data collection, she may sometimes actively try to minimize or even eliminate her reactive influence on the research participants. That is, if the researcher thinks reactivity occurs and has an idea about its cause, she may occasionally deem it feasible to reduce the impact of the cause or even eradicate it. Moreover, she may at times regard this as desirable because less or no reactivity will allow her to collect data that are (more) relevant from the perspective of answering her research question. By way of illustration, consider a researcher who realizes that the research participants wrongly regard her as a government spy and that this belief reactively affects their behavior. In response, she decides that it makes sense to try to convince them that their belief is false. As her attempt is successful, the research participants’ doings and sayings cease to be reactively influenced by the belief and the researcher is able to collect data that are more relevant to answering her research question. The latter concerns how the research participants manage to get by and, no longer perceiving her as a government spy, they are now willing to let her in on their strategies of survival.

As already indicated, it is far from always possible and desirable actively to take steps to minimize or eliminate reactivity. Still, I submit, whenever it makes sense to a researcher, she should give it a shot. By making a claim along these lines, I follow those proponents of the recent position who think that a researcher may occasionally try to reduce or even get rid of reactivity. As far as I can gather, those who maintain that a researcher should always let any reactivity be do not provide any reasons in support of their standpoint. Moreover, they fail to recognize that data should also be relevant to answering the researcher’s question and that data about situations with less or no reactivity may sometimes be more relevant from this perspective than data from situations with (more) reactivity.¹¹

Be that as it may, consider a researcher who only makes and confirms her reactivity assumptions after data collection has been terminated. Evidently, this researcher forfeits the opportunity actively to attempt to minimize or get rid of her reactive effects on the research participants.

There is no point denying that it may require an effort on the part of a researcher to make reactivity assumptions and provide second-order evidence in support of them. At the same time, note that a researcher need not necessarily formulate and confirm her reactivity assumptions anew each time she engages in an interview or participates in yet another situation. True reactivity assumptions may carry over from one data collection situation to the next. For example, once a researcher has confirmed the reactivity assumption that, because the research participants are now used to her, her being a stranger no longer causes any reactivity, she need not re-confirm this particular assumption when subsequently collecting data about them.

Additionally, and most importantly perhaps, a segment of data may perfectly well play the role of both first-order and second-order evidence: it may serve as evidence for a researcher’s answer to her research question as well as for other strings of data being reactivity transparent or good in some other respect. This means that a researcher need not generate all the second-order evidence in support of her reactivity assumptions on top of the data she generates in order to answer her research question. Rather, the data that function as second-order evidence will likely to a large extent overlap with the data that constitute her first-order evidence. These considerations show that it is less demanding than it might seem at first sight to require that data be supplemented with true reactivity assumptions. In any case, there is no way around this predicament in the sense that good data are reactivity transparent.

In closing, note that the foregoing reflections apply to all forms of qualitative data about situations in which a researcher may possibly have had a reactive effect on the research participants’ doings and sayings. There are also qualitative data about situations where no reactive effects were possible. For instance, a researcher may collect documents in the form of meeting minutes about meetings that took place prior to the onset of her research project. Accordingly, her research could not possibly have had any reactive impact on the meetings that the minutes describe. In cases like these, a researcher may, without further ado, regard the assumption that no reactivity occurred as confirmed.

6 Good qualitative data

Traditionally, philosophers of science have paid little attention to the philosophical issues surrounding data collection, including the question of what constitutes good data. Recently, this has changed, and the question of data quality has been examined in relation to big data, clinical data, and experimental data (see Illari, 2014; Leonelli, 2017; Stegenga, 2014; Feest, 2022). So far, however, the question of what constitutes good qualitative data has not been examined. In this section, I elaborate on the insight that good qualitative data are reactivity transparent by providing a partial framework for thinking about the quality of qualitative data. Moreover, I briefly note how these reflections relate to current philosophical discussions of data quality in other research contexts.

In the foregoing, I have argued that good qualitative data are reactivity transparent. More precisely, reactivity transparency is a good-making feature of data because the feature contributes to making data suited as evidence base for answers to research questions. Importantly, reactivity transparency is just one among multiple good-making features of qualitative data. For instance, in the previous section, I noted that good qualitative data are also relevant to providing an answer to the research question under study. In other words, relevance is a good-making feature of qualitative data too. Moreover, I have elsewhere pointed to further good-making features of qualitative data and data sets (Zahle, 2021). Now, the observation that qualitative data are good in virtue of having multiple good-making features is in line with analyses of other kinds of data. In her (2014), Phyllis Illari discusses how big data have multiple good-making features, just as Jakob Stegenga mentions various good-making features of clinical data as they figure in QUATs (quality assessment tests) in medical research (Stegenga, 2014).

On my account, it is worth stressing, reactivity transparency is always (in all research contexts) a good-making feature of qualitative data. It is not such that, in some contexts, reactivity transparency is a good-making feature of qualitative data, whereas in other contexts it isn’t. Likewise, the feature of relevance is a good-making feature of qualitative data in all research contexts: good qualitative data are always characterized by being relevant to answering the research question under study. And the same point applies to other good-making features of qualitative data (see Zahle, 2021).

Further, it bears repeating that reactivity transparency is a feature that qualitative data only have in conjunction with true reactivity assumptions. Not all good-making features are like this. For instance, intuitively, qualitative data in the form of field- and interview notes should also be descriptively adequate, that is, correctly describe what transpired while the researcher carried out participant observation or conducted a qualitative interview (Zahle, 2021). Yet, segments of data have (or fail to have) this feature independently of being supplemented by any true assumptions. The claim that good data may involve true assumptions is also made by Uljana Feest in her discussion of data quality in psychology experiments (Feest, 2022). She contends that good experimental data in psychology are data supplemented by various true assumptions about the data production process (Feest, 2022:13).

As stated above, qualitative data being reactivity transparent contribute to their being suited to serve as evidence base for an answer to a research question. This, however, should not be taken to mean that reactivity transparency is a property that a data segment only has relative to a particular research question. Whether some data are reactivity transparent is independent of the particular research question being examined: it is solely a matter of the data being supplemented by true reactivity assumptions. By comparison, the good-making feature of relevance is a property that some pieces of data only have in virtue of being able to serve as evidence for an answer to a particular research question. Accordingly, the same data segment may be relevant relative to answering one research question, but not another. In her examination of big data, Illari likewise considers whether good-making features are only possessed relative to a particular research question (Illari, 2014). She reaches a slightly different conclusion, though, in that she makes it clear that, in a research context, big data have (or fail to have) all their good-making features in virtue of their relation to the specific question that a researcher aims to answer.

Lastly, note that my analysis has left it open whether reactivity transparency is a good-making feature of all types of data. Recent discussions of reactivity in experiments show that this is not the case. For instance, Maria Jiménez-Buedo contends that, in connection with single experiments in economics, it is only experimental data about nonreactive sayings and doings that should serve as a basis for the experimenter’s causal inferences (Jiménez-Buedo, 2021; see Feest, 2022 for a similar point). I take this point, viz. that reactivity transparency is not a good-making feature of all types of data, to be in line with Sabina Leonelli’s more general insight that the quality of big data in the life sciences can only be assessed relative to a range of factors including the kinds of methods and equipment used to produce them (Leonelli, 2017).

The preceding considerations add up to five claims that may serve a framework for the analysis of good qualitative data. Since the framework merely captures insights that appeared from my discussion of reactivity transparency, it makes no pretense to be complete. Here are the claims:

• 1. Qualitative data sets being good is a matter of their having multiple good-making features.

• 2. Many good-making feature of qualitative data sets are the same in all research contexts.

• 3. Some good-making features of qualitative data sets are ones they have only in conjunction with true assumptions.

• 4. Some good-making features of qualitative data sets are ones they have only relative to a particular research question.

• 5. The good-making features of qualitative data sets differ, to some extent, from those of data sets consisting of other types of data.

In my view, these claims may helpfully structure further exploration and thinking about what constitutes good qualitative data. Moreover, the fact that the framework is in line with various points made in philosophical discussions of other types of data suggests that it might be worth examining whether the framework may be developed into a framework for thinking about the quality of good scientific data more generally. This is a task that goes beyond the scope of the present paper.¹²

7 Conclusion

In this paper, I have examined reactivity in qualitative data collection. I began by presenting two methods of qualitative data collection and by characterizing reactivity in qualitative data collection. On that ground, I defended the claim that good qualitative data are reactivity transparent rather than reactivity free. Thus, I argued that qualitative data are reactivity transparent when supplemented by true reactivity assumptions about whether or not reactivity occurred and, if it did, about its nature and causes. And I showed that data being conjoined with true reactivity assumptions contribute to rendering them suited to serve as evidence base for answers to research questions. Next, I discussed various issues raised by the requirement to put forward true reactivity assumptions. Lastly, I elaborated on my claim that good qualitative data are reactivity transparent by providing a general framework for the analysis of good qualitative data.

In closing, let me briefly stress the significance of my discussion vis-à-vis the long tradition for holding that reactivity poses a threat to the quality of qualitative data. My defense of the position that good qualitative data are reactivity transparent means that this threat may be dismissed: as long as data about situations with and without reactivity are supplemented by true reactivity assumptions, there is no problem, from a reactivity perspective, with using the data as evidence for answers to research questions. In other words, following my analysis, it is not reactivity as such, but the lack of reactivity transparency, that constitutes a threat to data quality.