Is Quantitative Research Ethical? Tools for Ethically Practicing, Evaluating, and Using Quantitative Research
This editorial offers new ways to ethically practice, evaluate, and use quantitative research (QR). Our central claim is that ready-made formulas for QR, including ‘best practices’ and common notions of ‘validity’ or ‘objectivity,’ are often divorced from the ethical and practical implications of doing, evaluating, and using QR for specific purposes. To focus on these implications, we critique common theoretical foundations for QR and then recommend approaches to QR that are ‘built for purpose,’ by which we mean designed to ethically address specific problems or situations on terms that are contextually relevant. For this, we propose a new tool for evaluating the quality of QR, which we call ‘relational validity.’ Studies, including their methods and results, are relationally valid when they ethically connect researchers’ purposes with the way that QR is oriented and the ways that it is done—including the concepts and units of analysis invoked, as well as what its ‘methods’ imply more generally. This new way of doing QR can provide the liberty required to address serious worldly problems on terms that are both practical and ethically informed in relation to the problems themselves rather than the confines of existing QR logics and practices.
KeywordsQuantitative research Quantitative methods Statistics Probability Regression Research design Data analysis Inductive inference
This essay describes some significant ways that researchers can practice, evaluate, and use quantitative research (henceforth ‘QR’). Our central claim is that all ways of doing QR operate in an ethical domain. We initiate a new discussion about QR at the Journal of Business Ethics (JBE) to develop this claim. We show that acknowledging the ethics-laden nature of QR raises questions that are often absent in discussions of QR practice, evaluation, and use—not to mention the practices, uses, and forms of evaluation themselves. Although we offer reasons for this absence, our ultimate aim is to encourage new ways to understand and do QR that are attentive to ethics. In particular, we draw attention to questions about how the purposes and effects of QR are constituted in a situated manner. For what purpose is the research being done in this way? What are the practical effects of doing research in that way?
To respond to the questions raised by realizing that QR is a domain of ethics, we recommend rejecting typical ready-made formulas, decontextualized ‘rules of thumb,’ and universal ‘best practices’ for conducting and evaluating QR. Such abstracted solutions limit research and its interpretation, while ignoring how its ethical status can shape its worldly effects—as qualitative researchers note (e.g., Cunliffe 2003; Hardy et al. 2001; Jeanes 2016). Instead, we advocate ‘built for purpose’ approaches that situate QR in the pursuit of tackling serious worldly problems—including inequality, global warming, or the corruption of democracy by monied interests—while focusing on the ethics of this process and its implications for action (see also Ezzamel and Willmott 2014; Farjoun et al. 2015; Freeman 2002; *Greenwood 2016; Martela 2015; Wicks and Freeman 1998).
The intended outcome is an approach to QR focused on whether research works for specific purposes. Stated directly, it is specific purposes that should drive the production and use of quantitative research, not ready-made formulas applied without attention to purposes and their relations. This focus places the onus on authors, editors, reviewers, and readers to critically evaluate acts of QR based on how they work for specific purposes—an author’s, a reader’s, JBE’s, an external community’s, and so forth—while considering the relevance or value of these purposes for JBE and the worldly problems that are at stake. If QR achieves its purposes, and these purposes are worthwhile, then publication at JBE may be warranted.
In what follows, we first animate our position by critiquing typical QR approaches, which are often divorced from specific purposes in the name of an abstract normative agenda that masquerades as an epistemic pursuit of validity, objectivity, and the like. Our critique and focus on purposes also apply to categories like descriptive, predictive, or prescriptive research that some researchers may believe offer a priori separations between types of QR. Our goal is to disrupt the universality of such categories, so that QR can be built for specific purposes rather than merely retro-fitted with preexisting narratives or practices. To facilitate this, we offer two dimensions of QR practice that can be used to understand and guide QR: ‘orientations’ and ‘ways of doing’ QR. As we note, an orientation can be a way of realizing a purpose, and a way of doing QR can realize an orientation. By analogy, going to California (a purpose) may require facing West (an orientation) and walking quickly (a way of doing).
An example we treat later is Turker (2009), whose stated purpose was to create and validate a measure of corporate social responsibility (CSR), oriented toward representing a particular body of CSR literature and scholars as well as practitioners, using a psychometric logic of scale development as a way of doing QR. As this description of Turker’s paper implies, we use orientations and ways of doing to disrupt typical ways of conceptualizing QR—instead of starting with baggage-laden terms such as ‘constructs’ or ‘methods.’ In turn, connecting purposes with orientations and ways of doing QR prompts asking novel questions about the ethics involved in a QR project, which can be used to guide QR practices and their evaluation while avoiding a habitualized reliance on abstractions like validity or objectivity.
To clarify what we propose for QR, we use examples of possible QR approaches and describe them in relation to various purposes, orientations, and ways of doing. For this, we draw on existing JBE papers as examples (annotated with a *), which appear in an online special issue that allows readers to easily consult the JBE papers we cite.
In our conclusion, we emphasize that using, practicing, and evaluating QR requires hard work, treating each QR activity on its own terms and contextualizing it in relation to the ethical issues that it embodies. Inherent to this pursuit is a quest for coherence between the production of QR and the purposes for doing so. If purposes are to be meaningful in specific situations, then the choice and elaboration of particular methods over others should cohere with the contextualization of purposes (i.e., a study’s method can be used and justified for a specific purpose, and purposes are always contextually specific). ‘Best practices’ or ‘rules of thumb’ may be useful for the purpose of standardizing QR, but this typical imperative for decontextualized abstraction distracts from the task of putting QR to work for other purposes that are of greater concern—inequality, global warming, or corruption. For this reason, we conclude with what we mean by specific purposes and how QR can address them, recalling an original purpose of JBE: ‘to improve the human condition’ (*Michalos 1988, p. 1).
The Present Scene and Situation
QR is often done in terms of representation and correspondence (Zyphur et al. 2016). In this narrative, worldly phenomena are represented in research, including by theories, hypotheses, models, equations, samples, data, or parameter estimates. In turn, these can be true, valid, or unbiased by corresponding to their worldly counterparts, for example, when observed data correspond to what they are meant to measure or parameter estimates corresponding to correlations or causal effects in a population. This narrative helps constitute multiple epistemologies in core disciplines of business ethics research, including psychology, sociology, economics, statistics, and analytic philosophy (e.g., Gabbay et al. 2011; Pedhazur and Schmelkin 2013; Shadish et al. 2002; Wasserman 2013; Wooldridge 2010). The narrative does help to organize QR, but it causes two problems that we now describe: an ethic of probabilistic inference; and a simplistic understanding of QR.
An Ethic of Probabilistic Inference
With the goal of producing representations that correspond to worldly phenomena, quantitative researchers often describe their core purpose to be the production of true, valid, or unbiased inferences. By ‘inference,’ they typically mean an act that relates a representation (such as a sample) to its proposed worldly counterpart (such as a population), whereas ‘true,’ ‘valid,’ and ‘unbiased’ imply that a representation perfectly (or at least adequately in some way) corresponds to its proposed worldly counterpart. This manner of conceptualizing research creates a dilemma that QR practices are meant to address.
The dilemma is this: if the world can only be represented, then correspondence is always uncertain because the world can never be known ‘in itself’ (Hacking 2001, 2006). In turn, inferences that connect a representation to worldly counterparts are uncertain (i.e., correspondence is always uncertain). To address this uncertainty, most QR practices rely on probabilities that are used to guide and justify three stages of a QR process: research design, which is a process of generating representations (e.g., sampling and measurement); data analysis, which generates additional representations that synthesize those from the first stage (e.g., parameter estimation, such as a regression coefficient β); and inductive inference, which summarizes how representations and the world correspond (e.g., hypothesis tests with generalizations to a population). Probabilities and probability theories bind these three stages together in practices designed to maximize correspondence at each stage (Howie 2002).
The result is arguably the dominant ethic of QR practice, in which researchers ought to generate representations with the highest probability of correspondence (e.g., Panter and Sterba 2011). Consider three examples of what researchers are told they should do and why: sample randomly from a population or randomly assign people to experimental conditions because this decreases the probability of systematic bias; collect large samples because these increase the probability of true parameter estimates; and limit the number of inferences using a dataset because this decreases the probability of making false inferences. In such activities, the necessity of probability may be obscured because its historical production as the primary tool for justifying QR practices may be forgotten. Nevertheless, probability does guide QR practices via artifacts such as p-values, Type-I/II errors, or confidence intervals.
The historical genesis of an obsession with probabilistic inference has been well documented in the literature that convincingly shows how a set of value assumptions and their implications produce an ethic to guide and govern research (see Daston 1995, 2005; Dewey 1929; Hacking 1990, 2006; Gigerenzer et al. 1989; Shapin and Schaffer 1985). Today, this ethic is immensely powerful as a central feature of many professional academic institutions, including research ethics and journal reviewing procedures (Panter and Sterba 2011), as well as ‘style guides’ for papers (e.g., American Psychological Association 2009). For example, a recent analysis of papers in the Academy of Management Journal showed that an average of 89 p-values was reported in each published article (Gigerenzer and Marewski 2015).
Unfortunately, this ethic of probabilistic inference cripples quantitative researchers, who may find it difficult to consider ethical issues on terms that are contextually relevant rather than in relation to a ‘problem of inference’ (Wicks and Freeman 1998). In general, the problem is that focusing on representation and correspondence produces an orientation toward ‘facts’ rather than ‘values.’ The latter are then presumed to belong to the domain of ethics, which is further presumed to be irrelevant or secondary to representing a singular reality. Furthermore, probabilistic inference tends to universalize elements of its computation with conceptual tools such as ‘samples,’ ‘populations,’ and the like, which are (erroneously) not conceived of as being constructed based on values—and therefore ethics.
This way of understanding and doing QR is ethically consequential, causing problems that we discuss throughout our essay. Some examples include a consideration of: who will use QR and for what purposes once it is produced? how do the descriptions used in a QR project serve the people or groups who are studied? what are the ethical consequences of doing QR that uses notions of representation, correspondence, and probabilistic inference? and how can QR be done so that its purposes address matters of serious concern while attending to a broad range of ethical issues? Such questions are hard to address when the primary tools for grappling with ethical issues exist in relation to probabilistic inferences (e.g., Panter and Sterba 2011), for example, by resorting to discussions of generalization to an abstract population rather than the ethics of QR practices and their effects in specific cases.
In sum, the ethic of probabilistic inference distracts from many other ethical issues. Also, it offers few tools to grapple with worldly problems outside a logic of inference under uncertainty and its associated ethic of maximizing or minimizing probabilities. Clearly, quantitative researchers need new ways of understanding and doing QR, including new kinds of ethical commitments alongside tools for understanding and handling ethical conundrums.
A Simplistic Understanding of Quantitative Research
The second problem caused by QR narratives is a simplistic understanding of the QR process. By emphasizing formal logics such as statistics and probability, researchers can fail to notice the actual doing of research, including the production of representations and the creation and use of specific tools for testing correspondence. In turn, by overlooking how QR is done, many researchers fail to see how the theory of knowledge that accompanies QR binds narratives of representation and correspondence to the assumption that this is the only reasonable way to understand abstractions such as ‘knowledge’ or ‘truth.’ Critical inquiry into the practices of QR production becomes impossible. One consequence is that a theorized singular external world—or, simply, ‘reality’ in the representation and correspondence narrative—is often understood as being somehow naturally constituted rather than existing as the product of QR practices. In other words, researchers fail to see the rather obvious reality that they coproduce what they propose to merely represent, including populations, variables, statistical parameters, chance or probabilities, and constructs (Zyphur et al. 2016).
The net result is that while a representation is being actively produced, researchers are falsely construing themselves as being in a passive role that merely represents what existed ‘all along,’ or that is purported to exist outside of the descriptive processes that define QR. This is misguided and unhelpful. It is misguided because ‘[m]ethod and reality do not fit by good fortune or preestablished harmony. Each defines the other’ (Hacking 1990, p. 213). Rather than waiting to be observed, objects of research are entangled with the research process (Deetz 1996). Researchers use tools they develop for themselves, such as measures or conceptual artifices such as variables or constructs (Hacking 1992b; Latour and Woolgar 1986). A representation cannot be divorced from the practices that produced it, and quantification with statistics and probability does not change this (Hacking 1992a).
The simplistic story about merely representing the world is unhelpful because it disempowers researchers from actively producing representations that are designed to do practical work by achieving relevant purposes (Abrahamson et al. 2016). Although quantitative research produces images of various kinds (e.g., a parameter estimate), the content of an image is thought to be separate from the QR activities that could be (re)engineered to satisfy many potential purposes (Wicks and Freeman 1998). Indeed, beyond ‘mere’ representation, QR has vast untapped potential for active (re)description that can change the world by changing the way that people understand themselves, their organizations, their governments, their civic duties, and the like (Abrahamson et al. 2016).
In sum, the result of existing QR practice is an expanse of methods and results that not only tend to be dry and sterile, they are also repetitive. Unfortunately, QR usually seeks to merely reproduce familiar representations with familiar methods—such as measuring a well-known construct with a pre-validated measure. This hinders the ability of researchers to address matters of serious concern in novel and contextualized ways that can have impact by intervening in the world to change it—rather than merely attempting to ‘represent’ it. To emphasize our previous point, QR practitioners need new orientations and ways of doing QR.
An Open-Ended Starting Point for Quantitative Inquiry
Having stated issues with existing QR practices of representation and correspondence, we now introduce an alternative that initially serves to relegate the status of representation and correspondence to one set of research practices among many. More importantly, we seek to motivate a new overall starting point for QR. This starting point is neither a model on which to build another quantitative empire, nor is it a rejection of the important work that many quantitative researchers already do. Instead, our starting point gives way to a universe of worldly problems that most QR neglects, or has yet to tackle head-on.
To begin, we put forth two infinitely long and intersecting dimensions of QR practice that we call orientations and ways of doing, which connect purposes to QR practice. Instead of being ‘foundations’ or somehow fundamental in a representation correspondence sense, each category and its contents are akin to idioms or axiomatic lists that tend toward infinity because they can be populated indefinitely, limited only by the creativity of those who adopt them. They may also be orthogonal, indicating that each orientation can, at least in theory, be combined with any way of doing QR in order to achieve a given purpose. In what follows, we describe these dimensions, beginning to populate the lists that may constitute each dimension while illustrating the fruitfulness of combinations that emerge. However, there are two caveats to mentioned upfront which, if ignored, undermine our broader recommendations.
First, it is specific purposes that should drive the production and use of quantitative research. If readers ignore the centrality of purposes in the production and use of QR, then our point has been missed. Furthermore, attempting to retain purported a priori categories such as descriptive, predictive, or prescriptive QR also misses our point. We say more about this later, but for now we reiterate that our efforts are not ‘best practices’ or ‘rules of thumb,’ because such simplistic heuristics are the very problems we attempt to avoid. Second, we make convenient choices by using easily recognizable language (i.e., orientations and ways of doing) and by invoking a two-dimensional space to tabulate them. However, these terms, and our way of relating them, are only a convenient starting point. Our hope is that researchers will begin to set these terms (and others) free by making them available to any arrangements that make sense for the broader agenda we are outlining. To repeat, QR should be ‘built for purpose,’ and it is purpose, not dogma or habits of QR practice that should drive (and limit) the production and use of QR (similar to Cartwright 2006, 2007).
Achieving a given purpose requires practical activity to be organized, including thinking, speaking, materially acting, and the like. To describe the organization of QR activity, we invoke the notion of an orientation, which can be understood as a way of realizing the purpose(s) of a study. To establish an orientation, one must face in a specific direction or otherwise exist in a specific relation to specific things. In turn, an orientation helps to determine what is observed or found to exist in a research setting, as well as the meaning of what is observed or found vis-à-vis its implications for ways of doing QR.
The etymological root of ‘orientation’ is in the verb ‘to orient,’ which as a secular term means ‘to position or align (a structure, etc.) with, or in a particular way relative to, the points of the compass, or other specified points’ (OED 2016). As an action, ‘orienting’ thus allows us to discuss the placing of research and a researcher in relation to specific things, or the relative direction in which research moves. For example, when Hakala and Ylijoki (2001) asked senior researchers ‘for whom is research done?’ they found four general orientations: academic, civil society, governmental, and entrepreneurial. Each orientation differed with respect to the audience (e.g., scientists, the public, decision makers, ‘markets’), the kind of knowledge that was produced (e.g., theoretical, practical, instrumental, commercial) and the motivations for doing research (e.g., reputation, progress, information, production).
This schema is rudimentary, but the definition of a research orientation can provide ‘different answers to the question “for whom is research done?”’ (Hakala and Ylijoki 2001, p. 374). In turn, this connects the purposes of QR with the way it is oriented. To elaborate, if we take Hakala and Ylijoki’s question and subordinate it to one that asks ‘what is the purpose of this research?’ we can ask questions such as ‘for whom?’ but also ‘to what end?’ ‘in support of what?’ and so forth. This expanded scope prompts thinking broadly in terms of what is to be achieved or enacted via QR with respect to various purposes and uses, and there are infinite ways to orient research based on the practical effects it is meant to have. In turn, doing and evaluating QR becomes linked to the purposes and orientations of a QR project—are they worthwhile and have they been adequately described and addressed in a study?
At this point, quantitative researchers who prefer stable research frameworks, models, rules of thumb, and best practices may be asking for a point of reference. This is because the raison d’être of such structures is to enable the communal agreements that allow deploying notions of representation and correspondence with an ethic of probabilistic inference (e.g., agreement about what is being represented and how to test correspondence). Because we seek to overcome the necessity of these theories, we advise against a tautological return to preexisting QR standards and norms—how can the foundations for QR be critiqued and overcome if they are also applied to understand the critique and its result?
We agree that a point of reference is needed for QR and we understand anxieties that may emerge over the retraction of a stable and simple set of scripts for it. Let us therefore consider ways to orient research beyond representation, correspondence, and probability. To start, consider that many discussions related to QR are oriented toward repeatability or ‘replication.’ For example, psychologists are increasingly concerned that their QR practices result in findings that cannot be trusted to replicate (e.g., Pashler and Wagenmakers 2012). Similar concerns exist in strategic management, wherein the practices associated with replication have been treated as a criterion for evaluating the worthiness of a science, contrasted against the practices that lead to replication failures (e.g., Bettis et al. 2016).
There are good reasons why researchers may be oriented toward replication (see a JBE editorial by Roloff and Zyphur 2017). For example, one purpose of replicability might be to respond coherently to a perceived need for consistent, parsimonious knowledge claims. Such claims can be difficult to trust if their associated practices cannot reliably link the communal discourse of the claims to outcomes associated with communal QR practices, such as replicable estimates of ‘causal effects’ under some description and practice. Another purpose of replicability might involve attempts to tame the unruly nature of environments that continually disrupt attempts at stable QR results (Gelman 2015). Or, as with Bettis et al. (2016), the purpose might be to critique the fetishizing of newness in management research (after Davis 1971), with the hope of appearing more scientific. Alternatively, research can be empirically oriented toward replication, studying it with QR tools such as meta-analysis for the purpose of showing how QR practices determine results. For example, *Rathner (2013) meta-analytically examined QR studies on the effects of socially responsible investing and investment outcomes, showing that QR practices determine what is observed—similarly, Kaptein and Schwartz (2008) attempt to stabilize the way ‘business codes’ are studied, pointing out that ‘[t]he greater the variety of research methods for determining the effectiveness of business codes, the more the findings will fluctuate’ (p. 117).
Of course, any purpose might justify the orientation of replication, but researchers and reviewers/editors should also think broadly about the ethical consequences of such purposes and their associated orientations. For example, replication exists ‘inside’ existing QR logics, so it is institutionally reproductive and therefore serves a purpose of perpetuating notions of representation, correspondence, and probability, often with a focus on mechanistic law-like effects that distract researchers from addressing worldly problems. Different orientations with different purposes can actively reconstruct notions of replicability so that QR can focus more directly on matters of serious concern rather than having to be filtered through notions of replication. In Hakala and Ylijoki’s terms, the focus could be a form of public knowledge, or it could involve political advocacy, or it might be about upholding a kind of market ideology.
Alternatively, we may think of an orientation for QR that differentiates itself by being against existing practices in order to make a difference, including in ethics (e.g., Wicks and Freeman 1998). In this case, QR could critique existing norms within QR itself (an inward-facing orientation), or it could be used against established ideas about QR in other domains of social life (an outward-facing orientation). An example of the former is our paper, whereas examples of the latter are critiques of ‘measures’ as apolitically producing representations (see Law 2009; MacKenzie et al. 2007; Scott 1998). Work that straddles both may be *Collison et al. (2012), who show that QR studies of the effects of corporate governance laws on market-based outcomes are flawed because they fail to consider a larger conceptualization of what an ‘outcome’ may be. *Collison et al. show that ‘societal well-being’ outcomes offer fundamentally different pictures of how to govern and legally regulate companies. Their purpose is to influence researchers and policy makers.
Of course, our rather simplistically described orientations can be supplemented with more sophisticated terms drawn from across the social sciences. For example, practice theory has played a prominent role in the work of many scholars since at least the 1970s, influencing a variety of business subdisciplines to produce new avenues for realizing a host of new agendas (see Feldman and Orlikowski 2011). Practice theory could offer new orientations for quantitative researchers in a manner that forwards specific purposes as we suggest here, with a focus on the actual activities of QR rather than the abstractions usually desired by researchers, such as truth, validity, objectivity, and the like. This opens the door to more complex metaphors for conceiving the nature of social inquiry, such as the notion of QR practices producing different ‘images’ of organizations (see Morgan 2006).
Under this description, researchers’ observations are a function of how their practices are oriented, allowing organizations to appear as machines, organisms, cultures, and the like. Each orientation may aid many purposes, but the orientation determines what is observed and what it means for worldly action. As an example, consider *Prado and Woodside (2015), whose purpose is to address fair trade and work practices, with an interest in understanding the adoption of the practices across firms and countries. This purpose—contributing to general knowledge—is common, but their orientation is configurational, with organizations and causes conceptualized as sets of characteristics rather than levels of variables. In turn, the relationship between firm characteristics and the adoption of fair trade and work practices appears as complex and nonlinear. As others note (e.g., Misangyi et al. 2017), configurational orientations produce images of asymmetries that are otherwise missed. Thus, the orientation of the authors is a practice that allows the world to appear as configurational.
In a similar manner, if an instrumental orientation that promotes thinking about QR as a ‘device’ or as a ‘tool’ is adopted, this could produce a different approach to how QR is done (see Wicks and Freeman 1998; see also Erturk et al. 2013). Though it may be common to think of devices as machine-like (e.g., ‘rules of thumb’ or ‘best practices’), devices need not look like (or work like) machines. A device can be a ‘purposeful or planned contrivance constructed for a particular end that achieves that purpose by dividing, separating, and classifying,’ understood as ‘a set of implicit and explicit strategies that work more or less repetitively to order, sort, define and arrange a heterogeneous but relatively discreet social and material field’ (Singleton and Law 2013, p. 260). As we have already noted, QR can be viewed in relation to machine-like pursuits of representation and correspondence, but QR may also be used as a tool or device to achieve many other purposes.
For example, many studies in JBE use QR tools to predict well-being outcomes with various predictor variables related to ethics. Here, QR becomes an instrument to justify a focus on ethics-related predictors because of their effects on well-being. As an example, *Huhtala et al. (2011) show a relationship between ‘ethical culture’ and various measures of well being among managers. Instrumentally speaking, this paper works for the purpose of justifying a focus on ethical climate in organizations through the QR tools deployed.
In sum, there are many ways that QR researchers can orient themselves beyond mere representation and correspondence. To further illustrate how this may be done, we now shift our attention to strategies for QR, which we refer to as ‘ways of doing.’ Before proceeding, however, we acknowledge that some researchers may still feel that we are unclear regarding what an orientation is and how to adopt one. This feeling of uncertainty is expected from researchers who appreciate ready-made QR formulas, in place of having to consider the ethics of QR. As in life generally, there are no easy answers for how researchers should be oriented, but for JBE an overriding concern should be the ethical implications of orientations that drive the production of QR. Whose purposes are served by an orientation? In what ways is an orientation useful for addressing matters of worldly concern? There are no singularly right ways to understand or deploy orientations, but these are the kinds of questions that may be asked and answered in papers submitted to JBE.
Ways of Doing
To achieve a specific purpose requires a specific orientation. To quantitatively put an orientation into practice requires a way of doing QR. To offer guidance for QR practitioners, we now provide examples of how to connect purposes and orientations with what quantitative researchers typically think of as methods—although by ‘ways of doing,’ we will mean more than what is often implied by ‘quantitative methods.’ For analytic purposes, we map this section onto the three QR stages noted previously: research design; data analysis; and inductive inference. Yet, we do not endorse uncritically separating these stages, and we note that ‘inductive inference’ is itself often oriented toward abstraction, which is not our goal.
Also, to avoid singular notions of representation and correspondence, and connect our discussion to familiar QR concepts, we collapse and remake two key QR terms: internal and external validity. Although we could invoke other forms of validity (e.g., Hardy and Clegg 1997), these familiar terms are a useful starting point for our discussion because they refer to whether a study’s methods allow for causal inference (internal validity) or generalization to a population (external validity; e.g., Campbell 1957; Campbell and Stanley 1963). These terms partly derive from a pragmatist focus on experimentation to guide action (e.g., Campbell 1991), but they are ‘still burdened with residues of a positivist philosophy… [placing] unnecessarily severe constraints on the range of standards available to assess and certify claims’ (Dunn 1982, p. 295). Thus, a problem with these notions of validity is that they distract from whether or not a study works to ethically achieve its purposes.
A focus on valid inferences leads to abstractions that are distant from the values and ethics that a study embodies, particularly in the ways that conceptualizations of phenomena and people are deployed—what we are calling orientations. By separating facts from values, facts appear to be unrelated to ethics; and with a focus on facts, ethics appear irrelevant for QR validity (e.g., Pedhazur and Schmelkin 2013; Shadish et al. 2002). New understandings of validity are needed to address the ways that QR is an ethical act and ethically consequential. This ethicality may be unrelated to representation or correspondence, such as if QR is meant to produce images of society that change the way people think and act—an enactment of a reality that did not yet exist to be merely ‘represented’ (Abrahamson et al. 2016).
Thus, we propose the use of a new conception of validity to subsume others: studies are valid when they develop a relationship among relevant purposes and ethics by connecting worldly affairs and concepts (i.e., orientations) with appropriate QR practices and discourse (i.e., ways of doing QR). This is to say that validity is relational rather than being inherent in a QR study—a study may be valid for some purposes or people but not others. Thus, we call this type of validity ‘relational validity,’ and it can be used to evaluate whether a study is adequate or useful and worthy of publication (cf., Tuck and McKenzie 2015). This occurs if a study produces and connects relevant purposes with its orientation and ways of doing QR in an ethically informed manner. A study, or a claim, or a specific QR activity, is valid not because of what the study is or what researchers do abstractly—and not via inferences based on probabilities. Instead, studies are valid when issues are addressed in ways that are appropriate given their context and the purpose of the research, all of which are ethics-laden.
This is not to rebut other work on how to understand and do QR (e.g., Kaptein and Schwartz 2008; Shadish et al. 2002), which may help establish relational validity if the connection among purposes, orientations, and QR practices is contingent on drawing specific kinds of causal or other conclusions for audiences that appreciate them. Indeed, our point is that attention to such matters partly informs relational validity. To be clear, we are suggesting that something incorporating a broad notion of ethics should subordinate other notions of validity as being useful only when they fit with relevant purposes and orientations in ethically considered ways—thereby establishing relational validity. We now offer examples of this way of reasoning about validity for research design, data analysis, and inductive inference.
The design of QR is often treated in relation to three activities: sampling, measuring, and procedures for causal inference. These activities are usually described as if they allow for the production of a representation (of a population/phenomenon), whose correspondence can subsequently be estimated or otherwise tested (Shadish et al. 2002). Yet, in our view, this puts the cart before the horse—as if the world existed in one way, waiting for the researcher to stumble upon it so that it could be singularly described using QR tools. If researchers are going to have the liberty, creativity, and ethical reasoning required to establish relational validity, more socially and practically grounded understandings of QR are needed. To promote this, we now explore topics in sampling, measurement, and causal inference.
Samples and Populations
The formal basis of probabilistic inference with statistics requires that a population is adequately represented by a sample. If researchers want to rely on this logic, then they may. However, doing so overlooks important issues that are germane to connecting the ethics of purposes and orientations with ways of doing QR.
To start, even when deploying notions of representation, conceptualizing populations from which samples are drawn involves acts of classification, rendering research subjects and objects similar or different in specific ways. For example, the study of gender as a male–female distinction is common, but even for something as ‘natural’ as this divide, constructing and reifying this difference in any singular way has political and ethical implications (see Young 2011). The same is true for ‘organizations’ when they are treated as abstract entities.
Unfortunately, most QR studies ignore the way they actively construct similarity and difference through notions of samples and populations. Even popular-press articles point out this deficiency to explain recent decouplings of statistics from democratic institutions, as in Brexit and Donald Trump: ‘blindness to local cultural variability is precisely what makes statistics vulgar and potentially offensive. Regardless of whether a given nation ha[s] any common cultural identity, statisticians would assume some standard uniformity or, some might argue, impose that uniformity upon it… Part of the job of statisticians is to classify people by putting them into a range of boxes that the statistician has created’ (Davies 2017).
In turn, failing to critically engage with QR subjects/objects can lead to a focus on abstracted aggregates rather than relevant and specific purposes. For example, the common QR practice of meta-analysis often invokes a notion of ‘the population,’ but precisely what this might be and the ethics involved in statements about it are rarely considered. Indeed, the prevailing QR norm is to invoke almost any concepts necessary to deploy QR tools without considering the concepts or their ethical implications even on the terms of representation and correspondence that are meant to justify the QR tools themselves.
In our view, if a study is oriented toward representing a population, this should be stated and the nature of the population should be described and ethically treated. Methods for establishing the representativeness of a sample should be used, which may include sampling randomly or by using prototypical cases. Yet, even with a logic of representation, researchers have an obligation to conceptualize populations sensibly (see Zyphur et al. 2016). This requires a thorough understanding of the ethics of the process of categorization that allows the construction of notions of similarity and difference that produce ‘populations.’
For this and other purposes, we agree with *Greenwood (2016): ‘researchers should address three fundamental questions…: (1) What is the research project about including what is the involvement of human participants (the individual subject) and what is the purpose of the paper (the collective subject)? (2) What ethical issues… are raised by this project (for the individual and collective subject)? and (3) How does the researcher address these ethical problems?… Ethical considerations of management research should be embedded throughout the research process and should be written into any publications arising’ (pp. 518–519). These questions ask for more than what typical notions of representation and correspondence can answer. To establish relational validity requires that a study and its authors’ practices and discourse exist in relation to the production of subjects/objects, broadly working toward relevant and valued purposes through the orientations invoked in a specific QR study. This relation is ethical and ought to be thoroughly considered to merit publication in JBE.
Furthermore, if there is no clear population to which QR is generalizing other than ‘everyone (or everything) all the time’—which seems to be what many QR studies imply by the term ‘population’—then inferences to this population are hard to take seriously. In such cases, researchers and external stakeholders would be better served by moving away from representation and correspondence with an abstract population. Instead, QR can be oriented toward enacting or actively producing images of a specific situation in which it is sensible to invoke a description a category of subjects/objects. Even the nature of subjects/objects can be built for purpose if a situation or context is treated on terms that are designed to address a specific problem—it would hardly be ethical to categorize people as either ‘male’ or ‘female’ if researchers were studying a lesbian, gay, bisexual, transgender and intersex organization.
Conveniently, by taking research subjects/objects seriously, the particulars of a situation can be taken seriously to achieve a given purpose. In turn, the ethic of probabilistic inference can be discarded, focusing instead on questions like *Greenwood’s (2016). Such questions and their answers are at the core of relational validity. Without recourse to abstractions such as a ‘population’ and by taking research subjects/objects seriously, researchers can work ethically to connect purposes and orientations with ways of doing QR—this may have nothing to do with conceptions of a ‘sample’ drawn from a ‘population.’ In sum, we recommend that researchers take a reflexive view of themselves and their practices, working ethically to link a context of research subjects/objects with the purposes and orientations of QR, and explicitly describing these links as evidence of relational validity.
Measurement is often treated as a process that allows the numerical representation of natural or ‘objective’ features of subjects/objects of study. Yet, as history shows, the world does not come prestructured as descriptions or qualities that can or should be singularly described (Hacking 1999, 2002; Rorty 2009). Instead, communities of people produce local agreement regarding what exists and how to speak and act in relation to it. This is a process of orienting a researcher and a community of researchers, which takes hardwork before becoming a legitimate way of doing QR in the form of measurement instruments.
Thus, ‘measurement’ is more an accomplishment than it is a representation, because an environment has been rendered sensible by producing and agreeing upon specific tools for description (Hacking 1992a, b). For example, *Keeble et al. (2003) describe the creation and use of ‘sustainability performance’ indicators and the difficulties in adapting new descriptions and practices to the ongoing discourse and activities that define organizations. As the authors imply, choosing an indicator or calling it ‘valid’ is a process that does not map onto a singular reality. Rather, it is a challenge that requires connecting existing discourse and practices with a new reality that incorporates the indicators. Thus, there is nothing abstractly natural or objective about measures, because they are produced to fit specific situations that define the ongoing activity of a community, and images of a reality that is purportedly ‘measured’ are a function of a community’s values and interests. Indeed, it is the values and interests that a measure incorporates which makes it relevant, such that the results of measurement have a ‘looping’ effect by orienting people and institutions to a measure, only then making the images it produces appear ‘objective’ (Hacking 2002).
A common example includes the creation and ‘validation’ of measures. For example, as we noted earlier, *Turker (2009) created a measure of CSR, writing questions or ‘items’ by synthesizing the existing literature and concerns of researchers, including input from business professionals. Turker also used factor analysis to assess how items ‘hang together.’ The result was a set of items that measure different CSR ‘factors,’ ‘dimensions,’ or ‘constructs.’
This is a nice example of measurement and validity in the making, with *Turker (2009) relying on input from researchers and others to create a framework for CSR and to develop questions about it. Nowhere in this process is a singular external reality accessed. Instead, Turker rightly works to establish relational validity by incorporating the concerns and interests of relevant stakeholders in the measure, including researchers and professionals. Also, with factor analysis, Turker rightly examines the coherence of the scale items—if respondents do not react to the discourse of the items in consistent ways, how useful can their responses be? Again, factor analysis (or reliability analysis) says nothing about how a measure accesses a singular reality, because responses are a function of ways people speak and think, which serve as the basis for producing scale items in the first place.
One implication of this way of understanding measurement is that researchers already ignore the imperative of valid or objective measurement on naïve terms. Indeed, the basis for scale validation is connecting the values and interests of a community with the practices of measurement—which themselves must be agreed upon in a community as legitimate. As a process, this is self-reinforcing, with measures becoming institutionalized because they offer legitimate images of what is referred to as ‘reality’ by following the practices of measure validation that a community creates or adopts for its own purposes. For example, *Turker (2009) currently has over 600 Google Scholar citations, meaning his way of constructing images of CSR must be dealt with by anyone desiring to construct different images using a different measure, thus creating the ‘looping’ effect we noted previously.
As our description of measurement should make clear, we view measurement as a productive activity, and one that is rife with ethical quandaries as well as opportunities to establish relational validity by ethically enacting orientations that serve various purposes. To start, we direct the reader to the earlier questions posed by *Greenwood (2016), which allow researchers to grapple with how measures should be understood for any QR project at JBE. Her questions point to normative dimensions of measurement that are typically overlooked. For example, if a description in the form of QR measures and the data they produce are going to be used to represent reality, then whose interests are incorporated in the descriptions and for what purposes? Also, considering that measures are ways of enacting orientations, what kind (and whose) orientations are enacted in any specific case of QR, and what are the broader ethical and other implications of the results in a general sense?
As an example, *Kerssens-van Drongelen and Fisscher (2003) note two ways to understand the ethical dimensions of measurement as ‘role morality’ and ‘common morality’ (see also Werhane and Freeman 1999). In this framework, anyone doing measurement has ethical obligations defined by their role but also a wider community. In our view, researchers should consider themselves similarly, with relational validity requiring ethically connecting purposes, orientations, and ways of doing QR via measurement and other QR practices.
To this end, typical ways of assessing measures such as factor analysis can be used (e.g., *Turker 2009), but simplistic terms of representation and correspondence offer few tools for understanding the ethics of this process. Instead, measures should be understood in relation to how they ethically connect purposes and orientations through ways of doing QR, and one way to do this is to address *Greenwood’s (2016) questions. In doing so, researchers may consider for whom a measure is valid and whether or not people describe themselves in a way that is consistent with the measure. Again, relational validity implies that a measure may be valid for specific purposes, orientations, and groups of people, but not others. At JBE, explicitly arguing for the relational validity of a measure means justifying the connections that allow a QR paper to ethically fulfill the purposes it is intended for, whatever those are.
Procedures for Causal Inference
Causal inference is often a goal of QR, with researchers attempting to justify causal inferences by experiments and statistical methods designed to remove potential sources of causal ambiguity. Any approach to causal inference has benefits and drawbacks, which are often treated in relation to abstract ‘threats to validity’ or ‘endogeneity’ (e.g., Pedhazur and Schmelkin 2013; Wooldridge 2010). Yet, these concepts often fail to contextualize productions and uses of QR, leading to three related issues that can be addressed to help establish relational validity. First, seeking universalized causal inference assumes that mechanistic, law-like causal relations are of primary interest. Although causal reasoning can usefully address some problems, being forced to describe the world with law-like causal effects is troubling. Partly, the problem is that this leads to insensitivity to context (which becomes ‘noise’ or ‘error’ to ‘control’), ignoring things that are difficult to treat with metaphors of ‘mechanism’ (as if the world was defined by the physics of objects in motion).
Ethical dilemmas are always in situ and complex. Therefore, attempting to universally describe and control the world with ‘causal effects’ often means overlooking the nuance and ethics of worldly situations and actions, which then makes it difficult to see connections in a context where causality has been established and one in which a specific finding may be useful (Falleti and Lynch 2009). Again, notions of an abstract ‘population’ can do more harm than good here, especially when coupled with universalized and mechanical ‘causal effects.’
Second, even with an interest in causal effects, notions of valid causal inference often imply that there is a singular way to map discourse (e.g., ‘ethical conflict’) onto action and/or situations (e.g., specific case materials or levels of an observed variable). Yet, with an infinity of ways to describe a situation or activity, causal effect claims are often ambiguous about how and where a QR finding might work to achieve a specific purpose. This problem is compounded by the institutionalization of specific concepts in relation to a measurement or experimental intervention, creating a ‘looping’ effect that cements the link between concepts and specific things as conceived by researchers rather than in relation to worldly problems.
A classic example is the notion of ‘memory’ in psychology, which came to be associated with recalling random ‘units’ or ‘bits’ of ‘information’ in an experiment, rather than by conceiving memory in relation to worldly problems of action—with psychologists seemingly unaware of how they make their concepts for themselves (Rose 1985). To tackle this insularity and lack of reflexivity, researchers could ask how a specific conceptualization or finding can help address a relevant purpose in the varied world of experience. Doing this requires recognizing that ‘causal effects’ operate only in relation to specific descriptions of the world, and making a description relevant for a specific problem requires contextualization that is sensitive to the problem and the people involved with it (Falleti and Lynch 2009).
Third, work on causal inference typically assumes that causality is singular and that it is not political or ethics-laden. Yet, history shows that causality is pluralistic (Cartwright 2004, 2006, 2007), and therefore efforts to singularize causality by standardizing QR should be seen as political ventures designed to homogenize the ways researchers reason and behave. When this becomes dogmatic, it limits the kinds of problems and situations that can be addressed because notions of causality are ‘a constraint on the kinds of structures we will allow our models to have’ (Cartwright 1993, p. 423). This is important because ‘causality’ is used to assign responsibility and agency in order to further specific purposes in practical situations (Stone 1989), and this has ethical implications (for discussions in JBE, see *Painter-Morland 2011; *Soares 2003). Indeed, by invoking different kinds of entities such as individuals or organizations, researchers are able to populate the world with notions of causes and effects that can lead to disastrous outcomes—consider how the global financial crisis was aided by causal reasoning about free markets and the value of deregulation.
Establishing relational validity means ethically linking purposes, orientations, and ways of doing QR by addressing these three points, rather than starting with specific notions of causality or an interest in causal effects or law-like mechanisms and then proceeding to fashion images of the world via QR. This is to say that causal inference and its artifices—as ways of doing QR—may usefully establish connections among purposes and orientations, but this is an ethics-laden process. Indeed, when causal interference is a useful way of doing QR, this will be due to a contextualization of a causal inference that fits a particular context and purpose, helping readers understand and evaluate QR ethics. Thus, relational validity helps overcome a focus on representation and correspondence, affording researchers the liberty required to ethically achieve relevant purposes whether they involve causal inference or not. The implication is that even causal inference should be built for purpose.
Data analysis is typically described as a process of ‘estimation,’ with samples used to compute ‘statistics’ that estimate ‘parameters’ such as (co)variances or causal effects that define ‘populations’ (e.g., Shadish et al. 2002). Thus, an estimated statistic is described as being true or valid when—as a representation—it corresponds to a parameter, with estimation methods being justified when they maximize the probability of correspondence.
If researchers are oriented toward representation and correspondence, then this logic may suit their purposes. Yet, establishing relational validity requires more: ways of doing data analysis should be ethically built for the purposes that motivate a study, enacting an orientation by producing images that are useful for a purpose. For example, linear regression is often used to ‘estimate’ regression weights or ‘slopes’ linking ‘predictors’ and ‘outcomes.’ However, how this can ethically link purposes and orientations is rarely justified or explored.
The explain, any regression technique models average levels of an outcome variable at different levels of predictors (see Appendix). This focus on averages is consistent with an interest in avoiding specific ‘errors in inference’ linked to probabilities (Pedhazur and Schmelkin 2013; Wasserman 2013). Yet, as *Hill (2002) implies, a focus on averages is a utilitarian orientation rather than a social justice orientation wherein the least well-off (or perhaps the most harmful) people (or outcomes) are a focus of inquiry. Thus, when researchers describe regression coefficients as indicating potential changes in an outcome as a function of predictors, it is always average levels of the outcomes that are modeled and all inferences are being made and hypotheses tested in relation to these.
This is practically and ethically consequential for QR. For example, if the purpose of research is to address justice, then a distribution’s variance or its highest/lowest values may be of interest. Although some researchers acknowledge the potential flaw of averages (e.g., Kozlowski and Klein 2000), this insight is rarely coupled with the importance of linking ethics to the purposes that motivate a study. Although various regression methods can quantitatively predict variances or high/low values along a variable, simple graphs or plots of observed data may also be suited to the task. Again, this illustrates how typical QR practices often have little to do with ethically linking purposes and orientations and instead tend to rely on routinized habits of QR practice that are not explicitly justified in specific cases of QR.
The kind of critique we offer here and specific examples of it could continue almost indefinitely, but for the sake of concision we offer only one additional insight into how to establish relational validity during the process of data analysis—while, we hope, not being overly prescriptive regarding how this may be done in a specific case of QR. We recommend focusing on the kinds of things used as subjects/objects in data analysis and how these are ethically justified in connecting purposes and orientations via specific ways of doing QR. For example, *Ralston et al. (2014) examined within- and between-country proportions of variation in culture variables to justify a focus on individuals (within each country) rather than only conceptualizing these variables and ethical outcomes at the national level. Similarly, *Orlitzky et al. (2015) examined the effect of firm, industry, and national factors on corporate social performance, showing that firm-level factors were associated with the most variance in social performance.
Such ways of doing QR can be useful by invoking images of different kinds of entities and modeling them as having specific properties via the methods used. For example, by inferring the meaningfulness of individuals, firms, or countries based on degrees of variation, a specific way of differentiating individuals or firms versus nations is invoked. In turn, establishing relational validity requires that the ethics of this invocation as it relates to specific purposes and orientations should be addressed. For example, why is variation the distinguishing characteristic of the reality of a subject or object?
With a focus on (co)variance, typical QR practices tend to value variation for its own sake and equate variance with the meaningfulness of entities. However, if the purpose of research is to justify focusing on individuals or firms versus nations to achieve a given purpose, then it is possible that a lack of variation may also be of interest—for example, in homogeneous organizations that lack diversity. For QR to be relationally valid, it must address the ethics of a way of doing data analysis as it relates to purposes and orientations, rather than starting with ways of doing and their associated QR logic and then proceeding to purposes and orientations—as is often habitually done in QR.
Again, this puts the onus on authors and reviewers to attempt to connect purposes and orientations with ways of doing QR, so that a valued purpose can be addressed by whatever kinds of research subjects/objects may be most appropriate—individuals, firms, and the like. Imbuing agency and an independent reality to any such entity should be considered an active practice rather than such entities existing de novo or a priori to be ‘empirically studied’ or even ‘discovered’ as if these processes were done somehow in ethically neutral ways. To denote the existence of something is to actively conceptualize it in some way, which should be done with an eye to how this may be done in relationally valid ways.
Inductive inference is typically a final step of QR that describes something about the world as being true (or false), typically by hypothesis tests. For this, probabilities commonly play a role, often as p-values, confidence intervals, or estimates of statistical model fit, all of which are meant to assess the correspondence of representations. This orientation toward generalization does serve a research purpose, but it is often tied to abstract notions of validity that make it difficult to understand how inductive inferences may be related to using research for a specific purpose in a specific situation. The problem, in part, is that the goal of inductive inference often opposes doing something practical with the products of research. The result is that QR is done to make inductive inferences, but when talking about generalizing to a specific situation or context that would involve actually doing something to change the world, researchers are instructed to hedge by noting the ‘limitations’ of a study.
Treating this kind of disjunction, Schön (1992) notes that ‘[r]esearchers may choose to stay on the high, hard ground where they can conduct research of a kind the academy considers rigorous… Or they may go down to the swamp where they can devote themselves to the social problems they consider truly important, but in ways that are not rigorous in any way they know how to describe. They must choose whether to be rigorous on the high ground or relevant in the swamp’ (p. 120). For our discussion, ‘the high ground’ and ‘rigor’ are linked to inductive inference and often involve ways of doing QR which assume that the varieties of experience and worldly uncertainties could or should be singularly described and evaluated. Yet, practical action is about tackling specific problems in specific situations on always local terms, which may be unrelated to researchers’ hypothesis tests and measures of uncertainty. In turn, there will always be a disjunction between inductive inferences (which involve abstraction) and the practical uses of research (which involve something specific).
To address this disjunction while allowing researchers to retain notions of inductive inference, we propose that relational validity offers a novel response to the centuries-old problem of induction (see Gabbay et al. 2011). As we have noted, relational validity involves ethically connecting purposes, orientations, and ways of doing QR, including how inferences are made. For this purpose, no method of sampling, analyzing data, testing hypotheses, or quantifying probability allows predicting the future with certainty. The world is too varied, and uncertainties are too overwhelming for research to generate inductive inferences that will be true in an abstract sense. Indeed, there is no abstract ‘world’ or ‘reality,’ because real problems and actions are always local and specific. Thus, if the goal of QR is to generate inferences that allow accurate generalizations to specific situations, then researchers must actively work to make their inferences true by actively coupling research findings to external situations and contexts, thus working toward relational validity. The result is that inductive inference is not an act of making statements in a paper; instead, it is the work of actively connecting research results to worldly situations and problems.
This can be done via research design and data analysis, for example, by limiting the scope of a research purpose so that research subjects/objects are chosen to fit the context of a specific purpose. Yet, our suggestion goes beyond this. In our view, relationally valid inductive inferences should be understood as the end result of an interaction between a researcher, a manuscript (and perhaps other materials generated during research), as well as practical uses of the research to address a specific purpose in an ethically considered manner.
This proposal—connecting research to contexts of application—has already been made in various forms (e.g., Abrahamson et al. 2016). Yet, this work often fails to tackle powerful guiding concepts for QR, especially ‘validity’ as related to representation and correspondence. Under our conception, relational validity tears asunder the separation of inductive inference and specific generalizations, because a context of use plays a key role in inductive inference by requiring the active participation of research in a world of purposes. In this process, descriptions can be generated to address specific purposes in context, rather than being evaluated by whether representations are valid because they abstractly correspond.
Thus, although it is easy for us to propose the term relational validity, putting it into practice will require substantial effort. For example, *Knox and Gruar (2007) worked to apply a theory of stakeholders to the development of a marketing strategy in a nonprofit firm. Their work is relationally valid because the ethical implications of their work were a concern throughout the process of realizing their purpose with a specific orientation and a way of doing their research. To increase the relational validity and justify publication at JBE, other researchers might start by offering web-based tools to help organizations access their research and/or describe a campaign to alert organizations about their work. Importantly, establishing relational validity requires ethically linking purposes, orientations, and ways of doing QR. In turn, inductive inference means actively working to enact research purposes, making research ‘true’ by helping it to shape the world (Abrahamson et al. 2016).
To do this ethically, researchers must act in good faith, offering prescriptions and recommendations based on the best available information to fit a specific purpose. On this point, QR practitioners may be alarmed that we are obviating the need to rely on typical notions of ‘best practices’ to avoid ‘errors in inference’ and/or rejecting the best available evidence when intervening or making worldly decisions, which is not necessarily the case. For example, substantial research has been concerned with ‘questionable research practices’ wherein QR practitioners search for small p-values and then report their results as if they had not done this (Bettis et al. 2016). In our view, this kind of practice would also not be in the spirit of relational validity, which requires engaging with research subjects/objects ethically rather than treating them as tools for a fishing expedition to further researchers’ careerist ambitions while at the same time intentionally misleading journal reviewers and readers.
We are also not rejecting the usefulness of evidence-based practice, although we would critique ‘evidence hierarchies’ and other simplistic frameworks that are found in discussions of such practice—with randomized experiments or meta-analyses touted as being best no matter the purpose or context of application (e.g., Rousseau et al. 2008). As policy scholars note, it is ‘naive rationalism’ to decontextualize and attempt to universalize evidence in this way and suppose that it will be useful (Russell et al. 2008, p. 40). Relational validity will link evidence with purposes in specific contexts. For this, determining the value of whatever might be called ‘evidence’ can be done by logics of appropriateness that center on ethically connecting situation-specific interests, concerns, outcomes, and purposes. As Parkhurst and Abeysinghe (2016) note, ‘[a]n appropriate use of evidence, therefore, would be one which is transparent about the policy concerns at hand, which questions whether intervention effects will be expected in the target area, and which is critically aware of different ways to classify populations and… problems’ (p. 673).
In the end, establishing relationally valid inductive inference by actively working to fulfill a purpose does not amount to a rejection of empirical ideals or notions of evidence. Instead, it merely requires doing what seems most advisable given available theory, empirical findings, and the complexities of real-world situations in light of the ethics of specific ways of doing QR and realizing valued purposes. This, we hope, can define QR at JBE.
Ethically Building for Purpose
To conclude our paper, we first offer terms for an ongoing discussion of ‘purposes’ and how QR can be ethically built to fulfill them. We then touch on a few other relevant issues. As the reader may have noticed, we explicitly delineated various orientations that may have seemed like purposes, such as ensuring replicability or representing a population. This was intentional in order to maintain a space for purposes outside the typical goals of QR. Although researchers operating with theories of representation and correspondence may propose that their purpose is to singularly represent and establish correspondence among QR and the world (see *Holland and Albrecht 2013), we reject this conceptualization as a purpose. Indeed, substantial scholarship in management and elsewhere notes that such a purpose often decouples QR from worldly problems (see *Buchholz and Rosenthal 2008).
If JBE and other journals aim ‘to improve the human condition’ (*Michalos 1988, p. 1), then the primary purpose of research should in some way be centered on humans and their conditions. As such, we propose that developing and deploying specific descriptions of the world should be considered an orientation associated with a way of doing QR, which may work more or less well for achieving specific worldly purposes. The point is that any notion of purposes must be connected to worldly affairs rather than merely the goals of a research community. For example, psychologists may propose their purpose is to represent cognition or emotions; sociologists may propose their purpose is to represent institutions or social aggregates; economists may propose their purpose is to represent preferences and rationality, whereas critical scholars or social theorists may propose their purpose is to show power relations. However, by focusing on the production of images of the world that are consistent with the values and outlooks of each community, the central purpose of research as an effort to address relevant worldly problems on practical terms may end up being absent or ignored.
Therefore, without being overly restrictive regarding what a relevant or valued purpose may be, we propose that researchers should attempt to understand their activities and the discourse they produce—including their ‘observations’—as ways of being oriented rather than as purposes. This decouples purposes from specific notions of what is being represented in research, so that representations can appear as orientations that can work more or less well to achieve valued purposes. The net result, we hope, will be the liberty and freedom required to put valued purposes into action through QR and other forms of research, focusing debates on precisely what a research community’s or an individual QR project’s purposes should be, rather than having to filter this through a single way of being oriented toward the world vis-à-vis any narratives of representation and/or correspondence.
To this end, we propose that a valued purpose for QR at JBE might be to inquire about what kinds of orientations and ways of doing QR seem to work for realizing different purposes. Some insight has already been generated into the issues that business scholars care about (see *Holland and Albrecht 2013), but how purpose can be understood and put into QR practice has received little attention because of a focus on concepts such as QR ‘validity,’ ‘objectivity,’ ‘evidence,’ or ‘rigor’ on terms that we have critiqued in this paper. By focusing more directly on worldly problems that matter, and allowing these to be a central purpose for doing QR, quantitative researchers at JBE and elsewhere may be better placed to achieve the ethical possibilities of their orientations and ways of doing QR.
It is for this purpose that we have written this paper. However, for this vision to be developed a monumental shift in what many QR practitioners care about must occur. Instead of merely attempting to produce representations that correspond more or less well—as if the goal of research was to setup and manage a xerox copying facility—QR must be done in relation to matters of serious concern, including inequality, global warming, corruption, and the like. Assuming that such problems are merely being represented will not have the effect that relationally valid QR will ultimately produce if it is participating or intervening in reality by enacting a world that is somehow better than the one which currently presents itself to researchers for study. We hope this possibility provides enough motivation for quantitative researchers to critically question and begin to change their practices, and we welcome the ensuing discussion and debate that our paper and its recommendations invite.
This research was supported by Australian Research Council’s Future Fellowship scheme (project FT140100629).
|Funder Name||Grant Number||Funding Note|
|Australian Research Council|