Cognitive science is an interdisciplinary enterprise. That is, indeed, the entire point of distinguishing it from cognitive psychology narrowly conceived. Standard histories of cognitive science present it as a union of elements of psychology, computer science, linguistics, and philosophy of mind that was forged in the 1960s and 1970s as researchers acknowledged deep structural isomorphisms among information processing implementation in people, non-human animals, and built computers. Cognitive science arguably began on the basis of a relatively unified theory, the mathematical account of computation (Lewis and Papadimitriou 1981) combined with the then-nascent ontology of effective algorithms for Von Neumann architectures (Knuth 1968). However, by the end of the 1980s rooms were opening in the interdisciplinary mansion that relied on other foundations: parallel distributed processing as a generalization of perceptron architectures (Rummelhart et al 1986), behavior-based robotics (Brooks 2013), swarm intelligence (Bonabeau et al 1999), artificial life (Langton 1989), cognitive neuroscience (Trehub 1991, Churchland and Sejnowski 1992), behavioral economics (Kagal et al. 2007), (cross-species) comparative psychology (Griffin 1984), social–ecological cognition (Byrne and Whiten 1988), developmental cognition (Wellman 1990), and consciousness studies (Baars 1988; Dennett 1991). Contributing disciplines have expanded to include neuroscience, ethology, information science, economics, and sociology.
In this teeming research ecosystem, prospects for a general theory of cognition must now be regarded as exceedingly remote. Most contemporary philosophers of science would urge a relaxed attitude to this fact of life: The history of science is not mainly a story of intertheoretic reduction or unification, occasional triumphs of deep synthesis notwithstanding (Mitchell 2009). Newton’s great unification that marked the exemplary scientific achievement for a couple of centuries was in this respect a false portent. In one respect, the picture of science, and cognitive science in particular, as an archipelago instead of a tower makes the Popperian attitude more plausible, since only relatively isolated theories can face the tribunal of experience straightforwardly. A relatively unified edifice would be a densely connected ‘web of belief’ in which each piece would gain support from the rest and none might generate clear predictions that could be distinctively associated with them (Quine and Ullian 1970). The Popperian attitude could in such a hypothetical context only be easily maintained with regard to the foundations of the entire multi-discipline, and falsificationism would then amount to trying to provoke Kuhnian revolution. That was certainly not Popper’s conception. He would thus likely be encouraged by the state of play in cognitive science, where relatively isolated theories do periodically get rejected based on fresh evidence.
However, cognitive science is characterized by a higher order of complexity than the divergence of foundations on which its proliferation of theories rests. Contributing disciplines differ from one another in what their practitioners take theory to mean, in the operational sense. This, I suggest, is one of two key points in diagnosing the conflicting responses of Kirsch’s referees.
All scientists recognize that they must simplify their descriptions of reality in trying to produce generalized knowledge. After all, we have one perfectly accurate model of the world available to us, namely the world itself, but mute holistic appreciation is not scientific knowledge. We need to do analysis, and analysis involves idealized abstraction. Broadly speaking, we find two general strategies for going about this. Most disciplines combine the strategies to some extent. But for concreteness of description, and with a view to the specific methodological tensions that characterized the reviewing of Kirsch’s paper, I will illustrate the contrast by comparing psychology and economics as disciplines that are each unusually pure in overwhelmingly emphasizing one abstracting strategy or the other.
Psychologists seek generalizations about processes they typically cannot directly observe, but must infer from measurable effects. Their preferred methodology for trying to achieve stability and convergence in the face of this challenge is to have developed, and to enforce through peer review mechanisms, a close record of relationships between conservatively extended experimental protocols and theoretical constructs anchored to measurement scoring systems. Most readers of Cognitive Processing will be skilled operators and guardians of this record. The approach is derived from the venerable craft of clinical diagnosis, where practitioners have long needed to control subjectivity in conjecturing underlying diseases and syndromes from manifest symptoms. To this, the psychologist adds statistical testing of instrument and protocol reliability and validity with respect to construct application. Psychometrics is, to a first approximation, the statistical theory of measures of construct validity.
In this context, psychological theories are essentially hypotheses about which constructs are implicated in the production of which behaviors. Hypothesis testing very naturally recruits the Popperian attitude as a rationale for practical features of methodology. Identifying a null hypothesis and then designing an experiment that might refute it is falsificationism at work.Footnote 1 This encourages psychologists to develop highly trained and sensitive attunement to potential confounding causal influences identified ex ante during experimental design. In general, psychologists prefer that confounds that cannot be straightforwardly controlled in linear regressions should be shut out of the laboratory.
This in turn inclines psychologists away from trying to discover ‘laws of behavior.’ Many psychologists explain their aversion to large-scale generalizations by noting that human and animal behavior (and now even behavior of neural networks and multi-agent simulations) is non-deterministic. But various other disciplines (climatology, for example, and molecular genetics) are comfortable with broad-sweep generalizations that are stochastic. The work of the philosopher of science Nancy Cartwright points to a deeper explanation of psychologists’ scrupulous modesty: To the extent that a laboratory environment is designed to protect phenomena from the intrusive confounds that contaminate the wild environment, putative laws governing the messy outside seem like ‘lies’ (Cartwright 1983, 1989, 1999).
Let us now contrast the methodology of psychology with that of economics. To keep the contrast within useful bounds, I will consider only laboratory-based experimental economics. The language of ‘constructs’ is foreign to economists; referential terms in their models are taken to be directly isomorphic to real objects and processes. Of course, economists must simplify and idealize causal relationships just like all scientists. Their vehicle for performing this is the structural model. Most of the practical art of experimental economics consists in developing tasks for experimental subjects with incentives for behavior provided in such a way that the model can identify predicted changes in behavior that vary with changes in the experimenter-controlled incentives. Apprentice economists learn from experience how to design models with ‘the right’ degree of parametric structure: Too few parameters will misspecify phenomena, generating ‘specification error,’ and too many will undermine the likelihood of successful identification. Econometrics is the statistical theory of model specification and estimation.
‘Theory,’ for economists, does not refer to hypotheses about empirical relationships. In some respects, economists think of ‘theory’ in the way that mathematicians do: Theory is simply precise specification. However, in light of their methodology economists use theory at two levels, to construct two different kinds of models. At one level of theory are economic models that specify causal channels (or, in more metaphysically humble language that many economists prefer, channels of ‘influence’) in the world, and at another level are data models that specify relationships between observations and inferences that can be made about estimated variable coefficients under different (usually nonlinear) regression models. ‘Theory,’ then, refers to formal structural specification of some class of systems that are, relative to some robustness criterion across models (see below), an equivalence class. When one economist doubts that another economist’s model is ‘economically significant,’ the former will not complain about a ‘rejected hypothesis’; she might instead say that the project she criticizes is an exercise in ‘mere theory.’
Because economists estimate structural models, any unobserved influence that is correlated with any variable in the model amounts to a specification error and will generate biased estimation of regression coefficients. Therefore, when economists think of possibly causally relevant factors during experimental design, they ideally set up a new treatment group arm of the experiment where the factor in question can be independently varied. If observation conditions or budget limitations prevent this first-best approach, they will revisit their basic design and their data modeling to ensure that the factor can be identified and its influence estimated. So, whereas psychologists seek to exclude ‘confounds’ from the laboratory, economists come up with strategies to bring ‘confounds’ into the laboratory. The psychologist’s laboratory is a bunker; the economist’s is intended to be a microcosm of the world.
Economists are as reluctant as psychologists to speculate about sweeping laws. However, in the case of the former this really does mainly just reflect metaphysical squeamishness about the idea of stochastic causation. (The great philosopher of science C.S Peirce advised against such anxiety; but the idea that a ‘real’ cause must always exert its effects unless blocked by another identified cause is a bit of folk metaphysics that holds on even among many physicists; see Ladyman and Ross 2013.) Economists show their relative immodesty (by comparison with psychologists) about making inferences from data in the attention they give to estimating cross-experimental robustness of observed effect strengths relative to classes of data models (Neumayer and Plümper 2017). Their concern with establishing ‘external validity’ of experiments (Guala 2005) reflects their practice of trying to proxy the world in their laboratories, rather than trying to eliminate all but a selected aspect of the world from the laboratory.
Psychologists’ emphasis on isolated hypothesis testing gives a different flavor to their empiricism from that of economists. Economists’ practice of accepting observed effects because they can successfully specify and identify them, and then accepting them as general causal structures when they can specify, identify and estimate them robustly, is in much greater tension with the Popperian attitude. The leading economic methodologist of the mid-twentieth century, Mark Blaug (see Blaug 1980) was a committed Popperian and caused a generation of economics PhD students to be made to read selections from Popper and Lakatos. It is noteworthy that Blaug ultimately grew deeply dissatisfied with his discipline, and in his final writings (e.g., Blaug 2002) denounced its mainstream practitioners for seldom conducting themselves as Popper counseled. The later Blaug can be interpreted without much strain as wishing that economists did psychology instead. This proves, as it were, the force of the contrast.
Kirsch’s interest in specifying broad classes of heuristics that might characterize various cognitive systems, including humans, at an abstract level but perhaps not in specific detail, would not surprise researchers who approach the world in the way that economists do. Because they are not shy about estimating ‘externally valid’ effects, and because they are motivated to find the broadest range of phenomena that are equivalence classes from the point of view of a model or family of models, it would be natural for an economist to be interested in heuristics that might restrict choice behavior in both people and various kinds of artificial systems, even if processing details varied from instance to instance.
It may seem odd, even to a non-psychologist, to talk about shared heuristics that might yet involve varying processing details. After all, a heuristic is a restriction on processing in the first place. But specification can occur at any of a multiple range of scales of exactness. Suppose that, as a heuristic for not (indefinitely) failing to notice e-mails I every day review all messages still in my inbox from 1 month ago. Suppose that you instead do a daily review of every message from 2 months ago. Are we using different heuristics or two variants of the same heuristic? Let us change the dimension of imagined processing difference: Suppose I scan the set of old messages manually, while you subject exactly the same set to search by a program you wrote that checks to see whether you acted on them. Are those different heuristics because they implement different algorithms, or the same heuristic because they address the same problem, unreliable attention under pressure from interruptions, by the same general device, a review anchored to the calendar date?
Heuristics have in fact been studied by both psychologists and economists, sometimes acting in interdisciplinary teams (see, e.g., chapters in Gigerenzer and Selten 2001). All are interested in the efficiency of the heuristics they study. Efficiency, after all, is, on some construal or other, what makes a heuristic a heuristic. But psychologists typically concentrate on a heuristic’s efficiency in terms of how much computational effort it requires compared with alternatives, whereas the economists are more likely to focus on the ex ante reliability with which rival heuristics get the same job done. These different loci of attention will tend to lead to systematically different levels of specification of equivalence classes. The psychologist will naturally discriminate at a relatively granular scale and seek to test hypotheses that distinguish between variants at this scale, while the economist might aim to model the widest set of implementations that are, within the feasible set given the time or energy budget (i.e., the budget that rules out the first-best solution and motivates the heuristic approach), equally reliable as solutions to a common problem.