Empiricism, sciences, and engineering: cognitive science as a zone of integration

Abstract

An article by Alexandra Kirsch accepted for publication in Cognitive Processing occasioned debate among reviewers about broad methodological issues in cognitive science. One of these issues is the proper place of Popperian falsificationism in the interdisciplinary cluster. Another is the tension between abstract models and theories that apply to wide classes of cognitive systems, and models of more restricted scope intended to predict specifically human patterns of thought and behavior. The lead editorial in a Commentary debate invited by the journal’s editors considers these issues from the perspective of a pragmatist philosophy of science inspired by Herbert Simon’s classic (The sciences of the artificial (2nd edition 1981; 3rd edition 1996), MIT Press, Cambridge, 1969) reflections on the blurring of the distinction between science and engineering in cognitive science.

Introduction

Alexandra Kirsch’s submission of her article ‘A unifying computational model of decision making’ (2019) raised interesting policy issues for the editors of Cognitive Processing. These were triggered by reports from peer reviewers that diverged sharply in their recommendations, but did not revolve around the usual matters of referee judgment, that is, factual accuracy, and soundness of data interpretation and analysis. Instead, different reviewers defended conflicting philosophical views about what constitutes a genuine contribution to knowledge in cognitive science, and about the proper scope of a journal with the specific mission of advancing knowledge of (mainly) natural information processing. This unusual circumstance offered the editors an interesting opportunity to reflect on the challenges of integrating the disciplinary strands that make up cognitive science. The Editor-in-Chief consequently requested the Handling Editor to initiate a Commentary debate that begins with the perspective of Kirsch’s referees, but will subsequently be open to further contributions from interested readers of the journal.

Methodological disputes are ultimately resolved by the empirical progress of science, not by philosophical arguments. However, reflection on such arguments can efficiently guide research design and coordinate the collective critical interpretation of new findings. Such meta-scientific awareness is often particularly fruitful in scientific research clusters, such as cognitive science, that entangle various disciplinary norms and traditions.

I will not here review or summarize the reviewers’ perspectives as stated in their respective reports. Nor will I offer any critical view of the modeling or argumentation in Kirsch’s article, the scientific value of which will be measured, in the usual fashion, by the extent of its influence on further research. My purpose is instead to place the divergence of perspectives revealed by the refereeing process in the context of a large relevant literature in the philosophy of science generally, and in the philosophy of cognitive science specifically.

I proceed as follows. Section “Falsifiability and empiricism” reflects on a ‘Popperian’ critical assessment frame suggested by one of Kirsch’s reviewers. I will not pronounce for or against the status of this frame as a guide to how science ‘should’ be done; it is inadvisable for philosophers of science to position themselves as arbiters of ‘proper’ science, a role which no one has authority to concede. Instead, I will use these thoughts in the shadow of Popper as a basis for discussing, in section “Theory across disciplines,” the heterogeneity of conceptions of ‘theory’ in different rooms of the cognitive science mansion. These differences interact with orthogonal distinctions that sort scientific contributions along a continuum from relatively purely descriptive to primarily practical. This contrast, between ‘pure science’ and ‘engineering,’ will be the subject of section “Science and engineering.” Cognitive science incorporates work all along the continuum, even in those parts of it that focus on natural, as opposed to built, cognitive processes. Section “Conclusion” concludes.

The discussion to follow was reviewed by the other participants in the present Commentary debate prior to their preparation of their contributions. The Editors of Cognitive Processing invite and encourage further opinions from scientific readers, for consideration for publication in forthcoming issues.

Falsifiability and empiricism

The most famous philosopher of science among scientists is certainly Sir Karl Popper (1902–1994). One reason for this is that Popper was particularly concerned to demarcate science from other activities, especially from ‘pseudo-sciences’ that try to illegitimately borrow the epistemic authority that is institutionally conferred upon scientists. Most scientists are called upon from time to time to indicate at least rough demarcation criteria: Students receiving their first scientific training often ask for such markers, and in a world where highly visible and influential political actors constantly incite people to ignore or reject inconvenient scientific findings, a socially responsible scientist must have something to say when asked why her professional views should be given special weight. Popper’s demarcation criterion, though strictly negative, has enjoyed a hugely successful mimetic history. According to it, a necessary condition for a statement, hypothesis, or generalization to be scientific is that it comes attached with specified potential empirical observations that, if discovered through a replicable procedure, would demonstrate the statement, hypothesis, or generalization in question to be untrue. This is the requirement that a scientific proposition be falsifiable (Popper 1969). Popper does not demand that the scientist currently be practically able to perform a falsification test or position herself for a decisive observation. But her claims must imply a difference to some clearly identified empirically contingent state of affairs.

Few contemporary philosophers of science still baldly pronounce themselves ‘Popperians.’ I will not review the numerous respects, identified in the philosophical literature, in which Popper’s view is too simple, too sweeping, gives too much weight to unrepresentative exemplars from classical physics, and if taken literally would exclude too much activity that is obviously scientific. Interested non-philosophers can find a good short review of these problems in Catton (2004). At the very least, a viable Popperian philosophy must incorporate responses to Kuhn’s (1962) historicism, which was the project launched by Lakatos (1970) and subsequently extended by numerous philosophers of science.

Notwithstanding the complexities involved in what preoccupies philosophers, namely trying to exactly state a viable formulation of falsificationism, contemporary scientists often defend and impress upon their students a general ‘Popperian attitude’ that is unarguably a core part of scientific culture. Setting aside pure mathematics as a domain that receives its own extensive philosophical consideration, theory or speculation that floats free of empirical implications is not scientific. The relationship between a body of theory and its empirical evidence can be complex, be non-obvious, or take years to clearly establish, but it is a relationship that no scientist is institutionally allowed to disregard. This principle plays an important role in real disputes over the allocation of scientific resources. For example, critics of string theory as a unifying foundation for physics argue that it is mathematics for which no specific physical interpretation is ever likely to be established on the basis of experiment. Many physicists believe that if at some point, string theorists cannot demonstrate that this charge was premature, then physics departments should stop hiring them (Smolin 2006; but see Dawid 2014).

In my view, the most serious problem with consistently applying a generic Popperian attitude is that the borderline between mathematics and empirical science is both indeterminate and heterogeneous across disciplines. But fuzzy boundaries are still boundaries, as long as one can identify clear exemplars of work on either side of them. When I referee papers in economics, I sometimes recommend rejection on grounds that a piece of work, however elegant, is economically empty mathematics, and I do this with confidence despite recognizing that there are other papers about which a similar judgment would be irresolvably debatable.

Let us now descend from the rarified heights of philosophical generalization to more focused concerns around trying to give the Popperian attitude (not the literal Popperian dogma) its due in cognitive science.

Theory across disciplines

Cognitive science is an interdisciplinary enterprise. That is, indeed, the entire point of distinguishing it from cognitive psychology narrowly conceived. Standard histories of cognitive science present it as a union of elements of psychology, computer science, linguistics, and philosophy of mind that was forged in the 1960s and 1970s as researchers acknowledged deep structural isomorphisms among information processing implementation in people, non-human animals, and built computers. Cognitive science arguably began on the basis of a relatively unified theory, the mathematical account of computation (Lewis and Papadimitriou 1981) combined with the then-nascent ontology of effective algorithms for Von Neumann architectures (Knuth 1968). However, by the end of the 1980s rooms were opening in the interdisciplinary mansion that relied on other foundations: parallel distributed processing as a generalization of perceptron architectures (Rummelhart et al 1986), behavior-based robotics (Brooks 2013), swarm intelligence (Bonabeau et al 1999), artificial life (Langton 1989), cognitive neuroscience (Trehub 1991, Churchland and Sejnowski 1992), behavioral economics (Kagal et al. 2007), (cross-species) comparative psychology (Griffin 1984), social–ecological cognition (Byrne and Whiten 1988), developmental cognition (Wellman 1990), and consciousness studies (Baars 1988; Dennett 1991). Contributing disciplines have expanded to include neuroscience, ethology, information science, economics, and sociology.

In this teeming research ecosystem, prospects for a general theory of cognition must now be regarded as exceedingly remote. Most contemporary philosophers of science would urge a relaxed attitude to this fact of life: The history of science is not mainly a story of intertheoretic reduction or unification, occasional triumphs of deep synthesis notwithstanding (Mitchell 2009). Newton’s great unification that marked the exemplary scientific achievement for a couple of centuries was in this respect a false portent. In one respect, the picture of science, and cognitive science in particular, as an archipelago instead of a tower makes the Popperian attitude more plausible, since only relatively isolated theories can face the tribunal of experience straightforwardly. A relatively unified edifice would be a densely connected ‘web of belief’ in which each piece would gain support from the rest and none might generate clear predictions that could be distinctively associated with them (Quine and Ullian 1970). The Popperian attitude could in such a hypothetical context only be easily maintained with regard to the foundations of the entire multi-discipline, and falsificationism would then amount to trying to provoke Kuhnian revolution. That was certainly not Popper’s conception. He would thus likely be encouraged by the state of play in cognitive science, where relatively isolated theories do periodically get rejected based on fresh evidence.

However, cognitive science is characterized by a higher order of complexity than the divergence of foundations on which its proliferation of theories rests. Contributing disciplines differ from one another in what their practitioners take theory to mean, in the operational sense. This, I suggest, is one of two key points in diagnosing the conflicting responses of Kirsch’s referees.

All scientists recognize that they must simplify their descriptions of reality in trying to produce generalized knowledge. After all, we have one perfectly accurate model of the world available to us, namely the world itself, but mute holistic appreciation is not scientific knowledge. We need to do analysis, and analysis involves idealized abstraction. Broadly speaking, we find two general strategies for going about this. Most disciplines combine the strategies to some extent. But for concreteness of description, and with a view to the specific methodological tensions that characterized the reviewing of Kirsch’s paper, I will illustrate the contrast by comparing psychology and economics as disciplines that are each unusually pure in overwhelmingly emphasizing one abstracting strategy or the other.

Psychologists seek generalizations about processes they typically cannot directly observe, but must infer from measurable effects. Their preferred methodology for trying to achieve stability and convergence in the face of this challenge is to have developed, and to enforce through peer review mechanisms, a close record of relationships between conservatively extended experimental protocols and theoretical constructs anchored to measurement scoring systems. Most readers of Cognitive Processing will be skilled operators and guardians of this record. The approach is derived from the venerable craft of clinical diagnosis, where practitioners have long needed to control subjectivity in conjecturing underlying diseases and syndromes from manifest symptoms. To this, the psychologist adds statistical testing of instrument and protocol reliability and validity with respect to construct application. Psychometrics is, to a first approximation, the statistical theory of measures of construct validity.

In this context, psychological theories are essentially hypotheses about which constructs are implicated in the production of which behaviors. Hypothesis testing very naturally recruits the Popperian attitude as a rationale for practical features of methodology. Identifying a null hypothesis and then designing an experiment that might refute it is falsificationism at work.¹ This encourages psychologists to develop highly trained and sensitive attunement to potential confounding causal influences identified ex ante during experimental design. In general, psychologists prefer that confounds that cannot be straightforwardly controlled in linear regressions should be shut out of the laboratory.

This in turn inclines psychologists away from trying to discover ‘laws of behavior.’ Many psychologists explain their aversion to large-scale generalizations by noting that human and animal behavior (and now even behavior of neural networks and multi-agent simulations) is non-deterministic. But various other disciplines (climatology, for example, and molecular genetics) are comfortable with broad-sweep generalizations that are stochastic. The work of the philosopher of science Nancy Cartwright points to a deeper explanation of psychologists’ scrupulous modesty: To the extent that a laboratory environment is designed to protect phenomena from the intrusive confounds that contaminate the wild environment, putative laws governing the messy outside seem like ‘lies’ (Cartwright 1983, 1989, 1999).

Let us now contrast the methodology of psychology with that of economics. To keep the contrast within useful bounds, I will consider only laboratory-based experimental economics. The language of ‘constructs’ is foreign to economists; referential terms in their models are taken to be directly isomorphic to real objects and processes. Of course, economists must simplify and idealize causal relationships just like all scientists. Their vehicle for performing this is the structural model. Most of the practical art of experimental economics consists in developing tasks for experimental subjects with incentives for behavior provided in such a way that the model can identify predicted changes in behavior that vary with changes in the experimenter-controlled incentives. Apprentice economists learn from experience how to design models with ‘the right’ degree of parametric structure: Too few parameters will misspecify phenomena, generating ‘specification error,’ and too many will undermine the likelihood of successful identification. Econometrics is the statistical theory of model specification and estimation.

‘Theory,’ for economists, does not refer to hypotheses about empirical relationships. In some respects, economists think of ‘theory’ in the way that mathematicians do: Theory is simply precise specification. However, in light of their methodology economists use theory at two levels, to construct two different kinds of models. At one level of theory are economic models that specify causal channels (or, in more metaphysically humble language that many economists prefer, channels of ‘influence’) in the world, and at another level are data models that specify relationships between observations and inferences that can be made about estimated variable coefficients under different (usually nonlinear) regression models. ‘Theory,’ then, refers to formal structural specification of some class of systems that are, relative to some robustness criterion across models (see below), an equivalence class. When one economist doubts that another economist’s model is ‘economically significant,’ the former will not complain about a ‘rejected hypothesis’; she might instead say that the project she criticizes is an exercise in ‘mere theory.’

Because economists estimate structural models, any unobserved influence that is correlated with any variable in the model amounts to a specification error and will generate biased estimation of regression coefficients. Therefore, when economists think of possibly causally relevant factors during experimental design, they ideally set up a new treatment group arm of the experiment where the factor in question can be independently varied. If observation conditions or budget limitations prevent this first-best approach, they will revisit their basic design and their data modeling to ensure that the factor can be identified and its influence estimated. So, whereas psychologists seek to exclude ‘confounds’ from the laboratory, economists come up with strategies to bring ‘confounds’ into the laboratory. The psychologist’s laboratory is a bunker; the economist’s is intended to be a microcosm of the world.

Economists are as reluctant as psychologists to speculate about sweeping laws. However, in the case of the former this really does mainly just reflect metaphysical squeamishness about the idea of stochastic causation. (The great philosopher of science C.S Peirce advised against such anxiety; but the idea that a ‘real’ cause must always exert its effects unless blocked by another identified cause is a bit of folk metaphysics that holds on even among many physicists; see Ladyman and Ross 2013.) Economists show their relative immodesty (by comparison with psychologists) about making inferences from data in the attention they give to estimating cross-experimental robustness of observed effect strengths relative to classes of data models (Neumayer and Plümper 2017). Their concern with establishing ‘external validity’ of experiments (Guala 2005) reflects their practice of trying to proxy the world in their laboratories, rather than trying to eliminate all but a selected aspect of the world from the laboratory.

Psychologists’ emphasis on isolated hypothesis testing gives a different flavor to their empiricism from that of economists. Economists’ practice of accepting observed effects because they can successfully specify and identify them, and then accepting them as general causal structures when they can specify, identify and estimate them robustly, is in much greater tension with the Popperian attitude. The leading economic methodologist of the mid-twentieth century, Mark Blaug (see Blaug 1980) was a committed Popperian and caused a generation of economics PhD students to be made to read selections from Popper and Lakatos. It is noteworthy that Blaug ultimately grew deeply dissatisfied with his discipline, and in his final writings (e.g., Blaug 2002) denounced its mainstream practitioners for seldom conducting themselves as Popper counseled. The later Blaug can be interpreted without much strain as wishing that economists did psychology instead. This proves, as it were, the force of the contrast.

Kirsch’s interest in specifying broad classes of heuristics that might characterize various cognitive systems, including humans, at an abstract level but perhaps not in specific detail, would not surprise researchers who approach the world in the way that economists do. Because they are not shy about estimating ‘externally valid’ effects, and because they are motivated to find the broadest range of phenomena that are equivalence classes from the point of view of a model or family of models, it would be natural for an economist to be interested in heuristics that might restrict choice behavior in both people and various kinds of artificial systems, even if processing details varied from instance to instance.

It may seem odd, even to a non-psychologist, to talk about shared heuristics that might yet involve varying processing details. After all, a heuristic is a restriction on processing in the first place. But specification can occur at any of a multiple range of scales of exactness. Suppose that, as a heuristic for not (indefinitely) failing to notice e-mails I every day review all messages still in my inbox from 1 month ago. Suppose that you instead do a daily review of every message from 2 months ago. Are we using different heuristics or two variants of the same heuristic? Let us change the dimension of imagined processing difference: Suppose I scan the set of old messages manually, while you subject exactly the same set to search by a program you wrote that checks to see whether you acted on them. Are those different heuristics because they implement different algorithms, or the same heuristic because they address the same problem, unreliable attention under pressure from interruptions, by the same general device, a review anchored to the calendar date?

Heuristics have in fact been studied by both psychologists and economists, sometimes acting in interdisciplinary teams (see, e.g., chapters in Gigerenzer and Selten 2001). All are interested in the efficiency of the heuristics they study. Efficiency, after all, is, on some construal or other, what makes a heuristic a heuristic. But psychologists typically concentrate on a heuristic’s efficiency in terms of how much computational effort it requires compared with alternatives, whereas the economists are more likely to focus on the ex ante reliability with which rival heuristics get the same job done. These different loci of attention will tend to lead to systematically different levels of specification of equivalence classes. The psychologist will naturally discriminate at a relatively granular scale and seek to test hypotheses that distinguish between variants at this scale, while the economist might aim to model the widest set of implementations that are, within the feasible set given the time or energy budget (i.e., the budget that rules out the first-best solution and motivates the heuristic approach), equally reliable as solutions to a common problem.

Science and engineering

In her paper, Kirsch cites Herbert Simon’s classic book The Sciences of the Artificial, originally published in (1969) and revised twice subsequently (1981, 1996). Simon’s central question was how there could be a proper science of designed systems, as opposed to simply engineering of them. If we think of science as the disinterested examination of the natural world—whereby ‘natural’ people typically mean ‘independent of design by people’—then it is indeed puzzling that we might scientifically investigate systems in which the characteristics were chosen by people in the first place.

What the class of designed systems that Simon discusses have in common is that they are intended to optimize, or to reach a performance threshold short of optimization that a designer deems good enough, in conditions that involve adaptive interface with a class of environments. By way of contrast, the problem of building an excellent toaster or car is a relatively pure engineering problem: The only restriction on making it as good as you want is cost, because neither toasters nor cars are very sensitive to varying ‘interface conditions.’ This is not true of AI systems, or robots, or artificial markets.

There are generalizations to be discovered about adaptive interface. Indeed, there are limitless potential such generalizations because adaptive interfaces are scale sensitive and there are no in-principle limits to the scales at which an inquirer might aim her interest and try to solve her design problem. Discovering these generalizations is a matter for science because scale effects are typically nonlinear, and the locations of transition and inflection points need to be found through theoretically disciplined experimentation.

The sciences of the artificial blend into (some) sciences of the natural, because biologists are also in the business of seeking generalizations about adaptive interfaces. It might be said that the systems studied by biologists—and, as a special case, by psychologists—are different because they were not designed. Of course, it is important to stress this if by ‘designed’ we mean ‘assembled with a preconceived end in view.’ The idea that biological systems were designed in that sense is creationism, and anathema to science. But, as the philosopher of cognitive science Daniel Dennett (1987, 1995) has emphasized, natural selection is a kind of designer in the very important sense that it conserves better solutions to problems and tends to eliminate inferior solutions to problems. Furthermore, nature’s solutions often offer vital clues to engineers because natural selection runs prodigiously long and varied experiments, albeit with only rare instances where treatments and controls are isolated.

The domain of heuristics, Kirsch’s topic, is the perfect point of convergence of the sciences of the artificial and the biological (psychological). A solution to a problem is a heuristic just in case it trades off some accuracy in completeness of representation and/or cross-domain reliability for reduced cost (in energy or time) of processing. That is the only kind of solution that natural selection can produce in principle. So all of psychology is a study of heuristics. But I said above that the boundary between the natural and the artificial, in Simon’s sense, is not clean or clear. The environment in which the specifically human mind has evolved is a social environment, teeming since the great encephalization with engineered artifacts. Humans are niche constructors on an unprecedented scale (Odling-Smee et al 1996; Sterelny 2003), so the dance of adaptation between their minds and their environments is coevolutionary.

A matter of awkwardness for Kirsch’s referees is that her paper is vertiginously poised at what seems to be an unstable or arbitrary point of engagement on this fuzzy borderline between the naturally selected and the engineered. She studies a class of heuristics more general than the human mind, while wanting to make sure that human heuristic processes are a proper subclass of her target. But her target class cannot be every possible heuristic process, because that would amount to the unfathomable domain of every possible solution procedure that is budget constrained.

I doubt that it is possible in principle to pre-isolate the class of processes to which Kirsch’s intuitions direct her attention. In section “Theory across disciplines,” I contrasted the psychologist’s methodology with the economist’s. It might seem at first glance that both of these methodologies, in their opposite ways, avoid the arbitrariness that seems to haunt Kirsch’s project. The psychologist, it might be said, restricts herself to the actual heuristics that natural selection produced in humans. Thus, she avoids the problem of an indeterminate boundary in possibility space. The economist, on the other hand, begins from the mathematically identified ideal solution space and systematically introduces costs, according to a rational sequence governed by which solution functions mathematically nest which others. Thus, we have a (radically) different procedure that likewise is safe from charges of arbitrariness. Did Kirsch provoke a muddled refereeing process because her work falls into a ravine between well-motivated disciplinary alternatives?

Conclusion

In fact, I think that neither the psychologist’s apparent solution to the arbitrariness of the target space, nor the economist’s supposed way around it, can actually hold us on the firm dry ground conjured above.

First consider what the psychologist’s ‘solution’ must assume that we can coherently aim to study the heuristics our brains would have run naturally, by themselves, if they could have undergone development in a psychologist’s laboratory isolated from confounds. This is purely fantastical, meaning not merely practically impossible but not conceivable with the slightest degree of specifiable detail. All people have evolved the heuristics they use while entangled from infancy in a network of artifactual ones. We are, as the cognitive scientist Andy Clark (2003) puts it, ‘natural-born cyborgs.’

As for the economist’s solution, the ideal decision-theoretic space is usually thought to be the axiomatic expected utility theory of Savage (1954). However, Binmore (2009) has recently forcefully reminded economists of a point Savage made to begin with (and this time, to judge from the literature, the point is getting through). Application of expected utility theory is only ‘best’ in environments where all ex ante uncertainty can be quantified based on prior knowledge, i.e., by some procedure other than uninformed conjecture. These environments are what Savage called ‘small worlds.’ Humans use heuristics even in many small worlds, at least when these are novel to them, but the very novelty of such applications suggests that natural selection might not have pre-selected them. In any event, almost all of the decision worlds of interest to cognitive scientists—psychologists, AI researchers, and behavioral economists alike—are large worlds. There is no end to the useful economic modeling that can be applied to such worlds, but from the perspective of being able to elegantly pre-configure the boundaries of this study domain economists are in the same position as Kirsch.

So, I suggest, are we all. What is the domain of ‘cognitive processing’? We cannot even really say that we know it when we see it, because sometimes it turns up in places where no one would have expected it, e.g., the arms of octopi (Sumbre et al. 2001) or ‘zero-intelligence’ traders in just the right kind of market (Gode and Sunder 1993). There will be other submissions to Cognitive Processing that cause referees to scratch their heads in different ways. That is editorial life in an adaptively open world.