1 Introduction

Social and evolutionary science emerged at roughly the same time during the nineteenth century, and both soon took a keen interest in each other’s ideas. Sociologists from Marx and Durkheim to modern social philosophers like Popper, Rawls and Rorty were influenced by evolutionary thinking. On the other side, ever since Darwin, evolutionists were eager to apply their theory to human culture. Nonetheless, the gap between the two fields remained wide. Social scientists feared that evolutionists wanted to force the autonomy of human agency and its unique cultural products into the straitjacket of genetic determinism. Biologists, for their part, saw behind this apparent autonomy the hidden workings of inherited fitness effects.

In recent years evolutionary analyses have moved closer to acknowledging the constructive, generative influence of autonomous cultural behaviour on our evolution, and have moved away from the temptation to search for adaptive meaning in all we do. Similarly, sociologists have become less fearful of looking at culture from an evolutionary point of view. However, major disagreements remain. The most important of these was already noticed by Darwin. He was convinced that a single natural process, evolution, created all life, including human beings and their culture. There could be “no doubt that the difference between the mind of the lowest man and that of the highest animal is immense … (but) it is certainly one of degree and not of kind” (Darwin 1981 [1871], pp. 104, 105). At the same time, however, it seemed to him that in human culture “the strangest customs and superstitions, in complete opposition to the true welfare and happiness of mankind, have become all-powerful throughout the world” (Darwin 1981 [1871], p. 99). The apparent discrepancy between the slow, adaptive process of natural selection and the speed of cultural innovation and change, the nature and role of human intent in cultural selection, and the high rate of cultural traits that make neither evolutionary nor functional sense remains a major obstacle to deciding whether culture is an evolutionary or a uniquely human product.

This article tries to find more common ground between an evolutionary theory whose primary focus has been long-term genetic change, and social and historical research that has shown the extraordinary variability of recent human history and culture. Cultural evolution occurs in human lifetimes. It consists of countless variations and changes, many of which are quickly forgotten. Of those that survive longer, some clearly improve the quality of human lives, whereas others cause much harm. In human history benefits and costs seem to be arbitrarily distributed across different human populations. They rarely have a discernible impact on human reproductive success, and where they do, such as in choosing to have fewer children in exchange for economic success, it is often negative. Cultural choices are always selections from a range of alternatives, but they are guided by conscious preferences of cultural actors.

I argue that past efforts to explain human culture as a composite of traits that are direct expressions, extensions, analogues, runaway- or by-products of genetic variation and selective retention, are not sufficient to account for the complexity and volatility of cultural evolution. Instead, we have to look for a more fundamental unifying principle that underlies both genetic and cultural change. The common thread running through evolution is that all organisms, including humans, face the problem of living and surviving in uncertain environments. Evolution solves that problem by increasing the ability of organisms to respond to uncertainty by generating undirected variety in order to maximize the chance of finding opportunities and avoiding threats in an unpredictable world. Blind genetic mutation was an early form of this response. The subsequent evolution of behavioural and cognitive novelty gradually improved the ability of life to cope with uncertainty. The extraordinary variety of human culture is the latest manifestation of this systemic tension between external uncertainty and organismic innovation. I examine how this tension is experienced during the lifetime of organisms.

2 Darwin and Cultural Evolution

Darwin devoted two late works, The Descent of Man (1871) and The Expression of Emotions in Man and Animals (1882), to exploring links between animal and human evolution. Still unaware of the genetic side of variation and inheritance, he based these studies and The Origin of Species on observations of living organisms, including humans, in their natural environments, and on inferences to their lifetime behaviour drawn from collections and fossil remains. He carefully noted the variations in appearance and behaviour that met or collided with changes in the natural world, and the competition for resources that ranged from mortal combat to cooperation and the peaceful occupation of new environmental niches. He saw that adaptation could lead to extreme specialization with the attendant risk of extinction when crucial parts of its environment changed. In other cases, physical or behavioural flexibility allowed organisms to live successfully in an ever-changing world. In addition, evolution produced its share of imperfect solutions, such as non-functional by-products of new adaptations, and adaptations rendered vestigial in new environments or repurposed as workable but suboptimal tools for new ends.

Matches between inherited abilities and environmental opportunities could be entirely fortuitous and coincidental. Often, however, organisms could become active participants in selection. They had to discover, “seize” and “partake of” potential uses of newly inherited abilities and new niches that they found, and turn their discoveries into “habits” during their lifetime (Darwin 1958 [1859], p. 117). Although he was never fully clear about the respective role of “spontaneous variations”, “changes in the conditions of life”, and the effect of “use and disuse”, Darwin was sure that “a little dose of judgement” entered animals’ interaction with their environment early in evolutionary history. “… the senses and intuitions, the various emotions and faculties such as love, memory, attention, curiosity, imitation, reason, &c., of which man boasts, may be found in an incipient, or even sometimes in a well-developed condition, in the lower animals” (Darwin 1981 [1871], p. 105). In the Descent of Man Darwin used sexual selection as the bridge that connected animal to human cognition. These ideas did not sit well with later generations of evolutionists. Darwin’s Descent was politely dismissed as a Lamarckian faux pas committed by a respected author approaching his dotage.

Darwin’s argument that the differences between animal and human culture were of degree, not of kind, found much subsequent support in studies of proto-cultural behaviour in animals. Long suspected of hidden anthropomorphism and still operating on the margins of Neo-Darwinist orthodoxy, this field has shed much light on the beginnings of cultural evolution.Footnote 1 Neo-Darwinism considered cognition, whether animal or human, as a black box where external stimuli were transformed into adaptive behaviour. Tolman’s (1948) cognitive map theory, which broke with Skinnerian behaviourism and marked the beginning of cognitive psychology, argued instead that animals were able to form spatial working memories that enabled them to store location details and use this information to imagine places and situations in ways that went far beyond immediate stimulus-response reactions. Research on animal cognition confirmed this idea (Lee 2023) and provided much evidence of animal cognition in crucial areas of cultural evolution. Tool use by a variety of animal species revealed their understanding of the functional properties of physical materials and their ability to solve complex problems encountered in their environment. Primates retain the facial recognitions of others, use purposive, flexible gestures to respond to others, and anticipate their intentions (Genty et al. 2009). They assess the reliability of information used to make decisions and collect more information to achieve better results (Tomasello 2023). In the area of communication, so important for the evolution of language and the rapid acceleration of human cultural evolution, animals from birds (Suzuki et al. 2016) to primates combine calls into syntax-like semantic structures to communicate situational information to con-specifics (Arnold and Zuberbühler 2006; Berthet et al. 2019; Leroux et al. 2021) and do so intentionally (Schel et al. 2013).

The two perhaps most important dimensions of cultural evolution, creativity and cooperation, are also found among many animals. The ability to find utility in variable and novel surroundings by exploratory probing is highly developed in species such as octopi and New Zealand Keas, which show high levels of curiosity, anticipation of object uses, and understanding of means–ends connections (Huber and Gajdon 2006). An experiment by Lai et al. (2023) showed rats capable of volitionally activating remote place representations, suggesting the presence of intent, imagination and creativity, abilities that were long considered the sole purview of humans. As in human cultures, proto-cultural behaviour is often first acquired during the play of young animals (Palagi et al. 2016). It can be transmitted between generations through learning and socialization and can turn into cultural traditions (Aplin et al. 2015). The fundamental role of cooperation in the evolution of human culture was already noted by Darwin. Its proto-cultural forms are common among many animal species, mostly directed at close genetic kin, but also at unrelated individuals. Bonobos, our closest primate relatives, cooperate frequently and non-opportunistically with unrelated groups (Dugatkin 2002; Samuni and Surbeck 2023). Non-kin cooperation requires recognition of allies and opponents, recall of their prior actions, and understanding of the benefits of collaboration, although such mental processes are difficult to observe (Cheney 2011). In human societies, language, the growth of interaction networks and the invention of communication technologies have vastly increased the range of cooperation (as well as cheating and defection), but its precursors in animals are well documented.

Not all animal culture is adaptive. Contingent changes in physical or social environments can render inherited predispositions redundant or maladaptive, although these can endure for many generations. In addition, maladaptive traits can be correlated with more adaptive ones. Mouth-breeding fish, for instance, often have difficulty breathing and feeding with brood in their mouths and swallow or spit out 40–60% of the fertilized eggs. The limited size of working memory prevents disorientation and information overload but also makes animals like squirrels and jays forget much of the food they hid in the ground or the bark of trees. The rate of redundancy and error during animals’ daily behaviour, such as the occasionally fatal consequence of crows teasing predators, is likely much higher. Unfortunately, we have few prolonged continuous observations of animals in natural settings. Redundant actions had no place in the adaptive economy of Neo-Darwinism, and studies of animal behavior have tended to focus on actions deemed to be adaptive. We therefore do not know whether birds only sing to attract mates, raptors soar on thermals only to look for food, or what seals feel when they surf on California waves. If we did, we might find surprising affinities with their surfing human counterparts.

3 A Narrowing of Perspectives: Social Science and Neo-Darwinism

In view of these similarities between natural and cultural evolution the question is why Darwin’s argument that a single process links all evolution remains a contentious area in biological and social science research.

One reason is the contiguous path of social and evolutionary science in the twentieth century. Both opted for theoretical models that dismissed any intentional role of individuals in biological and cultural evolution. On the side of sociology, its early founders, from Marx to Spencer and Durkheim, wrote in a turbulent century, and the search for social order and stability, either in their own or a future, better society, preoccupied all of them. Darwin had argued that evolution was a pragmatic process that had no immanent direction. Adaptation worked for the good of each being, but created no hierarchies of adaptedness. Early social scientists, by contrast, thought that they discovered in Darwin’s theory the causes of class divisions and the law of progressive social development. Such beginnings left a permanent mark. Modern sociology developed its own positivist and rationalist versions of population thinking. The reasons were similar to those that also inspired the New Synthesis in evolutionary biology: the temptation to bring the apparent precision of the natural to the social sciences in order to make them look more scientific. Durkheim had argued that sociological research should not focus on individual actions but on large-scale “social facts”, measurable patterns of social behaviour in groups and societies. Behind this stood a view of social life as a vastly complex but potentially understandable structure of finite cause–effect relations. The cumulative results of statistical regularities would eventually reveal underlying causal mechanisms. Real individuals vanished from view.

In statistical affairs, the first care before all else is to lose sight of the man taken in isolation in order to consider him only as a fraction of the species. It is necessary to strip him of his individuality to arrive at the elimination of all accidental effects that individuality can introduce into the question (Hacking 1990, p. 81).

Social entities were treated as closed systems in order to allow the verification of hypotheses. The spontaneous, creative side of individual behaviour was dismissed as unexplained statistical variance, and the complexity of social processes, Max Weber’s “myriad of causes”, was simplified by investigating cause–effect relations “net of other variables”. By contrast, real natural and social systems were open systems. Random causation and unpredictable outcomes were common features, and the verification and validation of numerical models was therefore impossible (Oreskes et al. 1994; Glaesser 2023). On the theoretical side of sociology, functional and rational choice theories gave rational social behaviour much the same role as adaptive behaviour in evolution. Fanciful functional value was found in all kinds of social practices and structures, none more so than social inequality. There was no place for irrational conduct, error and unforeseen consequences, and the persistence of dysfunctional or harmful social choices. Today, North American sociology faces an accumulation of often disjointed and unreplicated empirical studies, and its theories seem mired in identity politics.Footnote 2

In evolutionary biology, too, the discovery of genetic mutation and inheritance replaced the lived, observable interplay between organisms and their natural environments with a paradigm where fortuitous variations of discrete genetic units controlled specific behavioural and morphological changes. For the Neo-Darwinist New Synthesis, which began to dominate evolutionary biology in the 1950s, intentional individuals were fictions of the sociological imagination. In reality they were composites of inherited traits. Evolution was measured in terms of the frequency of traits and genes in populations of organisms. For evolutionary psychologists, brains became special-purpose organs, bundles of pre-programmed behaviour modules that filtered information to bring it in line with inherited needs. What appeared to be feelings, emotions, self or consciousness, disguised deeper biological drives. When trying to extend this model to human cultural evolution, Neo-Darwinists encountered the strange customs and superstitions that had already worried Darwin. They were ignored, regarded as irregularities or outliers, or reinterpreted to prove that they were fitness-optimizing behaviours after all.

The modern synthesis … dealt with potentially destabilising new findings … by acknowledging that such processes happen sometimes (subtext: rarely), are useful to some specialists (subtext: obscure ones), but do not fundamentally alter the basic understanding of biology that descends from the modern synthesis (subtext: don’t worry about it, we can continue as before). In short, new discoveries were often dismissed as little more than mildly diverting curiosities. (Buranyi 2022)

More recently, evolutionists have widened their search for extra-genetic influences in evolution, taken a more organism-centred view of variation and selection, and applied these ideas to cultural evolution (e.g. Laland et al. 2014, 2015; Dennett 2017; Lewens 2019; Henrich 2016, 2020). In memetics, culture was viewed as the product of memes, which, as ideas, symbols or composite memeplexes, competed for acceptance and replication by spreading through imitation from person to person. In niche construction theories the ability of organisms to alter their environment was investigated and the selective effects of these changes on their subsequent evolution were examined. In dual-inheritance or gene-culture co-evolution theories this idea was applied to culture, arguing that genetic predispositions encouraged imitation of others who possessed presumably inherited superior ability, skill, or high social status. Learning thus guided cultural variation toward success-, conformist- or frequency-dependent or content biases that enhanced personal and cultural fitness. Copying errors could happen but would not endure. Henrich extended this view by arguing that, just like historical changes in diet affected changes in lactose tolerance or human dental structures, cultural advances such as literacy led to wide-ranging changes in neural structures of the human brain. Similarly, the prohibition of marriages between close kin by the Catholic church replaced early medieval family-based cooperative social structures that were primarily controlled by shame with a more individualistic culture obsessed with success and non-conformism. Here, guilt and self-blame for social failure acted as the main social control dynamic. Both shame and guilt were “genetically evolved psychological package(s)” (Henrich 2020, p. 34). Cultural change thus altered the expression of inherited emotions and laid the groundwork for the cultural and economic success of western industrial societies.

Gene-culture co-evolution theories moved closer to Darwin’s argument that a single evolutionary process underlies genetic and cultural evolution. Nevertheless, important problems remained. First, co-evolution theories have not coped well with the complexity and short-term variation of human culture. The few well-established examples of cultural change affecting human physiology took place over tens of thousands of years. Recent work such as Henrich’s, relies largely on mathematical models that employ a limited number of variables to project possible long-term interactions between genetic and cultural evolution. The difficulty they encounter comes from the extraordinary variety of human culture, and therefore the extraordinary diversity of possible causes. Human culture has evolved at an accelerating rate in the short time since about 5000 BC for which we have detailed historical evidence. For social scientists trying to make sense of this rapidly changing cultural landscape, co-evolution theory had little to offer. This is not the place to examine Henrich’s argument in detail.Footnote 3 The general problem is that complex evolutionary systems, and in particular human culture, are unstable, stochastic and therefore sensitive to initial conditions whose effects are inherently unpredictable. Human history is replete with event sequences where seemingly insignificant events had large, unanticipated consequences. Contingency and chance are pervasive in all evolution, a characteristic that evolutionary modelling and sociological statistics tend to ignore in their pursuit of certainty. It is tempting to contrast Henrich’s 2020 book with Piketty’s Capital in the 21st Century (2014) and Capital and Ideology (2020). All three works are of monumental size, use exhaustive data, and look at long-term cultural change. Henrich corrals a wide range of empirical material into computational conjectures, whereas Piketty lays out a complex web of causes and consequences to trace lineages of social and ideological change. Both pursue similar goals, but Piketty’s books provide a richer source of the contingent twists and turns in the evolution of social patterns and structures.

Second, co-evolution theories have little to say about the origins of evolutionary novelty. The general tenor is that “Our capacity to acquire valuable skills and information from more knowledgeable others, such as parents, teachers, or friends, as well as indirectly via artefacts such as books and computers, furnishes us with a short cut to adaptive (and sometimes maladaptive) behaviour.” (Laland and Brown 2002, p. 249). They do not tell us how these knowledgeable others acquired their valuable skills in the first place, except to say that their “prestige is deeply rooted in exceptional abilities or skills which are unevenly distributed across the members of social groups” (Henrich 2020, pp. 119, 316). Henrich realizes that this leaves the question of the sources of novelty unsolved, but contents himself with noting that “Social learners face a bootstrapping problem of where the initial knowledge comes from.” (Muthukrishna et al. 2018, p. 6). Meme theorists like Blackmore face the same difficulty. The generative power behind creativity is the competition between memes. “Memes fight it out to get passed on into another brain or book or object, and in the process cultural and mental design comes about … There is no need to call on the creative ‘power of consciousness’, for consciousness has no power” (Blackmore 1999, p. 236). For Dennett, at an unknown point evolution changed from a bottom up to a top down “intelligent design” process that marked the beginning of a human culture. “One way or the other, the mind was lifted (by good design) into a region of Design Space from which its further forays were admirably swift and effective”. Its origin “must ultimately be accounted for in the cascade of cranes that had been designed and erected over the last billion years” (Dennett 2017, p. 319). In these views, neither autonomous choice nor social structures has a selective impact on cultural evolution.

Third, the belief by co-evolutionists that cultural evolution has broadly adaptive or socially beneficial outcomes also shapes their view of the models who initiate the biased learning that guides cultural selection. As with the origin of novelty, co-evolution theories provide little concrete information about who these models are. They remain disembodied constructs. We neither find out how they gained their prestigious position nor how they perform their task. Henrich does draw an ideal type picture of WEIRD models. Their “impersonal prosociality is about fairness principles, impartiality, honesty, and conditional cooperation in situations and contexts where interpersonal connections and in-group membership are deemed unnecessary or even irrelevant.” (Henrich 2020, p. 300). In exchange for this selfless service, models can count on the respect of the recipients. “Humans reliably develop emotions and motivations to seek out particularly skilled, successful and knowledgeable models and then are willing to pay deference to those models in order to gain their cooperation, or at least acquiescence, in cultural transmission” (Henrich 2016, p. 119).

Finally, co-evolutionists are not unaware that such harmonious relationships between model and follower could be disturbed by maladaptive or functionally harmful behaviour. The general view, however, is that these are deviations around adaptive means and are of little theoretical interest because they cannot divert cultural evolution from returning to a fitness-enhancing pathway (Wilson 2012, pp. 236–240). Richerson and Boyd devote an entire chapter to explaining the emergence of maladaptive cultural variants, but only as outgrowths of adaptive biases anchored in the imitation of social models (Richerson and Boyd 2005). These ideas provide little guidance to the historian or social scientist trying to understand the recurrence of political domination or social inequities. Hodgson and Knudsen (2010, p. 203) simply conclude that social inequalities “notwithstanding challenges and rebellions … evolved in order to ensure the chances of survival of both the individual and the group”. Memetic theorists pay more attention to the possibility that signs and symbols can be deleterious or play truth or altruism tricks on people, but they do so solely for their own replication. Individuals are mere temporary conglomerations of these replicators (Blackmore 1999, p. 236). For Dennett, the evolution of memes was a gradual triumph of good design. Lesser brainchildren of that process may still “swim in an ocean of semi-intelligent design” of fads, fashions and other memetic parasites (Dennett 2017, pp. 330, 331), but we can see behind this “cultural junk” the “cultural treasures” forged by intelligent design which successful models will bequeath to future generations (Dennett 2017, p. 308). That cultural junk presumably also includes maladaptive aberrations of no evolutionary significance.

4 Contingency and Choice in Human Culture

The sociologist open to evolutionary thinking will find in these ideas some similarities to familiar sociological problems, but they are barely recognizable. The world of sociologists and historians does not fit into this framework. They see societies that undergo often rapid periods of stagnation, growth and destruction, and experienced relatively short-term changes in their environment that had a profound social impact (Büntgen et al. 2011). Sociologists want to know why “not all social norms are equally likely to evolve or remain stable” (Henrich 2020, p. 71), who selects the norms, and how cultural forms start, endure or disappear. They want to know why innovation seems to occur so unpredictably, and why some inventions languish unrecognized while others have entirely unanticipated consequences. The co-evolutionist construct of homogeneous societies of models and deferential followers would strike sociologists as naïve and simplistic. Their research tells them that societies are internally divided and unequal. In real human history the harmonious co-evolutionary relationship between models and obsequious learners all too often increased the wealth and prestige of the model at the expense of the learner, who had no choice but to go along with gritted teeth. Last but not least, sociologists would ask why the many advances during the short human history were accompanied by genocides, wars, environmental destruction, slavery, exploitation and technological failure. They would suspect that such dysfunctional events are not exceptional but integral parts of human culture.

If we want to understand these aspects of culture, and at the same time follow Darwin’s argument that they reflect a single process of evolution, we cannot do so by insisting that human cultural evolution, in whatever roundabout way, must resemble genetic inheritance and have predominantly adaptive or functional results. I want to argue instead that in the course of evolution, variation and selection itself evolved, moving from molecular changes to the inventive behavioural probing of environments by animals, and the eventual transfer of most of the process of cultural variation and selection to internal cognitive processes in the human brain. All of these forms of variation and selection, however, have a common characteristic: the production of blind variety in response to uncertain environments.

In evolution all change is a modification of what already exists. The same applies to ideas. We must therefore acknowledge the work of some post-Darwinian authors who were influenced by Darwin’s hunch that evolving organisms could have some autonomous influence on selection. Of particular importance for understanding the evolution of human culture is the work of William James (1842–1910), John Dewey (1859–1952), Donald Campbell (1916–1996) and Niklas Luhmann (1927–1998). All rejected the idea that human life was shaped by unavoidable external necessity or pre-determined historical trajectories. All saw environments as fluid and constantly changing, confronting individuals with ambiguous possibilities and unpredictable futures. Their writings remained outliers to both Neo-Darwinist and sociological orthodoxy.Footnote 4

Luhmann’s contribution to the analysis of cultural evolution was to show that contingency and unpredictability, which he defined as everything that was “neither necessary nor impossible”, was a fundamental part of human existence. Humans always encountered more options than they could use, and more outcomes than they could predict, a condition that, if left unresolved, would immobilize the individual and make living impossible. Reducing complexity in order to achieve a workable stability in one’s life was therefore a basic necessity for survival. Individually, this could be done by making selective sense of one’s environment. On the societal level, complexity was reduced by creating structures, institutions and legal routines that terminated the potentially endless weighing of alternatives, and guided decision making along known channels. Luhmann’s advance over sociological functionalism and rational choice theories was to show that complexity reduction was essential for making social life manageable, but by no means guaranteed optimal rational or functional results. Social life remained unavoidably encumbered by chance, error and unanticipated consequences.

Campbell recognized that adaptation to such contingent environments was not just a human predicament but had a long evolutionary history. Tolman had already suggested that animals probed their environment by trial and error, and built and memorized spatial maps of their surroundings to find novel routes. Campbell (1965, 1970) showed that such cognitive mapping began as early as the ability of protozoa to use random movements to avoid toxic locations and settle in nutritious ones. Evolution gradually increased organisms’ ability to scan environments vicariously in order to explore their benefits and dangers. Trial and error remained the central method of exploration, but the evolution of vision, hearing, sonar communication, sounds, calls and eventually human language greatly improved both the volume of information and the efficiency of collecting it without having to come into direct contact with its source. Improved tools for gathering information were accompanied by the parallel evolution of cognitive maps that allowed organisms to imagine and assess possible outcomes of choices before they were actually made. Reliance on virtual information and cognitively conceived choice scenarios became particularly important in social environments where the often oblique nature of social signals made impulsive action dangerous. The shift from genetic to cultural selection may in turn have accelerated the rate of anatomical and behavioural changes from primates to humans. Genetic selection did not disappear, but the lifetime adaptation of organisms, and eventually humans, to their natural and social environments was overlaid by cultural selection based on cognitively generated goals.

Prior to Campbell and Luhmann, James had already suggested that the human mind solved the problem of making choices in a contingent world with a constant undirected stream of “new conceptions, emotions, and active tendencies which … are originally produced in the shape of random images, fancies, accidental out-births of spontaneous variation in the functional activity of the excessively instable human brain.” These, he thought, were just like the “morphological and social variations due to molecular accidents of an analogous sort” (James 1880, p. 445).

These random thought components were then used to organize experience and preference into meaning and sense, and to make a “free choice” from alternatives that individuals saw in changing environments. Some of these choices were dictated by natural or social constraints, but this was not Spencer’s survival of the fittest. For James, choice was a creative force that gave rise to extraordinary cultural variety.

The social affections, all the various forms of play, the thrilling intimations of art, the delights of philosophic contemplation, the rest of religious emotion, the joy of moral self-approbation, the charm of fancy and of wit—some or all of these are absolutely required to make the notion of mere existence tolerable (James 1878, p. 13).

Choices offered only temporary stability in a fluid world. Cognition, whether everyday or scientific, was fallible and reversible, combining periods of innovation with periods of provisionally stable knowledge, “flights and perchings”. For Dewey, too, the world was not passively experienced and represented in the mind. Knowing was an active process. Choices were guided by “ends in view”, imagined practical solutions to conflicting experiences in daily life. The mind could store and organize them into habitual modes of response, but unlike fixed adaptive routines they could always be revised. Inflexible choices were of little use. Pragmatic truth emerged as a result of experimental activity to find solutions when uncertainty and problematic situations had created a conflict of meanings.

5 A New Paradigm of Internal Selection

These ideas shed new light on the continuity of genetic and cultural evolution. To understand what unites the two processes we must ask ourselves what it is like to evolve in a contingent world.

The first characteristic of that world is the ubiquity of chance. Darwin’s observation that “a grain in the balance may determine which individuals shall live and which shall die” (Darwin 1958 [1859], p. 433) applies to all evolution. The more social and interactive environments are, the more complex and unpredictable they become. The complexity of human environments increased, first slowly and then at an accelerating rate, as a result of demographic growth, wider social networks, the variability of social communications, and the sheer volume and variety of the products of cultural evolution. Evolution therefore early on favoured organs that allowed more efficient scanning of environments, and brains that permitted an increasingly flexible cognitive anticipation of the potential uses of the information thus gained. In human culture experience, its storage in personal memories, libraries and computers, and formalized in routines and rules, creates an appearance of order and certainty, but risk is never far away. Whether we stay healthy or get sick, whether our marriages are happy or end in divorce, or how our children turn out, are just some of many life experiences that we are unable to predict. Chance is our silent but ubiquitous companion.

Our responses to such uncertainty are thought trials that search for opportunities and dangers in an uncertain environment. In our daily routines at home or at work, they often turn into habitual responses to familiar settings where we do not anticipate change. The trial nature of human thinking becomes more evident when we encounter something unforeseen, or when we wrestle with an unsolved problem. Here, we begin to probe alternatives or look for workarounds. Success is unpredictable. Solutions, discoveries or innovations can come to us suddenly or after a long and torturous process plagued by mistakes and frustrating dead ends. Their consequences often diverge from their intended goals, the more so the longer the timeline we observe. The idea that serendipity plays a significant role in our creativity and achievements challenges human, and especially scientific, self-understanding. The standard view is that creativity is a uniquely human property that has cumulative beneficial effects. In reality, serendipitous beginnings and unanticipated outcomes are as typical of major inventions as of the ordinary discoveries we make in everyday life.

The unpredictability of novelty and its consequences manifests in an irregular and unanticipated historical path of cultural evolution. Cultural evolution is a diversifying, not an end-directed process. Its long-term course closely resembles the phylogenetic trees of natural selection. The historical branching of human languages, from the tentative words of early humans to the estimated 7000 to 8000 human languages known today, follows exactly that pattern. Even in the age of dictionaries and linguistic standardization, the everyday uses of language continue to branch and diversify, and new words emerge and disappear. Cultural selection adds new meaning to existing words, emojis suggest feelings, and abbreviations replace full spelling on social media. What is true for languages is true for the evolution of objects and ideas. Marx observed in 1867 that “In Birmingham alone 500 varieties of hammers are produced, and not only is each adapted to one particular process, but several varieties often serve exclusively for the different operations in one and the same process.” (Marx 1906, p. 375). The same diversifying path describes the evolution of pens, paper clips or zippers (Petroski 1989, 1992; see also Basalla 1988), the emergence of Christianity (Pagels 1981), the evolution of sexual morality in late antiquity and the rise and fall of the Roman empire (Harper 2013, 2017), and the growth of the internet.

Cultural selection retains only a fraction of the variations we produce and leaves behind many more that are redundant, obsolete or injurious. The abundance of such traits in natural and cultural evolution is a severely understudied area of evolutionary research. We use many artefacts that work exquisitely well, but there are as many that are badly designed, barely work or quickly fall apart. Most genetic mutations have no adaptive effect and become dormant or disappear through extinction. In human culture most patents are never used, most new products do not become a market success, most start-ups fail, and most academic publications are never cited (Blute 2010). Social preferences may reject potentially valuable innovations and defend inefficient traditions. In sixteenth century Europe, inventions such as gunpowder and firearms, printing, and navigation quickly spread and diversified because of their ready use in military conflict, religious quarrels, and colonial expansion. By contrast, mechanical inventions such as the gears and controls of automated birds or music boxes only served to entertain the aristocracy. The discovery of their numerous productive applications had to wait for the industrial revolution. The selection of the mediocre and superfluous, not the optimal, is the hallmark of natural as well as cultural evolution (Hallpike 1986, pp. 13, 208).

From the vantage point of the individual, cultural evolution thus appears as an unstable product of contingency and order, chance and intent. It is a risk-taking, opportunistic activity of muddling inventively through a contingent world in order to find workable solutions to the problems of living, with frequent good and bad surprises along the way. In many spheres of our life, past experience, stored knowledge or institutional rules make choices routine and outcomes reliable. Small margins of risk can be ignored. The more complex our objectives, and the further our plans extend into the future, the larger those margins become. When we invest our money or enter a new job, we cannot tell how it will work out. Dilemmas replace predictability. We can try to reduce the odds by trusting the people we deal with, hedge our bets by diversifying our investments, or learn from the successes or failures of others, but we never experience the comforting assurance of being guided around the shoals of uncertainty by the ‘models’ that populate co-evolutionist theories.

This look at the actual experience of cultural evolution suggests that selection itself evolved broadly in tandem with the increasing complexity of social compared with natural environments. Stable or slowly changing natural settings encouraged adaptation through molecular changes of genetic material, as well as their accumulation in genomes with large portions of dormant genes waiting to be activated when opportune environments arose. Wagner (2023) refers to these as “sleeping beauties” and gives them an equal or greater role in evolution as singular, step-wise mutations. With the emergence of social organisms and more variable social environments, selection favoured more flexible behavioural and cognitive responses. In human culture selection takes a primarily internal, cognitive form, and its results are transmitted through lifetime learning and cross-generational cultural inheritance. The adaptive gain of this shift from external to internal selection lies not in the genetic fitness effects of specific cultural choices but in the overall increase in the efficiency of interacting with rapidly changing and uncertain natural and social environments. Nonetheless, internal selection follows the first principle of all evolution: that blind variation, whether genetic, behavioural or cognitive, is the optimal response to an unpredictable world because it maximizes the chance to find utility in contingent environments. The production of contingency in response to uncertainty is the common denominator of biological and cultural evolution.

We can now take a closer look at three common features of these two processes.

5.1 Contingency and Blindness

Internal selection, like its genetic equivalent, works through trial and error. It relies on two resources: new or memorized information about our environments, and imagined thought experiments that examine that information for consistency or utility for our preferences. Like mutation and external environments in genetic selection, these resources are contingent or ‘blind’. The vast majority of natural and cultural information we encounter in our lifetime exists unrelated to our purposes, and we did not create it. Regardless of whether we search our memory, books or computers, whether we generate survey data or experiment in our labs, our work always incorporates pre-existing materials, artefacts and ideas. The invention of writing, printing and modern forms of electronic communication and data storage greatly increased access to these resources, but that did not change their blind character. Clicking the search button on our computer can easily create the illusion that we are responsible for what appears on the screen. In reality, we are merely using the technological equivalent of the evolved virtual scanning processes outlined by Campbell. Like our once-upon-a-time trips to the library, searches remain a foray into the unknown.

We scan this information with a second blind resource: imagined visions of what we consider possible or desirable, James’ spontaneous thought variations, which are now seen as the default state of the brain (Raichle 2015). They remain blind even when we focus our attention on a specific task, such as considering different moves in a game of chess or trying various solutions of a scientific problem. Our material, social or scientific environments always offer more information than we can handle, and more choices than we can use. We reduce this complexity by ignoring or forgetting much of what we encounter, by storing some of it in our memory or external devices, and by assembling a core of experiences we consider trustworthy and reliable guides for future action. But in contingent environments key parameters of our choices are always unknown, and their outcomes are always subject to degrees of uncertainty. Our thought trials therefore always contain blind components.

This double contingency is not experienced separately. Information-gathering and sense-making blend indistinguishably. Our perception runs all information through the filter of subjective meaning. Just as genes are not codes but carriers of potentials whose use is discovered by organisms, cultural information does not come with a given content but with a range of possible meanings selected by the cultural agent. Meaning, sense and symbols are artefacts, temporary stabilizations of their intrinsic instability. In natural selection we fail to see this if we are fixated on long-term adaptive outcomes rather than on organisms exploring their environments while they are alive. In cultural evolution we fail to see the ambivalence of information if we become used to finding hindsight reasons for events, and submit to the fallacy of attributing success to intent, and structure to purpose.

5.2 Discovery and Innovation

The double contingency of surrounding blind external information with virtually unlimited imagined use scenarios allows us to find a match between what we know and what we want, but there is no guarantee that we succeed. The moment can arrive seemingly without effort or after years of frustrating work, unexpectedly and in the most unlikely places, or not at all. As Max Weber put it,

Ideas occur to us when they please, not when it pleases us. The best ideas indeed occur to one’s mind … when smoking a cigar on the sofa; or as Helmholtz says with the precision of a natural scientist: when taking a walk on a slowly ascending street … In any case, ideas come when we do not expect them, and not when we are brooding and searching at our desks. Yet ideas would certainly not come to mind had we not brooded at our desks and searched for answers with passionate devotion (Weber 1968, p. 312).

This elusiveness, documented in the history of discoveries by Jennings, Pasteur, Roentgen and many others but often dismissed as anecdotal, is in fact an integral result of the dual blindness of the trial and error process of internal selection. Both environments and cognitive sense-making are highly complex and variable systems. Their sensitivity to changes in initial conditions introduces an unavoidable measure of uncertainty into our perception of environments, and an unavoidable measure of unpredictability into our choices. In a classical article Alchian (1950) showed that in contingent economic environments rational planning and optimization are impossible, that success becomes apparent only in hindsight, and that purely random actions can produce both success and long-lasting structures. The unexpected experience of error and novelty, including the occasional discovery of something startlingly new and exciting, is a natural corollary of the double contingency of cultural selection.

This challenges conventional views of human exceptionalism. Few areas of human conduct have been surrounded with so much mystification. Modern culture has glorified major discoveries and inventions but ignored the failures that paved their way. It celebrated the inventor as genius and treated creativity as an exceptional ability, but overlooked its pervasive importance in solving the mundane problems of everyday life. Past innovations by others and existing technologies were simply absorbed into the creative aura of the inventor. So was the labour of workers, lab assistants, graduate students and collaborators. Their contribution was ignored, undervalued and underpaid, and where it was eventually recognized, as in Rosalind Franklin’s role in the discovery of DNA, it was too late.

5.3 Selective Retention, Redundancy and Maladaptation

The long-term course of cultural evolution is path dependent. It frequently arrives at nodal points where chance, error or the limits of human foresight lead cultural evolution far from its intended objectives. Occasionally, our history is propelled by major discoveries and inventions, such as the emergence of agriculture, of counting, writing and printing, of steam engines and the internet. They unfolded in unconnected steps, and their contributors were often unaware of the eventual results. These changes did not require a rewiring of the brain and of cognitive abilities. The required cognitive abilities were already there. Like Wagner’s “sleeping beauties”, they lay dormant until changing climate, discoveries and new ideas allowed them to make more information available to more people and open up opportunities for increased productivity.

Together with technical and cultural advances, internal selection leaves behind unused or discarded alternatives, whether as dormant genes or abandoned projects. Error and failure are the indispensable companions of trial. During their lifetime, humans must be able to experiment with non-lethal consequences in order to find solutions to the problems of living. They must have room to learn from blunders and errors “… ohne bei jedem Irrtum sogleich die eigene Haut riskieren zu müssen (without having to risk one’s own skin whenever one makes a mistake)” (Riedl 1981, p. 41). The individual or social preferences that guide our choices are highly complex, variable and often ephemeral. Over the longer term most of what rises to cultural prominence is short-lived or forgotten. What survives the redundancy threshold is retained as technologies, laws or policies that are deemed advantageous or socially useful, but there is no guarantee they are adaptive or serve some standard of common good. In human culture, technical, scientific and social advances mix with technological failure, ideological obfuscation and fake news. Truly maladaptive behaviours—wars, conquest, slavery or genocides—are common in human history, as are their ideological justifications. Lasting maladaptive structures frequently emerge where selfish and collective interests collide. Darwin rightly considered cooperation a foundation of all social evolution, but the distribution of the results of collective achievements among the contributors has been one of the most divisive problems in cultural evolution.

At the same time, redundancy is not just a junk pile of useless or harmful human practices. Like Darwin’s carefully preserved “old and useless notes” and Wagner’s “sleeping beauties”, they form a reservoir of unused options that offer opportunities for future use. Basalla’s (1988, p. 208) observation that “every novel artefact has an antecedent” applies not just to technology but to all areas of cultural evolution. Social maladaptations can be overthrown, and their removal can lead to powerful ethical and political principles designed to prevent their recurrence. Like all such choices, however, they remain vulnerable to subversion and change.

6 From Contingency to Structure

The long-term evolution of cultural artefacts, ideas and structures is, like natural selection, circuitous and branching and unfolds in fits and starts. But it also creates stable customs and traditions, technologies and social structures of shorter or longer duration. The theoretical problem is “why and how, given the potential for radical discontinuities in system behavior, do some systems seem to evolve away from the extremes of complete order, inertia, and stasis on one hand, and complete randomness and chaos on the other.” (Mathews et al. 1999, p. 446).

There is no shortage of historical examples of variation and consolidation in cultural evolution. Sometime around 30 AD, the leader of a small Jewish sect appeared whose teachings spread rapidly through a Roman empire weakened by epidemics, external threats and climate change. His message of equality and charity found a receptive audience among many Romans, their vast number of slaves, and Roman client states, all of whom had become resentful of the excesses of imperial rule. Suspected of subverting the veneration of the Roman gods and the legitimacy of emperors, Jesus was crucified and early Christians suffered persecutions of varying intensity over a period of some 300 years. During this time they also produced an extraordinary effluence of creativity. Still unorganized and split into many groups, they developed innovative interpretations of the books of the apostles and the Old Testament. Debates arose over whether God was male or female, whether one should share everything one owned with others, whether true faith allowed marriage or required abstinence, and whether believers should trust their own reading of the sacred texts or seek guidance by priests and bishops, a question that would haunt the Catholic Church again more than a millennium later. This period of probing and exploration changed with the conversion of emperor Constantin after 312 AD. Christians now found themselves tolerated and supported, but at a price: an increasingly hierarchical Church, growing doctrinal intransigence and intolerance of ‘heresies’, and a view of sexuality as the sinful legacy of the fall of Adam and Eve.Footnote 5

The emergence and consolidation of early Christianity from seemingly trivial beginnings to a well-organized state religion shows typical features of a path-dependent process punctured by chance events and buffeted by expanding and contracting selective forces. It started with a precipitating new belief that modified existing Jewish and Roman religious ideas. Its spread was favoured by fortuitous social and natural circumstances and slowed down by persecution. Its period of great innovation opened intriguing what-if possibilities for a future evolution of the Church: gender equality versus paternalism, community versus hierarchy, accepted versus repressed sexuality. None of these choices was driven by a guiding rationality, nor did they bring consistent advantages to their adherents. The willing acceptance of torture and death by early Christians or their choice of poverty or sexual abstinence produced few cultural and no adaptive benefits. The path narrowed only when the concentration of political and religious power created selective pressure favouring religious and worldly hierarchies and gender inequity which, though not unchallenged, survives to this day.

Such path-dependent processes follow no predictable course. An evolutionary sociology can, however, look for regions of opportunity where variations emerge repeatedly, and for social dynamics with particularly potent order-creating effects. Elsewhere I have examined in greater detail the evolution of structures of social inequality in human history (Baldus 2015, 2017), one of the most contentious and conflict-prone products of cultural evolution, and a powerful source of selective forces extending into many spheres of human culture. I argued that inequality structures, from early to modern societies, emerged from two fault lines in social environments that were particularly rich sources of chance events that open up pathways to inequality. Along the first of these we find initial fortuitous variations in strategic resources that increase opportunities for further accumulation. Chief among these are control of material wealth, knowledge, authority and power, and the ability to exclude others from social relations. All have potential compounding, self-reinforcing effects. Early human settlements along rivers or fords can turn after many generations into stranglehold over river crossings and fertile riverine land. Early long-term investments in cleared fields, olive groves or fishing boats create attractive opportunities for take-overs and protection rackets by powerful outsiders (Gilman et al. 1981). Products or ideas that achieved early acceptance often maintain their dominance long after better alternatives become available. Winner-take-all competitions multiply fortuitous gains. Inherited wealth has time and again enhanced access to wealth, power, education and suitable marriage partners.

None of these consequences must occur, and they rarely follow a preconceived plan. Inequality structures are structures of opportunity, and their path is paved with errors, failures and reversals. But once under way, gains are not likely to be lost again as quickly as they were obtained because compounding and opportunity hoarding favour their increase. Self-reinforcing dynamics appear often where large initial gains allow recipients to monopolize markets, social capital or prestige, and where other participants are locked into a position of dependence from which they can withdraw only at considerable cost (Arthur 1994, p. 94).

The second fault line involves trust and cooperation on one side, and self-interested defection on the other. Both are likely inherited behavioural predispositions that lie dormant but become active in opportune circumstances. Trust permits us to extend our relationship with others beyond short-term self-interest. Trust in turn is indispensable for cooperation, which allows participants to achieve more than they can accomplish alone. However, they also create opportunities for defection and deception. Trust assumes that others will act in ways that correspond to our expectations. But the confidence we place in them also gives them freedom from scrutiny and facilitates abuse of trust. Similarly, cooperators assume that others will do their part in a joint project that creates value beyond what each partner could achieve. But that added value also creates an incentive for others to defect and walk away with our work without doing theirs.

Here, too, we can identify areas where breaches of trust and defection from cooperation have particularly frequently in human history initiated pathways to inequality. In early, relatively egalitarian societies, kinship ties and institutions of sharing and generalized reciprocity were especially vulnerable to subversion of trust and cooperation, and to the accumulation of prestige and clients, whereas initially communal offices opened up opportunities for the rise of chiefs, kings and ruling classes.Footnote 6

Inequalities that emerge from these regions of opportunity have their own inherent weakness. Inequity perceived as unfair can encourage resentment and resistance, and betrayers of trust face the prospect of killing the goose that lays the golden eggs. Both therefore require forms of social control that legitimate or disguise unequal social relations. Few social structures have generated as many justifications as structures of inequality, and they usually arise as soon as such structures appear. Whether they rely on coercion or secrecy, whether they depict inequality as a divinely ordained order or the result of the survival of the fittest, whether they promise the poor salvation in heaven or trickle-down economics in this life, all such efforts are designed to maintain and strengthen prevailing structures of inequality. The incentives are strong to protect and multiply what one has. It is here, not in their functional or adaptive benefits, that we find the cause of the recurrence and consolidation of inequality in human societies. Powerful as such ideological reconstructions are, they do not always work. Inequality structures are overthrown, ruling groups collapse under their own excess, and social policies can significantly reduce distributive inequity.

7 Evolution, Intention and Agency

An analysis of the lifetime experience of variation and selection gives us a better picture of our role in cultural evolution. It is more modest than the exceptionalist view of human nature, but more autonomous than biological determinism. It acknowledges the importance of what already exists for what we accomplish, as well as the frequent failure and harmful consequences of what we do. Within these limits, our contribution to cultural evolution has been the invention of ever more efficient ways to gather and store information, and to use our cognitive ability to imagine an ever greater variety of possible choices.

Finding sources of variation, identifying what social influences shape the criteria for selection, and reconstructing pathways of cultural evolution offer much scope for a closer collaboration between co-evolutionary theory and sociological research. It can encourage the former to become more attuned to the complexity of social life, the latter to become aware of the role of chance in the interaction between biological inheritance and social change, and both to confirm Darwin’s argument for the continuity of natural and cultural evolution.

There is one more implication to be drawn from our role as agents in cultural evolution. In the social sciences, evolutionary thinking has often laboured under the opprobrium of justifying social Darwinism or imprisoning human actors in the dictates of their genes. From the perspective of lived evolution we are intentional participants in cultural, and eventually also biological, evolution. We bear, therefore, considerable responsibility for its course. This idea was first pondered by Darwin who thought that parental instincts in animals evolved gradually through natural selection into “higher” forms of altruism and morality in human beings and would ultimately promote the common good of humanity.

In sociology there has been a longstanding debate between those who want to use their findings to advocate and work for social change, and others who insist on separating ‘value-free’ research from the social and political interests. Similar divisions are familiar to biologists discussing evolutionary ethics, and have led to starkly contradictory views.Footnote 7 Darwin’s warning that cultural evolution can create all-powerful conditions opposed to the true welfare and happiness of mankind has not lost its relevance. Cultural evolution has brought us to the point of interfering with the very process of evolution itself. Understanding culture as an evolutionary product assigns us a large measure of agency, and with it the ability to influence the direction that these changes will take.