1 Introduction

When researchers and innovators solve problems, they often use well established procedures. Much of applied science is like this; for example, the development in Mexico of higher-yield short-stemmed wheat by Norman Borlaug in the 1960s was accomplished mainly by using standard techniques of selective breeding.

Often involved in this process are insights—original thinking that helps to make sense of a problem or leads to a new method to solve it. Insights may occur at the start of research, for example by recognizing the need and scope for research into higher-yielding cereals, as Borlaug did. Insights may also occur in the course of research; for example, Borlaug realized that he could grow wheat twice a year, rather than once, and speed up the breeding process. Or insights may occur at the end of an investigation that has revealed facts that now demand an explanation; for example, when Roentgen discovered x-rays in 1885, an insight was needed to relate these x-rays to other forms of radiation. Sometimes, the insight is so novel that it shifts the direction of a whole field of research. Examples are the introduction of germ theory, the Darwinian revolution, and the idea of tectonic plates moving. Such ‘paradigm shifts’ (Kuhn 1962) are very important but quite rare and there are many less profound insights that can still serve to direct research and interpret findings. In addition to their contribution in science, insights help us to construct new products, form new social institutions, and to create prose and poetry. Without insights, progress in these fields would be stultified but here we focus on their role in science.

Sometimes insights are deductive, when we realize that logic or mathematics compels a particular solution to a problem, as in geometric theorems. In science, it is common to see insights as inductive: we somehow think of a premise from which, with other available information, we can infer a solution to a problem. As such, the inductive process would belong to psychology and Zahar (1983) notes that the Vienna circle of philosophers saw induction in this way. However, philosophers of science, such as Zahar and McLaughlin (1982) have argued that new ideas may be derived using logic or mathematics. In Zahar’s argument, the derivation is based on meta-principles governing knowledge, and deduction from previously established theory. We enter this debate only to point out that there may be a disjunctive fallacy here since both inductive and deductive routes are possible in different situations. The insights with which we are concerned deal with facts related to a problem (concomitants, antecedents, outcomes), as well the identification of questions needing an answer; there is no deductive link from these facts to the new idea.

It seems that important innovations are often long delayed, which suggests that the lateral thinking involved in induction is difficult to perform. For example, the idea that species evolve by natural selection seems to have taken its time. The similarity between men and apes was well known. Darwin travelled on the Beagle with a book by Humboldt that described ecological connections between species, which should have helped (Wulf 2015). He returned from the Beagle voyage in 1836 with evidence showing the adaptation of creatures to their environment but it appears that it was not until 1838 that he formulated the theory of natural selection after reading Malthus’ (1798) essay, where it was argued that the growth in population would produce competition for resources. Darwin then took his time before publication, probably fearful of the religious reaction. He refined the theory and, when Wallace independently proposed natural selection to him, he and Wallace both made presentations to the Linnean Society in 1858. Wallace had also gathered evidence showing the links between species and, similarly, had conceived of the idea of natural selection after reading Malthus. Klein (2013) points out that many people had read Malthus but only these two came up with the idea of natural selection. It is difficult to reconcile such a history with the view developed by Popper (1930, first published in English in 1978), that there are plenty of new ideas, many of which fail on test.

We can point to other innovations that seem delayed. The great pyramids in Egypt were constructed without the help of pulleys, which were invented some 700 years later. Mathematics would have benefited from an earlier use of zero in the number system. China was mass producing steel by the third century AD, which made other products feasible. One such product is the bicycle, but this did not appear until 1817. A major obstacle to the invention of the bicycle must have been the lack of insight into how balance could be maintained on two wheels. Some innovations are assisted by naturally occurring phenomena that show what is possible. Birds illustrate flight; ash seeds demonstrate a kind of propeller; the burrs on burdock gave the idea of Velcro. But there were no self-balancing two-wheeled vehicles in Nature to aid the conception of the bicycle.

Many technical innovations lead to the creation of more innovations, and discoveries in science often provide the basis for insights that lead to further discoveries. For example, Newton’s work in optics showed how white light was a composite of different colors with somewhat different focal lengths. This explained the limitations of simple lenses and led Newton to think of a mirror-based telescope.

In any review of insights, we can only deal with a truncated sample. Of necessity, we cannot consider the insights that people have not made because they were too difficult. However, if we can understand better the circumstances under which insights happen, we may be able to create conditions that help us to enlarge their number. What seems clear is that new ideas tend to rest on knowledge. The knowledge that stimulates insights must come from experience. In science, that experience is observation of phenomena that occur naturally or are created by experiment. Sometimes, we already have the experience and the insight comes from recasting it in some way. For example, we have all experienced the force of gravity but Newton had the insight that it applied to the whole cosmos, not just the Earth, and then used mathematics to work out the elliptical form of planetary motion that this implied, which could be checked against evidence.

2 Processes Underlying Insight

2.1 An Early Approach

There have been attempts to specify the process whereby insights come about. Graham Wallas (1926) proposed that innovative research arose in four stages: preparation, incubation, illumination, verification. The first three stages relate to the generation of insight with preparation involving the gathering of knowledge, incubation its processing and illumination the insight itself. Preparation and incubation concern us because they occur before illumination or the attainment of insight. However, some preparation can obstruct new thinking. Klein (2013:130–8) cites the discovery of the double helix gene structure as a case where this occurred. Watson and Crick had the right preparation and recognized the implications of Rosalind Franklin’s x-ray diffraction photograph of DNA, but Franklin was held back by flawed data from earlier research that implied that DNA lacked a helical structure; she had the photograph for 10 months without recognizing its implications. So prior beliefs may hinder as well as help. There are many instances in science where a new idea is resisted because it is incompatible with established thinking. One example was Wegener’s (1912) theory of continental drift, which was initially rejected by geologists; it took 50 years for the theory to become accepted. One of the reasons for the initial rejection was a prior theory in geology that the Earth’s contours were created by cooling. Another reason was the lack of any account of how the movement of the tectonic plates was energized, since, at that time, no one knew about the release of heat from radioactive decay.

An Exercise in Insight

Here is a problem that troubled one of the authors. It concerns the diffusion of innovation: the way new ideas, practices and modes of consumption spread through a social network. The predominant thinking in this area has been that advertising or another mass-medium communication gets some people to adopt the innovation and then these early adopters persuade others; some of whom persuade yet more people, and so on. Thus, after the initial adoptions, the innovation spreads like a disease. This is a matter that concerns social scientists, epidemiologists and computer modellers; it is, or should be, a major scientific issue. A popular approach here is agent-based modeling. Researchers specify the form of the social network and the pattern of human interaction in relation to the innovation, code these in a program, run the program and observe the outcome. Then experiments can be conducted by changing input elements and observing any changes in output

Agent-based modeling will fail if the specifications do not fit what people do and the author nursed an unease for a long time about the sufficiency of the disease model at the centre of this thinking. Eventually this crystalized as a doubt about the one-way nature of disease transmission: there is no feedback. Can you see a way in which feedback could occur in the diffusion process?

The second stage in Wallas’ model, incubation, is intuitively appealing because we like to think that we sometimes work away at problems unconsciously to reach insights. However, Klein, who collected 120 examples of insights that had occurred naturally, found relatively few cases of insight where this seemed to be involved. He also noted that about two thirds of the experiments on incubation indicated unconscious processing. Thus, it looks as though incubation sometimes occurs but is not necessary.

On this evidence, it seems that the insights at the third stage in Wallas’ model need not be derived in the way he describes.

2.2 Circumstances Inducing Insights

Whilst Wallas’ approach was analytical, Klein’s was empirical. Klein classified the circumstances under which his 120 cases had arisen under five headings: connections, coincidences, curiosities, contradictions and creative desperation (or impasses). Klein’s five circumstances are not mutually exclusive since many insights can be classified in more than one way. For example, when Lavoisier showed that substances, such as phosphorus, gained weight when they burnt and proposed that they combined with a substance in air, he contradicted the assumptions of phlogiston theory which implied that they would lose weight as they lost phlogiston. But he also put the phlogiston theorists in an impasse. The effect of penicillin mold on bacteria, noted by Fleming, was an interesting connection/curiosity. Marshall’s observation that a patient lost stomach pain when he took antibiotic was a coincidence/connection/curiosity that led him to think that stomach ulcers might be due to bacterial infection. Eventually, Marshall identified helicobacter pylori as the cause of stomach ulcers. Klein argued that, depending on the circumstances of the case, the insight that solves the problem requires either the addition of new information, the reappraisal of current knowledge, or its abandonment.

2.3 Introspection?

Now let us consider the problem set earlier about whether the one-way nature of disease transmission is an appropriate model for the diffusion of innovation (see An Exercise in Insight). Here, it is important to think of the interactions between consumers. The disease model is only part of the picture. Current users may introduce a product to other consumers, but they may also receive comments about the product from other users, and these comments may cause them to give further word of mouth to others, some of whom may not yet have adopted the product. Thus, feedback in the user group may occur and diffusion need not be all one-way. This seems obvious now, but it took the first author years to understand this, perhaps because the disease model clearly provided an explanation. You may have thought of this process, but you did benefit from having the problem defined as one of feedback.Footnote 1

Unfortunately, we are not conscious of the inductive process that generates the insight; it just happens. We know the ideas that we had and when the insight occurred but not the process whereby this insight was reached (Nisbett and Wilson 1977), so introspection is limited. Because of this, we have to work out how insights come about from a review of the conditions that preceded them. In the case of the diffusion of innovation, one effect of word of mouth is to generate more word of mouth and our research showed that positive word of mouth on products is almost entirely confined to users and ex-users of the products. This must have helped our thinking.

3 Factors Affecting Insight

Advance in science must rest heavily on the acquisition of knowledge using scientific procedures but there are psychological processes that may also affect the search for knowledge and its use. These may help or hinder insights.

3.1 The Salience of Information

In making inductive leaps, some types of knowledge may be given more weight than others. One bias of this sort is an effect called base-rate neglect (Tversky and Kahneman 1982). Phenomena that are general, continuous and unchanging tend to have less impact on our thinking than those that are specific, causal and changing. Base rates, such as the proportion of people in different occupations, belong to the former group. If people are told that a smartly dressed man who argues persuasively and fluently is either a teacher or a barrister, there is a tendency to guess that he is a barrister, despite the fact that there are many more teachers than barristers. This example shows how we give different weights to different types of information and, in this process, tend to neglect background frequencies. In science, the unchanging conditions under which phenomena occur may get less attention than the features of the environment that change, but both may be necessary for explanation.

There are other biases that may enhance or diminish the weight given to information. An important one is confirmation bias (Wason 1960), which is a tendency to seek support for accepted beliefs and reject information that disrupts this thinking. This occurs when new scientific theories challenge an incumbent theory as we saw in the rejection of Wegener’s proposal about continental drift.Footnote 2 By contrast, theories that resolve a problem are more easily accepted; for example, Newton’s three laws of motion were well received in the main because they addressed a recognized deficit in understanding.

One basis for confirmation bias may be a preference for certainty. This is shown by the observation that people will pay more to increase an outcome probability from 99 to 100% than they will to increase it from 50 to 51% although the increment in benefit is the same. An established theory provides a degree of certainty and may be preferred on this basis. A second basis is the availability heuristic, a tendency to assign a greater likelihood to more mentally accessible thoughts (Tversky and Kahneman 1982). An established theory, being well known, is more accessible and therefore more likely to be perceived as true than newer but less accessible ideas. The tendency to assign higher likelihood to more accessible thoughts probably arises because, in everyday life, likelihood and accessibility are often associated. Thus, biases exist in thinking that may favor current beliefs and make it harder for people to entertain insights that challenge the status quo.

3.2 Motivation and Habit

A second barrier to insight is lack of motivation. The aphorism ‘necessity is the mother of invention’ is often attributed to Plato, but this seems to be a mistranslation of what Plato stated (see https://en.wikipedia.org/wiki/Necessity_is_the_mother_of_invention). Klein (2013) found many cases where an immediate need impelled innovative thinking. However, in science, motivation seems to come more from curiosity, uncertainty and from career advancement. Interestingly, psychological research has demonstrated that when work is driven by intrinsic motivations such as curiosity and uncertainty, as opposed to externally driven motivations, the solution that emerges is likely to be more innovative (Amabile 1985). Motivation can stimulate greater reflection on problems, encourage exploratory investigations and focused conversations with others. All these activities may stimulate insights.

On the opposite side, habit is often the culprit when people lack the motivation to make sense of evidence. We settle into routines in which action is controlled by cues. Such habituation is useful in everyday life because it deals with routine activities with little effort, but it may prevent us from recognizing problems and seeing new ways of doing things.

3.3 Serendipity

Insights also arise from incidental experience in the course of research or practice. Roberts (1989) reports many discoveries in science that came about serendipitously. A recent example was the observation by Rubino (2017) that diabetics undergoing bariatric surgery often lose their diabetic symptoms very soon after surgery. It took Rubino some time to gather funding for systematic research into this matter since it seemed implausible to prospective funders that surgery could have such a speedy effect. When the evidence was finally gathered, Rubino found that as many as half of the diabetics lost symptoms within days. This finding shows that researchers do not fully understand how insulin is produced and transported by the body. Rubino suggests several hypotheses relating to this evidence.

3.4 The Impact of Current Knowledge

The most obvious basis for insights in science is the continuing flow of new materials and discoveries. In the case of materials, the insight may not be that difficult. Once Teflon was available, new applications from cookware to shoelaces followed. As micro circuitry developed, memory sticks and solid hard drives came into being. Similarly, once mobile phones could be geographically located, a series of applications became available, from route finding to location marketing. In this century, a new class of materials called multiferroics has been discovered. These are both permanently magnetic and permanently electrically charged (Ornes 2019). A whole range of applications of these materials from health to solar energy harvesting is now in prospect. New materials may also stimulate greater theoretical insights as scientists try to explain how these materials work.

Subjects evolve as discoveries are made and theories develop. Each new understanding lays the groundwork for more ideas and investigations. Some subjects such as archaeology, astronomy, social research, geology, geography and epidemiology rely heavily on observations and data analyses to generate findings, which then require explanation. Consider the recent evidence that a bacterium involved in gum disease, Porphyromonas gingivalis, may play a major role in the development of Alzheimer’s disease (Dominy et al. 2019). These bacteria were found by observation of tissue from the brains of Alzheimer’s patients and subsequent work has indicated how the bacteria could bring about the disease. It is difficult to see how the role of this bacterium could have been discovered in any other way. But, once the association between P. gingivalis and Alzheimer’s had been found, insight could play a part in working out the possible processes that produce the disease.Footnote 3 Such findings have led some researchers to question the assumption that theory normally precedes the search for evidence. It is argued that, at least in some subjects, the gathering of evidence mostly occurs before the generation of theories (Bass 1995; Ehrenberg 1995). This is a matter that can be investigated empirically but surprisingly little work of this sort has been done.

4 How Can We Increase the Frequency of Insights?

Klein devotes several chapters to increasing the flow of insights. However, his concern is broader than ours and here we focus on matters that affect the conduct of scientific work.

4.1 Adjusting the Mental Set

People can be more creative when they look at problems in an abstract way. According to Liberman et al. (2007) people may set problems in a localized or distant context and, as a result, construe them differently. A distant setting may be in terms of physical distance or time (Jia et al. 2009). In both cases the effect of the distant construal is to make people see the problem more abstractly and, as a result, creatively. This evidence suggests that care in the presentation of problems is required. We should present the more abstract and generalized aspects of problems and not dwell too much on their concrete specification if we want people to have insights.

4.2 Recognition of Bias

Earlier, we noted how base-rate neglect and confirmation bias could affect the way new evidence and ideas were evaluated. The history of scientific work contains many cases where there was excessive resistance to ideas that challenged established thinking. To a degree, a skeptical approach to new ideas is justified since the established thinking has often been carefully examined and the new ideas have yet to be fully evaluated. However, it does appear that psychological processes raise the level of skepticism and reduce the willingness to see problems in an established theory. For example, resistance to Wegener’s theory of continental drift was based on the idea that mountains were produced by the Earth cooling and that Wegener had not explained the source of energy for drift. But the cooling theory was quite feeble, given the way that mountain ranges are distributed; tests on cooling material designed to simulate the process of mountain formation would probably have failed to support the hypothesis. In addition, although Wegener had no theory to explain how plate movement was powered, the eruption of volcanoes and the power of earthquakes testified to enormous subterranean energy. This suggests that the scientific community needs to take a stance that is more open to challenges to orthodox views.

4.3 Non-Experimental Work

More encouragement should be given to methods that are likely to generate insights, and this must include non-experimental work. The conduct of science is a human activity and is the outcome of social and psychological processes. Whilst we may characterize our objectives as scientific, these objectives are still socially agreed, and the social consensus may be biased. We suggest that generating original ideas is not given enough weight because of the emphasis on testing ideas.

It is easy to see why scientists value testing. New ideas are often wrong. They must pass the test before we can accept them but, without new ideas, science is reduced to refining existing knowledge. Thus, methods that throw up new ideas have value and this value is greater if important insights are difficult to achieve, as has been argued here. Such methods include exploratory studies, observation, case analyses, surveys and modeling. There are fields such as medicine and psychology where there is strong emphasis on experiments compared with non-experimental methods. In such fields, more use of non-experimental methods could increase the pace of progress by stimulating more insights. Is not faster progress a goal that we seek? One non-experimental method of increasing value is the use of artificial intelligence (AI) analysis. Where large quantities of data exist in a computable form, an AI analysis may reveal unsuspected patterns. For example, the analysis of medical records may reveal co-morbidities or other disease associations that suggest further research.

Here, a supporter of the current emphasis on testing may point out that new ideas flow from test failures. What is the evidence for this? When a hypothesis fails, we may generate alternatives but these may not come from the test. Most tests are experimental and are narrowly focused so that they provide little additional information. By contrast, correlational studies show a range of associations and some of these may cause us to have insights, while observational studies may produce findings that were completely unforeseen, as in the case of P. gingivalis in the brains of Alzheimer’s patients. Such findings then raise further questions: how do the bacteria pass into the brain? Can the bacteria be eliminated with antibiotics? What are the bacterial processes that could damage the brain? Can these processes be counteracted? Researchers will have ideas about the answers to these questions. We cannot accept those ideas without further tests but there could be little to test in some subjects if observational and other non-experimental methods are avoided.

4.4 Re-define Unresolved Problems

Another approach is to redefine the problem. This may mean challenging established thinking. For example, much effort has been devoted to halting Alzheimer’s disease by developing drugs that could prevent the plaques forming in the brains of such patients. The orthodox explanation has been that the plaques caused the brain degradation. It now appears possible that the plaques are produced in the brain as a by-product of resistance to P. gingivalis and might not have a causative role in the disease. This was always possible, but it seems that a determined search for an organism like P. gingivalis was held back because a causative role for the plaques seemed more plausible. What is needed is a ‘challenger section’ in research bodies, tasked with defining new research hypotheses and how these could be tested, particularly where progress has been limited.

In the same vein, research funding should be directed to unresolved problems. Whilst research bodies do highlight and fund research in important fields, such research might be better focused on specific problems rather than fields. Those who advise research bodies may also be attached to orthodox thinking but a formal commitment by the research body to reviewing alternatives could help to oppose this, particularly if this commitment was focused on areas where progress had ground to a halt.

4.5 Expose New Findings, Materials and Theories to Wider Scrutiny

If insights are scarce, their number can be increased by increasing the number of people thinking about any development. Unfortunately, there may be an advantage from holding back publicity about new discoveries in science because this will delay the work of competitor groups and assist goals of fame, tenure, and funding. Ideally, new developments should be publicized when these have reached a stage when they can interest others. Then they should be reported in appropriate detail so that the contributions of others can be added. This can be done within departments but experts in the field need to be involved wherever they are. However, to ensure fairness, we may need more conventions about preliminary release of information and the appropriate recognition of substantial contributions made by others. All of this should be incorporated into indices of academic performance. The practice of making work available online as soon as it is accepted for publication has helped to speed up dissemination of ideas and, in life sciences and medicine, the sharing of pre-publication results on Internet sites like bioRixiv and medRxiv allow scientists to disseminate their findings and obtain feedback from the scientific community. However, more could be done to encourage the early diffusion of new ideas.

4.6 Research on the Discovery Process

We need evidence on the circumstances attending new insights in science. The formal research paper usually reports on problems with existing theories and tests of new theories but does not describe how a new theory was conceived – what the circumstances were when the research question was thought of by a specific person and what were the nagging doubts and puzzles. As a result, the discovery process is not documented in science though it may be described in biographies, for example, Lewis’ (2017) account of the collaboration between Kahneman and Tversky or Wulf’s (2015) story of the development of Humboldt’s thinking. In papers, a footnote about the gestation of a theory would be useful. This is important because, if particular circumstances are found to be common, it would be wise to highlight these to research teams working on related problems. (As an illustration, see the parenthetic note in the abstract).

Another question concerns the most effective number in the research team and the division of labor therein. Some famous scientists have worked alone: Newton, Einstein, Darwin (and Wallace) are well known examples. But this may reflect the way in which science was normally conducted in the past. In more recent times, teams are the norm and it seems probable that the pooling of ideas in a group increases the range of thinking. But teams work in different ways. Sometimes, particularly when a range of expertise is needed, research teams may divide the work quite evenly. This pattern fits the work of Watson and Crick who, with Wilkins, received the Nobel Prize for their discovery of the double helix structure of the genome. Similarly, in psychology, Tversky and Kahneman worked jointly and engaged in long discussions about research on the heuristic processes that bias judgment (Lewis 2017). In other areas, one person may do most of the work with others acting as discussants when called upon. Sometimes novel ideas are retained in their academic silo and their application in other disciplines is not recognized. To counter such effects and pool knowledge, the Institute of Advanced Study at Durham University, for instance, brings together scholars from different fields to collaborate on projects of major intellectual, scientific, political and practical significance. Perhaps more teamwork of this sort is justified but, currently, we do not know what sort of group structure is the most productive in generating insights and how it would need to vary for different fields.

5 Postscript

At the present time, global heating is creating an unprecedented threat to our way of life. The warming of the planet has been made possible by the wide application of technologies that are informed by science and these problems are exacerbated when necessary political changes are implemented too slowly. However, science and technology have also succeeded in mitigating environmental threats and some political change has happened. Further progress depends, in part, on innovations in science, technology and political institutions. This paper is concerned with the conditions that could speed up such progress.