4.1 Optimism, Overoptimism, and Overoptimistic Predictions

We usually say that a time prediction is overoptimistic when the actual time usage is greater than the predicted time usage. This does not mean that an optimistic or overoptimistic view on time usage was the cause of the too low time prediction. Lack of knowledge, miscalculation during the prediction process, and bad luck in the execution of the project are examples of alternative reasons for too low time predictions. Describing too low, or overoptimistic, time predictions as caused by overoptimism, in the rose-coloured glasses sense, not only is incorrect but may also stop us from seeking other explanations of time overrun besides overoptimism [1].

To clarify the differences between being optimistic or overoptimistic and making optimistic time predictions, let us start by looking into the concepts of optimism and overoptimism. The Oxford English Dictionary describes optimism as ‘hopefulness and confidence about the future or the success of something’ [2]. If optimism is expecting things to go well, which they sometimes do, we may use the word overoptimism when the expected positive outcome is unwarranted, for instance, due to thinking too highly about one’s own skill and likelihood of success. If Patricia is an average chess player but rates her chess skill too highly and predicts that she will win a chess tournament where she is far from the best player, she is overoptimistic. Even if it is very unlikely that Patricia will earn first place in the tournament, it is still possible that she will be extremely lucky and win and, consequently, gave a correct prediction. Overoptimism, as we use the term here, does not always lead to overoptimistic predictions.

People tend to be overoptimistic about many things. Married people predict a too low likelihood of getting divorced, car drivers think too highly of their driving skills, and students tend to believe they will obtain better grades on exams than they actually receive.Footnote 1 A typical finding is that most people regard themselves as better than average, particularly in completing relatively simple tasks and tasks where they have exceeded a minimum skill level. This phenomenon has been named the better-than-average effect and may be seen as a type of overoptimism. The better-than-average type of overoptimism can be illustrated by the perceived likelihood of company success. Entrepreneurs have a high probability of failure and earn, on average, 35% less than employees in similar jobs after 10 years in business [4]. They take a huge risk by investing their own and perhaps even their family members’ money and time on something they should know has a high probability of failure. However, if their visions come true, they may have created a new Google or Tesla and contributed to their own and their country’s wealth. What is a seemingly unwise decision at the individual level supported by a strong degree of overoptimism may not only be beneficial but also essential for society at large. Had the entrepreneurs been realistic about the amount of time to be spent and the true likelihood of success, they may not have started their companies in the first place.

Overoptimism may be present not only when evaluating our own abilities and opportunities but also when assessing those of people we like. American football supporters consistently predicted, over a period of 17 weeks, that their football team would win more often than they did, even when they could earn money from making more accurate predictions. The predictions made at the end of the season were just as overoptimistic as those at the beginning of the season, despite repeated feedback about actual outcomes [5].

To make the use of optimism-related terms even more confusing, optimism and overoptimism are related to but are not the same as the personality trait named optimism. This type of optimism is often described as the tendency to believe that one will generally experience good outcomes in life [6]. Such an optimistic life orientation is associated with a range of beneficial outcomes: a longer life, better recovery from disease, and extended survival times for cancer and AIDS patients [7]. Surprisingly, at least to us, the personality trait of optimism seems to be only weakly, if at all, related to overoptimistic predictions [8], including time predictions [9].

In short, there is more to optimism than overoptimistic time predictions and optimism is far from the only way to explain overoptimistic time predictions. People’s desire for success or motivation for a given outcome may produce overoptimistic predictions, but overoptimistic predictions may also be caused by other factors. In the following sections, we will describe in more detail potential causes of and variables related to overoptimistic time predictions.

  • Take home message: Optimism (i.e. hopefulness for a desired outcome), overoptimism (i.e. unwarranted confidence in a desired outcome), and trait optimism (a personality characteristic related to a positive view of life outcomes) are different aspects of what we typically call optimism. Optimism and overoptimism but not so much the trait of optimism may lead to overoptimistic time predictions (i.e., too low predictions), but there are numerous other possible reasons for overoptimistic time predictions.

4.2 The Benefits of Overoptimism

At first glance, it is hard to see how our ancestors, Stone Age humans, would have benefitted from being systematically overoptimistic about time usage or underestimating the risk of actions. Underestimating the consequences of fighting with a person from another tribe or how much effort hunting on the other side of the mountain will require may, for example, lead to an early death or a food shortage. New generations would then be more likely to have the genes of more realistic Stone Age humans able to predict time and risk accurately. However, it seems that many of us are descendants of overoptimistic Stone Age humans. Could it be that overoptimism is rational and adaptive?

In a classic set of experiments from the 1970s, students were repeatedly given the choice to press or not to press a button within a time frame of three seconds [10]. Pressing the button could have the outcome of a light being switched on or the outcome that nothing happened. The light could also switch on if the student chose not to press the button. After 40 trials of button pressing, the participants were asked how much control they thought they had over the light. Even in the experimental situation in which they had absolutely no control of the light switching on or off, the typical student ended up believing that he or she exerted a substantial amount of control, as long as the light was switched on frequently. Typically, the perceived amount of control, when there was no control, was believed to be about 40% on a scale from zero to 100%. Depressed students seemed to be the only group of individuals who were reasonably accurate in their assessment of their influence over the light.

This study was the beginning of research on so-called depressive realism. Depressed people seemed to be more realistic than other people, who were generally overoptimistic regarding their influence, their recollection of feedback, their chances of success, and their stance on positive attributes in comparison with others. Unfortunately, as with many other spectacular ‘facts’ from science, the effect of depression on realism turned out to be very small when all relevant studies were summarized in a meta-analysis [11]. Nevertheless, even a small effect illustrates that optimistic beliefs and predictions may be associated with positive things in life, such as being less depressed. This is further illustrated by studies reporting that unrealistic optimism, or so-called positive illusions, may be good for your mental health [12].

It is even possible that strong overoptimism has helped us survive and reproduce [13]. Assume a very small chance of success for some effortful action, for example, an attempt to make a new, efficient type of weapon that would make hunting easier. Each separate choice of not engaging in this action may seem like a good decision, assuming that success is very unlikely and costly in terms of effort and materials. One day, however, someone will succeed in making a better tool or weapon and this is likely to be one of those overoptimistic persons pursuing the near impossible. Now the optimists, or at least one of them, have an advantage in the battle of survival.

Similarly, if all the costs, efforts, delays, and frustrations of a company’s ambitious projects were known beforehand, the projects might perhaps never have been initiated. When completed, most of the ambitious projects may have become complete failures, but perhaps one of them became a great success, giving the company a tremendous advantage that more than compensated for all the failures. Preceding many great successes are usually a high number of failures with strongly overoptimistic cost–benefit predictions.

The economist Albert O. Hirschman proposed an argument in favour of being overoptimistic [14]. He argued that people tend to underestimate their creativity and ability to find solutions to problems that emerge. Overoptimistic time and cost predictions may compensate for this overpessimism. Thus, overoptimism is the hiding hand that helps us conceal difficulties and encourages us to complete worthwhile projects that we never would have initiated had we known how problematic they would become.

The researchers Flyvbjerg and Sunstein [15] showed that in most large infrastructure projects, difficulties were indeed underestimated, as evident by the strong tendency to experience cost overruns. The benefits, for example, in terms of the number of users of a service, were, on the other hand, typically overestimated. If we overestimate benefits and underestimate costs, we may tend to start projects that are not worthwhile, and the hiding hand will not be as helpful as Hirschman assumed. Flyvbjerg and Sunstein claimed that the negative hiding hand, which they termed the malevolent hiding hand, applied in 78% of the projects they studied, while Hirschman’s positive hiding hand, which they termed the benevolent hiding hand, applied to only 22% of the projects.

Even if negative effects of overoptimism should be more common than positive effects in infrastructure and other types of projects, there may still be more benefits than costs across all such projects for society at large. With a few projects yielding immense benefits and fewer projects being detrimental, it would not be that bad to have a high number of overoptimistic cost–benefit analyses after all. However, we have yet to see systematic studies on the distribution of benefit outcomes for large projects.

A tendency towards overoptimism may be a result of a supportive and friendly environment that does not punish failures too much. Overpessimism, on the other hand, may be a result of a more unfriendly environment. There is a fascinating line of research that shows how humans and animals, for instance, dogs, rats, and starlings, seem to develop more overoptimistic expectancies in better environments [16].

In a study of birds’ behaviour, the researchers exposed starlings to both bad and good cage conditions. The bad conditions included unpredictable feeding and cleaning times in a small, boring cage. The good cage conditions included continuous access to water baths, perches made from natural branches, and bark chipping on the floor. The birds first learned that they could receive food pellets if they pressed one lever when they saw a light for two seconds and another lever when the light appeared for 15 seconds. From previous studies, the researchers knew that the birds preferred to receive food immediately, making the two-seconds lever a more positive experience. On some occasions, the light appeared for intermediate intervals, between two and 15 seconds. These were durations for which the birds did not have pre-learned responses. Would the birds believe that the intermediate durations were signals for the pleasant, immediate rewards or the less pleasant, delayed reward? The researchers found that the birds in the good cage conditions tended to interpret the ambiguous light duration as belonging to the most positive outcome and pressed the lever for the immediate food response outcome more often, while those in the more depressive cage conditions interpreted ambiguous light duration more accurately and pressed the 15-second lever when the duration was closer to 15 seconds and the two-second lever when the duration was closer to two seconds. In this context, the less optimistic birds actually received more food due to their more accurate perceptions.

What can we learn from this? Friendly environments and healthy minds may result in overoptimistic predictions, for both birds and people, even when one would benefit from more realistic predictions. If you always see the world through rose-coloured glasses, you can reassure your perhaps somewhat annoyed friends, colleagues, and family that it is more likely a sign of a happy and supportive environment than a diagnostic criterion for a delusional disorder.

  • Take home message: There are several positive sides of being overoptimistic regarding time usage. Overoptimistic predictions may:

  • Compensate for pessimistic beliefs regarding our creativity and problem-solving abilities. If problems happen along the way, we have a good chance of solving them.

  • Be essential for innovations.

  • Benefit society at large.

  • Be a sign of a friendly and supportive environment.

4.3 The Desire to Control Time

Motivation to perform a task efficiently may result in lower time predictions, as well as lower actual time usage. The evidence suggests that the decrease in time predictions due to higher motivation to perform efficiently is frequently not matched by the decrease in actual time usage. Consequently, strong motivation to quickly complete work tends to lead to overoptimistic time predictions.

This tendency was documented in a study where participants were divided into two groups. The participants in one group received monetary rewards for being among the fastest performers of an origami (paper-folding) task. Participants in the other group were given the same task but did not receive any monetary rewards. The time predictions of the participants in the first group were, as expected, typically lower than those of the other group. The actual time usage, however, was, on average, about the same for both groups, leading to more overoptimistic time predictions for those who could earn rewards for fast performance [17].

A similar effect may be at play when people are motivated to finish tasks quickly because the tasks are perceived as important [18]. It makes sense that people think they will work harder and more efficiently if tasks are important or if they are rewarded for fast performance. The main mistake seems to be an overestimation of the flexibility of the work and performance. Many types of work and performance are not flexible enough to be completed substantially faster than normal, even when the motivation is strong.

The tendency of higher optimism when motivated to finish fast is more pronounced for people with a desire for control [19]. When motivated to finish fast, those scoring high on the personality trait desire for control predicted lower time usages even for a task that was completely uncontrollable, such as the time it would take to watch a bowl being filled with tap water. Thus, there seems to be a link between overoptimistic time predictions and a general motivation to control things. This is further suggested by results from a study on power and time predictions. One group of people was influenced to think that they were powerful by writing a vivid report of a situation in which they had power over another individual. Another group was influenced to think that they lacked power by writing about a situation in which other people had power over them. All the participants were then asked to predict the time needed to complete a formatting task. Those who were influenced to feel powerful gave more optimistic predictions regarding the time it would take to complete the task. In addition, those with a general feeling of being powerful, based on self-ratings on statements such as ‘I can get others to do what I want’, were found to give more optimistic time predictions [20].

  • Take home message: The motivation to do something quickly and the desire for control tend to result in lower time predictions. These two factors may lead to a reduction in the actual time usage through increased efficiency, but this reduction is typically not enough to compensate for the lower time predictions.

4.4 Motivation to Make Accurate Time Usage Predictions

Does greater motivation to produce accurate time predictions lead to higher accuracy? Previously (Sect. 3.7), we gave an example of how rewarding project leaders with high time prediction accuracy led to higher time predictions and a reduction in productivity to make the actual time usages fit the time predictions. Is this what normally happens?

In a study on incentives, some participants were told that they would receive $2 if they managed to predict their time usage to be within one minute of the actual time usage and $4 if they predicted it to be within 30 seconds [21]. Participants in a control group were also asked to predict time usage but received no monetary incentives for accuracy. The participants in the first group, with incentives to provide accurate time predictions, tended to predict too high and less accurate time usages than those in the control group. In this context, the participants did not lower their productivity to make their predictions more accurate, as in the previously described real-world case with project managers, indicating that the effect of higher motivation for time prediction accuracy on work productivity is task and context dependent.

Non-monetary incentives are also candidates for increasing the accuracy of time predictions. Time prediction accuracy was, for example, reported to increase when software developers and managers were made more accountable, that is, when the time prediction accuracy was part of the company’s performance review of those in charge of the time predictions [22]. Similar results have been reported by other studies. Note that the higher accuracy may not mainly be a consequence of better time predictions or higher time predictions; it may be, instead, a consequence of a stronger emphasis on adjusting, typically reducing, the work content and quality to fit the time prediction [23].

When overoptimistic time predictions are penalized but accurate time predictions are not necessarily rewarded, we will most likely see a tendency towards more pessimistic, higher time predictions. An example is that waiters who predict customers’ waiting times to be seated tend to exaggerate their time predictions to avoid the negative consequences of angry customers [24].

  • Take home message 1: Rewards for accurate time predictions tend to produce higher but not necessarily more accurate predictions.

  • Take home message 2: It is risky to reward accurate time predictions or penalize overoptimistic time predictions. Attempts to optimize prediction accuracy may come at the expense of other goals, such as productivity and quality.

4.5 Selection Bias

‘We only win contracts when we have been overoptimistic about how much time it takes to complete the work’, was the frustrated comment of a software company manager. His company had, for the nth time, won a large contract only to discover that the prediction of time usage, the basis of the company’s price offer, was far too low. In situations in which the company had a great amount of relevant experience and was more likely to be realistic about the time usage, it was less likely to be selected due to its higher bid price.

This real-life experience suggests that, even in a world without a tendency towards overoptimistic time predictions, we will still observe more time overruns than underruns. This phenomenon may sound strange but it is a logical consequence of selection bias and that we only observe the accuracy of time predictions that lead to action. The projects most likely to be started are those assessed to have the most favourable prospects, typically including overoptimistic views of the required time usage. Such selection bias, when applied to bidding at auctions or offering a price in competition with others, is usually termed the winner’s curse. The frustrated software company manager may think that the company has had bad luck because it only wins projects when it is being overoptimistic. The client may think that nearly all software companies have overoptimistic time usage predictions, because they nearly always incur time overruns. In reality, both are causing the observed tendency towards overrun: the software company, because it is not compensating for the winner’s curse (selection bias) when pricing its projects,Footnote 2 and the client, by emphasizing low price as a selection criterion.

To better understand the importance of selection bias in creating time and cost overruns, assume 10 painters offer to paint your house. Each of them predicts the time they will need for the job and they give you a price based on that. The painters vary in how fast they work and none of them knows exactly how much time they will need to complete the job. For the sake of simplicity, assume that it is just as likely that a time prediction is too high as it is too low, that is, unbiased time predictions, and that the painters do not react strategically to your selection strategy. Consider the effects of the following painter selection strategies.

  • Select the best painter: You do not care about the price and select the painter you think is the best. If everybody did this, there would be no tendency towards overoptimistic time predictions for paint jobs. In most cases, however, it is unreasonable to assume that price is no issue. Completely ignoring price has, for example, the unfortunate side effect that the price of hiring a painter soon goes through the roof.

  • Select the painter with the average price: The strategy of selecting the average price may seem strange but it has been used as an approach to reduce the problems related to selecting overoptimistic bidders [25]. This strategy may remove the tendency towards overoptimistic time predictions but may also lead to price manipulations in bidding contexts. If a painter collaborates regarding the bid price (colludes) with another painter, they may manipulate the average price and make higher bids more likely to be selected. It might also be difficult to motivate clients to select bids substantially higher than the lowest ones.

  • Select the painter with the lowest price among those assessed to have sufficient competence: This strategy, which may be the most common, typically means selecting a painter with a lower than average price. This means being more likely to select among those painters with overoptimistic time predictions. You will consequently tend to experience overoptimistic time predictions even if there is no general tendency towards overoptimistic time predictions among the painters.

If we emphasize a low predicted time usage or a corresponding low price when choosing painters, carpenters, software companies, and so forth, we are very likely to experience the world as a place full of overoptimistic time and cost predictions. We may complain about painters who never finish in time or carpenters who provide much lower quality than expected due to overoptimistic time predictions but, in fact, it is very much us and our strategy of selecting the lowest offers that have created this world of overoptimistic time predictions. The degree of this overoptimism increases with greater prediction uncertainty, a stronger emphasis on low price when selecting bidders, and higher numbers of predictions from which to select [26].

An underlying problem is that we typically do not know the extent to which a low time prediction is a reflection of higher competence or a consequence of greater overoptimism. Skilled people can do the work in less time and should predict lower than average time usages. Lower time predictions may, on the other hand, also be a consequence of lack of skill in terms of not realizing the extent and complexity of the work. The latter case is called the Dunning–Kruger effect: those who know less know less about what they do not know; in short, they are unskilled and unaware of it [27].

The Dunning–Kruger effect was illustrated in a study of 35 software companies predicting the time usage and cost of the same software project [28]. Seventeen of the companies were allowed to gain more knowledge about the project by participating in a pre-study phase. Those companies made bids, on average, as much as 70% higher than the companies with less knowledge about the required time usage to complete the work. The study also found that the companies with experience from similar projects made bids about 60% higher than those without this experience. If you are a client with a strong focus on low price and a correspondingly lower focus on selecting workers based on their competence, you may consequently experience a tendency towards not only overoptimistic time predictions but also towards selecting less competent workers.

That said, if you exclude potential providers because they gave the lowest time predictions, this may have the unfortunate consequence of you bypassing the most skilled providers. We once asked seven companies to bid for and then complete the same software development project. In this case, the company with the lowest time prediction by far (only 18% of the median time prediction) turned out to be the most competent. That company managed, to our great surprise as clients, to finish on time, with the predicted use of work hours, and with good quality [29]. Not knowing the reason for a very low time prediction, which could be either the presence or absence of high competence, makes the lives of those who select workers difficult.

When we make decisions, such as who to hire for a job or what activity to start, we often try to optimize something, such as value for money, worker competence, or the best quality for a given amount of time usage. A counterintuitive consequence of optimization is that the more we optimize, the more likely we are to be disappointed! Optimizing typically improves the quality of the choice but, since the world is uncertain and our beliefs are inaccurate, the option we select as the best will tend to be less favourable than expected. This effect is referred to as the optimizer’s curse or post-decision surprise [30] and is based on the same statistical phenomenon as the winner’s curse.

Given that the best project is typically not as good as it seems, should we then pick the second or third most promising project? Probably not. The best-looking project still tends to be the best. The point is that picking the best alternative makes it more likely that you will be disappointed when comparing your expectations. If you do not like disappointments, be aware that high uncertainty, many options, and a strong focus on optimization lead to projects and options looking better than they actually are.

A solution to the problem of selection bias in the case of time predictions would be everyone increasing their time predictions when expecting to be in a winner’s curse situation. Such an increase sometimes develops naturally in mature markets and is then called a winner’s curse effect, that is, an effect of an awareness of the winner’s curse. In situations with more alternatives, greater uncertainty, and stronger competition, the bidding will then be more conservative and, in some contexts, even have the consequence that increased competition actually leads to higher, not lower, time predictions or prices [31].

It is challenging for service or product providers to determine how much to add to the time prediction or cost estimate to avoid the winner’s curse. This typically requires information one normally does not have. We found that, for certain typical software development project situations, with five to 10 companies competing for the same project, it may make sense to add about 15% to the time predictions to compensate for the winner’s curse [26]. Different situations, however, require different adjustments. In many situations, a better solution than increasing the time prediction might be to emphasize high time prediction accuracy, for example, by providing more and better information and motivate bidders to spend more resources and using better techniques for the time prediction work. When time predictions are accurate, the selection bias effect will be low.

  • Take home message 1: Even if people’s time predictions have no tendency towards overoptimism, the world would still appear to be full of overoptimistic time predictions. This is simply because tasks and options with overoptimistic time predictions are more likely to be selected and lead to action, which is a requirement to be evaluated in terms of the degree of overoptimism. Options with overpessimistic time predictions are less likely to lead to action and will consequently not be evaluated.

  • Take home message 2: The more we optimize, for example, the more strongly we emphasize low time predictions and low prices when selecting among alternative investments, the more likely the outcome will be worse than predicted. This does not mean that we should avoid optimizing when choosing—only that we should prepare for an outcome less positive than predicted.

4.6 Deception

Norway has a long tradition of winter sports and the city of Lillehammer, with a population of less than 25,000 inhabitants, became the host of the 1994 Winter Olympic games. The original cost prediction of the Olympics was around €100 million, which would result in a financial profit and not cost the Norwegian taxpayers a thing. The International Olympic Committee required the application for the Olympics to have a financial guarantee by the state, which was decided at €180 million. The Norwegian Parliament had no problem providing this guarantee. One reason for this was that the minister responsible was so confident regarding the accuracy of the early cost prediction that he claimed this guarantee would under no circumstances be effectuated. The parliament gave the financial guarantee without much discussion.Footnote 3

In reality, the Lillehammer Olympics cost close to €1 billion and incurred a financial loss of at least €500 million [32]. When we look at the early cost prediction process, it is apparent that there were strong incentives for underpredicting the costs. The initial prediction of €100 million was provided by the county of Lillehammer. Since Lillehammer would bear little of the cost and receive most of the benefits from the large investments, it had strong incentives to provide low initial cost estimates and to add new cost items after the application was sent to the International Olympic Committee [33]. Similar situations have arisen in many recent infrastructure and construction projects, where parties with a strong interest in starting the projects were involved in the early stages of time usage and cost predictions [34].

Does the very strong underprediction of time usage and cost in the Lillehammer Olympics and other project situations mean that people are lying to have projects started and investments accepted? Or is it the case that people actually believe—perhaps because they want to—in their overly optimistic time and cost predictions?

It would be naïve to think that people never deliberately underpredict time usage or costs to have projects, actions, or investments approved [35]. In a study on infrastructure projects, a transportation planner stated [36]:

You will often, as a planner, know the real costs. You know that the budget is too low but it is difficult to pass such a message to the counsellors [politicians] and the private actors. They know that high costs reduce the chances of national funding.

Another study, with software professionals, reported that some managers intentionally produced too low time predictions to create projects that would look more attractive for top management making the decisions [37].

The strategy of making projects look better than they are by lowering the time or cost prediction is an instance of deception if people give predictions in which they do not believe. It may also count as deception if management simply lowers the predictions given by their project managers or developers without a reasonable argument for lower time usage or costs. A decrease in the time predictions would, however, be the result of good intentions and not deception when based on plausible arguments, such as suggestions related to reducing the content of the delivery or simplifying the solution.Footnote 4 So, how much of the overoptimistic predictions are due to deception or lying?

Attempts have been made to quantify the amount of lying in time predictions. In a survey of information technology (IT) professionals, 66% reported having lied in relation to predictions of cost and time [39]. The category most frequently chosen by the respondent was that they believed—but did not know for sure or could not document—that lying occurred in 50% or more of projects. The perception of the frequency of lying in software project time predictions varied greatly, from very low (0–10%) to very high (90–100%). Another study of IT professionals found that time prediction distortions, which, in this context, are more or less the same as deception or lying, seldom happened [40]. However, instances in which management required too low time predictions, with no argument on how to reduce the time usage, were believed to be common.

The samples of these two studies were not large and we do not know how representative the selected contexts were or what the lies were about. Nevertheless, the studies demonstrated that overoptimistic time and cost predictions are sometimes best described as lies, deceptions, or deliberate distortions of the time predictions. There are researchers claiming that deception is the most important reason for overoptimistic time and cost predictions [41]. While people no doubt sometimes deliberately distort predictions to gain approval of projects from their boss, their colleagues, the public, or politicians, the claim that deception is the most important reason is, at least for time predictions, not well documented.

One potential indicator of deception in time and cost predictions is found in the typical increase of predicted costs from the initial stage, where the decision to invest or not is taken, to the project’s startup stage, where the project execution is planned and budgeted [42]. A report on 31 large road projects in Norway found that the average cost overrun was 0%, with a worst case overrun of 37%, when compared with the budgets established at the time of project startup. However, when tracking the history of the projects and looking at the early cost predictions presented to parliament as a basis for the decision to start the project or not, the predictions were much more overoptimistic [43]. On average, the projects were now 53% more expensive than initially predicted and the worst case was a project with an actual cost 137% higher than the initial prediction.

The obvious reason why it is difficult to document deceptions is that they are not meant to be disclosed or discovered. Another reason is that it is sometimes hard to establish to what degree a too low time or cost prediction is caused by judgement biases, selection biases, bad luck, or deception. A very low but not impossible time prediction may not always be considered a lie. How unlikely should a time prediction be to be perceived or determined as a lie?

When people in charge of the time predictions openly admit that they gave lower time usage predictions than they believed were possible, we can know for sure that deception was involved—but this rarely happens. Thus, the explanation that people frequently lie when giving time predictions must usually compete with other explanations of overoptimistic time predictions.

  • Take home message: We do not know how often or when deception is the main reason for overoptimistic time predictions. There is, on the other hand, no doubt that deception arises and does so more frequently when the person or organization making the time prediction has a strong incentive to give a low time prediction, for example, to increase the likelihood of being allowed to start their pet projects.

4.7 Who Makes the Most Realistic Time Predictions?

A famous economist once visited a company producing jeans priced as high as $250 [44]. He concluded that there must be a lot of money out there, given the willingness to pay so much for jeans. Partly based on this insight, he predicted that the inflation rate would become unusually high in the near future and he turned out to be amazingly correct. His prediction led to fame as a prediction expert, especially since most of the other experts made more conservative and less accurate inflation rate predictions. But how good a prediction expert was this economist in reality? Was he right for the wrong reasons?

Forecasters, such as the famous economist above, successful at predicting large changes, have, on average, the least accurate predictions [45]. This phenomenon may sound counterintuitive but it has a natural explanation. Predicting large changes from historical performance typically means emphasizing specific information and neglecting aggregated information, such as long-term trends or average productivity. The strategy of reacting strongly to one or very few indicators is more likely to lead to spectacular predictions and, hence, fame if the predictions are correct. Such a strategy, on the other hand, is also more likely to lead to incorrect predictions, due to its overreaction to specific information. Returning to our example, the famous economist was among those with the least accurate predictions in the period after his fame as a spectacular forecaster. This suggests that the main reason for his fame was not exceptional prediction skills, but rather luck and a willingness to use specific information (e.g. sales figures for expensive jeans) instead of aggregated information.

There are several domains where luck and bad luck—and not prediction skill—explain most of the variance in the prediction performance. There are even domains, such as stock market investment, where all or nearly all of the variance of forecasters’ prediction performance seems to be random.Footnote 5 Time prediction performance may not be as bad as that, but there seems to be large element of random variance in who makes the most accurate time predictions.

Even if we cannot expect anyone to consistently predict time usage accurately in contexts with uncertainty, it is meaningful to try to find out more about the characteristics of people who tend to give more accurate time predictions and in what settings they are more accurate. The research on this does not give very clear answers. Nevertheless, some connections, and lack of connections, between personal characteristics and prediction error are worth reporting.

Women versus men: We all know it: women are more rational than men. They fight less, care more about other people, and perform better than men in most tasks that require a brain. For example, an analysis of the Programme for International Student Assessment data from 2000 to 2010 shows that girls outperform boys in 70% of the countries, whereas boys outperform girls in only 4% of the countries [47]. Should we perhaps leave the time prediction and planning work to women?

Most time prediction studies, even those where the participants are asked to indicate their gender, omit reporting gender differences. This suggests typically no systematic difference in time prediction skill between men and women, because researchers tend to publish anything they find statistically significant, even if it is not part of their original hypotheses. A few studies, on the other hand, have suggested that men are more—but only a little more—overoptimistic in situations involving incentives for fast performance or when performance can be interpreted as a form of achievement [48, 49]. In total, however, we have not been able to find good documentation of when or why men will be less accurate or more overoptimistic than women in time prediction contexts. Most likely, gender is not a good indicator of time prediction accuracy.

Intelligent versus less intelligent people: How about those who perform better at school or in IQ tests? Are they better at making time predictions than other people? Probably not, according to research [50, 51]. At best, they may be slightly more accurate in predicting time for academic tasks. These are tasks they are better at performing in the first place, so if they happen to be slightly better at predicting time for such tasks, this is not very impressive.

People who believe they are good at planning and predicting: Asking people about how good they think they are at predicting time usage does not appear to be a reliable way of finding people who make accurate predictions. On the contrary, overconfidence in prediction abilities may be associated with overoptimistic time predictions. Despite not being a bulletproof personnel selection method for time prediction tasks, there may be something to self-reports of prediction accuracy. Those who say that they are good at planning and at deciding how long it will take to complete a task actually tend to be slightly better than those who do not agree with this characterization [52, 53]. Confidence in the accuracy of a particular time prediction, on the other hand, is not a robust indicator of time prediction accuracy [9].

Conscientious people: People who are generally more careful, diligent, and painstaking in life tend to give themselves more time to complete tasks. This can, in some cases, reduce the risk of too low time predictions but, in other cases, it can increase the risk of too high time predictions [54, 55].

Happy-go-lucky people: People who are better at enjoying the present moment and who do not worry much about the future or about past mistakes tend to predict that boring tasks will be completed relatively early. However, such people actually finish later than those who do not possess such qualities [56]. In other words, they are not the type of people you should trust the most when requiring time predictions.

People trained in time prediction: If we could train people to become better at time prediction, this could reduce problems related to cost and time overruns. When time predictions are required, we would then simply select those who have been trained for such work. How do we train people in time prediction and how effective is the time prediction training?

Training in the sense of learning about biases and the use of proper judgement and decision methods seems to have—but beware of the limited evidence—a positive effect on judgement and decision quality [57]. However, the standard training method of learning through feedback and reflection does not seem to work well for time prediction. As an illustration, a learning process called the Personal Software Process [58] includes a time prediction training program with feedback of actual time usage and analysis of reasons for deviation between predicted and actual time usage. Independent empirical studies analysing this training found no improvement in time prediction accuracy from following this program [59, 60]. Similar results were found in another study on the effect of a feedback and lessons learned training program.Footnote 6 Those who followed this training program were instructed to spend time reflecting on what went wrong, what went well, and what could be learned from previous accurate or inaccurate time predictions. The training program did not lead to improvements in time prediction accuracy compared to individuals who underwent no such training program.

Amount of task experience: Generally, more experience in completing a task tends to lead to more accurate time predictions. Research participants without experience in completing the Tower of Hanoi puzzle [62] had, for example, more overoptimistic time predictions than those with some experience [63]. It is, however, not always the case that more experience in completing tasks leads to higher time prediction accuracy. Experience may change the way people think about a task. As people gain experience, their mental representation of the task tends to become more abstract, with information stored in a few larger chunks rather than as many smaller entities that represent details of the task. This has the paradoxical effect that more experience can lead to more overoptimistic time predictions. In a study on predictions of the duration of piano tunes, participants brought their own sheet music and were asked to predict the time it would take to play tunes that were recently learned, well learned, and extremely well learned [64]. The piano players gave 48% too high time predictions for recently learned pieces, 11% too high time predictions for well-learned pieces, and 19% too low time predictions for extremely well learned pieces. This pattern was apparent for both novices with an average of six months of experience and advanced players with an average of 11 years of experience. The results suggest that the piano players ‘chunked’ together larger parts of the well-learned pieces, whereas they were more conscious about the different parts of the more recently learned tunes.

More overoptimistic time predictions among those with more experience were also found in a study where participants were instructed to fold a paper rabbit (an origami task) [65]. The participants were first asked to produce one (low-experience group), three (medium-experience group), or nine (high-experience group) rabbits. None of the participants received feedback on their time usage. The participants were then asked to predict how long it would take to make three more rabbits. Participants in the low-experience group tended to predict too high a time usage, whereas those in the medium-experience group and, to an even greater extent, those in the high-experience group predicted too low a time usage.

Before we interpret the results above as evidence of a detrimental effect of experience on time prediction accuracy, we should take a closer look at the studies. The piano and rabbit origami studies showed more underestimation with experience but the accuracy of the time predictions actually improved with experience. Although the more experienced participants tended to be more overoptimistic, they also held a more accurate view of their own performance. Furthermore, the participants in the two experiments did not receive any feedback on actual time usage. Perhaps the optimism in the high-experience group could have been reduced with feedback [66].

In professional life, the tasks are usually not as well defined as in the rabbit origami and piano studies. Each new action, for example, does not usually follow the previous one in a predetermined sequence. More complex situations seem to increasingly favour the time predictions of people with more experience. In contrast to the piano and rabbit origami experiments, a study of software professionals found that years of relevant project experience improved the accuracy of time predictions [67]. Other studies in similar contexts have shown that professionals with more relevant experience produce higher and more realistic, time predictions [68]. In particular, more relevant experience seems to be related to better ability in identifying required activities to complete a larger task or project and in assessing their complexity [28]. Ironically, this consequence of more experience is not always an advantage for those making the predictions. The ability to identify more required activities and complexities may be associated with higher time predictions and higher predicted costs. If a client selects someone based on low time predictions or price, they may select those with the least relevant experience, because the providers with the lowest bids may have forgotten to include essential activities in their time predictions.

It is essential to distinguish between the relevance of experience, in terms of similarities between the previous and current tasks, and the amount of experience, for example, the number of years solving tasks in the same domain. While the relevance of the experience has a clear relation to better time prediction accuracy, the amount of more general, not closely related experience usually leads to very little, if any, improvement [69].

Previous time prediction accuracy: A reasonably good and perhaps the best indicator of how much you can trust a person’s time predictions is the person’s past time prediction accuracy. In a study of software developers, previous time prediction overoptimism was the best indicator of future time prediction overoptimism [9]. Being the best indicator does not mean, however, that this indicator is very good and the study results suggest a large random element in time prediction accuracy, which should not be surprising given the typical wide and long-tailed distribution of time usage for similar tasks. Nevertheless, even an imperfect indicator is better than no indicator and, of all the indicators discussed in this section, this one seems to be the best, together with the amount of highly relevant experience.

Superforecasters: Tetlock and his research team completed a series of studies on what they referred to as superforecasters [70]. This work does not concern time predictions but is very interesting and impressive and has parallels to the findings cited earlier in this section. Tetlock’s research team recruited thousands of highly educated people from various professional- and science-related arenas (alumni networks, science blogs, etc.) for a massive prediction contest. The participants made a range of geopolitical predictions, such as ‘will the official euro to U.S. dollar exchange rate exceed 1.40 before December 31, 2014?’ After one year, the very best of the participants were selected as superforecasters and allowed to continue the contest in an environment where they could post questions and comments to other superforecasters. The superforecasters turned out to outperform the other contestants by a substantial margin in the second and third years of the tournament, that is, they were not just lucky. Even more surprisingly, they beat the predictions of professional intelligence analysts with access to classified information.

Why did the superforecasters perform better than the other forecasters? The study did not give a definite answer, but what distinguished the superforecasters from the other contestants was the following: they were highly motivated, highly intelligent and knowledgeable, enjoyed solving problems, were open-minded, had a scientific worldview, were thinking probabilistically, were willing to admit errors and change course, and were eager to obtain and share information. The research on superforecasters seems to reflect some of the points made above regarding who makes the best time predictions. For instance, time prediction accuracy was higher among those who were good at prediction in the past, accuracy was higher among highly knowledgeable individuals (i.e. specific task experience). Furthermore, the superforecasters were better at thinking probabilistically and using historical information (the essential ingredients of good predictions pointed out in the Chaps. 3 and 6), and information sharing in groups seemed to contribute to better predictions (see Sect. 7.6).

  • Take home message 1: Much of what we consider good prediction skills is likely the result of random variation and luck.

  • Take home message 2: There are only a few reliable indicators of who will give the most accurate time predictions and none of these indicators are very good. You will obtain somewhat better time predictions from people with highly relevant experience and from people who have made accurate time predictions in the past.