Predicting in shock: on the impact of negative, extreme, rare, and short lived events on judgmental forecasts

Abstract

The occurrence of unexpected events that are extreme in magnitude, rare in frequency, and short-lived in duration poses distinctive challenges to decision makers and planners. In this paper we examine the impact of negative versions of these events, which we term “shocks”, on the judgmental forecasts of subjects experiencing them. A behavioral experiment asking participants to forecast monthly time series in the presence of temporary but extreme decreases in those series is used. Average changes to annual prediction intervals and 1-month ahead forecasts were much smaller than the magnitude of the shock and occurred in proportion to the size of the shock. Changes to prediction intervals were more persistent for moderate than large shocks, and larger for shocks occurring a second time. Our results provide supporting evidence for the view that decision makers underweight rare and extreme events rather than overweight them, consistent with a discounting or forgetting effect. The behavioral findings are relevant to operations researchers involved in expert judgment elicitation and in supporting decision making.

Appendix 1: Study materials

This appendix contains materials used during the experiment. At the start of the experimental session, each subject was given a handout with the introductory material contained in Sect. 7.1 and asked to read this carefully. Once they had read the information sheet, students started the task by clicking a “start” button on a spreadsheet. They were then asked for estimates using prompt boxes, using the wording given in Sect. 7.2. A sample spreadsheet, showing a partially completed task, is shown in Fig. 6.

1.1 Introductory material

Please read the following instruction carefully.

Please consider the following situation: You have just been hired as an analyst by a trading company. Your task is to forecast monthly share prices for four different shares.

The company’s policy is not to tell you whether it has bought or sold shares based on your forecasts so that you can focus solely on accurately forecasting prices and not worry about the company’s performance. It is also company policy to only buy and sell shares in very large quantities, so the stakes are high and it is crucial that you are as accurate as possible in your forecasts.

Your forecasts should be based only on the past prices of each share. That is, imagine that you have no information about the traded company, the market it operates in, the stock market where it is listed, or any other information whatsoever. The shares you will see are not related to one another in any way.

For each share, you will be asked to do the following:

• 1. View a graph of monthly past share prices, from August 2003 to July 2011.

• 2. Give some forecasts. Begin by clicking on the “Start Simulation” button in the top-right corner of the graph of share prices. During the task, you will be asked for two types of forecasts:

• 3. Your best prediction for the next month’s share price. You will be asked for this estimate every month, for 24 months. After giving your estimate for a month, you will be shown the true price for that month, along with your prediction error, and then asked about the following month. Please be sure to check your estimates before clicking OK. Once you have clicked OK you will not be allowed to go back and make any changes.

• 4. The highest and lowest share prices that you think are possible for the next 12 months (such that you are very confident that no share price in the next 12 months will be higher or lower than the values you give). You will only be asked for this estimate once per year (each July). Again, please check your estimates before clicking OK.

• 5. At some stages during the task, you may be asked about how much you would be willing to pay to purchase an investment. This question is not about your mathematical skills, and there are no right or wrong answers. Instead, we ask you to estimate intuitively at which price you would be neither happy nor unhappy about the purchase of the investment.

Please note that you have plenty of time for this task (roughly 40 min in total), so there is no need to hurry. The student in the course giving the most accurate predictions overall (by the criterion of ‘mean square error’) will receive a prize as a reward.

Now please download and open the file “ShareForecasts.xlsm”, enable macros, and click on the “Start Simulation” button. You will then be asked for an identification number. Your identification number is X.

1.2 Question wording during task

Upon commencing with the forecasting task subjects were prompted the following:

Your boss asked you for some first year estimates before you start forecasting monthly prices. Click OK to provide them.

They were then asked to

• estimate the lowest share price for the next 12 months (e.g. until Aug/2013), such as you are very confident it will not be below it.

• Estimate the highest share price for the next 12 months (e.g. until Aug/2013), such as you are very confident it will not be above it.

This sequence of prompts/questions was repeated every 12 months. After giving the estimates above, subjects were repeatedly prompted for monthly point forecasts using the following:

Please estimate share price ($) for next month (e.g. Jul/2012)

Figure 6 shows a partially completed forecasting task.

Fig. 6

Screenshot showing the Excel interface used for the experiment. The task is partially complete