Letters in Spatial and Resource Sciences

, Volume 10, Issue 1, pp 109–127 | Cite as

Relating cost-benefit analysis results with transport project decisions in the Netherlands

  • Jan Anne Annema
  • Koen Frenken
  • Carl Koopmans
  • Maarten Kroesen
Open Access
Original Paper

Abstract

This paper relates the cost-benefit analysis (CBA) results of transportation policy proposals in the Netherlands with the decision to implement or abandon the proposal. The aim of this study is to explore the relation between the CBA results and decision-making. Multinomial logit regression models and Latent Class Analysis are used in this paper as the statistical tools to identify associations between CBA results and decisions and to reveal unobservable classes underlying the CBA results and the decisions for projects. Analysis was carried out on 106 Netherlands CBA reports (2000–2012) containing 454 observations. Each observation is a CBA result of a transport project variant. In line with most of the international literature, this study cannot find a significant association between Net Present Values (NPVs) and the variants chosen in political decisions (after controlling for other relevant variables). However, a positive NPV does keep variants ‘pending’, preventing a negative decision.

Keywords

Cost-benefit analysis Transportation Decision-making 

JEL Classification

D61 D62 

1 Introduction

In many countries, cost-benefit analysis (CBA) of transport projects has become common practice over the last two decades (Willis et al. 1998; Lee 2000; Hayashi and Morisugi 2000; Nakamura 2000; Odgaard et al. 2005; Annema et al. 2007; Mackie and Kelly 2007; World Bank 2010). However, the role of the CBA in political decision-making processes is a complex one. A priori, the expectation is that transport project proposals with a larger expected social return have a higher chance of being approved by politicians. Nevertheless, as our literature review shows, there is an increasing amount of evidence that CBA outcomes have only a limited impact on political decision-making.

Our study analyses the CBA’s for transport projects in The Netherlands between 2000 and 2012. In contrast to most earlier studies (Table 1), which focused on road projects, we include all types of transport projects. Furthermore, we have collected data for both national and regional projects. As such, our study is the first large-scale, systematic statistical analysis on the impact of a CBA outcome on political decision-making.

The paper is structured as follows. We present a literature review in Sect. 2 and produce some contextual information in Sect. 3. Section 4 discusses the methods and Sect. 5 the results. We end with some concluding remarks (Sect. 6).

2 Literature overview

Since the 1970s several scholars have studied the relationship between CBA outcomes and transportation decision-making (Table 1). Only the oldest two studies found (McFadden 1975, 1976) show a clear relation: decisions on highway routing in the 1970s in the US could be partly explained by the BCRs of these highway routings. All other and more recent international studies show a limited or no statistical relation between decisions and composite CBA information such as the benefit to cost ratio (BCR) and the net present value (NPV), as shown in Table 1. Interestingly, some studies conclude that other (CBA) information than the composite results was used in the decision such as in the studies by Fridstrom and Elvik (1997), Nellthorp and Mackie (2000), and Odeck (2010). For example, Fridstrom and Elvik (1997) concluded that costs were weighed relatively heavy and that smaller projects were preferred to larger projects, given the BCR. Inspired by these international studies we will also not only evaluate composite CBA information but also CBA components.

Concerning the usage of CBAs, a World Bank (2010) study also showed that their use of CBAs dropped considerably in the past decades. And, Proost et al. (2011) make clear that political decisions for multi-billion EU transport infrastructure projects are even made without support from CBA results. Here, the Dutch CBA practice is different (see also next section), because since the start of the Dutch CBA practice in 2000 CBAs have been made mandatory for large projects and have also been carried a substantial number of smaller projects and for a variety of transportation project categories. Possibly, due to this popularity, we might expect a priori that in Dutch decision-making CBA results play a more influential role in decision-making.

Nevertheless, in general the international literature points out that CBA results hardly affect actual decision-making. Although there is a huge knowledge base giving explanations why decision-makers use knowledge to a limited extent (e.g., Bax 2011; Landry et al. 2001; Kasemir et al. 2003), literature which explains why CBA results are hardly influential is scarce. Annema (2013) speculates that five barriers might explain the limited use of CBA results. One barrier might be related to a CBA often being just one part of a considerable amount of information about projects which decision-makers have to digest. So, he thinks it plausible to assume that some decision-makers just miss the CBA information, forget about it when the time of final decision-making arrives or do simply not have the time to really scrutinize the CBA information received. A second barrier is possibly a lack of understanding. A third family of barriers mentioned might be related to acceptance and trust. Politicians—even the ones who actually understand the CBA—may not accept the normative premises of the technique or they may not trust the effects estimated. A broad fourth barrier category distinguished by Annema (2013) was denoted as ‘political’. For example, some politicians may just think that expert techniques should not provide the final answer. Or, politicians may take only the sentiments of the regional or local community into account when making a decision on building new infrastructure. Annema deems it logical to assume that the four barriers mentioned might reinforce each other, resulting in a fifth ‘reinforcing’ barrier. Perhaps, as (Annema 2013, p. 306) states ‘it is more a miracle that CBA information has been used in some mega-project cases at all’.

If in the Netherlands practice between 2000 and 2012 CBA information was used at all, we will now analyze.

3 Context

For most transportation projects in the Netherlands it is compulsory to carry out a CBA in the decision-making process. However, it is not mandatory to act according to the CBA results. Consequently, a project with a highly negative Net Present Value (NPV) can still be implemented if the project receives a majority vote in parliament and/or in the regional councils. In other words, in the Dutch political context CBA is seen as a mandatory input in the decision-making process, but the tool is not meant to replace the political decision-making.

CBA is used for a wide range of projects. In 2000 the Dutch government decided that conducting a CBA was mandatory for large infrastructure transportation projects and should follow a CBA guide (Eijgenraam et al. 2000). Like in the newest guide to Cost-Benefit Analysis of the European Commission (EC 2014), the Dutch CBA practice relates to a broad welfare analysis, thus, including evaluation of non-market impacts of transport projects. In the period 2000 and 2012 it became customary to also apply CBA to smaller investment projects, for non-investment projects such as road pricing, for regional projects and so forth. In 2012 CBAs were made mandatory for all national projects included in the so-called ‘Long-range Plan for Spatial and Transport Projects’ (MIRT in Dutch) and for all regional projects which require a subsidy from the national government exceeding 112.5 million Euros1 (in 2010 prices, Ministry of Infrastructure and the Environment 2012)2 In the Netherlands a CBA is used to evaluate individual projects and not, as is sometimes the case in Scandinavia (Table 1), as a tool for ranking projects. Cost and benefits are computed from an economic point of view, including non-monetary effects such as travel time for consumers or CO\(_{2}\) emissions. The CBA metric used in Dutch practice is NPV, as this is prescribed in Netherlands CBA guidelines. In some CBA reports additional metrics are presented, such as the BCR or the (internal) rate of return, but this is not standard practice. Therefore, our analysis solely focusses on NPV as a main potential explanatory variable (see next section) for decisions and not on other CBA metrics such as the (economic) rate of return (see Del Bo and Florio 2010, for an analysis of this metric in ex ante project appraisal).
Table 1

Overview international studies relating CBA results to decision-making

Study

Main aim

Method

Main conclusion

McFadden (1975, (1976)

To test the form of benefit-cost calculus utilized by the California Division of Highways

Multinomial Logit Modelling (MNL)

The highway routing decisions (1958–1966) could mainly be explained by economic criteria, particularly the benefit-cost ratio (BCR)

Nilsson (1991)

To elucidate the influence of principles of welfare maximization on Swedish road investments in 1986

Binomial Logit Modelling (BLM)

It proved difficult to explain decision-making behavior by referring principles of welfare maximization

Odeck (1996)

To explore the priorities for regional road investments in Norway, with particular emphasis on the use of benefit-cost calculus

MNL and interviewing

Only in four out of fifteen regions BCR was found to be a significant explanatory variable. This ratio explained more than 30 % of the observed variation in only one region. The reasons given for this modest BCR impact by the road authorities was that several important impacts are not valued monetarily and therefore not included in the BCR

Fridstrom and Elvik (1997)

To reveal the preference of the regional Norwegian public road administration

MNL modelling based on the National Road Plan 1990–1993

Costs and benefits appeared to be of only marginal importance for the priorities set. More weight is attached to costs than to benefits. Smaller projects are preferred to larger projects, given the BCR

Nellthorp and Mackie (2000)

Reproducing the pattern of decision-making on 68 UK road schemes

Binomial Logit Modelling

The researchers were unable to find any statistical evidence that the pattern of decision-making was systematically related to Net Present Values or BCRs. Decision makers used other appraisal factors that were not valued in monetary terms, such as the scheme’s impact on noise, landscape, heritage, reliability and regeneration

Odeck (2010)

To examine which factors determine the rankings of road projects by Norwegian decision-makers

MNL modeling

CBAs per se did not matter for decision-making, but their components were taken into account in non-monetary units or partially rather than in a composite BCR or NPV

Eliasson and Lundberg (2012)

To study how CBA results were used (Swedish National Transport Investment Plan 2011–2012)

Document analysis and interview

Planners used CBA results primarily as a screening tool to avoid investment with negative net benefits. Politicians selected investment essentially at random, from a pure CBA perspective

Welde et al. (2013)

To explore how benefit-cost efficiency affects road investment decision in Sweden and Norway

Document analysis; binary logit models

Sweden was found to have a more systematic use of CBA compared with Norway. The conclusion drawn was that economic profitability seems to matter more in Sweden

Homan (2014), Homan et al. (2014)

US studies to determine, amongst others, if the most deserving surface transportation infrastructure projects, based on an applicant’s CBA and the likelihood that benefits exceed costs, are more likely to receive grant funding

MNL modeling

The outcome of the benefit-cost analysis was not a statistically significant factor. The results suggested that policy considerations other than a comparison of benefits versus costs were the primary driver in project selection

For more technical details on the Dutch CBA practice we refer to the CBA guides (Eijgenraam et al. 2000; Romijn and Renes 2013). Basically, the Dutch practice entails a state-of-the-art approach from problem analysis, establishing baseline alternatives, defining policy alternatives, estimating and valuing costs and effect of the policy alternatives compared to the baseline alternatives (for transport impacts often using transport modelling), calculating all costs and benefits discounted to the same base year and calculating the NPV. Annema et al. (2007) have evaluated 7 years of Dutch CBA transportation evaluation in depth. They have analyzed both contents as well as process related issues. For a discussion on attitudes of key actors in the Dutch Cost-Benefit Analysis (CBA) practice towards the role of CBA in the decision-making process for infrastructure projects, we refer to Mouter et al. (2013).

In this research 106 CBA reports are analyzed concerning transport policies (see next section). As remarked earlier ex ante transportation CBAs in the Netherlands are used in the decision-making process for a wide range of transport projects: from relatively small local projects such as new locks in inland waterways (tens of million euro investments) to very large and expensive high speed rail infrastructure projects connecting the Netherlands with Belgium or Germany (billions of euro). Next to investment projects also transport policies such as road pricing, increasing speed limits and abandoning Liquefied Petroleum Gas (LPG) as transport fuel have been scrutinized with CBA. The median investment costs of all transport projects considered in this analyses is 0.4 billion (Table 3). The Netherlands decision-makers do not receive only CBA information to make their decision but also Environmental Impact Assessment reports and eventually other reports on, for example, regional employment impacts, EU funding opportunities and so forth.

4 Methods

The methods used in this paper include document analysis, Multinominal Logit Modeling (MNL) and Latent Class Modeling (LCM). Document analysis is used to collect quantitative CBA information (e.g., Net Present Value) for each project evaluated using CBA in the period 2000–2012 and to determine the political decisions per project. This information is used, amongst others, to estimate the MNL and LC models. We will now elaborate on the three methods used.

Document analyses to find information Quantitative potential decision-influencing information was distilled out of 106 CBA reports.3 Our aim was to include as many transportation CBA reports as possible, for which a snowball method was used. We started with two smaller existing CBA report collections (Rienstra 2008 and TU Eindhoven) and by using internet and the authors’ network the collection was extended. Our collection is large but cannot be considered to be the whole population of projects for which a CBA was carried out.

The 106 CBA reports analyzed contain 454 project variants, or specific designs. For example, one project aimed to expand the highway network around Amsterdam Airport Schiphol. This expansion can be designed in many different ways such as two or three lanes or trajectories more to the north or to the south of the airport, and so forth. Each ‘different way’ of designing this project scrutinized with a CBA is a so-called project variant.
Table 2

Overview of project categories, reports, variants and the average amount of variants per report per project category, spatial scale and CPB involvement

 

CBA reports

Project variants

Category as assigned to in the MNL model and LCA

Project variant

   

Roads

26

152

Road

Rail persons

15

70

Rail

Road pricing (including toll roads)

11

50

Excluded in the models

Inland waterways

14

35

Other

Public transport (no heavy rail)

7

43

Other

Rail freight

6

25

Rail

Seaports

8

22

Other

Construction works (e.g., bridges)

9

31

Other

Airports

3

14

Other

Other (e.g., increasing speed limits)

7

12

Other

Total

106

454

 

Spatial scale

   

International

20

73

International

National

45

194

National

Regional

41

187

Regional

Total

106

454

 

CPB involvement

37

200

CPB involved

Table 2 presents descriptive statistics of the various project variants and three categorical variables considered in this paper as decision-influencing information (explanatory variables). The first categorical variable is the project category which may influence decisions as politicians may prefer certain project categories regardless of the CBA outcomes. As shown in Table 2, in the MNL and LC models used in this study the project categories are aggregated into three categories: road, rail and other transport projects such as sea ports, air ports and inland waterways projects. This avoids empty cells in the cross-tabulation between this variable and the dependent variable (the no go/go decision), which would lead to estimation problems. We have tried to estimate models with more then three project categories but these attempts did not result in meaningful results. The project category ‘road pricing’ is excluded from the models (see Table 2) because the variants were all rejected, despite the fact that 85 % of the pricing project variants scrutinized showed positive to very positive NPVs. Including this category in the analysis distorted the regression and LCA. More importantly, modelling this category provides little information: for this category there is hardly a logical relation between CBA results and decision-making.

The second categorical explanatory variable is the spatial scale of the project variants proposed which might also influence decisions regardless of the CBA outcomes. The scale ‘International’ concerns transport policy proposals aiming to improve the connection between the Netherlands and foreign countries, e.g. a high speed rail link, airport improvements. The national government is the level of decision-making for these international projects. ‘National’ projects refer to transport policy proposals intended to improve the national transport system (e.g., highway projects, pricing, implementing other speed limits on highways). Also for these national spatial scale projects the national government makes the decisions. For a regional transport policy proposal, the regional authorities (often combined with the national decision-making level for additional funding) are the main decision-makers.

A third categorical variable which might influence decision-making is ‘CPB involvement’. CPB stands for the Netherlands Bureau for Economic Policy Analysis, an independent economic research institute which both carries out CBAs themselves or provides second opinions on CBAs made by others. The CPB is regarded in the Netherlands as an institution delivering high-quality CBAs or as ‘the big stick’ encouraging other CBA authors to deliver a high quality CBA, knowing that the CPB will carry out a second opinion. Moreover, the CPB results often get a lot of media attention. Thus, a priori, our hypothesis is that ‘CPB involvement’ may influence the quality of a CBA and the CBA’s impact on decision making. Therefore, we included this category in our models. In around 45 % of the project variants the CPB was involved.
Table 3

Overview of CBA variables used

 

N

Mean

Median

Standard deviation

Billion Euro

   Investment costs (PV)

454

1.1

0.4

1.7

   Transport benefits (PV)

440

1.2

0.3

3.2

   Indirect effects (PV)

215

−0.2

0.1

2.3

   External effects (PV)

286

0.3

0.0

2.1

   Net present value (NPV)

454

−0.1

0.0

3.0

   Relative NPV

454

0.6

0.3

4.9

Ratio

    

   Inverse of number of variants

454

0.27

0.20

0.25

Table 3 presents the continuous explanatory variables used in our models. These variables capture the main results of the CBA reports. Five variables can be found in the CBA reports which may influence decision-making: the Net Present Value (NPV) and the Present Values (PVs) of the initial investment costs, transport benefits (mainly travel time gains), indirect effects and external effects (mainly environmental impacts). In the Netherlands the indirect effects relate to the impact of a project on markets other than the transport market. Additionally, we constructed three extra variables which might affect decision-making: ‘relative NPV’ signifies the difference between the NPV of a project variant and the NPV of the project variant with the highest NPV. Our assumption was that decision-makers were more likely to choose the variant with the highest relative NPV. The ‘Inverse of number of variants’ variable represents the amount of variants scrutinized per project. A priori, our assumption was that a variant has a lower chance of being chosen if it is part of a relatively large collection of variants. To be clear, the lower the inverse number for a project is, the more variants there are. Finally, we used the variable ‘year’ signifying the ‘age’ of the CBA report. The assumption is that decision-making in the early days of the CBA practice (in the early 2000s) might differ from the more recent decision-making practice.

We did not evaluate the quality of the CBAs. Nor were any corrections on the data carried out. The reason is that we aim to relate the CBA outcomes as presented to the decision-maker to their decisions, regardless of whether this information was correct or not.

Some original CBA data was processed. In about 75 % of the 106 CBA reports point estimates of the present values for the variables distinguished were presented in the summary and conclusions. In the other 25 % the results were presented in bandwidths, for example, using a low, middle and high economic growth scenario. In this way the CBA authors were able to indicate to the decision-makers the influence of future uncertainty on the PV outcomes. The reports which only presented point estimates sometimes included a more subtle picture by addressing future uncertainty somewhere in the report (e.g., qualitatively or in an appendix). For our analysis we used the main outcomes as presented in the summary and conclusion as input, on the assumption that decision-makers also focused on the main results presented in the CBA reports. In our models point estimates are only used for the five variables (Table 3). The data presented in a bandwidth was therefore changed into point estimates using the middle scenario estimates, or the average of low and high if only those two estimates were presented, or with only the low or high estimate if only the low or high estimate was presented in the summary or conclusion.

As Table 3 shows, transport benefits, indirect and external effects could not be found in the CBA reports of all 454 cases. Sometimes results were presented on an aggregated level which meant that we (and, of course, also the decision-maker using these CBAs) could not clearly distinguish between these types of effects. In addition, indirect and external effects were not always included or were only included in a qualitative way.

After exclusion of the pricing projects, missing values only occurred on the variables indirect effects and external effects. Inclusion of these missing values would result in a sample size of only 125 (i.e. after listwise deletion). To still be able to take these variables into account in the analysis, we replaced missing values by zero’s. We note that the impact of CBA on decision makers is determined by their impression of CBA results, which is not necessarily the same as the CBA results themselves. We assumed that missing values for indirect and/or external effects give decision makers the impression that these effects are zero or negligible. Below, we return to this matter.

Document analysis was used to determine the final decision of each project. Three categories of decisions were distinguished: ‘go’, ‘no go’ or ‘pending/unknown’. Official government documents and newspaper items were used for each project variant to determine whether a final go or no go decision was taken, or whether the decision was still pending at the time of our research (in 2013).4

Multinomial logit models To assess the influence of the independent variables (Tables 2, 3) on the final decision (‘go’/‘no go’/‘pending’) several multinomial logit models were estimated. In the computation of standard errors it is normally assumed that each observation is independent of all other observations in the data set. However, in our particular case, the project variants of a certain project are likely to be correlated. To account for these intra-class correlations clustered robust standard errors were computed (Rogers 1994) using the projects as the clustering variable.

The latent class cluster model In addition to running standard MNL regressions, we were also interested in examining our dataset in a more explorative way. In particular, we were interested in identifying clusters of projects that were similar with respect to the various characteristics (Tables 2, 3) and see what relation these clusters had to the final decision (‘go’/‘no go’/‘pending’). Our aim with this analysis was to shed light on the question as to what types of projects exist (i.e. with specific configurations with respect to the various characteristics) and which project-types are most/least likely to be implemented. To this end, a latent class cluster model was specified and estimated.5 To account for the correlations between the observations the projects (and not the project variants) were identified as the primary sampling units.

Latent class analysis (LCA) is a model-based (probabilistic) clustering technique (Magidson and Vermunt 2004; McCutcheon 1987). As such, it has several benefits over traditional clustering techniques like K-means cluster analysis, namely that (1) misclassification biases are reduced, (2) statistical criteria can be used to judge the optimal number of classes, (3) the significance of the model parameters can be computed and assessed and (4) variables of the mixed-scale type can be accommodated (Magidson and Vermunt 2002). This latter benefit is especially relevant to our application given that the project characteristics (Tables 2, 3) represent a mix of nominal and continuous variables.

In LCA the goal is to find the most parsimonious model, i.e. with the smallest number of latent classes which adequately describe the associations between the indicators. To identify the optimal model, subsequent models were estimated with 1 through 10 latent classes. The Log-Likelihood did not improve greatly after the 5-class solution. Additionally, beyond the 5-class models the clusters also started to overlap in a qualitative sense. Therefore, the choice was made to present the 5-class model in the results (see below).

5 Results

The relationship between NPV and the decision To gain an initial sense of whether and to what extent there is a relationship between the NPV and the final project decision, we dichotomized the NPV into two categories (positive and negative) and cross-tabulated the distribution with the decision (‘go’ and ‘no go’). The results are presented in Table 4 (for 65 variants the decision-making was still pending or unknown at the time of analysis and are excluded. Also the pricing policies are excluded). We can see that in the Dutch practice non-pricing project variants with a negative NPV are associated with a higher percentage (83 %) no go decisions compared to non-pricing variants with a positive NPV (70 % no go decision). A chi-square test indicates that the bivariate association between the two variable is significant (\(\chi ^{2} = 7.89,\) p < 0.05). However, this effect should be controlled for the influences of other (possible confounding) variables, which is done in the next section.
Table 4

The relation between NPV and ‘go’ and ‘no go’ decision (excluding decisions pending and unknown and pricing projects)

 

Decision ‘no go’ (%)

Decision ‘go’ (%)

Total

NPV negative

83

17

216

NPV positive

70

30

98

Total

79

21

314

Results of the multinominal logit model We tested several specifications of the multinomial logit model (pricing policies excluded) with CBA information as explanatory variables. Model specification C will be explained below in ‘missing values analysis’. Models with interaction effects were also tested, but none of these were found to be significant. Table 5 presents the main results. Specification A shows that a positive Net Present Value (NPV) for a project variant seems to increase the probability of ‘keeping the variant alive’ (Pending vs. No go), but not to be decisive in the final project variant choice (Go vs. No go). ‘Keeping a project variant alive’ may also be caused by specific costs and benefits, as shown in specification B. In particular, low investment costs and high transport benefits appear to keep a project Pending rather than leading to a No go decision. An alternative hypothesis is that the decision is not influenced by the NPV of the specific variant, but by the highest NPV of all the variants. However, specification D shows that the maximum NPV of all the variants and the difference between a variant’s specific NPV and the maximum NPV are not significant.

The probability of a Go decision for a specific variant is higher if the inverse of the number of project variants is higher. This signifies that project variants in projects with a relatively low number of variants (meaning a relatively high inverse number) have a higher chance that their decision is Go compared to project variants in projects with a relatively large number of project variant alternatives. The negative coefficient for the variable ‘Year’ in Pending versus No go (top half of Table 5) indicates that the later the CBA was carried out, the higher the chance that the decision is still Pending.

Table 5 also shows that rail projects have a significantly higher probability of getting a Pending decision compared to No go, but also a lower probability of getting a Go (see the switch in signs for Rail in Table 5). Apparently it was difficult for Dutch decision-makers to be decisive with this project category. Possibly, this reflects the larger technological complexity and associated risks with rail projects compared to road projects (Flyvbjerg et al. 2002). There are no significant differences between international, national and regional projects. Also, the involvement of the CPB has no significant effect.
Table 5

Cluster-robust multinomial logit model

Specification

A

B

C

D

 

\(\beta \)

SE\(^{\mathrm{a}}\)

\(\beta \)

SE\(^{\mathrm{a}}\)

\(\beta \)

SE\(^{\mathrm{a}}\)

\(\beta \)

SE\(^{\mathrm{a}}\)

‘Pending’ decision (compared to ‘No go’)

   NPV

0.449

(0.207)*

      

   Relative NPV

      

0.297

(0.212)

   Investment costs

  

\(-\)0.628

(0.296)*

\(-\)0.662

(0.253)*

  

   Transport benefits

  

0.592

(0.267)*

0.528

(0.264)*

  

   Indirect benefits

  

\(-\)0.538

(1.356)

    

   External benefits

  

1.125

(3.423)

    

   Inverse no. of variants

\(-\)0.366

(1.392)

\(-\)0.285

(1.445)

\(-\)0.344

(1.389)

\(-\)0.457

(1.386)

   Year

\(-\)0.311

(0.106)*

\(-\)0.305

(0.109)*

\(-\)0.302

(0.108)

\(-\)0.296

(0.113)*

Type of project (reference: other transport projects)

   Road

\(-\)0.519

(0.751)

\(-\)0.419

(0.781)

\(-\)0.400

(0.760)

\(-\)0.636

(0.747)

   Rail

1.954

(0.834)*

2.082

(0.846)*

2.151

(0.835)*

1.294

(0.745)

Spatial scale (reference: international)

   Regional

0.222

(1.084)

0.284

(1.033)

0.357

(1.040)

\(-\)0.070

(1.007)

   National

\(-\)0.710

(0.985)

\(-\)0.578

(1.147)

\(-\)0.559

(1.129)

\(-\)0.656

(0.961)

   CPB involvement

\(-\)0.473

(0.913)

\(-\)0.324

(0.913)

\(-\)0.298

(0.909)

\(-\)0.422

(0.887)

   Constant

0.561

(1.234)

0.373

(1.287)

0.311

(1.222)

0.914

(1.169)

‘Go’ decision (compared to ‘Nogo’)

   NPV

0.073

(0.126)

      

   Relative NPV

      

\(-\)0.015

(0.202)

   Investment costs

  

\(-\)0.189

(0.142)

\(-\)0.209

(0.137)

  

   Direct benefits

  

0.016

(0.138)

\(-\)0.008

(0.103)

  

   Indirect benefits

  

\(-\)0.348

(0.622)

    

   External benefits

  

0.230

(1.186)

    

   Inverse no. of variants

3.095

(0.560)*

3.262

(0.599)*

3.222

(0.586)*

3.121

(0.588)*

   Year

\(-\)0.067

(0.040)

\(-\)0.055

(0.041)

\(-\)0.056

(0.037)

\(-\)0.070

(0.040)

Type of project (reference: other transport projects)

   Road

\(-\)0.280

(0.294)

\(-\)0.205

(0.297)

\(-\)0.183

(0.295)

\(-\)0.301

(0.287)

   Rail

\(-\)1.118

(0.542)*

\(-\)1.067

(0.509)*

\(-\)1.04

(0.511)*

\(-\)1.274

(0.493)

Spatial scale (reference: international)

   Regional

0.240

(0.423)

0.383

(0.447)

0.0426

(0.441)

0.144

(0.441)

   National

\(-\)0.626

(0.485)

\(-\)0.276

(0.533)

\(-\)0.255

(0.533)

\(-\)0.698

(0.521)

   CPB involvement

0.192

(0.332)

0.488

(0.378)

0.517

(0.369)

0.171

(0.335)

   Constant

\(-\)1.646

(0.493)

\(-\)1.912

(0.540)

\(-\)1.944

(0.531)

\(-\)1.547

(0.503)

   Pseudo R\(^{2}\)

0.187

 

0.203

  

0.201

0.172

 

   Observations

404

 

404

  

404

404

 

* Significant at 5 %

\(^\mathrm{a}\) Cluster-robust standard errors between brackets

Results of the Latent Cluster Analysis (LCA) The parameter estimates related to the indicators were all significant at the 5 % level. Based on the parameter estimates the profile of each class can be calculated. These profiles are shown in Table 6. In the present case, the profiles contain three kinds of parameters. Firstly, the unconditional class membership probabilities, indicating the probabilities that a randomly drawn project belongs to one of the five classes. These probabilities can be interpreted as the size of each cluster. Secondly, the (conditional) response probabilities are shown. These are estimated for the four nominal indicators (decision, project category, CPB involvement and spatial scale) and indicate the probability of a particular response given that a project belongs to a particular class. For example, a project belonging to the first class has a probability of 51 % of having a ‘no go’ decision. And thirdly, the (conditional) response means, which are estimated for the eight continuous variables (investment, transport benefits, indirect effects, external effects, NPV, relative NPV, the inverse number of variants and year). These indicate the expected mean values given that a project belongs to a particular class.
Table 6

Latent class profiles

 

Cluster 1

Cluster 2

Cluster 3

Cluster 4

Cluster 5

Cluster size (row total \(=\) 1)

0.37

0.24

0.16

0.15

0.08

Decision (column total \(=\) 1)

     

   No go

0.51

0.68

0.84

0.36

0.90

   Go

0.24

0.24

0.03

0.46

0.00

   Pending

0.25

0.07

0.12

0.18

0.10

CBA information

     

   Investment (mean billion €)

0.1

0.5

3.3

1.4

2.7

   Transport benefits (mean billion €)

0.1

0.3

2.5

1.2

2.0

   Indirect effects (mean billion €)

0.0

0.0

0.5

0.0

0.0

   External effects (mean billion €)

0.0

\(-\)0.0

\(-\)0.1

\(-\)0.0

0.0

   NPV (mean billion €)

\(-\)0.0

\(-\)0.2

\(-\)0.8

\(-\)0.4

\(-\)0.5

   Relative NPV (mean billion €)

\(-\)0.0

\(-\)0.2

\(-\)1.5

\(-\)0.7

\(-\)2.3

   Inverse number of variants (mean)

0.3

0.2

0.3

0.4

0.1

Project category (total \(=\) 1 over columns)

     

   Rail

0.03

0.44

0.44

0.43

0.00

   Road

0.49

0.29

0.24

0.17

1.00

   Other transport projects

0.48

0.26

0.33

0.40

0.0

CPB involvement (“yes \(+\) no” \(=\) 1 over columns)

     

   Yes, involved

0.05

0.53

0.56

0.38

1.00

   Year (mean)

4.3

6.2

7.5

4.6

6.4

Spatial scale (total \(=\) 1 over columns)

     

   Regional

0.84

0.32

0.04

0.45

0.00

   National

0.11

0.17

0.85

0.42

1.00

   International

0.05

0.52

0.11

0.13

0.00

The explorative latent class analysis indicates the existence of different classes (or subsets) underlying the CBA outcomes and decision-making in the Netherlands. The five clusters are interpreted below.

Cluster 1 is the largest cluster. The cluster represents relatively small project variants (average investment costs of around 100 million Euros; mainly road and ‘other’ projects) with a small mean negative NPV. The project variants in this cluster (together with cluster 4) have a lower chance of a ‘no go’ decision compared with clusters 2, 3 and 5. This cluster contains many regional project variants. It has the lowest CPB involvement of all the clusters and is the ‘youngest’ cluster with an average mean age for drafting the CBAs of 4.3 years ago (compared to 2012).

Cluster 2 is the second largest cluster, and not very clear. The investment costs are (again) relatively low and the mean NPV is negative but compared to clusters 3, 4 and 5 quite modestly negative. The cluster has the highest share of international project variants of all the clusters. In this cluster ‘pending’ is chosen relatively less often compared with the other clusters. The share of the ‘go’ decision is the same as in cluster 1. This cluster has a more or less even mixture of all project variant categories.

Cluster 3 is a relatively small but clear cluster. It represents the very large project variants (mean investment costs of 3.3 billion Euros) with a relatively high negative NPV (a mean of −0.8 billion Euro). In line with the MNL model results these project with very large investment costs have a higher chance of a ‘no go’ than a ‘go’. The involvement of the CPB in the CBAs for these projects is relatively high. The project variants are relatively old and the share of regional projects in this cluster is very low.

Cluster 4 is also relatively small. It represents moderately expensive projects (mean investment costs of 1.4 billion Euros). The mean NPV is negative: −0.4 billion Euros. Interestingly, this cluster has the highest share of ‘go’ compared to all the other clusters. If we compare this cluster with clusters 3 and 5 (also large investments projects albeit both with bigger projects) the main differences are that cluster 4 has a relatively high share of regional projects, relatively low CPB involvement and a relatively low share of road projects.

Finally, cluster 5 is the smallest cluster containing expensive national road project variants (mean investments costs of 2.7 billion Euros). For these variants ‘go’ was never chosen. The CPB was involved in all the CBAs for these variants.

Additionally to the MLN modelling, the cluster analysis shows that there are different types of transport projects that experience a different type of political process. In particular, there are two clear dynamics. First, there are many small, regional projects that have, relatively speaking, a lower chance of being rejected while large, national (rail) projects tend to have a strong negative NPV and, in line with the CBA, tend to be rejected relatively often.

Missing values analysis As stated in the method section, missing values on the variables ‘indirect effects’ and ‘external effects’ were replaced by zero’s. While these variables did not have significant effects on the decision (in the MNL model) and also did not discriminate strongly between the latent classes (in the LC model), we ran an additional MNL model (specification C, Table 5) excluding these variables to assess whether their inclusion did not ‘distort’ the effects of the other variables. This proved not to be the case, as differences between this model (specification C) and the presented model (Table 5, specification B) are minimal. With respect to the latent class model, we retained the variables (‘indirect benefits’ and ‘external benefits’) in their original form, so with missing values defined as missing values. This is because the latent class model can easily handle missing values, as this model is estimated using full information maximum likelihood. Hence,the parameters are estimated using all available information for each of the cases. In essence, missing values are skipped in the ML estimation (see page 51 of Vermunt and Magidson 2005).

6 Concluding remarks

Based on our large-scale study of CBA reports for all transport proposals in the Netherlands during the period 2000–2012, we did not find a significant statistical relation between the welfare assessment of transport projects using a CBA and the variants chosen in political decisions (after controlling for other relevant variables). However, a positive NPV does keep variants ‘pending’, preventing a negative decision at least temporarily. In addition, we found that rail projects faced a higher chance of being rejected.

It is up to the politicians to decide whether and to what extent the efficiency criterion is important. Thus, the question of whether a stronger relation between aggregate CBA results and decision-making is required cannot be answered scientifically. However, it is interesting to note that politicians decided in the year 2000 to make the CBA tool mandatory but, subsequently, no significant relation between aggregate CBA results and decisions can be found (at least for the CBAs carried out in the period 2000–2012). Our conclusion for the case of the Netherlands fits in—broadly speaking—with the international literature (see Sect. 2). A recommended next step would be to research the question ‘why’? Annema (2013, see section 2) gives some speculative reasons but to really understand this limited CBA use in-depth research is required. Is it due to the quality of the CBAs? Is it their ‘black box’ character? Are there political reasons? Is there information overload? Is it due to a combination of reasons? The answers to these questions might improve (if desired) the position of CBA in decision-making for transportation projects.

Finally, we also noted that decision-making for smaller regional projects seems to have other characteristics compared to national and international transport projects. Thus, in further research, it may be wise to give special attention to the position of CBAs in decision-making for these smaller (often regional) transportation projects.

Footnotes

  1. 1.

    For urbanized regional regions around Amsterdam, The Hague and Rotterdam the cut-off is 225 million Euros.

  2. 2.

    In 2013 the Dutch government published a letter (with a new general CBA guideline as appendix) in which they announced that CBA will also be used in decision-making outside the domain of transport (Ministry of Finance 2013).

  3. 3.

    Thus, qualitative information, if included, was ignored. Some cost or benefit items are hard to monetize and were given labels such as PM (‘Pro Memorie’ is Latin for ‘keep in mind’) or plusses or minuses in the main CBA table. In order to carry out a quantitative analysis we did not include this qualitative information in our database.

  4. 4.

    It should be noted that it was not always clear which project variant was finally chosen. For example, one project proposal related to extending the highway capacity around Schiphol Amsterdam Airport and the city of Amsterdam. 16 different highway project variants were evaluated using CBA. One main variant related to building new highway stretches in 8 different sub-variants. Another main variant related to only extending existing highways with extra lanes, also with 8 sub-variants. In the final official policy document it became clear which main variant was chosen but to identify with 100 % certainty which sub-variant (one of the 8) was actually chosen was not possible, not even after calling the people who were actually involved in the decision. Thus, some errors on sub-variant level may have been made in our analysis. Also, categorizing decisions as ‘pending’ or ‘no go’ is less clear-cut than might be expected. As a rule we took the final decision document or newspaper clipping as the starting point. If it was made clear on paper that the final decision was not clear yet, the project decision was categorized as ‘pending’. If it was made clear that the political body which decides (e.g., the city of Groningen municipality council voted against building a city tram for which a CBA had been carried out) said ‘no go’, the decision was categorized as ‘no go’. Even if the newspaper carrying this news (for example) said that it was imaginable that after the election the new council (with a different political composition perhaps) might still decide to implement the project.

  5. 5.

    Using the dedicated software package Latent Gold 5 (Vermunt and Magidson 2005).

Notes

Acknowledgments

We thank Erik de Best for work on the database and dr. Sytze Rienstra for sharing his large collection of Dutch CBAs with us. We would also like to thank the two anonymous reviewers for their comments that have helped to improve the paper.

References

  1. Annema, J.A.: The use of CBA in decision-making on mega-projects: empirical evidence. In: Priemus, H., van Wee, B. (eds.) International Handbook on Mega-Projects. Edward Elgar, Cheltenham/Northampton (2013)Google Scholar
  2. Annema, J.A., Koopmans, C., Van Wee, B.: Evaluating transport infrastructure investments: the Dutch experience with a standardised approach. Transp. Rev. 27(2), 125–150 (2007)CrossRefGoogle Scholar
  3. Bax, C.: Processes and patterns: the utilization of knowledge in Dutch road safety policy. Stichting Wetenschappelijk Onderzoek Verkeersveiligheid, Leidschendam (2011)Google Scholar
  4. Del Bo, C., Florio, M.: Cost-benefit analysis and the rates of return of infrastructure projects: evidence from international organizations. Transit. Stud. Rev. 17(3), 587–610 (2010)CrossRefGoogle Scholar
  5. EC: Guide to cost-benefit analysis of investment projects: economic appraisal tool for cohesion policy 2014–2020 (2014). http://ec.europa.eu/regional_policy/sources/docgener/studies/pdf/cba_guide.pdf. Site Accessed Jan 2016
  6. Eijgenraam, C.J.J., Koopmans, C.C., Tang, P.J.G., Verster, A.C.P.: Evaluation of Infrastructural Projects; Guide for Cost-Benefit Analysis. CPB Netherlands Bureau for Economic Policy Analysis, The Hague (2000)Google Scholar
  7. Eliasson, J., Lundberg, M.: Do cost-benefit analyses influence transport investment decisions? Experiences from the Swedish transport investment plan 2010–21. Transp. Rev. 32(1), 29–48 (2012)CrossRefGoogle Scholar
  8. Flyvbjerg, B., Holm, M.K.S., Buhl, S.L.: Underestimating costs in public works projects: error or lie? J. Am. Plan. Assoc. 68(3), 279–295 (2002)CrossRefGoogle Scholar
  9. Fridstrom, L., Elvik, R.: The barely revealed preference behind road investment priorities. Public Choice 92(1/2), 45–168 (1997)CrossRefGoogle Scholar
  10. Hayashi, Y., Morisugi, H.: International comparison of background concept and methodology of transportation project appraisal. Transp. Policy 7, 73–88 (2000)CrossRefGoogle Scholar
  11. Homan, A.C.: Role of BCA in TIGER grant reviews: common errors and influence on the selection process. J. Benefit Cost Anal. 5(1), 111–135 (2014)CrossRefGoogle Scholar
  12. Homan, A.C., Adams, T.M., Marach, A.J.: A statistical analysis of the role of benefit-cost analysis in awarding TIGER grants. Public Works Manag. Policy 19(1), 37–50 (2014)CrossRefGoogle Scholar
  13. Kasemir, B., Jäger, J., Jaeger, C.C., Gardner, M.T.: Public Participation in Sustainability Science. A Handbook. Cambridge University Press, Cambridge (2003)CrossRefGoogle Scholar
  14. Landry, R., Amara, N., Lamari, M.: Utilization of social science research knowledge in Canada. Res. Policy 30(2), 333–349 (2001)CrossRefGoogle Scholar
  15. Lee, D.B.: Methods for evaluation of transportation projects in the USA. Transp. Policy 7, 41–50 (2000)CrossRefGoogle Scholar
  16. Mackie, P., Kelly, C.: Transport Appraisal in other countries: lessons for the NATA Refresh—Specification. Report produced for Department for Transport, University of Leeds (2007)Google Scholar
  17. Magidson, J., Vermunt, J.: Latent class models for clustering: a comparison with K-means. Can. J. Mark. Res. 20(1), 36–43 (2002)Google Scholar
  18. Magidson, J., Vermunt, J.K.: Latent class models. In: Kalpan, D. (ed.) The Sage handbook of quantitative methodology for the social sciences, pp. 175–198. Sage Publications, Thousand Oaks (2004)Google Scholar
  19. McCutcheon, A.L.: Latent Class analysis, vol. 64. Sage Publications, Thousand Oaks (1987)CrossRefGoogle Scholar
  20. McFadden, D.: The revealed preferences of a government bureaucracy: theory. Bell J. Econ. 6(2), 401–716 (1975)CrossRefGoogle Scholar
  21. McFadden, D.: The revealed preferences of a government bureaucracy: empirical evidence. Bell J. Econ. 7(1), 55–72 (1976)CrossRefGoogle Scholar
  22. Ministry of Finance: Kabinetsbrief bij de algemene MKBA Leidraad [Government letter related to the general CBA guidelines]. IRF/2013/993, 6th of December 2013, The Hague (2013) (in Dutch)Google Scholar
  23. Ministry of Infrastructure and the Environment: KBA bij MIRT-Verkenningen. Kader voor het invullen van de OEI-formats [CBA in National Spatial and Infrastructure Planning Studies], 1st of September 2012, The Hague (2012)Google Scholar
  24. Mouter, N., Annema, J.A., van Wee, B.: Attitudes towards the role of cost-benefit analysis in the decision-making process for spatial-infrastructure projects: a Dutch case study. Transp. Res. Part A: Policy Pract. 58, 1–14 (2013)Google Scholar
  25. Nakamura, H.: The economic evaluation of transport infrastructure: needs for international comparisons. Transp. Policy 7, 3–6 (2000)CrossRefGoogle Scholar
  26. Nilsson, J-E.: Investment decisions in a public bureaucracy. A case study of the Swedish road planning practices. J. Transp. Econ. Policy 25(2), 163–175 (1991)Google Scholar
  27. Nellthorp, J., Mackie, P.: The UK Roads Review—a hedonic model of decision making. Transp. Policy 7(2), 127–138 (2000)CrossRefGoogle Scholar
  28. Odeck, J.: Ranking of regional road investment in Norway. Transportation 23(2), 123–140 (1996)CrossRefGoogle Scholar
  29. Odeck, J.: What determines decision-makers’ preferences for road investments? Evidence from the Norwegian road sector. Transp. Rev. 30(4), 473–494 (2010)CrossRefGoogle Scholar
  30. Odgaard, T., Kelly, C., Laird, J.: HEATCO work package 3: current practice in project appraisal in Europe, HEATCO (Developing Harmonized European Approaches for Transport Costing and Project Assessment), Deliverable 1/Volume 1. COWI A/S, Denmark, Kongens Lyngby (2005)Google Scholar
  31. Proost, S., Dunkerley, F., De Borger, B., Günhneman, A., Koskenoja, P., Mackie, P., van der Loo, S.: When are subsidies to Trans European Network projects justified? Transp. Res. A Policy Pract. 45(3), 161–170 (2011)CrossRefGoogle Scholar
  32. Rienstra, S.: De rol van kosten-batenanalyse in de besluitvorming [The role of cost-benefit analysis in the decision-making process]. KIM the Netherlands Institute for Transport Policy Analysis, The Hague (2008) (in Dutch)Google Scholar
  33. Rogers, W.: Regression standard errors in clustered samples. Stata Tech. Bull. 3(13), 19–23 (1994)Google Scholar
  34. Romijn, G., Renes, G.: General Guidance for Cost-Benefit Analysis. CPB Netherlands Bureau for Economic Policy Analysis and PBL Netherlands Environmental Assessment Agency, The Hague (2013)Google Scholar
  35. Vermunt, J.K., Magidson, J.: Technical Guide for Latent GOLD 5.0: Basic, Advanced, and Syntax. Statistical Innovations Inc, Belmont (2005)Google Scholar
  36. Welde, M., Eliasson, J., Odeck, J., Börjesson, M.: The Use of Cost-Benefit Analyses in Norway and Sweden: A Comparison. Association for European Transport (2013). https://abstracts.aetransport.org/paper/index/id/198/confid/1. Site Accessed April 2015
  37. Willis, K.G., Garrod, G.D., Harvey, D.R.: A Review of cost-benefit analysis as applied to the evaluation of new road proposals in the U.K. Transp. Res. D 3(3), 141–156 (1998)CrossRefGoogle Scholar
  38. World Bank: Cost-Benefit Analysis in World Bank Projects. The International Bank for Reconstruction and Development/The World Bank, Washington, DC (2010)Google Scholar

Copyright information

© The Author(s) 2016

Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Authors and Affiliations

  • Jan Anne Annema
    • 1
  • Koen Frenken
    • 2
  • Carl Koopmans
    • 3
    • 4
  • Maarten Kroesen
    • 1
  1. 1.Delft University of TechnologyDelftThe Netherlands
  2. 2.Utrecht UniversityUtrechtThe Netherlands
  3. 3.VU UniversityAmsterdamThe Netherlands
  4. 4.SEO Economic ResearchAmsterdamThe Netherlands

Personalised recommendations