1 Introduction

Ever since states have existed, they have sought to count and track their citizens in order to tax and control them [1]. Modern states record their own activities in great detail. Just as the digital traces of individuals contain valuable insights about everything from their health to their socioeconomic status [2], the administrative big data maintained by the state can be used to analyze its successes and failures [3]. One such failure that is surprisingly persistent even among economically advanced and democratic states is corruption, which we frame as the “deliberate restriction of open and fair access to public resources for the benefit of connected actors” [4]. Corruption is indeed a significant failure of governance, as it has been shown to slow growth [5] and innovation [6] and to subvert democracy [7], while compounding inequality [8].

Despite its importance, corruption is difficult to measure because individuals engaging in corruption obviously want to keep it a secret. Ground truth examples of convicted corrupt actors are hard to come by and anyway form a biased sample: large-scale corruption often goes unpunished because the state, including law enforcement and the judiciary, has been captured by corrupt interests [9]. For these and other reasons, corruption is traditionally measured via surveys which suffer from well-documented shortcomings [10, 11].

The significant amount of administrative data collected by governments presents an opportunity to study corruption risk from a new perspective using new tools of data science [12]. In this paper, we mine a big administrative dataset with over 4 million public procurement contracts in the EU for insights on the organization of corruption. Public procurement, the process by which governments purchase goods, services and construction works from the private sector, accounts for up to 20% of GDP [13] and is known to be vulnerable to corruption [14]. We proxy for corruption risk at the contract level with a binary indicator that is 1 if there was no competition for the contract. Such single-bidder contracts have been shown to predict significant risk of corruption [15, 16]. Aggregated to the country level, the rate of single bidding correlates significantly with commonly used survey-based measures of corruption and predicts overpricing on the auction level.

With micro-level data in hand, we apply the tools of network science to map and analyze the distribution and structure of corruption risks in different European countries. We represent markets as bipartite networks of the issuers (public institutions, sometimes referred to as buyers) and winners (firms, sometimes referred to as suppliers) of procurement contracts. These bipartite networks have many qualitative characteristics in common with other empirical networks [17] that distinguish them from random networks. For instance, they have heterogeneous degree distributions and significant local correlations.

These networks are more than just maps of markets. We argue that they provide an ideal framework for studying the organization of corruption. There are many reasons to consider corruption as a fundamentally networked phenomenon. The most significant disclosed examples of corruption, for instance the recent scandal involving the Brazilian state-owned oil company Petrobras [18, 19], involve hundreds of individuals, firms, and institutions. The Petrobras scheme entailed the exchange of billions of dollars in bribes and kickbacks for the award of public contracts. The size of the conspiracy, of which nearly 100 conspirators—including the former president of Brazil—have been convicted, suggests that it involved a sophisticated and collective effort. Likewise, network studies of organized crime [20] and terrorism [21] demonstrate that illegal activity tends to leave complex traces that networks are especially suited to describe and explain.

In recent years, network analyses of big relational datasets have delivered important insights into social phenomena. Networks can highlight important signals of incoming financial crises [22], predict how economies will develop [23], and suggest how social organization can facilitate cooperation [24]. As macro-level phenomena such as inequality tend to emerge from many micro-level interactions [25], network representations are especially useful to analyze macro-outcomes when micro-level data are available.

In this paper, we use a network approach to describe the distribution of corruption risk in different procurement markets from two perspectives: the extent to which it is centralized, that is, how prevalent it is between actors in the core of the market versus its periphery, and how clustered it is, describing the extent to which corruption risk is bunched in different parts of the network. We find that in some EU countries corruption risk is more common in the core of the market, while in others it is more common in its periphery. In every country in our database, corruption risk is clustered, though the magnitude of this effect varies significantly. Our results are compared to realistic null models, which take into account the overall magnitude of corruption risk as well the propensity of governments in different countries to purchase different goods and services. Our findings have significant implications for both the academic study of corruption and practical anti-corruption efforts. Observing that corruption seems to be organized in different ways suggests that different anti-corruption strategies may be effective in different contexts.

The paper is organized as follows. Section 2 surveys related work, including recent work on the measurement of corruption risk via public procurement contracts. Section 3 describes the data and corruption risk indicator. Section 4 introduces the network measures of the distribution of corruption risk in markets. Section 5 concludes and suggests ideas for future research.

2 Related work

Big data has been applied to a variety of social and economic problems in domains including health [26], scientific research [27], urban mobility [28, 29], and development [30]. The digitization of our social and professional lives has facilitated most of this work, providing researchers access to high-resolution trace data about human behavior and activities. Often the data analyzed are “found data”—data which are not originally collected for the purpose of the research. Data collected from public administrative databases, for example public procurement, fall into this category [31].

The study of corruption using big administrative data is a new and evolving area of research with its own challenges. It represents a major departure from previous empirical studies of corruption which leverage perception-based surveys for macro-studies and experiments at the micro-level. Below, we briefly survey these approaches to the research of corruption.

The most prominent and longest running survey-based measures of corruption are Transparency International’s  Corruption Perceptions Index (CPI) [32] and the World Bank’s  Worldwide Governance Indicators (WGI) [33], both available since the 1990s. Both measures, quantifying perceptions of corruption and quality of government at the national level, are composite indicators, mixing general population and expert surveys. The measures are highly correlated (\(\rho > \) .9) and are designed to be consistent over time.

The significant complexity of weighing components and correlating observations year to year has lead some researchers to criticize the validity and broad application of these measures [10, 34]. More recent attempts to resolve some of these issues or to create more consistent measures include the  Varieties of Democracy (V-DEM) indicator of political corruption [35] and the Quality of Government Institute’s (QoG)  European Quality of Government Index (EQI) [36].

Experimental studies of corruption have the advantage of allowing researchers to directly measure causes or levels of corruption in specific contexts by precise specification. For example, to study the influence of culture on corruption Cameron et al. [37] observed individuals from different cultures playing games with both an economic incentive to cheat and a mechanism to punish cheaters. Interestingly, they found greater variation in the propensity to punish corruption than to engage in it across cultures. Weisel and Shalvi [38] observe pairs of individuals playing a dice-rolling game in which pairs can increase their payout by cheating as a team, and find that pairs tend to cheat more often than individuals playing a similar game alone.

These studies have the clear limitation that they happen in artificial environments. Some experiments are carried out in the field, though coming with significant higher costs and risks. Perhaps the most famous of these relating to corruption are Olken’s field experiments in Indonesia [11, 39]. Olken designed a series of interventions to test the effects of both audits and grassroots organization on corruption outcomes in 600 Indonesian villages during a national road construction project. Crucially, Olken had the resources to independently assess the expected cost of the roads in order to estimate actual observed corruption in each village. He found that pre-announced audits reduce corruption, but that community organizing induced no change. Such studies are a gold-standard in causal inference, but suffer from significant costs and limitations in scope.

Some observational studies study corruption using conviction data. Most of these studies focus on the USA, exploiting the federal structure of government and making the assumption that independent federal corruption prosecutions give an accurate measurement of corruption at the state level [40]. The correlation between state-level corruption perception and federal conviction data is low (roughly 0.3) [41], suggesting that these two measures of corruption are quantifying different things. Conviction data can be biased by selective enforcement or geographic factors—for example if more competent enforcement is clustered in certain regions. The value of conviction data as a proxy for corruption is even more limited when legal interventions happen at the same level as the corrupt behavior. For example, in a country with high levels of corruption, the courts themselves may be corrupt—a fact that would skew conviction rates.

Observing the shortcomings of other approaches, researchers have increasingly turned to big data methods to quantify and study corruption in the public sector. These studies tend to leverage found data collected and curated by governments for other purposes. They rely heavily on new trends for open government and e-government [42]. As citizens increasingly demand transparency and public-sector use of information and communication technologies proliferates, it is reasonable to expect that more data will become available in the future [43]. Data in the USA on lobbying [44] and campaign contributions [45, 46] have been used to measure the influence of money in politics. Internationally, large-scale firm ownership data reveal how firms avoid taxes [47]. Crowdsourced data on convicted public officials, for instance extracted from Wikipedia, have been used to study the emergence of systemic corruption [19]. Big data has also been leveraged to study topics related to corruption like nepotism [48].

As mentioned in Introduction, our study leverages big data on public procurement. The scale and scope of procurement and its status as a key interface between the public and private sectors have made it a popular topic of research. How do researchers quantify corruption using public procurement data? Fieldwork suggests that corrupt officials steer contracts to favored firms by restricting competition, insuring high profits [49]. In general, traces of the strategies used to restrict competition can be extracted from administrative data on the contracting process, enabling researchers to score individual contracts for corruption risk. The most general indicator tracks the competitive outcome directly: Whether the contract attracted a single bidder. Single-bid contracts represent a clear corruption risk.

Single-bid contracting rates have been used as an effective proxy for corruption in various contexts, including studies on the relationship between corruption and political incumbency [15], the importance of meritocracy in bureaucratic outcomes [16], the impact of social networks on local corruption [50, 51], and the effect of campaign contributions on corruption [52]. Procurement-based corruption risk indicators have the advantage that they apply to micro-level transactions, enabling the study of corruption at multiple scales. Several previous studies study the distribution of corruption risk in procurement markets represented as networks. One such study finds that repeated interactions are significantly related to corruption risk [53], while another demonstrates how procurement markets change when corruption risk becomes endemic [14].

Despite the significant interest in corruption as a problem and the proliferation of public procurement-based big data indicators, we are not aware of research that explores the distributional and structural properties of corruption in different countries based on the patterns of interactions between public and private actors. This gap in the literature is surprising given that social scientists have been categorizing countries by the organization of corruption in their public sectors for decades [54, 55]. We fill this gap by leveraging the tools of network science, providing an analytic framework to both study the distribution of corruption risks in countries and to provide tailor suggestions on how to combat it.

3 Data and framework

3.1 Procurement data

Our analytic framework measures corruption risk at a transaction level using data from public procurement contracts. We collected data on all public procurement contracts published in Tenders Electronic DailyFootnote 1 (TED), the official journal of public procurement contracts of the European Union, from 2008 to 2016. Both calls for tenders and the announcement of their award are published in TED. EU law requires that any public procurement contract exceeding an estimated value of 5.2 million Euro for works and 135 thousand Euro for services and supplies must be published on TED. TED estimates that over 400 billion Euro of total contract value is published in its pages every year, accounting for nearly 3% of EU GDP. Though the high thresholds exclude a significant number of contracts from our data, using only data from TED maximizes the international comparability of our analyses.

Our final dataset consists of 4,098,771 contracts awarded in 2008–2016. We consider 26 member states of the EU, excluding Luxembourg because of the relatively small number of contracts awarded there and Croatia because it only joined the EU in 2013. We also exclude contracts awarded by EU institutions (the Commission, Parliament, Council) as we aim to compare countries. The data are available for download at https://zenodo.org/record/3537986#.XcmMPkX0mgk.

We processed the dataset to deduplicate the identities of each contract’s issuing buyer (i.e., public institution such as a ministry or city hall) and supplying winner (i.e., a private firm). Given that our analytic approach requires an accurate map of the interactions between issuers and winners, we developed a pipeline to maximize the accuracy of our deduplication, following the approach of Christen [56]. For each country, we preprocessed the text data for each entity, used machine learning to select both optimal string similarity measures and blocking methods, and selected a clustering threshold maximizing accuracy on a manually labeled subsample using the Dedupe computer software [57]. To give an example, the number of unique suppliers of contracts in France decreased from 364,125 to 200,584. For a more detailed summary of the procedure, see [58].

Besides the issuing buyer and winning supplier of each contract, we were able to extract several additional pieces of information, including the year the contract was awarded, the Common Procurement Vocabulary (CPV) code—a EU-wide taxonomy of procurement contracting goods and services [59], and the number of bidders competing for the contract. We use the latter information to score each contract for corruption risk.

3.2 Corruption risk

We quantify the corruption risk of procurement contracts using a binary indicator tracking whether the contract attracted only a single bid in its competition. The use of single bidding as a signal for corruption risk or “undetected fraud” has recently been suggested by the European Court of Auditors [60]. Nevertheless, it is certainly not the case that any single instance of single bidding can be used as evidence for corrupt behavior. For instance, it may be the case that the government is purchasing a niche good or service with few suppliers on the market or under emergency circumstances—certainly it is more likely for such contracts to be awarded to a single bidder.

Two aspects of our data mitigate these limitations to the validity of single bidding as a signal of corruption risk. The first is that the high threshold of contract values insures both visibility and interest from the private sector. Secondly, the null models we create to benchmark our measures consider the CPV code of the contract, distinguishing between contracts for different categories of goods and services, such as medicine and furniture. In the case when data on the number of bidders are missing from a contract, we impute the country-level average single-bidding rate. This imputation strategy likely leads to an underestimate of corruption risk.

Fig. 1
figure 1

Single-bidding rates on procurement contracts on TED, 2008–2016 by country

Full size image

We plot the single-bidding rates with of each country over the 2008–2016 period in Fig. 1. We see significant variation, with single-bidding rates below 10% in some countries and over 30% in others. How do these measures of corruption risk correlate with the survey-based perception indicators mentioned in the previous section? In Fig. 2, we correlate the average single-bidding rates from 2008 to 2016 with Transparency International’s Corruption Perception Index (TI CPI) [32], the World Bank’s Control of Corruption Index (WB CoC) [33], Varieties of Democracy’s Corruption Index (V-Dem Corruption Index) [35], and the Quality of Government (QoG) Institute’s European Quality of Governance Index (European Qual. Gov. Index) [36] each measured in 2013 (this was the most recent year for which all five indicators were available). In each case, we find a significant correlation between country-level single-bidding rates and worse corruption or quality of government outcomes (Pearson correlation between .66 and .72, Spearman correlation between .74 and .76 in magnitude, all correlations significant at \(p< 0.05\)). We also find a significant correlation ( around − 0.7) between single-bidding rates and the recently developed Index of Public Integrity [61], which quantifies the extent to which the institutional framework of a country fosters integrity and blocks corruption.

Fig. 2
figure 2

The 2008–2016 single-bidding rate of each country correlated with various perception-based measures of corruption, all measured in 2013. When correlations are negative, the perception-based measure refers to positive outcomes, for instance quality of government or control of corruption. We report both Pearson and Spearman correlations

Full size image

Our data offer several advantages over the survey-based indicators. Surveys encode subjective perceptions about corruption that may be biased. For instance, past research by Olken found that perceived corruption exceeds actual corruption when ethnic diversity is high [11]. This is a significant obstacle when we would like to compare the rate of corruption across countries. Moreover, the way such indicators are calculated is often modified from year to year, again making comparison difficult. From our perspective, however, the most significant advantage of the single-bidder measure of corruption risk is that it is observed at a granular level; surveys are expensive and, with few exceptions [36], are not carried at the sub-national, e.g., regional level. We exploit that our indicator is available at the transaction level in the following section by introducing our network science-based analytic framework.

4 Network analysis

Public procurement markets can be described as bipartite networks because of the clear division of their participants into issuers and winners. Bipartite networks have been applied to the study of systems including two kinds of actors including flowers and their pollinators [62], cities and industries present in them [63], and buyers and sellers in markets [64]. We apply the same paradigm to our data on procurement markets: For each country and year in our dataset, we create bipartite networks in which issuers and winners are connected by a weighted edge, with weight counting the number of contracts between the two entities.

The standard analysis of a complex network is based on null models, which are randomized versions of the empirical system. The statistical observations made on the empirical network are compared to those obtained on the randomized versions. Significant differences can be interpreted as results of correlations, which are eliminated by the randomization process.

In Table 1, we report summary of the raw statistics of each country’s procurement market network, averaged over 2008–2016. In all cases, the average weighted degrees of nodes (counting the number of contracts the node is involved in) are smaller than their standard deviations, indicating heterogeneous degree distributions.

Table 1 Summary statistics of procurement market networks, averaged over 2008–2016
Full size table

We inspect the degree heterogeneity of each country graphically in Fig. 3. Plotted on a log–log scale, the degree distributions of both issuers and winners are heterogeneous in all countries: Some rare issuers and winners are involved in hundreds, even thousands of contracts across the time span of our data, while the majority are involved in only a few contracts. This heterogeneity cannot be considered as a signature of any corruption; it is more related, e.g., to the broad size distribution of firms and institutions [66].

Fig. 3
figure 3

The distribution of the number of contracts awarded and won, for issuers (in gray) and winners (in white), respectively, of procurement contracts in each country, aggregated from 2008 to 2016. The distributions are plotted on a log–log scale. We report the alpha parameter of a power-law degree distribution fitted to both distributions in the plot [67]

Full size image

We also report the Robins–Alexander clustering of each network in Table 1. Robins–Alexander clustering is defined as the number of cycles of length four in the network divided by the number of paths of length three [65]. Given that a firm wins contracts from two issuers, and that another firm wins a contract from one of these two issuers, Robins–Alexander clustering can be interpreted as the probability that this second firm also wins a contract from the other issuer. This measures the tendency for local clustering in the market analogous to the clustering coefficient in monopartite networks. The expected Robins–Alexander clustering of random bipartite networks tends to their density as they get large [65]. As seen in Table 1, the observed clustering is typically an order of magnitude greater than the density of the observed networks. This indicates the presence of significant local correlations in the markets.

These descriptive statistics indicate that our networks have rich structure which deviates significantly from random behavior. In the next subsections, we introduce two measures of the structure of the market: its centralization and the extent too which it is clustered. We then exploit these measures to describe the distribution of corruption risk in each market.

4.1 Core-periphery analysis

A common task of network analysis is to highlight the most central and active nodes. Some empirical networks have a group of densely connected actors at their centers, sometimes called cores [68]. Besides highlighting important actors, methods to detect network cores can be used to compare the degree of centralization in a network. In this subsection, we adapt a method known as k-shell decomposition to weighted bipartite networks in order to rank issuers and winners by their centrality in the network. We say that the most central actors form the core of the market. We measure and compare the relative sizes of the cores of each procurement market, interpreting a larger core as a signal of procurement market centralization. We find a strong correlation between market centralization and corruption risk.

In generic unweighted graphs, the concept of a coreness is defined iteratively [69]. To start, all nodes of degree 1 are assigned a core number of 1 and then removed from the graph. In the next step, all remaining nodes with degree 1 in the trimmed graph are also assigned core number 1 and then removed. This process repeats until all nodes have at least degree 2. Nodes with degree 2 are assigned core number 2, and they are the first nodes to be removed in the next iteration of removals. This process yields a hierarchical decomposition of the nodes in the network by their core number. The k-core of the graph refers to the subgraph consisting of nodes with core number at least k [70].

Our networks have two features that distinguish them from the graphs considered in this example. The first is that the edges include weights, encoding the volume of contracts between the focal issuer and winner. The second aspect is that the different node sets have different degree distributions. We tweak the concept of the core number and k-core in order to apply it to our networks.

Our method modifies the iterative procedure described above to consider the weighted degrees, defined as the count of contracts they are involved in as either issuers or winners, respectively, of nodes. Otherwise the method is identical. As the edge weights in our networks are integers (describing contracting volume), no further modification is necessary. This is a simplified version of the weighted k-shell decomposition of Garas et al. [71].

Fig. 4
figure 4

The weighted core of the Hungarian market from 2008 to 2016. Gray nodes denote issuers of contracts, while white nodes denote winners. Nodes are included in the core if they engage in many contracts with other highly active nodes, defined iteratively. Edges are colored red if the rate of single bidding on contracts between the issuer and winner exceeds 50%). Note that the core index of each node considers edge weights encoding the count of contracts between issuers and winners

Full size image

This procedure assigns each node in our network a weighted core number, measuring its centrality in the network. As we would like to partition the network into a core and periphery, we need to determine a cutoff for a node to be considered a member of the core. This cutoff should be a function of the size of the network as it would not be appropriate to use the same cutoff for networks of vastly different sizes. We also address the concern that issuers and winners have different degree distributions by setting two different cutoffs for core membership: one for issuers and one for winners. In the end, we choose to consider issuers as core issuers if their weighted core number exceeds the average degree of issuers. As winners tend to have lower average contract win counts, we only consider those winning twice the average degree as core winners.

We visualize the Hungarian core in Fig. 4, marking edges with above-market average single-bidding rates in red. The visualization highlights the importance of considering the volume of contracting between issuers and winners in our measure. Issuers and winners can have only a single neighbor and still be considered for core membership because they may have a high volume of contracts with that neighbor.

We observe that the core has a consistent size over time and remains densely connected in all years. We confirm that in general the weighted core subgraph has a much higher density in a summary statistics table included in Appendix. In general, between 20 and 60% of all contracts awarded in a given country are between core issuers and winners.

We now pose two questions about the relationship between network centralization and corruption risk measured by single bidding. First: What is the relationship between the degree of centralization of a market and its corruption risk. In Fig. 5, we plot the relationship between centralization, quantified by the share of all contracts between core issuers and winners, against corruption, measured by the overall single-bidding rate and the survey-based EQI measure. We find a significant positive relationship between procurement market centralization and corruption. In the political economy literature, there is an ongoing debate about the relationship between the centralization of government power and corruption. Our finding supports the side of the debate claiming that centralization facilitates corruption [72].

Fig. 5
figure 5

Comparing the centralization of the procurement markets, measured as the share of all contracts which are between core issuers and winners, and corruption risk quantified by single bidding and the European Quality of Government Index (EQI), respectively. All measures are averaged over the years 2008–2016. We report Pearson and Spearman correlations, both significant at \(p<.01\) and errors indicate 95th percentile bootstrapped confidence intervals. Note Greece and Norway are excluded from the EQI data

Full size image

We are also interested in the distributional nature of corruption risk. Does centralization induce corruption by fostering corruption among core issuers and winners? We probe this question by comparing the rate of single bidding in the core against a null model that randomizes the distribution of single bidding. Specifically, we preserve network structure and the label of nodes as core or periphery, but randomly shuffle the single-bidding labels. We restrict the randomization by permuting single-bidder labels only across contracts with the same two-digit CPV code. This takes into account market-specific effects. For each market, we divide the observed rate of single bidding in its core \(SB_{\mathrm{core}}\) by the average rate of single bidding in its core over 1000 CPV-preserving randomizations \(\mu (SB_{\mathrm{core}}^{rand})\). We plot the results averaged over all years for each country in Fig. 6.

Unlike in the previous figure which indicated a strong relationship between market centralization and corruption risk, no clear pattern emerges. In some countries, corruption risk in the form of single bidding is overrepresented in the core. In other countries, it is underrepresented. For example, Czech Republic and Hungary have similar overall corruption risk scores, but in the former single bidding is more common between core issuers and winners, while in the latter it is more common between actors on the periphery. In the second panel of Fig. 6, we show that the size of the core does not have any straightforward relationship with the overrepresentation of corruption risk in the core.

Fig. 6
figure 6

Comparing the relative prevalence of single bidding in the core of EU procurement markets with their overall single-bidding rates and the relative core sizes, respectively. The dotted line represents the null model: The expected prevalence of single bidding in the core when single bidding is randomized within sectors

Full size image

We draw several conclusions from this analysis. The first is that, as mentioned, there is a strong correlation between centralization in procurement markets and their overall procurement risk. What is less clear is by what mechanism this effect might emerge. Our latter findings show that centralization is not related to higher corruption risk in either the core or periphery in general. The observed heterogeneity in the tendency of corruption risk to accumulate in the core or periphery of different countries has important policy implications because the strategies used by corrupt actors to extract rents are likely very different.

4.2 Edge clustering

Another dimension of potential heterogeneity in the distribution of corruption risk is its clustering. Previous research has found that corruption is significantly correlated in the network, with high corruption risk edges typically bunched together [14]. To demonstrate this idea, we plot the Hungarian procurement market in 2014 in Fig. 7. Coloring edges with above-market average rates of single bidding, a clear pattern emerges: The top left cluster of nodes has significantly higher rates of single bidding than other parts of the network. We propose to quantify this tendency, as well as the overall tendency of a procurement market to exhibit topological clustering using community detection. We first group the edges of the network into communities and measure the quality of this partition. We then calculate the variation of single-bidding rates across communities as a measure.

Fig. 7
figure 7

The 2014 Hungarian procurement market. We plot the largest connected component of the network, filtering out nodes involved in less than three contracts for the sake of visualization. Nodes are buyers and suppliers of contracts, connected by an edge if they contract with one another. Edges are colored red if the single-bidding rate on the edge exceeds the average rate of single bidding that year. Single bidding is significantly overrepresented among the edges in the top left cluster

Full size image

Community detection is a distinguished area of research interest in network science [73]. There are many methods to partition the nodes of networks into communities. As our object of interest, corruption risk, is an attribute of network edges, we should rather partition the edges of the network into communities. There are several well-studied methods used to cluster the edges of networks into so-called link communities. We adopt one approach based on line graphs [74, 75]. The line graph of a network is the network obtained if the edges if the original graph are considered as nodes and then connected if they share a node in the original graph. We transform each of our networks into the corresponding line graph, obtaining a new network for each one in which the nodes correspond to the original network’s edges or contract relationships. We can then apply any standard clustering algorithm on the new network to obtain an assignment of each issuer–winner edge to a community. We apply the Louvain method, a computationally fast and accurate method commonly used to detect communities in networks [76].

We can also use the partition of the line graphs into communities to describe the extent to which the networks are topologically clustered. We measure the tendency of edges to be within rather than between communities detected by the Louvain method using modularity, a quality function of network partitions. Modularity varies between − 1 and 1, with negative scores indicating that edges are rather present between “communities” of the given partition than within them, scores around 0 indicating that there is no difference in the frequency of inter- vs intra-community edges, and higher values indicating that edges are much more likely to be between nodes of the same community rather than across communities. In other words, higher modularity scores indicate that the network has more distinct and separated groups of nodes.

There is significant topological clustering in all countries and years, with modularity ranging from .55 to .80. For reference, the 2014 Hungarian market plotted in Fig. 7 has a modularity of .71. Given the partition of the edges of each network into communities, we can now quantify the extent to which single bidding varies across the different communities of a network.

Fig. 8
figure 8

Clustering of single bidding in procurement market networks by country, averaged over 2008–2016. Higher values indicate higher variation of single-bidding rates across communities in the market relative to a shuffled null model, indicated by the baseline value of 1

Full size image

The partition of edges in a network, denoting contracting relationships between issuers and winners naturally gives us a partition of all contracts awarded in a given market. We then calculate the coefficient of variation of single bidding across the clusters, defined as the standard deviation of the single-bidding rates across the contract clusters over the average. As clusters can have significantly different numbers, we actually calculate the weighted standard deviation \(\sigma ^{W}_{SB}\) and mean \(\mu ^{W}_{SB}\) of single bidding across clusters, defined as follows:

$$\begin{aligned} \sigma ^{W}_{SB} = \sqrt{ \frac{ \sum _{c\in C} |c| \left( sb_{c} - \mu ^{W}_{SB}\right) ^2 }{ \frac{(|C|-1)}{|C|} \sum _{c\in C} |c| } }, \end{aligned}$$

and

$$\begin{aligned} \mu ^{W}_{SB} = \frac{\sum _{c \in C} |c| sb_{c}}{\sum _{c \ in C} |c|}, \end{aligned}$$

where C denotes the set of contract clusters, c is a specific cluster, and \(sb_{c}\) is the rate of single bidding in the cluster c. The weighted coefficient of variation, which in our context we refer to as the clustering of single bidding, is simply the ratio \(\sigma ^{W}_{SB}/\mu ^{W}_{SB}\).

As with our measure of centralization, we compare our observed clustering of single-bidding measure against a suitable null model. We again opt to randomize the single-bidder label on contracts within the 2-digit CPV classes to create a realistic yet randomized distribution to compare against the empirical data. Thousand times we shuffle the single-bidder label on all contracts and recalculate the clustering of single bidding for these randomized markets. We divide the observed clustering of single bidding by the average of the 1000 randomized clustering of single-bidding scores. We plot the result by country, averaged over 2008–2016, in Fig. 8. In every case, single bidding is non-trivially clustered within communities in the network. The magnitude of observed clustering ranges from 2 to over 8 times higher than expected in the sector-preserving randomization. We interpret this as significant evidence that corruption is not randomly distributed in the public sector.

The extent to which corruption risk clusters in the market has important policy implications. The greater the clustering of single bidding in a market, the more likely it is that investigations of the network neighbors of known corrupt actors will be successful. For example, returning to our visualization of the Hungarian market in 2014 (see Fig. 7), we do not only suggest that corruption risk is significantly higher in the northwestern community, but also that investigators of corruption in Hungary should consider this pattern of clustering in the future to shape their strategy.

We now compare our two measures of the distribution of corruption risk in markets. For simplicity, we consider only those countries with above average single-bidding rates in our data. We plot the centralization of corruption risk against its tendency to cluster for these countries in Fig. 9. We observe that although corruption risk is high in all countries, corruption risk within each country is distributed differently. Countries in the bottom right corner of the plot, such as Portugal and Italy, have higher corruption risk in the core of their markets and a weaker tendency for risk to cluster. Countries in the top and center of the plot such as Poland and Latvia have a neutral distribution of risk across the core and periphery of their markets, but overall risk has a strong tendency to cluster. Finally, countries such as Hungary, Slovakia, and Estonia have higher corruption risk in their peripheries and a moderate tendency for risk to cluster.

Fig. 9
figure 9

The relative prevalence of corruption risk in the core of the market plotted against its tendency to cluster, plotted for countries with above average single-bidding rates in the EU, 2008–2016

Full size image

This emphasis on the distribution of corruption risk presents a novel way to think about corruption in a comparative manner. Though it may not be accurate to say that each country with high corruption risk is corrupt in its own way, these deviations suggest that corruption may be organized in different ways, likely reflecting political and economic constraints. The differences in topological structure (i.e., the degree of centralization, measured by the share of contracts in the core of the market, and the clustering of edges into communities measured by modularity) of the markets also indicate that interesting structures emerge in the mesoscopic level of these data: between the micro-level of individual transactions and the global picture of the entire market.

5 Discussion and future work

In this paper, we applied network science methods to mine a big administrative dataset on public procurement contracts for insights on the distribution of corruption risk. We found that countries may have similar levels of corruption risk but significantly different distributions of that risk. In some countries, corruption risk is more prevalent in the center of the network, while in others among peripheral actors. We also found that the degree of centralization in procurement markets overall is a strong predictor of their global corruption risk and low quality of government. Another dimension of heterogeneity is the degree of clustering of corruption risk: Though corruption risk is significantly clustered in all countries in our data, the extent of the clustering varies between 2 to 8 times what is expected under a sector-preserving randomization.

These heterogeneities are not merely curious artifacts in the data. They have significant implications for anti-corruption policy. For example, when corruption risk is centralized, it is possible that the central government itself is a hotbed of corruption and should not be trusted to address the problem. If corruption risk is highly clustered, successful investigations of corruption should snowball by following up with “nearby” actors. Overall, our findings serve as a complement rather than a substitute to traditional research on the study of corruption risks. They represent an opportunity to examine corruption risk from a new perspective that effectively leverages a new emerging data source. We also note that our measure of corruption risk is only a proxy—there are other potential reasons that competition is lacking in a market. This is another reason why our approach should not replace but rather enhance previous work.

We suggest several directions to extend our findings. For instance, considering that corruption is known to vary considerably at the regional level [36, 51], geographic information about the locations of issuers and winners could enhance our analysis [53, 77]. Before we can adjust the granularity of our study, more work is needed to define markets is a rigorous way. For example, markets for different goods and services likely have different characteristic geographic sizes: IT consulting services markets are likely more geographically diverse than markets for road repair, where transportation and fuel costs are a significant part of the budget. Defining market boundaries from the data can be misleading.

More work is needed to understand the evolution of corruption within countries. Our approach aggregates both corruption risk and network structure over time—certainly it is the case that corruption can change over time. Past work on the response of corrupt networks to political turnover suggests that procurement markets are significantly rewired following changes of government [78]. Methods to analyze temporal networks are becoming increasingly sophisticated [79, 80]. Comparative analyses and case studies using these data and these new methods promise to enhance our understanding of the different ways corruption works, and potentially how it can be limited.

Policymakers are of course not only interested in measuring and understanding corruption, but mitigating it. Future work should describe and test possible interventions that leverage network maps of corruption risk. Given the significant clustering of corruption risk in all the countries we studied, it would be of interest to move authorities responsible for anti-corruption across geographic and market boundaries, observing whether certain individuals are more effective anti-corruption actors. The major challenge in this area is not that we lack ideas for experimentation, but that obtaining buy-in from officials is difficult, especially in countries where corruption is a significant problem.

More broadly, our work demonstrates the potential of the application of a data science perspective on newly available administrative data. We agree with Rademacher’s claim that open data are fundamental for open societies [12]. As access to data becomes more abundant, the value of statistically sound, ethically responsible data science as an input to policymaking will only increase. The value of high quality public administrative data can also be increased by integrating it into broader research frameworks [81], enabling scientists to solve real-world problems by combining data in novel ways. For example, previous attempts to model the relationship between corruption and economic growth [82, 83] and complexity [84, 85] could be revisited using our fine-grained indicators of corruption risk.