Mapping the international health aid community using web data

2.1 Entity definition

The first step for extracting relations between entities from web pages is to define which are our entities. Entities are coded through keywords. Here we focus on three types of entities: the aid organizations part of our study, the issues related to health and aid, and the set of countries in which the organizations operate. All lists of entities have been curated by domain experts.

We start from the latter mentioned: the countries. This is arguably the less problematic task among the three, because countries are generally identified by their name. Thus, the list of country entities is simply the list of country names on which we want to focus. There are few exceptions, for instance in the case of Jordan we make sure the webpage is writing about the kingdom of Jordan and not about a person called Jordan. In the case of the Kyrgyz Republic, we have to be aware that, in some cases, it is referred to as “Kyrgyzstan”. Table A1 reports the list of countries that were selected in this study as examples, to illustrate how the organizations relate in specific country-settings based on variety in income level, fragility status, and size of the World Bank Group support to health services. The countries were selected among the subsample of countries that had received World Bank and IFC financial support to improve health services during the 2005–2016 period, in order to cover all World Bank Group geographical regions; different income levels and fragility.

Next, we define the international aid organizations that are part of the study. This list has two ends: the aid organizations are the subject of the study, but at the same time they are the source of the data of the study. It is necessary not only to refer to them in an unambiguous way, but also to identify their website. In many cases, it was possible to name an agency which did not have a website, or it did not have a website in English (more on this requirement in the next section), and thus the organization was dropped from the study.

For each organization, we scan the webpages in search of its full name. Since it is rare that organizations are mentioned by their full name—in some cases it can be quite long—we look also for its official acronym. Moreover, we define, where appropriate, the colloquial and most common handle for the organization: for instance the “Bill & Melinda Gates Foundation” is usually mentioned simply as the “Gates Foundation”.

There are some cases in which an organization can have multiple websites. For instance, the World Health Organization has five of them. In that case, all websites are crawled and the information gathered from all of them is connected to WHO. Table A2 reports the list of countries on which we focus in this study, along with their website’s URL.

Finally, we define the issues we want to study. In this paper, we are focusing on health issues and organizations, so our keywords are narrowed to this domain. Like in the case of countries, we selected keywords that characterize World Bank Support to health services provided during the 2005–2016 period. Each keyword is reduced to lower case. This means that the crawler will match a page with a keyword regardless of its case.

While also the organization names are case insensitive, this does not apply to the acronyms, which have to match also the case. This is to avoid to match “WHO” to the pronoun “who”. This measure is necessary because we do not perform standard stopwording to the text, meaning that we do not remove syntactic words (like “the”, “and”, etc.) from the webpage text. The reason is that organization names contain stopwords, and we would miss matches if we removed them.

A domain expert defined the keywords to avoid ambiguity. For instance, the simple word “health” is not a keyword, because the word can be used ambiguously as a metaphor (e.g. in the formula “the health of a country’s economy”). Table A3 reports the list of keywords on which we focus in this study.

2.2 Web crawling

The data is collected by a standard web crawler, implemented in Python using the scrapy package.² Table A2 lists the starting URLs, which are the input of the crawler. The crawler is instructed never to leave the starting domain, so no page outside the selected websites is considered. If a domain has a sub-domain (e.g. www.worldbank.org contains data.worldbank.org) the sub-domain is also crawled. We impose no depth limit, meaning that the domains are fully crawled, without discarding pages that are more than n clicks away from the home page.

In some cases, webpages need JavaScript to properly render their content and hyperlinks to other pages. We use a headless web browser—PhantomJS³—able to execute the code to render the page.

Each page is firstly tested against the list of our issues, as defined in Table A3. If the page does not contain any of the specified issues, it is discarded and not used for any of the analysis in this paper. If the page does contain at least one issue, then we store all the keywords—issues, countries and organizations—mentioned, and all hyperlinks to other websites included in the study.

In all three cases, we would obtain inflated counts for term co-appearances. Our counter-measures are as follows.

Regarding issue #1, an easy solution would be to automatically filter navigation menus in headers/footers. However, given differences and lack of standards in web design, this is not feasible when crawling multiple websites. We are then forced to perform a post-process of the crawled data. We observe that, if terms are appearing in structural components of a website, they will always appear the same number of times in each page. In other words, menu items will result in a background occurrence frequency for some keywords. To take this into account, we establish the mode appearance of a word in a website. If a page mentions the word less than or equal to the website mode, the mention is discarded, unless the mode happens rarely, thus it cannot be associated with a menu item.

Regarding issue #2, we also propose a post-process cleaning operation. If the same page was crawled n times from n different URLs, then we will find n copies of the same sets of terms, in the same order and with the same frequency. We scan all the crawled pages from a website and, if this is the case, the n copies of the pages will be counted only as one. This is an imperfect solution, because there could be two genuinely different pages with the very same term composition, but we assume that such coincidence will be rare and unable to affect significantly our results.

We address issue #3 in the next section, because its solution is implemented in the process that transforms the crawled webpages into the edges of the networks we build.

2.3 Building the networks

Once the web crawling is complete, the result is a set of pages—cleaned with the post processing procedure explained in the previous section. A page p is defined as a set of entities that co-appear on the page, among which at least one has to be a health issue, otherwise the page is discarded. Every time two entities—say i and j—co-appear in a page of a website w, we increase their co-occurrence counter \(n^{w}_{ij}\) by one. We keep a different co-occurrence counter for each website to address the issue #3 mentioned in the previous section.

\(\mathit{WS}_{ij}\) still provides a higher connection strength from w if w has many pages mentioning i and j, but it is difficult for it to dwarf smaller websites that could mention i more often with a third entity k rather than with j. Note that \(\mathit{WS}_{ij}\) is symmetric, meaning that \(\mathit{WS}_{ij} = \mathit{WS}_{ji}\). This is because co-appearance is a symmetric relationship (\(n^{w}_{ij} = n^{w}_{ji}\)).

In this paper we also look at a citation network, connecting organization i to organization j if i cites or links j. The connection strength is still estimated using \(\mathit{WS}_{ij}\), replacing \(n^{w}_{ij}\) with \(c^{i}_{j}\). Rather than recording the number of co-appearances of i and j, \(c^{i}_{j}\) records the number of times i mentions or links j in its website. In this case, symmetry is broken, as \(c^{i}_{j} \neq c^{j}_{i}\): even if i links to j, it is possible that j does not link back.

\(\mathit{WS}_{ij}\) is at the basis of all our networks with a single layer (i.e. the ones we aggregate from the entire crawled data): if i and j are entities of the same type (say they are both countries), then the network is unipartite. If i and j’s type is different—say one is an organization and the other a issue—then the resulting network is bipartite.

We can create a version of \(\mathit{WS}_{ij}\) for generating multilayer networks. If we focus on a single country c by selecting exclusively the pages mentioning c, we can create a derived \(\mathit{WS}^{c}_{ij}\). If we generalize this by collecting all possible \(\mathit{WS}^{c}_{ij}\)\(\forall c \in C\), the result is a multilayer network whose layers are the network for a given c, which represent how that particular country is “seen” by the international aid community. The same operation can generate multilayer networks whose layers are co-occurrences coming from a single website w, which represent how that particular organization “sees” the international aid community on health.

The last step in the network generation process is network backboning. Web data is very noisy, meaning that a page might mention two entities in very different parts of it, and the relation between the two might be weak. Network backboning is a cleaning procedure designed specifically to identify and remove from the network spurious communities. For most of our networks, we use the noise-corrected backbone, which establishes the expectation for an edge weight and keeps it in the network only if the actual edge weight is higher than this expectation [20]. This method is specifically designed for count networks, i.e. networks whose edge weights are results of a discrete counting process. This is the case for the web networks studied here, as each co-appearance is increasing a counter.

The citation network is a special case for two reasons. First, citing and linking a website is an act less prone to noise: a hyperlink is a stronger signal than the co-appearance of two terms in a text. Second, it is a network whose higher-order is more meaningful than in the co-appearance network. The number of edges required to go from i to j in a citation network is meaningful, because it is proportional to how difficult it is for information to flow from i to j. If two entities are not connected in a co-appearance network it means they are never mentioned together, so it makes little difference if their network distance is two or four edges. For these reasons, this backbone is instead generated using the high-salience skeleton [21], which is built exactly looking at shortest paths in the network.

2.4 Postprocess blacklisting

Even if we use proper names and acronyms to distinguish an organization’s name from the rest of a page text, our procedure is not exempt from possible mistakes. Names like “World Bank” or “Plan International” are phrases that can be used colloquially, without necessarily referring to those organizations in particular. We create a procedure to blacklist organizations in case our pre- and post-process were unable to perform this distinction properly.

For this phase, we use specifically the citation information. Each organization can be cited in two ways: (i) in the text of a page and (ii) by a hyperlink. While the former way is affected by the aforementioned problem, the latter is not: a link cannot be used colloquially in a text without wanting to explicitly refer to that particular website. Therefore, if an organization is disproportionally cited through mechanism (i), then we can argue that those documents were using the sentence without really referring to the organization itself. Those organizations are likely to introduce noise in the networks and have to be dropped.

Our test is performed using a simple linear regression, given the assumption of the strong relation between mechanisms (i) and (ii). In practice, we want to use the number of times an organization is cited by hyperlink to predict how many times it will be cited by name. Organizations with a high residual are candidates to be dropped from our networks—although they are kept as data sources, since the content of their website is not affected by this issue.

When we look at the top residuals of this regression, there is one organization that stands out: Plan International. Its residual is 68% higher than the second largest residual. The average of this ratio for the other residuals in the top 10 is only 12%. As a consequence, the only organization we drop from our networks after this procedure is Plan International.

3.1 Unipartite networks

We start our analysis by looking at the unipartite networks. The first on which we focus is the network of co-citations of international aid organizations. In this network, two organizations are linked to each other if they are mentioned in the same web page a significant amount of times.

Figure 1 depicts this network. The first noticeable property of the network is the tendency of organizations to cluster with similar organizations. This homophily can be estimated by calculating the modularity of the partition obtained by classifying organizations according to their class (as described in Sect. 2.1). Modularity is a measure estimating how many more edges are inside a node community than we would expect [22]. Modularity takes values between −1 and 1. In this case, we observe a value of 0.34. This is remarkably high considering that we obtain a modularity of 0.47 when we detect node communities with an actual community discover algorithm (Infomap [23]). This means that the organization class is a reasonable way to partition this network, i.e. organizations of similar classes tend to cluster together.

Open image in new window

Note that this is a significance map, meaning that we only see the associations that are stronger than expected. There are strong associations that are not in this network, because they are a reflection of business-as-usual for the connected organizations. The distribution of edge weights is broad, and we depict it in Fig. 2. The distribution has a strong exponential cutoff. This means that very well cited organizations fail to be mentioned together as much as we would expect—as this would generate a power-law. Table 1 reports the top ten organizations associated with the World Bank according to the WS measure.

Open image in new window

Table 1

Top ten organizations according to WS for the World Bank in the co-citation network

Rank	Organization	WS
1	WHO	92.69
2	European Union	91.07
3	UNICEF	78.52
4	UNDP	69.99
5	USAID	58.23
6	IMF	56.66
7	UNF	53.71
8	AfDB	49.82
9	DfID	49.57
10	Gates Foundation	48.74

Keeping our attention on the organization-organization network, we now focus on its citation variant: here organizations are connected not if they co-appear in a page, but if they directly mention (or link) one another. In this case, connections are not symmetric: the edges have a direction going from the citer to the citee.

Note that being the most central node in the network does not necessarily mean that the node is the most cited. In fact, the third most cited organization (according to the sum of the WS of the edges pointing to it) is the World Health Organization, which is eighth most central. UNDP and FAO received less citations, but they have higher betweenness centrality. This means that the additional citations come from either the same edge or from other organizations that do not contribute to the centrality of the WHO.

In the citation network, another important question regards connected components: is there a citation path going from an organization to any other organization? A disconnected space would mean that there is information that cannot reach every organization in the system.

The citation network has only one weakly connected component. This means that there is a path between any two organizations, provided that we ignore the direction of the edge [25]. The majority of the network is composed by nodes that are target of a citation, but do not cite back in sufficient amount to clear the backboning threshold. There is a core of 15 aid organizations part of the only non-trivial strongly connected component (i.e. a component containing more than one node). With the exception of Save the Children, all organizations in Table 2 are part of this core of the network.

We conclude the section on unipartite networks by looking at the other two types of networks we generated: the issue–issue and country-country networks. We do not perform analysis on them as this paper is more focused on aid organizations. However, they are objects with interesting descriptive power. They tell us how aid organizations talk about health issues and countries.

Figure 3 depicts the issue network. Two issues are connected if they co-appear in webpages more often than we would expect given their popularity. There are two interesting aspects. First, terms that are synonyms or quasi-synonyms are connected in this network—such as “DAH” and “Development Assistance for Health”, or “TB” and “Tubercolosis”. This is a sign that the methodology is valid, as it was able to catch such relationships.

Open image in new window

The second aspect concerns our ability of defining clusters of issues. We perform a simple community discovery with the Infomap algorithm [23] and it returned us the clusters depicted in the figure. The communities help us in making sense of different issues as being part of macro categories. For instance, we can delineate a reproductive health macro category (in orange) as distinct from the one focused on infrastructure and governmental cooperation (in red).

Similar considerations can be made for the country network. Figure 4 depicts it. This network can also be used as a mean of validation of our Web data. We expect to find connections between countries that share a similar geographical and cultural position. Countries affected by the same issues should also be closely connected.

Open image in new window

In fact, we see that there is a strong geographical component, showing an African, American, Middle East and Far East cluster. Romania is the only European country, and thus has smaller connections. Yet, its most significant connection is with its neighbor Turkey.

3.2 Bipartite networks

We now turn our attention to bipartite networks. In a bipartite network, we have two classes of nodes. Edges only connect nodes belonging to unlike classes. In this work we have three possible types of bipartite networks: organizations connected to keywords, organizations connected to countries, and countries connected to keywords. The connection criterion is the same used for unipartite networks: the co-appearance of the two terms in the same webpage.

The first network we analyze is the one connecting organizations to keywords. Our first question is: how can we estimate which issues are the most associated with an organization? To answer this question we create a derived WS measure. \(\mathit{WS}_{oi}\) is the total number of co-appearances of organization o and issue i. We can define \(\mathit{WS}_{i}\) as the total number of times i was mentioned. Note that \(\mathit{WS}_{i} \neq\sum_{\forall o \in O} \mathit{WS}_{oi}\), because we can have a page mentioning i and two different organizations, so that page would be double counted.⁴

We can visualize the relationship between the World Bank and all issues on the issue network we built in the previous section. In Fig. 5 (left), we use the same layout used in Fig. 3. The only difference is in the color of the nodes. Instead of showing network communities, we show the value of WS% (\(= 100 * \mathit{WS}_{oi} / \mathit{WS}_{i}\)). From the figure, we can see that the World Bank tends to have lower WS% only for very popular issues (large nodes), and it tends to be mentioned disproportionally often with smaller issues.

Open image in new window

Is there an alignment between what the World Bank is talked about (i.e. the terms with which it co-appears) and what the World Bank is talking about (i.e. the terms it mentions in its website)? Fig. 5 (right) is an attempt to answer this question. The WS% indicator as calculated so far is equal to \(\mathit {WS}_{oi} / \mathit{WS}_{i}\). If we substitute the numerator, we can estimate the share of pages mentioning i coming from a given organization website o: \(\mathit{WS}^{o}_{i} / \mathit{WS}_{i}\). In a perfectly aligned world, these two ratios should be the same: an organization is mentioned together with an issue exactly as much as this organization itself mentions the issue. Disagreements in these two ratios imply that o thinks some other issue is more relevant to itself than i.

Figure 5 (right) shows that there is a correlation between the two ratios, but it is not perfect. The vast majority of issues is clustered into a clear linear relationship; however, there are a handful of outliers. Secondary/Tertiary Care and Avian Influenza are mentioned by the World Bank more than the World Bank is mentioned with them; on the other hand, the World Bank mentions Infant Survival and Neglected Disease proportionally less than it is mentioned about them.

A plausible reason to explain why in the World Bank site “Infant Survival” is found more often with other partners (i.e. high co-appearances relative to citation), while other terms, such as “Avian Flu” is found more often in isolation (i.e. high citation relative to co-appearances) could be that the World Bank operates to improve “Infant Survival” often with other health organizations (e.g. UNICEF and WHO). On the other hand, the World Bank often operates in the “Avian Flu” domain with no-(human)health organizations (e.g. OIE—World Organisation for Animal Health) as avian flu is relevant both to human and animal health.

We can repeat the same exercise in the organization-country bipartite network. Table 4 reports the results of the same analysis performed for Table 3; Fig. 6 depicts the same scattergram shown in Fig. 5 (right), shifting the attention from issues to countries.

Open image in new window

Table 4

The ten most associated countries to the World Bank according to WS

Rank	Keyword	WS%
1	Vietnam	4.33%
2	Kyrgyzstan	4.33%
3	Haiti	4.27%
4	Turkey	4.24%
5	Peru	4.24%
…	…	…
13	Sierra Leone	4.14%
14	Sri Lanka	4.11%
15	Indonesia	4.11%
16	Philippines	4.08%
17	Nigeria	4.02%

The main difference between the two bipartite networks is that WS% distributes differently. There is less variation in the organization-country network than in the organization-keyword network. For the World Bank, the difference between the most associated country (Vietnam) and the least associated country (Nigeria) is just above 0.3 percentage points.

Just as in the case with issues, also for countries there is a strong correlation between the number of times the World Bank is mentioned about a country and the number of times it mentions the country itself—with the presence of some outliers. Romania is mentioned by the World Bank more than we would expect given the speech of the rest of the aid community, while Jordan and Egypt are mentioned fewer times.

We conclude this section by looking at the country-issue bipartite network. In this case we cannot compare the association as emerging from co-occurrences against direct mentions, because we do not have a source of data on what a country talks about. For this reason, we limit ourselves to providing a simple ranking. We want to answer the question: which issue is most related to which country?

3.3 Multilayer networks

In this section we focus on multilayer networks, i.e. networks whose nodes can be connected with different criteria. We focus on a single variant on all the possible multilayer networks we could build. Analysis similar to the one delineated in this section can easily be carried on the other types of multilayer networks. Here we are interested in investigating the network connecting issues together.

The layers of the network come from the different websites crawled. In other words, we build a unipartite issue–issue network just like we did in the Unipartite Networks section, by looking exclusively at data coming from a website at a time. In this way, we have a layer showing us the issue–issue network in the eyes of the World Bank, a layer for UNICEF, a layer for the WHO, and so on. Figure 7 (left) shows an example of such structure.

Open image in new window

In the Unipartite Networks section we saw that there is an issue–issue network arising from the combination of the discourse of all international aid organizations. From that network, we discovered that—for instance—HIV is related to Maternal Health. From where is this connection coming? How do the networks emerging from the speech of each aid organization combine to form the big picture shown in Fig. 3?

In [17], the authors introduced two concepts that are useful for thinking about this question. These concepts were “complementarity” and “redundancy”. If we look at all the edges in the multilayer network, we can record in how many layers a connection is reproduced. If on average all connections are present in most layers, then we can conclude that there is a high degree of redundancy: the deletion of an edge in a specific layer does not affect the connectivity of the multilayer network as a whole. However, if most edges are present in a single layer, then the deletion of a single edge from i to j in the only layer in which they are connected has a dramatic effect. If we perform such deletion, there is no direct way to go from i to j.

In other words, \(\rho_{ij}\) is the average number of layers in which i is connected to j.

If we calculate the average \(\rho_{ij}\) value for all the edges in the issue–issue network shown in Fig. 3, we obtain a value of 0.035, meaning that the average edge appears in only 3.5% of all layers. This seems to suggest that the network is characterized by a certain level of complementarity. The intuition is confirmed if we look at the edges with the highest value of redundancy. The top eight edges, the ones present in 13 or more layers, are either synonyms or antonyms frequently used together: Malnutrition and Nutrition, Communicable Diseases and Non-communicable Disease, AIDS and HIV, and so on. If we were to collapse these issues as one, the average redundancy would be even lower than 0.035.

Moreover, there are 19 connections in the issue–issue network as shown in Fig. 3 that do not appear in a single layer. This means that it is the aggregation of correlated discourses that make these connections emerge, even if there is not a single organization for which the two issues are co-mentioned a significant amount of times. Table A4 reports the full list.

Another multilayer network analysis focuses on estimating the similarities between each layer and the whole structure. We can calculate two related, but complementary measures: Alignment and Impact. Alignment tells us how much a layer is similar to the whole network. To estimate Alignment, we calculate the correlation coefficient between the global issue–issue WS values and the ones coming from a single layer. A high Alignment value means that the connection strength between two issues in the eyes of an organization is correlated with the strength in the overall network. In other words, the organization agrees with the community about the relation between the two issues.

On the other hand, Impact tells us how much of the global structure is due to the organization itself. We define Impact as the normalized difference between Alignment and Alignment∗. Alignment∗ is the correlation between the WS values of the layer and the ones coming from the global network minus the layer itself.⁵ A high Impact value means that, if we were to remove the layer from the global network, the network would change into something rather different than the layer itself. Impact equal to zero would imply that removing the layer from the network would not change the distribution of the WS values at all.

Unsurprisingly WHO, the main actor in health, is in the leader quadrant: its direction emerges as a defining force of the global health arena. Many organizations score a higher impact than WHO, for instance UNICEF and Doctors without Borders, as they likely use their influence in a more visible way for the population at large. The position of World Bank as explorer also makes sense: the World Bank is a large organization which can coordinate a significant human and economic effort, however it is not traditionally thought of as a health provider, and thus has a lower alignment level with the community.

3.4 Case study: World Bank

We now focus our attention on a single organization: the World Bank. The aim of this section is to provide some examples of organization-centric analyses. We focus on the World Bank because it is one of the largest organizations in international aid. Among its many commitments—helping nations to build more equitable societies and to improve fiscal performance and country competitiveness—the World Bank Group (WBG) also works in the health sector. To support countries in reaching the goal of UHC by 2030 (all people and communities can use the promotive, preventive, curative, rehabilitative and palliative health services they need, of sufficient quality to be effective, while also ensuring that the use of these services does not expose the user to financial hardship), the Bank provides financing, state-of-the-art analysis, and policy advice to improve service delivery and expand access to quality, affordable health care. During the period from fiscal year 2000 to 2016, the World Bank invested US$35 billion in the Health, Nutrition and Population (HNP) thematic areas. Over this period, the average annual lending increased significantly from US$1.3 billion to US$2.4 billion. The Bank currently manages an active HNP portfolio of $11.9 billion.

The WBG has focused its health sector investments and research in areas that are especially vital to helping countries achieve UHC by 2030, working closely with donors, development partners, governments, and the private sector. Some of these focus areas include ending preventable maternal and child mortality; reducing stunting and improved nutrition for infants and children; strengthening health systems and health financing; ensuring pandemic preparedness and response; promoting sexual and reproductive health and rights; and the prevention and treatment of communicable diseases.

In light of this, the World Bank is among the main actors in this area, and one of considerable impact—as highlighted in Fig. 7 (right). Moreover, we have access to additional data sources that make the analysis more substantive.

We can correlate World Bank’s Involvement with WS. Figure 8 shows their direct relationship. The Spearman rank correlation between these two measures is 0.6, with a \(p\mbox{-value} \sim0.01\). Obtaining an almost significant p-value with only 17 observations is remarkable, even more so considering that there is one clear outlier (Romania).

Open image in new window

To conclude this section, we perform an analysis borrowing tools from network epidemics [19]. Real communication channels between organization do not usually take place on the Web, and thus are outside the data we analyze. However, we can interpret the organization-organization citation network as a trace of past communication: if two organizations are talking about each other and they link each other, it means that they are aware of their actions in the field, and are paying attention at each other discourse. In this light, the citation network can be used as a forensic tool to analyze influence after it took place. Note that, in the original network, we establish a directed edge going from organization i to organization j if i cites j. Here, we are interested in knowing that j influenced i, so we reverse the direction of all edges, transforming the citation network into an influence network.

Susceptible–Infected (SI) models have been defined for network data [26]. In these models, nodes are assigned to one of two classes: Susceptible if the node can be infected with a disease, and Infected if the infection happened. In the most simple model—which we use here—a parameter β is specified: if more than a β fraction of the incoming edge weight of a Susceptible node comes from Infected nodes, then the node turns from Susceptible to Infected. These models have been successfully used to track the spread of information in social networks [27].

Here we assume that at time step zero only the World Bank is “infected” with a message it wants to spread. We run an SI model, recording at each time step the fraction of nodes that are part of the Infected pool. Here β represents the share of “infected” cited pages by an organization i necessary to infect i itself. If \(\beta = 0.1\), to infect i we require that at least 10% of the pages cited by i have to come from infected organizations. In practice, higher βs mean that we require more citations between organizations A and B to say that A has influenced B in the past.

If we assume that all influence connections in the network can be used, the World Bank is able to infect almost the entire network (\({\sim}94\%\) of the nodes). The leftover nodes have no sufficiently strong incoming connections, and thus cannot be influenced. This is tested across a variety of β values. Figure 9 (left) depicts the result of these simulations. The higher the β parameter the harder it is for an infection to spread. However, in the case of the World Bank this only affects the speed of propagation of the information (from three to seven steps), not the final coverage. This implies that, according to this model, if the World Bank sends a message to the international aid community, likely 94% of organizations will receive it eventually, assuming that the real unobservable epidemics parameter β is equal to or lower than 0.15.

Open image in new window

However, not all messages are equal. The content of the message likely influences its chances to be passed or not. We can simulate also this case, by creating a multilayer view of the influence network, where each layer only contains citations made from pages containing only a specific keyword. We can run the SI model using exclusively edges coming from a single layer, which will now inform us about the power of the World Bank to influence organizations exclusively about a specific issue.

Figure 9 (right) depicts the result of these simulations. We also report the result of the simulation using all the layers, for reference. We can see that there are significant differences between issues. The World Bank is able to reach the vast majority of nodes in some cases, for instance when talking about Public Health (the final infected share of nodes is \({\sim}88\%\)). However, in the case of Avian Influenza the message finds a bottleneck in the multilayer network, and only reaches a third of the network.

How does the World Bank compare to other organizations? Here we choose three comparisons: WHO, UNICEF and USAIDS. We run the SI model using all connections from all keywords, and fixing β once again to 0.1. Figure 10 (left) depicts the result. We can see that the World Bank is noticeably slower than WHO, which reaches saturation faster. However, the World Bank outperforms UNICEF. When considering all keywords at the same time, the messages coming from USAID are dwarfed and reach a negligible portion of the network.

Open image in new window

However, as pointed out before, these diffusion patterns are highly dependent on which keyword we are focusing. Figure 10 (right) depicts the information spreading results when focusing on a specific one: “Nurse”. In this case, the four organizations are hardly distinguishable, with USAID having an influence potential on par with the World Bank and WHO.

3.5 Stability

The Web is notoriously dynamic: there is no guarantee that a webpage you can visit now is going to be online tomorrow. In the face of such dynamism, one could argue that any snapshot gathered from the Web could be noisy to the point of not being a good representation of the system.

Our original crawl was performed in February 2017. Eleven months after, on January 2018, we perform a second crawl of a selected number of websites to test the stability of our results. We selected roughly half of the websites and apply our data cleaning pipeline. The question is: is the 2017 WS measure a good predictor of 2018 WS? We answer by doing two analysis.

First, we perform a simple regression analysis. Regressing 2017 WS against 2018 WS yields an \(R^{2} \sim74.7\%\). This means that the vast majority of the variance in the 2018 measure can be predicted using 2017 data. This result reassures us that, overall, the networks we build are stable over time. A perfect correlation would have been as problematic as a low one: it would imply that the international aid community does not change which would be a puzzling property for such a complex system. Since the correlation is not perfect, we are reassured that we are able to capture the evolution of the system.

We also provide a visual representation of the relationship between the 2017 and 2018 versions of WS, in Fig. 11. Most of the data points lie in the diagonal, which means that most edges maintained the same weight.

Open image in new window