1 The Data Demand Challenge

The outbreak of the Ebola virus disease in West Africa in 2014 constituted one of the gravest global health emergencies of recent years.Footnote 1 The Ebola outbreak originated in rural Guinea in December 2013, and then spread across the country and to the neighboring countries of Liberia and Sierra Leone. The pandemic continued for two years and the World Health Organization (WHO) only declared Liberia free of Ebola in May 2015, Sierra Leone in November 2015, and Guinea in December 2015. By the end of the crisis, the epidemic had claimed more than 11,300 lives in these three countries, including over 500 frontline healthcare workers.Footnote 2

In addition to its effects on people’s health, Ebola caused widespread economic disruption. At the height of the epidemic, schools, and markets were closed, government workers were placed on furlough, social gatherings were banned, transportation restrictions were placed on people and goods, and international borders were closed. Therefore, in addition to the health monitoring by the WHO, there was an urgent need for just-in-time data in order to monitor the economic impact of Ebola on livelihoods and wellbeing. Given the epidemic, however, it was impossible to deploy enumerators to the field to collect information from households and communities through face-to-face interviews.

The solution to this challenge came from the realization that the rapid spread of mobile phone coverage had created possibilities to monitor the crisis through mobile phone interviews. Mobile phones are particularly useful in situations in which data must be collected rapidly, at low cost, and/or in situations where traditional face-to-face interviews are not possible. In Sierra Leone and neighboring Liberia, it allowed for a timely response by providing critical data to decision makers about household welfare at the height of the crisis and during its aftermath (Fig. 1).

Fig. 1
figure 1

(Note Shading reflects dates of data collection. Source Authors calculations based on WHO Sit Rep data)

Timing of Sierra Leone and Liberia high-frequency mobile phone surveys (Color figure online)

Full size image

2 The Innovation

The proliferation of mobile phone networks and inexpensive handsets has opened up new possibilities for data collection. Since 2012, the Africa region of the World Bank supports a mobile phone survey initiative called Listening to Africa (L2A). L2A collaborates with statistical agencies and offers the possibility to complement face-to-face household surveys with mobile data collection.Footnote 3

The standard L2A approach starts with a face-to-face household survey that serves as a baseline. This baseline survey ensures that the randomly drawn sample is representative of the target population. During this survey, each respondent receives a simple mobile phone and when necessary, a solar charger. The respondents then receive calls from a call center every month, which conducts the mobile phone interviews. Survey questions are programmed in computer-assisted telephone interview software, allowing questions to be posed, and answers to be simultaneously recorded. The phone interviews are short so that data can be collected quickly, and respondents do not become overly fatigued. Data, once collected, are made available to the public.

The L2A approach has been introduced in several countries, including Madagascar, Malawi, Mali, Senegal, Tanzania, and Togo, and the L2A team has prepared a handbook documenting its experiences.Footnote 4,Footnote 5 A two-minute video explaining the L2A approach can be found on the World Bank’s website.Footnote 6

While typical L2A questionnaires are fixed ahead of time, the instrument is flexible and can adapt to unforeseen needs. In particular, the high-frequency collection was well-suited to monitor food security, and the L2A team was able to respond to the unfolding situations in Malawi, Senegal, and Madagascar (Fig. 2). A sample questionnaire with food security questions that can be used for mobile phone interviews is presented in the annex to this chapter.

Fig. 2
figure 2

(Source Authors’ calculations from the Malawi, Madagascar, and Senegal L2A surveys)

Responses on food security issues from various L2A surveys

Full size image

When the Ebola crisis began in 2014, the World Bank team had accumulated several years of experience with mobile phone surveys. Building on the L2A model, high-frequency mobile phone interventions were designed to provide rapid monitoring of the socio-economic impacts of Ebola in Liberia and Sierra Leone.Footnote 7 As the L2A approach had shown, baseline information was needed to anchor estimates in a representative dataset. Fortunately, there were recent surveys in both countries that could serve this purpose. In Liberia, the Household Income and Expenditure Survey (HIES) was being conducted as the crisis broke out, and was forced to curtail its fieldwork in August 2014. Though only about half of the sample (4075 households) were surveyed, it was nationally representative, and despite not being planned as a panel survey, had collected phone numbers and contact information for respondents. Overall, 57% of HIES households reported a mobile phone number for at least one household member. This database of phone numbers and household characteristics became the sample frame for the mobile phone survey sample. In total, five rounds of phone interviews were completed between October 2014 and March 2015. Data were collected by the Gallup Organization from their US-based call centers, as there was no suitably experienced call center on the ground in Liberia, and it was not possible to bring in international experts due to the travel ban. While using an external call center posed several challenges, including a lack of proficiency in local languages, unwillingness of respondents to speak to strangers, and a high costs of calling, the survey was able to conduct 2781 interviews with 1082 unique households over the five rounds.

In Sierra Leone, the 2014 Labour Force Survey (LFS) was also being carried out during the Ebola crisis, with fieldwork completed in July 2014. The LFS is a nationally representative survey, with a sample size of 4188 households. It was planned as a panel survey, and had therefore collected phone numbers and contact information, with 66% of LFS households reporting a mobile phone number for a least one household member. Using this database, three rounds of data collection were completed between November 2014 and May 2015. Data were collected through a call center at the national statistics bureau, Statistics Sierra Leone, supervised by Innovations for Poverty Action for the first two rounds and supervised directly by the World Bank for round three. The survey was able to reach 2111 respondents over the three rounds (Himelein et al. 2015).

3 Results from the Ebola Surveys

The Ebola surveys covered a wide range of topics, employment, agriculture, food security and prices, social assistance, remittances, migration, education, and health facility utilization. The team deliberately avoided asking questions directly related to illness within the household. Such questions were omitted for two reasons: first, to prevent non-response if households feared the authorities would come to remove ill members and, second, because the nature of the national sample was not well-suited to surveying disease incidences. The survey also included topics that were kept consistent in every round for monitoring purposes, such as those related to food security and economic activity, and some that were included in only one or two rounds based on the evolving situation. For example, the first round included questions as to whether the respondent had ever heard of Ebola and what sources of information they had on prevention. In later rounds, questions related to education were added, as schools reopened and social assistance as safety nets projects were rolled out.

The results from the survey yielded several important findings related to the economic situation. In both Sierra Leone and Liberia, the surveys found significant declines in employment during the crisis, but the effects were not significantly higher in places with higher numbers of Ebola cases. This indicates an overall economic slowdown caused by the nationwide precautionary measures, particularly the closure of markets, had more of an impact on employment than direct cases of Ebola. Moreover, in both countries, women were more likely to have stopped working during the crisis, and less likely to have returned to work by the end of data collection period. In Sierra Leone, income and labor force participation (hours) for both men and women remained below baseline levels at the end of data collection, although the overall percentage of individuals working had largely rebounded. In addition, many workers had switched sectors during the crisis, generally moving to positions with lower productivity (Fig. 3).

Fig. 3
figure 3

(Source Sierra Leone high-frequency mobile phone survey and Liberia high-frequency mobile phone survey)

Evidence from the Sierra Leone and Liberia phone surveys

Full size image

Beyond the findings related to labor markets, the surveys provided important insights related to prices, food security, coping strategies, education, avoidance of healthcare facilities, and perceptions of public safety and trust in institutions. The surveys were able to monitor the usage of healthcare facilities for non-Ebola medical care. For example, the percentage of women in Sierra Leone giving birth in the previous two months in a hospital or clinic increased from 28% in November 2014 to 64% in February 2015 to 89% in May 2015. In some cases, these findings conflicted with the anecdotal evidence that had been previously guiding policy. In agriculture, farmers in both countries estimated that the production had declined, but to a lesser extent than had been feared, and with no evidence of the widescale abandonment that had been previously reported. In Sierra Leone, a delay in the arrival of seasonal rains also played a role. In education, once schools reopened, most students returned, 87% in Sierra Leone and 73% in Liberia. Of those that did not, the reason cited was monetary rather than fear of infection.

4 Implementation Challenges, Lessons Learned, and Next Steps

Although they cannot replace face-to-face household surveys in all contexts, mobile phone surveys offer substantial benefits in specific circumstances and for specific data collection needs. Advantages include the ability to collect data in volatile and high-risk environments (such as during political crises or epidemics), flexibility and responsivity to new data needs, timeliness, cost effectiveness, and utility for monitoring and impact evaluation. However, this approach remains challenging, and several lessons have been learned.

The risk of non-response and attrition applies to all panel surveys but is more likely for high-frequency mobile phone panel surveys. In the case of L2A, several strategies were undertaken to minimize these risks. Because sample selection did not consider prior ownership of a mobile phone, some households, particularly the poorest ones, had access to a mobile phone network but did not actually own mobile phones. To overcome this, mobile phones were distributed to all selected households, regardless of whether they already owned one, and respondents received training on various aspects of mobile phone ownership. In addition, the frequent power cuts in survey locations meant that phones could not be recharged, which could then lead to non-participation. To address these power cuts, small solar chargers were provided to allow households to charge their phones and receive follow-up calls.

In L2A, respondents were compensated each time they completed a phone interview, receiving a small amount of airtime credit transferred directly to their phones. This was both to compensate respondents for their participation, thereby encouraging them to stay involved, and to prevent the cancellation of phone numbers, which is a risk for those who do not ‘top up’ their phones after a certain period (usually 90 days). The lag period between the baseline survey and the first phone interview was also kept short. During the baseline survey, phone numbers were collected for all household members to increase the chances of reaching the respondent, and respondents were asked for their preferred call times. Efforts to track and trace hard-to-reach respondents also continued throughout implementation.

Response rates for the L2A surveys were generally high, reflecting the numerous measures taken to minimize non-response. In the Ebola surveys, however, other than providing limited compensation to respondents, it was not possible to take any of the above mitigation strategies. This was compounded by low network coverage rates, particularly in rural areas, and led to low response rates and issues with sample representativeness. For those baseline survey households that did not respond in some of all of the cell phone rounds, analysts attempted to mitigate the impact of attrition by adjusting the weighting of the data. The correct weighting depends on whether cross-sectional or panel analysis is being conducted, and, in the case of panel analysis, which rounds of the survey are being compared. In the Sierra Leone and Liberia mobile phone surveys, multiple sets of weights were necessary depending on the combination of rounds. While the distribution of respondents in the mobile phone survey by age, gender, county, and sector of employment were similar to those found in the HIES and LFS samples, response rates were far lower in rural areas—compared with urban areas—due to limited network coverage. To adjust for differences in characteristics between the baseline and subsequent rounds, it was necessary to apply an attrition adjustment to the baseline survey weights. The adjustment included a propensity score adjustment, which uses the available characteristics of the household head from the baseline survey (age, gender, location, and sector of employment), and a post-stratification adjustment. This increased the total weighting of each stratum to match the distribution found in the last census. Full details of the weighting methodology can be found in World Bank (2014), and each report contains a table showing the regression results underlying the propensity score calculations on which the weighting adjustments were based. Even after taking into account these adjustments, however, careful review is necessary to determine if the results from the mobile phone survey can truly be considered representative, as opposed to merely indicative (Fig. 4).

Fig. 4
figure 4

(Source Authors’ calculations)

Response rates for the high-frequency mobile phone surveys in Sierra Leone and Liberia

Full size image

Another lesson learned was to keep the survey short. While households can and will participate in a mobile phone interview, the questionnaire must be kept short to minimize respondent fatigue, which can be a cause of attrition and non-response. Mobile phone-based surveys are therefore not appropriate for lengthy interviews or complex questions, such as those relating to household consumption. Mobile phone surveys also cannot substitute in-depth information that can be collected in face-to-face household surveys.

While fielding new ad hoc surveys to monitor an evolving crisis is possible (see Chapter 3), a more systematic approach is clearly preferable. If a representative mobile phone survey could be carried out on short notice, this would not only provide valuable real-time information, but could also be used to mount an effective response. The high-frequency mobile phone surveys to monitor the socio-economic impacts of Ebola in Sierra Leone and Liberia were possible because the most recent national household surveys had collected contact information. A proactive approach to crisis monitoring would start with the systematic creation (and maintenance) of databases with phone numbers and core household respondent characteristics. Another lesson from the Ebola crisis is that setting up a call center is relatively straightforward and can even be done from abroad.