Rapid Emergency Response Survey

Abstract

In 2017, a prolonged drought severely affected several countries in Africa and the Middle East. For less resilient countries, food shortages escalated into catastrophes, and famine-like conditions were created in Nigeria, Somalia, South Sudan, and Yemen, urging a rapid humanitarian response, but also developmental interventions to create resilience in the long term. The Rapid Emergency Response Survey was designed to collect data to inform the developmental response. Results were generated over an eight-week period, using mobile phone interviews that were designed to be applicable to other emergencies and crises. This chapter describes the methodology, sampling, and key results of the Rapid Emergency Response Survey, and draws lessons for the use of an adaptive questionnaire design to find a better trade-off between depth and comprehensiveness of data.

1 The Data Demand and Challenge

In 2017, the United Nations (UN) stated that the world was facing the most serious humanitarian crisis since the Second World War, with over 20 million people at risk of starvation and famine.¹ The crisis was concentrated in four countries: Nigeria, Somalia, South Sudan, and Yemen. Alongside hunger, large portions of the population in these countries were facing deteriorating living conditions and threatened livelihoods.²

The crisis was driven by both drought and conflict to differing degrees in the four countries. In Nigeria, the Boko Haram conflict contributed to poor market access, severe food shortages, and disruption of livelihoods in the North-Eastern States.³ For the Somali population, the dry agricultural season contributed to high food prices, livestock losses, and displacement.⁴ In South Sudan, below-average crop production and inter-communal violence contributed to famine in the former Unity State; in addition, 70% of the population of South Sudan was in serious need of humanitarian assistance.⁵ In Yemen, airstrikes and violent clashes on the ground kept food prices high and resulted in high dependency on food imports and emergency aid.⁶

The crisis required a response along the humanitarian-development nexus, to address urgent humanitarian needs while working toward short- to medium-term socio-economic development goals. The UN and the World Bank have worked to synchronize their responses to crises to the greatest extent possible.⁷ Greater development can improve resilience and reduce fragility , so that future shocks do not automatically lead to humanitarian catastrophes.⁸

During the crisis, rapid data collection was required to assess the population at risk of famine. Traditional survey methods were unsuitable for a variety of reasons. First, results were needed urgently, so lengthy household questionnaires were inappropriate. Second, funding constraints meant that costly traditional surveys were also unfeasible. Third, a significant portion of the affected populations was believed to be located in conflict-affected areas, where face-to-face data collection is very risky. Given the context, there was a need for a survey that was low-cost, fast, and technically feasible. Data collection needed to be launched and completed in a matter of days, while also ensuring the safety of the implementing teams. Convincing sampling frames had to be obtained in environments where existing data was scarce. Finally, the crisis was unfolding in four different contexts, and country-specific approaches were required that were both standardized yet adaptive.

2 The Innovation

The Rapid Emergency Response Survey (RERS) was designed with standardized survey protocols that can be implemented quickly in times of crises. It was designed as a phone survey to allow rapid access to populations at risk of famine, and can be carried out by local call-centers at low cost.⁹ During the crisis, enumerators recorded data digitally and uploaded it every day to a cloud-based server, in order to map and update data trends on a daily basis.

The questionnaire was quick to administer, yet still included a broad range of development topics that might need to be better understood during a crisis. A maximum administering time of about 20 minutes was necessary for many reasons: Phone networks often have weak connectivity, making long interviews difficult, respondents have shorter attention spans over the phone compared to face-to-face interviews, and minimizing respondent fatigue was crucial to increasing the accuracy of the data and to avoid burdening potentially stressed respondents. However, the questionnaire must also provide a wide snapshot of the population’s conditions, investigating a comprehensive set of topics including education, livelihoods, health, market access, food security, and water, in order to identify which have been most affected by the crisis and to inform a response.

The survey covered the mobile phone users, with a focus on areas deemed to be in ‘emergency’ or worse by the Integrated Phase Classification (IPC).¹⁰ In order to participate in the survey, households had to own a mobile phone, have network coverage, and a means to charge the phone. As such, one key limitation of mobile surveys is that it excludes households too poor to have a mobile phone , or households that are too isolated to live in areas with coverage. Despite this shortcoming, the survey allows for an immediate, ground-level assessment of challenges related to the crisis, and the survey’s results can be considered conservative estimates of how the entire population is affected, leading to insightful policy interventions.

Sampling strategies must be adaptable to local contexts. In Nigeria, an ideal starting point was to call respondents from previous surveys who represented the intended population, since phone numbers had been collected for previous waves of this survey .¹¹ This approach allowed for the comparison of RERS estimates to estimates from the existing survey, which included the non-phone-using segment of the population. Household characteristics can thus be compared between the sample from the previous survey and the RERS, so that the representativeness could be assessed. About 80% of the phone numbers called resulted in successful interviews.

In the absence of existing surveys, a comprehensive list of phone numbers disaggregated by region would provide the best sampling frame ; however, such lists are often unavailable, unreliable, or outdated. A bulk SMS to mobile phone users asking for consent to participate in a survey can provide an alternative sampling frame, from which respondents can be randomly selected. To ensure that the crisis-affected population is represented, it is crucial that any bulk SMS can geographically target crisis-affected regions: This approach was followed in Somalia. While this methodology is effective, it further compromises the representativeness of the survey by requiring respondents to reply to a text message before being interviewed. More than 65% of the numbers called resulted in successful interviews, allowing for fast execution of the survey. However, the actual number of recipients of the bulk SMS is unknown, making it difficult to calculate the percentage of SMS recipients who were interested in participating.

Random digit dialing (RDD) is a tool that randomly generates phone numbers, and can be a practical solution when phone number lists from surveys or bulk SMS dashboards are not available. This approach was followed in South Sudan and Yemen. Machine-learning algorithms can generate random-digit sequences based on a small set of verified existing numbers to create new numbers that are likely to exist. This reduces the loss of time that results from calling non-existent numbers. However, response rates are still unpredictable, especially if the survey targets specific geographic areas. On average, 10% of the numbers called resulted in successful interviews, prolonging the survey’s duration.

3 Key Results

This section describes the collected data and highlights selected trends, starting with similarities between the four countries, and followed by selective country-specific deep-dives. In Nigeria, the survey involved households located in the North-East, North-Central, and South-South zones; of these, the North-East zone includes states classified to be under the emergency phase as per the IPC. For Somalia and South Sudan, only areas declared to be in a state of emergency or worse were surveyed. In Yemen, the survey covered all regions, stratified into emergency and non-emergency areas: Non-emergency regions are sampled because they had pockets of highly food-insecure households.

The proportion of highly food-insecure households was found to be large, but varied widely between the four countries, ranging from 30% in Somalia, to around 50% in Nigeria and Yemen, to over 70% in South Sudan (Fig. 1).¹² Higher food insecurity was recorded in those countries that faced conflict during the crisis. A high incidence of conflict was reported in Nigeria, South Sudan, and Yemen, while in Somalia, the crisis was primarily due to dry agricultural seasons and a lack of resilience.

Fig. 1

(Source Author’s calculations based on RERS data)

Food security score by country

In addition to food insecurity , the populations surveyed also faced a range of developmental challenges. Livelihoods were affected in all four countries, with large portions of the populations (ranging from 31% in Nigeria to 84% in Yemen) facing a decrease in income and a change in their main source of livelihood (ranging from 13% in Nigeria to 31% in Somalia). Poor health, insufficient access to water, and low preparedness for drought were also common in all four countries. Other issues such as school attendance and livestock loss were more context-specific (Table 1).

Table 1 Developmental gaps among the famine-risk population

In Nigeria, one in five households lost income over the previous 12 months. Highly food-insecure households were more likely to experience a decrease in income than food-secure households (39 and 21% respectively; Fig. 2). Over one in three households did not store food for future use. Highly food-insecure households were most likely to not store food (39%) compared to households with low or medium food insecurity (28 and 29% respectively; Fig. 2). Early warning systems for drought preparedness and food-storage capabilities might allow for higher resilience and reduce the need for desperate coping strategies.

Fig. 2

(Source Author’s calculations based on RERS Nigeria data [Conducted with the National Bureau of Statistics of the Federal Government of Nigeria under Poverty and Conflict Monitoring Systems])

Trends in income and food storage

In Somalia, surveyed household had lost livestock and changed their employment activities. Over 30% of the Somali population owned livestock in the previous 12 months. However, among households that owned livestock, four in five faced a decrease in livestock holdings, with the primary reason being death or disease (66%). Livestock had also been depleted from being sold off (9% of livestock-owning households; Fig. 3). Economic assistance to compensate for livestock losses was clearly necessary to increase household income. The survey found that about 15% of households changed their main employment activity, with a shift away from farm-based employment (15–10% in farm labor; 20–13% in own-account farming), while the number of respondents involved in non-farm businesses increased sharply (17–28%; Fig. 3). Households responded to the drought by compensating for losses in agricultural income through shifts in the labor market.

Fig. 3

(Source Author’s calculations based on Somali RERS)

Trends in livestock losses and employment (Somalia)

In South Sudan, most households rely on markets to buy food. However, while food is generally available in markets, households often cannot afford to buy food because of high prices (Fig. 4). These high food prices are not surprising, given South Sudan’s recent period of high inflation. To improve access to food, the survey findings emphasize the importance of vouchers as opposed to food imports.

Fig. 4

(Source Author’s calculations based on RERS South Sudan)

Challenges in accessing food markets and livestock losses (South Sudan)

Almost 40% of the South Sudanese population owned livestock in the previous 12 months. Of these, almost two in three households lost livestock due to death or disease, and theft (25 and 21% respectively; Fig. 4). Livestock restoration could help livestock-based agricultural livelihoods to be regained faster.

In Yemen, food prices and in particular, and school attendance were found to be affected by the crisis. Most households used markets to access food; as such, an increase in food prices was the key challenge in accessing the market for both food-secure and food-insecure households (Fig. 5). Again, this suggests that future interventions should be based on food vouchers rather than food imports. About one in four households had not sent all their children to school regularly in the previous year, largely due to unsafe routes to school and school closures. Safety issues and school closures resulted in low school attendance for both boys and girls (Fig. 5). This underlines the detrimental effects of insecurity on future generations, and the need to restore educational infrastructure.

Fig. 5

(Source Author’s calculations based on RERS Yemen)

Challenges in accessing food markets and reasons for low school attendance (Yemen)

4 Implementation Challenges, Lessons Learned, and Next Steps

The results of the Emergency Response Survey showcased above emphasize the importance of understanding the local situation, which is context- and country-specific. A quick turnaround from survey inception to results and analysis is a key factor in the usefulness of the data to inform disaster responses. The survey was completed the quickest in Somalia (2600 interviews in ten days), and slowest in Nigeria (around 600 interviews in 25 days; Fig. 6), with the response rates and the size of the enumeration team being key factors in the speed of data collection. In Somalia, the enumeration team consisted of 25 enumerators, which was five times the size of the team in Nigeria. In South Sudan and Yemen, response rates were less than 15%, which increased the survey’s duration. However, despite these various constraints, data collection was quick enough to generate results for each country in eight weeks. The case of Somalia further demonstrates that very rapid data collection can be done with a reasonably sized team of enumerators, even in a constrained environment.

Fig. 6

(Source Author’s calculations based on RERS)

Survey duration and implementation costs

This survey methodology can be deployed rapidly while keeping costs low. Operating through country-based call centers cost roughly $50,000 per country, and the cost per interview was less than $35 in all countries except for Nigeria. The interviews were most cost-effective in Somalia ($23 per interview) and most expensive in Nigeria ($86 per interview; Fig. 6). The bulk of the costs were fixed, and thus a larger sample size drove down the cost per interview. This fixed-cost structure allowed for increasingly cheap future rounds of the survey once the call center infrastructure has been established.

Statistical infrastructure, such as a list of respondents with phone numbers, can accelerate data collection and improve the representativeness of the survey . In Nigeria, phone numbers collected as part of a nationally representative household survey were used to select respondents. Such approaches can save time in preparing the sampling frame for the survey, compared to negotiating with mobile phone providers to provide randomized lists of phone numbers or to send a bulk SMS. It can also minimize the legal implications of mobile phone-based surveys, as some countries do not allow large numbers of unsolicited phone calls. Using data from a nationally representative survey, as was the case in Nigeria, the representativeness of the collected data can be assessed by comparing respondents who participated in the phone survey with respondents who could not be reached, as well as to respondents who either did not provide or did not have a phone number.

Questionnaire design is a crucial step in preparing a survey . In the absence of quantitative data about the impact of the drought on a variety of areas, the questionnaire was designed to explore diverse developmental topics including education, livelihoods, health, remittances, prices, and market access. Survey data clearly indicated that certain topics were more seriously affected than others, warranting more detailed exploration that was impossible in the context of the initial survey. For example, in South Sudan, more than 90% of households had suffered an illness in the previous three months (Table 1); however, while the questionnaire was able to collect details about the most recent illness in the household, the module was insufficiently deep to allow specific conclusions regarding health interventions to be drawn. In hindsight, additional information on household member-specific and less recent illnesses would have been valuable. However, the design of the questionnaire traded in-depth exploration against comprehensive thematic coverage. Another finding from South Sudan was that remittances were not severely affected by the crisis; again, in hindsight, the questionnaire could have been optimized by adding questions on remittances. However, it is impossible to make these choices a priori, especially during an unfolding crisis situation.

The use of an adaptive questionnaire is a promising approach to escape this limitation. The premise is to adapt the questionnaire while collecting data. The first round of the questionnaire should cover a broad range of developmental topics, with an emphasis on preliminary questions assessing the extent to which each topic is affected. After around 500 interviews, trends from the data collection will indicate which topics warrant more exploration.¹³ A survey conducted through a call center allows for the rapid adaptation of the questionnaire, as well as splitting the sample into individually representative parts at no extra cost. Thus, the questionnaire can be adapted after every 500 interviews, with increasing levels of detail on relevant topics and subtopics (Fig. 7, colored green). Less relevant topics can be dropped to keep the duration of the interview manageable (Fig. 7, colored gray). Even saving five minutes by skipping preliminary questions on irrelevant themes can save crucial time in a 20-minute interview for more in-depth exploration of relevant topics.

Fig. 7

Proposed adaptive questionnaire design (Color figure online)

Adaptive questionnaire design fits well within the fast, low-cost survey methodology of the RERS. Enumerators can be trained on the full, detailed version of the questionnaire before data collection to allow quick adaptation of relevant and irrelevant topics (Fig. 8). The design will create systematically missing values for detailed questions in interviews conducted at the beginning of data collection, and for explorative questions later in the implementation. The random sequence of interviews, however, ensures that the missing data is not biased and, thus, can be analyzed by ignoring missing values. While this will affect standard errors, a large sample, such as the 2500 interviews in Somalia, can ensure sufficiently narrow confidence intervals, even after several adaptations of the questionnaire.

Fig. 8

Proposed timeline for adaptive questionnaire design with RERS methodology

By presenting the example of a pilot survey , this chapter provides proof-of-concept that quantitative data collection via phone surveys is feasible, cost-effective, and informative in the context of shock responses. The pilot highlighted the importance of using an effective questionnaire design, like adaptive questionnaires, to balance the need to comprehensively cover a wide range of topics with the need to collect detailed information on specific sub-topics identified as part of the survey. Although smart and innovative designs can optimize trade-offs, emergency response surveys are no substitute for face-to-face household surveys based on representative sampling frames , ensuring that everybody is included, even the poorest and most vulnerable, who might not own mobile phones . However, compromises need to be made to provide a timely response following a shock, creating a niche for emergency response surveys as presented in this chapter.

Such emergency phone surveys can be prepared and implemented at global and national levels. At the global level, an adaptive questionnaire template can be prepared before emergency situations occur. This will reduce the preparation time needed to adapt the questionnaire template to a country and a specific crisis. At the country level, the groundwork for a survey can be prepared by collecting phone numbers of potential respondents. Questions about phone numbers and the willingness to participate in future survey interviews should be included by default in nationally representative surveys at both the household and firm levels. Lists of phone numbers of respondents who are knowledgeable about specific topics can further add value and allow for more in-depth interviews. These can be obtained by reaching out to sector ministries to collect phone numbers for their staff, which may already be integrated into the HR system. The ability to call, for example, health workers or police officers across an entire country allows for many more monitoring options that would not only be relevant in emergency situations. National statistics offices often maintain such (sufficiently anonymized) phone number databases, and provide them in emergency situations. In crisis-prone countries, the establishment of a call center and, potentially, the use of continuous phone surveys can also further accelerate implementation and provide baseline data.

Emergency phone surveys can play a critical role in crisis analytics, especially if integrated with other data sources that are either recurrently collected in a country, available upon demand, or typically collected during a crisis. For example, market price data is collected in most countries, whether by national statistics offices, UN agencies, or both, and can be triangulated geographically with household interviews. Satellite images can also provide additional context specifically to household interviews, for example, by gathering information about agricultural activity or damage to physical infrastructure. Furthermore, social network data or mobile phone usage data can provide invaluable insight. However, access to such datasets often takes time, making pre-emergency agreements necessary. UN agencies have developed standard data collection methodologies for crisis situations, including the WFP’s Vulnerabilities Assessment and Mapping (VAM), and IOM’s Displacement Tracking Matrix (DTM). For a meaningful integration, the different underlying data sources should be readily available and anonymized at the micro level, including sufficiently disaggregated geographical indicators. This requires pre-emergency agreements between involved agencies, ideally at both global and national levels. This is a valuable effort for the avoidance of redundancies and the creation of new synergies. Thus, technological advancements make crisis analytics an extremely powerful tool to inform crisis responses. To harvest their full potential, more pilots must be carried out, and specific statistical infrastructure as well as collaborations and agreements between different stakeholders will be needed.