1 Introduction

Mali is a sparsely populated, predominantly desert country with an undiversified economy. It is particularly vulnerable to commodity price fluctuations (gold is a major export), and to the consequences of climate change. Mali has a population of 15 million, 10% of whom are living in the three northern regions of Gao, Kidal, and Timbuktu. High population growth rates, low agricultural productivity, and weather shocks fuel food insecurity, poverty, and instability. The delivery of services within this large territory is challenging and affects geographic equity and social cohesion.

Mali’s political and security situation became volatile in 2012 when the northern regions were occupied by rebel and criminal groups who threatened to take over the country in a coup. These events led to a coup and the deployment of French-led military forces in January 2013. In July 2013, the United Nations Multidimensional Integrated Stabilization Mission in Mali (MINUSMA) took over security measures from the French forces. Constitutional order was restored when two-round presidential elections were held in July and August 2013, with a turnout of 49 and 46% of eligible voters, respectively.

A Peace Accord between the government and two rebel coalitions, known as the “Platform” and “Coordination” groups, was signed by the government and the Platform group on 15 May 2015, and by the government and the Coordination group on 20 June 2015. However, its implementation remains challenging. Security, which is critical to ensuring economic recovery and poverty reduction, remains fragile, with continuing attacks on the UN forces and the Malian army by jihadist groups in the north. There are also attacks on civilians in Bamako, the most recent of which targeted the Radisson Blu Hotel in November 2015, the Nord-Sud Azalai Hotel in March 2016, and a holiday resort near Bamako in June 2017.

Following the presidential elections, a Mali donor conference was organized in Belgium. At the conference, the international community confirmed its continued support, and aid flows, which had declined following the coup, resumed. Following the conference, development partners including the World Bank started to prepare new projects, many focusing on the still insecure northern part of the country. With this refreshed engagement came an increased commitment to project performance.

Information on project implementation is typically captured by project monitoring systems. These monitoring systems track progress but are also expected to flag potential shortcomings or problems. In practice, most monitoring systems do not act as independent rapporteurs, but focus on producing progress indicators for midterm and final reviews. Even this reduced role is not always well-executed and reports often come too late to help projects improve. Supervision missions offer another source of information on project performance, but there is a limit to the information such missions obtain. After all, why show a team of visiting project supervisors an activity that is facing problems?

Less biased information about the effectiveness of projects comes from evaluations by non-project staff. Typically, these take the form of randomized control trials, or large-scale surveys, such as the Service Delivery Indicator (SDI) Surveys, which measure the quality of service delivery in health and education, or Public Expenditure Tracking Surveys (PETS). The challenge of these data-intensive approaches is not their reliability, but that they are expensive and therefore not able to be repeated frequently. Moreover, they are time-consuming and rarely deliver quick results; sometimes, results only become available after the project has finished.

2 The Innovation

For project managers who want to use monitoring data, information obtained through iterative feedback loops is to be preferred over data from infrequent surveys. After all, if the aim is to improve outcomes, it is important not only to establish what a project’s problems are, but also to act to address them and to assess whether the action resolved the issue. The idea behind an iterative feedback loop is to allow a project team to learn lessons from a project’s shortcomings and improve its performance. Once action has been taken, one must assess whether the identified deficiencies have been resolved. To allow for regular feedback, data collection should be affordable and focused. Reliable, regular, and inexpensive data are the ideal (see also Box 1).

To meet these requirements, a beneficiary feedback system was designed that is light and low-cost, focused on a select set of issues, and implemented by an independent entity with no stake in the outcomes of the project. This approach has been labeled: Iterative Beneficiary Monitoring or IBM. By keeping data collection focused (few research questions and small samples), IBM facilitates timely data analysis and the rapid preparation of reports. By keeping data collection costs down, frequent data collection becomes feasible. The IBM approach reflects a major difference from more typical monitoring systems that collect the bulk of their information at the beginning, in the middle, and at the end of the project. The approach fits within the thinking on adaptive project design as well as complexity, approaches to project design and implementation that stress the importance of context, collecting feedback and demonstrating flexibility in design and implementation.Footnote 1

Box 1 Beneficiary monitoring is not a new concept, but light monitoring is

IBM is not the first time projects systematically seek feedback from beneficiaries during project implementation. A 2002 social development paper presented lessons learned from Beneficiary Assessments that aimed to amplify the voice of the people for whom development is intended. In the report, Beneficiary Assessment is presented as a tool for managers who wish to improve the quality of development operations. The approach, which is rarely used today, has been applied to over 300 projects in 60 countries; it is qualitative, and relies on a combination of direct observation, conversational interviews, and participant observation.

This qualitative approach differs from IBM in important ways. IBM samples tend to be much smaller, its reports shorter, more factual, and produced within weeks of data collection. The cost of the qualitative approach is also much higher. Where IBM costs never more than $5000 per round of data collection, the average cost of qualitative Beneficiary Assessments was $40,000 per round of data collection. For these reasons the qualitative approach is less suited to serve as an iterative feedback loop that is repeated regularly.

Source L. F. Salmen (2002).

How does IBM work in practice? An iterative feedback loop begins with gaining intimate knowledge of a project. This implies discussions with the project manager and those responsible for project implementation (such as the Project Implementation Unit) to establish trust and to identify issues in need of investigation.Footnote 2 Project staff are in an excellent position to reflect on the factors that may be hampering successful project implementation.

Core project documents need to be read, starting with the Project Appraisal Document (i.e. the document describing the project, its objectives, and modes of implementation). The Implementation Manual is another important document because it describes how the project is expected to operate in practice. It can also be invaluable for identifying sources of information or standards that can be used to assess the project. Supervision reports, aide memoires, and mission reports help to identify issues of potential concern. Project familiarization is time-consuming and, in itself, an iterative process. It is indispensable if an effective approach to data collection is to be designed, and because it builds trust with the project staff, laying the groundwork for follow-up once results have been produced.

Collecting information from beneficiaries and others at the front-line of service provision (such as staff working in schools, clinics, or farmers’ organizations) is at the heart of the iterative feedback approach. Their experience with the project is what ultimately matters. IBM thus focuses on obtaining direct feedback from these beneficiaries. Identifying what information to obtain from whom is an important step in the design of a feedback system. For instance, in a project offering meals to students, the perspective of parents and guardians is critical because they can ascertain that children have eaten. Students can give their views on the quantity and quality of the food and how often they receive it. Head-teachers can confirm whether the money to buy the food arrives on time, Parent Teacher Associations can explain whether procedures are being followed, and those who prepare the food are well-placed to report whether the money they receive is sufficient.

It is thus critical that the iterative system is developed in close collaboration with project managers. They need to provide access to project files (including beneficiary databases needed for sampling) and to validate the methodology and instruments for data collection. If this is not carefully done, project managers may eventually contest the validity of the results, and little follow-up can be expected. While the monitoring team will need to collaborate closely with project management, the team will also need to ensure that the identity of respondents and the locations where data are collected are kept confidential. If this is not done, there is a risk that the results will be biased.

It is important to keep the data collection exercise light, and to resist the temptation to collect more information than is strictly necessary. A project manager’s capacity is often constrained, and a project team can only handle so many issues at a time. Given that the approach is iterative, new issues can be addressed in subsequent rounds of data collection and not all issues need not be investigated in the first iteration. This gives the project team the option to prioritize what is most critical or most easily addressed. By keeping the data collection exercise light, the design of data collection instruments is relatively straightforward. Nonetheless, validation of the data collection instruments by project management remains an essential step. This includes pre-testing in a real-life setting and discussing the instruments with key project staff to assure that the right issues are captured in an appropriate way (Fig. 1).

Fig. 1
figure 1

Five steps of the IBM approach

Full size image

The design phase of the iterative approach is typically the most time-consuming phase, and hence, the most resource intensive. Rapport must be built with project staff and analysts need to familiarize themselves with the details of the project and develop, discuss, and test data-collection instruments and approaches. In comparison, data collection itself is relatively inexpensive. The “golden rule” of IBM is that each round of data collection should cost less than $5000. This is an arbitrary number which is kept deliberately small to force IBM designers to focus on key issues and affordable samples. Given this cost structure, the iterative feedback loop differs fundamentally from typical survey exercises, where data collection is the costliest part of the process. Keeping data collection costs low is of primordial importance to the success of IBM, because in its absence, frequent data collection would not be affordable and its iterative character lost.

Data are typically collected by enumerators specifically hired and trained for the task. Data can be collected using face-to-face interviews, but due to the high transportation costs of survey data collection, samples need to be kept to a minimum. This need not be a problem. When project-related issues are widespread, or when standards or deadlines must be met (as set out in the Implementation or Operations Manual), a small number of deviations may pinpoint a problem. Irrespective of sample size, attention needs to be paid to the sample design to ensure that the results are representative; this implies identifying a database from which the sample can be drawn. This is usually not a problem, as most projects maintain a database of beneficiaries. Additional decisions may also have to be made to keep costs down, but these should always be discussed with project staff, to ensure that such decisions are acceptable. For instance, it may be proposed to sample only from one small geographic area. This might be acceptable, for instance, if this area reflects an upper bound, meaning that the effects of any of the project’s shortcomings are likely to be worse in other areas. For example, if it takes a long time to transfer money to schools close to the capital, then it is plausible to assume that the situation is worse in more remote areas.

Figure 2 illustrates a case in Tanzania, generated as a precursor to IBM by one of the authors. It shows how a small number of water kiosks (24 observations), drawn randomly from a database of all water kiosks, already shows that official tariffs set by the regulator are ignored.

Fig. 2
figure 2

(Source Uwazi 2010)

Small samples may suffice to uncover problems

Full size image

Technology can be used to enhance efficiency and reduce cost. If projects collect phone numbers of beneficiaries, information can be collected rapidly and in a cost-effective manner by enumerators who call beneficiaries on their mobile phones (see Chapters 2 and 3 on data collection using mobile phone interviews). This allows for larger samples while remaining within the $5000 data collection budget and is particularly important in a context of insecurity, or when the population may be hostile to authorities and their activities. Mobile phone-based data collection is also a solution when beneficiaries are mobile, as is the case for displaced populations or nomads (Chapter 4). Because collecting data over the phone is inexpensive, collecting phone numbers of beneficiaries simplifies the creation of an iterative feedback loop.

Box 2 How IBM compares to project monitoring

Iterative beneficiary monitoring is an agile, inexpensive way to obtain feedback on project implementation. IBM can be considered a complement to project monitoring in the following ways:

First, while traditional project monitoring is used to continuously assess overall implementation progress and tends to produce voluminous progress reports at fixed points in time, IBM is demand-driven, produces short reports, can be repeated as often as is needed and is focused on diagnosing specific barriers to effective implementation.

Second, project monitoring provides progress reports to the project manager, while IBM reports to the person responsible for the project in the donor organization. IBM thus functions as an independent check on project monitoring systems, much in the same way that financial audits serve as an independent check on companies’ regular financial reports. Within the World Bank, IBM is carried out by non-project staff, who do not bear responsibility for supervising the project. Though IBM has never been applied in this manner, it could be viewed as means to assess the ability of an MIS system to identify pertinent issues. By engaging non-project staff, project teams tend to benefit from a fresh perspective that helps teams improve, even in well-established projects.

Third, relative to a field supervision mission by the project lead, IBM is project supervision “on steroids” as IBM obtains feedback from a much larger sample of beneficiaries than could possibly be covered by a supervision mission visiting two or three project sites. When IBM goes to project sites, it typically visits some 20–30 sites. When beneficiaries are interviewed by phone, sample sizes lie between a couple of hundred and one thousand. IBM also collects data from randomly selected activities, hence avoiding selection bias.

Once collected, data are analyzed and offered as feedback to project managers and project leaders. Given that the dataset is kept small, analysis is rapid. IBM reports are specific, factual and short, and typically less than ten pages. As reports are likely to reveal a project’s shortcomings, care is taken to ensure the highest standards of accuracy. Where World Bank projects are concerned, management is copied as a matter of procedure. Often, results will also be discussed with those responsible for the project in the client government. These authorities may request that the project team take the steps required to address the issues but rarely is this needed as project teams tend to be responsive to IBM findings. Another round of data collection will follow sometime later (generally after a few months), with the aim of measuring improvements and, to assess whether new issues may have arisen. The reporting process is the same as for the earlier round. This cycle is repeated on a regular basis until the end of the project.

Reports remain internal, intended for use by the client government, project managers, and supervisors. Disclosing negative facts publicly could have unintended negative consequences, and as is not an objective of IBM.Footnote 3 The experience with water price monitoring (as shown in Fig. 2) is illustrative in this regard. Light monitoring principles were applied, but instead of working to address the issue with the regulator, those in charge of the monitoring process sought media attention. Public pressure and parliamentary questions led to corrective action, but these were of an ad hoc and symbolic in nature. Certain responses even aggravated the situation, as some water kiosks were closed because they had been overcharging, leaving those dependent on water kiosks with fewer options than they had previously. After the initial media interest, there was no systematic follow-up, and overcharging continued unabated.

3 Key Results

The IBM approach was first introduced in Mali, offering feedback to an education project (school feeding), an agriculture project (electronic subsidies or e-vouchers), a social protection project (cash transfer), and also to activities managed by the Malian Authorities such as the provision of health insurance to the extreme poor and the functionality of newly established land commissions. In the case of school feeding, the project supervisor expressed concern that only part of the money allocated to this activity was being used. To explore this issue, a clear division of tasks was agreed: the team member from the Poverty Practice of the World Bank would take charge of all issues related to data collection and reporting, while the supervisor from the Education Practice of the Bank would facilitate all interactions with the Ministry of Education and the Project Implementation Unit. The collaboration was smooth, and after some introductory and follow-up meetings, the National Centre of School Canteens at the Ministry of National Education shared the database of schools benefiting from the school feeding program. This database was used to draw a sample of beneficiary schools. To assure ownership and accuracy, officials from the Ministry and the Centre actively participated in the preparation and validation of the survey methodology and tools but were not provided the list of schools included in the sample.

The first round collected data in 20 randomly selected schools. Two enumerators were trained and traveled to each of the schools to carry out face-to-face interviews with head teachers, managers of school canteens, and a subsample of parents. It cost less than US$5000 to complete the data collection exercise, and the report took little time to prepare, as information had only been collected on a limited set of issues. Officials from the National Centre of School Canteens were informed about the main results together with the project manager. Results were shared with the Country Director and the Minister of National Education.

Results showed that it took more than four months to transfer money from the Ministry of National Education to schools. Consequently, much of the money for school feeding arrived after the school year had started, jeopardizing one of the objectives of the program, namely increasing enrolment rates. Moreover, the amount of money sent to schools was insufficient to feed all students during the envisaged period, and some schools were forced to offer food less than five days per week, reducing the incentive for students to remain in school (Fig. 3).

Fig. 3
figure 3

(Source Authors’ calculations based on IBM data)

Regular follow-up improved school feeding performance

Full size image

Transfers were expected every quarter, but their real frequency was lower. Also, procedures as described in the operations manual were not followed exactly. Amounts transferred were supposed to reflect enrolment rates for instance, but often they deviated and were much higher or lower than they should have been.

The IBM report was discussed with the project staff, and the Minister of National Education, who followed-up by sending letters to project officials demanding improvements. Additional supervision missions were initiated, and school enrolment information was updated to ensure the correct amounts were transferred.

Six months later, a second round of data was collected, this time in 30 schools randomly selected from a list that excluded the schools that have been interviewed in the first round. Results showed it now took much less time for money to arrive at the schools. Most schools received close to the exact amount that was expected, and all money that was disbursed by the Ministry arrived in the schools. The second report thus showed significant improvements in project implementation, through certain issues persisted (Table 1).

Table 1 Iterative feedback approach for a school feeding project in Mali
Full size table

The success of the use of this data collection approach in the education sector aroused interest from other project supervisors. The approach was then introduced to an agriculture project that distributed subsidies in the insecure north of the country using electronic vouchers (e-vouchers). E-voucher beneficiaries had been registered and their phone numbers and core characteristics captured in a database. This information was used to send them vouchers by text message. Upon receipt of their vouchers, beneficiaries could buy specific products, typically fertilizers and livestock products, at designated retail locations at a discount.

Project management expressed concern about the limited uptake of the subsidies. A supervision mission had reported that during the first wave only a fraction of the beneficiaries who had been sent an e-voucher had collected their products, even when they were free of charge. The suggestion was that there might be problems with the distribution system, or that there was a lack of interest among the beneficiaries in the products on offer. Identifying the exact nature of the problems was clearly important for the success of the project.

Because the project had a database with phone numbers of its beneficiaries, and as the areas of intervention were insecure, the team opted to use telephone interviews for data collection. Project management shared its database and participated in working sessions to validate the methodology and survey instruments and to select a representative sample of 100 beneficiaries who were interviewed by phone. Inspection of the shared database revealed the presence of many duplicate phone numbers, allocated to different people in different villages. While the procedural manual permits different beneficiaries to use the same phone number, as not everyone owns a phone, they would be expected to live in the same village. However, the duplicates identified in the database were not in the same location. After four attempts to call a respondent, only 40% had been reached, raising questions about network coverage in villages where beneficiaries live, the accuracy of the phone numbers in the database, and/or the location of beneficiaries, as some people might have left their initial locations due to insecurity.

The initial results showed that all the beneficiaries who had received e-vouchers had collected their products, suggesting that the low uptake of the products was not for a lack of interest. As a significant proportion of beneficiaries could not be reached by phone, it was not possible to know whether all the e-vouchers had been successfully delivered. It seemed plausible that, like the failed telephone interviews, many e-vouchers had failed to reach their intended beneficiaries, suggesting a communication problem between the e-voucher platform and the beneficiaries. Finally, many beneficiaries indicated not having received the full quantity of (free) products indicated on their vouchers. Nor had they been compensated for any items not received.

Following these results, the Bank’s team contacted the project and telecom providers to discuss the findings and to address certain issues, including the number of duplicate phone numbers in the database, the inability to send a high number of text messages per second, and the absence of a “text message received” message.

A second round of data collection was carried out five months later. The sample was increased, as there was a need to assess whether the approach was working and how well it worked, as the successful implementation of the e-voucher scheme was a condition for a budget support operation to the government of Mali. More information was needed than a simple understanding of whether the approach was working, and evidence had to be collected on the percentage of beneficiaries in different districts, and the application of targeting criteria. The second round showed that the management of the system had improved. The database was cleaner, more respondents could be reached, more messages could be sent per second, and receipt messages were now received. However, the results also showed that the roll-out of the scheme still left much to be desired. Not all the agreed zones were covered, and e-vouchers had been sent late, three months after the start of the agricultural season. Moreover, e-vouchers were distributed for fertilizers that could not be used given the stage of the growing season. Finally, fertilizer suppliers turned out to have been selected using a non-competitive method. These findings led to high-level discussions between Work Bank management and the Malian authorities (Table 2).

Table 2 Iterative feedback for a project distributing agricultural inputs using electronic vouchers
Full size table

IBM, because it collects evidence directly from beneficiaries, has proved to be effective at monitoring gender outcomes of projects. In a number of instances, pertinent and concerning gender biases were uncovered. Beneficiaries of a cash transfer program turned out to be mostly men, as were the beneficiaries of the e-voucher program. Land commissions lacked almost any female members (Fig. 4). The adverse gender results uncovered by IBM were not the consequence of bad intentions. Projects were often designed with gender in mind, and in some instances, even employed gender specialists. Invariably, project staff responded positively to the findings when they received them and corrective actions followed. In the latest iteration of IBM, approaches to asking sensitive questions (discussed in Chapter 11) are used to assess from project beneficiaries whether Gender Based Violence might be in issue. Particularly for infrastructure projects in fragile or remote settings this is at times a concern.

Fig. 4
figure 4

(Source Hoogeveen et al. 2018)

Selected gender outcomes uncovered by different IBM activities

Full size image

4 Implementation Challenges, Lessons Learned, and Next Steps

IBM’s iterative feedback approach is relatively straightforward, but applying it successfully requires care. Build a good rapport with a project team is critical, and nobody likes to receive negative feedback, although this is precisely what an iterative feedback system often does. Confidentiality, good relations with project staff and the government, and agreement on the shared objectives of the monitoring process are essential. Once it is evident that the objectives of the IBM team are aligned with those of the people responsible for project implementation, reticence typically disappears.

Integration of an iterative monitoring approach at the project design stage has the benefit of being able to identify possibilities for beneficiary monitoring early on. Small changes in project design or in the procedural manual can greatly facilitate iterative monitoring. For instance, it makes a difference when procedural manuals stipulate that phone numbers and core characteristics of beneficiaries need to be captured in an electronic database that can be accessed for sampling and (anonymized) monitoring. It also makes a major difference when a procedural manual stipulates that certain benefits need to be distributed by a certain date, as this then offers a clear point in time when progress toward project objectives can be measured.

Even if an iterative monitoring approach is only designed during the project implementation phase, ways can be found to make follow-up monitoring easier. Registering the phone numbers of respondents in face-to-face interviews allows for easy follow-up. Indeed, during each round of the school feeding IBM exercise, phone numbers of respondents (canteen managers, head teachers, and households) were collected for future follow up. Sometimes feedback is offered spontaneously, with beneficiaries volunteering information to the project team, often by text message, about instances when the money for school feeding was exhausted before the expected date, about whether or not the money arrived on time, or about other issues affecting the functioning of the canteen. When such information is received and deemed relevant, the project team can use the phone numbers of other beneficiaries to verify whether what has been reported is a unique case, or an indicator of a more generalized problem.Footnote 4

Another issue for consideration is who should conduct the monitoring. Unsatisfactory results with existing monitoring systems suggest that much is to be said for monitoring by an independent third party. In Mali, staff from the Poverty Practice were responsible for data collection, while staff from the Education respectively Agricultural Practices who were responsible for project implementation, facilitated dialogue with project staff. Working with staff from the Poverty Practice had major advantages. Its micro-economists are experienced in sampling, designing instruments for data collection, training enumerators, and executing primary data collection activities, as well as in data analysis and reporting. Moreover, its staff is familiar with prevailing operating procedures but does not bear responsibility for the success or failure of a project. This facilitates giving independent, unfiltered feedback.

Local presence is another important element for success. Presence facilitates building trust with the project teams and an understanding of how the project operates, and makes it much easier to have discussions about results and corrective actions. Presence close to the location of implementation also increases responsiveness, which is important when issues need to be identified and addressed quickly: after all, lost days cannot be made up, missed meals cannot be replaced, and agricultural inputs distributed late are of little use to farmers.

Familiarity with project procedures and staff facilitates the design of an iterative loop, and as such, outsourcing the approach in the same way as financial audits are outsourced is likely to be a challenge. An intermediate approach, however, could work. Design of instruments and reporting could be left to staff familiar with household survey design and analysis, and dialogue with the client left to those responsible for the project, while data collection could be outsourced. Such an institutional set-up underscores the respective responsibilities of the recipient government, those responsible for project implementation, for project supervision, and for offering beneficiary feedback. It assures a separation of roles which helps avoid reporting bias.