Skip to main content

When do performance assessments influence policy behavior? Micro-evidence from the 2014 Reform Efforts Survey

Abstract

Sovereign governments, multilateral institutions, and non-governmental organizations have created a wide array of tools to assess the policy performance of developing countries, including conditional aid and debt relief programs, blacklists, watch lists, performance-based organizational accession procedures, cross-country benchmarking exercises, and country-specific diagnostics. Scholars and policymakers generally agree that these types of performance assessments can influence the policy priorities and actions of public sector decision-makers in low-income and middle-income countries. However, there is little systematic evidence about the specific conditions under which performance assessments instigate changes in policy behavior. There is also a lack of understanding about the causal mechanisms through which different types of performance assessments facilitate policy changes. We seek to close this evidence gap by leveraging a survey of 1784 government officials that provides comparative data on the agenda-setting influence and reform design influence of more than 100 performance assessments in 123 low-income and middle-income countries. Using a multilevel linear model that accounts for the hierarchical structure of the survey data, we find robust evidence that performance assessments yield greater policy influence when they make an explicit comparison of government performance across countries and allow assessed governments to participate in the assessment process. This pattern is consistent with the theoretical argument that policymakers carefully weigh the credibility signaling benefits and policy autonomy costs of addressing performance assessment requirements.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Notes

  1. Among other things, the Saakashvili administration fired 15,000 members of the notoriously corrupt police force, dramatically increased the salaries of 10,000 public servants to combat petty corruption, reduced the time and cost of property registration by 75%, cut the corporate income tax rate from 20% to 15%, streamlined customs by reducing the number of days it takes to export from 54 to 10, and slashed the number of activities requiring a business license from 909 to 159 (Bagaudinova et al. 2007; World Bank 2012; Bendukidze and Saakhashvili 2014; Light 2014). These and other reforms resulted in major improvements on the World Bank’s Doing Business Indicators, and USAID (2009) celebrated Georgia’s rapid rise on the Doing Business indicators as evidence of “the broadest, deepest, fastest business climate reforms [undertaken by] any country in the last 50 years.”

  2. Several studies indicate that the Doing Business Report directly inspired and guided the design and implementation of business environment reforms by the Saakashvili administration (IEG 2008: 46; IEG 2009: 36; USAID 2009; Schueth 2011: 64). Evidence from the 2014 Reform Efforts Survey also suggests that the Doing Business Report exerted outsized influence on the policy priorities and plans of Georgian officials during the Saakashvili administration (see Figure I-2).

  3. During the same period of time, the Saakashvilli administration participated in various IMF- and World Bank-monitored reform programs. It signaled its interest in joining the EU and deepened its ties with EU by agreeing to a European Neighborhood Policy Action Plan in 2006 and an Association Agreement in 2010 (Di Puppo 2014). It also made clear that it hoped to join NATO and was willing to adjust its policies to comply with NATO’s expectations. Many rationalist and constructivist IR scholars would interpret such actions as efforts to signal commitment and credibility to external actors and thereby unlock material and reputational rewards (Chwieroth 2013; Girod and Tobin 2016; Buntaine et al. 2017). Yet curiously, the Georgian government did not choose to emphasize any of these efforts in their publicity campaign. They instead focused almost exclusively on the country’s performance vis-à-vis its competitors and peers, as measured by World Bank’s Doing Business Report. This strategic choice calls attention to two closely-related puzzles that we try to solve in this article: why are cross-country PAs increasingly popular credibility signaling devices, and when and why do countries use cross-country PAs to guide their reform priorities and plans?

  4. Here we use the term “performance assessments” to refer to discrete, codified, and regularized assessments of government performance that are conducted by one or more third-party entities in one or more policy domains. We only consider PAs that are national in scope. We do not consider assessments that evaluate specific programs, projects, or subnational entities.

  5. These policy outcomes include, among other things, the passage of anti-trafficking and anti-money laundering legislation, increased rates of primary school enrollment, lower levels of public sector corruption, and higher levels of state compliance with international labor standards.

  6. Elaborating on this point, Grindle and Thomas (1991: 68) explain that “[one] needs to be centrally concerned with the attitudes, perceptions, and actions of these elites, since they [are] fundamental to any efforts to initiate and sustain reform.”

  7. To date, this literature has largely focused on establishing the existence of causal relationships between cross-country PAs and policy outcomes of interest. This initial focus on whether PAs influence final policy outcomes is understandable a first-order question is whether PAs are epiphenomenal or inconsequential. However, now that we know “assessments matter,” it has become increasingly clear that we need to need to better understand when these assessments matter and the causal processes through which they operate.

  8. States, of course, also seek to send credibility signals to foreign donors (Dreher et al. 2012; Knack 2013; Buntaine et al. 2017). However, in allocating the money and external validation that recipient countries value, donors respond primarily to factors other than recipient performance (Alesina and Dollar 2000; Easterly and Williamson 2011). Therefore, incentives to send signals of credibility, or good performance, may be weaker for donors than for investors.

  9. There is both observational and experimental evidence that foreign investors use the Doing Business Report in this way. Martin et al. (2014) find that investor sentiment, as measured by local stock market indices, improves in response to improvements on the World Bank’s Doing Business indicators. Corcoran and Gillanders (2015) provide evidence that improvements on the World Bank’s Doing Business indicators result in higher FDI inflows. Kelley et al. (2017) conducted an experiment with 150 professional investors and found that exposure to information about country rankings on World Bank’s Doing Business indicators substantially affected the willingness of investors to make investments in emerging markets. There is also evidence that cross-country PAs other than the Doing Business Report have similar information-updating impacts among donors, creditors, and investors (Öhler et al. 2012; Morse 2017).

  10. According to Acemoglu et al. (2013), “[r]eform programs in numerous countries use the Doing Business measures as an input into policymaking. Over 350 regulatory reforms across the [w]orld have been inspired by Doing Business since 2003. Some 35 governments currently maintain inter-ministerial committees considering reforms informed by Doing Business. More broadly every year many governments have undertaken reforms leading to improvements of the indicators without explicitly considering the Doing Business report. The total of reforms tracked by Doing Business annually exceeds 200 globally—over 2000 since the report tracks them. This suggests that between 15 and 20% of all such reforms were affected by the report itself” (Acemoglu et al. 2013).

  11. There are also journalistic accounts that suggest foreign investors pay attention to the Doing Business Report and consider it to be a meaningful signal. For example, in 2012, the New York Times published a story about an investment fund manager who decided to double down on investments in Egypt after monitoring the country’s performance on the World Bank’s EDBI. He said that “[o]ne of the things that attracted us to Egypt was that the World Bank had improved its ‘ease of doing business’ rating. ... From 2004 to 2007, it had the single greatest improvement in ‘ease of doing business’ of any country in the world” (Luongo 2012).

  12. The fact that many policymakers perceive cross-country PAs as imposing unreasonably high domestic policy autonomy costs is clearly evident in the debates that have surrounded the World Bank’s Doing Business Report. In 2013, the World Bank commissioned an independent panel review of this particular cross-country PA. A number of the developing countries, including China, used the opportunity of the independent panel review to advocate for the elimination of the Doing Business Report (Harding 2013). One interview that the independent panel conducted with a senior official from a World Bank borrower country nicely summarizes the “one-size-fits-all” criticism, which motivated the effort to abolish the PA: “The report is fundamentally biased. It assumes that deregulation is the more appropriate approach to follow and limited liability companies are the more appropriate business form to take. It also expects meaningful changes every year regardless of whether these changes fit into the long-term and more consistent reform program and regulatory framework of countries. Yet the regulatory function of the procedures and the standards of implementation and enforcement of regulations, which are equally if not more important, are not taken into consideration …” (World Bank 2013).

  13. Of course, while participatory PAs have knowledge and legitimacy advantages, it is also possible that involving assessed entities in assessment processes could lead to capture, manipulation, and status quo bias (Rodrik 1996; Swedlund 2013). However, our study is focused on explaining the determinants of assessment influence. On the broader issue of whether external assessments of government performance lead to negative, unintended consequences, see Parks et al. (2015).

  14. Indeed, the evidence uptake and organizational learning literatures suggests that the most influential diagnostic and advisory tools are those that involve “the assessed” in evaluation design, data collection, and evidence interpretation processes (Patton 2008; Russ-Eft and Preskill 2009).

  15. In Section 5 of this study, we analyze attributional responses from policymakers who have firsthand knowledge of the reasons why their governments recalibrated reform priorities in response to cross-country PAs and country-specific PAs. We find that respondent evaluations of the “reasons for assessment influence” in the cross-country benchmarking cohort are statistically less likely than the comparison group (respondent evaluations of PAs that did not involve cross-country benchmarking) to indicate that these assessments were “seen as respecting the [country’s] sovereign authority over final policy decisions.” A likely explanation for this empirical pattern is that government officials perceive cross-country benchmarking exercises as promoting one-size-fits-all policy solutions (Arruñada 2007; Chhotray and Hulme 2009; Goldsmith 2011; World Bank 2013).

  16. An extensive literature in international relations demonstrates that bad reputations and good reputations are economically and politically consequential (e.g., Crescenzi 2007). For example, if a state has a reputation for default on it loans, it will be significantly more difficulty accessing international capital markets and securing new loans from external sources on favorable terms (Tomz 2007).

  17. A focus on these inputs often involves changing de jure systems (e.g., rules, institutions, or procedures). Outputs or outcomes may or may not respond to the specific input-based changes that are prioritized. By way of illustration, the establishment of an anti-corruption commission is an input-based change while reductions in petty or grand corruption are outcomes that the commission might seek to achieve. In order to achieve these outcomes, the commission would need to produce a set of outputs (e.g., more criminal convictions, more seizures of ill-gotten proceeds).

  18. Reform is more likely when it becomes “the business of a wider range of insiders” (Jacoby 2006: 638) and when the public places pressure on the authorities to follow-through on their reform commitments (Abiad and Mody 2005).

  19. These survey data are valuable in that they capture the experiences and observations of public sector officials from the governments that assessment producers ultimately wish to influence. While we cannot be certain that all survey respondents provided answers on the basis of policy decisions that they themselves made or directly witnessed, the types of respondents who participated in the survey are substantially more proximate to the empirical phenomena of interest (PA influence on policy decisions) than any other population. A public opinion survey or a survey of experts would have been significantly less useful, as these populations generally know little about the inner-workings of government (Hoffmann-Lange 2007; Tansey 2007).

  20. See Online Appendix A for more information on how the sampling frame was constructed for this survey and how the survey was conducted.

  21. Most of these studies explicitly account for self-selection bias—i.e., the fact the policymakers might use PAs to validate reforms that they were going to undertake anyway.

  22. In Figure I-1, we report the average levels of reported agenda-setting influence (across our full sample of 123 countries) for the performance assessments in our sample.

  23. See Parks et al. (2015) for more details on the design and content of the 2014 Reform Efforts Survey.

  24. Readers might question whether these survey-based measures of PA influence capture any meaningful information about the actual implementation of de facto reforms. Indeed, since individual respondents evaluated PA influence based on their own personal observations and experiences, it is possible that these evaluations do a poor job of capturing the true scope of influence that a specific PA exerted on a specific reform outcome or set of reform outcomes. Therefore, to test the construct validity of dependent variable, we examined whether our survey-based measure of perceived PA influence strongly correlates with objective measures of de facto reform implementation. To do so, we selected a PA that was evaluated by a large number of our survey respondents (across many countries and years) and that itself produces objective measures of reform implementation for most of the same countries and years: the World Bank’s Doing Business Report. If it is true that our dependent variable measures what it purports to measure, one would expect respondent evaluations of the Doing Business Report’s policy influence to correlated with objective measures of business environment reform implementation from the Doing Business Index. In Online Appendix G, we confirm that respondent evaluations of the policy influence of the Doing Business Report are strongly and positively correlated with the actual implementation of business environment reforms as reflected in the actual changes in Doing Business Index. Parks and Davis (forthcoming) conduct a similar construct validity test for a survey-based measure of the policy influence of the Millennium Challenge Corporation’s eligibility criteria and country scorecards (another PA included from our sample). They too report a strong, positive correlation between respondent evaluations of PA influence and independently-generated measures of reform adoption and implementation.

  25. The fact that the sample for this study consists of 1784 government officials rather than the 3400 government officials who participated in the survey is mostly a reflection of the fact that not all government officials were knowledgeable about individual PAs. The 2014 Reform Efforts Survey was designed to only allow those government officials who were knowledgeable about individual PAs to answer questions about their perceived level of agenda-setting influence (in question 31) and reform design influence (in question 32). Tables A-1, A-2 and A-3 in Online Appendix A provide summary statistics on the characteristics of the 1784 government officials who participated in the 2014 Reform Efforts Survey and assessed at least one of the individual assessments.

  26. Survey participants were first asked (in question 29) to identify a list of PAs with which they were familiar. They selected these assessments from a fixed list (see Parks et al. 2015 and Online Appendix C for more details). They were subsequently asked (in question 30) to write-in any additional PAs that they believed might have informed the government’s reform efforts (in the respondent’s particular area of policy expertise. They then individually evaluated (in question 31) the agenda-setting influence of each of these assessments on a scale of 0–5.

  27. In Online Appendix H, we report the average agenda-setting influence score of each of the 103 assessments as evaluated by public officials who took part in the 2014 Reform Efforts Survey.

  28. All the country-level covariates (e.g., AID DEPENDENCY, INCOME, POPULATION, DEMOCRACY) are merged with the survey data based on the starting year of respondents’ evaluations. Respondents’ assessments of PAs’ influence are based on the duration of time through which they held a government position in their respective countries between 2004 and 2013. For instance, if a given respondent held a position from 2007 to 2009, the respondent’s evaluations assessed to what extent PAs yielded policy influence in those three years for which the respondent served the country in the capacity specific to her/his position.

  29. Readers will recall from section II of this article that we have no theoretical basis to expect the credibility signaling benefits of cross-country benchmarking exercises to be larger—or smaller—than their policy autonomy costs. Without detailed information about the size of the benefits of credibility signaling and the costs of foregone policy autonomy, one cannot know with any certainty whether benefits exceed costs (and the cross-country PA will exert influence) or costs exceed benefits (and the cross-country PA will fail to exert influence). Therefore, the direction of the relationship between cross-country benchmarking and policy influence is itself an empirical question to be tested.

  30. In cases when the assessed government can increase its policy autonomy (i.e., higher sovereignty and lower domestic audience costs) and reduce its uncertainty about the achievability of the net benefits (of signaling credibility to investors and donors), we expect risk-adjusted cost-benefit calculations to shift in the direction of taking policy action to address the performance requirements of a cross-country PA. This also means that when a cross-country benchmarking exercise allows for assessed governments to engage in the assessment process, it is effectively reducing the risk adjustment factor that assessed governments used in their cost-benefit calculation (of whether or not to address the performance requirements of the assessment). This risk adjustment factor, in effect, represents the imperfect information that the assessed government possesses about the likelihood of reaping the credibility signaling benefits that cross-country assessments can provide.

  31. Some of the PAs that focus exclusively on inputs include Extractive Industries Transparency Initiative (EITI) requirements, the membership requirements of the Egmont Group of Financial Intelligence Units, and the World Trade Organization’s Accession Working Party Reports and Accession Protocols.

  32. Roughly 49% of the variation in our dependent variable can be “explained” with respondent-fixed effects, which suggests that it is important that we account for respondent-level attributes.

  33. All of the regressions also control for respondents’ policy areas of focus and the types of positions that they held. However, for the sake of brevity, the regression coefficients of these respondent-level variables are not reported in our regression tables. Each respondent was asked to select his or her primary policy area of focus (from a fixed list 23 different policy areas) and specify a position that he or she held during their period of public service in a given country between 2004 and 2013 (which were subsequently recoded into a standardized set of “position types”; see Table A-2 in the Online Appendix). Respondents were then asked to evaluate each PA from the vantage point of his or her policy area of focus and country in that particular position and time period (Parks et al. 2015).

  34. Our models also reveal several other interesting results. Consistent with previous research, we find that smaller countries, poorer countries, and more aid-dependent countries are more responsive to external sources of data and analysis (Andrews 2011; Parks et al. 2015; Custer et al. 2016; Parks and Davis forthcoming; Buntaine et al. 2017). We also find some evidence for the claim that domestic political support for reform conditions assessment influence (Chassy 2014). More specifically, we find that reform support by the Office of the Head of State and/or Office of the Head of Government (EXECUTIVE SUPPORT) has a statistically significant and positive effect on assessment influence (p value<0.01). These findings attest to the centrality of the executive branch in leading and shepherding (PA-informed) reform processes. We also find that presence of reform supporters within the judicial branch (JUDICIAL SUPPORT) and domestic civil society organizations (CIVIL SUPPORT) can expand or narrow the scope of influence that performance assessments exert on reform priorities and processes. These results may point to “pass-through effect” whereby a country’s propensity to undertake reform amplifies or dampens the policy influence of PAs (Chassy 2014).

  35. These tests are described at greater length in Online Appendix F.

  36. The 2014 Reform Efforts Survey measured the influence of PAs in two different ways: agenda-setting influence and reform design influence. The former (question 31) captures whether PAs influenced specific policy priorities, whereas the latter (question 32) captures the influence of PAs on the design features of specific reform prioritized by the authorities. There is no reason to believe that our theoretical prediction only applies to the agenda-setting stage of the policymaking process. Our argument about PA influence should be equally applicable at the reform design stage of the policymaking process.

  37. Participants in this survey were only asked to answer this question for PAs that received the highest sum of ratings in question 31 (agenda-setting influence) and 32 (design influence). When they tied in the sum, respondents were asked to answer this question only for the assessment that received a higher rating in question 31. If they still tied in question 31, one of the assessments was selected at random.

  38. In question 34, respondents were asked to identify the primary reason for assessment influence from a fixed list of fifteen options. Therefore, to construct the 15 binary dependent variables that are used in these logistic regression models, we determined whether a response option was identified as the primary reason for assessment influence. If it was, the dependent variable corresponding to that response option was coded 1 and otherwise 0.

  39. These controls are included to account for characteristics of respondents or the countries where respondents live and work that might systematically bias the ways in which individuals characterize the reasons for PA influence (e.g., social desirability bias).

  40. Survey responses from those with firsthand knowledge of the reasons why their governments recalibrated reform priorities in response to cross-country PAs suggest that policymakers weigh the benefits of credibility signaling to investors against the costs of foregone policy autonomy. However, it is important to note that there is no evidence that these same considerations influence the decision-making of governing elites in developing countries when they evaluate if it is in their best interests to address the performance requirements of a country-specific PA.

  41. Earlier in the survey, respondents were informed that the term “development partners” referred to “international organizations, foreign embassies, and development finance agencies.”

  42. The primary factors that shape the cross-country allocation of aid are donor interests and recipient needs (Alesina and Dollar 2000; Easterly and Williamson 2011).

  43. In cases when donors do allocate aid on the basis of recipient performance, they often link their financial disbursements to performance on country-specific assessments rather than cross-country benchmarking assessments (Savedoff 2011).

  44. Indeed, none of the country-specific assessments in our sample (see Online Appendix C) are produced by foreign investors.

  45. See also Table G-1 in the Online Appendix.

  46. Kakha Bendukidze, the country’s State Minister of Reform Coordination, seized upon the cross-country rankings in the World Bank’s annually-generated Doing Business Report as a uniquely useful means by which the government could signal its credibility to foreign investors (Schueth 2011; De Waal 2011).

  47. They did so with support from USAID’s Georgia Business Climate Reform (GBCR) project (USAID 2009). As explained by Schueth (2011: 64), “GBCR was the preferred partner of [Minister] Bendukidze and [the Office of the State Minister on Reforms Coordination] because its mandate enabled the flexible allocation of resources in support of the government’s changing priorities. By June 2006, continuous consultation and collaboration between [the Office of the State Minister on Reforms Coordination] and GBCR led to the convergence of both organizations’ priorities on making sure that EDBI reflected the progress already being made in regulatory reform. GBCR replicated EDBI’s methodology using data that it collected independently using business process mapping methods—e.g., GBCR staff counted the number of documents and days necessary to clear goods through Georgian customs terminals, and visited the offices, filled the forms, and paid the fees necessary to register a new business. GBCR compared the results of its independent research with the indicators published for Georgia in the 2006 edition of EDBI.”

  48. According to a 2008 report from the “Doing Business [DB] creates its annual country rankings using information supplied by persons deemed knowledgeable about selected laws and regulations in each country covered. The DB team identifies individual lawyers, notaries, officials, and firms and requests that they provide information on one or more specified DB topics” (IEG 2008: 13). The same report notes that approximately 18% of the contributors to the Doing Business Report are government officials from assessed countries.

  49. We are not suggesting that the Georgian government engaged in statistical fraud or otherwise compromised the credibility of the World Bank’s assessment. Rather we are arguing that the Doing Business Report and other cross-country PAs that allow for or encourage involvement of the assessed governments in the assessment process are characterized by evaluative information traveling in two directions: from the assessment provider to the assessed government, and from the assessed government to the assessment provider.

  50. Some argue that these material consequences motivate policy responses by states (Noland 1997; Öhler et al. 2012). Others argue that rather than making a rational calculation of whether a particular type of policy adjustment serves their own interests or the national interest, public sector decision-makers take “cognitive shortcuts” and look to their neighbors or other peers for information about the most “modern” and “appropriate” types of policies (Weyland 2005; Gray 2013).

  51. In fact, the collection and publication of information about state performance can lead to “gaming” or other forms of opportunistic behavior (Sandefur and Glassman 2015; Kerner et al. 2017; Buntaine et al. 2017).

References

  • Abiad, A., & Mody, A. (2005). Financial reform: What shakes it? What shapes it? American Economic Review, 95(1), 66–88.

    Article  Google Scholar 

  • Acemoglu, D., Collier, P., Johnson, S., Klein, M., Wheeler, G. (2013). A review of doing business. Mimeo.

  • Alesina, A., & Dollar, D. (2000). Who gives foreign aid to whom and why? Journal of Economic Growth, 5(1), 33–63.

    Article  Google Scholar 

  • Andrews, M. (2011). Which organizational attributes are amenable to external reform? An empirical study of African public financial management. International Public Management Journal, 14(2), 131–156.

    Article  Google Scholar 

  • Andrews, M. (2013). The limits of institutional reform in development: Changing rules for realistic solutions. Cambridge: Cambridge University Press.

    Book  Google Scholar 

  • Arndt, C., & Oman, C. (2006). Uses and abuses of governance indicators. Paris: OECD.

    Google Scholar 

  • Arruñada, B. (2007). Pitfalls to avoid when measuring institutions: Is doing business damaging business? Journal of Comparative Economics, 35(4), 729–747.

    Article  Google Scholar 

  • Bagaudinova, S., Omran, D., Shavurov, U. (2007). Georgia: Licensing 159 activities—Not 909. In Celebrating Reform 2007. Washington DC: World Bank.

  • Bendukidze, K., Saakhashvili, M. (2014). Georgia The Most Radical Catch-Up Reforms, In Anders Aslund and Simeon Djankov, eds. The Great Rebirth: The Victory of Capitalism over Communism, Peterson Institute for International Economics, November.

  • Besley, T. (2015). Law, regulation, and the business climate: The nature and influence of the World Bank doing business project. Journal of Economic Perspectives, 29(3), 99–120.

    Article  Google Scholar 

  • Besley, T., & Burgess, R. (2002). The political economy of government responsiveness: Theory and evidence from India. The Quarterly Journal of Economics, 117(4), 1415–1451.

    Article  Google Scholar 

  • Biglaiser, G., & DeRouen, K. (2010). The effects of IMF programs on U.S. foreign direct Investment in the Developing World. Review of International Organizations, 5(1), 73–95.

    Article  Google Scholar 

  • Bisbee, J.H., Hollyer, J.R., Rosendorff, B.P., Vreeland, J.R. (2017). The millennium development goals and education: Accountability and substitution in global assessment. Mimeo.

  • Brooks, S., Cunha, R., & Mosley, L. (2015). Categories, creditworthiness and contagion: How investors’ shortcuts affect sovereign debt markets. International Studies Quarterly, 59(3), 587–601.

    Article  Google Scholar 

  • Bueno de Mesquita, B., & Smith, A. (2010). Leader survival, revolutions and the nature of government finance. American Journal of Political Science, 54, 936–950.

    Article  Google Scholar 

  • Buntaine, M., Parks, B., & Buch, B. (2017). Aiming at the wrong targets: The domestic consequences of international efforts to build institutions. International Studies Quarterly, 61(2), 471–488.

    Article  Google Scholar 

  • Busia, K. (2014). The Search for Domestic Accountability in Development Aid: What Role for the APRM in Reshaping Governance in Africa? In M. Ndulo & N. van de Walle (Eds.), Problems, Promises, and Paradoxes of Aid: Africa’s Experience (pp. 171–210). Newcastle upon Tyne: Cambridge Scholars Publishing/Institute for African Development.

    Google Scholar 

  • Büthe, T., & Milner, H. V. (2008). The politics of foreign direct investment into developing countries: Increasing FDI through international trade agreements. American Journal of Political Science, 52(October), 741–762.

    Article  Google Scholar 

  • Chassy, A. (2014). Civil Society and Development Effectiveness in Africa. In Problems, Promises, and Paradoxes of Aid: Africa’s Experience. Newcastle upon Tyne: Cambridge Scholars Publishing.

    Google Scholar 

  • Chhotray, V., & Hulme, D. (2009). Contrasting visions for aid and governance in the 21st century: The white house millennium challenge account and DFID’s drivers of change. World Development, 37(1), 36–49.

    Article  Google Scholar 

  • Chwieroth, J. (2007). Neoliberal economists and capital account liberalization in emerging markets. International Organization, 61(2), 443–463.

    Article  Google Scholar 

  • Chwieroth, J. M. (2013). The silent revolution: How the staff exercise informal governance over IMF lending. Review of International Organizations, 8(2), 265–290.

    Article  Google Scholar 

  • Corcoran, A., & Gillanders, R. (2015). Foreign direct investment and the ease of doing business. Review of World Economics, 151(1), 103–126.

    Article  Google Scholar 

  • Crescenzi, M. (2007). Reputation and international conflict. American Journal of Political Science, 51, 382–396.

    Article  Google Scholar 

  • Custer, S., Masaki, T., Sethi, T., Latourell, R., Rice, Z., & Parks, B. (2016). Governance Data: Who Uses It and Why? Williamsburg: AidData and the Governance Data Alliance.

    Google Scholar 

  • David-Barrett, E., & Okamura, K. (2016). Norm diffusion and reputation: The rise of the extractive industries transparency initiative. Governance, 29(2), 227–246.

    Article  Google Scholar 

  • De Waal, A. (2011). Georgia’s choices: Charting a future in uncertain times. Washington DC: Carnegie Endowment for International Peace.

    Google Scholar 

  • Di Puppo, L. (2014). The construction of success in anti-corruption activity in Georgia. East European Politics, 30(1), 105–122.

    Article  Google Scholar 

  • Dreher, A., & Voigt, S. (2011). Does membership in international organizations increase governments’ credibility? Testing the effects of delegating powers. Journal of Comparative Economics, 39(3), 326–348.

    Article  Google Scholar 

  • Dreher, A., Nunnenkamp, P., & Öhler, H. (2012). Why it pays for aid recipients to take note of the Millennium Challenge Corporation: Other donors do! Economics Letters, 115, 373–375.

    Article  Google Scholar 

  • Easterly, William. And Williamson, Claudia R. 2011. Rhetoric vs. reality: The best and worst of aid agency practices. World Development 39: 1930–1949.

    Article  Google Scholar 

  • Elkins, Z., Guzman, A. T., & Simmons, B. A. (2006). Competing for capital: The diffusion of bilateral investment treaties, 1960-2000. International Organization 60(4), 811–846.

  • Elkins, Z., Guzman, A. T., & Simmons, B. A. (2008). Competing for capital: The diffusion of bilateral investment treaties, 1960­2000. In The Global Diffusion of Markets and Democracy. Cambridge: Cambridge University Press.

    Google Scholar 

  • Etter, L. (2014). Can transparency reduce corruption? Evidence from firms in Peru and Mali on the impact of the extractive industries transparency initiative (EITI) on corruption. Paper presented at The Doing Business Conference at Georgetown University in Washington DC on 2014 February 20–21.

  • Extractive Industries Transparency Initiative (EITI) International Secretariat. (2015). Quantifying intangibles using the EITI in credit rating assessments. Oslo: EITI International Secretariat.

    Google Scholar 

  • Fariaa, A., & Mauro, P. (2009). Institutions and the external capital structure of countries. Journal of International Money and Finance, 28(3), 367–391.

    Article  Google Scholar 

  • Fenton Villar, P., & Papyrakis, E. (2017). Evaluating the impact of the extractive industries transparency initiative (EITI) on corruption in Zambia. The Extractive Industries and Society, 4(4), 795–805.

    Article  Google Scholar 

  • Flores, T., Lloyd, G., Nooruddin, I. (2015). The Technocratic Advantage: Does Technocratic Leadership Attract International Financing? Working Paper.

  • Gainer, M., Chan, S., & Skoet, L. (2016). Faster together: A one-stop shop for business registration in Senegal, 2006–2015. Princeton: Innovations for Successful Societies.

    Google Scholar 

  • Garriga, A. C., & Phillips, B. J. (2013). Foreign aid as a signal to investors predicting FDI in post-conflict countries. Journal of Conflict Resolution, 58(2), 280–306.

    Article  Google Scholar 

  • Girod, D. M., & Tobin, J. (2016). Take the money and run: The determinants of compliance with aid agreements. International Organization, 70(1), 209–239.

    Article  Google Scholar 

  • Goldsmith, A. A. (2011). No country left behind? Performance standards and accountability in US foreign assistance. Development Policy Review, 29, s157–s176.

    Article  Google Scholar 

  • Gray, J. (2009). International organization as a seal of approval: European Union accession and investor risk. American Journal of Political Science, 53(4), 931–949.

    Article  Google Scholar 

  • Gray, J. (2013). The company states keep. International economic organization and sovereign risk in emerging markets. Cambridge: Cambridge University Press.

    Google Scholar 

  • Gray, J., & Slapin, J. (2012). How effective are preferential trade agreements? Ask the experts. Review of International Organizations, 7(3), 309–333.

    Article  Google Scholar 

  • Gregory, N. (2012). Doing business and doing development. World Economics, 13(3), 13–26.

    Google Scholar 

  • Grindle, M. S., & Thomas, J. W. (1991). Public choices and policy change: The political economy of reform in developing countries. Baltimore: John Hopkins University Press.

    Google Scholar 

  • Hafner-Burton, E. M. (2008). Sticks and stones: Naming and shaming the human rights enforcement problem. International Organization, 62(4), 689–716.

    Article  Google Scholar 

  • Hallward-Driemeier, M., & Pritchett, L. (2015). How business is done in the developing world: Deals versus rules. Journal of Economic Perspectives, 29(3), 121–140.

    Article  Google Scholar 

  • Harding, R. (2013). China seeks to water down key World Bank report. Financial Times May 6, 2013.

  • Hoffmann-Lange, U. (2007). Methods of elite research. In R. J. Dalton & H.-D. Klingemann (Eds.), Oxford handbook of political behavior (pp. 910–927). Oxford: Oxford University Press.

    Google Scholar 

  • Hollyer, J. R., & Rosendorff, B. P. (2012). Leadership survival, regime type, policy uncertainty and PTA accession. International Studies Quarterly, 56(4), 748–764.

    Article  Google Scholar 

  • Hollyer, J. R., Rosendorff, B. P., & Vreeland, J. R. (2011). Democracy and transparency. Journal of Politics, 73(4), 1191–1205.

    Article  Google Scholar 

  • Independent Evaluation Group of The World Bank (IEG). (2008). Doing Business: An Independent Evaluation. Taking the Measure of The World Bank-IFC Doing Business Indicators. Washington D.C.: World Bank.

    Google Scholar 

  • Independent Evaluation Group of The World Bank (IEG). (2009). The World Bank in Georgia 1993–2007: Country Assistance Evaluation. Washington D.C.: World Bank.

    Google Scholar 

  • International Monetary Fund. (2009). Georgia: 2009 article IV consultation and second review under the stand-by arrangement—Staff report; press release and public information notice on the executive board discussion; and statement by the executive director for Georgia. IMF Country Report No. 09/127.

  • Jacoby, W. (2006). Inspiration, coalition, and substitution: External influences on Postcommunist transformations. World Politics, 58(4), 623–651.

    Article  Google Scholar 

  • Jensen, N. M. (2003). Democratic governance and multinational corporations: Political regimes and inflows of foreign direct investment. International Organization, 57(3), 587–616.

    Article  Google Scholar 

  • Johnson, K., Greenseid, L. O., Toal, S. A., King, J. A., Lawrenz, F., & Volkov, B. (2009). Research on Evaluation use: A review of the empirical literature from 1986 to 2005. American Journal of Evaluation, 30(3), 377–410.

    Article  Google Scholar 

  • Kapstein, E. B., & Converse, N. (2008). The fate of young democracies. Cambridge: Cambridge University Press.

    Book  Google Scholar 

  • Kelley, J. G. (2017). Scorecard diplomacy: Grading states to influence their reputation and behavior. Cambridge: Cambridge University Press.

    Book  Google Scholar 

  • Kelley, J. G., & Simmons, B. A. (2015). Politics by number: Indicators as social pressure in international relations. American Journal of Political Science, 59(1), 55–70.

    Article  Google Scholar 

  • Kelley, J.G., Simmons B.A. (2016). Introduction: Global assessment power in the twenty-first century. Paper prepared for the annual meeting of the American Political Science Association. Philadelphia, PA. September 1–4, 2016.

  • Kelley, J.G., Simmons, B.A., Doshi, R. (2017). The power of ranking: The ease of doing business Indicator as a form of social pressure. Mimeo.

  • Kerner, A., Jerven, M., & Beatty, A. (2017). Does it pay to be poor? Testing for systematically underreported GNI estimates. The Review of International Organizations, 12(1), 1–38.

    Article  Google Scholar 

  • Kijima, R., Lipscy, P.Y. (2017). The politics of international testing. Mimeo.

  • Knack, S. (2013). Aid and donor Trust in Recipient Country Systems. Journal of Development Economics, 101, 316–329.

    Article  Google Scholar 

  • Knecht, T. (2010). Paying Attention to Foreign Affairs: How Public Opinion Affects Presidential Decision Making. University Park: Penn State University Press.

    Google Scholar 

  • Koliev, F., Sommerer, T., Tallberg, J. (2017). Reporting matters: Performance assessment and compliance in the ILO. Mimeo.

  • Krasner, S. D. (2011). Foreign aid: Competing paradigms. Journal of Intervention and Statebuilding, 5(2), 123–149.

    Article  Google Scholar 

  • Kydd, A. (2000). Trust, reassurance, and cooperation. International Organization, 54(2), 325–357.

    Article  Google Scholar 

  • Lake, D., & Farris, C. J. (2014). Why international trusteeship fails: The politics of external Authority in Areas of limited statehood. Governance, 27(4), 569–587.

    Article  Google Scholar 

  • Lebovic, J. H., & Voeten, E. (2009). The cost of shame: International organizations and foreign aid in the punishing of human rights violators. Journal of Peace Research, 46(1), 79–97.

    Article  Google Scholar 

  • Lee, C.-y. (2017). Terrorism, counterterrorism aid, and Foreign Direct Investment. Foreign Policy Analysis, 13(1), 168–187.

    Article  Google Scholar 

  • Licht, A. (2010). Coming into money: The impact of foreign aid on leader survival. Journal of Conflict Resolution, 54(1), 58­87.

    Article  Google Scholar 

  • Light, M. (2014). Police reforms in the republic of Georgia: The convergence of domestic and foreign policy in an anti-corruption drive. Policing and Society, 24(3), 318–345.

    Article  Google Scholar 

  • Loayza, N., Rigolini, J., & Llorente, G. (2012). Do middle classes bring institutional reforms? World Bank policy research working paper 6015. Washington DC: World Bank.

    Google Scholar 

  • Lombardi, D., & Woods, N. (2008). The politics of influence: An analysis of IMF surveillance. Review of International Political Economy, 15(5), 709–737.

    Article  Google Scholar 

  • Luongo, M. (2012). A rush to do business in the Middle East. July 16, 2012. New York Times.

  • Martin, F., Mondragón-Vélez, C, Vila, L. (2014). Investing on Business Climate and Reform in Emerging Markets: An equity portfolio performance Evaluation. Paper presented at the doing business: Past, present and future of business regulation conference at Georgetown University’s McDonough School of Business. Washington, DC. February 20–21, 2014.

  • Mcmillan, J., & Zoido, P. (2004). How to subvert democracy: Montesinos in Peru. Journal of Economic Perspectives, 18(4), 69–92.

    Article  Google Scholar 

  • Merchant-Vega, N., & Malesky, E. (2011). A peek under the engine Hood: The methodology of subnational economic governance indices. Hague Journal on the Rule of Law, 3(2), 186–219.

    Article  Google Scholar 

  • Mitchell, L. A. (2009). Uncertain democracy: U.S. foreign policy and Georgia's rose revolution. Philadelphia: University of Pennsylvania Press.

    Book  Google Scholar 

  • Morse, J.C. (2017). Institutional monitoring and market enforcement: Generating compliance in the regime to combat terrorist financing. Mimeo.

  • Murdie, A., & Peksen, D. (2015). Women’s rights INGO shaming and the government respect for Women’s rights. Review of International Organizations, 10(1), 1–22.

    Article  Google Scholar 

  • Nelson, S.C. (2009). Endangered species: Exploring the political survival of economic policymakers. Paper presented at the Cornell PSAC seminar. April 2009.

  • Nelson, S. C. (2017). The currency of confidence: How economic beliefs shape the IMF’s relationship with its borrowers. Ithaca: Cornell University Press.

    Book  Google Scholar 

  • Noland, M. (1997). Chasing phantoms: The political economy of USTR. International Organization, 51, 365–387.

    Article  Google Scholar 

  • Öhler, H., Nunnenkamp, P., & Dreher, A. (2012). Does conditionality work? A test for an innovative US aid scheme. European Economic Review, 56, 138–153.

    Article  Google Scholar 

  • Parks, B., Davis, C. (forthcoming). When do governments trade domestic reforms for external rewards? Explaining policy responses to the Millennium Challenge Corporation’s eligibility standards. Governance..

  • Parks, B., Rice, Z., & Custer, S. (2015). Marketplace of ideas for policy change: Who do developing world leaders listen to and why? Williamsburg: AidData and the College of William and Mary.

    Google Scholar 

  • Patton, M. Q. 2008. Utilization focused evaluation. 4th Edition. Saint Paul, MN: Sage Publishing.

  • Pevehouse, J. C. (2002). With a little help from my friends? Regional organizations and the consolidation of democracy. American Journal of Political Science, 46(3), 611–626.

    Article  Google Scholar 

  • Pitlik, H., Frank, B., & Firchow, M. (2010). The demand for transparency: An empirical note. Review of International Organizations, 5, 177–195.

    Article  Google Scholar 

  • Pop-Eleches, G. (2009). From Economic Crisis to Reform: IMF Programs in Latin American and Eastern Europe. Princeton: Princeton University Press.

    Google Scholar 

  • Quiroz Flores, A., & Smith, A. (2013). Leader survival and natural disasters. British Journal of Political Science, 43(4), 821–843.

    Article  Google Scholar 

  • Roberts, J., Tellez, J. (2017). Freedom House’s scarlet letter: Assessment power through transnational pressure. Mimeo.

  • Rodrik, D. (1996). Why is there multilateral lending? In B. M. Arvin (Ed.), Annual World Bank conference on development economics. Westport: Praeger.

    Google Scholar 

  • Rodrik, D. (2014). When ideas trump interests: Preferences, worldviews, and policy innovations. Journal of Economic Perspectives, 28(1), 189–208.

    Article  Google Scholar 

  • Russ-Eft, D., & Preskill, H. (2009). Evaluation in organizations: A systematic approach to enhancing learning, performance, and change (Second ed.). Cambridge: Perseus.

  • Sandefur, J., & Glassman, A. (2015). The political economy of bad data: Evidence from African survey and administrative statistics. Journal of Development Studies, 51(2), 116–132.

    Article  Google Scholar 

  • Savedoff, W. (2011). Incentive Proliferation? Making Sense of a New Wave of Development Programs. Washington D.C.: Center for Global Development.

    Google Scholar 

  • Schueth, S. J. (2011). Assembling international competitiveness: The republic of Georgia, USAID, and the Doing Business Project. Economic Geography, 37(1), 51–77.

    Article  Google Scholar 

  • Solt, F. (2008). Economic inequality and democratic political engagement. American Journal of Political Science, 52(1), 48–60.

    Article  Google Scholar 

  • Svensson, J. (1998). Investment, property rights and political instability: Theory and evidence. European Economic Review, 42, 1317–1341.

    Article  Google Scholar 

  • Swedlund, H. (2013). The domestication of governance assessments: Evidence from the Rwandan joint governance assessment. Conflict, Security & Development, 13(4), 449–470.

    Article  Google Scholar 

  • Tansey, O. (2007). Process tracing and elite interviewing: A case for non-probability sampling. PS: Political Science and Politics, 40(4), 765–772.

    Google Scholar 

  • Tomz, M. (2007). Reputation and international cooperation: Sovereign debt across three centuries. Princeton: Princeton University Press.

    Google Scholar 

  • Ubaldi, B. (2013). Open government data: Towards empirical analysis of open government data initiatives. OECD Working Papers on Public Governance, No. 22.

  • UNDP. (2008). A users’ guide to measuring corruption. UNDP Oslo Governance Centre.

  • USAID. (2009). Georgia: Opened for business. Georgia business climate reform final report, Washington DC: USAID. Accessed at http://pdf.usaid.gov/pdf_docs/Pdacn591.pdf

  • Vreeland, J. R. (2003). The IMF and economic development. Cambridge: Cambridge University Press.

    Book  Google Scholar 

  • Weyland, K. G. (2005). Theories of policy diffusion: Lessons from Latin American pension reform. World Politics, 57(2), 262–295.

    Article  Google Scholar 

  • World Bank. (2007). Celebrating Reform 2007: Doing Business Case Studies. Washington D.C.: World Bank.

    Book  Google Scholar 

  • World Bank. (2012). Fighting corruption in public services: Chronicling Georgia’s reforms. Washington DC: World Bank.

    Book  Google Scholar 

  • World Bank Independent Review Panel. (2013). Independent panel review of the doing business report. Washington, DC: World Bank.

    Google Scholar 

  • Zabanova, Y. (2011). Georgia's post-Soviet Libertarian elite: Choosing a development model and marketing the strategy. Working paper at the conference “The Caucasus and Central Asia, twenty years after independences: Questioning the notion of 'South countries', August 25–27, 2011, Almaty, Kazakhstan.

Download references

Acknowledgments

We thank Dan Honig, Eddy Malesky, Elise Le, Steven Bernstein, Anton Strezhnev, Judith Kelley, and Beth Simmons for providing helpful comments on previous drafts of the manuscript. We also thank NORC at the University of Chicago—in particular, Renee Hendley, Kate Hobson, David Kordus, Elise Le, Aparna Ramakrishnan, Alex Rigaux, Bhanuj Soni, and Stacy Stonich—for their survey design and data collection assistance. Additionally, we owe a debt of gratitude to the nearly 6,750 development policymakers and practitioners from 126 countries who participated in the 2014 Reform Efforts Survey. Were it not for their willingness to graciously share of their experiences and insights, this study would not have been possible. This study was also made possible through generous financial support from the John Templeton Foundation and the Smith Richardson Foundation. The findings, interpretations, and conclusions expressed in this publication are those of the authors and should not be attributed in any manner to the College of William and Mary, AidData, The World Bank, its Board of Executive Directors, or the governments they represent.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Takaaki Masaki.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

ESM 1

(DOCX 863 kb)

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Masaki, T., Parks, B.C. When do performance assessments influence policy behavior? Micro-evidence from the 2014 Reform Efforts Survey. Rev Int Organ 15, 371–408 (2020). https://doi.org/10.1007/s11558-018-9342-3

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11558-018-9342-3

Keywords

  • Performance assessment
  • Sovereignty
  • Credibility
  • Reform
  • FDI
  • Elite survey