Abstract
Governments typically introduce new programs because they want to achieve some sort of positive change. While policy makers may have a clear idea of what they want to achieve, there is often less certainty about the most effective way of achieving desired policy goals or even whether achieving a policy goal is possible given current financial constraints and/or institutional arrangements. Even when desired policy goals are achieved, it may be difficult to isolate which factor (or combination of factors) had the most significant impact. This difficulty is known as the dependent variable problem.
After introducing and defining the dependent variable problem, the chapter identifies the sort of research and policy questions that are explicitly concerned with causality and the different methods or techniques that policy makers can use to increase their understanding of what factor (or combination of factors) is facilitating, or impeding, desired change. The chapter discusses the strengths and limitations of randomized controlled trials where outcomes for two groups, one of whom underwent intervention, are compared, quasi-experimental approaches, and cross-case comparisons of sequential events. The chapter makes the point that, in the real world, causality is often complex, nonlinear, and/or influenced by macro-level factors beyond the control of policy makers responsible for a particular policy or program. For this reason, policy makers need to think carefully about the particular combination of techniques capable of generating the sort of information that will help them understand causal linkages.
Keywords
This is a preview of subscription content, log in via an institution.
References
American Economic Association (AEA n.d.). https://www.socialscienceregistry.org
Bastagli, F. 2008. Conditionality in public policy targeted to the poor: Promoting resilience? Social Policy & Society 8 (1): 127–140.
Bennett, A., and C. Elman. 2006. Complex causal relations and case study methods: The example of path dependence. Political Analysis 14 (3): 250–267.
Braithwaite, V. 1995. Games of engagement: Postures within the regulatory community. Law and Policy 17 (3): 225–255.
Braithwaite, J. (2016). Restorative justice and responsive regulation: The question of evidence. RegNet working paper no. 51. School of Regulation and Global Governance, https://johnbriathwaite.com/wp-content/uploads/2016/10/SSRN_2016BraithwaiteJ-revised-51.pdf. Accessed 2 Apr 2019.
———. 2017. Types of responsiveness. In Regulatory theory: Foundations and applications, ed. P. Drahos, 117–132. Canberra: ANU Press.
Brown, H. 2013. Racialized conflict and policy spillover effects: The role of race in the contemporary U.S. welfare state. American Journal of Sociology 119 (2): 394–443.
Cairney, P. 2013. What is evolutionary theory and how does it inform policy studies? Policy & Politics 41 (2): 279–298.
Capano, G. 2009. Understanding policy as an epistemological and theoretical problem. Journal of Comparative Policy Analysis: Research and Practice 11 (1): 7–31.
Capano, G., and M. Howlett. 2009. Introduction: The determinants of policy change; Advancing the debate. Journal of Comparative Policy Analysis: Research and Practice 11 (1): 1–5.
Crépon, B., E. Duflo, M. Gurgand, R. Rathelot, and P. Zamora. 2013. Do labor market policies have displacement effects? Evidence from a clustered randomized experiment. The Quarterly Journal of Economics 128 (2): 531–580.
Dupuis, J., and R. Biesbroek. 2013. Comparing apples and oranges: The dependent variable problem in comparing and evaluating climate change adaptation policies. Global Environmental Change 23 (6): 1476–1487.
Eisenstadt, N. 2011. Providing a sure start: How government discovered early childhood. Bristol: The Policy Press.
Farrow, K., S. Hurley, and J. Sturrock. 2015. Grand Ali29 August 2019bis: How declining public sector capability affects services for the disadvantaged. cpd.org.au/wp-content/uploads/2015/12/Grand-Albis-Final.pdf
Frey, B.S., and A. Stutzer. 2005. Beyond outcomes: Measuring procedural utility. Oxford Economic Papers 57 (1): 90–111.
Gerring, J. 2006. Single-outcome studies: A methodological primer. International Sociology 21 (5): 707–734.
Haydu, J. 1998. Making use of the past: Time periods as cases to compare and as sequences of problem solving. American Journal of Sociology 51 (3): 688–701.
Kaplan, T.J. 1986. The narrative structure of policy analysis. Journal of Policy Analysis and Management 5 (4): 761–778.
McCarney, R., J. Warner, S. Iliffe, R. van Haselen, M. Griffin, and P. Fisher. 2007. The Hawthorn Effect: A randomised controlled trial. BMC Medical Research Methodology. https://doi.org/10.1186/1471-2288-7-30.
McKenzie, D. 2012. A pre-analysis plan checklist. http://blogs.worldbank.org/impactevaluations/a-pre-analysis-plan-checklist. Accessed 29 Aug 2019.
Mossberger, K., and H. Wolman. 2003. Policy transfer as a form of prospective policy evaluation: Challenges and recommendations. Public Administration Review 63 (4): 428–440.
Nevile, A. 2013a. Reframing rights as obligations: Implications for service users’ ability to exercise their rights. Australian Journal of Human Rights 19 (2): 147–164.
———. 2013b. The curse of accountability: Assessing relationships in the delivery of employment services. The Economic and Labour Relations Review 24 (1): 64–79.
Nevile, A., E. Malbon, A. Kay, and G. Carey. 2019. The implementation of complex social policy: Institutional layering and unintended consequences in the National Disability Insurance Scheme. Australian.
Paloyo, A.R., S. Rogan, and P. Siminski. 2016. The effect of supplemental instruction on academic performance: An encouragement design experiment. Economics of Education Review 55: 57–69.
Parker, C., and V.L. Nielsen. 2017. Compliance: 14 questions. In Regulatory theory: Foundations and applications, ed. P. Drahos, 217–232. Canberra: ANU Press.
Pearl, J., and D. Mackenzie. 2018. The book of why: The new science of cause and effect. New York: Basic Books.
Pollitt, C. 1995. Improvement strategies. In Quality improvement in European public services, ed. C. Pollitt and G. Bouckaert, 131–161. London: SAGE.
Rayner, J. 2009. Understanding policy change as a historical problem. Journal of Comparative Policy Analysis: Research and Practice 11 (1): 83–96.
Tarrow, S. 2010. The strategy of paired comparison: Toward a theory of practice. Comparative Political Studies 43 (2): 230–259.
Torgerson, D., and C. Togerson. 2008. Designing randomised trials in health, education and the social sciences: An introduction. Houndmills. Palgrave Macmillan.
van der Heijden, J., and J. Kuhlmann. 2017. Studying incremental institutional change: A systematic critical meta-review of the literature from 2005–2015. Policy Studies Journal 87 (3): 509–538.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Section Editor information
Rights and permissions
Copyright information
© 2021 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this entry
Cite this entry
Nevile, A., Biddle, N. (2021). The “Dependent Variable Problem”: How Do We Know What Caused Desired Change?. In: Sullivan, H., Dickinson, H., Henderson, H. (eds) The Palgrave Handbook of the Public Servant. Palgrave Macmillan, Cham. https://doi.org/10.1007/978-3-030-29980-4_29
Download citation
DOI: https://doi.org/10.1007/978-3-030-29980-4_29
Published:
Publisher Name: Palgrave Macmillan, Cham
Print ISBN: 978-3-030-29979-8
Online ISBN: 978-3-030-29980-4
eBook Packages: Political Science and International StudiesReference Module Humanities and Social SciencesReference Module Business, Economics and Social Sciences