Abstract
This article examines policy experimentation in the context of policy learning in Canadian cultural policy. Despite the attraction of experimentation to encourage learning and thus improved policy outcomes, much of the literature on experimentation does not give sufficient attention to how it is operationalized in practice. Drawing from a novel dataset based on interviews with key actors, this article focuses on how the governance of experimentation impacts learning resulting from experimentation. Findings ultimately demonstrate that while learning occurred, it was constrained overall by a hierarchical, top-down approach to experimentation. Lessons from this case study can therefore be useful for both policy scholars and public administrations embarking on experimentation or other types of public sector innovation in Canada and beyond.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Availability of data and material
Not applicable.
Change history
24 September 2021
In the PDF version of the original publication, the placements for Tables 1 and 2 were incorrectly displayed. Now this has been fixed.
Notes
Cultural policies are generally agreed to be those associated with the arts (museums, visual and performing arts, heritage, and literature and poetry), and may also extend to other areas such as sport, languages, libraries, zoos, botanical gardens, fairs and festivals, folklore, and crafts (Mulcahy, 2006).
A keyword search of ‘experimentation’ and ‘public policy’ into Web of Science shows experiment used in a variety of ways, including an experimental approach to governance (Sabel and Zeitlin, 2010); to describe scientific work undertaken exogenously to government, the results of which are then used to inform policy-making (Stoker, 2010); and as a tool for learning to inform policy-making (McFadgen and Huitema, 2017), to use just three examples.
For this literature review, I first searched “policy experiments” on Web of Science. I discounted sources that did not address the governance of experiments. I then made a list of factors that were important to the realization of experiments and loosely grouped them according to the four themes.
It should be noted that most public servants did not conceptualize success this way; for them, it was mostly about completing a task. Thanks to an anonymous reviewer for drawing my attention to this.
The emphasis on evidence-based results and outcomes, as well as increased monitoring and reporting, is indicative of neoliberal approach to public sector innovation more broadly. Space constrains this discussion in this article, but future work can and should expand on this (see also Birch and Jacob, 2019).
In the end, PCH had 51 experiments, but six were created after the publication of the plan.
A full outline of the department’s experiments can be found in the Report on Experimentation (PCH 2018b).
These included the 2015 mandate letter from Trudeau to Brison; the 2016 Privy Council directive on experimentation (Privy Council 2016); the Impact and Innovation unit’s annual reports; and the Experimentation Works (2019) blog. In PCH, key documents included the 2018 Report on Experimentation (PCH 2018b); the 2018–2019 department report (PCH, 2018a); and the department’s evaluation plan (PCH, 2018c).
Using the report on experimentation (PCH, 2018b), I emailed an interview invite to all programs that had an identifiable team member listed in the government’s Electronic Services Directory of Public Servants. Of the programs I did not reach, some did not respond to requests for an interview, and for others I could not find contact information.
“Reporting” for most interviewees consisted of all administrative work outside of the experiment itself, such as documents for TBS, surveys from the Innovation and Experimentation Team (twice per year), as well as Management Accountability Framework reporting, an annual government-wide assessment of management practices and performance (Treasury Board Secretariat, 2016).
References
Ansell, C., & Bartenberger, M. (2016). Varieties of experimentalism. Ecological Economics, 130(October), 64–73.
Ansell, C., & Bartenberger, M. (2017). The diversity of experimentation in the experimenting society. In I. van de Poel, L. Asveld, & D. C. Mehos (Eds.), New perspectives on technology in society: Experimentation beyond the laboratory (pp. 36–58). Routledge.
Aucoin, P., Smith, J., & Dinsdale, G. (2004). Responsible government: Clarifying essentials, dispelling myths and exploring change. Canadian Centre for Management Development.
Baird, J., Plummer, R., Huag, C., & Huitema, D. (2014). Learning effects of interactive decision-making processes for climate change adaptation. Global Environmental Change, 27(July), 51–63.
Barber, M. (2015). How to run a government: So that citizens benefit and taxpayers don’t go crazy. Penguin.
Birch, L., & Jacob, S. (2019). “Deliverology” and evaluation: A tale of two worlds. Canadian Journal of Program Evaluation/la Revue Canadienne D’évaluation De Programme, 34(2), 303–328.
Bos, J. J., Brown, R. R., & Farrelly, M. A. (2013). A design framework for creating social learning situations. Global Environmental Change, 23(2), 398–412.
Brock, K. L. (2021). Policy labs, partners and policy effectiveness in Canada. Policy Design and Practice. Early view. Retrieved from https://www.tandfonline.com/doi/full/https://doi.org/10.1080/25741292.2021.1880063
Brodkin, E. L., & Kaufman, A. (2000). Policy experiments and poverty politics. Social Service Review, 74(4), 507–532.
Brown, H. S., & Cohen, M. J. (2019). Climate-governance entrepreneurship, higher-order learning, and sustainable consumption: The case of the state of Oregon United States. Climate Policy, 19(6), 739–755.
Campbell, D. (1969). Reforms as experiments. American Psychologist, 24(4), 409–429.
Canada, Government of (2020) 2019 Public Service Employee Survey Results for Canadian Heritage. Retrieved from https://www.tbs-sct.gc.ca/pses-saff/2019/results-resultats/bq-pq/20/org-eng.aspx
Dewey, J. (1991). The public and its problems. Swallow Press.
Dunlop, C., & Radaelli, C. (2018). The lessons of policy learning: Types, triggers, hindrances and pathologies. Policy and Politics, 46(2), 255–272.
Experimentation Works (2019, July 12) Experimentation Works Impact Report (2018–2019). Retrieved from https://medium.com/@exp_works/experimentation-works-impact-report-2018-2019-18434dd3d8fc
Farrelly, M., & Brown, R. (2011). Rethinking urban water management: Experimentation as a way forward? Global Environmental Change, 21(2), 721–732.
Heikkila, T., & Gerlak, A. (2013). Building a conceptual approach to collective learning: Lessons for public policy scholars. Policy Studies Journal, 41(3), 484–512.
Heilmann, S. (2008). Policy experimentation in China’s economic rise. Studies in Comparative International Development, 43(1), 1–26.
Heiskanen, E., Hyvönen, K., Laakso, S., Laitila, P., Matschoss, K., & Mikkonen, I. (2017). Adoption and use of low-carbon technologies: lessons from 100 finnish pilot studies, field experiments and demonstrations. Sustainability, 9(5), 1–20.
Hildén, M., Jordan, A., & Huitema, D. (2017). The search for climate change and sustainability solutions—The promise and the pitfalls of experimentation. Journal of Cleaner Production, 169, 1–7.
Howlett, M. (2012). The lessons of failure: Learning and blame avoidance in public policy-making. International Political Science Review, 33(5), 539–555.
Hoytema, J. (2019, May 23). Privy Council clerk Ian Shugart urges public service to innovate—cautiously. Ottawa Citizen. Retrieved from https://ottawacitizen.com/news/local-news/privy-council-clerk-ian-shugart-urges-public-service-to-innovate-cautiously
Huitema, D., Jordan, A., Munaretto, S., & Hilden, M. (2018). Policy experimentation: Core concepts, political dynamics, governance and impacts. Policy Sciences, 51(2), 143–159.
Huitema, D., Cornelisse, C., & Ottow, B. (2010). Is the jury still out? Toward greater insight in policy learning in participatory decision processes—The case of Dutch citizens’ juries on water management in the Rhine Basin. Ecology and Society, 15(1), 16.
Mahler, J. (1997). Influences of organizational culture on learning in public agencies. Journal of Public Administration Research and Theory, 7(4), 519–540.
McFadgen, B. (2019). Connecting policy change, experimentation, and entrepreneurs: Advancing conceptual and empirical insights. Ecology and Society, 24(1), 30.
McFadgen, B., & Huitema, D. (2017). Stimulating learning through policy experimentation: A multi-case analysis of how design influences policy learning outcomes in experiments for climate adaptation. Water, 9(9), 1–22.
Mulcahy, K. (2006). Cultural policy: Definitions and theoretical approaches. The Journal of Arts Management, Law, and Society, 35(4), 319–330.
Newig, J., Kochskämper, E., Challies, E., & Jager, N. W. (2016). Exploring governance learning: How policymakers draw on evidence, experience and intuition in designing participatory flood risk planning. Environmental Science and Policy, 55(2), 353–360.
O’Brien, D. (2010). Measuring the value of culture: A report to the Department for Culture, Media and Sport.
Paquette, J., Beauregard, D., & Gunter, C. (2017). Settler colonialism and cultural policy: The colonial foundations and refoundations of Canadian cultural policy. International Journal of Cultural Policy, 23(3), 269–284.
Patrimoine Canada/Department of Canadian Heritage (PCH) (2020) Raison d’être, mandate and role. Retrieved from https://www.canada.ca/en/canadian-heritage/corporate/mandate.html
Patrimoine Canada/Department of Canadian Heritage (PCH). (2018a). 2018–2019 Departmental Plan. Canadian Heritage/Patrimoine canadien.
Patrimoine Canada/Department of Canadian Heritage (PCH) (2018b) A Plan for Experimentation at Canadian Heritage. Ottawa: Canadian Heritage/Patrimoine canadien. Retrieved from http://publications.gc.ca/collections/collection_2018/pch/CH1-40-2018-eng.pdf
Patrimoine Canada/Department of Canadian Heritage (PCH) (2018c). Departmental Evaluation Plan 2018–2019 to 2022–2023. Retrieved from https://www.canada.ca/en/canadian-heritage/corporate/publications/plans-reports/departmental-evaluationplan-2018-19-2022-23.html
Privy Council Office (2016). Experimentation direction for deputy heads. Retrieved from https://www.canada.ca/en/innovation-hub/services/reports-resources/experimentation-direction-deputy-heads.html
Privy Council Office (2019a) Deputy Ministers Task Force on Public Sector Innovation: Report to the Clerk of the Privy Council 2018–2019. Retrieved from https://www.canada.ca/content/dam/ih-ci/documents/pdfs/DM-TF-PSI-Report-AUG08.pdf
Privy Council Office (2019b) 2018–2019 Impact & Innovation Unit Annual Report. Retrieved from https://www.canada.ca/content/dam/ih-ci/documents/pdfs/rpt/impact-and-Innovation-Unit-2018-19-annual-report.pdf
Radaelli, C. (2003). The open method of coordination: A new governance architecture for the European Union? Swedish Institute for European Policy Studies (SIEPS), Stockholm.
Rocle, N., & Salles, D. (2018). “Pioneers but not guinea pigs”: Experimenting with climate change adaptation in French coastal areas. Policy Sciences, 51(2), 231–247.
Sabel, C., & Zeitlin, J. (Eds.). (2010). Experimentalist governance in the European Union: Towards a new architecture. Oxford University Press.
Sanders, A. J. P, Ford, R. M., Keenan, R. J., & Larson, A. M. (2020). Learning through practice? Learning from the REDD+ demonstration project, Kalimantan Forests and Climate Partnership (KFCP) in Indonesia. Land Use Policy 91, 104285.
Stoker, G. (2010). Translating experiments into policy. The ANNALS of the American Academy of Political and Social Science, 628(1), 47–58.
Suškevičs, M., Hahn, T., & Rodela, R. (2019). Process and contextual factors supporting action-oriented learning: A thematic synthesis of empirical literature in natural resource management. Society & Natural Resources, 32(7), 731–750.
Treasury Board of Canada (2016). Management Accountability Framework. Retrieved from https://www.canada.ca/en/treasury-board-secretariat/services/management-accountability-framework.html
Treasury Board of Canada Secretariat (2019). Population of the federal public service by department. Retrieved from https://www.canada.ca/en/treasury-board-secretariat/services/innovation/human-resources-statistics/population-federal-public-service-department.html
Trudeau, J. (2015, November 12). Mandate letter: Treasury Board of Canada. Retrieved from https://pm.gc.ca/eng/president-treasury-board-canada-mandate-letter-november-12-2015
Van der Heijden, J. (2014). Experimentation in policy design: Insights from the building sector. Policy Sciences, 47(3), 249–266.
van Doren, D., Driessen, P.P.J., Runhaar, H.A.C., & Giezen, M. (2020). Learning within local government to promote the scaling-up of low-carbon initiatives: A case study in the City of Copenhagen. Energy Policy, 136, 111030.
Witting, A. (2017). Ruling out learning and change? Lessons from urban flood mitigation. Policy and Society, 36(2), 251–269.
Acknowledgements
Thank you to the Canada-UK Foundation for their financial support in completing the fieldwork for this project. Thank you as well to Meghan Alexander, Marc Geddes, Nick Or, Jonathan Paquette, Emily St.Denny, Ellen Stewart, James Weinberg, and Matthew Wood, as well as my UEA colleagues who gave advice on earlier drafts.
Funding
The fieldwork for this project received funding from the Canada-UK Foundation.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflicts of interest
Author declare that they have conflict of interest.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Mattocks, K. Policy experimentation and policy learning in Canadian cultural policy. Policy Sci 54, 891–909 (2021). https://doi.org/10.1007/s11077-021-09433-3
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11077-021-09433-3