Skip to main content
SpringerLink
Log in

Supervised scaling of semi-structured interview transcripts to characterize the ideology of a social policy reform

  • Published:
Quality & Quantity Aims and scope Submit manuscript

Abstract

Automated content analysis methods treat “text as data” and can therefore analyze efficiently large qualitative databases. Yet, despite their potential, these methods are rarely used to supplement qualitative analysis in small-N designs. We address this gap by replicating the qualitative findings of a case study of a social policy reform using automated content analysis. To characterize the ideology of this reform, we reanalyze the same interview data with Wordscores, using academic publications as reference texts. As expected, the reform’s ideology is center/center-right, a result that we validate using content, convergent and discriminant strategies. The validation evidence suggests not only that the ideological positioning of the policy reform is credible, but also that Wordscores’ scope of application is greater than expected.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Similar content being viewed by others

Notes

  1. Although we can edit the interview transcripts to improve the signal/noise ratio, this requires time and resources, thereby decreasing the comparative advantage of Wordscores for analyzing interview data.

  2. Although the ideological underpinnings of activation is to some extent a matter of debate, we argue that it epitomizes a predominantly centrist, third way ideology (see Driver and Martell 2000; Van Berkel and Møller 2002; White 2004; Huo 2009).

  3. The policy paradigms are “types” and are therefore qualitatively distinct. However, their ideological characterization by Daigneault’s (2014, 2015) allows us to position them on the same left–right dimension.

  4. The word counts for reference and virgin texts are those after preprocessing.

  5. The word frequencies displayed in Tables 2 and 3 and discussed in the text are calculated from all the texts. For example, the word frequency for “think” is 647, which means that this word was scored 647 times in the virgin and reference texts.

References

  • Adcock, R., Collier, D.: Measurement validity: a shared standard for qualitative and quantitative research. Am. Polit. Sci. Rev. 95(03), 529–546 (2001)

    Article  Google Scholar 

  • Baturo, A., Mikhaylov, S.: Life of Brian revisited: assessing informational and non-informational leadership tools. Polit. Sci. Res. Methods 1(01), 139–157 (2013)

    Article  Google Scholar 

  • Benoit, K., Laver, M.: Compared to what? A comment on “A robust transformation procedure for interpreting political text” by Martin and Vanberg. Polit. Anal. 16(1), 101–111 (2008)

    Article  Google Scholar 

  • Benoit K., Nulty P. Quanteda: Quantitative Analysis of Textual Data. Version 0.9.9-24. (2017)

  • Benoit, M., Cantin, M.-H., Duchesne, É.: Une nouvelle culture émergente? L’évaluation du positionnement des groupes d’intérêt anglophones face au Cadre stratégique agricole canadien de 2007. Can. J. Polit. Sci. 46(04), 921–950 (2013)

    Article  Google Scholar 

  • Braun, V., Clarke, V.: Using thematic analysis in psychology. Qual. Res. Psychol. 3(2), 77–101 (2006)

    Article  Google Scholar 

  • Brier, A., Hopp, B.: Computer assisted text analysis in the social sciences. Qual. Quant. 45(1), 103–128 (2011)

    Article  Google Scholar 

  • Charbonneau, É.: Talking like a tax collector or a social guardian? The use of administrative discourse by U.S. State lottery agencies. In: Imbeau, L.M. (ed.) Do they Walk Like They Talk? Speech and Action in Policy Processes, pp. 223–240. Springer, New York (2009)

    Chapter  Google Scholar 

  • Collette, B., Pétry, F.: Comparing the position of Canadian political parties using French and English manifestos as textual data. In: Kaal, B., Maks, I., Elfrinkhof, A.V. (eds.) From Text to Political Positions: Text Analysis Across Disciplines, pp. 27–44. John Benjamins, Amsterdam (2014)

    Chapter  Google Scholar 

  • Collier, D., Brady, H.E., Seawright, J.: Outdated views of qualitative methods: time to move on. Polit. Anal. 18(4), 506–513 (2010)

    Article  Google Scholar 

  • Cousins, K., Mcintosh, W.: More than typewriters, more than adding machines: integrating information technology into political research. Qual. Quant. 39(5), 581–614 (2005)

    Article  Google Scholar 

  • Daigneault, P.-M.: Three paradigms of social assistance. Sage Open 4(4) (2014). https://doi.org/10.1177/2158244014559020

    Google Scholar 

  • Daigneault, P.-M.: Ideas and welfare reform in Saskatchewan: entitlement, workfare or activation? Can. J. Polit. Sci. 48(1), 147–171 (2015)

    Article  Google Scholar 

  • Daigneault, P.-M., Duval, D., Imbeau, L.M.: L’analyse de contenu automatisée et les entretiens font-ils bon ménage? Caractériser l’idéologie d’une réforme de politique sociale à l’aide de Wordscores. In: Daigneault, P.-M., Pétry, F. (eds.) L’analyse textuelle des idées, du discours et des pratiques politiques, pp. 267–292. Presses de l’Université Laval, Québec (2017)

    Google Scholar 

  • Dean, H.: Social policy and human rights: re-thinking the engagement. Soc. Policy Soc. 7(01), 1–12 (2008)

    Article  Google Scholar 

  • Driver, S., Martell, L.: Left, right and the third way. Policy Polit. 28(2), 147–161 (2000)

    Article  Google Scholar 

  • Galli, E., Grembi, V., Padovano, F.: Would you trust an Italian politician? Evidence from Italian regional politics. In: Imbeau, L.M. (ed.) Do They Walk Like They Talk? Speech and Action in Policy Processes, pp. 109–129. Springer, New York (2009)

    Chapter  Google Scholar 

  • Gauvin, J.-P., Chhim, C., Medeiros, M.: Did they mind the gap? Voter/party ideological proximity between the BQ, the NDP and Quebec voters, 2006–2011. Can. J. Polit. Sci. 49(2), 289–310 (2016)

    Article  Google Scholar 

  • Grimmer, J., Stewart, B.M.: Text as data: the promise and pitfalls of automatic content analysis methods for political texts. Polit. Anal. 21(3), 267–297 (2013)

    Article  Google Scholar 

  • Herzog, A., Benoit, K.: The most unkindest cuts: speaker selection and expressed government dissent during economic crisis. J. Polit. 77(4), 1157–1175 (2015)

    Article  Google Scholar 

  • Huo, J.: Third Way Reforms: Social Democracy After the Golden Age. Cambridge University Press, Cambridge (2009)

    Book  Google Scholar 

  • Imbeau, L.M.: Dissonance in fiscal policy: a power approach. In: Imbeau, L.M. (ed.) Do They Walk Like They Talk? Speech and Action in Policy Processes, pp. 167–184. Springer, New York (2009)

    Chapter  Google Scholar 

  • Jenson, J., Saint-Martin, D.: Building blocks for a new social architecture: the LEGO paradigm of an active society. Policy Polit. 34(3), 429–451 (2006)

    Article  Google Scholar 

  • Johnson, R.B., Onwuegbuzie, A.J., Turner, L.A.: Toward a definition of mixed methods research. J. Mixed Methods Res. 1(2), 112–133 (2007)

    Article  Google Scholar 

  • Klüver, H.: Measuring interest group influence using quantitative text analysis. Eur Union Polit. 10(4), 535–549 (2009)

    Article  Google Scholar 

  • Krouwel, A., van Elfrinkhof, A.: Combining strengths of methods of party positioning to counter their weaknesses: the development of a new methodology to calibrate parties on issues and ideological dimensions. Qual. Quant. 48(3), 1455–1472 (2014)

  • Laver, M., Benoit, K.: Locating TDs in policy spaces: the computational text analysis of Dáil speeches. Irish Polit. Stud. 17(1), 59–73 (2002)

    Article  Google Scholar 

  • Laver, M., Benoit, K., Garry, J.: Extracting policy positions from political texts using words as data. Am. Polit. Sci. Rev. 97(02), 311–331 (2003)

    Article  Google Scholar 

  • Levitas, R.: The Inclusive Society? Social Exclusion and New Labour. Palgrave Macmillan, New York (2005)

    Google Scholar 

  • Lowe, W.: Understanding wordscores. Polit. Anal. 16(4), 356–371 (2008)

  • Martin, L.W., Vanberg, G.: A robust transformation procedure for interpreting political text. Polit. Anal. 16(1), 93–100 (2008)

    Article  Google Scholar 

  • Marzagão, T.: Using NLP to measure democracy. https://arxiv.org/abs/1502.06161 (2015)

  • Montpetit, É.: In Defense of Pluralism: Policy Disagreement and its Media Coverage. Cambridge Studies in Comparative Public Policy. Cambridge University Press, Cambridge (2016)

    Book  Google Scholar 

  • Morel, N.: Social rights. In: Fitzpatrick, T., Kwon, H.-J., Manning, N., Midgley, J., Pascall, G. (eds.) International Encyclopedia of Social Policy, pp. 1291–1294. Routledge, New York (2006)

    Google Scholar 

  • O’Connor, B.: The intellectual origins of ‘welfare dependency’. Aust. J. Soc. Issues 36(3), 221–236 (2001)

    Article  Google Scholar 

  • Ralph, D., Stobbe, M.: Welfare reform as moral reform in Saskatchewan. In: Biggs, L., Stobbe, M. (eds.) Devine Rule in Saskatchewan: A Decade of Hope and Hardship, pp. 267–281. Fifth House Publishers, Saskatoon (1991)

    Google Scholar 

  • Ruedin, D.: The role of language in the automatic coding of political texts. Swiss Polit. Sci. Rev. 19(4), 539–545 (2013)

    Article  Google Scholar 

  • Slapin, J.B., Proksch, S.-O.: A scaling model for estimating time-series party positions from texts. Am. J. Polit. Sci. 52(3), 705–722 (2008)

    Article  Google Scholar 

  • Stephens, J.D.: The social rights of citizenship. In: Castles, F.G., Leibfried, S., Lewis, J., Obinger, H., Pierson, C. (eds.) The Oxford Handbook of the Welfare State, pp. 511–525. Oxford University Press, Oxford (2010)

    Google Scholar 

  • Toff, B.J., Kim, Y.M.: Words that matter: twitter and partisan polarization. Paper presented at the UW-Madison’s Political Behavior Research Group meeting, Madison, Wisconsin (2013)

  • van Berkel, R., Møller, I.H.: The concept of activation. In: van Berkel, R., Møller, I.H. (eds.) Active Social Policies in the EU: Inclusion Through Participation?, pp. 45–71. The Policy Press, Bristol (2002)

    Chapter  Google Scholar 

  • Walters, W.: The ‘active society’: new designs for social policy. Policy Polit. 25(3), 221–234 (1997)

    Article  Google Scholar 

  • Warwick, P.V.: Public opinion and government policy in Britain: a case of congruence, amplification or dampening? Eur. J. Polit. Res. 54(1), 61–80 (2015)

    Article  Google Scholar 

  • Weinberg, M.: Measuring governors’ political orientations using words as data. State Polit. Policy Q. 10(1), 96–109 (2010)

    Article  Google Scholar 

  • Welshman, J.: Workfare. In: Fitzpatrick, T., Kwon, H.-J., Manning, N., Midgley, J., Pascall, G. (eds.) International Encyclopedia of Social Policy, pp. 1559–1561. Routledge, New York (2006)

    Google Scholar 

  • White, S.: Welfare philosophy and the third way. In: Lewis, J., Surender, R. (eds.) Welfare State Change: Towards a Third Way?, pp. 25–46. Oxford University Press, Oxford; New York (2004)

    Chapter  Google Scholar 

  • Williams, C.C.: Active welfare. In: Fitzpatrick, T., Kwon, H.-J., Manning, N., Midgley, J., Pascall, G. (eds.) International Encyclopedia of Social Policy, pp. 7–8. Routledge, New York (2006)

    Google Scholar 

Download references

Acknowledgements

A previous version of this article was presented at the annual conference of the Société québécoise de science politique (SQSP), at Université Laval, Quebec City, on May 20 2016, and preliminary results were published in a book chapter (Daigneault et al. 2017). We thank Ken Benoit and Michael Laver for their generosity in answering general questions about Wordscores, Daniel Béland for validating the placement of academic texts, and Marc André Bodet, Dominic Forest and François Pétry for their comments on a previous version of this paper. We also thank Karen Taylor for her thorough text editing of a previous version.

Funding

Funding from the Research Team Support Program of the Fonds de recherche du Québec—Société et culture (FRQSC) is gratefully acknowledged.

Author information

Authors and Affiliations

  1. Center for Public Policy Analysis (CAPP) and Department of Political Science, Université Laval, Pavillon Charles-de Koninck, Office 4459, 1030 Avenue des Sciences-Humaines, Québec, QC, G1V 0A6, Canada

    Pierre-Marc Daigneault, Dominic Duval & Louis M. Imbeau

Authors
  1. Pierre-Marc Daigneault
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Dominic Duval
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Louis M. Imbeau
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Pierre-Marc Daigneault.

Ethics declarations

This research project does not require ethics review because it involves publicly available information (i.e., publications) and the secondary use of anonymous data that cannot generate identifiable information about participants (i.e., concatenated interviews in which word order was randomized).

Conflict of interests

The authors declare that there is no conflict of interest.

Electronic supplementary material

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Daigneault, PM., Duval, D. & Imbeau, L.M. Supervised scaling of semi-structured interview transcripts to characterize the ideology of a social policy reform. Qual Quant 52, 2151–2162 (2018). https://doi.org/10.1007/s11135-017-0650-0

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11135-017-0650-0

Keywords

Search

Navigation