Skip to main content
SpringerLink
Log in

estimating interest groups’ policy positions through content analysis: a discussion of automated and human-coding text analysis techniques applied to studies of EU lobbying

  • Symposium
  • Published:
European Political Science Aims and scope Submit manuscript

Abstract

The promises and pitfalls of automated (computer-assisted) and human-coding content analysis techniques applied to political science research have been extensively discussed in the scholarship on party politics and legislative studies. This study presents a similar comparative analysis outlining the pay-offs and trade-offs of these two methods of content analysis applied to research on EU lobbying. The empirical focus is on estimating interest groups’ positions based on their formally submitted policy position documents in the context of EU policymaking. We identify the defining characteristics of these documents and argue that the choice for a method of content analysis should be informed by a concern for addressing the specificities of the research topic covered, of the research question asked and of the data sources employed. We discuss the key analytical assumptions and methodological requirements of automated and human-coding text analysis and the degree to which they match the identified text characteristics. We critically assess the most relevant methodological challenges research designs face when these requirements need to be complied with and how these challenges might affect measurement validity. We also compare the two approaches in terms of their reliability and resource intensity. The article concludes with recommendations and issues for future research.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Notes

  1. See Grimmer and Stewart (2013: 268) for an excellent overview of automated content analysis methods for political texts. Supervised and unsupervised text-scaling algorithms are the two main approaches used for ideological scaling.

  2. This distinction differs thus from the classical dichotomy of qualitative versus quantitative content analysis (Krippendorff, 2004: 87–98). Our analysis is based on the fundamental assumption that ‘[a] content analysis has as its goal a numerically based summary of a chosen message set. It is neither a gestalt impression not a fully detailed description of a message or a message set’ (Krippendorff, 2004: 87–89).

  3. We note though that the application of automated content analysis techniques usually requires a careful preparation of analysed texts in terms of removing uninformative words, numbers, figures and punctuation marks. This step can also be labour intensive and relies exclusively on the efforts of researchers.

  4. See however the very recent attempts made to develop ‘automated multilingual content analysis techniques’ by Proksch et al (2015).

  5. Slapin and Proksch (2014: 137) refer to these as ‘stopwords’, that is, words that have no ideological content such as ‘prepositions and conjunctions’.

References

  • Alexa, M. and Zuell, C. (2000) ‘Text analysis software: Commonalities, differences and limitations: The results of a review’, Quality and Quantity 34(3): 299–321.

    Article  Google Scholar 

  • Benoit, K. and Laver, M. (2006) Party Policy in Modern Democracies, London: Routledge.

    Google Scholar 

  • Benoit, K. and Laver, M. (2007) ‘Estimating party policy positions: Comparing expert surveys and hand-coded content analysis’, Electoral Studies 26(1): 90–107.

    Article  Google Scholar 

  • Benoit, K. and Laver, M. (2012) ‘The Dimensionality of Political Space: Epistemological and Methodological Considerations’, European Union Politics 13(2): 194–218.

    Article  Google Scholar 

  • Boräng, F., Eising, R., Klüver, H., Mahoney, C., Naurin, D., Rasch, D. and Rozbicka, P. (2014) ‘Identifying frame: A comparison of research methods’, Interest Groups and Advocacy 3(2): 188–201.

    Article  Google Scholar 

  • Budge, I. and Pennings, P. (2007) ‘Do they work? Validating computerized word frequency estimates against policy series’, Electoral Studies 26(1): 121–129.

    Article  Google Scholar 

  • Bunea, A. (2013) ‘Issues, preferences and ties: Determinants of EU interest groups’ preference attainment in the environmental policy area’, Journal of European Public Policy 20(4): 552–570.

    Article  Google Scholar 

  • Bunea, A. (2014) ‘Explaining interest groups’ articulation of policy preferences in the European commission’s open consultations: An analysis of the environmental policy area’, Journal of Common Market Studies 52(6): 1224–1241.

    Article  Google Scholar 

  • Bunea, A. and Baumgartner, F.R. (2014) ‘The state of the discipline: Authorship, research designs, and citation patterns in studies of EU interest groups and lobbying’, Journal of European Public Policy 21(10): 1412–1434.

    Article  Google Scholar 

  • Bunea, A. and Ibenskas, R. (2015) ‘Quantitative text analysis and the study of EU lobbying and interest groups’, European Union Politics 16(3): 429–455.

    Article  Google Scholar 

  • Caldeira, G.A. and Wright, J.R. (1990) ‘Amici Curiae before the supreme court: Who participates, when and how much?’ Journal of Politics 52(3): 782–806.

    Article  Google Scholar 

  • Diermeier, D., Godbout, J.F., Bei, Y. and Kaufmann, S. (2012) ‘Language and ideology in congress’, British Journal of Political Science 42(1): 31–55.

    Article  Google Scholar 

  • Eising, R. (2016, this issue of European Political Science) ‘Studying interest groups: Methodological challenges and tools’, European Political Science, doi: 10.1057/eps.2016.14.

  • Eising, R., Rasch, D. and Rozbicka, P. (2015) ‘Institutions, policies and arguments: Framing activities in the EU multi-level system’, Journal of European Public Policy 22(4): 516–533.

    Article  Google Scholar 

  • Evans, D. (1996) ‘Before the roll-call: Interest group lobbying and public policy outcomes in house’, Political Research Quarterly 49(2): 287–304.

    Article  Google Scholar 

  • Grimmer, J. and Stewart, B.M. (2013) ‘Text as data: The promise and pitfalls of automatic content analysis methods for political texts’, Political Analysis 21(3): 267–297.

    Article  Google Scholar 

  • Hayes, A.F. and Krippendorff, K. (2007) ‘Answering the call for a standard reliability measure for coding data’, Communication Methods and Measures 1(1): 77–89.

    Article  Google Scholar 

  • Klüver, H. (2009) ‘Measuring interest group influence using quantitative text analysis’, European Union Politics 10(4): 535–549.

    Article  Google Scholar 

  • Klüver, H. (2013) Lobbying in the European Union: Interest Groups, Lobbying Coalitions and Policy Change, Oxford: Oxford University Press.

    Book  Google Scholar 

  • Klüver, H. and Mahoney, C. (2015) ‘Measuring interest group framing strategies in public policy debates’, Journal of Public Policy 35(2): 223–244.

    Article  Google Scholar 

  • Klüver, H., Mahoney, C. and Opper, M. (2015) ‘Framing in context: How interest groups employ framing to lobby the European commission’, Journal of European Public Policy 22(4): 481–498.

    Article  Google Scholar 

  • Krippendorff, K. (2004) Content Analysis: An Introduction to Its Methodology, Thousand Oaks, CA; London: SAGE Publications.

    Google Scholar 

  • Laver, M., Benoit, K. and Garry, J. (2003) ‘Extracting policy positions from political texts using words as data’, American Political Science Review 97(2): 311–331.

    Article  Google Scholar 

  • Lowe, W. and Benoit, K. (2013) ‘Validating estimates of latent traits from textual data using human judgment as a benchmark’, Political Analysis 21(3): 298–313.

    Article  Google Scholar 

  • Mikhaylov, S., Laver, M. and Benoit, K. (2012) ‘Coder reliability and misclassification in the human coding of party manifestos’, Political Analysis 20(1): 78–91.

    Article  Google Scholar 

  • Nelson, D. and Yackee, S.W. (2012) ‘Lobbying coalitions and government policy change: An analysis of federal rulemaking’, Journal of Politics 74(2): 339–553.

    Article  Google Scholar 

  • Neuendorf, K.A. (2002) The Content Analysis Guidebook, London: SAGE Publications.

    Google Scholar 

  • Proksch, S., Lowe, W. and Soroka, S. (2015) ‘Automated multilingual content analysis: A new approach to estimating conflict from parliamentary speeches across languages’, Paper presented at the fifth annual conference of the European Political Science Association, Vienna, 25–27 June.

  • Proksch, S.O. and Slapin, J.B. (2012) ‘Institutional foundations of legislative speech’, American Journal of Political Science 56(3): 520–537.

    Article  Google Scholar 

  • Rasmussen, A. (2015) ‘Participation in written government consultations in Denmark and the United Kingdom: System and actor-level effect’, Government and Opposition 50(2): 271–299.

    Article  Google Scholar 

  • Rasmussen, A. and Alexandrova, P. (2012) ‘Foreign interests lobbying Brussels: Participation of non-EU members in commission consultations’, Journal of Common Market Studies 50(4): 614–631.

    Article  Google Scholar 

  • Rasmussen, A. and Carroll, B.J. (2014) ‘Determinants of upper-class dominance in the heavenly chorus: Lessons from European union online consultations’, British Journal of Political Science 44(2): 445–459.

    Article  Google Scholar 

  • Roberts, C.W. (2000) ‘A conceptual framework for quantitative text analysis on joining probabilities and substantive inferences about texts’, Quality and Quantity 34(3): 259–274.

    Article  Google Scholar 

  • Skalski, P.D. (2002) ‘Computer Content Analysis Software’, in K.A. Neuendorf (ed.) The Content Analysis Guidebook, London: SAGE Publications, pp. 225–235.

    Google Scholar 

  • Slapin, J.B. and Proksch, S.O. (2008) ‘A scaling model of for estimating time-series party positions from texts’, American Journal of Political Science 52(3): 705–722.

    Article  Google Scholar 

  • Slapin, J.B. and Proksch, S.O. (2014) ‘Words as Data: Content Analysis in Legislative Studies’, in S. Martin, T. Saalfeld and K.W. Strom (eds.) Oxford Handbook of Legislative Studies, Oxford: Oxford University Press, pp. 126–144.

    Google Scholar 

  • Vannoni, M. (2016, this issue of European Political Science) ‘Studying preference attainment using spatial models’, European Political Science, doi: 10.1057/eps.2016.13.

  • Voltolini, B. (2016, this issue of European Political Science) ‘Framing processes and lobbying in EU foreign policy: Case study and process tracing methods’, European Political Science, doi: 10.1057/eps.2016.18.

  • Yackee, J.W. and Yackee, S.W. (2006) ‘A bias towards business? Assessing interest group influence on the US bureaucracy’, Journal of Politics 68(1): 128–139.

    Article  Google Scholar 

Download references

Acknowledgements

The authors are grateful to Helene Helboe Pedersen and Rainer Eising for their excellent comments on previous versions of this study, as well as to the participants in the 2014 ECPR workshop on methods to study interest groups, for their great insights and distinghuished company. They would also like to thank Jon Slapin and Ken Benoit for introducing them to the world of automated text analysis. Both authors gratefully acknowledge the financial support received from the European Commission’s Marie Sklodowska-Curie Actions Programme through their individual Intra-European Fellowships while writing this article (A. Bunea: Grant no. 622661; R. Ibenskas: Grant no. 330446).

Author information

Authors and Affiliations

  1. Department of Political Science, University College London, 31 Tavistock Square, London, WC1H 9QU, UK

    adriana bunea

  2. Department of Politics and International Relations, University of Southampton, Highfield Campus, University Road, Southampton, SO17 1BJ, UK

    raimondas ibenskas

  3. Department of Political Science, Aarhus University, Bartholins Allé 7, Aarhus C, DK-8000, Denmark

    anne s binderkrantz

Authors
  1. adriana bunea
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. raimondas ibenskas
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. anne s binderkrantz
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to adriana bunea.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

bunea, a., ibenskas, r. & s binderkrantz, a. estimating interest groups’ policy positions through content analysis: a discussion of automated and human-coding text analysis techniques applied to studies of EU lobbying. Eur Polit Sci 16, 337–353 (2017). https://doi.org/10.1057/eps.2016.15

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1057/eps.2016.15

Keywords

Search

Navigation