The Comparative Interest Group-survey project: design, practical lessons, and data sets

Abstract

This article discusses the methodology and practice behind planning and executing the Comparative Interest Group-survey project (CIG-survey). The CIG-survey includes surveys among national populations of organized interests in 9 European countries and at the European Union level. Although surveys are a useful and reliable way to collect data on a variety of topics, there are also numerous pitfalls and challenges in surveying interest groups, especially across multiple countries. Despite the prominent use of surveys in interest group research, systematic reflections on this method are scarce and data sets are not always properly archived or openly accessible. This article elaborates upon the practical implications and reflects on the lessons learnt during from the implementation of the CIG-survey. Moreover, we highlight how the fuzzy boundaries of interest communities obfuscate sampling and that surveying interest organizations requires researchers to navigate through a specific organizational context to reach and motivate respondents. We also demonstrate how a careful survey plan can positively affect response rates and enable the creation of robust comparative data sets.

Introduction

Analysing the management, political strategies and organizational development of interest groups is vital to get a better understanding of systems of interest representation. Studying organized interests from different countries also adds considerably to our comparative knowledge, especially when we are interested in the activities of national and European interest groups in multilevel policymaking. This paper outlines the Comparative Interest Group-survey project that addressed these issues (CIG-survey hereafter). The CIG-survey is an offspring of the INTEREURO project (2011–2015) that entailed the mapping and surveying of EU-level interest associations (see www.intereuro.eu for the publicly available data archive). Building on this successful experience we decided to strengthen the comparative leverage of the project by replicating it in several European countries, and hence, the CIG-survey project was launched. The core of the project is funded by grants from the Research Foundation-Flanders (FWO-V), the Slovenian Research Agency and the European Research Council (ERC); separate national CIG-surveys depended on grants from national research agencies.

Herein we present the methodology and some of the lessons drawing from these projects. The overarching aim was to collect systematic data on various organizational aspects of interest groups in a wide range of European political systems. We conducted 9 country online surveys, in Belgium, The Netherlands, Sweden, Lithuania, Slovenia, Portugal, Italy, Poland, Czech Republic and one survey at the European Union (EU) level. In total more than 15,000 organizations were invited to participate, and almost 5000 completed the questionnaire with an average response rate of 36%. The data collected allow us to answer a number of theoretical and empirical questions with respect to levels of mobilization, advocacy targets of interest organizations, the use of political strategies, organizational development, internal democracy, as well as the critical resource dependencies these organizations face. In addition to this, a major aim was to archive and share the collected data in order to facilitate cumulative and replication research.

Online surveys are an efficient and cost-effective way of collecting detailed organization-specific data on a large scale (Dillman et al. 2014, p. 301). However, despite the widespread use of surveys producing ground-breaking insights in interest group research (for an overview see Marchetti 2015), there remains a paucity of research examining the practical and technical challenges interest group researchers face, especially compared to other disciplinary areas (e.g. psychology, management, economics). While we can learn a lot from various adjacent fields, surveying interest organizations presents its own unique challenges and requires navigating through a specific organizational context. In this article we discuss and present the CIG-survey project and provide an overview of some of the challenges we faced. More precisely, we clarify different methods of sampling organized interests, how we identify and characterize national interest communities, the survey plan and recruitment of interest group leaders as our respondents. We also provide an overview of the available data sets that are publicly available via a project website (see www.cigsurvey.eu).

Why Comparative Interest Group surveys?

When it comes to large-N quantitative research on organized interests we can broadly identify three different research designs predicated on: (1) unobtrusive observational data; (2) data collected through interviews centred around concrete policy issues; and (3) large-scale survey data.

First, much group research relies on observational data drawn from registries, often supplemented with information from organizational websites. Such data are useful in order to map a population of organized interests and characterize individual organizations. In addition to this, many scholars combine this type of data with publicly available evidence on policymaking processes—such as government web pages on consultation or the news media (e.g. Klüver 2018). One advantage of using unobtrusive data is that it is rather easily collected and the chances of respondent-induced bias are absent. Indeed, much ground-breaking interest group research is predicated on observational data (e.g. Lowery and Gray 2000). However, this approach relies heavily on publicly available sources; the availability and quality of these sources can vary considerably between countries. Moreover, public sources usually lack contextualized information, especially on the internal functioning and organizational development of groups, or how interest groups interact informally with policymakers. These sources can also suffer from reliability problems—e.g. organized interests may present themselves on their website in a biased way and public registries might contain incomplete or outdated evidence.

Second, in order to obtain more contextualized data on how groups engage in policy processes, how they seek influence, build coalitions and spend their political resources, interest group scholars have developed projects in which a substantial number of interest group officials are interviewed face to face (e.g. Beyers et al. 2014, p. 176). These projects generate information not available via public sources—such as informal networks, the mobilization on specific issues, perceptions and political attitudes, political ideas and policy positions—and have demonstrably advanced our knowledge (e.g. Baumgartner et al. 2009; Dür et al. 2019; Heinz et al. 1993). Despite its advantages, such as high response rates and face-to-face contact with respondents, this approach has some drawbacks. Interviewing is prone to various sources of bias such as acquiescence, self-presentation, post hoc justifications and socially acceptable responses. It is also a costly method—in terms of time, labour and money—most notably because researchers have to work around respondents’ schedules.

Third, many group scholars have collected observational data via large scale and in recent years mostly web-based surveys. Online surveys allow researchers to reach a large number of interest organizations in relatively affordable way (e.g. Dür and Mateo 2016). It enables the collection of systematic data on the internal functioning, political activities, management and relations with members and other interest organizations. Furthermore, the ‘breadth of coverage of many organisations means it is more likely than some other approaches to obtain data based on a representative sample, and can therefore be generalizable to population’ (Kelley et al. 2003, p. 262).

While the INTEREURO project combined these three research designs, the CIG-survey is an example of a large-scale online survey project. Despite several advantages there are three main challenges with large-scale online surveys: i) the type of research questions that can be analysed; ii) identifying the relevant population; and iii) the implementation of the survey. To begin with, the method is limited in terms of research questions that can be analysed. For instance, it is less suitable for policy-centred research focusing on how groups seek to influence specific policies. Next, representative surveys rely on a robust identification of interest group populations and sampling. However, the fuzzy boundaries of systems of interest representation, as well as the ambiguous identity of some organized interests, can greatly obfuscate sampling. Several other biases can emerge. Interest group populations are highly diverse, especially when aiming to conduct comparative research, which means that particular terms used in questionnaires might be understood in different ways across countries. Finally, the quality of survey data sets also relies on how surveys are implemented. Much depends on identifying the most appropriate respondent within the organization—in terms of knowledge and experience—as well as achieving high response rates. Despite many challenges, surveying interest organizations in a rigorous and robust manner can result in detailed and reliable organizational-specific data.

Turning specifically to comparative aspects, political scientists commonly characterize political systems and state–society relations by applying encompassing macro-conceptual labels—e.g. neocorporatist, statist or pluralist. The political structures of countries, be it federal or unitary, ethno-linguistic divisions, etc., or the multi-level nature of the EU has a significant impact on interest group systems—i.e. the differential patterns of access, influence and organizational structures across various countries. Political culture and political development are also crucial when comparing interest group systems in mutating West-European democracies and diverse democracies in Central, Eastern or Southern Europe. These macro-conceptualizations enhance our theoretical understandings of how political systems function and the way (and extent to which) state–society interactions are structured.

However, macro-characterizations of entire political systems are not unproblematic because state–society relations are not stable, vary across policy areas and may change considerably overtime. Different policy sectors exhibit varying patterns of mobilization and representation, and most importantly, macro-concepts do not provide us with a micro-view on key operational aspects of interest group systems and policymaking processes. Furthermore, while interactions between policymakers and interest groups are shaped by system-level rules and procedures, the aggregation of all these interactions at the micro-level can also shape overall interest representation systems. We actually know little about how practices and beliefs at the level of individual interest groups are intertwined with system-level features of interest representation. Additionally, while labels such as neocorporatism or pluralism are frequently used to describe and classify countries, systematic comparative assessments of their key expectations regarding state–society interactions are relatively rare (an exception is Grote et al. 2008). To test the validity of these concepts, one should address a range of comparative empirical questions, including how many interest organizations mobilize, which interests participate and seek influence in policymaking processes and why we observe variance in density and diversity across policy domains and political systems. The CIG-survey aimed to address these issues.

Finally, there is one further innovative feature of the CIG-survey. It includes a combination of countries that are seldom studied in the interest group field (smaller continental European and Central, Eastern or Southern Europe countries). Most political science research focuses on the usual suspects—the USA, the European Union and West-European countries such as the UK, Denmark, The Netherlands or Belgium. Including these varying political contexts provides an opportunity to study the extent to which previous insights are also applicable in these countries and may also facilitate the further refinement of insights developed in previous research.

Mapping interest group communities and sampling interest groups

In contrast to other survey research (e.g. public opinion), where registers of citizens are usually available, comprehensive registers of interest organization populations are rare. This means that researchers have to invest a significant amount of time in constructing a representative overview of interest group populations. Furthermore, the absence of reliable registries makes the use of standard sampling techniques like stratified or random sampling difficult. The limited systematic mapping of interest intermediation systems in many countries means that these studies, in contrast to those on parties, government formation, legislative or electoral politics, often lack sound comparative empirical links to the way societal interests are represented via organized interests (see Halpin and Jordan 2011 for an overview). This is accentuated in the case of smaller and Central-European countries.

A continuing and thorny challenge facing interest group researchers involves the conceptual ambiguity that surrounds the main concept (Beyers et al. 2008; Jordan et al. 2004). Defining concepts such as an ‘interest organization’ or ‘interest community’ is not straightforward. Although different definitions and conceptualizations are not in themselves a problem, scholars are often unclear about which kind of organizations are included or excluded in an appropriate sample. Conceptual clarity (or lack of thereof) has important consequences for the interpretation of research results and generalizability.

One commonly used definition of an organized interest includes three criteria: (1) being organized; (2) aiming to influence public policy and (3) achieving political objectives through informal and formal political engagements outside the electoral arena (Beyers et al. 2008; see also Andrews and Edwards 2004; Jordan et al. 2004). The latter component sets interest groups apart from political parties because interest groups do not seek office through elections, but try to achieve their goals through formal (e.g. advisory councils) or informal engagements (e.g. lobbying) with policymakers. The first component refers to the fact that a minimal level of organization is required, thus excluding broad movements, waves of public opinion and/or the political activities of individual citizens. Finally, these organizations should show some level of political activity or articulate a public policy interest, which implies that they aim to influence policy outcomes.

Importantly, many politically active organizations are not primarily established with the aim to represent some interest, cause or constituency, but rather for other purposes. This is not only the case for firms and institutions (whose main goal is, for example, to produce cars, govern a territory, providing education), but also for organizations (membership-based and memberless) that focus mainly on service provision or charity work. Political representation can be both an Olsonion (1965) by-product of their core business, and/or accord with Truman’s (1951) disturbance theory—i.e. if their interests are threatened they mobilize politically. Often, the expertise such actors have at their disposal, as well as their economic significance (in terms of employment and/or investments), means they play a crucial role in any political system (Salisbury 1984; see also Lowery 2007). Yet, companies, hospitals or universities have no political representative function and their potential political activities are in most cases largely self-regarding (see cell 3 and 4 in Table 1). As our project aimed to develop a better understanding of the tensions organized interests experience between organizational maintenance, political insiderness and responsiveness to their members/supporters and goals, we decided not to include these pressure participants in the survey (Jordan et al. 2004).

Table 1 Typology of organized interests

Depending on which conceptual component—organizing for collective action or political advocacy—is emphasized, interest group scholars tend to apply two kinds of data collection strategies for mapping interest group populations, focusing either on behavioural (advocacy or lobbying for policy influence) or organizational aspects (mobilizing a constituency) (Berkhout et al. 2018). Scholars studying policymaking processes are inclined to adopt a top-down mapping strategy using registers of politically relevant organizations (for examples see Baumgartner et al. 2009; Beyers et al. 2014; Dür et al. 2019; Fraussen et al. 2015; Hanegraaff et al. 2015; Klüver 2012). Although a behavioural top-down approach is well-suited for research on interest organizations’ policymaking participation, it is somewhat less suitable for studying how organized interests act—or do not act—as intermediary transmission belts between citizens and the state. More concretely, the CIG-survey focuses on membership organizations or ‘groups of’ and ‘solidarity organizations’, the so-called ‘groups for’ that can have members, but that seek collective rather than sectional goods. These solidarity organizations can be divided between primarily politically active cause organizations (like Greenpeace, World Wildlife Fund and Friends of the Earth) and primarily service providing organizations (Red Cross, Doctors without Borders). Many of these latter groups that represent a collective constituency interests or cause can be labelled ‘latent interest organizations’ as they are politically less visible and/or active. However, the fact that they often provide important public goods, makes them politically relevant.

Accordingly, the CIG-survey sampling relied largely on bottom-up mapping and different country surveys relied on directories, registries or encyclopaedias of organizations that list all the associations in a particular political system (see Table 2). Most of the sources used had no threshold in terms of political activities; all organizations were included, irrespective of their involvement in public policymaking. Some CIG-surveys used a top-down mapping that may approximate to the results of a bottom-up approach as the threshold for political activity is very low. For instance, in the Swedish case colleagues used a database with all associations that had sent one letter to a government agency in a time span of several years (and they combined this with other sources). In short, we deliberately sought to include constituency-based organizations, including those that exhibit lower or less visible levels of political activity. Making valid claims about balancing organizational maintenance, constituencies’ interests and engaging with policymakers would be rendered impossible if one of the main outcomes (rarely or seldom being politically active) was systematically excluded. Second, empirically, some problems would have arisen when determining a specific cut-off point of political activity, which is further complicated when trying to make this assessment based on the website of an organization.

Table 2 Overview of data sets

Overall survey plan

Constructing a well-defined survey plan is critical for three main reasons: (1) to obtain a viable response rate—Marchetti (2015) estimated that the average response rate in interest organization surveys is c. 41%; (2) to reduce survey bias due to a lack of response from specific respondents (Dillman et al. 2014); and (3) to combat survey fatigue—many interest groups are frequently invited by researchers and policymakers to take part in various research projects. One of the unique features of the CIG-surveys is that highly similar surveys were undertaken in various European countries. This entailed establishing a coherent and equivalent approach within each country taking into account local circumstances. We cannot report in detail on each separate CIG-survey (see www.cigsurvey.eu for an exhaustive overview of how national surveys were conducted). Here we sketch the general contours of the overall survey plan which entails developing an appropriate questionnaire, identifying respondents within organizations and detailing how to approach and motivate these respondents.

Questionnaire development

Based on our experiences within previous projects, especially INTEREURO, we developed a core questionnaire that included supplementary space for country-specific questions relevant to different national contexts (Beyers et al. 2016). The core questionnaire was piloted among a small group of respondents in Belgium, the EU, Slovenia and The Netherlands. Following these pilots, we refined and tightened the questionnaire. In total, 12 colleagues from various participating countries were involved in creating an English version of the core questionnaire that was translatable into eleven languages. The following colleagues were involved in developing the core questionnaire: Joost Berkhout, Patrick Bernhagen, Jan Beyers, Frida Boräng, Caelesta Braun, Danica Fink-Hafner, Marcel Hanegraaff, Frederik Heylen, William Maloney, Daniel Naurin, Meta Novak and Dominique Pakull. The questionnaire was translated in Czech, Dutch, English, French, Italian, Lithuanian, Polish, Portuguese, Slovenian and Swedish.

Great care was taken to ensure that the questions were framed in a way that would make responses comparable. For this, we took inspiration from a large number of excellent survey projects that have been conducted during the past decades (e.g. the INTERARENA project led by Anne Binderkrantz; Binderkrantz et al. 2015). The CIG-survey addresses a number of questions that have hitherto not been extensively discussed in the empirical literature, e.g. how decisions within groups are made; advocacy activities; how members get involved; and dependence on institutional funding. Some questionnaires also included questions on rarely studied topics, including group ideological positions, and interactions with the judiciary. Table 3 gives an overview of the main dimensions of the questionnaire.

Table 3 Dimensions covered in the questionnaire

One important challenge in designing interest group questionnaires is the fact that many concepts and terms are value-loaded. Terms such as ‘lobbying’ or ‘interest group’ have a different, and sometimes pejorative, connotations in various national contexts. Many group representatives, in particular those who work for civil society organizations, recoil at being referred to as ‘lobbyists’ who work in ‘interest groups’. Thus, care had to be taken to employ the most appropriate label like civil society organisations in English, middenveldorganisatie in Dutch and organisations intermédiaires, in French, and instead of using the term ‘lobbying’, we used more neutral language, such as ‘seeking to influence public policy’ or ‘informing politicians’.

Also when designing the questionnaire, care should be taken as the language should find a balance in its specificity. For instance, as some groups are only marginally politically active, specialized language or jargon directed at highly politically active groups might bias responses towards these group types. In addition, we needed to take into account a considerable variation among our respondents; some small civil society groups are led by a handful of volunteers, while others are highly differentiated and have a large cohort of professional staff. Accordingly, the questionnaire allowed for the possibility that it could be completed by different people within one organization.

Identifying respondents

Interest group representatives are often embedded in a professional environment that is internally differentiated. Many, especially larger, organizations have a front office, a number of policy experts (in some cases working groups/committee structures), a secretary-general, a board of directors headed by a president supported by auxiliary staff, interns and volunteers. Furthermore, large peak associations are usually clusters of organizations with a national office and regional/local chapters. In such organizations, it is not always easy to locate the most appropriate respondent and sometimes it is unlikely that one person will be able to answer a wide variety of questions. In contrast, smaller organizations, especially those located in consolidating democracies, tend to not employ professional staff. Organizational complexity and diversity is a critical consideration when seeking to identify potential respondents.

It is crucial to personalize communications—i.e. try to avoid sending invitation letters to a general e-mail address. Personalization can engender ties and trust between the research team and the respondent (Dillman et al. 2014; Cook et al. 2000; Cycyota and Harrison 2002). Accordingly, in each case we identified an organizational official or spokesperson (e.g. the chairperson, the director, the secretary-general). In most countries, it is considered polite and courteous to use full names and titles when approaching potential respondents. It also demonstrates that the research team has done its homework and due diligence on basic organizational facts before inviting experts to take part in a survey.

The national CIG-survey teams invested a significant amount of time creating lists with names of key spokesperson, and in approximately 95% of the cases, it was possible to identify at least one person (e.g. chairperson or director) as well as their e-mail address. A small number of cases (less than 1%) were dropped from the sample because we could not identify an individual or an e-mail address. For large organizations we attempted to identify two individuals (e.g. the president and the director). Efforts to identify key spokespersons included website searches and telephone calls to the organizations (if such information was not available on the website). A personalized approach required collecting evidence on the name of the organization, abbreviation, full name of the respondent, e-mail address, telephone number and gender. Great care was taken when collecting contact information as errors in gender, and the related grammatical errors or spelling mistakes may invariably lead to non-responses.

Approaching and motivating respondents

It is crucial to have a well-thought through plan on how to approach and motivate respondents. In contacting respondents, the CIG-survey teams adopted various techniques aimed at reducing barriers and positively influencing respondent’s willingness to take part in the survey (Cycyota and Harrison 2006; Frohlich 2002). The most important aspect here is the careful planning and implementation of follow-up reminders via e-mail and/or telephone (Roth and Be Vier 1998). Experimental research shows that repeated contacts signal the legitimacy of a survey project and the willingness of the researcher to invest time and resources in reaching out to respondents (Sauermann and Roach 2013). Initial invitation letters help to establish trust by providing contact e-mails, telephone numbers and the project website. Several CIG-surveys used postal letters (with university logos) as the initial form of communication. Sending personalized prenotification letters might make a survey stand out and shows the commitment of the research team, especially if the letter is signed by the project leader. All the CIG-surveys adopted procedures involving repeated interactions with respondents, and this led to substantial improvements in response rates. However, care was taken to avoid being seen as ‘pushy’ and we limited our reminder contacts to four (via e-mail and/or telephone). The final reminder included a closing date. Our exchanges with our respondents were sensitive to the varying national contexts. For instance, the Slovenian invitation letter stressed that this was a major international project, while the Polish project emphasized that the survey was being led by researchers based at a Polish University. Finally, we incentivized our respondents in a variety of ways, in the Belgium and EU surveys the research teams made a one Euro donation to a charity for each completed survey, other incentives included a report on preliminary research results or an invitation to the presentation of the first results.

Concluding remarks

The ten survey data sets (in csv format) have all been uploaded on a project website and can be freely used for replication purposes (www.cigsurvey.eu). The project website contains information on, the sampling frame, questionnaires in the original language and the core questionnaire, report of the fieldwork and data report. Parts of the data are not publicly accessible because of data-protection reasons (e.g. names of individual respondents, organizational budget), but the principal investigators are prepared to provide limited access to some of these data if specific needs would arise.

One of the most valuable characteristics of the data collected under the auspices of the CIG project is that we gathered data on comprehensive populations of interest groups. Accordingly, the survey results offer considerable generalizability and comparative leverage: first, in terms of interest group activity and their level of mobilization in multilevel policymaking comparing the regional, national and European levels, and second, related to different political and administrative contexts and state–society relations, e.g.: neocorporatist, statist and pluralist; federal or unitary structures; longer-standing Western democracies and newer South European and postsocialist Central and Eastern Europe. Some of these issues have been addressed in previous research, however, not always in a systematic way. To date (2019) the CIG data have produced some outstanding research outputs, e.g. the role of interest groups as transmission belt in representative democracy (Albareda 2018); party–interest group interactions (Berkhout et al. 2019); the effect of public opinion on organizational survival (Hanegraaff and Poletti 2019); coalition building (Hanegraaff and Pritoni 2019): interest groups system density and access (Hanegraaff et al. 2019): mortality anxiety of interest groups (Heylen et al. 2018); the funding of interest groups (Heylen and Willems 2019); interest group access (Willems 2020) and development of national interest representation systems (Novak and Fink-Hafner 2019a, b). Numerous other publications are being prepared and/or are under review.

The general lesson so far has been the experiences with: (1) dividing the research work among national teams that individually implemented the survey, while they were able to use data from other national surveys for comparative research; (2) combining the provision of a common framework and the core survey instrument to all national teams while allowing for adaptations in population mapping and the implementation of the survey in particular national contexts; (3) close cooperation among research teams to exchange good practice and provide guidance and assistance to the teams that joined the project later; and (4) the central coordination of the preparations of national survey data for publications and further replication research. However, we have also faced some challenges that should be addressed in future research, e.g. (1) there should be detailed agreements on data sharing and use before conducting cross-national surveys; (2) identifying and rewriting questions that can be differently understood in various national contexts; (3) update and extend the questionnaire on topics not included in the current survey (such as the use of social media, advocacy during different stages of policy process); and (4) adapt the questionnaire to make it more suitable for different interest group types (ranging from small, unprofessionalized to highly professionalized and internationalized groups).

We strongly encourage researchers to use the CIG questionnaire (or parts of it) for replication purposes on the condition that the authors of the questionnaire are consulted in advance and the original questionnaire is cited (see project website for details on how to replicate and cite the databases). If comparable data would be collected in future projects, it would be a contribution to the public good if also these data would become also publicly available.