1 Preface

The empirical social sciences are in a period of change and renewal that is characterized by efforts to improve the reliability and availability of empirical findings and—ultimately—the credibility of the theories that are based on these findings. Increasing the transparency of research practices is at the heart of many of these efforts. This involves methodological as well as cultural changes within research disciplines. Two cornerstones of these developments are preregistration and registered reports. While preregistration entails specifying and publishing hypotheses, research designs, and analysis plans prior to data collection or data access, registered reports are a publication format that takes the practice of preregistration one step further. In the registered reports model, studies are not only preregistered but also peer reviewed, revised, and accepted in principle for publication prior to data access. Hence, the editorial decision to offer in-principle acceptance of an article is based solely on the merits of the research question, research design, and analysis plan rather than on whether the empirical findings eventually turn out to be in line with predictions, ambiguous, clear, or surprising.

While the registered reports format has been increasing in popularity in various scientific disciplines in recent years, its application in quantitative political science has been tested only in a few instances. To further increase its visibility and popularity in quantitative political science, we encourage researchers and journals in our scientific discipline to consider registered reports as an appropriate publication format. To this end, we initiated the GLES Open Science Challenge 2021, a pilot project that combined the rigor of registered reports with quantitative political science in the context of the 2021 German federal election. We show that transparent research that follows open science principles is feasible and contributes to increasing the reproducibility and credibility of findings. We further demonstrate that this kind of research can make substantial contributions to our discipline.

This special issue is a compilation of the articles that resulted from the GLES Open Science Challenge 2021. All articles included in this special issue are registered reports based on German Longitudinal Election Study (GLES) data collected primarily in the weeks prior to the 2021 German federal election. The emphasis of the GLES Open Science Challenge was on the methodological innovation of facilitating the use of registered reports in quantitative political science. Given this methodological rather than thematic focus, the articles included in this special issue address a broad range of topics. Some focus on general topics recently discussed in political science, such as polarization, societal recognition, populism, and inequality, while others address topics that are directly related to the context of the 2021 German federal election, such as electoral integrity, coalition preferences, and the effects of published opinion polls.

In the following, we first illustrate the rationale for relying on registered reports to improve the quality of empirical research; we then describe the procedure of the GLES Open Science Challenge 2021; and, finally, we give an overview of the articles featured in this special issue as the outputs of this—ultimately highly successful—initiative.

2 Registered Reports: A Promising Approach to Overcoming Selection Bias in Scientific Publications

Science is collaborative, cumulative, and dynamic. Theories, research methods, and paradigms are constantly being proposed, tested, and revised—and sometimes abandoned. There is a consensus that objectivity, transparency, and reproducibility are essential to science. When science follows these principles, the cumulative progress of scientific knowledge is possible. However, translating these abstract principles into concrete practices in specific scientific disciplines has been the subject of controversial debate.

One major challenge for the empirical foundation of scientific theories is the “results paradox” (Chambers and Tzavella 2022). This notion refers to the observation that academic success relies at least as much on publishing novel and statistically significant results as on applying rigorous research practices. Results-centeredness discourages researchers from submitting empirical analyses with mixed results or null findings to scientific journals and discourages journals from publishing such findings. This facilitates publication bias, that is, the underreporting of null results (Esarey and Wu 2016; Franco et al. 2014; Gerber and Malhotra 2008; Peplow 2014). Furthermore, it can incentivize the use of questionable research practices such as p-hacking, which is the post hoc adaptation of data analysis procedures in order to produce a desired result (Simmons et al. 2011), or HARKing (Banks et al. 2016; Kerr 1998), which is the formulation of hypotheses after the results are known while presenting them as if they had been generated a priori. Such research practices lead to empirical results that are not replicable by other scientists (Blaszczynski and Gainsbury 2019; Dreber and Johannesson 2019; Shrout and Rodgers 2018).

However, efforts to improve the transparency and replicability of research processes are gaining traction (Ioannidis 2005; Smith 2019). Subscribing to the open science movement, researchers are discussing how to improve the robustness of empirical research findings by increasing the transparency of research procedures. Present in many fields (Fraser et al. 2018), the movement has also reached the social sciences and, more recently, political science (Burlig 2018; Freese and Peterson 2017; Monogan 2013; Wuttke 2019).

One promising tool to overcome the results paradox and to increase the transparency of research practices is the registered reports publication format (Chambers 2013; Chambers et al. 2015, 2014; Chambers and Tzavella 2022; Nosek and Lakens 2014). In stage 1 of a two-stage publication process, authors submit a manuscript comprising an introduction, research questions and hypotheses, research design, and analysis plan, which is peer reviewed prior to data collection or access—and thus before the results are known. Stage 1 manuscripts include all parts of a typical manuscript except for the results and discussion sections. After stage 1 peer review, the manuscript is accepted in principle for publication or is rejected, or the authors are invited to revise and resubmit. As this first review round is completed before data collection or access, the analysis plan can be modified following the reviewers’ recommendations. Once in-principle acceptance has been obtained, the authors will proceed to conduct the study, adhering exactly to their peer-reviewed procedures (see Chambers 2013). After completing the paper by adding the results and discussion sections, the authors submit their final manuscript for re-review (stage 2). Subject to passing quality checks, and provided the findings are meaningfully interpreted, the manuscript is published regardless of the empirical direction, strength, or statistical significance of the results.

Registered reports have several advantages (Cook et al. 2021; Reich 2021; Reich et al. 2020; Syed and Donnellan 2020): First, hypotheses and methods must be preregistered, thereby ensuring that the decisions about data analyses are specified in stage 1 of the publication process. Divergences from preregistered procedures must be transparently documented in the resulting research. Overall, this prevents researchers from engaging in questionable research practices, such as searching for operationalizations and model specifications that support specific hypotheses or adapting hypotheses to the available data post hoc (i.e., confounding prediction and postdiction; see Nosek et al. 2018). Furthermore, it ensures that data analyses are conducted in accordance with the preregistered research plan. Second, the registered reports format entails peer review of the analysis plan before data collection or access. Peer review of an analysis plan at this early stage ensures that suggested revisions are incorporated into the data collection processes (if applicable) and the data analysis methods. Therefore, registered reports can strengthen research procedures by encouraging closer, stepwise cooperation between authors and reviewers. Third, registered reports ensure a results-blind stage 1 review and editorial decision process. As positive outcomes or significant effects cannot play a role in stage 1, registered reports help prevent publication biases (Scheel et al. 2021; Soderberg et al. 2021).

3 Open Science Initiatives in Quantitative Political Science

Although registered reports improve the replicability and transparency of research practices and help avoid publication biases in scientific journals, the spread of this publication format is still relatively limited (Montoya et al. 2021). In quantitative political science, in particular, it is quite rare. However, two pilot projects aimed at improving the transparency of research practices in quantitative political science inspired the GLES Open Science Challenge 2021.

In 2015, Comparative Political Science (CPS) issued an open call for submissions of full research designs for prospective projects or already conducted projects that contained no mention of the actual results (Findley et al. 2016). Of the 19 original submissions, three were eventually granted in-principle acceptance, and, in 2016,Footnote 1CPS published a special issue featuring these three articles (Bush et al. 2016; Hidalgo et al. 2016; Huff and Kruszewska 2016). Although the editors acknowledged the benefits of the results-blind review process for high-quality publications, they also observed some biases and shortcomings introduced by the publication process (Findley et al. 2016). First, submissions were biased toward quantitative research approaches (mainly survey or field experiments): “We did not receive a single qualitative submission” (Findley et al. 2016, p. 17). Second, all of the authors in the special issue were early-career scholars. In conclusion, Findley et al. (2016) noted that when implementing results-blind peer review, (1) the number of submissions with null findings may rise disproportionately because scholars might intentionally select journals that offer results-blind peer review to publish such studies; (2) circulating and discussing articles widely with colleagues is not possible, which might negatively affect the quality of submissions; and (3) scheduling reviews prior to conducting a study prolongs the research process.

The first time registered reports were applied in electoral research was in the context of the 2016 U.S. general election, when Arthur Lupia, Brendan Nyhan, and David Mellor organized the 2016 Election Research Preacceptance Competition.Footnote 2 Scholars were invited to conduct a study on the 2016 election using American National Election Study (ANES) data and to preregister their analysis plan before the data were publicly available. Nine high-ranked political science journals took part in this initiative and agreed to review and—if deemed worthy—accept in principle registered reports submitted prior to data availability.Footnote 3 Although more than 50 analysis plans were preregistered with a public registry, only two papers were eventually published in this competition (Enders and Scott 2019; Monogan 2020). The organizers (personal communication) suspect that the low number of publications was due to the fact that most preregistrations were submitted by early-career scholars who might have lacked experience on how to implement their ideas. In retrospect, the organizers concluded that the initiative could have benefited from early-career scholars receiving advice and mentoring from experienced researchers at the crucial stages of the process.

By now, a small number of journals in political science have adopted registered reports as a standard submission option in their journal policies. They include, for example, the Japanese Journal of Political Science, the Journal of Experimental Political Science, and Politics and the Life Sciences.Footnote 4 Further, the Journal of Politics has announced that it will adopt registered reports as a submission format starting in January 2023.Footnote 5 These developments and the aforementioned initiatives show that advocacy for more transparency in quantitative political science is growing. At the same time, new publication formats such as registered reports raise new questions and challenges. As mentioned above, one of the central questions is how to convince members of the scientific community across all experience levels to adopt such formats and to extend their application to a broad range of research questions and designs. Moreover, efforts are needed to familiarize researchers with the particularities of registered reports submission and review and to create opportunities for them to gain experience with the challenges, benefits, and possible limitations of this publication format. In light of these challenges, registered reports have also been referred to as a methodological innovation and part of a cultural revolution in empirical research, as researchers have to learn how to follow open science principles and apply transparent research practices (Chambers 2019).

4 The GLES Open Science Challenge 2021

The GLES Open Science Challenge 2021 aimed to give quantitative political science researchers an opportunity to experience that registered reports are an appropriate and beneficial publication format for their studies. In initiating the project, we built on the experiences of the previous initiatives aimed at maximizing transparency in political science described above. By adopting the registered reports format rather than relying only on results-blind peer review, we were also able to reduce concerns raised by the editors of the CPS special issue (Findley et al. 2016) regarding the potential pitfalls. As both the reviewers and the authors were unaware in stage 1 of the outcomes of the analyses, intentionally submitting articles with null findings in order to get them published was not possible.Footnote 6 On the basis of the points raised in a private communication by the organizers of the 2016 Election Research Preacceptance Competition, we consider lack of expertise in applying transparency and replicability in the research process to be one possible reason why this publication format has rarely been used so far. To reduce the hurdles for both authors and reviewers in considering registered reports as an appropriate publication format in the field of quantitative political science, we supported the authors and reviewers taking part in the GLES Open Science Challenge 2021 at every stage of the submission and review process. To this end, we implemented multiple feedback rounds throughout the process. For example, because authors had to submit an abstract prior to starting work on the registered reports (stage 0), they had already received feedback on their initial ideas and the feasibility of their analyses before embarking on stage 1. We maintained close contact with the authors and reviewers throughout the editorial process. By demonstrating that registered reports bring about substantial contributions to our discipline, we raise awareness of this publication format and contribute to ongoing efforts to make transparent research practices more popular.

4.1 Bringing Together Registered Reports and GLES Data

All registered reports considered in the GLES Open Science Challenge 2021 rely on data from the German Longitudinal Election Study (GLES) collected in the context of the 2021 German federal election. One particularity of this project was that the organizers, who are also the editors of this special issue, were at the same time part of the GLES team responsible for the conceptualization of the study and for administering the collection, documentation, and publication of GLES data. The overlap of simultaneously being editors and data providers allowed us to coordinate the processes of data collection and data publication with the different stages of the GLES Open Science Challenge. This in turn enabled a fast-track publication process, with only approximately 12 months between abstract submission and publication of the final manuscripts. The fact that the organizers were very familiar with the data sets used by the authors to conduct their studies also allowed for providing in-depth support in setting up the registered reports. The decision to use GLES data for a preregistration challenge stemmed from a desire to show that this publication format can also be applied to secondary data. Although registered reports are most often applied in experimental contexts or to data collected for one specific study (van der Akker et al. 2019), our project shows that large surveys are also suitable for the application of transparent research practices. Given that the pre-election data are freely available to the scientific community and are released shortly after election day, and that the surveys capture a wide range of concepts and survey instruments to study political attitudes and behavior, the GLES is very appropriate for conducting an open science challenge. As data providers, the organizing team was, of course, also interested in incentivizing the use of GLES data by the scientific community. This challenge helps GLES gain visibility in quantitative political science, shows that the GLES data are suitable for different research purposes, and encourages researchers to use these data for their research. For the GLES Open Science Challenge 2021, researchers could choose between two different data sets, the GLES Pre-Election Cross-Section or the GLES Rolling Cross-Section.

4.1.1 The GLES Pre-Election Cross-Section

The GLES Pre-Election Cross-Section (GLES 2022a) is a data source that is very well suited to analyzing the voting behavior of the German electorate. The survey includes a wide range of questions on the political attitudes and behavior of the respondents, as well as on their sociodemographic characteristics. Striving for a temporally and internationally comparative perspective, many questions included in this survey are based on established concepts used in electoral research. However, the GLES also adopts innovative and current perspectives of international election and attitudinal research by including new instruments. For example, the 2021 Pre-Election Cross-Section included questions about electoral-integrity beliefs and perceived societal recognition.

The GLES Pre-Election Cross-Section (GLES 2022a) is a probabilistic survey fielded in the four weeks prior to election day. Because of the COVID-19 pandemic, the 2021 survey was conducted in a self-administered mixed-mode design consisting of computer-assisted web interviews and paper-and-pencil interviews. Ultimately, 5116 at least partial interviews were conducted between August 26 and September 25, 2021. Of those, 3716 (72.63%) respondents (aged 16 years and older) participated via web, while 1400 (27.37%) respondents completed the paper-and-pencil questionnaire.

4.1.2 The GLES Rolling Cross-Section

The GLES Rolling Cross-Section (GLES 2022b) is designed to record cross-sectional developments in political attitudes and behavior during the electoral campaign period. It is a probabilistic population survey based on computer-assisted telephone interviews that is divided into structurally equivalent daily samples according to a rolling cross-section (RCS) design (Johnston and Brady 2002). The design enables campaign-induced changes in public opinion to be captured. Voter reactions to events during the campaign can be registered, and the decay or stability of these effects can be revealed. Because the RCS study includes a post-election panel wave of the survey, it is also possible to investigate intraindividual changes between the two waves.

The pre-election data collection took place during the final 55 days of the run-up to election day and was complemented by a post-election panel wave in which available respondents from the pre-election interviews were re-interviewed. The gross sample of telephone numbers comprised 60% landline numbers and 40% mobile numbers drawn from sampling frames of the Arbeitsgemeinschaft Deutscher Markt- und Sozialforschungsinstitute. The exact number of telephone numbers to be entered into the system was determined daily, considering the current dynamics of the field period. In total, 7068 complete interviews were conducted in the pre-election period (a daily average of 128.51 interviews); 4446 (62.9%) of the pre-election respondents also completed the post-election survey.Footnote 7

In addition to questions on key aspects of electoral research, special features of the 2021 GLES Rolling Cross-Section survey were questions with a focus on perceptions and assessments of the candidates for the chancellorship; questions to capture media use, especially reception of the TV debates; and questions related to the use of absentee voting.

4.2 Procedure

As outlined above, the motivation behind this special issue was to encourage researchers to consider registered reports as an appropriate publication format for their studies. Further, all registered reports featured in this special issue rely on survey data provided by GLES. To combine the rigor of registered reports with GLES data, we had to align data collection and preparation schedules with the submission and review processes of the GLES Open Science Challenge. Specifically, first, we published questionnaires and study documentation of the GLES surveys before the data were collected in order to give participating researchers enough time to set up their studies and become familiar with the data sets. Second, we aligned the submission and review processes with the data collection and publication processes to allow for a results-blind preparation and review of stage 1 registered reports. Third, we scheduled submission and review processes around the 2021 German federal election. Fig. 1 gives an overview of the coordination of the scheduling of the data collection and provision steps with the GLES Open Science Challenge submission and review processes.

Fig. 1
figure 1

Stages of the GLES Open Science Challenge 2021

Full size image

4.2.1 Publication of Questionnaires and Study Documentation

To facilitate the preparation of registered reports, the GLES team published the questionnaires and study documentation of the GLES Pre-Election Cross-Section and the GLES Rolling Cross-Section surveys more than a month before the start of data collection. The published information included questionnaires, codebooks, survey design information, and the targeted sample size. In anticipation of unforeseeable developments during the campaign, and pending the results of pretests, the complete set of questions was not provided. Instead, the questionnaires included several placeholders. All information was published in English and was freely accessible on the GLES homepage. In addition, a fixed date for the data release was specified. This was necessary for scheduling the first round of reviews and implementing any revisions resulting from the reviews before the data were published.

4.2.2 Submission of Abstracts

In stage 0 of the GLES Open Science Challenge, we called for submissions of open topic abstracts providing a general overview of the planned study. Authors who submitted an abstract received editorial feedback on the feasibility and suitability of their contribution for the special issue. Because of coordination of the scheduling of the submission and review process with data collection and publication, abstracts had to be submitted by July 12, 2021, and thus over two months before the 2021 German federal election on September 26. As the questionnaires were published in late June, the authors had only three weeks to familiarize themselves with the questions, to come up with a research question related to the questionnaires, and to prepare and submit an abstract. We received 32 abstracts from 58 authors, which can be seen as a sign of considerable interest in the GLES Open Science Challenge.

As the abstracts had to be submitted at a time when the election campaign had not reached its political climax, the submissions covered a wide range of research questions related to elections, political attitudes, and behavior rather than focusing on the particularities of the 2021 election. Of the 32 study proposals submitted, 17 received positive feedback, and the authors were encouraged to submit a stage 1 manuscript.

4.2.3 Stage 1 Manuscript Submission and Review

Following the initial feedback on their planned studies, researchers had two months to prepare their stage 1 manuscripts. Besides standard criteria of article evaluation, the subsequent evaluation of the stage 1 manuscript submissions focused on whether the proposed analyses offered an adequate and appropriate test of the preregistered hypotheses.Footnote 8 To enable us to align the submission of stage 1 manuscripts with the collection of GLES survey data, authors had to submit their stage 1 manuscript by September 25, 2021—the day before election day, which was also the last day of data collection for the pre-election surveys. The period between the last day of data collection for the pre-election surveys and the publication of datasets was two months. Thus, the review process of stage 1 manuscripts was scheduled in such a way that it allowed authors to incorporate reviewers’ feedback into their preregistration before accessing the data in November/December 2021. We received 13 stage 1 manuscripts from 23 authors. Seven manuscripts were accepted in principle for publication, and the authors were invited to resubmit their articles as stage 2 manuscripts after data publication and completion of the preregistered analyses.

4.2.4 Reviewer Recruitment and Provision of Information

The strict timeline of the GESIS Open Science Challenge resulted in tightly scheduled review rounds. On average, the reviewers had only two weeks to review the stage 1 manuscripts. To ensure that the review process would work under these constraints, a call for reviewers was published well in advance of the review process. The recruitment of the reviewers in that way was highly successful. By the time the stage 1 manuscripts were submitted, we had already recruited a large pool of reviewers with expertise in different subfields of electoral research, with different degrees of experience in the academic system, and from many countries. We were overwhelmed by the breadth of the response and the great willingness of scholars to participate in the GLES Open Science Challenge as reviewers. As we assumed that neither the authors nor the reviewers would have much experience with registered reports, we provided guidelines on our homepageFootnote 9 that addressed the particularities of this article format from an author’s and a reviewer’s perspective.

4.2.5 Stage 2 Manuscript Submission and Review

Once the data were published, authors were invited to submit their stage 2 manuscripts, including the data analysis, reporting of results, and discussion. Additionally, authors were required to submit detailed analysis scripts to make each step of data cleaning, preparation, and analysis replicable. In stage 2, submissions were reviewed with regard to the implementation of the preregistered analysis plan, the interpretation of the results, and the discussion. All seven stage 1 manuscripts that obtained in-principle acceptance were resubmitted as stage 2 manuscripts and are now part of this special issue.

5 The GLES Open Science Challenge: Participating Authors

Restricting submissions to our special issue to research that used GLES data may have limited the diversity of participating scholars. Because researchers already familiar with GLES data had to invest less effort in familiarizing themselves with the data, they may have been more likely to submit their ideas.Footnote 10 We tried to overcome this risk of possible in-group bias by circulating the call for papers for the GLES Open Science Challenge through various channels and by encouraging researchers from different subdisciplines of political science and across different countries to participate. Nonetheless, it is important to take a look at some background information on the participating authors.

Table 1 provides key characteristics of participating authors at the different stages of the challenge—namely, their gender, academic status, (former) membership on the GLES team, and (former) affiliation with the University of Mannheim.Footnote 11 Although we aimed to encourage the participation of a broad and diverse range of researchers, we have to acknowledge that many of the participating authors had some proximity to the GLES through their current or former affiliations. Further, we observe a low (and over submission stages decreasing) share of professors and female researchers. While it is difficult to say whether and to what extent this observation indicates a problem, similar future projects should invest efforts in attracting researchers without prior links to the institution running the project, researchers of different career levels, and researchers from a broad diversity of backgrounds.

Table 1 Key characteristics of the participating authors in the GLES Open Science Challenge 2021
Full size table

6 The GLES Open Science Challenge 2021: Contributions

With the empirical analyses based on the two GLES data sets described in the previous sections, this special issue brings together a diverse set of studies on relevant questions during political change and crises in the context of the 2021 German federal election. Election years not only allow for analyzing electoral behavior, short-term developments, and campaign dynamics, but they also present an opportunity to monitor (the polarization of) opinions across issues, populist sentiments, and behavioral implications of attitudes and beliefs. This diversity of analytical approaches is also reflected in the contributions in this special issue, which address various aspects of political attitudes and behavior in the 2021 German federal election.

The first three articles address topics related to electoral behavior in the context of the 2021 German federal election. Robert Welz investigated the impact of issue distance, intracoalition heterogeneity, and salience on voters’ coalition preferences (Welz 2022). Fabienne Unkelbach, Melvin John, and Vera Vogel focused on the role of voters’ social class in poll effects in the 2021 German federal election (Unkelbach et al. 2022). And Christian Schnaudt assessed the behavioral implications of electoral-integrity beliefs in Germany (Schnaudt 2022).

The remaining four articles deal with political attitudes more broadly. The first two studies are aimed at describing and analyzing populist attitudes. While Nils Steiner, Christian Schimpf, and Alexander Wuttke assessed the multiple roots of populism in feelings of lacking societal recognition (Steiner et al. 2022), Robert Huber, Michael Jankowski, and Carsten Wegscheider investigated the impact of policy discontent and representation on populist attitudes (Huber et al. 2022). The final two studies explain attitudes toward two different issues in depth: Jan Menzner and Richard Traunmüller assessed why citizens think they cannot speak freely (Menzner and Traunmüller 2022), and Denis Cohen investigated the context effects of rent control preferences (Cohen 2022).

The GLES Open Science Challenge 2021 represents the first collection of registered reports published in the German Political Science Quarterly (PVS). With this special issue, PVS joins the growing number of scientific journals across different disciplines that publish articles in the registered report format, which reached 278 in June 2021 (Montoya et al. 2021). The articles in this special issue are testament to the fact that the application of registered reports in quantitative political science is feasible and desirable and that it results in relevant contributions to our discipline.