1 Preface

This special issue includes registered reports produced in the context of the German Longitudinal Election Study (GLES) Open Science Challenge 2021. The project aimed to demonstrate that registered reports are an appropriate publication format for research in quantitative political science that can successfully be applied to secondary data. Registered reports help increase transparency and replicability in the research process and can thus make substantial and relevant contributions to our discipline. The core benefits of this relatively new publication format stem from the fact that the merits of studies are assessed based on their research questions and the rigor of their proposed analysis plans rather than on their results. One particularity of this project is that all submitted manuscripts—and thus all registered reports in this special issue—were required to use data from the German Longitudinal Election Study (GLES) collected in the context of the 2021 German federal election. The present article concludes this special issue by critically evaluating the GLES Open Science Challenge 2021 in hindsight. In the following, we look at the project from the perspectives of (1) the authors, (2) the reviewers, and (3) the editorial team. By reflecting on these experiences, we aim to provide insights and guidance for similar future projects.

2 The Authors’ Perspective

Most of the authors who participated in the GLES Open Science Challenge 2021 had no previous experience in authoring a registered report. Hence, it is very likely that our project helped motivate authors to accept the challenge of trying out this publication format. In general, the authors of the articles included in this special issue evaluated the experience of participating in the GLES Open Science Challenge 2021 positively. The present section summarizes more specific feedback and comments provided by these authors regarding their experiences.

2.1 Preregistration Is Demanding

Not surprisingly, most of the authors of the articles in this special issue stressed in their comments that preregistration was difficult. As one author wrote:

Not knowing your data beforehand requires you to think more intensively and deeply about your planned analysis, and in particular about any contingencies that may arise after observing the data. … Anticipating possible intricacies related to unknown data and having prepared a contingency plan as part of the preregistered analysis plan are two of the specific challenges when writing a registered report.

Several authors conceded that this process of formulating hypotheses and specifying analysis plans without access to data departed from the usual research procedure. For example, one author noted the following:

Even though you generally want to be strict with yourself and not let the data inform your theoretical decisions, there are moments when you just quickly want to check if theory is broadly in tune with empirical evidence.

As further comments along these lines provided by other authors show, this does not necessarily imply that the usual research procedure is circular to the point of deriving a hypothesis from the same data that are then used to “test” the hypothesis in the statistical analyses. Instead, specifying the operationalizations and making the decisions necessary to translate a hypothesis deduced from theory into a concrete analytic procedure can be very difficult without information on the data structure, which can often be obtained only from the respective data set. The following comment by the authors of one of the articles exemplifies some of these difficulties:

A particular challenge is to anticipate and think through potential problems with the statistical analysis that one would usually (i.e., in an analysis without a pre-analysis plan) just rush over when conducting the analysis. This includes issues such as the proper coding of variables into sensible categories that are not too sparsely populated or the use of factor scores to operationalize constructs when not knowing in advance whether variables do indeed load on a single scale.

Another author mentioned the following questions as a challenge for preregistration:

Will there be sufficient cases for subgroup analyses? Will there be sufficient variation in the core variables of interest? Will items form a coherent scale according to theoretical expectations, or do they need to be analyzed separately instead?

However, authors also mentioned that facing these challenges had benefits:

Having no access [to the data] requires basing all arguments on theoretical, logical thinking without lending inspiration from peaks at the data. This way, developing the theoretical argument and ensuring its internal consistency receives the attention it deserves, driven by the motivation to make predictions that might fail but are sufficiently thought through that they will hold up to the data.

Another team of authors noted the following:

It was particularly positive to focus on the theoretical ideas first, without having to evaluate the quality and analysis of the data as well as the overlap between the two at the same time. This approach allows certain parts of the text to be largely completed at a given point in time, which at the same time makes it easier to focus on the parts of the text that still need to be worked on, thus making it easier to organize and divide the work (within a team).

2.2 Particularities of Submitting a Registered Report to the GLES Open Science Challenge

Several authors highlighted difficulties specifically associated with preregistering an observational study rather than experimental research:

Scholars have much more leeway in how they operationalize certain variables and what methods are used in observational studies. … This makes preregistration of observational studies more challenging, as all of these steps must be hypothesized.

Another team of authors commented that

It is not trivial at all to decide when to reject or accept a hypothesis when running several model specifications (versus having a clearly defined experiment with a straightforward manipulation that directly speaks to the hypothesis). In hindsight, this seems an obvious problem, but how many scholars really give it a thought in their everyday analyses? … In this sense, the GLES [Open Science Challenge] was a true eye-opener.

Some challenges mentioned by the authors refer specifically to the use of registered reports in electoral research. For example, at the point in time when the research idea had to be formulated (mid-July 2021), the ultimate outcome of the federal election on September 26 appeared very unlikely to most observers, and crucial events and developments during the electoral campaign were unknown. One author noted the following:

One was not only blind to data but also somewhat blind to election results and context while preparing the manuscript. However, this double blindness both forced and motivated one to reflect very thoroughly on theory and to be as rigorous as possible in order to derive robust expectations. For me, this was highly instructive.

Another team of authors stressed that

Some theoretical ideas emerged only after the stage 1 manuscript had to be submitted. Although we still had not looked at the data, we could not preregister any of these ideas. Instead, we think that theoretical arguments require time and reflection, and this process is somewhat constrained if there is a fixed deadline by which all theoretical arguments have to be developed.

2.3 Comments Relating to the Organization of the GLES Open Science Challenge

Some points raised by the authors referred specifically to the organization of the GLES Open Science Challenge 2021. Several authors noted that it would have helped to have had more time for the preparation of the stage 1 manuscript, whereas in the later phases of working on the manuscript they had more than enough time. While the strict time schedule was described as both challenging and helpful by several authors, one author raised the question of whether knowledge of the time constraints on authors might have encouraged reviewers to give more friendly reviews. Related to that, the author in question raised concerns regarding the balance between a thorough review of the analysis code and fundamental conceptual feedback in the reviews:

As much as I liked the thorough code review, I remain skeptical whether this is a reasonable allocation of time and resources when paired with (what I consider to be) a fairly quick and lenient approach to peer review that eschews challenging authors on major, time-consuming work on theory and research design. Sure, it ensures that authors execute the research exactly as proposed at stage 1. But if the stage 1 review isn’t equally strict when it comes to aspects of research design, it seems (metaphorically speaking) like this merely forces somebody to walk in a straight line without worrying too much whether they are headed in the right direction to begin with.

2.4 Suggestions for Future Open Science Initiatives

Some authors came up with ideas for similar projects in the future. Questioning the strong focus of political science on confirmatory research, one author suggested that the value of empirically rigorous exploratory work should also be highlighted in future projects:

If the next GLES [Open Science Challenge] invited exploratory contributions along with registered reports, and explicitly encouraged the co-existence of both approaches (while of course explicitly discouraging exploratory-dressed-as-confirmatory research), this would really take us further as a discipline.

Other authors recommended promoting exchanges between participants in future Open Science Challenges in a workshop- or conference-like format:

We would recommend holding an author workshop or similar before submitting the final version of the theoretical work. This would make it possible to see how others approach the challenge, both in terms of content and their preregistration strategy. At the same time, presentations, particularly of theoretical work, help to reflect on and improve the argumentation. Given the strict timeline of the GLES Open Science Challenge, it was impossible to present the paper at a normal conference.

Finally, the provision of a synthetic data set was proposed:

A “fake data set” resembling the final data structure that can be released prior to the actual data would [be] very helpful. In this data set, the distribution of the variables should approximate the actual distribution in the final data set, but the correlations between the variables should not be present.

3 The Reviewers’ Perspective

We conducted a short online survey to investigate how the reviewers who participated in our project evaluated the review process. Invitations to take part in the survey were sent to all reviewers who participated in the GLES Open Science Challenge directly after the first review of the stage 2 manuscripts. Of the 26 reviewers invited, 19 reviewers took part in the survey. Regarding their career level, 56% of the reviewers were postdoctoral researchers, 22% were professors, 17% were doctoral students, and 5% were researchers for whom none of these categories apply. In our survey, we asked the reviewers to assess different aspects of the review process. This involved providing a general assessment of the review process, rating the information we provided for reviewing submissions to the Open Science Challenge, rating the schedule of the review process, giving their general views on reviewing registered reports, and providing some information on their previous experience with transparent research practices. In what follows, we report on the results of this survey.

3.1 Feedback Related to the GLES Open Science Challenge 2021

Overall, the respondent reviewers enjoyed taking part in the GLES Open Science Challenge. All respondents assessed their participation in our challenge as either very good (72%) or good (28%).Footnote 1 Most of the respondents (83%) had not had previous experience reviewing registered reports. Therefore, it is not surprising that all respondents consulted the reviewer guidelines that we provided. These guidelines were rated as a helpful tool in the review process and were perceived as easy to access (94%), well structured (100%), and containing all information needed (94%). One point that was particularly stressed by the respondents in the open comments was that clear and early communication of the time frames was essential for the review process to go smoothly. Other respondents pointed out that it should have been made clearer at the reviewer recruitment stage that participating in the project as a reviewer would entail reviewing both stage 1 and stage 2 manuscripts, as well as clarifying which rounds of review were obligatory for all manuscripts or necessary only in the case of specific editorial decisions (minor/major revisions). As reported by some reviewers, having this information well in advance would have been helpful in structuring their work. Therefore, we encourage editors of future projects of this kind to communicate as clearly as possible to the reviewers how many review rounds are scheduled and when and in what cases they will take place.

3.2 General Assessment of Registered Reports

Regarding the general assessment of registered reports, most of the reviewers who participated in our survey indicated that they valued registered reports and believed that this publication format could help reduce publication bias in quantitative political science. Concerning the review process of registered reports, some respondent reviewers expressed the belief that reviewing registered reports required specific expertise. One reviewer made the following comment:

Registered reports are a great opportunity to decrease publication bias regarding null results. However, I found it probably takes some experience reviewing them. This applies mostly to assessing the completeness and soundness of analysis plans before any results are published (spotting loopholes, etc.).

When evaluating the time needed to review a registered report, some respondents felt that reviewing registered reports was more time-consuming than reviewing conventional articles. However, for most respondents, this was the first time they had reviewed a registered report. This assessment may therefore be related to a lack of experience in reviewing journal articles in this format and may become less relevant in the future.

4 The Editors’ Perspective

In reporting our experiences as editors of the contributions submitted to the GLES Open Science Challenge 2021, we will focus, first, on the particularities of handling registered reports generally. Because the GLES Open Science Challenge combined the rigor of registered reports with GLES data, we will then address the particularities of handling this new publication format in the context of secondary data use.

4.1 Particularities of Handling Registered Reports

When conducting the GLES Open Science Challenge, it became clear that changing research practices in the direction of more transparency and replicability involves not only methodological innovations but also profound cultural changes. For all of those involved in an editorial capacity, the project was one of their first experiences of publishing registered reports. Hence, we all learned a lot in the process.

4.1.1 The Importance of Supporting Authors

As we did not expect authors or reviewers to have much experience with registered reports, we provided guidelines on preparing and evaluating manuscripts and preregistering analysis plans. However, during the editorial process, it became apparent that, in addition to providing these guidelines, we had to be available as contact persons to help solve some of the problems encountered during the submission process. Starting with technical support—some authors had technical problems handling the Open Science Framework (OSF) platform—it turned out that the editorial system we used was not designed for registered reports, as implementing in-principle acceptance was not possible. Further, we were confronted not only with technical issues but also with substantive issues regarding the implementation of registered reports. For example, authors raised questions regarding the level of detail of the preregistration plan. While issues could be resolved thanks to the excellent collaboration among guest editors, journal editors, and the editorial office, as well as with authors, these examples show that implementing registered reports requires additional efforts compared with regular publications. Because registered reports are becoming more common, we are confident that the prevalence of such issues will decrease as standards become established.

4.1.2 Strictly Separating Prediction and Postdiction

Discussing registered reports and their contribution to quantitative political science, all of the editors considered the increased transparency in the entire research and publication process to be very positive. However, not only the authors and reviewers had limited experience with registered reports. For the editorial team, too, it was their first time handling this publication format. Assessing divergences from the preregistered analysis plans after data access posed a particular challenge for us. What happens to the registered report if it turns out that the theoretical model is not empirically testable? What if the analysis plan has errors that were overlooked by authors, reviewers, and editors, or if the final data deviate substantially from the formulated expectations? We acknowledged that strictly adhering to open science practices, and formulating hypotheses and creating analysis plans without any knowledge of the data, means that a wide range of possible distributions/outcomes must be considered. This is very challenging and requires sound knowledge of the research question as well as the statistical methods used in the study. Confronted with having to make decisions on these issues, we realized once more that strictly separating prediction and postdiction—that is, confirmatory and exploratory research—differs more from common research practice than we expected.

4.1.3 A Time-Consuming Evaluation Process

The handling of registered reports was a steep learning curve for all participants. Our experience showed that—compared with regular publications—handling registered reports requires (1) more review stages (with available reviewers at all stages), (2) more editorial decisions, and (3) more formal checks for concordance of the preregistered analysis plans with the stage 2 manuscript and the supplied code. This makes handling registered reports in the editorial review process more complex and time-consuming than handling conventional articles.

4.2 Bringing Together Registered Reports and Secondary Data

In the GLES Open Science Challenge, all submitted manuscripts were required to use GLES data for their empirical analyses. Limiting data analysis to 2021 GLES survey data meant synchronizing the processes of data collection and publication with the submission and review processes of the present special issue. More practically, it meant that all steps of these processes were fixed in advance, and postponements were impossible. In the present section, we discuss some potential implications of these constraints.

4.2.1 Mutually Dependent Schedules of Data Release and Review Process

The schedule for the publication of the GLES questionnaires and data put some constraints on the structure of the GLES Open Science Challenge, and vice versa. First, the strict schedule was challenging for the organization of the publication of GLES study information because preparations for and the actual process of data collection can always result in short-term changes, necessary adjustments, or late delivery of information. To provide all the necessary information to the authors in time, we published several updated versions of the questionnaires and study documentation during the data collection process. Although this meant additional work compared with publishing the final version of the questionnaire along with the publication of the data, we argue that inevitable changes do not militate against publishing study information prior to the field period or data release. On the contrary, it is a necessary prerequisite for preregistration and for registered reports with secondary data.

Second, the tight schedule of the review process before data access was a result of balancing the interests of the submitters of registered reports and of regular data users. In particular, the time between stage 1 manuscript submission and data publication was short. Therefore, only a limited number of short review rounds within 2 months were possible before the decisions on in-principle acceptance of the manuscripts were made. To ease the time pressure, it would have been beneficial to increase the time between the publication of questionnaires and study information on the one hand and data dissemination on the other. However, delaying data releases would have been at the expense of other data users who had not submitted registered reports or preregistered their studies and who wanted to gain insights into public opinion close to election day. To overcome this dilemma and still guarantee a high quality of the evaluation process, at least two guest editors commented on each article in addition to the reviews provided. We also postponed the release of the data by one week but included more data preparation steps than in a regular prerelease.

In light of this, we strongly encourage other data providers to make their information material on questionnaires, study implementation, and dates of data publication publicly available as early as possible. Even if time schedules are tight, data providers can thereby make preregistration based on secondary data possible. However, one downside of our strict schedule was that factors such as possibly time-intensive revisions or the acceptability of failure to implement reviewer recommendations due to time constraints had more influence on the editorial decisions than would have been the case in normal publication processes. Consequently, exciting submissions that would have needed more extensive and time-intensive revision could not be given in-principle acceptance under the existing time limitations. That said, we are aware that this special issue is not an exception and that this trade-off is faced by all fast-track publications: Early access to scientific results comes at the price of a time-constrained evaluation process.

4.2.2 A Fixed Schedule as a Limitation on the Analytical Potential

Setting a preregistration date prior to data access also means that analyses of rare, random events that occur during the field period cannot usually be preregistered. Especially in electoral research, single events or short-term developments often turn out to be important in retrospect. These explanatory approaches can rarely be analyzed as registered reports. This issue was intensified by the particularities of the GLES Open Science Challenge, as we adapted the submission and review processes of the studies to the data collection and publication processes of two GLES surveys. Naturally, the scope of the present special issue was reduced by the data at hand.

Furthermore, the deadline for the submission of abstracts (stage 0) was scheduled for 3 months before the election, and stage 1 manuscripts had to be submitted by the day before election day. This schedule ensured that manuscripts received in-principle acceptance before data were accessible. Adjusting the submission and review processes to reflect the outcome of the election would have resulted in a delay in data publication, which we aimed to avoid. However, this design significantly impacted the analytical potential of the articles submitted. It was impossible to conduct studies that addressed the short-term dynamics in the 2021 campaign related to events and the final election results or that foresaw shifts in public opinion. The premise of the GLES is to make the data available to the scientific community very soon after collection. Hence, the problem of registering hypotheses without contextual knowledge was further exacerbated. Generally, the question of how to bring together registered reports and analyses of rare, random events must be discussed more intensively and resolved.

4.2.3 The Issue of Longitudinal Investigations

One further issue was whether we should allow longitudinal investigations of changes in political attitudes and behavior as part of the Open Science Challenge. As the GLES has been conducted for every federal election in Germany since 2009, and question wordings are intentionally harmonized over time, its database offers excellent opportunities for analyzing time trends. Against the backdrop of preregistration, one of the drawbacks of longitudinal investigations is that parts of the data are already accessible and have been analyzed intensively. Hence, predictions of researchers familiar with the database might no longer be blind to the data. The issue of applying preregistration when using longitudinal data has been discussed rather peripherally to date. However, some initial efforts are being made to develop principles and create preregistration templates that consider the problem of preregistration when using longitudinal data (Mroczek 2019). In our case, the Open Science Challenge had a strict focus on the 2021 German federal election and many of the studies in this special issue drew on newly introduced GLES items and questions. Furthermore, the main hypotheses of the studies were to be answered with data from two new GLES surveys. Thus, this obstacle had no direct impact on the registered reports featured here.

4.2.4 Making Possible the Submission of Rejected Manuscripts Elsewhere

Another issue discussed by the editorial team was the possibility for authors to submit rejected registered reports to other journals. According to our strict time plan, the final in-principle acceptance decision was sent out the day before data publication. This means that authors with rejected stage 1 manuscripts were unable to submit their manuscripts as registered reports elsewhere. We tried to overcome this possible obstacle by requiring authors to submit an abstract in stage 0 of the challenge so that we could give feedback on the feasibility and fit of the research idea and the methodological approach prior to the completion and submission of the stage 1 manuscript. Nonetheless, six manuscripts were rejected in stage 1. Once their manuscripts were rejected, the authors were able to submit these manuscripts as preregistered studies to other journals. However, because the data were published soon afterwards, there was no possibility to submit them elsewhere as registered reports. In future projects, editors and data providers alike should consider how to enable the submission of rejected registered reports to other scientific journals in cases in which data sets become available after rejection or during the manuscript revision phase.

4.2.5 Intricacies

It is important to highlight that registered reports and preregistration are only one possible approach to making research processes more transparent and that the Open Science Challenge clearly focused on the application of registered reports in quantitative, deductive political science. Thus, the project was geared toward contributions from only a subset of methodological approaches within political science. All contributions in the special issue used a hypothetico-deductive research approach. Of course, many innovative political science studies employ explorative or qualitative methods. Admittedly, the GLES Open Science Challenge was not suitable for research using these approaches because the GLES allows only for quantitative analyses, and space for non-preregistered exploratory analyses was limited. Evaluating qualitative submissions in an open science framework would have been beyond the expertise of the editorial team. However, discussions are ongoing across different scientific disciplines on applying preregistration and registered reports to exploratory research (Dirnagl 2020) and qualitative research (Jacobs 2020; Kapiszewski and Karcher 2021; Haven and van Grootel 2019). Hopefully, transparent and replicable research and publication processes will be implemented for all methodological approaches represented in political science. Consequently, we encourage not only researchers who use a hypothetico-deductive approach but also representatives of other methodological approaches to explore new ways of moving toward more transparency in political science.

5 Concluding Remarks

The GLES Open Science Challenge shows that conducting research that strictly follows open science principles is feasible in quantitative political science that uses secondary data from a large-scale election study. As this special issue shows, this way of doing research can bring about substantial contributions to our discipline, in which the underlying research process is completely transparent. Hence, the necessary additional efforts are absolutely worthwhile. By conducting the GLES Open Science Challenge and sharing the experiences of individuals involved in the project in different roles, we hope to contribute to the current discussions on open science research practices and to encourage similar future initiatives.