Introduction

In preparing for this special issue, the editorial team appreciated the unique challenges that lay in store. To borrow heavily from Seinfeld, we were proposing the equivalent of a journal volume about nothing. Our friends and colleagues called us crazy (not necessarily a novel accusation for at least one of us). Others suggested that we were undertaking a task that would largely be a fool’s errand. At least one said, “Sometimes the road less traveled, is less traveled for a reason.” Almost to a person, they were cautioning us against pursuing this project because null results are simply not to be celebrated, or at least published.

To be sure, many people with whom we discussed the theme of this special issue offered something in the way of encouragement. Comments such as “I wish there were an outlet for publishing null results that are, nonetheless, meaningful.” One downside, however, is that by asking for papers in which null results take center stage, we anticipated being flooded with a tsunami of submissions from which we might find a thimble full of quality manuscripts. We thought that we would receive every poorly conceived, badly measured, underpowered, and otherwise worthless study ever conducted, opening the proverbial floodgates. Anyone with a degree of good judgment would have simply dropped this idea and moved on to other pursuits.

We wanted to be sure that we did not experience an administrative nightmare and have to process hundreds of papers, so we asked that authors send us short proposals that we could use to screen for relevance and fit. As the story turned out, we did not receive an overwhelming number of proposals. Indeed, we experienced just the opposite. We found that even though we often think that null results are out there just waiting to be shopped to journals—that did not happen. Specifically, we received a total of 31 proposals. From that pool, approximately half (15) of the authors were asked to submit full manuscripts for review. Ultimately, our special issue slowly turned into the special "half" issue that you see before you.

In the remainder of this editorial statement, we would like to address the broad issue of publishing null results and provide a summary of how the papers in this special half-issue are examples of how null results inform our science.

What Happened?

In thinking about why we received relatively few proposals, several explanations seem possible. For example, perhaps there are simply few times when we conduct research and actually observe null results. This, of course, is patently absurd. Another potential explanation is that our call for proposals went unnoticed and was simply not communicated widely and loudly enough. Though this possibility is more realistic than the first, our call went through the same channels as almost all other special issue calls for proposals, so it also seems an unlikely reason. A third possible explanation is that we, as a field, are simply not conditioned to respond to a call as outlandish as one asking for null results. Indeed, the request for null results may have struck many researchers as so "out there" as to strain credulity. Perhaps people thought this call for proposals was a product of The Onion?

Most compelling to us, perhaps the reason we received the relatively few proposals that we did is a product of (a) being conditioned to view null results as worthless and (b) recognition (self-awareness) of the methodological deficiencies in our own work that may be driving null results. If the first reason was the primary driver of individuals’ decisions not to submit to this special issue, we need to more broadly consider the value of research and not to be so quick to assume that null findings mean, we learn nothing from the study (Landis and Rogelberg 2013). If the second reason is the explanation, then we should be collectively proud that we recognize null results frequently are produced by measures with poor reliability, research designs that are not appropriate for answering the question at hand, low statistical power, and a litany of other methodological flaws.

Why Encourage Publication of Null Results?

Why might we encourage null results and want a home for them? We can think of several possible reasons. First, what if an organization was choosing between investing resources in one of two training programs? Imagine further that one of the interventions was relatively inexpensive (e.g., $50 per employee) and the other was relatively expensive (e.g., $100 per employee). If data collected from multiple experiments revealed that there was no significant difference in the effect of these training programs on valued outcomes (e.g., employee performance), such results would be quite valuable to organizational decision makers.

A second scenario in which we might find null results not only valuable, but also even ideal, is when we create a test. With almost all such cases, we hope our test does not show differences across subgroups and that the weight associated with the interaction term in our regression model was nonsignificant. Extending this example to a related context, imagine a researcher testing a structural model in which she hopes to find evidence that the set of relations is similar between males and females. In testing this model, the researcher compares two models. One of the models allows for the parameters to freely vary while the second constrains the loadings to be equal across groups. If the fit of the second model were not significantly different than the fit of the first, the researcher would be pleased.

A third scenario may involve the testing of a theory and it’s various propositions. Although finding data supportive of particular propositions is certainly noteworthy, so is the case of finding that certain propositions are not supported. In other words, the researcher’s null findings serve as key information for the usefulness and/or validity of the theory in question.

The Elephant in the Room

To this point, we have painted a somewhat rosy picture of null results. We cannot ignore, however, the fact that null results may often be the product of poor research design, inadequate measurement, or a host of other methodological shortcomings. In a recent commentary, Leavitt (2013) succinctly summarized why uncritical interpretation of null results may lead to a far worse scientific corpus than ignoring meaningful null results ever would. Rather than assuming that all null results are to be discarded, however, we believe that those produced through rigorous designs may provide important information.

A More Complete Picture

Ultimately, our science is better to the extent that we publish robust findings that emerge in our research studies regardless of whether those results are "significant" or "nonsignificant." The spirit behind this special issue is to suggest that we need a complete picture of our phenomena of interest to truly advance our scientific knowledge. Not publishing meaningful null results is like painting a picture of a landscape on a clear day missing the color blue. We can appreciate that editorial decisions would certainly be rendered simpler by applying a shorthand rule of thumb like, “Papers that report null results should be rejected.” Such a rule of thumb, however, is an oversimplification and one that may have a deleterious effect on our accumulated knowledge. In the following sections, we offer some recommendations to editors, reviewers, and authors that may facilitate the appropriate evaluation of null results in our literature.

Suggestions for Changing the Status Quo and Better Leveraging Null Results

One of the intended goals of this special issue is to engage our field in a discussion about the role of null results in our accumulated knowledge base. Though we agree with one another in our private conversations, our practices are not likely to change without a concerted effort on the part of journal editors, reviewers, and authors. With that in mind, we offer some suggestions for the primary stakeholder groups involved in the publication process.

Editors/Reviewers

As the primary gatekeepers, editors, and reviewers have enormous control and influence over not only what gets published but also what gets submitted. If we agree that null results can have value, the following recommendations might help advance our science:

  • Be open to null results. This is perhaps easier said than done. Because null results may often occur in the presence of methodological shortcomings (e.g., lack of power, poor measures), there may be a knee-jerk reaction to equate null results with poor research. Reviewers should actively resist the temptation to assume that null results are necessarily due to poorly conducted research. Instead, the conceptual and methodological quality of the research should substantially determine the fate of a manuscript.

  • Closely related to the first point, we certainly want to hold authors to high standards with regard to methodological rigor, whether null or significant results are reported. Because no study is perfect, editors, and reviewers regularly determine the appropriate latitude to afford when evaluating significant results. The same latitude should be afforded to papers that report null results. That is, the bar cannot be unrealistically high for null work. We must be careful to avoid the application of a double standard.

  • Reviewers and editors often request that information deemed nonessential be removed during the revision process. Not surprisingly, this information is often associated with null results. We should not sacrifice information for brevity. Unless truly unrelated to the primary research question, do not remove nonsignificant results from manuscripts. More information is always better than less information.

Authors

Individual researchers conducting primary studies fundamentally create our corpus of knowledge. These individuals make choices regarding what to study, how to study it, and where to report the results. Because of their critical role, we believe that authors can take actions that facilitate the acceptance of null results as meaningful and, therefore, publishable.

  • First and foremost, researchers must pursue important questions regardless of whether the questions purport differences or similarities. If we collectively ask uninteresting questions, who really cares whether we find statistically significant results, let alone null results?

  • Closely related to the first point, we must conduct our research with the highest levels of methodological rigor. Not only does poorly conducted research (e.g., low power, weak manipulations, inadequate psychometric properties, poor internal, and external validity) lead to Type-II errors (null results) but also to Type-I errors (spurious “significant” findings). Strong methodological rigor will aid in the interpretation of null results.

  • Accurately explain, interpret, and characterize results. Investigate whether null results have methodological or substantive explanations. We do not recommend drawing strong inferences if null results can be clearly attributed to methodological shortcomings (i.e., do not oversell results). Alternatively, when guided by strong, a priori theory and sufficient methodological rigor, the interpretation of null results should not be overlooked (i.e., do not undersell results).

  • Be thoughtful in research design. Multitrait-multimethod logic (i.e., convergent and discriminant validity) has guided research for more than 50 years. Within such a framework, null results may be more meaningfully interpreted. If two variables are nonsignificantly related to one another, but both are significantly related to other variables, the null result may have different meaning than if only the nonsignificant correlation is presented.

  • We are not advocating that all papers with null results be submitted to journals. Of course, if all of us submitted everything we ever did, our journals would be clogged and the system would suffer. Instead, we are simply suggesting that all of us keep an open mind that scientific progress can take many forms. Rather than discouraging the publication of work that does not confirm a priori hypotheses, we should instead be encouraging researchers to ask interesting and important questions, use rigorous methods and appropriate statistical analyses, and to share those results no matter how small or large the p value.

The Current Papers

Although all of the papers that comprise this special issue have null results as the focus, they are each unique in how null results are incorporated. In the following section, we provide a brief summary of each paper and highlight how each furthers the aims of the special issue.

The paper by Mueller et al. (2014) evaluated whether different online survey implementation strategies were associated with differences in perceived anonymity and employee response behavior in organizational surveys. Their results showed that different implementation strategies had no substantial effects on perceived anonymity, nonresponse, or the responses of survey participants to closed-ended and open-ended survey questions. This paper is a terrific example of how null results may be important to consider. In practical contexts, there are often different methods for administering surveys, tests, appraisal systems, etc. If there are no differences between different approaches, organizations may chose to use methods that are cheaper or practically more efficient.

The paper by Kepes et al. (2014) provides a slightly different window on the importance of null results. Meta-analysis is a powerful analytic tool for cumulating results across a body of studies. Null results from primary studies are as meaningful and important to include in a meta-analysis as are "significant" results. Only by including results (including null results) from all primary studies, can a meta-analyst have confidence in the accuracy of the results.

Carter et al. (2014) provide still another example of a research question and context in which null results may be expected. Specifically, test developers generally hope their measures work equally well for all individuals in a given population. For example, one would hope that a given test of knowledge measures the same underlying construct to the same degree for men and women. Stated another way, we hope that our measures display measurement invariance/equivalence (MI/E) across groups (or perhaps time). In such studies of MI/E, the researcher hopes for “null” results as such results indicate the test works equally well for all test takers.

The paper by LoPilato et al. (2014) is focused on the performance of college football teams and addresses the question of whether performance variability is related to particular outcomes. Providing still another example of how null results may be valuable, this study is perhaps a "classic" case of how many of us think about null results. That is, unlike the other papers in this issue, the Overstreet et al. paper reports an empirical study in which the proposed hypotheses are not supported by the observed results. That is, there is a failure to reject the null hypothesis. Given the rigor of the study, however, the observed null results seem unlikely to be due to methodological flaws. As a result, the null results may instead signal that, perhaps, there really is not a meaningful relationship between the variables of interest as some have believed. Many of us may be quick to discard such results, but we believe that this is instead an opportunity to more carefully consider the theory from which the hypotheses were developed. Perhaps the data are inconsistent with the theory, because the theory is incorrect? We believe this is a question worth considering and that this paper serves as a good example of when nothing may be something.

Conclusions

We believe the papers included in this special issue are terrific examples of how null results can make strong contributions to our science. Taken together, these articles illustrate contexts in which null results are (a) explicitly proposed and expected, (b) important for accumulating results across an area of the literature, (c) unexpected, yet potentially interesting, and (d) practically important.

In addition to providing a forum for null results, a second goal of this special issue is to offer some recommendations for editors, reviewers, and authors for how null results may be meaningfully incorporated into our literature. For authors, do not be afraid to submit papers with null results as long as the study is theoretically and methodologically rigorous. For editors and reviewers, we recommend that we be open to null findings from well-conducted studies to the extent that these findings inform the field.

Before concluding this editorial, we very much want to stress that while we did this special feature to bring attention to, and highlight the importance of research around null results, this journal has and will continue to be open to publishing high quality research demonstrating null effects. In just 2013 alone, two such papers were published (Major et al. 2013; Woehr et al. 2013). These papers and those included in this special issue make meaningful contributions to our literature and serve as important examples of how "null" results inform our science.