Skip to main content
SpringerLink
Log in

Answers to 18 Questions About Open Science Practices

  • Original Paper
  • Published:
Journal of Business and Psychology Aims and scope Submit manuscript

Abstract

Open science refers to an array of practices that promote openness, integrity, and reproducibility in research; the merits of which are being vigorously debated and developed across academic journals, listservs, conference sessions, and professional associations. The current paper identifies and clarifies major issues related to the use of open science practices (e.g., data sharing, study pre-registration, open access journals). We begin with a useful general description of what open science in organizational research represents and adopt a question-and-answer format. Through this format, we then focus on the application of specific open science practices and explore future directions of open science. All of this builds up to a series of specific actionable recommendations provided in conclusion, to help individual researchers, reviewers, journal editors, and other stakeholders develop a more open research environment and culture.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • American Psychological Association. (2010). Publication manual of the American Psychological Association (6th ed.). Washington, DC: American Psychological Association.

  • Anderson, K. (2016). 96 things pubishers do. The scholarly kitchen. Retrieved on March 9, 2017 from https://scholarlykitchen.sspnet.org/2016/02/01/guest-post-kent-anderson-updated-96-things-publishers-do-2016-edition/

  • Anderson, M. S., Martinson, B. C., & De Vries, R. (2007). Normative dissonance in science: Results from a national survey of US scientists. Journal of Empirical Research on Human Research Ethics, 2, 3–14.

    Article  PubMed  Google Scholar 

  • Antonakis, J. (2017). On doing better science: From thrill of discovery to policy implications. The Leadership Quarterly, 28, 5–21.

    Article  Google Scholar 

  • Baker, M. (2016). Is there a reproducibility crisis? A nature survey lifts the lid on how researchers view the crisis rocking science and what they think will help. Nature, 533, 452–455.

    Article  PubMed  Google Scholar 

  • Bakker, M., van Dijk, A., & Wicherts, J. M. (2012). The rules of the game called psychological science. Perspectives on Psychological Science, 7, 543–554.

  • Bakker, M., & Wicherts, J. M. (2014). Outlier removal and the relation with reporting errors and quality of psychological research. PLos One.

  • Bandura, A. (2001). Social cognitive theory: An agentic perspective. Annual Review of Psychology, 52, 1–26.

    Article  PubMed  Google Scholar 

  • Banks, G. C., Kepes, S., & McDaniel, M. A. (2015). Publication bias: Understanding the myths concerning threats to the advancement of science. In C. E. Lance & R. J. Vandenberg (Eds.), Statistical and methodological myths and urban legends: Doctrine, verity, and fable in organizational and social sciences (pp. 36–64). New York: Routledge.

    Google Scholar 

  • Banks, G. C., & McDaniel, M. A. (2011). The kryptonite of evidence-based I-O psychology. Industrial and Organizational Psychology: Perspectives on Science and Practice, 4(1), 40–44. https://doi.org/10.1111/j.1754-9434.2010.01292.x.

    Article  Google Scholar 

  • Banks, G. C., & O’Boyle Jr., E. H. (2013). Why we need I-O psychology to fix I-O psychology. Industrial and Organizational Psychology: Perspectives on Science and Practice, 6, 291–294.

    Article  Google Scholar 

  • Banks, G. C., O’Boyle Jr., E., Pollack, J. M., White, C. D., Batchelor, J. H., Whelpley, C. E., Abston, K. A., Bennett, A. A., & Adkins, C. L. (2016). Questions about questionable research practices in the field of management: A guest commentary. Journal of Management, 42, 5–20.

    Article  Google Scholar 

  • Banks, G. C., Pollack, J. M., Bochantin, J. E., Kirkman, B. L., Whelpley, C. E., & O’Boyle, E. H. (2016). Management’s science practice gap: A grand challenge for all stakeholders. Academy of Management Journal, 59, 1–27.

    Article  Google Scholar 

  • Banks, G. C., Rogelberg, S. G., Woznyj, H. M., Landis, R. S., & Rupp, D. E. (2016). Evidence on questionable research practices: The good, the bad, and the ugly. Journal of Business and Psychology, 31, 323–338.

    Article  Google Scholar 

  • Bedeian, A. G., Taylor, S. G., & Miller, A. N. (2010). Management science on the credibility bubble: Cardinal sins and various misdemeanors. Academy of Management Learning & Education, 9, 715–725.

    Google Scholar 

  • Benjamin, D. J., Berger, J. O., Johannesson, M., Nosek, B. A., Wagenmakers, E.-J., Berk, R., … Camerer, C. (2017). Redefine statistical significance. Nature Human Behaviour.

  • Bhattacharjee, Y. (2013). The mind of a con man. The New York Times. Retrieved March 9, 2017 from http://www.nytimes.com/2013/04/28/magazine/diederik-stapels-audacious-academic-fraud.html?pagewanted=all&_r=0.

  • Biemann, T. (2013). What if we were Texas sharpshooters? Predictor reporting bias in regression analysis. Organizational Research Methods, 16, 335–363.

    Article  Google Scholar 

  • Bosco, F. A., Aguinis, H., Field, J. G., Pierce, C. A., & Dalton, D. R. (2016). HARKing’s threat to organizational research: Evidence from primary and meta-analytic sources. Personnel Psychology, 69, 709–750.

    Article  Google Scholar 

  • Bosco, F. A., Aguinis, H., Singh, K., Field, J. G., & Pierce, C. A. (2015). Correlational effect size benchmarks. Journal of Applied Psychology, 100, 431–449.

    Article  PubMed  Google Scholar 

  • Bosco, F. A., Steel, P., Oswald, F. L., Uggerslev, K., & Field, J. G. (2015). Cloud-based meta-analysis to bridge science and practice: Welcome to metaBUS. Personnel Assessment and Decisions, 1, 3–17.

    Article  Google Scholar 

  • Butler, D. (2016). Dutch lead European push to flip journals to open access. Nature News, 529(7584), 13.

    Article  Google Scholar 

  • Carey, B. (2015). Journal science releases guidelines for publishing scientific studies. The New York Times. Retrieved from http://www.nytimes.com/2015/06/26/science/journal-science-releases-guidelines-for-publishing-scientific-studies.html?_r=0.

  • Carpenter, N. C., Son, J., Harris, T. B., Alexander, A. L., & Horner, M. T. (2016). Don’t forget the items: Item-level meta-analytic and substantive validity techniques for reexamining scale validation. Organizational Research Methods, 19, 616–650.

    Article  Google Scholar 

  • Cashen, L. H., & Geiger, S. W. (2004). Statistical power and the testing of null hypotheses: A review of contemporary management research and recommendations for future studies. Organizational Research Methods, 7, 151–167.

    Article  Google Scholar 

  • Caulfield, T., Harmon, S. H., & Joly, Y. (2012). Open science versus commercialization: A modern research conflict? Genome Medicine, 4, 1.

    Article  Google Scholar 

  • Cortina, J. M. (2015). A revolution with a solution. Philadelphia, PA: Opening plenary presented at the meeting of the Society for Industrial/Organizational Psychology.

    Google Scholar 

  • Cortina, J. M., Green, J. P., Keeler, K. R., & Vandenberg, R. J. (2017). Degrees of freedom in SEM: Are we testing the models that we claim to test? Organizational Research Methods, 20, 350–378.

    Article  Google Scholar 

  • Derksen, M., & Rietzschel, E. F. (2013). Surveillance is not the answer, and replication is not a test: Comment on Kepes and McDaniel,“How trustworthy is the scientific literature in I–O psychology?”. Industrial and Organizational Psychology, 6, 295–298.

    Article  Google Scholar 

  • Donnellan, M. B., Lucas, R. E., & Cesario, J. (2015). On the association between loneliness and bathing habits: Nine replications of Bargh and Shalev (2012) Study 1. Emotion, 15, 109–119.

    Article  PubMed  Google Scholar 

  • Eich, E. (2014). Business not as usual. Psychological Science, 25, 3–6.

    Article  PubMed  Google Scholar 

  • Ethiraj, S. K., Gambardella, A., & Helfat, C. E. (2016). Replication in strategic management. Strategic Management Journal, 37, 2191–2192.

    Article  Google Scholar 

  • Fang, F. C., & Casadevall, A. (2015). Competitive science: Is competition ruining science? Infection and Immunity: IAI. 02939–02914.

  • Findley, M. G., Jensen, N. M., Malesky, E. J., & Pepinsky, T. B. (2016). Can results-free review reduce publication bias? The results and implications of a pilot study. Comparative Political Studies, 1–37.

  • Friesike, S., Widenmayer, B., Gassmann, O., & Schildhauer, T. (2015). Opening science: Towards an agenda of open science in academia and industry. The Journal of Technology Transfer, 40, 581–601.

    Article  Google Scholar 

  • Gabriel, A. S., & Wessel, J. L. (2013). A step too far? Why publishing raw datasets may hinder data collection. Industrial and Organizational Psychology: Perspectives on Science and Practice, 6, 287–290.

    Article  Google Scholar 

  • Grahe, J. E. (2014). Announcing open science badges and reaching for the sky. The Journal of Social Psychology, 154, 1–3.

    Article  PubMed  Google Scholar 

  • Grand, J. A., Rogelberg, S. G., Allen, T. D., Landis, R. S., Reynolds, D. H., Scott, J. C., Tonidandel, S., Truxillo, D. M. (2017). A systems-based approach to fostering robust science in industrial-organizational psychology. Industrial and Organizational Psychology: Perspectives on Science and Practice. 1–39.

  • Grand, J. A., Rogelberg, S. G., Banks, G. C., Landis, R. S., & Tonidandel, S. (in press). From outcome to process focus: Fostering a more robust psychological science through registered reports and results-blind reviewing. Perspectives on Psychological Science.

  • Healy, L. W. (2015). Tipping point: Information industry outlook. Retrieved from http://info.outsellinc.com/rs/422-MBV-091/images/Outsell_CEO_Topics_01oct2015_Info_Industry_Outlook_2016_Tipping_Point.pdf.

  • Hollenbeck, J. R., & Wright, P. M. (2017). Harking, sharking, and tharking: Making the case for post hoc analysis of scientific data. Journal of Management, 43, 5–18.

    Article  Google Scholar 

  • Internal Revenue Service. (2014). Form 990: Return of organization exempt from income tax. Retrieved from http://www.irs.gov/pub/irs-pdf/f990.pdf.

  • Ioannidis, J. P. A. (2005). Why most published research findings are false. PLoS Medicine, 2, e124.

    Article  PubMed  PubMed Central  Google Scholar 

  • Janssen, M., Charalabidis, Y., & Zuiderwijk, A. (2012). Benefits, adoption barriers and myths of open data and open government. Information Systems Management, 29, 258–268.

    Article  Google Scholar 

  • Jebb, A. T., Parrigon, S., & Woo, S. (in press). Exploratory data analysis as a foundation of inductive research. Human Resource Management Review.

  • John, L. K., Loewenstein, G., & Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science, 23(5), 524–532. https://doi.org/10.1177/0956797611430953.

  • Jones, J. W., & Dages, K. D. (2013). A new era of large-scale data sharing: A test publisher’s perspective. Industrial and Organizational Psychology, 6, 309–312.

    Article  Google Scholar 

  • Kepes, S., Banks, G., C., McDaniel, M. A., & Whetzel, D. L. (2012). Publication bias in the organizational sciences. Organizational Research Methods, 15, 624–662. https://doi.org/10.1177/1094428112452760.

  • Kepes, S., & McDaniel, M. A. (2013). How trustworthy is the scientific literature in I-O psychology? Industrial and Organizational Psychology: Perspectives on Science and Practice, 6, 252–268.

    Article  Google Scholar 

  • Kerr, N. L. (1998). HARKing: Hypothesizing after the results are known. Personality and Social Psychology Review, 2, 196–217. https://doi.org/10.1207/s15327957pspr0203_4.

    Article  PubMed  Google Scholar 

  • Korn, M. (2014). Management research is fish, says new management research. The Wall Street Journal. Retrieved from http://blogs.wsj.com/atwork/2014/02/18/management-research-is-fishy-says-new-management-research/.

  • Lakens, D., Adolfi, F. G., Albers, C. J., Anvari, F., Apps, M. A. J., Argamon, S. E., … Zwaan, R. A. (2017, September 18). Justify your alpha: A response to “redefine statistical significance”. Retrieved from psyarxiv.com/9s3y6

  • Leavitt, K. (2013). Publication bias might make us untrustworthy, but the solutions may be worse. Industrial and Organizational Psychology: Perspectives on Science and Practice, 6, 290–295.

    Article  Google Scholar 

  • LeBreton, J. M. (2016). Editorial. Organizational Research Methods, 19, 3–7.

    Article  Google Scholar 

  • Locke, E. A. (2007). The case for inductive theory building. Journal of Management, 33, 867–890.

    Article  Google Scholar 

  • Lykken, D. T. (1968). Statistical significance in psychological research. Psychological Bulletin, 70, 151–159.

    Article  PubMed  Google Scholar 

  • Maxwell, S. E. (2004). The persistence of underpowered studies in psychological research: Causes, consequences, and remedies. Psychological Methods, 9, 147–163.

    Article  PubMed  Google Scholar 

  • McCook, A. (2016). Duke fraud case highlights financial risks for universities. Science, 353, 977–978.

    Article  PubMed  Google Scholar 

  • McKiernan, E. C., Bourne, P. E., Brown, C. T., Buck, S., Kenall, A., Lin, J., et al. (2016). How open science helps researchers succeed. eLife, 5, e16800.

    Article  PubMed  PubMed Central  Google Scholar 

  • Munafò, M. R., Nosek, B. A., Bishop, D. V., Button, K. S., Chambers, C. D., du Sert, N. P., Simonsohn, U., Wagenmakers, E.-J., Ware, J. J., & Ioannidis, J. P. (2017). A manifesto for reproducible science. Nature Human Behaviour, 1, 0021.

    Article  Google Scholar 

  • Nosek, B. A., Alter, G., Banks, G. C., Borsboom, D., Bowman, S. D., Breckler, S., Buck, S., Chambers, C., Chin, G., Christensen, G., Contestabile, M., Dafoe, A., Eich, E., Freese, J., Glennerster, R., Goroff, D., Green, D. P., Heese, B., Humphreys, M., Ishiyama, J., Karlan, D., Kraut, A., Lupia, A., Marbry, P., Madon, T., Malhotra, N., Wilson, E. M., McNutt, M., Miguel, E., Paluck, E. L., Simonsohn, U., Soderberg, C., Spellman, B. A., Tornow, J., Turitto, J., VandenBos, G. R., Vazire, S., Wagenmakers, E. J., Wilson, R., & Yarkoni, T. (2015). Promoting an open research culture: Author guidelines for journals to promote transparency, openness, and reproducibility. Science, 348, 1422–1425.

    Article  PubMed  PubMed Central  Google Scholar 

  • Nosek, B. A., & Bar-Anan, Y. (2012). Scientific utopia: I. Opening scientific communication. Psychological Inquiry, 23, 217–243.

    Article  Google Scholar 

  • Nosek, B. A., Spies, J. R., & Motyl, M. (2012). Scientific utopia: II. Restructuring incentives and practices to promote truth over publishability. Perspectives on Psychological Science, 7, 615–631.

    Article  PubMed  Google Scholar 

  • Nuzzo, R. (2015). How scientists fool themselves-and how they can stop. Nature, 526, 182–185.

    Article  PubMed  Google Scholar 

  • O'Boyle, E. H., Banks, G. C., Carter, K., Walter, S., & Yuan, Z. (2018). A 20-year review of outcome reporting bias in moderated multiple regression. Journal of Business and Psychology. https://doi.org/10.1007/s10869-018-9539-8.

  • O'Boyle, E. H., Banks, G. C., & Gonzalez-Mule, E. (2017). The Chrysalis effect: How ugly initial results metamorphosize into beautiful articles. Journal of Management, 43, 400–425.

    Google Scholar 

  • Office of Science and Technology Policy. (2000). Federal research misconduct policy. Federal Register., 65(235), 76260–76264.

    Google Scholar 

  • Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349, aac4716.

    Article  Google Scholar 

  • Pordes, R., Petravick, D., Kramer, B., Olson, D., Livny, M., Roy, A., Avery, P., Blackburn, K., Wenaus, T., & Würthwein, F. (2007). The open science grid. Journal of Physics: Conference Series IOP Publishing, 12057, 140–146.

    Google Scholar 

  • Resnik, D. B., Neal, T., Raymond, A., & Kissling, G. E. (2015). Research misconduct definitions adopted by U.S. research institutions: Introduction. Accountability in Research, 22(1), 14–21. https://doi.org/10.1080/08989621.2014.891943.

    Article  PubMed  PubMed Central  Google Scholar 

  • Rowhani-Farid, A., & Barnett, A. G. (2016). Has open data arrived at the British Medical Journal (BMJ)? An observational study. BMJ Open, 6, e011784.

    Article  PubMed  PubMed Central  Google Scholar 

  • Savage, C. J., & Vickers, A. J. (2009). Empirical study of data sharing by authors publishing in PLoS journals. PLoS One, 4, e7078.

    Article  PubMed  PubMed Central  Google Scholar 

  • Schmidt, F. L., & Hunter, J. E. (2015). Methods of meta-analysis: Correcting error and bias in research findings (3rd ed.). Newbury Park: Sage.

    Book  Google Scholar 

  • Schmidt, F. L., & Oh, I.-S. (2016). The crisis of confidence in research findings in psychology: Is lack of replication the real problem? Or is it something else? Archives of Scientific Psychology, 4(1), 32.

  • Schwab, A., & Starbuck, W. (2017). A call for openness in research reporting: How to turn covert practices into helpful tools. Academy of Management Learning & Education, 16, 125–141.

    Article  Google Scholar 

  • Spector, P. E., Rogelberg, S. G., Ryan, A. M., Schmitt, N., & Zedeck, S. (2014). Moving the pendulum back to the middle: Reflections on and introduction to the inductive research special issue of Journal of Business and Psychology. Journal of Business and Psychology, 29, 499–502.

    Article  Google Scholar 

  • Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22, 1359–1366.

    Article  PubMed  Google Scholar 

  • Sliter, M., Yuan, Z., & Boyd, E. M. (2013). Let’s be honest: Evidence for why industrial–organizational psychology research is trustworthy. Industrial and Organizational Psychology, 6, 273–276.

    Article  Google Scholar 

  • Sterling, T. D., Rosenbaum, W. L., & Weinkam, J. J. (1995). Publication decisions revisited: The effect of the outcome of statistical tests on the decision to publish and vice versa. The American Statistician, 49, 108–112.

    Google Scholar 

  • Tanner, A. (2013). How a zip code can tell a marketer exactly who you are. Forbes Retrieved from http://www.forbes.com/sites/adamtanner/2013/07/22/how-just-a-zip-code-can-tell-a-marketer-exactly-who-you-are/#1a25491b12a7

  • Tenopir, C., Allard, S., Douglass, K., Aydinoglu, A. U., Wu, L., Read, E., Manoff, M., & Frame, M. (2011). Data sharing by scientists: Practices and perceptions. PLoS One, 6, e21101.

    Article  PubMed  PubMed Central  Google Scholar 

  • Vermeulen, F. (2005). On rigor and relevance: Fostering dialectic progress in management research. Academy of Management Journal, 48, 978–982.

    Article  Google Scholar 

  • Wagenmakers, E.-J. & Dutilh, G. (2016). Seven selfish-reasons for pre-registration. Association for Psychological Science, 1–6.

  • Wagenmakers, E.-J., Wetzels, R., Borsboom, D., van der Maas, H. L., & Kievit, R. A. (2012). An agenda for purely confirmatory research. Perspectives on Psychological Science, 7, 632–638.

    Article  PubMed  Google Scholar 

  • Wetzels, R., Matzke, D., Lee, M. D., Rouder, J. N., Iverson, G. J., & Wagenmakers, E.-J. (2011). Statistical evidence in experimental psychology: An empirical comparison using 855 t-tests. Perspectives on Psychological Science, 6, 291–298.

    Article  PubMed  Google Scholar 

  • Wicherts, J. M. (2016). Data re-analysis and open data. In J. Plucker & M. Makel (Eds.), Doing good social science: Trust, accuracy, transparency. American Psychological Association: Washington.

    Google Scholar 

  • Wicherts, J. M., & Bakker, M. (2012). Publish (your data) or (let the data) perish! Why not publish your data too? Intelligence, 40, 73–76.

    Article  Google Scholar 

  • Wicherts, J. M., Bakker, M., & Molenaar, D. (2011). Willingness to share research data is related to the strength of the evidence and the quality of reporting of statistical results. PLoS One, 6, e26828.

    Article  PubMed  PubMed Central  Google Scholar 

  • Wicherts, J. M., Borsboom, D., Kats, J., & Molenaar, D. (2006). The poor availability of psychological research data for reanalysis. American Psychologist, 61, 726–728.

    Article  PubMed  Google Scholar 

  • Woo, S. E., O'Boyle, E. H., & Spector, P. E. (in press). Best practices in developing, conducting, and evaluating inductive research. Human Resource Management Review.

Download references

Author information

Authors and Affiliations

  1. University of North Carolina at Charlotte, Charlotte, NC, 28223, USA

    George C. Banks & Steven G. Rogelberg

  2. West Virginia University, Morgantown, WV, USA

    James G. Field

  3. Rice University, Houston, TX, USA

    Frederick L. Oswald

  4. Indiana University, Bloomington, IN, USA

    Ernest H. O’Boyle

  5. Illinois Institute of Technology, Chicago, IL, USA

    Ronald S. Landis

  6. Purdue University, West Lafayette, IN, USA

    Deborah E. Rupp

Authors
  1. George C. Banks
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. James G. Field
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Frederick L. Oswald
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ernest H. O’Boyle
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Ronald S. Landis
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Deborah E. Rupp
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Steven G. Rogelberg
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to George C. Banks.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Banks, G.C., Field, J.G., Oswald, F.L. et al. Answers to 18 Questions About Open Science Practices. J Bus Psychol 34, 257–270 (2019). https://doi.org/10.1007/s10869-018-9547-8

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10869-018-9547-8

Keywords

Search

Navigation