Advertisement

Journal of Business and Psychology

, Volume 34, Issue 3, pp 257–270 | Cite as

Answers to 18 Questions About Open Science Practices

  • George C. BanksEmail author
  • James G. Field
  • Frederick L. Oswald
  • Ernest H. O’Boyle
  • Ronald S. Landis
  • Deborah E. Rupp
  • Steven G. Rogelberg
Original Paper

Abstract

Open science refers to an array of practices that promote openness, integrity, and reproducibility in research; the merits of which are being vigorously debated and developed across academic journals, listservs, conference sessions, and professional associations. The current paper identifies and clarifies major issues related to the use of open science practices (e.g., data sharing, study pre-registration, open access journals). We begin with a useful general description of what open science in organizational research represents and adopt a question-and-answer format. Through this format, we then focus on the application of specific open science practices and explore future directions of open science. All of this builds up to a series of specific actionable recommendations provided in conclusion, to help individual researchers, reviewers, journal editors, and other stakeholders develop a more open research environment and culture.

Keywords

Open science Philosophy of science Questionable research practices Research ethics 

References

  1. American Psychological Association. (2010). Publication manual of the American Psychological Association (6th ed.). Washington, DC: American Psychological Association.Google Scholar
  2. Anderson, K. (2016). 96 things pubishers do. The scholarly kitchen. Retrieved on March 9, 2017 from https://scholarlykitchen.sspnet.org/2016/02/01/guest-post-kent-anderson-updated-96-things-publishers-do-2016-edition/
  3. Anderson, M. S., Martinson, B. C., & De Vries, R. (2007). Normative dissonance in science: Results from a national survey of US scientists. Journal of Empirical Research on Human Research Ethics, 2, 3–14.CrossRefGoogle Scholar
  4. Antonakis, J. (2017). On doing better science: From thrill of discovery to policy implications. The Leadership Quarterly, 28, 5–21.CrossRefGoogle Scholar
  5. Baker, M. (2016). Is there a reproducibility crisis? A nature survey lifts the lid on how researchers view the crisis rocking science and what they think will help. Nature, 533, 452–455.CrossRefGoogle Scholar
  6. Bakker, M., van Dijk, A., & Wicherts, J. M. (2012). The rules of the game called psychological science. Perspectives on Psychological Science, 7, 543–554.Google Scholar
  7. Bakker, M., & Wicherts, J. M. (2014). Outlier removal and the relation with reporting errors and quality of psychological research. PLos One.Google Scholar
  8. Bandura, A. (2001). Social cognitive theory: An agentic perspective. Annual Review of Psychology, 52, 1–26.CrossRefGoogle Scholar
  9. Banks, G. C., Kepes, S., & McDaniel, M. A. (2015). Publication bias: Understanding the myths concerning threats to the advancement of science. In C. E. Lance & R. J. Vandenberg (Eds.), Statistical and methodological myths and urban legends: Doctrine, verity, and fable in organizational and social sciences (pp. 36–64). New York: Routledge.Google Scholar
  10. Banks, G. C., & McDaniel, M. A. (2011). The kryptonite of evidence-based I-O psychology. Industrial and Organizational Psychology: Perspectives on Science and Practice, 4(1), 40–44.  https://doi.org/10.1111/j.1754-9434.2010.01292.x.CrossRefGoogle Scholar
  11. Banks, G. C., & O’Boyle Jr., E. H. (2013). Why we need I-O psychology to fix I-O psychology. Industrial and Organizational Psychology: Perspectives on Science and Practice, 6, 291–294.CrossRefGoogle Scholar
  12. Banks, G. C., O’Boyle Jr., E., Pollack, J. M., White, C. D., Batchelor, J. H., Whelpley, C. E., Abston, K. A., Bennett, A. A., & Adkins, C. L. (2016). Questions about questionable research practices in the field of management: A guest commentary. Journal of Management, 42, 5–20.CrossRefGoogle Scholar
  13. Banks, G. C., Pollack, J. M., Bochantin, J. E., Kirkman, B. L., Whelpley, C. E., & O’Boyle, E. H. (2016). Management’s science practice gap: A grand challenge for all stakeholders. Academy of Management Journal, 59, 1–27.CrossRefGoogle Scholar
  14. Banks, G. C., Rogelberg, S. G., Woznyj, H. M., Landis, R. S., & Rupp, D. E. (2016). Evidence on questionable research practices: The good, the bad, and the ugly. Journal of Business and Psychology, 31, 323–338.CrossRefGoogle Scholar
  15. Bedeian, A. G., Taylor, S. G., & Miller, A. N. (2010). Management science on the credibility bubble: Cardinal sins and various misdemeanors. Academy of Management Learning & Education, 9, 715–725.Google Scholar
  16. Benjamin, D. J., Berger, J. O., Johannesson, M., Nosek, B. A., Wagenmakers, E.-J., Berk, R., … Camerer, C. (2017). Redefine statistical significance. Nature Human Behaviour.Google Scholar
  17. Bhattacharjee, Y. (2013). The mind of a con man. The New York Times. Retrieved March 9, 2017 from http://www.nytimes.com/2013/04/28/magazine/diederik-stapels-audacious-academic-fraud.html?pagewanted=all&_r=0.Google Scholar
  18. Biemann, T. (2013). What if we were Texas sharpshooters? Predictor reporting bias in regression analysis. Organizational Research Methods, 16, 335–363.CrossRefGoogle Scholar
  19. Bosco, F. A., Aguinis, H., Field, J. G., Pierce, C. A., & Dalton, D. R. (2016). HARKing’s threat to organizational research: Evidence from primary and meta-analytic sources. Personnel Psychology, 69, 709–750.CrossRefGoogle Scholar
  20. Bosco, F. A., Aguinis, H., Singh, K., Field, J. G., & Pierce, C. A. (2015). Correlational effect size benchmarks. Journal of Applied Psychology, 100, 431–449.CrossRefGoogle Scholar
  21. Bosco, F. A., Steel, P., Oswald, F. L., Uggerslev, K., & Field, J. G. (2015). Cloud-based meta-analysis to bridge science and practice: Welcome to metaBUS. Personnel Assessment and Decisions, 1, 3–17.CrossRefGoogle Scholar
  22. Butler, D. (2016). Dutch lead European push to flip journals to open access. Nature News, 529(7584), 13.CrossRefGoogle Scholar
  23. Carey, B. (2015). Journal science releases guidelines for publishing scientific studies. The New York Times. Retrieved from http://www.nytimes.com/2015/06/26/science/journal-science-releases-guidelines-for-publishing-scientific-studies.html?_r=0.
  24. Carpenter, N. C., Son, J., Harris, T. B., Alexander, A. L., & Horner, M. T. (2016). Don’t forget the items: Item-level meta-analytic and substantive validity techniques for reexamining scale validation. Organizational Research Methods, 19, 616–650.CrossRefGoogle Scholar
  25. Cashen, L. H., & Geiger, S. W. (2004). Statistical power and the testing of null hypotheses: A review of contemporary management research and recommendations for future studies. Organizational Research Methods, 7, 151–167.CrossRefGoogle Scholar
  26. Caulfield, T., Harmon, S. H., & Joly, Y. (2012). Open science versus commercialization: A modern research conflict? Genome Medicine, 4, 1.CrossRefGoogle Scholar
  27. Cortina, J. M. (2015). A revolution with a solution. Philadelphia, PA: Opening plenary presented at the meeting of the Society for Industrial/Organizational Psychology.Google Scholar
  28. Cortina, J. M., Green, J. P., Keeler, K. R., & Vandenberg, R. J. (2017). Degrees of freedom in SEM: Are we testing the models that we claim to test? Organizational Research Methods, 20, 350–378.CrossRefGoogle Scholar
  29. Derksen, M., & Rietzschel, E. F. (2013). Surveillance is not the answer, and replication is not a test: Comment on Kepes and McDaniel,“How trustworthy is the scientific literature in I–O psychology?”. Industrial and Organizational Psychology, 6, 295–298.CrossRefGoogle Scholar
  30. Donnellan, M. B., Lucas, R. E., & Cesario, J. (2015). On the association between loneliness and bathing habits: Nine replications of Bargh and Shalev (2012) Study 1. Emotion, 15, 109–119.CrossRefGoogle Scholar
  31. Eich, E. (2014). Business not as usual. Psychological Science, 25, 3–6.CrossRefGoogle Scholar
  32. Ethiraj, S. K., Gambardella, A., & Helfat, C. E. (2016). Replication in strategic management. Strategic Management Journal, 37, 2191–2192.CrossRefGoogle Scholar
  33. Fang, F. C., & Casadevall, A. (2015). Competitive science: Is competition ruining science? Infection and Immunity: IAI. 02939–02914.Google Scholar
  34. Findley, M. G., Jensen, N. M., Malesky, E. J., & Pepinsky, T. B. (2016). Can results-free review reduce publication bias? The results and implications of a pilot study. Comparative Political Studies, 1–37.Google Scholar
  35. Friesike, S., Widenmayer, B., Gassmann, O., & Schildhauer, T. (2015). Opening science: Towards an agenda of open science in academia and industry. The Journal of Technology Transfer, 40, 581–601.CrossRefGoogle Scholar
  36. Gabriel, A. S., & Wessel, J. L. (2013). A step too far? Why publishing raw datasets may hinder data collection. Industrial and Organizational Psychology: Perspectives on Science and Practice, 6, 287–290.CrossRefGoogle Scholar
  37. Grahe, J. E. (2014). Announcing open science badges and reaching for the sky. The Journal of Social Psychology, 154, 1–3.CrossRefGoogle Scholar
  38. Grand, J. A., Rogelberg, S. G., Allen, T. D., Landis, R. S., Reynolds, D. H., Scott, J. C., Tonidandel, S., Truxillo, D. M. (2017). A systems-based approach to fostering robust science in industrial-organizational psychology. Industrial and Organizational Psychology: Perspectives on Science and Practice. 1–39.Google Scholar
  39. Grand, J. A., Rogelberg, S. G., Banks, G. C., Landis, R. S., & Tonidandel, S. (in press). From outcome to process focus: Fostering a more robust psychological science through registered reports and results-blind reviewing. Perspectives on Psychological Science.Google Scholar
  40. Healy, L. W. (2015). Tipping point: Information industry outlook. Retrieved from http://info.outsellinc.com/rs/422-MBV-091/images/Outsell_CEO_Topics_01oct2015_Info_Industry_Outlook_2016_Tipping_Point.pdf.
  41. Hollenbeck, J. R., & Wright, P. M. (2017). Harking, sharking, and tharking: Making the case for post hoc analysis of scientific data. Journal of Management, 43, 5–18.CrossRefGoogle Scholar
  42. Internal Revenue Service. (2014). Form 990: Return of organization exempt from income tax. Retrieved from http://www.irs.gov/pub/irs-pdf/f990.pdf.
  43. Ioannidis, J. P. A. (2005). Why most published research findings are false. PLoS Medicine, 2, e124.CrossRefGoogle Scholar
  44. Janssen, M., Charalabidis, Y., & Zuiderwijk, A. (2012). Benefits, adoption barriers and myths of open data and open government. Information Systems Management, 29, 258–268.CrossRefGoogle Scholar
  45. Jebb, A. T., Parrigon, S., & Woo, S. (in press). Exploratory data analysis as a foundation of inductive research. Human Resource Management Review.Google Scholar
  46. John, L. K., Loewenstein, G., & Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science, 23(5), 524–532.  https://doi.org/10.1177/0956797611430953.
  47. Jones, J. W., & Dages, K. D. (2013). A new era of large-scale data sharing: A test publisher’s perspective. Industrial and Organizational Psychology, 6, 309–312.CrossRefGoogle Scholar
  48. Kepes, S., Banks, G., C., McDaniel, M. A., & Whetzel, D. L. (2012). Publication bias in the organizational sciences. Organizational Research Methods, 15, 624–662.  https://doi.org/10.1177/1094428112452760.
  49. Kepes, S., & McDaniel, M. A. (2013). How trustworthy is the scientific literature in I-O psychology? Industrial and Organizational Psychology: Perspectives on Science and Practice, 6, 252–268.CrossRefGoogle Scholar
  50. Kerr, N. L. (1998). HARKing: Hypothesizing after the results are known. Personality and Social Psychology Review, 2, 196–217.  https://doi.org/10.1207/s15327957pspr0203_4.CrossRefGoogle Scholar
  51. Korn, M. (2014). Management research is fish, says new management research. The Wall Street Journal. Retrieved from http://blogs.wsj.com/atwork/2014/02/18/management-research-is-fishy-says-new-management-research/.
  52. Lakens, D., Adolfi, F. G., Albers, C. J., Anvari, F., Apps, M. A. J., Argamon, S. E., … Zwaan, R. A. (2017, September 18). Justify your alpha: A response to “redefine statistical significance”. Retrieved from psyarxiv.com/9s3y6
  53. Leavitt, K. (2013). Publication bias might make us untrustworthy, but the solutions may be worse. Industrial and Organizational Psychology: Perspectives on Science and Practice, 6, 290–295.CrossRefGoogle Scholar
  54. LeBreton, J. M. (2016). Editorial. Organizational Research Methods, 19, 3–7.CrossRefGoogle Scholar
  55. Locke, E. A. (2007). The case for inductive theory building. Journal of Management, 33, 867–890.CrossRefGoogle Scholar
  56. Lykken, D. T. (1968). Statistical significance in psychological research. Psychological Bulletin, 70, 151–159.CrossRefGoogle Scholar
  57. Maxwell, S. E. (2004). The persistence of underpowered studies in psychological research: Causes, consequences, and remedies. Psychological Methods, 9, 147–163.CrossRefGoogle Scholar
  58. McCook, A. (2016). Duke fraud case highlights financial risks for universities. Science, 353, 977–978.CrossRefGoogle Scholar
  59. McKiernan, E. C., Bourne, P. E., Brown, C. T., Buck, S., Kenall, A., Lin, J., et al. (2016). How open science helps researchers succeed. eLife, 5, e16800.CrossRefGoogle Scholar
  60. Munafò, M. R., Nosek, B. A., Bishop, D. V., Button, K. S., Chambers, C. D., du Sert, N. P., Simonsohn, U., Wagenmakers, E.-J., Ware, J. J., & Ioannidis, J. P. (2017). A manifesto for reproducible science. Nature Human Behaviour, 1, 0021.CrossRefGoogle Scholar
  61. Nosek, B. A., Alter, G., Banks, G. C., Borsboom, D., Bowman, S. D., Breckler, S., Buck, S., Chambers, C., Chin, G., Christensen, G., Contestabile, M., Dafoe, A., Eich, E., Freese, J., Glennerster, R., Goroff, D., Green, D. P., Heese, B., Humphreys, M., Ishiyama, J., Karlan, D., Kraut, A., Lupia, A., Marbry, P., Madon, T., Malhotra, N., Wilson, E. M., McNutt, M., Miguel, E., Paluck, E. L., Simonsohn, U., Soderberg, C., Spellman, B. A., Tornow, J., Turitto, J., VandenBos, G. R., Vazire, S., Wagenmakers, E. J., Wilson, R., & Yarkoni, T. (2015). Promoting an open research culture: Author guidelines for journals to promote transparency, openness, and reproducibility. Science, 348, 1422–1425.CrossRefGoogle Scholar
  62. Nosek, B. A., & Bar-Anan, Y. (2012). Scientific utopia: I. Opening scientific communication. Psychological Inquiry, 23, 217–243.CrossRefGoogle Scholar
  63. Nosek, B. A., Spies, J. R., & Motyl, M. (2012). Scientific utopia: II. Restructuring incentives and practices to promote truth over publishability. Perspectives on Psychological Science, 7, 615–631.CrossRefGoogle Scholar
  64. Nuzzo, R. (2015). How scientists fool themselves-and how they can stop. Nature, 526, 182–185.CrossRefGoogle Scholar
  65. O'Boyle, E. H., Banks, G. C., Carter, K., Walter, S., & Yuan, Z. (2018). A 20-year review of outcome reporting bias in moderated multiple regression. Journal of Business and Psychology.  https://doi.org/10.1007/s10869-018-9539-8.
  66. O'Boyle, E. H., Banks, G. C., & Gonzalez-Mule, E. (2017). The Chrysalis effect: How ugly initial results metamorphosize into beautiful articles. Journal of Management, 43, 400–425.Google Scholar
  67. Office of Science and Technology Policy. (2000). Federal research misconduct policy. Federal Register., 65(235), 76260–76264.Google Scholar
  68. Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349, aac4716.CrossRefGoogle Scholar
  69. Pordes, R., Petravick, D., Kramer, B., Olson, D., Livny, M., Roy, A., Avery, P., Blackburn, K., Wenaus, T., & Würthwein, F. (2007). The open science grid. Journal of Physics: Conference Series IOP Publishing, 12057, 140–146.Google Scholar
  70. Resnik, D. B., Neal, T., Raymond, A., & Kissling, G. E. (2015). Research misconduct definitions adopted by U.S. research institutions: Introduction. Accountability in Research, 22(1), 14–21.  https://doi.org/10.1080/08989621.2014.891943.CrossRefGoogle Scholar
  71. Rowhani-Farid, A., & Barnett, A. G. (2016). Has open data arrived at the British Medical Journal (BMJ)? An observational study. BMJ Open, 6, e011784.CrossRefGoogle Scholar
  72. Savage, C. J., & Vickers, A. J. (2009). Empirical study of data sharing by authors publishing in PLoS journals. PLoS One, 4, e7078.CrossRefGoogle Scholar
  73. Schmidt, F. L., & Hunter, J. E. (2015). Methods of meta-analysis: Correcting error and bias in research findings (3rd ed.). Newbury Park: Sage.CrossRefGoogle Scholar
  74. Schmidt, F. L., & Oh, I.-S. (2016). The crisis of confidence in research findings in psychology: Is lack of replication the real problem? Or is it something else? Archives of Scientific Psychology, 4(1), 32.Google Scholar
  75. Schwab, A., & Starbuck, W. (2017). A call for openness in research reporting: How to turn covert practices into helpful tools. Academy of Management Learning & Education, 16, 125–141.CrossRefGoogle Scholar
  76. Spector, P. E., Rogelberg, S. G., Ryan, A. M., Schmitt, N., & Zedeck, S. (2014). Moving the pendulum back to the middle: Reflections on and introduction to the inductive research special issue of Journal of Business and Psychology. Journal of Business and Psychology, 29, 499–502.CrossRefGoogle Scholar
  77. Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22, 1359–1366.CrossRefGoogle Scholar
  78. Sliter, M., Yuan, Z., & Boyd, E. M. (2013). Let’s be honest: Evidence for why industrial–organizational psychology research is trustworthy. Industrial and Organizational Psychology, 6, 273–276.CrossRefGoogle Scholar
  79. Sterling, T. D., Rosenbaum, W. L., & Weinkam, J. J. (1995). Publication decisions revisited: The effect of the outcome of statistical tests on the decision to publish and vice versa. The American Statistician, 49, 108–112.Google Scholar
  80. Tanner, A. (2013). How a zip code can tell a marketer exactly who you are. Forbes Retrieved from http://www.forbes.com/sites/adamtanner/2013/07/22/how-just-a-zip-code-can-tell-a-marketer-exactly-who-you-are/#1a25491b12a7
  81. Tenopir, C., Allard, S., Douglass, K., Aydinoglu, A. U., Wu, L., Read, E., Manoff, M., & Frame, M. (2011). Data sharing by scientists: Practices and perceptions. PLoS One, 6, e21101.CrossRefGoogle Scholar
  82. Vermeulen, F. (2005). On rigor and relevance: Fostering dialectic progress in management research. Academy of Management Journal, 48, 978–982.CrossRefGoogle Scholar
  83. Wagenmakers, E.-J. & Dutilh, G. (2016). Seven selfish-reasons for pre-registration. Association for Psychological Science, 1–6.Google Scholar
  84. Wagenmakers, E.-J., Wetzels, R., Borsboom, D., van der Maas, H. L., & Kievit, R. A. (2012). An agenda for purely confirmatory research. Perspectives on Psychological Science, 7, 632–638.CrossRefGoogle Scholar
  85. Wetzels, R., Matzke, D., Lee, M. D., Rouder, J. N., Iverson, G. J., & Wagenmakers, E.-J. (2011). Statistical evidence in experimental psychology: An empirical comparison using 855 t-tests. Perspectives on Psychological Science, 6, 291–298.CrossRefGoogle Scholar
  86. Wicherts, J. M. (2016). Data re-analysis and open data. In J. Plucker & M. Makel (Eds.), Doing good social science: Trust, accuracy, transparency. American Psychological Association: Washington.Google Scholar
  87. Wicherts, J. M., & Bakker, M. (2012). Publish (your data) or (let the data) perish! Why not publish your data too? Intelligence, 40, 73–76.CrossRefGoogle Scholar
  88. Wicherts, J. M., Bakker, M., & Molenaar, D. (2011). Willingness to share research data is related to the strength of the evidence and the quality of reporting of statistical results. PLoS One, 6, e26828.CrossRefGoogle Scholar
  89. Wicherts, J. M., Borsboom, D., Kats, J., & Molenaar, D. (2006). The poor availability of psychological research data for reanalysis. American Psychologist, 61, 726–728.CrossRefGoogle Scholar
  90. Woo, S. E., O'Boyle, E. H., & Spector, P. E. (in press). Best practices in developing, conducting, and evaluating inductive research. Human Resource Management Review.Google Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  • George C. Banks
    • 1
    Email author
  • James G. Field
    • 2
  • Frederick L. Oswald
    • 3
  • Ernest H. O’Boyle
    • 4
  • Ronald S. Landis
    • 5
  • Deborah E. Rupp
    • 6
  • Steven G. Rogelberg
    • 1
  1. 1.University of North Carolina at CharlotteCharlotteUSA
  2. 2.West Virginia UniversityMorgantownUSA
  3. 3.Rice UniversityHoustonUSA
  4. 4.Indiana UniversityBloomingtonUSA
  5. 5.Illinois Institute of TechnologyChicagoUSA
  6. 6.Purdue UniversityWest LafayetteUSA

Personalised recommendations