Grounds for Ambiguity: Justifiable Bases for Engaging in Questionable Research Practices

Abstract

The current study sought to determine research scientists’ sensitivity to various justifications for engaging in behaviors typically considered to be questionable research practices (QRPs) by asking them to evaluate the appropriateness and ethical defensibility of each. Utilizing a within-subjects design, 107 National Institutes of Health principal investigators responded to an invitation to complete an online survey in which they read a series of research behaviors determined, in prior research, to either be ambiguous or unambiguous in their ethical defensibility. Additionally, each behavior was paired with either an ostensibly sound or unsound reason for the behavior. Consistent with hypotheses, the results indicated that scientists perceived QRPs as more appropriate and defensible when paired with a justifiable motive relative to when paired with a clearly unethical motive, particularly for QRPs that are more ambiguous in their ethicality. In fact, ambiguous QRPs were perceived as categorically defensible when given a justifiable motive. This suggests scientists are sensitive to contextual factors related to QRPs’ appropriateness, which could inform how institutions develop appropriate training modules for research integrity.

This is a preview of subscription content, log in to check access.

Fig. 1

Notes

  1. 1.

    A recent report by the National Academies of Science, Engineering, and Medicine refer to QRPs as Detrimental Research Practices, thereby emphasizing the potential deleterious consequences of these behaviors on scientific research (NASEM 2017).

  2. 2.

    This research project was reviewed and approved by the University of Southern Mississippi Institutional Review Board (Protocol Number: CH2-16110904).

  3. 3.

    Reliabilities were computed for both types of motives for UU- and AU-QRPs for the appropriateness and defensibility scales. For UU-QRPs’ appropriateness, both justifiable (α = 0.33) and unjustifiable motives (α = 0.33) produced low reliabilities. The reliabilities were also low in defensibility for justifiable (α = 0.41) and unjustifiable motives (α = 0.41). With AU-QRPs’ appropriateness, justifiable motives elicited a low reliability (α = 0.58), whereas unjustifiable was acceptable (α = 0.70). For defensibility, unjustifiable (α = 0.74) and justifiable motives (α = 0.64) were acceptable. The low reliabilities of the UU-QRP items were not substantially improved when removing less reliable items from the analyses (αs < 0.50) and the justifiable motives reliability for AU-QRPs did not improve following removal of any single item (αs < 0.59). Because reliabilities did not improve into an acceptable range for any of the low-reliability subscales (Cronbach and Meehl 1955), and we wanted to eliminate as much of an imbalance between responses for subscales as possible, we opted to report analyses including all tested items. These low alphas, particularly for ostensibly justifiable motives, may reflect the actual variability in perceptions of how acceptable certain research practices are, even when considering these motives.

References

  1. Anderson, M. S., Horn, A. S., Risbey, K. R., Ronning, E. A., De Vries, R., & Martinson, B. C. (2007). What do mentoring and training in the responsible conduct of research have to do with scientists’ misbehavior? Findings from a national survey of NIH-funded scientists. Academic Medicine, 82, 853–860.

    Article  Google Scholar 

  2. Cronbach, L. J., & Meehl, P. E. (1955). Construct validity in psychological tests. Psychological Bulletin, 52(4), 281–302.

    Article  Google Scholar 

  3. Fanelli, D. (2010). Do pressures to publish increase scientists’ bias? An empirical support from US States Data. PLoS ONE, 5, e10271.

    Article  Google Scholar 

  4. Fanelli, D. (2011). Negative results are disappearing from most disciplines and countries. Scientometrics, 90, 891–904.

    Article  Google Scholar 

  5. Fanelli, D., Costas, R., & Larivière, V. (2015). Misconduct policies, academic culture and career stage, not gender or pressures to publish, affect scientific integrity. PLoS ONE, 10, e0127556.

    Article  Google Scholar 

  6. Festinger, L., & Carlsmith, J. M. (1959). Cognitive consequences of forced compliance. The Journal of Abnormal and Social Psychology, 58, 203–210.

    Article  Google Scholar 

  7. Franco, A., Malhotra, N., & Simonovits, G. (2014). Publication bias in the social sciences: Unlocking the file drawer. Science, 345, 1502–1505.

    Article  Google Scholar 

  8. Goldstein, N. J., Griskevicius, V., & Cialdini, R. B. (2007). Invoking social norms: A social psychology perspective on improving hotels’ linen-reuse programs. Cornell Hotel and Restaurant Administration Quarterly, 48, 145–150.

    Article  Google Scholar 

  9. Ioannidis, J. P. (2012). Why science is not necessarily self-correcting. Perspectives on Psychological Science, 7, 645–654.

    Article  Google Scholar 

  10. John, L. K., Loewenstein, G., & Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science, 5, 524–532.

    Article  Google Scholar 

  11. LaRose, R., & Kim, J. (2006). Share, steal, or buy? A social cognitive perspective of music downloading. CyberPsychology & Behavior, 10, 267–277.

    Article  Google Scholar 

  12. Leary, M. R., & Kowalski, R. M. (1990). Impression management: A literature review and two-component model. Psychological Bulletin, 107, 34–47.

    Article  Google Scholar 

  13. Loken, E., & Gelman, A. (2017). Measurement error and the replication crisis. Science, 355, 584–585.

    Article  Google Scholar 

  14. Lynott, D., Corker, K. S., Wortman, J., Connell, L., Donnellan, M. B., Lucas, R. E., et al. (2014). Replication of “Experiencing physical warmth promotes interpersonal warmth” by Williams and Bargh (2008). Social Psychology, 45, 216–222.

    Article  Google Scholar 

  15. May, H. (2012). Nonequivalent comparison group designs. In H. Cooper, P. M. Camic, D. L. Long, A. T. Panter, D. Rindskopf, & K. J. Sher (Eds.), APA handbook of research methods in psychology, Vol 2: Research designs—Quantitative, qualitative, neuropsychological, and biological (pp. 489–509). Washington, DC: American Psychological Association.

    Google Scholar 

  16. National Academies of Science, Engineering, and Medicine. (2017). U.S. scientific research enterprise should take action to protect integrity in research; New advisory board on research integrity should be established. http://www8.nationalacademies.org/onpinews/newsitem.aspx?RecordID=21896. Accessed 23 July 2018.

  17. Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 34, aac4716.

    Article  Google Scholar 

  18. Paulhus, D. L. (2002). Socially desirable responding: The evolution of a construct. The Role of Constructs in Psychological and Educational Measurement, 4969, 49–69.

    Google Scholar 

  19. Prelec, D. (2004). A Bayesian truth serum for subjective data. Science, 306, 462–466.

    Article  Google Scholar 

  20. Ramalingam, S., Bhuvaneswari, S., & Sankaran, R. (2014). Ethics workshops-are they effective in improving the competencies of faculty and postgraduates? Journal of Clinical and Diagnostic Research: JCDR, 8, XC01.

    Google Scholar 

  21. Ritchie, S. J., Wiseman, R., & French, C. C. (2012). Failing the future: Three unsuccessful attempts to replicate Bem’s ‘Retroactive Facilitation of Recall’ effect. PLoS ONE, 7, e33423.

    Article  Google Scholar 

  22. Roig, M. (2015). Avoiding plagiarism, self-plagiarism, and other questionable writing practices: A guide to ethical writing. Office of Research Integrity. https://ori.hhs.gov/education/products/roig_st_johns/Salami%20slicing.html. Accessed 7 Mar 2018.

  23. Sacco, D. F., Bruton, S. V., & Brown, M. (2018). In defense of the questionable: Defining the basis of research scientists’ engagement in questionable research practices. Journal of Empirical Research on Human Research Ethics, 13, 101–110.

    Article  Google Scholar 

  24. Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22, 1359–1366.

    Article  Google Scholar 

  25. Steele, K. M., Bass, K. E., & Crook, M. D. (1999). The mystery of the Mozart effect: Failure to replicate. Psychological Science, 10, 366–369.

    Article  Google Scholar 

  26. Tijdink, J. K., Verbeke, R., & Smulders, Y. M. (2014). Publication pressure and scientific misconduct in medical scientists. Journal of Empirical Research on Human Research Ethics, 9, 64–71.

    Article  Google Scholar 

  27. Todd, E. M., Torrence, B. S., Watts, L. L., Mulhearn, T. J., Connelly, S., & Mumford, M. D. (2017). Effective practices in the delivery of research ethics education: A qualitative review of instructional methods. Accountability in Research, 24, 297–321.

    Article  Google Scholar 

  28. Watts, L. L., Mulhearn, T. J., Medeiros, K. E., Steele, L. M., Connelly, S., & Mumford, M. D. (2017). Modeling the instructional effectiveness of responsible conduct of research education: A meta-analytic path-analysis. Ethics and Behavior, 27, 632–650.

    Article  Google Scholar 

  29. Wicherts, J. M., Veldkamp, C. L., Augusteijn, H. E., Bakker, M., Van Aert, R., & Van Assen, M. A. (2016). Degrees of freedom in planning, running, analyzing, and reporting psychological studies: A checklist to avoid p-hacking. Frontiers in Psychology, 7, 1832.

    Article  Google Scholar 

Download references

Funding

The authors disclose that this research was funded by grants awarded to the first and third author from the Department of Health and Human Services’ Office of Research Integrity (Grant Nos. 1 ORIIR170035-01-00 and 1 ORIIR160021-01-00).

Author information

Affiliations

Authors

Corresponding author

Correspondence to Donald F. Sacco.

Appendix

Appendix

UU-QRPs with Unjustifiable Motives

  1. 1.

    A researcher is coauthoring a manuscript using data he collected and used previously for another study and separate paper. He does not indicate the origin of his data, because the hypotheses and variables analyzed in the current manuscript differ from those in the prior work.

  2. 2.

    A researcher submits a paper for publication without receiving direct permission from all coauthors, since as the lead author, she believed she could speak for all of her coauthors.

  3. 3.

    A researcher investigating the preliminary effects of a new pharmaceutical withholds journal submission of a manuscript at the request of the corporate sponsor, because the sponsor wants the paper’s publication to coincide with the release of the new drug.

  4. 4.

    A researcher’s coauthor wants to conduct his own analyses on some data recently collected by the research team, but the Principal Investigator refuses to share the raw dataset to ensure that he alone has access to the findings.

  5. 5.

    A researcher fails to replicate a finding that she found recently in another study. She decides to publish the results from the first study alone, without reporting the failed replication attempt, since she is confident that the first study represents the truth.

  6. 6.

    An employee of a corporate sponsor of the research analyzes the data for publication and is listed as one of the study’s co-authors. Though the sponsor’s involvement in the project is clearly indicated in the manuscript, the employee’s relationship to the sponsor is not disclosed.

UU-QRPs with Justifiable Motives

  1. 1.

    A researcher is coauthoring a manuscript using data he collected and used previously for another study and separate paper. He does not indicate the origin of his data, because his graduate school mentor used to do the same thing, leading him to infer that this is an acceptable practice.

  2. 2.

    A researcher submits a paper for publication without first receiving direct permission from all coauthors, because the coauthors had approved earlier, similar drafts.

  3. 3.

    A researcher investigating the preliminary effects of a new pharmaceutical withholds journal submission of a manuscript at the request of the corporate sponsor, because the sponsor wants additional evidence that the drug has no significant side effects.

  4. 4.

    A researcher’s coauthor wants to conduct his own analyses on some data recently collected by the research team, but the Principal Investigator refuses to share the raw dataset in order to protect participants’ confidential information.

  5. 5.

    A researcher fails to replicate a finding that she found recently in three previous studies. She decides to publish the results from the original three studies alone, without indicating the failed replication attempt, since she is confident that the first three studies represent the truth.

  6. 6.

    An employee of a corporate sponsor of the research analyzes the data for publication and is listed as one of the study’s co-authors. The sponsor’s involvement in the project is clearly indicated in the manuscript, and the employee’s relationship to the sponsor is disclosed.

AU-QRPs with Unjustifiable Motives

  1. 1.

    The results from a researcher’s most recent study are trending toward significance, which prompts the researcher to collect data from an additional ten participants beyond what was estimated from an a priori power analysis. The original data set was sufficiently powered to detect effects and no participants were excluded from analyses. Following this extra data collection, the results become conventionally significant and are then published, but without disclosing the post hoc modification of the research plan.

  2. 2.

    Upon a preliminary analysis of a data set, a researcher observes that she is about 20 observations away from what her original study design called for, but since her hypothesis is already supported by the existing data, she stops data collection to write up the results for publication immediately.

  3. 3.

    So as to meet a target journal’s standards for statistical significance, and because the journal requires reporting p values only to the hundredth place, a researcher reports the p value of a finding as p = 0.05, rounding down the actual significance observed, which was p = 0.053.

  4. 4.

    In a submitted manuscript, a researcher acknowledges a statistician for her technical assistance without asking the statistician for permission to do so, simply assuming the statistician would appreciate the acknowledgement.

  5. 5.

    A researcher publishes statistically significant results, but statistical significance was attained only by excluding data from two participants after the researcher becomes convinced that these participants probably should not have been included in the study to begin with.

  6. 6.

    A researcher conducts a study in which two outcome measures out of four yield significant results. She reports the results of only the two significant measures, and fails to mention the two others, so as to present the results as more consistent and impressive than the entirety of her data supports.

  7. 7.

    A researcher obtains unexpected results from what she predicted for a study and drafts a manuscript reporting these results as having been predicted from the start, reporting the results in the discussion as support for predicted relationships.

  8. 8.

    A researcher conducts a study with two independent variables, with an a priori prediction that there should be a statistical interaction between the two factors. However, the statistical interaction is not conventionally significant. He omits the results of this analysis in his paper in order to conduct post hoc tests, which provide support for more specific predictions, and does not disclose that he predicted a statistical interaction between the two variables.

  9. 9.

    Several of a researcher’s colleagues have questioned why she has not yet submitted for publication a paper they had all recently coauthored. Because she is under pressure from her department chair to publish in high-impact journals, she is delaying submission because she hopes to discuss the paper at an upcoming conference with the editor of a journal she has in mind. She hopes that by delaying publication, it will increase the likelihood of publication in a high-impact journal.

  10. 10.

    A researcher conducts two different but conceptually similar studies that she intends to make into two different papers. For convenience, she re-uses the literature review from the first-drafted paper for the second without disclosing the reuse of this material.

  11. 11.

    So as to garner more publications, a researcher separates out several aspects of a large study he recently conducted, making each an independent manuscript, but without disclosing the relationship of the papers to each other in each manuscript.

  12. 12.

    At the suggestion of a corporate sponsor of the research, a researcher changes her planned methods of statistical analysis, even though she believes her original methods more accurately represent the data.

AU-QRPs with Justifiable Motives

  1. 1.

    The results from a researcher’s most recent study are trending toward significance, which prompts the researcher to collect data from an additional ten participants beyond what was estimated from an a priori power analysis. Ten participants from the initial sample were excluded from analysis after failing an attention check, thus necessitating the additional ten participants for the study to attain adequate statistical power. The results then become conventionally significant and are published; the researcher discloses details regarding post hoc modification of the research plan.

  2. 2.

    Upon a preliminary analysis of a data set, a researcher observes that she is about 20 observations away from what her original study design called for, but since her hypothesis is already supported by the existing data, she stops data collection to preserve unused research funds for a follow-up study that will determine the reliability of the original findings.

  3. 3.

    Because a target journal limits reporting p values to the hundredth place, a researcher reports the p value of a finding as p = 0.05, rounding down the actual significance observed, which was p = 0.053. He acknowledges the finding was marginal but did not reach conventional statistical significance.

  4. 4.

    In a submitted manuscript, a researcher acknowledges a statistician for her technical assistance without asking the statistician for permission to do so, unaware that it is impolite to acknowledge someone without her permission.

  5. 5.

    A researcher publishes statistically significant results, but statistical significance was attained only by excluding data from two participants after a statistical test for outliers confirmed the appropriateness of removing these participants’ data.

  6. 6.

    A researcher conducts a study in which two outcome measures out of four yield significant results. She reports the results of only the two significant measures, but mentions the other two in a description of her methods, despite not reporting findings from them, because these other measures had insufficient reliability.

  7. 7.

    A researcher obtains unexpected results from what she predicted for a study and drafts a manuscript reporting these results as not having been predicted from the start, offering a tentative post hoc explanation for the findings in the paper’s discussion.

  8. 8.

    A researcher conducts a study with two independent variables, with an a priori prediction that there should be a statistical interaction between the two factors. However, the statistical interaction is not conventionally significant. He reports in the manuscript that although the statistical interaction was not significant, he conducted post hoc tests, which supported his more specific predictions, but indicated that such findings should be interpreted cautiously in light of the non-significant statistical interaction.

  9. 9.

    Several of a researcher’s colleagues have questioned why she has not yet submitted for publication a paper they had all recently coauthored. She hopes to collect data from a few more studies to confirm the initial results, which she hopes would help place the paper in a high-impact journal.

  10. 10.

    A researcher conducts two different but conceptually similar studies that she intends to make into two different papers. As she had seen her graduate school mentor do several times, she re-uses the literature review from the first-drafted paper for the second without disclosing the reuse of this material.

  11. 11.

    Consistent with what often occurs in his field, a researcher separates out several aspects of a large study he recently conducted, making each an independent manuscript, but without disclosing the relationship of the papers to each other.

  12. 12.

    At the suggestion of a journal manuscript peer reviewer, a researcher changes her planned methods of statistical analysis, even though she believes her original methods more accurately represent the data.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Sacco, D.F., Brown, M. & Bruton, S.V. Grounds for Ambiguity: Justifiable Bases for Engaging in Questionable Research Practices. Sci Eng Ethics 25, 1321–1337 (2019). https://doi.org/10.1007/s11948-018-0065-x

Download citation

Keywords

  • Questionable research practices
  • Ethics
  • Integrity
  • Motives