Abstract
Scientific uncertainty is an integral part of the research process and inherent to the construction of new knowledge. In this paper, we investigate the ways in which uncertainty is expressed in articles and propose a new interdisciplinary annotation framework to categorize sentences containing uncertainty expressions along five dimensions. We propose a method for the automatic annotation of sentences based on linguistic patterns for identifying the expressions of scientific uncertainty that have been derived from a corpus study. We processed a corpus of 5956 articles from 22 journals in three different discipline groups, which were annotated using our automatic annotation method. We evaluate our annotation method and study the distribution of uncertainty expressions across the different journals and categories. The results show a predominant concentration of the distribution of the scientific uncertainty expressions in the Results and Discussion section (71.4%), followed by 12.5% of expressions in the Background section, and the largest proportion of uncertainty expressions, approximately 70.3%, are formed as author(s) statements. Our research contributes methodological advances and insights into the diverse manifestations of scientific uncertainty across disciplinary domains and provides a basis for ongoing exploration and refinement of the understanding of scientific uncertainty communication.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Data availability
The data supporting the paper is publicly accessible on the Zenodo platform at https://doi.org/https://doi.org/10.5281/zenodo.8024787. Furthermore, the system used for automatic annotation in this study can be accessed for demonstration at https://bit.ly/unscientify-demo.
References
Ascough II, J., Maier, H., Ravalico, J., & Strudley, M. (2008). Future research challenges for incorporation of uncertainty in environmental and ecological decision-making. Ecological Modelling, 219, 383–399. https://doi.org/10.1016/j.ecolmodel.2008.07.015
Atanassova, I., Rey, F., & Bertin, M. (2018). Studying Uncertainty in Science: A distributional analysis through the IMRaD structure. In 7th International workshop on mining scientific publications (WOSP 2018) at 11th edition of the language resources and evaluation conference (LREC 2018), 2018, Miyazaki, Japan. http://lrec-conf.org/workshops/lrec2018/W24/pdf/10_W24.pdf
Blanchemanche, S., Rona-Tas, A., Cornuéjols, A., Duroy, A., & Martin, C. (2013, January). An ontology of scientific uncertainty: Methodological lessons from analyzing expressions of uncertainty in food risk assessment. In SAESA Amsterdam conference, 2013.
Bongelli, R., Riccioni, I., Burro, R., & Zuczkowski, A. (2019). Writers’ uncertainty in scientific and popular biomedical articles. A comparative analysis of the British Medical Journal and Discover Magazine. PLoS ONE, 14(9), e0221933.
Budescu, D. V., & Wallsten, T. S. (1995). Processing linguistic probabilities: General principles and empirical evidence. Psychology of Learning and Motivation, 32, 275–318.
Candlin, C. N., & Hyland, K. (Eds.) (2014). Writing: Texts, processes and practices. Routledge.
Candlin, F. (2000). Practice-based doctorates and questions of academic legitimacy. Journal of Art and Design Education, 19(1), 96–101.
Chen, C., Song, M., & Heo, G. E. (2018). A scalable and adaptive method for finding semantically equivalent cue words of uncertainty. Journal of Informetrics, 12(1), 158–180.
Cordner, A., & Brown, P. (2013). Moments of uncertainty: Ethical considerations and emerging contaminants. Sociological Forum, 28(3), 469–494. https://doi.org/10.1111/socf.12034
Courtney, H. (2001). 20/20 Foresight: Crafting strategy in an uncertain world. Harvard Business Press.
Dosi, G., & Egidi, M. (1991). Substantive and procedural uncertainty: An exploration of economic behaviours in changing environments. Journal of Evolutionary Economics, 1, 145–168.
Fijnvandraat, M. L. (2009). Shedding light on the black hole: The roll-out of broadband access networks by private operators. Doctoral Dissertation, Delft University of Technology. Next Generation Infrastructure Foundation. ISBN 978-90-79878-03-1.
Fox, C. R., & Ülkümen, G. (2011), Distinguishing two dimensions of uncertainty. In W. Brun, G. Kirkebøen & H. Montgomery (Eds.), Essays in judgment and decision making. Universitetsforlaget. https://doi.org/10.2139/ssrn.3695311
Friedman, M., & Kandel, A. (1999). Introduction to pattern recognition: Statistical, structural, neural, and fuzzy logic approaches (Vol. 32). World Scientific.
Funtowicz, S. O., & Ravetz, J. R. (1990). Uncertainty and quality in science for policy (Vol. 15). Springer.
Guillaume, J. H., Helgeson, C., Elsawah, S., Jakeman, A. J., & Kummu, M. (2017). Toward best practice framing of uncertainty in scientific publications: A review of Water Resources Research abstracts. Water Resources Research, 53(8), 6744–6762. https://doi.org/10.1002/2017WR020609
Helton, J. C. (1994). Treatment of uncertainty in performance assessments for complex systems. Risk Analysis, 14(4), 483–511.
Hyland, K. (1996). Talking to the academy: Forms of hedging in science research articles. Written Communication, 13(2), 251–281.
Hyland, K. (1998). Hedging in scientific research articles. In Hedging in scientific research articles (pp. 1–317). John Benjamins. https://doi.org/10.1075/pbns.54
Hyland, K. (2000). Developments in English for Specific Purposes: A multi-disciplinary approach. English for Specific Purposes, 19(3), 297–300.
Jauch, L. R., & Kraft, K. L. (1986). Strategic management of uncertainty. Academy of Management Review, 11(4), 777–790.
Kahneman, D., & Tversky, A. (1982). Variants of uncertainty. Cognition, 11(2), 143–157.
Light, M., Qiu, X. Y., & Srinivasan, P. (2004). The language of bioscience: Facts, speculations, and statements in between. In HLT-NAACL 2004 workshop: Linking biological literature, ontologies and databases, 2004 (pp. 17–24). https://aclanthology.org/W04-3103
Medlock, B., & Briscoe, T. (2007, June). Weakly supervised learning for hedge classification in scientific literature. In ACL, 2007 (Vol. 2007, pp. 992–999).
Meijer, I. S., Hekkert, M. P., Faber, J., & Smits, R. E. (2006). Perceived uncertainties regarding socio-technological transformations: Towards a framework. International Journal of Foresight and Innovation Policy, 2(2), 214–240.
Morgan, M. G., Henrion, M., & Small, M. (1992). Uncertainty: A guide to dealing with uncertainty in quantitative risk and policy analysis. Cambridge University Press.
Ningrum, P. K., & Atanassova, I. (2023). Dataset for multidisciplinary uncertainty mining—Ver1. https://doi.org/10.5281/zenodo.8024787
Ningrum, P. K., Mayr, P., & Atanassova, I. (2023). UnScientify: Detecting scientific uncertainty in scholarly full text. arXiv [Cs.CL]. http://arxiv.org/abs/2307.14236
Pinto, M. D. G., Osorio, P., & Martins, F. (2014). A theoretical contribution to tackling certainty and uncertainty in scientific writing: Four research articles from the journal Brain in focus. In Communicating certainty and uncertainty in medical, supportive and scientific contexts (pp. 291–308). John Benjamins Publishing Company.
Rammel, C., & van den Bergh, J. C. (2003). Evolutionary policies for sustainable development: Adaptive flexibility and risk minimising. Ecological Economics, 47(2–3), 121–133.
Refsgaard, J. C., van der Sluijs, J. P., Højberg, A. L., & Vanrolleghem, P. A. (2007). Uncertainty in the environmental modelling process—A framework and guidance. Environmental Modelling and Software, 22(11), 1543–1556. https://doi.org/10.1016/j.envsoft.2007.02.004
Rey, F. C., Bertin, M., & Atanassova, I. (2018, June). Une étude de l'incertitude dans les textes scientifiques: vers la construction d'une ontologie. In TOTh 2018 Terminology and Ontology: Theories and applications (pp. 229–242). Presses universitaires Savoie Mont Blanc.
Rubin, V. L., Liddy, E. D., & Kando, N. (2006). Certainty identification in texts: Categorization model and manual tagging results. In Computing attitude and affect in text: Theory and applications (pp. 61–76). Springer.
Salager-Meyer, F. (1997). I think that perhaps you should: A study of hedges in written scientific discourse. In T. Miller (Ed.), Functional approaches to written text: Classroom applications (pp. 105–118). English Language Programs: United States Information Agency.
Stocking, S. H., & Holstein, L. W. (1993). Constructing and reconstructing scientific ignorance: Ignorance claims in science and journalism. Knowledge, 15(2), 186–210.
Swales, J. M. (2014). 1990. Genre analysis: English in academic and research settings. Cambridge: Cambridge University Press, selected 45–47, 52–60. In The discourse studies reader (pp. 306–316). John Benjamins Publishing Company.
Thomson, J., Hetzler, E., MacEachren, A., Gahegan, M., & Pavel, M. (2005). A typology for visualizing uncertainty. Proceedings of SPIE: The International Society for Optical Engineering, 5669, 146–157, Article 16. https://doi.org/10.1117/12.587254
Van Asselt, M. (2000). Perspectives on uncertainty and risk: The PRIMA approach to decision support. Springer.
Van Asselt, M., & Rotmans, J. (2000). Uncertainty in integrated assessment, a bridge over troubled water (p. 60). ICIS (International Centre for Integrative Studies), Maastricht University.
van der Bles, A. M., van der Linden, S., Freeman, A., & Spiegelhalter, D. (2018). 18 The effects of communicating uncertainty about facts and numbers. Evidence-Based Medicine, 23(Suppl 1), A9–A10.
van der Sluijs, J. P. (1997). Anchoring amid uncertainty. In On the management of uncertainties in risk assessment of anthropogenic climate change. CIF-Gegevens Koninklijke Bibliotheek.
van Witteloostuijn, A. (1986). Choice-theory versus vs. behaviourism: A paradox (pp. 1–16). University of Groningen.
Vincze, V., Szarvas, G., Farkas, R., Móra, G., & Csirik, J. (2008). The BioScope corpus: Biomedical texts annotated for uncertainty, negation, and their scopes. BMC Bioinformatics, 9(11), 1–9. https://doi.org/10.1186/1471-2105-9-S11-S9
Walker, W. E., Harremoës, P., Rotmans, J., Van Der Sluijs, J. P., Van Asselt, M. B., Janssen, P., & Krayer von Krauss, M. P. (2003). Defining uncertainty: A conceptual basis for uncertainty management in model-based decision support. Integrated Assessment, 4(1), 5–17. https://doi.org/10.1076/iaij.4.1.5.16466
Yao, M., Wei, Y., & Wang, H. (2023). Promoting research by reducing uncertainty in academic writing: A large-scale diachronic case study on hedging in Science research articles across 25 years. Scientometrics, 128(8), 4541–4558. https://doi.org/10.1007/s11192-023-04759-6
Acknowledgements
The authors gratefully acknowledge the French ANR for funding this research under the ANR InSciM Project. Special thanks to the CRIT Laboratory at the Université de Franche-Comté for their support and the provision of conducive working facilities. Additionally, this paper represents a substantially extended version (at least 25% new material) of the ISSI2023 conference paper, and the authors appreciate the insights gained from the conference that contributed to the refinement of the research. The original conference paper entitled “Scientific Uncertainty: an Annotation Framework and Corpus Study in Different Disciplines” is accessible at: https://www.conftool.pro/issi2023/index.php?page=browseSessions&form_session=41.
Funding
This work is supported by the ANR InSciM Project (2021–2025), funded by French ANR, Grant Number ANR-21-CE38-0003-01.
Author information
Authors and Affiliations
Contributions
Both authors contributed equally to the conception, design, and execution of this research project. Panggih Kusuma Ningrum and Iana Atanassova collaborated seamlessly in collecting and analyzing data, developing methodologies, and interpreting results. Their shared intellectual input significantly contributed to the creation of the manuscript, encompassing drafting, revising, and finalizing the content. The authors jointly participated in critical discussions, ensuring the scientific rigor and validity of the study. The allocation of responsibilities and tasks was mutual, and both authors actively engaged in the synthesis of ideas, forming the cohesive narrative present in this manuscript. Consequently, Panggih Kusuma Ningrum and Iana Atanassova share equal credit for the intellectual content and the final version of this research work.
Corresponding author
Ethics declarations
Conflict of interest
The authors affirm that there are no conflicts of interest or competing interests associated with the completion of this research work and the development of the manuscript. They declare that no financial, personal, or professional circumstances exist that could be perceived as having influenced the conduct or reporting of the research. The authors are committed to upholding the highest standards of integrity and transparency in the communication of their findings. This statement serves as an unequivocal declaration that, to the best of their knowledge, there are no affiliations, financial involvements, or other relationships that could be perceived as constituting a conflict of interest or influencing the impartiality of the research presented in this manuscript.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Ningrum, P.K., Atanassova, I. Annotation of scientific uncertainty using linguistic patterns. Scientometrics (2024). https://doi.org/10.1007/s11192-024-05009-z
Received:
Accepted:
Published:
DOI: https://doi.org/10.1007/s11192-024-05009-z