1 Introduction

Sharing of digital objects produced during the course of reported research has been proposed to be essential for scientific reproducibility and integrity since over a decade [1]. Likewise, still often forgotten in the shadow of data availability options, the majority of computational science publications lack code sharing, which limits the reuse of scripts and models, and moreover, limiting reproducibility [2]. Standards for making code available are necessary in order to report all aspects of computational analyses that enable replication by other scientists [3].

Fig. 1
figure 1

Scientific articles are queried for code availability using various search engines and open data. Corresponding availability statements are inspected over time, and results differ between de facto shared code (no code/code), proper documentation (doc), persistent identifiers (PID), or possible execution (exe), such as ease of installation, provided environment, or even peer-reviewed code

Full size image

The topic of reproducibility has been of concern years before 2012 [4, 5]. The term reproducibility in science has already been known around 1660 and became a matter of interest for computational sciences since everybody had access to the internet [6]. All materials and methods leading to scientific results should be available to everyone to ensure that results are reproducible; however, publishing false positives due to publication bias and other factors contributed to the reproducibility crisis [7]. Reproducibility key points include the necessity for data readability and the use of standardized dictionaries, licenses, search and citation options, machine-readable meta-data, original raw data if possible and associated processing scripts next to the final data used in analyses [8]. In general, computational code should not only be easily readable but also follow basic programming and commenting styles as well as standardized nomenclatures [9,10,11,12].

Publishers’ and funding agencies’ policies on public code availability have been suggested, as it has been shown in the past, that numerous publications on computational research had not shared the underlying developed source code [13, 14]. Likewise, institutional policies such as university research data management policies can foster corresponding best practices [15, 16]. There are some references indicating a trend towards sharing data and providing source code, but also highlighting the necessity of preservation [17, 18]. This trend may be a result of introduced policies. Data sharing requirements set by publishers can, and do in some cases, significantly effect sharing practices towards open science [19]. In this regard, it has also been shown that data sharing leads to additional scholarly citations [20]. However, simple data availability statements do not guarantee long-term availability of the indicated resources, since the lack of persistent identifiers hinder data access over time [21]. On top of that, quality of available resources is not checked in most cases, such as data or code peer review, that would contribute to trust in available online resources [22]. Additionally, non-compliance with data availability statements, such as "available upon request", has been observed [23]. On the other hand, data sharing policies by funding agencies could also influence the adoption of open science principles, though this matter is controversially discussed based on the argument that top down enforced rules are prone to constrain researchers instead of proposing alternatives, resulting in exacerbating existing practices [24].

In the past, several publishers have introduced recommendations and requirements anchored in policies and guidelines on appropriate data description, standardization, archiving and sharing of digital objects [20, 25]. Many publishers primarily provide policies and guidelines on data sharing, and only in addition, they recommend to share the underlying code of scientific findings from computational analyses, but do not provide guidelines on specific steps for code sharing. Such vague recommendations have yet to be structured and specified in detail, also in a more stringent form, in order to avoid a wide margin of interpretation.

The Transparency and Openness Promotion (TOP) guidelines define several levels for reproducibility and open science principles on digital object sharing, including code [26]. This development was driven by researchers in psychology due to the reproducibility crisis [27, 28]. Level 0 encourages code sharing or does not require authors to add a code availability statement. Level 1 requires some code availability statement. Level 2 for code deposited in a trusted repository. Level 3 adds the replication of code during the review phase of the publication next to code deposition in trusted repositories. This is partly in line with the reproducibility spectrum presented in Fig. 1. Examples for all levels translated to data sharing can be found among Wiley journals [29]. Still, there is only a introductory note that research data include methods, algorithms and software, while the term code is not mentioned. Examples for level 2 are given by Nature [30] or PLOS [31]. The latter provides specific instructions on disclosure. NeurIPS is one rare example for level 3 [32]. In exceptional domain-specific cases, guidelines for structuring code submission have been established together with code reviews during the peer review process [33].

Disciplinary differences in research practices may require domain-specific standards for disclosure of digital objects. Some research areas have broadly accepted open science as standard practice, such as for social sciences where data or code sharing practices have been used by less than half of authors in 2010 and currently this number has increased to 87% [34]. However, this does not indicate that they share code or data in each of their research works. Another study highlighted up to 18% articles to share the underlying code in case of publications on computational models [35]. In distinct disciplines, for example across medical research, data sharing rates have been shown to be consistently low at around 8% with much lower rates for public code sharing [36, 37]. A meta-analysis in the domain of medical and health sciences argues that declarations of data sharing have increased since 2016, while declared and actual public data sharing significantly differed, and code sharing prevalence did not increase, other than in the domain of computer sciences [36]. Interestingly, several factors might contribute to this effect, such as data privacy when working with data from human participants, but also cultural differences among domains.

In computational research, source code constitutes the documented specification of methods, equivalently comparable to mathematical formulas in theoretical science  [38]. Thereby, in a reproducible research setting the clarity of presentation becomes a new priority. Decision making has to be documented alongside source code, while versioning systems, such as GIT, facilitate transparency [39, 40]. Software has to be documented in a instructional style, to an extent that it is described sufficiently clearly and in detail to be traceable, interpretable and reproducible. There are differences in the perception as well as presentation of data between science and technology research teams [41]. This results in a different readability of source code, since development teams start scripting by rapid prototyping and then continue to optimize for performance, whereas science teams use data and code for providing their hypotheses [38].

Readability of source code is based on comprehensibility how to understand, reuse and/or change existing source code [42]. Different levels of documentation can support this process. Unfortunately, documentation is often considered a hurdle in software development, rather than recognized as benefit. Hence, documentation issues include lack of correctness, completeness, up-to-dateness and readability [43]. Source code and its documentation should be complete, consistent and accessible [44], or high-quality clean code[45]. Provenance documentation should be made available too [46]. While working on source code documentation it is beneficial to use coding guidelines, conventions, checklists, tools such as Doxygen, as well as to write tests [47, 48]. Additionally, code readability testing has been introduced, such as pair programming and reviewing source code by others, as method to improve software quality and its documentation [49].

FAIR principles for research software have been introduced on the basis of community-endorsed principles for research software by the FAIR for Research Software working group [50]. The working group has defined research software to include source code files, algorithms, scripts, computational workflows and executables that were created during the research process, further indicating the differentiation between disciplines for software components, such as operating systems, libraries, dependencies and packages, that should be considered as software and not research software [51]. The principles can be summarized to the following high-level key messages. First, research software and associated metadata have to be easily findable for both humans and machines. The citation file format (cff) as exemplary best practice provides citation metadata for research software [52]. Secondly, it has to be retrievable via standardized protocols. Third, research software should be interoperable with other software by exchanging (meta)data through interaction via standardized application programming interfaces. Fourth, it should be both usable, in terms of that it can be executed and reusable so that it can be understood, modified, built upon, or incorporated into other software. Further details and examples for adopting the principles have been outlined in [50]. This is in line with other guidelines, such as on open research software, including the key points of downloadability, installability on different platforms, defined conditions of use, open-source code with version history and proper description of dependencies [53]. FAIR, in contrast to the term open, entails metadata such as good documentation of setup, configuration and versions [54]. In the field of computer science and related fields several guidelines have been made available how to handle software artefacts that could be further refined and integrated into discipline-specific guidelines on disclosing code along scholarly publications [55, 56]. Long-term and archivable reproducibility of analysis pipelines has been evaluated for exemplary solutions, including environment and package management [57]. These suggest several requirements, such as completeness, modular designs and scalability, minimal complexity, verifiable input and output, version control and linking descriptive narratives to code. The provision of containerized applications represents a suitable example, but also web-based and desktop applications can be processed to meet the criteria by their registration in certified or domain-specific best-practice catalogues using unique and persistent identifiers, next to the provision of a proper licence and comprehensive descriptions also on all dependencies [50].

This work, summarized in Fig. 1, investigates code availability in publications in regard to possible changes over the last decade. This longitudinal analysis on practices for publishing source code of computational studies also explores disciplinary differences and indicates several open issues on source code sharing for reproducible computational scientific publications.

2 Materials and methods

In order to search for publications based on computational scripts, we take on the example of life science research where data have to be computationally analysed, limiting the search for publications by the key term "computational model", making use of PubMed Central (PMC) which comprises millions of citations for biomedical literature from MEDLINE, life science journals and online books. Since 2018 the National Library of Medicine (NLM), the world’s largest medical library operated by the US’ federal government, released a novel functionality of PMC. This feature of associating datasets to publications is based on the aggregation of data citations and data availability statements as well as supplementary materials from secondary source databanks as ClinicalTrials.gov, GenBank, figshare, Dryad, in an "associated data" box in the page navigation bar [58].

Results of web search engines for scientific publications were tested, namely Scopus next to PMC.

The PMC search summary resulted from the following parameters: Search term "computational", and article attributes "Free full text", "Associated data”, for the years 2012–2023. Results were collected on 2024-07-08.

This search strategy was complemented with a similar query using the same search attributes without any search term restriction in order to compare results not limited to "computational".

For manual control of code availability, the number of hits in results was previously further narrowed down by using the search query "computational model" instead of "computational", collected on 2023-12-27.

The Scopus search was based on the following input query:

ALL("code availability") OR ALL("code available") OR ALL("code can be found") OR ALL("code provided") OR ALL("code is deposited"). Results were collected 2023-01-18.

For further comparison Scopus search was extended to the input query ALL("computational") in comparison with:

ALL("computational") AND (ALL("code availability") OR ALL("code available") OR ALL("code can be found") OR ALL("code provided") OR ALL("code is deposited").

Data were collected 2023-02-13. There can be slight changes in the number of results returned depending on the time of data collection because of entry numbers being continuously refined by Scopus.

Since most references on the open issue of lacking code availability originate from around 2013, and the reproducibility crisis was coined around 2012 [27], timeseries data including publication dates for the respective articles were collected in order to assess possible changes during the last years. Publications from the last decade have been investigated in the cases of search queries via Scopus and PMC. Additionally, a recently updated open dataset [59] has been utilized to compare results for code availability statements isolated from the search queries described above. This includes data from the last 5 years only.

Data were exported as or manually noted in comma-separated spreadsheets from Scopus and PMC and further processed via Python (vers. 3.8.3) available in JupyterLab (vers. 3.2.5). Statistical analysis was carried out using pandas (vers. 1.4.2) method for Pearson’s correlation. Plots were compiled using matplotlib (vers. 3.5.0), pandas (1.3.4) and seaborn (0.13.1). Output and code can be found at https://www.doi.org/10.3217/gfpp0-2vf87.

3 Results

Various forms of code availability statements can be found. These include a separate section after data availability or code availability as part of data availability. Moreover, information on code availability can be found in the methods section. And, there are other publications without any code availability statement, though authors have used forms of software artefacts for their research. Some statements refer to as "upon request", "supplementary“, "download“, "can be found“, "available in“/"available at“, or "not applicable“, instead of linking to web resources. Long-term code accessibility constitutes one important criterion to be investigated, and several code repositories were identified, such as Dryad [60], GitHub linked to Zenodo [61], Code.europa.eu [62], Code-Ocean [63], Kaggle [64], R-universe, CRAN [65], CPAN [66], PyPi [67], SourceForge [68], Software Heritage [69] and various disciplinary source code libraries, indexed, e.g. in re3data [70].

To investigate the common practice in more detail, we collected results from specified search queries as shown in the next subsections.

3.1 Specified search queries on code availability among various disciplines

Analysis of indications on code availability in scientific publications has been a manual process. We have used a specified search query in Scopus [71].

Fig. 2
figure 2

Scopus search results: all documents listed upon search query for code availability (blue), and corresponding articles only out of all documents (orange) (colour figure online)

Full size image
Fig. 3
figure 3

Scopus search results: all documents listed upon search query "computational" (blue), and articles only out of all documents on "computational" (orange), in comparison with documents on "computational" and the search query for code availability (green) and articles, respectively (red) (colour figure online)

Full size image

Isolating code availability statements from publications by using specified search strings in Scopus are presented in Fig. 2. The isolated fraction of scientific publications holding statements on code availability in various formats highlights an increasing number of articles in the last decade. Still, the general number of publications also increases over time, presented in Fig. 3. The subfraction of articles referring to associated code correlates stronger to years than the fraction of general documents, as shown in Table 1. The category of documents comprises other types of publications, such as books and chapters, conference papers, erratums, reviews, notes and letters, next to research articles. Since many of those publications, such as books and narrative reviews, will possibly not use computational analyses, the number of articles was isolated and is presented separately.

Table 1 Correlation of Scopus search results: factors between 0 (without any correlation) and 1 (for identical numbers), highlighting hits on keyword "computational" for all documents, or articles only, respectively, without or with specified search strings on code availability
Full size table

Documents of the Scopus search on articles with associated code comprise several disciplines; the most frequent ones are highlighted in Figs. 4A, B. The results present all articles that include the search query for textual indications on potential code availability. Figure (A) on top presents the individual number of documents out of the hits for a given discipline. The most documents on code availability can be found for computer science, followed by mathematics and engineering. Disciplines of medicine and life sciences have been combined and also show an increasing number of documents over the last years. Figure (B) indicates the ratio of documents from a given discipline over the overall number of documents from the given search query on code availability for each year. In this case the ratio increases over time for the discipline of computer science. There is no significant increase of this ratio for medicine and life sciences. Since there are an increasing number of documents based on bioinformatics and data science experiments in medicine and life sciences due to the growing amount of biomedical data produced, code availability should be a major topic for this discipline. The Scopus search on code availability was not targeted disciplinary-specific. Another search web resource specific for the latter disciplines has been utilized and is presented in the next subchapter.

Fig. 4
figure 4

Scopus search results of search query for code availability: A with corresponding disciplines presented as number of documents, B ratio of discipline-specific hits in comparison with the yearly overall document number resulted from the given search query on code availability. A and B share the same legend (colour figure online)

Full size image

3.2 Open dataset on code availability in publicly available articles from PubMed Central

The comparator dataset [59] provided by PLOS has been reanalysed specifically for code availability statements as presented in Fig. 5. PLOS is a multidisciplinary publisher with a history to focus the fields of medicine and biology. Results are comparable to the disciplinary view in Scopus for medicine and life sciences. In case of the comparator dataset, the ratio of selected publications that share the underlying code of models or analyses has doubled since the past 5 years from 12 to almost 24%. An online location of the underlying code has been indicated in a smaller number of publications therefrom, while this number has also doubled from 10 to almost 20% between the years 2018 and 2023. Next to an indicated URL for online source code, some articles used the supplementary information for code descriptions, in many cases without presenting source scripts. The remaining subset of listed publications refer to code in the methods section without any detailed disclosure.

Fig. 5
figure 5

Comparator dataset: ratios of publications indicating code availability (blue), or indicating an online location of code (orange), in comparison with all publications that include results based on code (colour figure online)

Full size image

3.3 Specified search queries on code availability using PubMed Central

NCBI PMC provides literature from biomedical and health disciplines as well as related fields of life sciences, behavioural sciences, chemical sciences and bioengineering [72]. This was used as basic setting to search for publications on biological computational models. Free full text search results for publications with the search string "computational" in all fields are presented in Fig. 6. In addition, the NCBI filters in PMC searches allow to narrow down hits to publications with associated data. Manual inspection of hits on "computational model" indicates possibly wrong attributed associated data in case of year 2021 and years between 2013 and 2019. The data filter on "associated data" has first been introduced in 2018 retrospectively [73]. A random sample subset of publications (\(N=9\)) from those years did not include any availability statements or associated data to their experimental results. The search query using the term "computational" is further compared as ratio to all articles using resulting from no restriction in search strings, presented in Fig. 7. These lines point out the given trend of a possible wrong allocation of associated data availability to articles in all documents listed in PMC. Results from 2019, 2020, 2022 and 2023 suggest a constant number and maybe fraction of publications would include references to digital objects associated to their experimental findings. Further manual inspection of fractions in 2022 and 2023 revealed code availability statements in various formats, as part of data availability statements, separate section or part of methods. Eight and nine free full text publications from 2022 to 2023, respectively, include a code availability statement specifying a corresponding location, such as on github, figshare, OSF or others, as presented in Table 2. The table lists the publications with associated data and the repositories used for data as well as code. If mentioned in the respective manuscript, archives are also indicated. In summary for 2023 and 2022 this makes only 17 out of 31 manuscripts that provided URLs to the respective manuscript’s code. Two of the 17 URLs indicated have not been archived and were unfortunately not available anymore or at the time of analysis.

Fig. 6
figure 6

PubMed Central search query "computational": all documents listed (red), open access (OA) (yellow), or those with attributed data availability (blue), together with OA (green) (colour figure online)

Full size image
Fig. 7
figure 7

PubMed Central search results as ratio between hits from search query "computational" and no restriction: all documents (red), open access (OA) (yellow), associated data availability (blue), together with OA (green) (colour figure online)

Full size image
Table 2 Data and code availability in search results from search query = "computational model", open access publications with associated data as of 2023 and 2022: associated data details on data and code and optional code archive; digital object identifier (DOI), not available (na)
Full size table

4 Discussion

Reproducibility of scientific results is essential for trust in science [74]. Scholarly research articles should therefore include all necessary descriptions as well as materials, including relevant digital objects, such as code, as suggested a decade ago. Data availability has been the primary focus of several investigations, initiatives and funding organization or publisher terms. This work intends to highlight code sharing practices in scientific publications over the last years.

Results from Scopus search query and the reanalysed comparator dataset highlight code availability statements of publications in the range of 1–24% depending on discipline. Notably, PMC search queries, given the feature of isolating publications with associated data, show inconsistent results over the last decade in the percentage of shared digital objects and therefore had to be manually inspected. The comparably low number of the corresponding publications in the years 2010 to 2012 and 2019 to 2020 as well as 2022 to 2023 points to resulting percentages between 1 and 3%. These low numbers are more realistic than the resulting high rates from the years 2013 to 2017, which have been randomly checked and identified to be falsely annotated by the search engine in several cases. This feature of associating datasets to publications provided by PMC is a valuable function for refining search queries and, however, has yet to be optimized and consequent results should be interpreted carefully and double-checked. From another point of view, the comparison of the search string "computational" to all documents listed in PMC, presented in Fig. 7, could suggest a slightly higher trend of open access documents in computational science. Still, this trend would have to be carefully reevaluated for the recent and upcoming years.

In general, both search queries in PMC and Scopus have their limitations. Next to a potential weakness of the feature for associated data, search strings used in both databases are not possible be complete: It is not possible to list all articles that, both, are based on computational analyses and include the associated published computational code.

In the cases of the Scopus query and the comparator dataset, there has been a tendency of a growing number of publications that include code availability over the last decade. Still, the disciplinary view on publications has not revealed this tendency to be true for several disciplines except for the field of computer science. This is also in line with the results from the PMC query that together with the Scopus query do not show any improvement over time towards increased code availability in life sciences and medicine. There is a disciplinary common practice to make use of certain publishers which have different policies on data or code availability in place that can range from no indication of code, to code submission and reproducibility policies, and even the provision of checklists for included source code during the article submission process [75]. Compliance to such policies would need to be checked during the peer review process.

To ensure reproducibility of published results based on computational analyses it is essential to support source code persistence and to present reusable methods. This is indeed a matter of concern, since providing URLs without archives does not suffice, since the reference needs to persist in future, ideally without access restrictions. Still, as observed, many publications providing code availability statements do not use persistent identifiers and archives for long-term accessibility of code. Corresponding examples of the manual inspection of code availability among isolated publications include several possible descriptions, such as no reference to code location, supplementary files accompanying the publication, links to github repositories, as well as common archives, such as figshare or dryad, the code-specific archive of software heritage and disciplinary curated web resources in selected cases. Notably, the open science framework (OSF) platform was indicated multiple times, showing a current trend towards adoption of open science ideals. Since data archiving has already been described in many journal policies and recommendations, these should also integrate more detailed explanatory notes on sharing code and research software.

Good quality of code documentation is another important criterion to not only help the original researchers but also other scientists in understanding and reusing the underlying scripts. Results from search queries on publications and their associated availability statements indicate that there is still plenty of room for improvement. Nevertheless, using assistive tools such as CoPilot or other code-specialized generative AI powered tools can already facilitate source code refactoring, reviewing and its documentation [76,77,78]—a step that should not be forgotten at the end of any busy day of (data) science work [47, 79]. Still, in times where systems get more complex, and highly abstracted libraries are used to work on even more complex ideas, it is essential to keep provenance in mind to be accessible as well as traceable [46].

The manual analysis of code availability statements pointed out the use of various programming languages and tools for analysis and models. Future studies could expand on this topic giving an overview on disciplinary preferences, and ease of installation, as well as provision of runtime environments and platform-independent executables, such as containerized applications. Also, the feasibility of semi-automatic approaches to such an analysis could be studied. Future research could further focus automatic generated documentation, community reviews, or more detailed statistics on preferences, such as choice of repository.

The comparison of interdisciplinary to disciplinary results on code availability among scholarly publications implicates an importance for differently elaborated recommendations and guidelines for various research fields. These could be integrated in funder and journal directives. Ultimately, semi-automatic compliance checks and adoption of policies by editorial boards and reviewers are necessary to replicate scientific results.