Introduction

Complete reporting is essential for translating results of clinical research into scientifically robust evidence to support decision-making in daily practice [1]. Reporting guidelines are developed to serve as useful tools to enhance the quality and transparency of clinical research [2, 3], while it is still a long-standing and widespread issue that the reporting quality is suboptimal [4, 5]. It is no wonder that a series of studies have repeatedly stressed the unsatisfied endorsement of reporting guideline in clinical journals [6,7,8,9,10,11,12,13,14,15,16,17,18,19], since the journals’ endorsement of reporting guidelines may raise the stakeholders’ awareness of them and is related to the completeness of reporting in medical journals [20].

The reporting quality of diagnostic accuracy tests and systematic reviews in Radiology, European Radiology, and Korean Journal of Radiology has been improved since it became mandatory to use the corresponding reporting guidelines [21,22,23]. It is reasonable that the change of the editorial policy of one specific journal significantly influences the reporting quality. On the other hand, the reporting quality of the abstracts of randomized controlled trials in the field of interventional radiology did not show obvious improvement after the update of corresponding guidelines [24]. One of the potential reasons for this difference is that the level of endorsement of reporting guidelines are different among the 61 interventional radiological journals that had been investigated. Therefore, it is necessary to investigate the endorsement of reporting guidelines, and sequent call those journals with low endorsement to make it mandatory. The earlier use of reporting guidelines during research process is considered to have greater impact on the final manuscript in radiological journals, suggesting that there is a need for enhanced education on the use of reporting guidelines [25]. The reporting guidelines should be introduced to the researchers to educate them about the benefits, be used by the reviewers during their review process, and be endorsed by journals to guarantee the adherence.

The artificial intelligence (AI) has engendered a rapid increasing medical application, especially in medical imaging [26,27,28,29,30]. The clinical translation of these academic research on AI should be based on scientific publication with enough details to allow readers to determine the rigor, quality, and generalizability. These publications of AI research in medical imaging calls for specific reporting guidelines for transparent and organized AI research reporting [31,32,33,34,35]. In addition to general reporting guidelines, the endorsement of these AI reporting guidelines has potential to improve the reporting quality and transparency of AI research, and reshape the current radiological practice. So far, the endorsement of AI reporting guidelines has not been assessed.

Therefore, this study aimed to investigate the endorsement of general reporting guidelines and those for AI applications in medical imaging in radiological journals, and explore associated journal characteristic variables.

Materials and methods

Study design

Our study is a meta-research study, and corresponding reporting guideline is currently not available [36,37,38,39,40,41]. Ethical approval or written informed consent were not required for this study because no human or animal subjects have been included in this study. We did not register the study protocol since there were no appropriate platform. However, we have drafted a protocol for this cross-sectional meta-research study (Supplementary Note S1). The sample selection, data extraction, and guideline endorsement assessment were conducted by two independent reviewers (JYZ and YX). The statistical analysis was performed by one reviewer (JYZ) under supervision of a methodologist (JJL). Any discrepancies will be resolved by discussion or consulting with the review group. Our study group is consisted of reviewers with diverse background and knowledge, including radiologists, and health professionals from multiple disciplines. All the reviewers have experience in manuscript drafting and publishing. Some of the reviewers have served as reviewers for radiological journals and evidence-based medicine journals, as well as editorial board members for radiological journals.

Sample journals

The journals from the Radiology, Nuclear Medicine & Medical Imaging category, Science Citation Index Expanded of the 2022 Journal Citation Reports were identified via Clarivate website [42], and assessed for eligibility. The exclusion criteria were (1) journals not publishing original research; (2) journals published in non-English languages; and (3) journals lacking instructions for authors.

Data extraction

The following bibliometrics information was directly downloaded via Clarivate: journal name, 2022 Journal Impact Factor (JIF), and the JIF quartile (Q1, Q2, Q3, Q4). The following journal characteristics were extracted: publication region, publication institution/publisher, publication language, publication frequency, type of access, whether the journal is only in the Radiology, Nuclear Medicine & Medical Imaging category, and whether the journal is owned by an academic society [43, 44]. The official website address of each journal was recorded. The search and data extraction via Clarivate was carried out on 20 July 2023.

Endorsement assessment

The endorsement of fifteen general reporting guidelines [45,46,47,48,49,50,51,52,53,54,55,56,57,58,59] and ten reporting guidelines for AI applications in medical imaging [60,61,62,63,64,65,66,67,68,69] was rated using a 5-level tool [19]. The 15 general reporting guidelines were selected since they are considered as the most frequently used for main study types, and are highlighted on the Enhancing the QUAlity and Transparency Of health Research (EQUATOR) Network website [70]: (1) CONSORT (Consolidated Standards of Reporting Trials) for randomized trials [45], (2) STROBE (STrengthening the Reporting of Observational Studies in Epidemiology) for observational studies [46], (3) PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) for systematic reviews [47], (4) SPIRIT (Standard Protocol Items: Recommendations for Interventional Trials) for study protocols [48], (5) PRISMA-P (Preferred Reporting Items for Systematic Review and Meta-Analysis Protocols) for study protocols [49], (6) STARD (Standards for Reporting of Diagnostic Accuracy) for diagnostic/prognostic studies [50], (7) TRIPOD (Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis) for diagnostic/prognostic studies [51], (8) CARE (CAse REport guidelines) for case report [52], (9) AGREE (Appraisal of Guidelines, Research, and Evaluation) for clinical practice guidelines [53], (10) RIGHT (Reporting Items for Practice Guidelines in Healthcare) for clinical practice guidelines [54], (11) SRQR (Standards for Reporting of Qualitative Research) for qualitative research [55], (12) COREQ (COnsolidated criteria for REporting Qualitative research) for qualitative research [56], (13) ARRIVE (Animal Research Reporting of In Vivo Experiments) for animal pre-clinical studies [57], (14) SQUIRE (Standards for QUality Improvement Reporting Excellence) for quality improvement studies [58], and (15) CHEERS (Consolidated Health Economic Evaluation Reporting Standards) for economic evaluations [59]. The 10 reporting guidelines for AI applications in medical imaging were identified and chosen according to relevant reviews and expert’s opinions [31,32,33,34,35]: (1) CONSORT-AI (Consolidated Standards of Reporting Trials involving Artificial Intelligence) [60], (2) SPIRIT-AI (Standard Protocol Items: Recommendations for Interventional Trials involving Artificial Intelligence) [61], (3) FUTURE-AI (Fairness Universality Traceability Usability Robustness Explainability Artificial Intelligence solutions) [62], (4) MI-CLAIM (Minimum Information about Clinical Artificial Intelligence Modeling) [63], (5) MINIMAR (Minimum Information for Medical AI Reporting) [64], (6) CLAIM (CheckList for Artificial Intelligence in Medical imaging) [65], (7) MAIC-10 (Must Artificial Intelligence Criteria-10) [66], (8) RQS (Radiomics Quality Score) [67], (9) IBSI (Image Biomarker Standardization Initiative) [68], (10) CLEAR (CheckList for EvaluAtion of Radiomics research) [69]. The reporting guidelines included in our study did not cover all the available AI reporting guidelines. Therefore, we also recorded the extra identified AI reporting guidelines if the journal websites mentioned it. Unfortunately, there was no extra AI reporting guidelines identified.

The modified 5-level tool rated the endorsement into: (1) “active strong”: journal requires completed checklist and/or flow diagram with article submission; (2) “active weak”: journal encourages to flow a specific guideline; (3) “passive moderate”: journal only requires the abstract to follow a specific guideline; (4) “passive weak”: journal encourages to prepare manuscripts according to EQUATOR Network website [70] or the International Committee of Medical Journal Editors document [71]; and (5) “none”: journal does not mention any reporting guideline [19]. The assessment on journal’s endorsement of reporting guidelines were based on their documents on submission (instructions for authors, submission guideline, editorial policies, etc.). The endorsement of reporting guidelines was assessed from 22 July 2023 to 23 July 2023.

Statistical analysis

The statistical analysis was conducted by using R language version 4.1.3 within RStudio version 1.4.1106 [72,73,74,75]. The endorsement types of “active strong”, “active weak”, “passive moderate”, and “passive weak” were considered as a positive outcome, while the type of “none” was treated as a negative outcome. The journal was considered as positive if at least one reporting guideline was positive; otherwise, the journal was treated as negative. The journal characteristics between the positive and negative groups were compared. The journal characteristics were further evaluated by logistic regression analysis to tell whether they are associated with the reporting guidelines endorsement. The factors associated with the endorsement of general reporting guidelines and those for AI applications in medical imaging were assessed together, because the endorsement of reporting guidelines for AI applications in medical imaging were very low. All of the statistical tests were two-sided. The alpha level for statistically significance is set at 0.05, if not stated otherwise.

Results

Journal inclusion

There were 135 journals in the Radiology, Nuclear Medicine & Medical Imaging category, Science Citation Index Expanded of the 2022 Journal Citation Reports. We excluded 10 journals that only publish reviews, 7 non-English journals, and 1 journal without available website. Finally, 117 radiological journals were included (Fig. 1).

Fig. 1
figure 1

Flowchart for radiological journal inclusion

Full size image

Journal characteristics

The mean ± standard deviation, median (range) of 2022 JIF of included journals was 3.7 ± 2.8, 2.9 (0.6 to 19.7) (Table 1). The journals were most likely belonged to JIF Q2 (29.9%, 35/117), North America (47.9%, 56/117), and not only included in the Radiology, Nuclear Medicine & Medical Imaging category (54.7%, 64/117). Most of the journals were published by Springer (25.6%, 30/117), with a frequency of ≥ 12 issue/year (40.2%, 47/117), supporting hybrid access mode (70.1%, 82/117), and were owned by academic societies (71.8%, 84/117) (Fig. 2). The characteristics of each journal are presented in Supplementary Tables S1 and S2. The original data for analysis are presented in Supplementary Data Sheet.

Table 1 Characteristics of included radiological journals
Full size table
Fig. 2
figure 2

Sankey diagram of journal characteristics. Abbreviation: JIF = Journal Impact Factor, LWW = Lippincott Williams & Wilkins

Full size image

Endorsement of reporting guidelines

The journals were divided into positive (61.5%, 72/117) and negative (38.5%, 45/117) groups (Table 1). The endorsement level of general reporting guidelines was most likely to be “active weak” (12.8% to 32.5%, 15/117 to 38/117) (Table 2). The top-five general reporting guidelines were CONSORT (58.1%, 68/117), PRISMA (54.7%, 64/117), STROBE (51.3%, 60/117), STARD (50.4%, 59/117), and ARRIVE (35.9%, 42/117).

Table 2 Endorsement levels of reporting guidelines in radiological journals
Full size table

The most implemented reporting guidelines for AI applications in medical imaging was CLAIM (1.7%, 2/117), while other nine artificial intelligence reporting guidelines were not mentioned in documents on submission of radiological journals. The examples for five levels of endorsement are presented in Supplementary Table S3. The endorsement of reporting guidelines of each journal is presented in Fig. 3 and Supplementary Table S4.

Fig. 3
figure 3

Endorsement of each reporting guideline according to radiological journals. The left part presents the endorsement of 15 general reporting guidelines. The right part presents the endorsement of 10 reporting guidelines for AI applications in medical imaging. Abbreviation: Q1 to Q4 = the first to the forth Journal Impact Factor quartile

Full size image

Factors associated with the endorsement of reporting guidelines

The journal characteristics did not show difference between the positive and negative groups, except for the distribution of publisher (Table 1). The multivariable logistic regression analysis showed that JIF quartile and publisher were associated with the endorsement of reporting guidelines in radiological journals (Table 3). The JIF Q2 journals were more likely to endorse the reporting guidelines than JIF Q1 journals (odds ratio 7.83, 95% confidence interval 1.70–36.20, P = 0.008). The journals that are published by academic societies (0.17, 0.04–0.64, P = 0.009), Lippincott Williams & Wilkins (0.13, 0.03–0.68, P = 0.016), and other publishers (0.03, 0.002–0.33, P = 0.004) were less likely to endorse the reporting guidelines than those published by Springer.

Table 3 Factors associated with the endorsement of reporting guidelines
Full size table

Discussion

The endorsement of general reporting guidelines was lowest for SPRIT, PRISMA-P, AGREE, RIGHT, and COREQ (all 32.5%) in radiological journals, and highest for CONSORT (58.1%). Only two journals suggested to implant the CLAIM (1.7%), while the other nine evaluated reporting guidelines for AI applications in medical imaging were not mentioned. The JIF quartile and publisher were associated with the endorsement of reporting guidelines in radiological journals.

The CONSORT statement was one of the most early developed reporting guidelines for randomized clinical trials, followed by a brunches of reporting guidelines published covering main study types. In accordance to the previous studies, the CONSORT statement has been most commonly endorsed by journals in varying specialties [6,7,8,9,10,11,12,13,14,15,16,17,18,19]. The methodology of randomized clinical trials was less difference among journals from different specialties, and the CONSORT statement met the requirements for most of the journals. Likewise, the reporting guidelines for systematic reviews, observational studies, and animal studies were also with relatively high endorsement, because their methodology were also of less variety. The radiological journals further recommend to use STARD statement, since the diagnostic accuracy test plays an important role in radiology [21, 22]. However, the TRIPOD statement has not been widely accepted by the radiological journals. It is possible that the study type of multivariable prediction model is less conducted than the diagnostic accuracy test. The reporting guidelines for other study types were less endorsed by radiological journals. It is possible that the protocol, and case report are not an acceptable study type for some of the journals, and guidelines, qualitative research, qualitative research, quality improvement, and economic evaluations were not common study types for radiological journals.

In addition to general reporting guidelines, the reporting guidelines for AI applications in medical imaging were also evaluated in our study. Unfortunately, the endorsement of investigated AI reporting guidelines was extremely low in radiological journal. Only European Radiology and Journal of the American College of Radiology recommended CLAIM for AI studies, although the CLAIM was published on the Radiology: Artificial Intelligence [65]. The MAIC-10 and CLEAR were two guidelines recently published on Insights into Imaging for AI studies and radiomics studies [66, 69], and the IBSI statement for radiomics was introduced by Radiology [68]. However, they have not been widely implanted by radiological journals even the journals published them. In contrast to the low endorsement in radiological journals, these reporting guidelines were usually applied for systematic reviews [76,77,78,79,80,81,82,83,84,85]. These systematic reviews found that the adherence rate of RQS, IBSI, and CLAIM were suboptimal for radiomics and AI studies for medical imaging. It is not weird since most of the radiological journals did not endorse these reporting guidelines. The reporting transparency has been improved after the introduction of reporting guidelines [20,21,22,23]. It is expectable to make the reporting guidelines for AI applications in medical imaging mandatory to improve the awareness and application of them, in order to achieve transparent AI and radiomics study reporting. The AI and radiomics community should understand the importance of proper self-reporting, and encourage researchers, journals, editors, and reviewers to take action to ensure the proper usage of checklists [82,83,84,85]. We plan to investigate the influence of the endorsement of the reporting guidelines for AI applications in medical imaging on the quality of study reporting.

We found that the JIF quartile and publisher were associated with the endorsement of reporting guidelines in radiological journals. The journals with higher JIF were generally more likely to endorse the reporting guidelines than lower ones [6, 15, 17, 18]. It is reasonable that the journals with higher impact factors have higher endorsement of reporting guidelines, since they were considered to have higher quality. However, our study showed that the JIF Q2 journals were more likely to endorse the reporting guidelines than JIF Q1 ones in radiological journals. The underlying reason is not yet clear. The journals that were published by Springer showed higher endorsement of reporting guidelines in both surgery and radiological journals [17]. We infer that the higher endorsement was benefited by the unified editorial policy of Springer that recommended the specific reporting guidelines and EQUATOR Network website. The surgical journals from United Kingdom and Europe were more likely to endorse the reporting guidelines than those from North America [17], but we did not find the influence of publication region in radiological journals.

The requirement of adherence to reporting guidelines may improve reporting quality [20,21,22,23], and the agreement among journals on the endorsement of reporting guidelines could improve the quality of research publishing [86]. Rather than prioritizing additional studies on the poor quality of health research reporting, interventions are needed to improve reporting [87]. However, only a limited number of tools has been raised for this purpose, and a smaller number of them has been evaluated [88]. There was the only one that showed a statistically significant effect on reporting quality was a Consort-based WEB tool [89, 89]. This tool supports adherence at the manuscript writing stage. The earlier the reporting guidelines were used, the more impact on the final manuscript and higher perceived value [25]. Therefore, it has been repeatedly suggested to enhanced education on the use of these guidelines. It seemed to be most pivotal to support journals to include hard-wiring adherence to reporting guidelines into their editorial policy. It may be an effective way to make the reporting guidelines mandatory, and ask the reviewers to use the reporting guidelines during the review process. The reporting quality has been improved if the journal required authors to incorporate section headings that reflected CONSORT items into their manuscripts [90]. Although it has not been found that these interventions are effective in radiological journals, the authors, reviewers, and editors may actively use the reporting guideline to guide themselves in the study design, conduction, drafting, reviewing, and revision. Further survey is need to identify the obstacles against the endorsement of reporting guidelines in journals. The potential reasons for suboptimal endorsement of reporting guidelines includes the insufficient resources for mandatory use, the needs to reduce barriers to submission and review, the unique editorial perspective, or the use of alternative to improve the study quality [16, 17]. As the current study may aware the radiological community on the issue of suboptimal endorsement of reporting guidelines. We plan to re-evaluate the endorsement in the future to find out whether the publication of radiological research is reshaped.

Our study has limitations. First, our study only included the radiological journals in the Science Citation Index Expanded. We selected these radiological journals as a representative sample of the high-quality journals of this field. Our results may over-estimate the journals’ endorsement of the reporting guideline. Second, our study is a cross-sectional study only presenting the current endorsement of reporting guidelines in radiological journals is suboptimal. An update study should be conducted in the future. Third, we relied on online documents to assess the endorsement without verifying the additional instructions potentially appear during the manuscript submission and review process. Although we collected and cross-checked the journal information, the journal websites may be update afterwards. Changes in editorial policies may not be timely updated online, which may influence on our conclusion. Fourth, we only investigated a limited number of general reporting guidelines and those for AI applications in medical imaging. There are a lot of reporting guidelines developed for more specific purpose and are available on EQUATOR Network website. Although our study did not cover all the available guidelines, we considered that the included reporting guidelines can at least represent the usually-used ones. We did find many extra AI guidelines which was not included in our study [91,92,93,94,95]. It would be interesting to investigate the endorsement of AI reporting guidelines which is not specifically designed for AI application medical imaging. Finally, our study only emphasized the potential of making reporting guidelines for AI applications in medical imaging mandatory to improve awareness and application in order to achieve high-quality study reporting. Nevertheless, there are currently too many different reporting guidelines for the AI research domain that is difficult for authors, reviewers, and editors to choose [31,32,33,34,35, 60,61,62,63,64,65,66,67,68,69, 91,92,93,94,95]. It is the next step of study to assess how relevant and how actual are these developed and developing guidelines to the current state of development of AI.

Conclusion

As a summary, our study found that the general reporting guideline endorsement are suboptimal in radiological journals. The implementation of artificial intelligence reporting guidelines was extremely low. Radiological journals may consider making general and artificial intelligence reporting guidelines mandatory to improve their awareness and application, in order to achieve high-quality and transparent radiological study reporting.