Content-based image retrieval systems (CBIRS) are a new and potentially impactful tool for radiological reporting, but their clinical evaluation is largely missing. This study aimed at assessing the effect of CBIRS on the interpretation of chest CT scans from patients with suspected diffuse parenchymal lung disease (DPLD).
Materials and methods
A total of 108 retrospectively included chest CT scans with 22 unique, clinically and/or histopathologically verified diagnoses were read by eight radiologists (four residents, four attending, median years reading chest CT scans 2.1± 0.7 and 12 ± 1.8, respectively). The radiologists read and provided the suspected diagnosis at a certified radiological workstation to simulate clinical routine. Half of the readings were done without CBIRS and half with the additional support of the CBIRS. The CBIRS retrieved the most likely of 19 lung-specific patterns from a large database of 6542 thin-section CT scans and provided relevant information (e.g., a list of potential differential diagnoses).
Reading time decreased by 31.3% (p < 0.001) despite the radiologists searching for additional information more frequently when the CBIRS was available (154 [72%] vs. 95 [43%], p < 0.001). There was a trend towards higher overall diagnostic accuracy (42.2% vs 34.7%, p = 0.083) when the CBIRS was available.
The use of the CBIRS had a beneficial impact on the reading time of chest CT scans in cases with DPLD. In addition, both resident and attending radiologists were more likely to consult informational resources if they had access to the CBIRS. Further studies are needed to confirm the observed trend towards increased diagnostic accuracy with the use of a CBIRS in practice.
• A content-based image retrieval system for supporting the diagnostic process of reading chest CT scans can decrease reading time by 31.3% (p < 0.001).
• The decrease in reading time was present despite frequent usage of the content-based image retrieval system.
• Additionally, a trend towards higher diagnostic accuracy was observed when using the content-based image retrieval system (42.2% vs 34.7%, p = 0.083).
Content-based image retrieval (CBIR) is an application of machine learning that attempts to alleviate the challenge of searching for digital images in large databases. In comparison to conventional search techniques that analyze text, keywords or other metadata, CBIR analyzes the “content” of the image. In general, CBIR allows the automated search of similar images to a chosen image or part of an image. As an example, this would allow to browse any accessible database for cases similar to the present case based on patterns and other imaging features. Further, this search could be narrowed by adding traditional non-imaging information, as long as this information is available, and the specific CBIR-system was trained on such data. In the radiology domain, it allows radiologists to query an image from a current study and automatically receive similar cases from the local picture archiving and communication system (PACS) or online image databases . In addition to providing similar cases, some CBIR systems do also provide case-specific educational content or links to relevant online educational information linked to the case which may help the radiologists to narrow the differential diagnosis. Beyond case databases, this may accelerate the process of accessing relevant information from sources such as Radiopedia.org . There, an abundance of educational cases is available, yet search is based on textual entry and not linked to the image information of a case at hand.
Thus far, research on CBIR has primarily focused on technical capabilities such as retrieval performance. However, studies evaluating the benefit of CBIR systems in a clinical setting are scarce [3,4,5,6,7]. A previous work assessed CBIR applications for diagnosing pulmonary pathologies in chest CTs in an experimental setting, specifically complex pathologies such as diffuse parenchymal lung diseases (DPLDs) [8,9,10]; however, studies adopting a practical approach towards the reading of chest CTs in a realistic setting with a wide variety of possibly rare findings that would demand considerable experience are yet lacking. This evidence of the potential impact of CBIR on the clinical routine including time savings or improved accuracy is necessary for its informed adoption. This study therefore aimed at assessing the impact of a CBIR diagnostic support system (CBIRS) on the reading time and diagnostic accuracy with the diagnosis of chest CT scans from patients with suspected DPLDs.
The study was approved by the institutional review board (protocol number 1288/2018), and the need for informed consent was waived due to the retrospective nature. The study consisted of two phases: baseline and intervention. In the baseline phase, participants read chest CTs without support from the CBIRS but were allowed to use any additional sources they typically use during clinical routine. In the subsequent intervention phase, participants had access to the CBIRS while reading the chest CTs (Fig. 1).
The participants consisted of four radiology residents and four attending radiologists without sub-specialization in pulmonary imaging with a mean of 2.1 ± 0.7 and 12 ± 1.8 years of experience in reading chest CT scans, respectively; their median number of chest CT reports were 528 ± 135 and 2050 ± 1110, respectively (Table 1).
To simulate a clinical reporting situation, the assessment was performed at a dedicated clinical picture archiving and communication system (PACS) workstation. For the intervention phase, a shortcut integrated into the PACS opened the CBIRS for the respective case within 5 s. The participants could place a region of interest (ROI) within the selected CT examination. After activating the ROI-drawing mode by pressing the appropriate hotkey, a rectangular ROI of freely adjustable size can be placed within an axial image. The location is fully up to the user, and, in general, will be somewhere in or around a possibly relevant pulmonary CT-pattern. The CBIRS then identified similar cases stored in the PACS, and provided additional information related to the pattern defined by the ROI (Fig. 2).
All participants were briefed on the CBIRS’ functionality before the intervention phase and had the chance to use the CBIRS on three cases that were not part of the evaluation. Reporting was done based on the image data of the respective cases without access to previous examinations (including previous radiological reports), or non-imaging data (e.g., laboratory tests). During both phases, participants were allowed to use any source of information (e.g., online resources or books). In the intervention phase, the participants had access to the above-described CBIRS.
In both phases, every participant read 27 cases, and the time between retrieving the image data and the completion of the reading was measured for each case. Overall, every case was read four times (but never twice by the same radiologist): by one radiology resident and one attending radiologist in the baseline phase, and by a different resident and different attending radiologists in the intervention phase.
The reading time was taken by an independent observer. The same observer also noted whether the reader searched for additional information.
The database consisted of 108 individual thin-section CT scans — 100 (92.6%) with confirmed DPLDs and 8 (7.4%) with inconspicuous pulmonary CT scans. None of the 108 cases was used for training the CBIRS. Chest CT scans performed between 1.1.2018 and 31.12.2018 on one single scanner (CT Somatom Drive, Siemens Healthineers) were included. Exclusion criteria included the following: (a) acute pulmonary pathologies; (b) patients with any kind of pulmonary surgery; (c) all inconspicuous pulmonary CTs after the first eight cases without any pathological imaging findings. Most of the indications for the 108 CT scans were either a follow-up examination in case of an already known disease or the primary CT-scan in case of a clinically suspected disease. In some cases, the CT findings were incidental, and the scan was conducted for another reason not covered by the exclusion criteria. The final database resulted in 108 cases (Table 2, Supplemental Table 1). The eight inconspicuous cases without any pulmonary pathological imaging findings served as two “healthy” cases per participant per session. Participants were told that their case sets may include “healthy” cases.
For each case, the final diagnosis was confirmed by one sub-specialized thoracic radiologist (H.P., 20 years of experience) where available using prior and follow-up examinations, clinical symptoms, pathology and histology reports, and interdisciplinary board decisions.
All cases were reconstructed using a sharp (I70f) kernel and 1-mm slice thickness. The acquisition was done with activated automatic tube modulation, 90, 100, or 140 kV and either without contrast agent (n = 58), during a pulmonary arterial phase (n = 15; total of 50 ml of 400 mg iodine per ml contrast agent with a threshold trigger at 115 HU and a 5-s delay) or during a venous phase (n = 35; total of 70 ml of 400 mg iodine per ml contrast agent after a 50-s delay).
A research prototype of contextflow SEARCH Lung CT (contextflow GmbH) was used as the CBIRS. After manually marking a region of interest (ROI), it searches a large database of 6542 anonymized thin-section CT scans (2528 female, 4014 male, ages 5 to 100 years, acquired over 2.5 years) for similar imaging patterns. Each of the CTs in this training dataset included expert labels for 19 lung-specific patterns. A content-specific similarity function based on these expert-labeled data was developed and used to compare image patterns. For any marked region of interest, the CBIRS provides the three most likely disease patterns present. For every pattern class, it additionally provides the cases that most closely resemble the marked ROI in the current scan (Fig. 2).
The image retrieval algorithm searches for image patches similar to the region of interest image patch (ROI) marked by the user in the current case. This ROI serves as a query, and retrieval identifies the most similar patches in a large pre-curated database of patches extracted from lung CTs. We created this search database of over 5,000,000,000 such patches, involving two steps: (1) learning a similarity function that puts patches of the same disease pattern in close proximity, so that the nearest neighbors of a newly observed patch have the same disease pattern, and (2) creating an index, so that during actual search the comparison of the ROI with the database patches can be performed rapidly (within 1 s).
The similarity function is learned from training data consisting of thin-section CTs. Lungs are segmented and from within the lung region patches are extracted. The entirety of these patches is used to learn an embedding of patches into an embedding space. There, each patch is represented by a coordinate, so that the Euclidean distance between patch representatives can serve as a similarity function. The embedding is performed by a feed forward convolutional neural network, its input is the image patch, and the output the coordinates of the representative in the embedding space.
A direct comparison of a query patch with all database patches calculating billions of distances during the interactive use is not feasible. Instead, an index of the coordinates in the embedding space is built to accelerate retrieval to less than 1 s. A separate index is built for each disease pattern class. To this end the embedding is subdivided in a hierarchical fashion to build a two-level data structure, containing information on the coarse and fine location of individual patches in the embedding space.
During search, first the ROI is embedded using the feed forward network, the three most likely pattern classes are estimated, and then search for the most similar patches is conducted in the hierarchical index for each pattern class.
The 19 lung pattern classes are as follows: airway wall thickening, atelectasis, bronchiectasis, bulla, consolidation, cysts, effusion, emphysema, ground-glass opacities, honeycombing, mass, mosaic attenuation pattern, nodular pattern, nodule, pneumothorax, pulmonary cavity, reticular pattern, tree in bud, non-specific (includes inconspicuous lung regions).
All statistical computations were performed using SPSS v26.0 (version 26.0, IBM Corp.). p-values equal or below 0.05 are considered statistically significant. A sample size calculation was conducted to establish the necessary number of cases. For the variable “correct diagnosis,” a power of 80%, two-sided alpha of 5% and 20% increase in correct diagnoses were chosen. Under the assumption of 60% diagnostic accuracy for different DPLDs , this equals an increase of 12 percentage points, resulting in 83 necessary cases when analyzed with McNemar’s test. Rounded up, we included 100 cases with lung pathologies. Continuous variables were tested for normal distribution using Shapiro-Wilk test and via visual assessment per subgroup and are displayed either as mean ± standard deviation or as median [interquartile range] as appropriate. To evaluate differences in time-to-diagnosis and correct diagnoses, three-way linear mixed models with an unstructured covariance matrix were used. The three factors were as follows: “phase” (baseline vs. intervention), “reader” (eight different readers subdivided into two groups of experience), and “if the participant looked for additional information” (yes vs. no). All main effects and two-way interactions were included. For post hoc analyses, Bonferroni multiplicity corrections were assessed.
Modeled overall time used per case was reduced by 31.3% when using the CBIRS (p < 0.001), controlling for individual participants, experience level, and whether they looked for information (Fig. 3). The reduction in time taken per case from baseline to intervention was more distinct for cases where the participants looked for information compared to where they did not (110 vs 39 s saved, p = 0.002) (Table 3).
Overall diagnostic accuracy was higher in the interventional phase compared to the baseline phase without reaching statistical significance (42.2% vs 34.7%, p = 0.083).
Participants used additional information (e.g., internet searches) more frequently during the intervention phase compared to the baseline phase (154 [72%] vs 95 [43%], p < 0.001). Of the 154 information searches, 108 (70.1%) were conducted solely with the CBIRS and 46 (29.9%) with the CBIRS plus traditional means.
Despite searching for additional information more frequently, both the radiology residents and attending radiologists showed a decrease in reading time when the CBIRS was available, and there was a tendency towards a stronger decrease in reading time for senior radiologists (27% vs 35%, p = 0.078).
In this study we found that the integration of a CBIRS into the clinical workflow of radiologists reading chest CTs resulted in a significant decrease in average reading time.
The main hypothesized advantage of the CBIRS is the streamlined provision of relevant information for the case the radiologist is currently reporting on. One of the most acclaimed informational sources is Radiopaedia, providing over 45,000 cases where quality and actuality are ensured by the editorial board consisting of radiological experts. Several studies have shown Radiopaedia to be the top or one of the most used informational sources [12, 13]. Furthermore, an educational benefit was demonstrated for integrating Radiopaedia-based training in a medical curriculum , underlining the potential long-term on radiological experience. Similar to how contemporary search engines work, we aimed at closing the gap between the image and the information that is necessary to provide the correct diagnosis by providing both no-click results after the search and a direct link to the most relevant informational site (i.e., the corresponding Radiopaedia article). Indeed, both for radiology residents and attending radiologists, the decrease in average reading time was more pronounced when they looked for additional information. This is particularly relevant as the participants used additional information more frequently during the intervention phase compared to the baseline phase (72% vs. 43%, p < 0.001). The CBIRS was used in all of these instances (70.1% of the information searches were conducted solely with the CBIRS and 29.9% with the CBIRS plus traditional means). This indicates a more focused and efficient search for information when the CBIRS is available.
Although participants were given different cases for each phase, reading times may still have been affected by familiarity with the study process and the individual drive to finish the readings. However, even after adjusting for alternative factors in our model, including phase, reader type, and additional information search, the overall 31.3% decrease in reading time remained and was concurrent with a trend towards higher diagnostic accuracy when the CBIRS was available (42.2% vs 34.7%, p = 0.083).
Reporting was done based on the image data of the respective cases without access to previous examinations (including previous radiological reports), or non-imaging data (e.g., laboratory tests). This limits the highest achievable accuracy for complex cases such as in interstitial lung disease. Although the participants may not have had a confident diagnosis available, they still had to commit to giving one, resulting in the shown overall diagnostic accuracy (which may be quite low even for experienced thoracic specialists as reported by Walsh et. al ).
Previous studies assessing CBIRS in lung imaging have demonstrated significant improvements in diagnostic accuracy: 13% for pulmonary nodules  and 15–62% for various disease patterns and diagnoses [8, 10, 16]. However, most of these studies were met with significant limitations, including application solely for pulmonary nodules, low number of test cases, and inclusion of individual images (rather than volumes) in the training data. This study aimed to simulate clinical routine, leaving participants the decision whether and how to search for information (including all options in both phases). This design avoids possible biases and more closely reflects routine workflow, in contrast to forcing radiologists to use information search. While previous studies report a benefit mainly for less experienced participants , this study showed significant time savings both for junior and senior radiologists. A similar study showed the feasibility of improving diagnostic accuracy while investigating a CBIR for aiding the diagnosis of ILD . The CBIR applied in the referenced paper had an additional query database with labeled diagnoses for the query cases, which was not in place yet for the CBIR used during our study. For further studies we suggest adding well-curated diagnosis labels to the query database.
As we encountered delays as a result of the ongoing SARS-CoV-2 pandemic, time between the two phases ranged from 9 and 15 months for individual participants. Thus, increasing experience between phases constitutes another potential bias. However, the results for radiology residents and attending radiologists were similar, even though the latter had a markedly less pronounced relative increase in years of professional experience and finished reports (Table 1).
In conclusion, the presence of a CBIRS when reading chest CTs for patients with suspected DPLD resulted in an overall 31% decrease in reading time despite participants searching for additional information more frequently. This is of particular interest, as a recent study  reports that about 86% of studies using AI could lead to an increase in radiologists’ workload. In addition, we noted a non-significant trend towards improved diagnostic accuracy. These findings hold true for both junior and senior radiologists. Such optimistic outcomes warrant additional research of CBIRS in clinical settings.
Content-based image retrieval system
Diffuse parenchymal lung disease
Picture archiving and communication system
Region of interest
Cao Y, Steffey S, He J et al (2014) Medical image retrieval: a multimodal approach. Cancer Informat 13:125–136
Depeursinge A, Fischer B, Müller H, Deserno TM (2011) Prototypes for content-based image retrieval in clinical practice. Open Med Inform J 5:58–72
Kashif M, Raja G, Shaukat F (2020) An efficient content-based image retrieval system for the diagnosis of lung diseases. J Digit Imaging 33:971–987
Ma L, Liu X, Fei B (2020) A multi-level similarity measure for the retrieval of the common CT imaging signs of lung diseases. Med Biol Eng Comput 58:1015–1029
Muramatsu C (2018) Overview on subjective similarity of images for content-based medical image retrieval. Radiol Phys Technol 11:109–124
Depeursinge A, Vargas A, Gaillard F et al (2012) Case-based lung image categorization and retrieval for interstitial lung diseases: clinical workflows. Int J Comput Assist Radiol Surg 7:97–110
Aisen AM, Broderick LS, Winer-Muram H et al (2003) Automated storage and retrieval of thin-section CT images to assist diagnosis: system description and preliminary assessment. Radiology 228:265–270
Endo M, Aramaki T, Asakura K et al (2012) Content-based image-retrieval system in chest computed tomography for a solitary pulmonary nodule: method and preliminary experiments. Int J Comput Assist Radiol Surg 7:331–338
Oosawa A, Kurosaki A, Kanada S et al (2019) Development of a CT image case database and content-based image retrieval system for non-cancerous respiratory diseases: method and preliminary assessment. Respir Investig 57:490–498
Silva CIS, Müller NL, Lynch DA et al (2008) Chronic hypersensitivity pneumonitis: differentiation from idiopathic pulmonary fibrosis and nonspecific interstitial pneumonia by using thin-section CT. Radiology 246:288–297
Derakhshani A, Ding J, Vijayasarathi A (2021) On-call radiology 2020: where trainees look for help in a high stakes and time sensitive environment. Clin Imaging 77:219–223
Ozutemiz C, Dicle O, Koremezli N (2015) How Turkish radiology residents access information related to their profession in this social media and smartphone era. Eur J Radiol Open 2:129–133
El-Ali A, Kamal F, Cabral CL, Squires JH (2019) Comparison of traditional and web-based medical student teaching by radiology residents. J Am Coll Radiol 16:492–495
Walsh SLF, Calandriello L, Sverzellati N et al (2016) Interobserver agreement for the ATS/ERS/JRS/ALAT criteria for a UIP pattern on CT. Thorax 71:45–51
Choe J, Hwang HJ, Seo JB et al (2022) Content-based image retrieval by using deep learning for interstitial lung disease diagnosis with chest CT. Radiology 302:187–197
Kwee TC, Kwee RM (2021) Workload of diagnostic radiologists in the foreseeable future based on recent scientific advances: growth expectations and role of artificial intelligence. Insights Imaging 12:88
Open access funding provided by Austrian Science Fund (FWF). This study has received partial funding from the European Commission (Bigmedilytics 780495), the Austrian Science Fund (FWF) (P 35189), and the Vienna Science and Technology Fund (WWTF) Project Nr. LS20-065 (PREDICTOME).
The scientific guarantor of this publication is Dr. Georg Langs.
Conflict of interest
The authors of this manuscript declare relationships with the following companies:
Sebastian Röhrich: Consulting activities for contextflow GmbH;
Markus Krenn: Chief Product Officer of contextflow GmbH
Georg Langs: Chief Scientist / Co-Founder of contextflow GmbH, Speaker fees: Roche, Siemens, Boehringer-Ingelheim;
Julie Sufana: Chief Marketing Officer of contextflow GmbH
Rui Zhang: Chief Medical Officer of contextflow GmbH
Jakob Scheithe: Developer / Application Engineer of contextflow GmbH
Helmut Prosch: Speakers fees: Boehringer-Ingelheim, Roche, Novartis, MSD, BMS, GSK, Chiesi, AstraZeneca.
Statistics and biometry
One of the authors has significant statistical expertise.
Written informed consent was waived by the IRB.
Institutional Review Board approval was obtained.
• performed at one institution
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Röhrich, S., Heidinger, B.H., Prayer, F. et al. Impact of a content-based image retrieval system on the interpretation of chest CTs of patients with diffuse parenchymal lung disease. Eur Radiol 33, 360–367 (2023). https://doi.org/10.1007/s00330-022-08973-3
- Lung diseases, Interstitial
- Tomography, X-ray computed
- Diagnosis, Computer assisted
- Artificial intelligence