Skip to main content
SpringerLink
Log in

Generate Structured Radiology Report from CT Images Using Image Annotation Techniques: Preliminary Results with Liver CT

  • Original Paper
  • Published:
Journal of Digital Imaging Aims and scope Submit manuscript

Abstract

A medical annotation system for radiology images extracts clinically useful information from the images, allowing the machines to infer useful abstract semantics and become capable of automatic reasoning and making diagnostic decision. It also supplies human-interpretable explanation for the images. We have implemented a computerized framework that, given a liver CT image, predicts radiological annotations with high accuracy, in order to generate a structured report, which includes predicting very specific high-level semantic content. Each report of a liver CT image is related to different inhomogeneous parts like the liver, lesion, and vessel. We put forward a claim that gathering all kinds of features is not suitable for filling all parts of the report. As a matter of fact, for each group of annotations, one should find and extract the best feature that results in the best answers for that specific annotation. To this end, the main challenge is discovering the relationships between these specific semantic concepts and their association with the low-level image features. Our framework was implemented by combining a set of the state-of-the-art low-level imaging features. In addition, we propose a novel feature (DLBP (deep local binary pattern)) based on LBP that incorporates multi-slice analysis in CT images and further improves the performance. In order to model our annotation system, two methods were used, namely multi-class support vector machine (SVM) and random subspace (RS) which is an ensemble learning method. Applying this representation leads to a high prediction accuracy of 93.1% despite its relatively low dimension in comparison with the existing works.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

References

  1. Atam P Dhawan. Medical image analysis, volume 31. John Wiley & Sons, 2011.

  2. Loveymi S, Shadgar B, Osareh A: An efficient approach to automated medical image annotation. International Review on Computers and Software 6(5):749–759, 2011

    Google Scholar 

  3. Tommasi T, Orabona F, Caputo B: Discriminative cue integration for medical image annotation. Pattern Recognition Letters 29(15):1996–2002, 2008

    Article  Google Scholar 

  4. Tatiana Tommasi, Francesco Orabona, and Barbara Caputo. An svm confidence-based approach to medical image annotation. In Workshop of the Cross-Language Evaluation Forum for European Languages, pages 696–703. Springer, 2008.

  5. Tatiana Tommasi, Barbara Caputo, Petra Welter, Mark Oliver Guldld, and Thomas M Deserno. Overview of the clef 2009 medical image annotation track. In Workshop of the Cross-Language Evaluation Forum for European Languages, pages 85–93. Springer, 2009.

  6. Dimitrovski I, Kocev D, Loskovska S, Dzeroski S: Hierarchical annotation of medical images. Pattern Recognition 44(10-11):2436–2449, 2011

    Article  Google Scholar 

  7. Demner-Fushman D, Antani S, Simpson M, Thoma GR: Annotation and retrieval of clinically relevant images. International Journal of Medical Informatics 78(12):e59–e67, 2009

    Article  Google Scholar 

  8. Zehra Camlica, Hamid R Tizhoosh, and Farzad Khalvati. Medical image classification via SVM using LBP features from saliency-based folded data. In 2015 IEEE 14th International Conference on Machine Learning and Applications (ICMLA), pages 128–132. IEEE, 2015.

  9. Riadh Bouslimi, Abir Messaoudi, and Jalel Akaichi. Using a bag of words for automatic medical image annotation with a latent semantic. arXiv preprint arXiv:1306.0178, 2013.

  10. Alaidine Ben Ayed, Mustapha Kardouchi, and Sid-Ahmed Selouani. Rotation invariant fuzzy shape contexts based on eigenshapes and Fourier transforms for efficient radiological image retrieval. In 2012 International Conference on Multimedia Computing and Systems, pages 266–271. IEEE, 2012.

  11. Zare MR, Mueen A, Seng WC: Automatic medical x-ray image classification using annotation. Journal of Digital Imaging 27(1):77–89, 2014

    Article  Google Scholar 

  12. Mueen A, Zainuddin R, Baba MS: Automatic multilevel medical image annotation and retrieval. Journal of digital imaging 21(3):290–295, 2008

    Article  CAS  Google Scholar 

  13. Wang Y, Wang L, Rastegar-Mojarad M, Moon S, Shen F, NaveedAfzal SL, Zeng Y, Mehrabi S, Sohn S et al.: Clinical information extraction applications: a literature review. Journal of biomedical informatics 77:34–49, 2018

    Article  Google Scholar 

  14. Kokciyan N, Turkay R, Uskudarli S, Yolum P, Bakir B, Acar B: Semantic description of liver ct images: an ontological approach. IEEE journal of biomedical and health informatics 18(4):1363–1369, 2014

    Article  Google Scholar 

  15. Gao H, Aiello Bowles EJ, Carrell D, Buist DSM: Using natural language processing to extract mammographic findings. Journal of biomedical informatics 54:77–84, 2015

    Article  Google Scholar 

  16. Castro SM, Tseytlin E, Medvedeva O, Mitchell K, Visweswaran S, Bekhuis T, Jacobson RS: Automated annotation and classification of bi-rads assessment from radiology reports. Journal of biomedical informatics 69:177–187, 2017

    Article  Google Scholar 

  17. Imane Nedjar, Said Mahmoudi, Mohammed Amine Chikh, Khadidja Abi-Yad, and ZouheyrBouafia. Automatic annotation of liver ct image: ImageCLEFmed 2015. In CLEF (Working Notes), 2015.

  18. Neda Barzegar Marvasti, Maria del Mar Roldan Garcia, Suzan Uskudarli, Jose Francisco Aldana Montes, and Burak Acar. Overview of the ImageCLEF 2015 liver ct annotation task. In CLEF (Working Notes), 2015.

  19. Kumar A, Dyer S, Kim J, Li C, Leong PHW, Fulham M, Feng D: Adapting content-based image retrieval techniques for the semantic annotation of medical images. Computerized Medical Imaging and Graphics 49:37–45, 2016

    Article  Google Scholar 

  20. Zhiyun Xue, Sameer Antani, L Rodney Long, and George R Thoma. Automatic multi-label annotation of abdominal ct images using CBIR. In Medical Imaging 2017: Imaging Informatics for Healthcare, Research, and Applications, volume 10138, page 1013807. International Society for Optics and Photonics

  21. Depeursinge A, Kurtz C, Beaulieu C, Napel S, Rubin D: Predicting visual semantic descriptive terms from radiological image data: preliminary results with liver lesions in ct. IEEE transactions on medical imaging 33(8):1669–1676, 2014

    Article  Google Scholar 

  22. Xu Y, Lin L, Hu H, Wang D, Zhu W, Wang J, Han X-H, Chen Y-W: Texture-specific bag of visual words model and spatial cone matching based method for the retrieval of focal liver lesions using multiphase contrast-enhanced ct images. International journal of computer-assisted radiology and surgery 13(1):151–164, 2018

    Article  Google Scholar 

  23. Pol Cirujeda, Henning Muller, Daniel Rubin, Todd A Aguilera, Billy W Loo, MaximilianDiehn, Xavier Binefa, and Adrien Depeursinge. 3d Riesz-wavelet-based covariance descriptors for texture classification of lung nodule tissue in ct. In 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pages 7909– 7912. IEEE, 2015.

  24. Arai K, Herdiyeni Y, Okumura H: Comparison of 2d and 3d local binary pattern in lung cancer diagnosis. Int J Adv Comput Sci Appl 3(4):89–95, 2012

    Google Scholar 

  25. Zhang J, Xia Y, Xie Y, Fulham M, Feng DD: Classification of medical images in the biomedical literature by jointly using deep and handcrafted visual features. IEEE Journal of biomedical and health informatics 22(5):1521–1530, 2018

    Article  Google Scholar 

  26. Nanni L, Paci M, Brahnam S, Ghidoni S: An ensemble of visual features for Gaussians of local descriptors and non-binary coding for texture descriptors. Expert Systems with Applications 82:27–39, 2017

    Article  Google Scholar 

  27. Gang Wang, David Forsyth, and Derek Hoiem. Comparative object similarity for improved recognition with few or no examples. In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages 3525–3532. IEEE, 2010.

  28. Carlotta Domeniconi and Bojun Yan. Nearest neighbor ensemble. In Proceedings of the 17th International Conference on Pattern Recognition, 2004. ICPR 2004., volume 1, pages 228–231. IEEE, 2004.

  29. Castellano G, Fanelli AM, Sforza G, Torsello MA: Shape annotation for intelligent image retrieval. Applied Intelligence 44(1):179–195, 2016

    Article  Google Scholar 

  30. Varish N, Pal AK: A novel image retrieval scheme using gray level co-occurrence matrix descriptors of discrete cosine transform based residual image. Applied Intelligence 48(9):2930–2953, 2018

    Article  Google Scholar 

  31. Soh L-K, Tsatsoulis C: Texture analysis of SAR sea ice imagery using gray level co-occurrence matrices. IEEE Transactions on geoscience and remote sensing 37(2):780–795, 1999

    Article  Google Scholar 

  32. Janis Fehr and Hans Burkhardt. 3d rotation invariant local binary patterns. In 2008 19th International Conference on Pattern Recognition, pages 1–4. IEEE, 2008.

  33. Vladimir Vapnik. The nature of statistical learning theory. Springer science & business media, 2013.

  34. Wu T-F, Lin C-J, Weng RC: Probability estimates for multi-class classification by pairwise coupling. Journal of Machine Learning Research 5(Aug):975–1005, 2004

    Google Scholar 

  35. Ludmila I Kuncheva. Combining pattern classifiers: methods and algorithms, 2nd Ed. John Wiley & Sons, 2014.

  36. Image Nedjar, Said Mahmoudi, and Mohammed Amine Chikh. Content-based medical image retrieval for liver ct annotation. Transactions on Machine Learning and Artificial Intelligence, 5(4), 2017.

  37. Assaf B, Spanier NC, Sosna J, Acar B, Joskowicz L: A fully automatic end-to-end method for content-based image retrieval of ct scans with similar liver lesion annotations. International journal of computer-assisted radiology and surgery 13(1):165–174, 2018

    Article  Google Scholar 

  38. Kurtz C, Depeursinge A, Napel S, Beaulieu CF, Rubin DL: On combining image-based and ontological semantic dissimilarities for medical image retrieval applications. Medical image analysis 18(7):1082–1100, 2014

    Article  Google Scholar 

  39. Spanier AB, Cohen D, Joskowicz L: A new method for the automatic retrieval of medical cases based on the RadLex ontology. International journal of computer assisted radiology and surgery 12(3):471–484, 2017

    Article  CAS  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Computer Engineering Department, Bu-Ali Sina University, Shahid Fahmideh blvd., Hamedan, Iran

    Samira Loveymi, Mir Hossein Dezfoulian & Muharram Mansoorizadeh

Authors
  1. Samira Loveymi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mir Hossein Dezfoulian
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Muharram Mansoorizadeh
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Muharram Mansoorizadeh.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendix

Appendix

Table 5 List of liver CT annotations [18]
Full size table
Table 6 Accuracy of feature groups and classifiers for each question
Full size table
Table 7 Accuracy of feature groups and classifiers for each question
Full size table

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Loveymi, S., Dezfoulian, M.H. & Mansoorizadeh, M. Generate Structured Radiology Report from CT Images Using Image Annotation Techniques: Preliminary Results with Liver CT. J Digit Imaging 33, 375–390 (2020). https://doi.org/10.1007/s10278-019-00298-w

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10278-019-00298-w

Keywords

Search

Navigation