Skip to main content
SpringerLink
Log in

Deep Learning-Based Detect-Then-Track Pipeline for Treatment Outcome Assessments in Immunotherapy-Treated Liver Cancer

  • Published:
Journal of Imaging Informatics in Medicine Aims and scope Submit manuscript

Abstract

Accurate treatment outcome assessment is crucial in clinical trials. However, due to the image-reading subjectivity, there exist discrepancies among different radiologists. The situation is common in liver cancer due to the complexity of abdominal scans and the heterogeneity of radiological imaging manifestations in liver subtypes. Therefore, we developed a deep learning-based detect-then-track pipeline that can automatically identify liver lesions from 3D CT scans then longitudinally track target lesions, thereby providing the evaluation of RECIST treatment outcomes in liver cancer. We constructed and validated the pipeline on 173 multi-national patients (344 venous-phase CT scans) consisting of a public dataset and two in-house cohorts of 28 centers. The proposed pipeline achieved a mean average precision of 0.806 and 0.726 of lesion detection on the validation and test sets. The model’s diameter measurement reliability and consistency are significantly higher than that of clinicians (p = 1.6 × 10−4). The pipeline can make precise lesion tracking with accuracies of 85.7% and 90.8% then finally yield the RECIST accuracies of 82.1% and 81.4% on the validation and test sets. Our proposed pipeline can provide precise and convenient RECIST outcome assessments and has the potential to aid clinicians with more efficient therapeutic decisions.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Data Availability

The LiTS data presented in this study are openly available in https://competitions.codalab.org/competitions/17094. All in-house data in this study cannot be public due to the privacy regulation, but are available from the corresponding author on reasonable request.

Code Availability

The python program that implemented the pipeline as well as the pre-trained nnDetection models are open-source on GitHub: https://github.com/alibool/detect-then-track.

Abbreviations

AP:

Average precision

CNN:

Convolutional neural networks

CT:

Computed tomography

FROC:

Free-response Receiver Operating Characteristic

HCC:

Hepatocellular carcinoma

IoU:

Intersections over unions

LiTS:

Liver tumor segmentation

RECIST:

Response Evaluation Criteria in Solid Tumors

SOD:

Sum of the diameter

References

  1. ClinicalTrials.gov. CTG Labs - NCBI. ClinicalTrials.gov. https://clinicaltrials.gov/ [Accessed December 1, 2023].

  2. Eisenhauer E, Therasse P, Bogaerts J, Schwartz LH, Sargent DJ, Ford R, et al. New response evaluation criteria in solid tumours: Revised RECIST guideline (version 1.1). European Journal of Cancer [Internet]. 2009;45(2):228–47. Available from: https://doi.org/10.1016/j.ejca.2008.10.026

  3. Hadjiiski LM, Weizer AZ, Alva A, Caoili EM, Cohan RH, Cha K, et al. Treatment response assessment for bladder cancer on CT based on computerized volume analysis, World Health Organization Criteria, and RECIST. American Journal of Roentgenology [Internet]. 2015;205(2):348–52. Available from: https://doi.org/10.2214/ajr.14.13732

  4. Lau DH, Seibert JA, Gandara DR, Laptalo L, Geraghty E, Coulon C. Computer-Assisted image analysis of bronchioloalveolar carcinoma. Clinical Lung Cancer [Internet]. 2005;6(5):281–6. Available from: https://doi.org/10.3816/clc.2005.n.006

  5. Gonen KLCM, Ford R. Monitoring Reader Metrics in Blinded Independent Central Review of Oncology Studies. Journal of Clinical Trials [Internet]. 2015;05(04). Available from: https://doi.org/10.4172/2167-0870.1000230

  6. Rr F, Sc M, Fraunberger J. Adjudication Rates between Readers in Blinded Independent Central Review of Oncology Studies. Journal of Clinical Trials [Internet]. 2016;06(05). Available from: https://doi.org/10.4172/2167-0870.1000289

  7. Saxena S, Jena B, Gupta N, Das S, Sarmah D, Bhattacharya P, et al. Role of artificial intelligence in radiogenomics for cancers in the era of Precision Medicine. Cancers [Internet]. 2022;14(12):2860. Available from: https://doi.org/10.3390/cancers14122860

  8. Hamamoto R, Suvarna K, Yamada M, Kobayashi K, Shinkai N, Miyake M, et al. Application of Artificial Intelligence technology in Oncology: Towards the establishment of precision Medicine. Cancers [Internet]. 2020;12(12):3532. Available from: https://doi.org/10.3390/cancers12123532

  9. Bhinder B, Gilvary C, Madhukar NS, Elemento O. Artificial intelligence in cancer research and precision medicine. Cancer Discovery [Internet]. 2021;11(4):900–15. Available from: https://doi.org/10.1158/2159-8290.cd-21-0090

  10. Jiang H, Diao Z, Shi T, Zhou Y, Wang F, Hu W, et al. A review of deep learning-based multiple-lesion recognition from medical images: classification, detection and segmentation. Computers in Biology and Medicine [Internet]. 2023;157:106726. Available from: https://doi.org/10.1016/j.compbiomed.2023.106726

  11. Ren S, He K, Girshick R, Sun J. Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. IEEE Transactions on Pattern Analysis and Machine Intelligence [Internet]. 2017;39(6):1137–49. Available from: https://doi.org/10.1109/tpami.2016.2577031

  12. Lin TY, Goyal P, Girshick R, He K, Dollár P. Focal loss for dense object detection. IEEE Transactions on Pattern Analysis and Machine Intelligence [Internet]. 2020;42(2):318–27. Available from: https://doi.org/10.1109/tpami.2018.2858826

  13. Nasrullah N, Sang J, Alam MS, Mateen M, Cai B, Hu H. Automated Lung Nodule Detection and Classification Using Deep Learning Combined with Multiple Strategies. Sensors [Internet]. 2019;19(17):3722. Available from: https://doi.org/10.3390/s19173722

  14. Pehrson LM, Nielsen MB, Lauridsen CA. Automatic Pulmonary nodule Detection Applying deep learning or machine learning algorithms to the LIDC-IDRI Database: A Systematic review. Diagnostics [Internet]. 2019;9(1):29. Available from: https://doi.org/10.3390/diagnostics9010029

  15. Sourlos N, Wang J, Nagaraj Y, Van Ooijen PMA, Vliegenthart R. Possible bias in supervised deep learning algorithms for CT lung nodule detection and classification. Cancers [Internet]. 2022;14(16):3867. Available from: https://doi.org/10.3390/cancers14163867

  16. Tang Y, Yan K, Xiao J. One click lesion RECIST measurement and segmentation on CT scans. In: Lecture Notes in Computer Science [Internet]. 2020. p. 573–83. Available from: https://doi.org/10.1007/978-3-030-59719-1_56

  17. Gul S, Khan MS, Bibi A, Khandakar A, Ayari MA, Chowdhury MEH. Deep learning techniques for liver and liver tumor segmentation: A review. Computers in Biology and Medicine [Internet]. 2022;147:105620. Available from: https://doi.org/10.1016/j.compbiomed.2022.105620

  18. Survarachakan S, Prasad PJR, Naseem R, De Frutos JP, Kumar RP, Langø T, et al. Deep learning for image-based liver analysis — A comprehensive review focusing on malignant lesions. Artificial Intelligence in Medicine [Internet]. 2022;130:102331. Available from: https://doi.org/10.1016/j.artmed.2022.102331

  19. Bilic P, Christ PF, Vorontsov E, Chlebus G, Chen H, Dou Q, et al. The Liver Tumor Segmentation Benchmark (LITS). Medical Image Analysis [Internet]. 2023;84:102680. Available from: https://doi.org/10.1016/j.media.2022.102680

  20. Yushkevich PA, Piven J, Hazlett HC, Smith RG, Ho S, Gee JC, et al. User-guided 3D active contour segmentation of anatomical structures: Significantly improved efficiency and reliability. NeuroImage [Internet]. 2006;31(3):1116–28. Available from: https://doi.org/10.1016/j.neuroimage.2006.01.015

  21. Baumgärtner M, Jäger PF, Isensee F, Maier-Hein KH. NNDetection: a self-configuring method for medical object detection. In: Lecture Notes in Computer Science [Internet]. 2021. p. 530–9. Available from: https://doi.org/10.1007/978-3-030-87240-3_51

  22. Jaeger PF, Kohl S a. A, Bickelhaupt S, Isensee F, Kuder TA, Schlemmer H, et al. Retina U-Net: Embarrassingly Simple Exploitation of Segmentation Supervision for Medical Object Detection. arXiv (Cornell University) [Internet]. 2018; Available from: https://arxiv.org/pdf/1811.08661

  23. Lin TY, Dollár P, Girshick R, He K, Hariharan B, Belongie S. Feature Pyramid Networks for Object Detection. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) [Internet]. 2017; Available from: https://doi.org/10.1109/cvpr.2017.106

  24. Ronneberger O, Fischer P, Brox T. U-NET: Convolutional Networks for Biomedical Image Segmentation. In: Lecture Notes in Computer Science [Internet]. 2015. p. 234–41. Available from: https://doi.org/10.1007/978-3-319-24574-4_28

  25. Cardoso MJ, Li W, Brown R, Ma N, Kerfoot E, Wang Y, et al. MONAI: An open-source framework for deep learning in healthcare [Internet]. arXiv.org. 2022. Available from: https://arxiv.org/abs/2211.02701

  26. Sobek J, Inojosa JRM, Inojosa BJM, Rassoulinejad-Mousavi SM, Conte GM, Lopez-Jimenez F, et al. MedYOLO: a medical Image object Detection framework [Internet]. arXiv.org. 2023. Available from: https://arxiv.org/abs/2312.07729

  27. Wittmann B, Navarro FA, Shit S, Menze B. Focused decoding enables 3D anatomical detection by transformers. The Journal of Machine Learning for Biomedical Imaging [Internet]. 2023;2(February 2023):72–95. Available from: https://melba-journal.org/2023:003

  28. Heinrich MP, Jenkinson M, Brady M, Schnabel JA. MRF-Based Deformable Registration and Ventilation Estimation of lung CT. IEEE Transactions on Medical Imaging [Internet]. 2013;32(7):1239–48. Available from: https://doi.org/10.1109/tmi.2013.2246577

  29. Avants B, Tustison NJ, Song G. Advanced Normalization Tools: v1.0. Insight Journal [Internet]. 2009; Available from: https://doi.org/10.54294/uvnhin

  30. Dong X, Zhou Y, Wang L, Peng J, Lou Y, Fan Y. Liver cancer detection using hybridized fully convolutional neural network based on deep learning framework. IEEE Access [Internet]. 2020;8:129889–98. Available from: https://doi.org/10.1109/access.2020.3006362

  31. Vorontsov E, Cerny M, Régnier P, Di Jorio L, Pal C, Lapointe R, et al. Deep Learning for Automated Segmentation of Liver Lesions at CT in Patients with Colorectal Cancer Liver Metastases. Radiology [Internet]. 2019;1(2):180014. Available from: https://doi.org/10.1148/ryai.2019180014

  32. Sun C, Guo S, Zhang H, Li J, Chen M, Ma S, et al. Automatic segmentation of liver tumors from multiphase contrast-enhanced CT images based on FCNs. Artificial Intelligence in Medicine [Internet]. 2017;83:58–66. Available from: https://doi.org/10.1016/j.artmed.2017.03.008

Download references

Acknowledgements

All computations were run on the Siyuan-1 cluster supported by the Center for High Performance Computing at Shanghai Jiao Tong University.

Funding

This study was supported by National Natural Science Foundation of China (No. 12171318), Shanghai Science and Technology Commission (No. 21ZR1436300), Shanghai Jiao Tong University STAR Grant (No. 20190102), Medical Engineering Cross Fund of Shanghai Jiao Tong University (No. YG2023ZD21), Collaboration of SJTU-Beigene (No. 21H010104287).

Author information

Author notes

  1. Jie Zhou and Yujia Xia contributed equally.

Authors and Affiliations

  1. Department of Statistics, School of Mathematical Sciences, Shanghai Jiao Tong University, Shanghai, China

    Jie Zhou & Zhangsheng Yu

  2. Department of Bioinformatics and Biostatistics, School of Life Sciences and Biotechnology, Shanghai Jiao Tong University, Shanghai, China

    Yujia Xia & Zhangsheng Yu

  3. SJTU-Yale Joint Center for Biostatistics and Data Science, School of Life Sciences and Biotechnology, Shanghai Jiao Tong University, Shanghai, China

    Jie Zhou, Yujia Xia & Zhangsheng Yu

  4. Global Statistics and Data Science, BeiGene, Shanghai, China

    Xiaolei Xun

  5. Clinical Research Institute, Shanghai Jiao Tong University School of Medicine, Shanghai, China

    Zhangsheng Yu

Authors
  1. Jie Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yujia Xia
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xiaolei Xun
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Zhangsheng Yu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Zhangsheng Yu.

Ethics declarations

Ethics Approval and Consent to Participate

The study data passed the ethics review (Application No. I2021173I) and were approved by the Human Genetic Resource Administration of China (Approval No. [2021] GH5565). All participants provided informed consent.

Conflict of Interest

The authors declare no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhou, J., Xia, Y., Xun, X. et al. Deep Learning-Based Detect-Then-Track Pipeline for Treatment Outcome Assessments in Immunotherapy-Treated Liver Cancer. J Digit Imaging. Inform. med. (2024). https://doi.org/10.1007/s10278-024-01132-8

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10278-024-01132-8

Keywords

Search

Navigation