Abstract
Early screening is an important way to reduce the mortality of hepatocellular carcinoma (HCC) and improve its prognosis. As a noninvasive, economic, and safe procedure, B-mode ultrasound is currently the most common imaging modality for diagnosing and monitoring HCC. However, because of the difficulty of extracting effective image features and modeling longitudinal data, few studies have focused on early prediction of HCC based on longitudinal ultrasound images. In this paper, to address the above challenges, we propose a spatiotemporal attention network (STA-HCC) that adopts a convolutional-neural-network–transformer framework. The convolutional neural network includes a feature-extraction backbone and a proposed regions-of-interest attention block, which learns to localize regions of interest automatically and extract effective features for HCC prediction. The transformer can capture long-range dependencies and nonlinear dynamics from ultrasound images through a multihead self-attention mechanism. Also, an age-based position embedding is proposed in the transformer to embed a more-appropriate positional relationship among the longitudinal ultrasound images. Experiments conducted on our dataset of 6170 samples collected from 619 cirrhotic subjects show that STA-HCC achieves impressive performance, with an area under the receiver-operating-characteristic curve of 77.5%, an accuracy of 70.5%, a sensitivity of 69.9%, and a specificity of 70.5%. The results show that our method achieves state-of-the-art performance compared with other popular sequence models.
Y. Zhang and C. Hu are the co-first authors.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Bruix, J., Reig, M., Sherman, M.: Evidence-based diagnosis, staging, and treatment of patients with hepatocellular carcinoma. Gastroenterology 150(4), 835–853 (2016)
Cho, K., et al.: Learning phrase representations using RNN encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078 (2014)
Cui, Y., Jia, M., Lin, T.Y., Song, Y., Belongie, S.: Class-balanced loss based on effective number of samples. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9268–9277 (2019)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)
Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)
Hu, J., Shen, L., Sun, G.: Squeeze-and-excitation networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7132–7141 (2018)
Ioannou, G.N., et al.: Assessment of a deep learning model to predict hepatocellular carcinoma in patients with hepatitis c cirrhosis. JAMA Netw. Open 3(9), e2015626–e2015626 (2020)
Kanda, T., Goto, T., Hirotsu, Y., Moriyama, M., Omata, M.: Molecular mechanisms driving progression of liver cirrhosis towards hepatocellular carcinoma in chronic hepatitis b and c infections: a review. Int. J. Mol. Sci. 20(6), 1358 (2019)
Li, X., Zhong, Z., Wu, J., Yang, Y., Lin, Z., Liu, H.: Expectation-maximization attention networks for semantic segmentation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9167–9176 (2019)
Marrero, J.A., et al.: Diagnosis, staging, and management of hepatocellular carcinoma: 2018 practice guidance by the American association for the study of liver diseases. Hepatology 68(2), 723–750 (2018)
Obi, S., et al.: Combination therapy of intraarterial 5-fluorouracil and systemic interferon-alpha for advanced hepatocellular carcinoma with portal venous invasion. Cancer Interdisc. Int. J. Am. Cancer Soc. 106(9), 1990–1997 (2006)
Papatheodoridis, G., et al.: PAGE-B predicts the risk of developing hepatocellular carcinoma in Caucasians with chronic hepatitis B on 5-year antiviral therapy. J. Hepatol. 64(4), 800–806 (2016)
Pham, T., Tran, T., Phung, D., Venkatesh, S.: Predicting healthcare trajectories from medical records: a deep learning approach. J. Biomed. Inform. 69, 218–229 (2017)
Poh, Z., et al.: Real-world risk score for hepatocellular carcinoma (RWS-HCC): a clinically practical risk predictor for HCC in chronic hepatitis B. Gut 65(5), 887–888 (2016)
Schuster, M., Paliwal, K.K.: Bidirectional recurrent neural networks. IEEE Trans. Signal Process. 45(11), 2673–2681 (1997)
Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: visual explanations from deep networks via gradient-based localization. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 618–626 (2017)
Sharma, D., Xu, W.: phyLoSTM: a novel deep learning model on disease prediction from longitudinal microbiome data. Bioinformatics 37(21), 3707–3714 (2021)
Tan, M., Le, Q.: Efficientnet: rethinking model scaling for convolutional neural networks. In: International Conference on Machine Learning, pp. 6105–6114. PMLR (2019)
Vaswani, A., et al.: Attention is all you need. In: Advances in Neural Information Processing Systems, pp. 5998–6008 (2017)
Wang, H., Li, C., Zhang, J., Wang, J., Ma, Y., Lian, Y.: A new LSTM-based gene expression prediction model: L-GEPM. J. Bioinform. Comput. Biol. 17(04), 1950022 (2019)
Wang, X., Girshick, R., Gupta, A., He, K.: Non-local neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7794–7803 (2018)
Wong, G.L.H., et al.: Liver stiffness-based optimization of hepatocellular carcinoma risk score in patients with chronic hepatitis B. J. Hepatol. 60(2), 339–345 (2014)
Wu, S., et al.: Hepatocellular carcinoma prediction models in chronic hepatitis B: a systematic review of 14 models and external validation. Clin. Gastroenterol. Hepatol. 19(12), 2499–2513 (2021)
Yang, J.D., Hainaut, P., Gores, G.J., Amadou, A., Plymoth, A., Roberts, L.R.: A global view of hepatocellular carcinoma: trends, risk, prevention and management. Nat. Rev. Gastroenterol. Hepatol. 16(10), 589–604 (2019)
Yao, Z., et al.: Preoperative diagnosis and prediction of hepatocellular carcinoma: radiomics analysis based on multi-modal ultrasound images. BMC Cancer 18(1), 1–11 (2018)
Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2921–2929 (2016)
Acknowledgements
This research is supported by the National Natural Science Foundation of China (No. 82172020 and 81772923) and Guangdong Provincial Key Laboratory of Medical Image Processing (No. 2020B1212060039).
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Zhang, Y. et al. (2022). Spatiotemporal Attention for Early Prediction of Hepatocellular Carcinoma Based on Longitudinal Ultrasound Images. In: Wang, L., Dou, Q., Fletcher, P.T., Speidel, S., Li, S. (eds) Medical Image Computing and Computer Assisted Intervention – MICCAI 2022. MICCAI 2022. Lecture Notes in Computer Science, vol 13433. Springer, Cham. https://doi.org/10.1007/978-3-031-16437-8_51
Download citation
DOI: https://doi.org/10.1007/978-3-031-16437-8_51
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-16436-1
Online ISBN: 978-3-031-16437-8
eBook Packages: Computer ScienceComputer Science (R0)
