Abstract
Medical data analysis often combines both imaging and tabular data processing using machine learning algorithms. While previous studies have investigated the impact of attention mechanisms on deep learning models, few have explored integrating attention modules and tabular data. In this paper, we introduce TabAttention, a novel module that enhances the performance of Convolutional Neural Networks (CNNs) with an attention mechanism that is trained conditionally on tabular data. Specifically, we extend the Convolutional Block Attention Module to 3D by adding a Temporal Attention Module that uses multi-head self-attention to learn attention maps. Furthermore, we enhance all attention modules by integrating tabular data embeddings. Our approach is demonstrated on the fetal birth weight (FBW) estimation task, using 92 fetal abdominal ultrasound video scans and fetal biometry measurements. Our results indicate that TabAttention outperforms clinicians and existing methods that rely on tabular and/or imaging data for FBW prediction. This novel approach has the potential to improve computer-aided diagnosis in various clinical workflows where imaging and tabular data are combined. We provide a source code for integrating TabAttention in CNNs at https://github.com/SanoScience/Tab-Attention.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Bano, S., et al.: AutoFB: automating fetal biometry estimation from standard ultrasound planes. In: de Bruijne, M., et al. (eds.) MICCAI 2021. LNCS, vol. 12907, pp. 228–238. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-87234-2_22
Benacerraf, B.R., Gelman, R., Frigoletto, F.D., Jr.: Sonographically estimated fetal weights: accuracy and limitation. Am. J. Obstet. Gynecol. 159(5), 1118–1121 (1988)
Campbell, S., Wilkin, D.: Ultrasonic measurement of fetal abdomen circumference in the estimation of fetal weight. BJOG: Int. J. Obstet. Gynaecol. 82(9), 689–697 (1975)
Chen, T., Guestrin, C.: XGBoost: a scalable tree boosting system. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 785–794. KDD ’16, ACM, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939785
Duanmu, H., et al.: Prediction of pathological complete response to neoadjuvant chemotherapy in breast cancer using deep learning with integrative imaging, molecular and demographic data. In: Martel, A.L., et al. (eds.) MICCAI 2020. LNCS, vol. 12262, pp. 242–252. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-59713-9_24
Guan, Y., et al.: Predicting esophageal fistula risks using a multimodal self-attention network. In: de Bruijne, M., et al. (eds.) MICCAI 2021. LNCS, vol. 12905, pp. 721–730. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-87240-3_69
Hadlock, F.P., Harrist, R., Sharman, R.S., Deter, R.L., Park, S.K.: Estimation of fetal weight with the use of head, body, and femur measurements-a prospective study. Am. J. Obstet. Gynecol. 151(3), 333–337 (1985)
Holste, G., Partridge, S.C., Rahbar, H., Biswas, D., Lee, C.I., Alessio, A.M.: End-to-end learning of fused image and non-image features for improved breast cancer classification from MRI. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3294–3303 (2021)
Huang, S.C., Pareek, A., Seyyedi, S., Banerjee, I., Lungren, M.P.: Fusion of medical imaging and electronic health records using deep learning: a systematic review and implementation guidelines. NPJ Digital Med. 3(1), 136 (2020)
Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. In: International Conference on Learning Representations (ICLR) (2015)
Liu, M., Zhang, J., Adeli, E., Shen, D.: Joint classification and regression via deep multi-task multi-channel learning for Alzheimer’s disease diagnosis. IEEE Trans. Biomed. Eng. 66(5), 1195–1206 (2018)
Lu, Y., Zhang, X., Fu, X., Chen, F., Wong, K.K.: Ensemble machine learning for estimating fetal weight at varying gestational age. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 9522–9527 (2019)
Pedregosa, F., et al.: Scikit-learn: machine learning in Python. J. Mach. Learn. Res. 12, 2825–2830 (2011)
Płotka, S., et al.: BabyNet: residual transformer module for birth weight prediction on fetal ultrasound video. In: Medical Image Computing and Computer Assisted Intervention-MICCAI 2022: 25th International Conference, Singapore, September 18–22, 2022, Proceedings, Part IV, pp. 350–359. Springer (2022). https://doi.org/10.1007/978-3-031-16440-8_34
Płotka, S., et al.: Deep learning fetal ultrasound video model match human observers in biometric measurements. Phys. Med. Biol. 67(4), 045013 (2022)
Pölsterl, S., Wolf, T.N., Wachinger, C.: Combining 3D image and tabular data via the dynamic affine feature map transform. In: de Bruijne, M., et al. (eds.) MICCAI 2021. LNCS, vol. 12905, pp. 688–698. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-87240-3_66
Pressman, E.K., Bienstock, J.L., Blakemore, K.J., Martin, S.A., Callan, N.A.: Prediction of birth weight by ultrasound in the third trimester. Obstet. Gynecol. 95(4), 502–506 (2000)
Salomon, L., et al.: ISUOG practice guidelines: ultrasound assessment of fetal biometry and growth. Ultrasound Obstet. Gynecol. 53(6), 715–723 (2019)
Shaw, P., Uszkoreit, J., Vaswani, A.: Self-attention with relative position representations. arXiv preprint arXiv:1803.02155 (2018)
Sherman, D.J., Arieli, S., Tovbin, J., Siegel, G., Caspi, E., Bukovsky, I.: A comparison of clinical and ultrasonic estimation of fetal weight. Obstet. Gynecol. 91(2), 212–217 (1998)
Tao, J., Yuan, Z., Sun, L., Yu, K., Zhang, Z.: Fetal birthweight prediction with measured data by a temporal machine learning method. BMC Med. Informa. Decis. Making 21(1), 1–10 (2021)
Tran, D., Wang, H., Torresani, L., Ray, J., LeCun, Y., Paluri, M.: A closer look at spatiotemporal convolutions for action recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6450–6459 (2018)
Vaswani, A., et al.: Attention is all you need. In: Advances in Neural Information Processing Systems 30 (2017)
Wang, X., Liu, D., Zhang, Y., Li, Y., Wu, S.: A spatiotemporal multi-stream learning framework based on attention mechanism for automatic modulation recognition. Digit. Signal Process. 130, 103703 (2022)
Woo, S., Park, J., Lee, J.Y., Kweon, I.S.: CBAM: convolutional block attention module. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 3–19 (2018)
Yadav, S., Rai, A.: Frequency and temporal convolutional attention for text-independent speaker recognition. In: ICASSP 2020–2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6794–6798. IEEE (2020)
Acknowledgements
This work is supported by the European Union’s Horizon 2020 research and innovation programme under grant agreement Sano No 857533 and the International Research Agendas programme of the Foundation for Polish Science, co-financed by the European Union under the European Regional Development Fund.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Grzeszczyk, M.K. et al. (2023). TabAttention: Learning Attention Conditionally on Tabular Data. In: Greenspan, H., et al. Medical Image Computing and Computer Assisted Intervention – MICCAI 2023. MICCAI 2023. Lecture Notes in Computer Science, vol 14226. Springer, Cham. https://doi.org/10.1007/978-3-031-43990-2_33
Download citation
DOI: https://doi.org/10.1007/978-3-031-43990-2_33
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-43989-6
Online ISBN: 978-3-031-43990-2
eBook Packages: Computer ScienceComputer Science (R0)
