Abstract
In a complex disease such as tuberculosis, the evidence for the disease and its evolution may be present in multiple modalities such as clinical, genomic, or imaging data. Effective patient-tailored outcome prediction and therapeutic guidance will require fusing evidence from these modalities. Such multimodal fusion is difficult since the evidence for the disease may not be uniform across all modalities, not all modality features may be relevant, or not all modalities may be present for all patients. All these nuances make simple methods of early, late, or intermediate fusion of features inadequate for outcome prediction. In this paper, we present a novel fusion framework using multiplexed graphs and derive a new graph neural network for learning from such graphs. Specifically, the framework allows modalities to be represented through their targeted encodings, and models their relationship explicitly via multiplexed graphs derived from salient features in a combined latent space. We present results that show that our proposed method outperforms state-of-the-art methods of fusing modalities for multi-outcome prediction on a large Tuberculosis (TB) dataset.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Asad, M., Mahmood, A., Usman, M.: A machine learning-based framework for predicting treatment failure in tuberculosis: a case study of six countries. Tuberculosis 123, 101944 (2020)
Baltrušaitis, T., Ahuja, C., Morency, L.P.: Multimodal machine learning: a survey and taxonomy. IEEE Trans. Pattern Anal. Mach. Intell. 41(2), 423–443 (2018)
Cozzo, E., de Arruda, G.F., Rodrigues, F.A., Moreno, Y.: Multiplex networks (2018). https://doi.org/10.1007/978-3-319-92255-3, http://link.springer.com/10.1007/978-3-319-92255-3
DeLong, E.R., DeLong, D.M., Clarke-Pearson, D.L.: Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics, 837–845 (1988)
Domenico, M.D., et al.: Mathematical formulation of multilayer networks. Phys. Rev. X 3, 041022 (2014). https://doi.org/10.1103/PHYSREVX.3.041022/FIGURES/5/MEDIUM, https://journals.aps.org/prx/abstract/10.1103/PhysRevX.3.041022
Ferriani, S., Fonti, F., Corrado, R.: The social and economic bases of network multiplexity: exploring the emergence of multiplex ties. Strateg. Org. 11, 7–34 (2013). https://doi.org/10.1177/1476127012461576
Gabrielian, A., et al.: TB depot (data exploration portal): a multi-domain tuberculosis data analysis resource. PLOS ONE 14(5), e0217410 (2019). https://doi.org/10.1371/journal.pone.0217410, http://dx.plos.org/10.1371/journal.pone.0217410
Huang, G., Liu, Z., Van Der Maaten, L., Weinberger, K.Q.: Densely connected convolutional networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4700–4708 (2017)
Jones, P., et al.: InterProScan 5: genome-scale protein function classification. Bioinformatics (Oxford, England) 30(9), 1236–40 (2014). https://doi.org/10.1093/bioinformatics/btu031
Kant, Y., et al.: Spatially aware multimodal transformers for TextVQA. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12354, pp. 715–732. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58545-7_41
Kipf, T.N., Welling, M.: Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907 (2016)
Kivelä, M., Arenas, A., Barthelemy, M., Gleeson, J.P., Moreno, Y., Porter, M.A.: Multilayer networks. J. Complex Netw. 2, 203–271 (2014). https://doi.org/10.1093/COMNET/CNU016, https://academic.oup.com/comnet/article/2/3/203/2841130
Lahat, D., Adali, T., Jutten, C.: Multimodal data fusion: an overview of methods, challenges, and prospects. Proc. IEEE 103(9), 1449–1477 (2015)
Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 (2017)
Maggioni, M.A., Breschi, S., Panzarasa, P.: Multiplexity, growth mechanisms and structural variety in scientific collaboration networks, 20, 185–194 (4 2013). https://doi.org/10.1080/13662716.2013.791124, https://www.tandfonline.com/doi/abs/10.1080/13662716.2013.791124
Manson, A.L., et al.: Genomic analysis of globally diverse mycobacterium tuberculosis strains provides insights into the emergence and spread of multidrug resistance. Nat. Genet. 49(3), 395–402 (2017)
Muñoz-Sellart, M., Cuevas, L., Tumato, M., Merid, Y., Yassin, M.: Factors associated with poor tuberculosis treatment outcome in the southern region of Ethiopia. Int. J. Tuberc. Lung Dis. 14(8), 973–979 (2010)
World Health Organization: Treatment of Tuberculosis: Guidelines. World Health Organization (2010)
Sargin, M.E., Erzin, E., Yemez, Y., Tekalp, A.M.: Multimodal speaker identification using canonical correlation analysis. In: 2006 IEEE International Conference on Acoustics Speech and Signal Processing Proceedings, vol. 1, p. I. IEEE (2006)
Sauer, C.M., et al.: Feature selection and prediction of treatment failure in tuberculosis. PLoS ONE 13(11), e0207491 (2018)
Scarselli, F., Gori, M., Tsoi, A.C., Hagenbuchner, M., Monfardini, G.: The graph neural network model. IEEE Trans. Neural Netw. 20(1), 61–80 (2008)
Schlichtkrull, M., Kipf, T.N., Bloem, P., van den Berg, R., Titov, I., Welling, M.: Modeling relational data with graph convolutional networks. In: Gangemi, A., et al. (eds.) ESWC 2018. LNCS, vol. 10843, pp. 593–607. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-93417-4_38
Seabolt, E.E., et al.: OMXWare, A Cloud-Based Platform for Studying Microbial Life at Scale (nov 2019), http://arxiv.org/abs/1911.02095
Subramanian, V., Do, M.N., Syeda-Mahmood, T.: Multimodal fusion of imaging and genomics for lung cancer recurrence prediction. In: 2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI), pp. 804–808. IEEE (2020)
Vu, T.D., Yang, H.J., Nguyen, V.Q., Oh, A.R., Kim, M.S.: Multimodal learning using convolution neural network and sparse autoencoder. In: 2017 IEEE International BigComp, pp. 309–312. IEEE (2017)
Wang, H., Subramanian, V., Syeda-Mahmood, T.: Modeling uncertainty in multi-modal fusion for lung cancer survival analysis. In: Proceedings - International Symposium on Biomedical Imaging 2021-April, 1169–1172 (2021). https://doi.org/10.1109/ISBI48211.2021.9433823
Wang, H., Subramanian, V., Syeda-Mahmood, T.: Modeling uncertainty in multi-modal fusion for lung cancer survival analysis. In: 2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI), pp. 1169–1172. IEEE (2021)
Wang, H., Yushkevich, P.: Multi-atlas segmentation with joint label fusion and corrective learning-an open source implementation. Front. Neuroinform. 7, 27 (2013)
Wáng, Y.X.J., Chung, M.J., Skrahin, A., Rosenthal, A., Gabrielian, A., Tartakovsky, M.: Radiological signs associated with pulmonary multi-drug resistant tuberculosis: an analysis of published evidences. Quant. Imaging Med. Surg. 8(2), 161 (2018)
Xu, K., Hu, W., Leskovec, J., Jegelka, S.: How powerful are graph neural networks? arXiv preprint arXiv:1810.00826 (2018)
Yuan, H., Yu, H., Wang, J., Li, K., Ji, S.: On explainability of graph neural networks via subgraph explorations. In: International Conference on Machine Learning, pp. 12241–12252. PMLR (2021)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
D’Souza, N.S. et al. (2022). Fusing Modalities by Multiplexed Graph Neural Networks for Outcome Prediction in Tuberculosis. In: Wang, L., Dou, Q., Fletcher, P.T., Speidel, S., Li, S. (eds) Medical Image Computing and Computer Assisted Intervention – MICCAI 2022. MICCAI 2022. Lecture Notes in Computer Science, vol 13437. Springer, Cham. https://doi.org/10.1007/978-3-031-16449-1_28
Download citation
DOI: https://doi.org/10.1007/978-3-031-16449-1_28
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-16448-4
Online ISBN: 978-3-031-16449-1
eBook Packages: Computer ScienceComputer Science (R0)
