Abstract
The discovery of patient-specific imaging markers that are predictive of future disease outcomes can help us better understand individual-level heterogeneity of disease evolution. In fact, deep learning models that can provide data-driven personalized markers are much more likely to be adopted in medical practice. In this work, we demonstrate that data-driven biomarker discovery can be achieved through a counterfactual synthesis process. We show how a deep conditional generative model can be used to perturb local imaging features in baseline images that are pertinent to subject-specific future disease evolution and result in a counterfactual image that is expected to have a different future outcome. Candidate biomarkers, therefore, result from examining the set of features that are perturbed in this process. Through several experiments on a large-scale, multi-scanner, multi-center multiple sclerosis (MS) clinical trial magnetic resonance imaging (MRI) dataset of relapsing-remitting (RRMS) patients, we demonstrate that our model produces counterfactuals with changes in imaging features that reflect established clinical markers predictive of future MRI lesional activity at the population level. Additional qualitative results illustrate that our model has the potential to discover novel and subject-specific predictive markers of future activity.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Baumgartner, C.F., Koch, L.M., Tezcan, K., Ang, J., Konukoglu, E.: Visual feature attribution using Wasserstein GANs. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Los Alamitos, CA, USA, pp. 8309–8319. IEEE Computer Society (2018). https://doi.org/10.1109/CVPR.2018.00867. https://doi.ieeecomputersociety.org/10.1109/CVPR.2018.00867
Charachon, M., Cournède, P.-H., Hudelot, C., Ardon, R.: Visual explanation by unifying adversarial generation and feature importance attributions. In: Reyes, M., et al. (eds.) IMIMIC/TDA4MedicalData 2021. LNCS, vol. 12929, pp. 44–55. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-87444-5_5
Cohen, J.P., et al.: Gifsplanation via latent shift: a simple autoencoder approach to progressive exaggeration on chest X-rays. CoRR abs/2102.09475 (2021). https://arxiv.org/abs/2102.09475
Dadar, M., Mahmoud, S., Narayanan, S., Collins, D.L., Arnold, D.L., Maranzano, J.: Diffusely abnormal white matter converts to T2 lesion volume in the absence of MRI-detectable acute inflammation. Brain 145, 2008–2017 (2022). https://doi.org/10.1093/BRAIN/AWAB448. https://academic.oup.com/brain/article/145/6/2008/6470368
Elliott, C., et al.: Slowly expanding/evolving lesions as a magnetic resonance imaging marker of chronic active multiple sclerosis lesions. Multiple Sclerosis 25, 1915–1925 (2019). https://doi.org/10.1177/1352458518814117. https://pubmed.ncbi.nlm.nih.gov/30566027/
Esmaeilzadeh, S., Belivanis, D.I., Pohl, K.M., Adeli, E.: End-to-end Alzheimer’s disease diagnosis and biomarker identification. CoRR abs/1810.00523 (2018). https://arxiv.org/abs/1810.00523
Goodfellow, I., et al.: Generative adversarial nets. In: Advances in Neural Information Processing Systems, vol. 27 (2014)
Goyal, Y., Wu, Z., Ernst, J., Batra, D., Parikh, D., Lee, S.: Counterfactual visual explanations. In: International Conference on Machine Learning, pp. 2376–2384. PMLR (2019)
Hartung, H.P., et al.: Predictors of disease activity in 857 patients with MS treated with interferon beta-1b. J. Neurol. 262(11), 2466–2471 (2015)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)
Hore, A., Ziou, D.: Image quality metrics: PSNR vs. SSIM. In: 2010 20th International Conference on Pattern Recognition, pp. 2366–2369. IEEE (2010)
Isensee, F., et al.: nnU-Net: self-adapting framework for U-Net-based medical image segmentation. arXiv preprint arXiv:1809.10486 (2018)
Jiang, H., et al.: A multi-label deep learning model with interpretable Grad-CAM for diabetic retinopathy classification. In: 2020 42nd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), pp. 1560–1563. IEEE (2020)
Mothilal, R.K., Sharma, A., Tan, C.: Explaining machine learning classifiers through diverse counterfactual explanations. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, pp. 607–617 (2020)
Panwar, H., Gupta, P., Siddiqui, M.K., Morales-Menendez, R., Bhardwaj, P., Singh, V.: A deep learning and grad-cam based color visualization approach for fast detection of Covid-19 cases using chest X-ray and CT-scan images. Chaos Solitons Fractals 140, 110190 (2020)
Pawlowski, N., Coelho de Castro, D., Glocker, B.: Deep structural causal models for tractable counterfactual inference. In: Advances in Neural Information Processing Systems, vol. 33, pp. 857–869 (2020)
Perez, E., Strub, F., De Vries, H., Dumoulin, V., Courville, A.: Film: visual reasoning with a general conditioning layer. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 32 (2018)
Reinhold, J.C., Carass, A., Prince, J.L.: A structural causal model for MR images of multiple sclerosis. CoRR abs/2103.03158 (2021). https://arxiv.org/abs/2103.03158
Romeo, M., et al.: Clinical and MRI predictors of response to interferon-beta and glatiramer acetate in relapsing-remitting multiple sclerosis patients. Eur. J. Neurol. 20(7), 1060–1067 (2013)
Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-CAM: visual explanations from deep networks via gradient-based localization. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 618–626 (2017)
Siddiquee, M.M.R., et al.: Learning fixed points in generative adversarial networks: from image-to-image translation to disease detection and localization. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 191–200 (2019)
Van Looveren, A., Klaise, J., Vacanti, G., Cobb, O.: Conditional generative models for counterfactual explanations. arXiv preprint arXiv:2101.10123 (2021)
Vollmer, T., et al.: A randomized placebo-controlled phase III trial of oral laquinimod for multiple sclerosis. J. Neurol. 261(4), 773–783 (2014)
Xia, T., Chartsias, A., Tsaftaris, S.A.: Adversarial pseudo healthy synthesis needs pathology factorization. arXiv preprint arXiv:1901.07295 (2019)
Zia, T., Murtaza, S., Bashir, N., Windridge, D., Nisar, Z.: VANT-GAN: adversarial learning for discrepancy-based visual attribution in medical imaging. Pattern Recogn. Lett. 156, 112–118 (2022)
Acknowledgements
The authors are grateful to the International Progressive MS Alliance for supporting this work (grant number: PA-1412-02420), and to the companies who generously provided the clinical trial data that made it possible: Biogen, BioMS, MedDay, Novartis, Roche/Genentech, and Teva. Funding was also provided by the Natural Sciences and Engineering Research Council of Canada, the Canadian Institute for Advanced Research (CIFAR) Artificial Intelligence Chairs program, and a technology transfer grant from Mila - Quebec AI Institute. S.A. Tsaftaris acknowledges the support of Canon Medical and the Royal Academy of Engineering and the Research Chairs and Senior Research Fellowships scheme (grant RCSRF1819 / 8 / 25). Supplementary computational resources and technical support were provided by Calcul Québec, WestGrid, and Compute Canada. This work was made possible by the end-to-end deep learning experimental pipeline developed in collaboration with our colleagues Justin Szeto, Eric Zimmerman, and Kirill Vasilevski. Additionally, the authors would like to thank Louis Collins and Mahsa Dadar for preprocessing the MRI data, Zografos Caramanos, Alfredo Morales Pinzon, Charles Guttmann and István Mórocz for collating the clinical data, Sridar Narayanan, Maria-Pia Sormani for their MS expertise.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Kumar, A. et al. (2022). Counterfactual Image Synthesis for Discovery of Personalized Predictive Image Markers. In: Kakileti, S.T., et al. Artificial Intelligence over Infrared Images for Medical Applications and Medical Image Assisted Biomarker Discovery. MIABID AIIIMA 2022 2022. Lecture Notes in Computer Science, vol 13602. Springer, Cham. https://doi.org/10.1007/978-3-031-19660-7_11
Download citation
DOI: https://doi.org/10.1007/978-3-031-19660-7_11
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-19659-1
Online ISBN: 978-3-031-19660-7
eBook Packages: Computer ScienceComputer Science (R0)
