Skip to main content

Advertisement

SpringerLink
Log in

MaPPeRTrac: A Massively Parallel, Portable, and Reproducible Tractography Pipeline

  • Research
  • Published:
Neuroinformatics Aims and scope Submit manuscript

Abstract

Large-scale diffusion MRI tractography remains a significant challenge. Users must orchestrate a complex sequence of instructions that requires many software packages with complex dependencies and high computational costs. We developed MaPPeRTrac, an edge-centric tractography pipeline that simplifies and accelerates this process in a wide range of high-performance computing (HPC) environments. It fully automates either probabilistic or deterministic tractography, starting from a subject’s magnetic resonance imaging (MRI) data, including structural and diffusion MRI images, to the edge density image (EDI) of their structural connectomes. Dependencies are containerized with Singularity (now called Apptainer) and decoupled from code to enable rapid prototyping and modification. Data derivatives are organized with the Brain Imaging Data Structure (BIDS) to ensure that they are findable, accessible, interoperable, and reusable following FAIR principles. The pipeline takes full advantage of HPC resources using the Parsl parallel programming framework, resulting in the creation of connectome datasets of unprecedented size. MaPPeRTrac is publicly available and tested on commercial and scientific hardware, so it can accelerate brain connectome research for a broader user community. MaPPeRTrac is available at: https://github.com/LLNL/mappertrac.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

Data availability

I have included the link for the HCP database in the Information Sharing Statement. The MRI data repositories for TRACK-TBI patients, TRACK-TBI control subjects, and TRACK-Pilot subjects will be publicly available in the near future.

Notes

  1. GitHub repo: https://github.com/LLNL/mappertrac

References

  • Babuji, Y., Woodard, A., Li, Z., Katz, D.S., Clifford, B., Kumar, R., ..., & Chard, K. (2019). Parsl: Pervasive parallel programming in python. In The 28th ACM international symposium on high-performance parallel and distributed computing (hpdc). https://doi.org/10.1145/3307681.3325400

  • Basser, P. J., Mattiello, J., & LeBihan, D. (1994). MR diffusion tensor spectroscopy and imaging. Biophysical Journal, 66(1), 259–267.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  • Behrens, T., Johansen-Berg, H., Jbabdi, S., Rushworth, M., & Woolrich, M. (2007). Probabilistic diffusion tractography with multiple fibre orientations. What can we gain? NeuroImage, 23, 144–155.

    Article  Google Scholar 

  • Behrens, T., Woolrich, M., Jenkinson, M., Johansen-Berg, H., Nunes, R., Clare, S., ..., Smith, S. (2003). Characterization and propagation of uncertainty in diffusion-weighted MR imaging. Magnetic Resonance in Medicine, 50 (5), 1077–1088. https://doi.org/10.1002/mrm.10609

  • Bodien, Y.G., McCrea, M., Dikmen, S., Temkin, N., Boase, K., Machamer, J., ..., Investigators, T.R.A.C.K.-T.B.I. (2018). Optimizing outcome assessment in multicenter tbi trials: Perspectives from track-tbi and the tbi endpoints development initiative. The Journal of Head Trauma Rehabilitation, 33(3), 147–157. https://doi.org/10.1097/HTR.0000000000000367

  • Boettiger, C. (2015). An introduction to Docker for reproducible research. ACM SIGOPS Operating Systems Review, 49(1), 71–79.

    Article  Google Scholar 

  • Conturo, T. E., Lori, N. F., Cull, T. S., Akbudak, E., Snyder, A. Z., Shimony, J. S., …, & Raichle, M. E. (1999). Tracking neuronal fiber pathways in the living human brain. Proceedings of the National Academy of Sciences, 96(18), 10422–10427.

  • Côté, M.-A., Girard, G., Boré, A., Garyfallidis, E., Houde, J.-C., & Descoteaux, M. (2013). Tractometer: Towards validation of tractography pipelines. Medical Image Analysis, 17(7), 844–857. https://doi.org/10.1016/j.media.2013.03.009

  • Cui, Z., Zhong, S., Xu, P., Gong, G., & He, Y. (2013). Panda: A pipeline toolbox for analyzing brain diffusion images. Frontiers in Human Neuroscience, 7, 42. https://doi.org/10.3389/fnhum.2013.00042

    Article  PubMed  PubMed Central  Google Scholar 

  • Dagum, L., & Menon, R. (1998). OpenMP: An industry standard API for shared-memory programming. IEEE Computational Science and Engineering, 5(1), 46–55.

    Article  Google Scholar 

  • Desai, N. (2005). Cobalt: an open source platform for hpc system software research. Edinburgh BG/L System Software Workshop (pp. 803–820).

  • Desikan, R.S., Ségonne, F., Fischl, B., Quinn, B.T., Dickerson, B.C., Blacker, D., ..., et al. (2006). An automated labeling system for subdividing the human cerebral cortex on MRI scans into gyral based regions of interest. NeuroImage, 31(3), 968–980.

  • Fischl, B. (2012). FreeSurfer. Neuroimage, 62(2), 774–781.

    Article  PubMed  Google Scholar 

  • Foster, M., & Deardorff, M. (2017). Open science framework (osf). Journal of the Medical Library Association, 105(2), 203. https://doi.org/10.5195/JMLA.2017.88

  • Gentzsch, W. (2001). Sun grid engine: Towards creating a compute power grid. Proceedings First IEEE/ACM International Symposium on Cluster Computing and the Grid (pp. 35–36).

  • Gorgolewski, K. J., Auer, T., Calhoun, V. D., Craddock, R. C., Das, S., Duff, E. P., ..., et al. (2016). The brain imaging data structure, a format for organizing and describing outputs of neuroimaging experiments. Scientific Data, 3(1), 1–9.

  • Gorgolewski, K.J., Alfaro-Almagro, F., Auer, T., Bellec, P., Capot ̆a, M., Chakravarty, M. M., ..., et al. (2017). Bids apps: Improving ease of use, accessibility, and reproducibility of neuroimaging data analysis methods. PLoS Computational Biology, 13(3), e1005209.

  • Jenkinson, M., Beckmann, C. F., Behrens, T. E., Woolrich, M. W., & Smith, S. M. (2012). FSL. Neuroimage, 62(2), 782–790.

    Article  PubMed  Google Scholar 

  • Karcher, N. R., & Barch, D. M. (2021). The ABCD study: understanding the development of risk for mental and physical health outcomes. Neuropsychopharmacol, 46, 131–142. https://doi.org/10.1038/s41386-020-0736-6

    Article  Google Scholar 

  • Kiar, G., Bridgeford, E. W., Chandrashekhar, V., Mhembere, D., Burns, R., Gray Roncal, W. R., & Vogelstein, J. T. (2017). A comprehensive cloud framework for accurate and reliable human connectome estimation and meganalysis. BioRxiv. https://doi.org/10.1101/188706

  • Kurtzer, G. M., Sochat, V., & Bauer, M. W. (2017). Singularity: Scientific containers for mobility of compute. PloS One, 12(5), e0177459.

  • León, E. A., D’Hooge, T., Hanford, N., Karlin, I., Pankajakshan, R., Foraker, J., ... & Leininger, M. L. (2020). TOSS-2020: A commodity software stack for HPC. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis (pp. 1–15). IEEE.

  • LLNL. (2021). TOSS description. Retrieved 2021–04–30, from https://computing.llnl.gov/projects/toss-speeding-commodity-cluster-computing

  • Madhyastha, T. M., Koh, N., Day, T. K. M., Hernández-Fernández, M., Kelley, A., Peterson, D. J., ..., & Grabowski, T. J. (2017). Running neuroimaging applications on amazon web services: How, when, and at what cost? Frontiers in Neuroinformatics, 11, 63–63. https://doi.org/10.3389/fninf.2017.00063

  • Maximov, I. I., Alnæs, D., & Westlye, L. T. (2019). Towards an optimised processing pipeline for diffusion magnetic resonance imaging data: Effects of artefact corrections on diffusion metrics and their age associations in uk biobank. Human Brain Mapping, 40(14), 4146–4162. https://doi.org/10.1002/hbm.24691

  • Moon, J. Y., Bremer, P.-T., Mukherjee, P., Markowitz, A. J., Palacios, E. M., Cai, L. T., ..., the TRACK-TBI Consortium. (2020). Mappertrac: A massively parallel, portable, and reproducible tractography pipeline. BioRxiv, 2020-12.

  • Moon, J. Y., Mukherjee, P., Madduri, R. K., Markowitz, A. J., Cai, L. T., Palacios, E. M., ..., Bremer, P.-T. (2022). The case for optimized edge-centric tractography at scale. Frontiers in Neuroinformatics, 16, 752471.

  • Mukherjee, P., Berman, J. I., Chung, S. W., Hess, C. P., & Henry, R. G. (2008a). Diffusion tensor MR imaging and fiber tractography: Theoretic underpinnings. American Journal of Neuroradiology, 29(4), 632–641.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  • Mukherjee, P., Chung, S. W., Berman, J. I., Hess, C. P., & Henry, R. G. (2008b). Diffusion tensor MR imaging and fiber tractography: Technical considerations. American Journal of Neuroradiology, 29(5), 843–852.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  • Owen, J. P., Wang, M. B., & Mukherjee, P. (2016). Periventricular white matter is a nexus for network connectivity in the human brain. Brain Connectivity, 6(7), 548–557. https://doi.org/10.1089/brain.2016.0431

    Article  PubMed  Google Scholar 

  • Owen, J. P., Chang, Y. S., & Mukherjee, P. (2015). Edge density imaging: Mapping the anatomic embedding of the structural connectome within the white matter of the human brain. NeuroImage, 109, 402–417. https://doi.org/10.1016/j.neuroimage.2015.01.007

  • Palacios, E. M., Yuh, E. L., Mac Donald, C. L., Bourla, I., Wren-Jarvis, J., Sun, X., ..., et al. (2022). Diffusion tensor imaging reveals elevated diffusivity of white matter microstructure that is independently associated with long-term outcome after mild traumatic brain injury: a TRACK-TBI study. Journal of Neurotrauma, 39(19–20), 1318–1328.

  • Payabvash, S., Palacios, E. M., Owen, J. P., Wang, M. B., Tavassoli, T., Gerdes, M., ..., Mukherjee, P. (2019). White matter connectome edge density in children with autism spectrum disorders: Potential imaging biomarkers using machine-learning models. Brain Connectivity, 9(2), 209–220, https://doi.org/10.1089/brain.2018.0658

  • Python Software Foundation. (n.d.). Python package index - pypi. Python Software Foundation. Retrieved from https://pypi.org/

  • Qi, X., & Arfanakis, K. (2021). Regionconnect: Rapidly extracting standardized brain connectivity information in voxel-wise neuroimaging studies. NeuroImage, 225, 117462. https://doi.org/10.1016/j.neuroimage.2020.117462

  • Raji, C. A., Wang, M. B., Nguyen, N., Owen, J. P., Palacios, E. M., Yuh, E. L., & Mukherjee, P. (2020). Connectome mapping with edge density imaging differentiates pediatric mild traumatic brain injury from typically developing controls: proof of concept. Pediatric Radiology, 50(11), 1594–1601. https://doi.org/10.1007/s00247-020-04743-9

    Article  PubMed  PubMed Central  Google Scholar 

  • Reber, J., Hwang, K., Bowren, M., Bruss, J., Mukherjee, P., Tranel, D., & Boes, A.D. (2021). Cognitive impairment after focal brain lesions is better predicted by damage to structural than functional network hubs. Proceedings of the National Academy of Sciences, 118 (19). https://doi.org/10.1073/pnas.2018784118

  • Rex, D. E., Ma, J. Q., & Toga, A. W. (2003). The LONI pipeline processing environment. NeuroImage, 19(3), 1033–1048. https://doi.org/10.1016/S1053-8119(03)00185-X

  • Roine, T., Jeurissen, B., Perrone, D., Aelterman, J., Philips, W., Sijbers, J., & Leemans, A. (2019). Reproducibility and intercorrelation of graph theoretical measures in structural brain connectivity networks. Medical Image Analysis, 52, 56–67. https://doi.org/10.1016/j.media.2018.10.009

  • Schirner, M., Rothmeier, S., Jirsa, V. K., McIntosh, A. R., & Ritter, P. (2015). An automated pipeline for constructing personalized virtual brains from multimodal neuroimaging data. NeuroImage, 117, 343–357. https://doi.org/10.1016/j.neuroimage.2015.03.055

  • Smith, S. M., Jenkinson, M., Woolrich, M. W., Beckmann, C. F., Behrens, T. E., Johansen-Berg, H., ..., et al. (2004). Advances in functional and structural MR image analysis and implementation as FSL. NeuroImage, 23, S208–S219.

  • Sporns, O. (2013). The human connectome: Origins and challenges. NeuroImage, 80, 53–61.

    Article  PubMed  Google Scholar 

  • Sporns, O., Tononi, G., & Kötter, R. (2005). The human connectome: a structural description of the human brain. PLoS Computational Biology, 1(4), e42.

    Article  PubMed  PubMed Central  Google Scholar 

  • Tannenbaum, T., Wright, D., Miller, K., & Livny, M. (2001). Condor – a distributed job scheduler. In T. Sterling (Ed.), Beowulf cluster computing with Linux. MIT Press.

    Google Scholar 

  • Thain, D., Tannenbaum, T., & Livny, M. (2005). Distributed computing in practice: The Condor experience. Concurrency - Practice and Experience, 17(2–4), 323–356.

    Article  Google Scholar 

  • Theaud, G., Houde, J.-C., Boré, A., Rheault, F., Morency, F., & Descoteaux, M. (2020). Tractoflow: A robust, efficient and reproducible diffusion MRI pipeline leveraging Nextflow & Singularity. NeuroImage, 218, 116889. https://doi.org/10.1016/j.neuroimage.2020.116889

  • Top500. (2023). Top 500 supercomputer sites, release: November 2023. Retrieved from https://www.top500.org/

  • Tournier, J. D., Mori, S., & Leemans, A. (2011). Diffusion tensor imaging and beyond. Magnetic Resonance in Medicine, 65(6), 1532.

    Article  PubMed  PubMed Central  Google Scholar 

  • Tournier, J.-D., Calamante, F., & Connelly, A. (2012). MRtrix: Diffusion tractography in crossing fiber regions. International Journal of Imaging Systems and Technology, 22(1), 53–66.

    Article  Google Scholar 

  • Van Essen, D. C., Smith, S. M., Barch, D. M., Behrens, T. E. J., Yacoub, E., Ugurbil, K., & The WU-Minn HCP Consortium. (2013). The WU-Minn human connectome project: An overview. NeuroImage, 80, 62–79. https://doi.org/10.1016/j.neuroimage.2013.05.041

  • Vingelmann, P., Fitzek, F. H., NVIDIA. (2020). CUDA, release: 10.2.89. Retrieved from https://developer.nvidia.com/cuda-toolkit

  • Volkow, N. D., Koob, G. F., Croyle, R. T., Bianchi, D. W., Gordon, J. A., Koroshetz, W. J., ..., & Weiss, S. R. (2018). The conception of the ABCD study: From substance use to a broad NIH collaboration. Developmental Cognitive Neuroscience, 32, 4–7.

  • Wilkinson, M. D., Dumontier, M., Aalbersberg, I. J., Appleton, G., Axton, M., Baak, A., ..., Mons, B. (2016). The FAIR guiding principles for scientific data management and stewardship. Scientific Data, 3(1), 160018. https://doi.org/10.1038/sdata.2016.18

  • Yoo, A.B., Jette, M.A., Grondona, M. (2003). Slurm: Simple linux utility for resource management. Workshop on Job Scheduling Strategies for Parallel Processing (pp. 44–60).

  • Yue, J. K., Vassar, M. J., Lingsma, H. F., Cooper, S. R., Okonkwo, D. O., Valadka, A. B., ..., & Sinha, T. K. (2013). Transforming research and clinical knowledge in traumatic brain injury pilot: multicenter implementation of the common data elements for traumatic brain injury. Journal of neurotrauma, 30(22), 1831–1844.

  • Yuh, E. L., Cooper, S. R., Mukherjee, P., Yue, J. K., Lingsma, H. F., Gordon, W. A., ..., Sinha, T. K. (2014). Diffusion tensor imaging for outcome prediction in mild traumatic brain injury: A TRACK-TBI study. Journal of Neurotrauma, 31(17), 1457–1477. https://doi.org/10.1089/neu.2013.3171

Download references

Acknowledgements

The research is funded by the United States Department of Energy under the DOE Office of Science, Advanced Scientific Computing Research. Support is organized under The Co-Design for Artificial Intelligence and Computing at Scale for Extremely Large, Complex Datasets projects (Grant #KJ040301). Geoffrey Manley and Pratik Mukherjee disclose grants from the United States Department of Defense – TBI Endpoints Development Initiative (Grant #W81XWH-14-2-0176), TRACK-TBI Precision Medicine (Grant #W81XWH-18-2-0042), and TRACK-TBI NETWORK (Grant #W81XWH-15-9-0001); NIH-NINDS – TRACK-TBI (Grant #U01NS086090); and the National Football League (NFL) Scientific Advisory Board – TRACK-TBI LONGITUDINAL. The United States Department of Energy supports Dr. Manley for a precision medicine collaboration. One Mind has provided funding for TRACK-TBI patients stipends and support to clinical sites. He has received an unrestricted gift from the NFL to the UCSF Foundation to support the research efforts of the TRACK-TBI NETWORK. Dr. Manley has also received funding from NeuroTrauma Sciences LLC to support TRACK-TBI data curation efforts. Additionally, Abbott Laboratories has provided funding for add-in TRACK-TBI clinical studies. Amy Markowitz receives funding from the Department of Defense TBI Endpoints Development Initiative (Grant #W81XWH-14-2-0176) and TRACK-TBI NETWORK (Grant #W81XWH-15-9-0001). Ms. Markowitz also receives salary support from the United States Department of Energy precision medicine collaboration and the philanthropic organization, One Mind. In addition, we are grateful that Sam Payabvash and his group at Yale University School of Medicine have joined the efforts to test the latest release of MaPPeRTrac and pursue its application to a broad range of neuropathological research using EDI and connectomes. The TRACK-TBI Investigators: Shankar Gopinath, MD, Baylor College of Medicine; Ramesh Grandhi, MD MS, University of Utah; C. Dirk Keene, MD PhD, University of Washington; Michael McCrea, PhD, Medical College of Wisconsin; Randall Merchant, PhD, Virginia Commonwealth University; Laura B. Ngwenya, MD, PhD, University of Cincinnati; Ava Puccio, PhD, University of Pittsburgh; David Schnyer, PhD, UT Austin; Sabrina R. Taylor, PhD, University of California, San Francisco; John K. Yue, MD, University of California, San Francisco; Esther L. Yuh, MD, PhD, University of California, San Francisco; Ross Zafonte, DO, Harvard Medical School.

Author information

Author notes

  1. Lanya T. Cai and Joseph Moon contributed equally to this work.

Authors and Affiliations

  1. Department of Radiology and Biomedical Imaging, University of California, San Francisco, 185 Berry St., San Francisco, CA, 94107, USA

    Lanya T. Cai, Eva M. Palacios & Pratik Mukherjee

  2. Lawrence Livermore National Laboratory, 7000 East Ave, Livermore, CA, 94550, USA

    Joseph Moon, Shivsundaram Shankar & Peer-Timo Bremer

  3. Beckman Institute, University of Illinois at Urbana-Champaign, 405 N Mathews Ave, Urbana, IL, 61801, USA

    Paul B. Camacho & Aaron T. Anderson

  4. Department of Radiology, Washington University in St. Louis, 510 S Kingshighway Blvd, St. Louis, MO, 63110, USA

    Won Jong Chwa

  5. Bioengineering Department, University of Illinois at Urbana-Champaign, 506 S Mathews Ave, Urbana, IL, 61801, USA

    Bradley P. Sutton

  6. Department of Neurosurgery, University of California, San Francisco, 400 Parnassus Ave, San Francisco, CA, 94143, USA

    Amy J. Markowitz & Geoffrey T. Manley

  7. Argonne National Laboratory, 9700 S Cass Ave, Lemont, IL, 60439, USA

    Alexis Rodriguez & Ravi K. Madduri

Authors
  1. Lanya T. Cai
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Joseph Moon
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Paul B. Camacho
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Aaron T. Anderson
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Won Jong Chwa
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Bradley P. Sutton
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Amy J. Markowitz
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Eva M. Palacios
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Alexis Rodriguez
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Geoffrey T. Manley
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Shivsundaram Shankar
    View author publications

    You can also search for this author in PubMed Google Scholar

  12. Peer-Timo Bremer
    View author publications

    You can also search for this author in PubMed Google Scholar

  13. Pratik Mukherjee
    View author publications

    You can also search for this author in PubMed Google Scholar

  14. Ravi K. Madduri
    View author publications

    You can also search for this author in PubMed Google Scholar

Consortia

the TRACK-TBI Investigators

  • Shankar Gopinath
  • , Ramesh Grandhi
  • , C. Dirk Keene
  • , Michael McCrea
  • , Randall Merchant
  • , Laura B. Ngwenya
  • , Ava Puccio
  • , David Schnyer
  • , Sabrina R. Taylor
  • , John K. Yue
  • , Esther L. Yuh
  •  & Ross Zafonte

Contributions

In alphabetical order: A.J.M., G.T.M., P.M., P.T.B., R.K.M., and S.S. contributed to the study conception and design. Data acquisition and management were led by A.J.M., G.T.M., P.M., and the TRACK-TBI Investigators. Software development and data analysis were performed by A.R., A.T.A., B.P.S., E.M.P., J.M., L.T.C., P.B.C., and W.J.C.. The first draft of the manuscript was written by J.M. and L.T.C.. B.P.S., P.M., and R.K.M. revised the manuscript. All authors read and approved the final manuscript.

Corresponding authors

Correspondence to Pratik Mukherjee or Ravi K. Madduri.

Ethics declarations

Competing interests

The authors declare no competing interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Information Sharing Statement

To access open-source software and public database used in the present study, please see the following resources for more information:

MaPPeRTrac, https://github.com/LLNL/mappertrac;

FSL, https://fsl.fmrib.ox.ac.uk/fsl/fslwiki/FSL;

FreeSurfer, https://surfer.nmr.mgh.harvard.edu/;

MRTrix3, https://www.mrtrix.org/;

Singularity, https://docs.sylabs.io/guides/3.5/user-guide/introduction.html;

HCP Database, https://db.humanconnectome.org/app/template/Login.vm.

A collaboration between the U.S. Department of Energy and TRACK-TBI (Transforming Research and Clinical Knowledge in Traumatic Brain Injury).

See Acknowledgements for the additional TRACK-TBI Investigators contributed to this work.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file1 (DOCX 69 KB)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Cai, L.T., Moon, J., Camacho, P.B. et al. MaPPeRTrac: A Massively Parallel, Portable, and Reproducible Tractography Pipeline. Neuroinform 22, 177–191 (2024). https://doi.org/10.1007/s12021-024-09650-0

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12021-024-09650-0

Keywords

Search

Navigation