Nonlinear Alignment of Whole Tractograms with the Linear Assignment Problem
- 254 Downloads
Abstract
After registration of the imaging data of two brains, homologous anatomical structures are expected to overlap better than before registration. Diffusion magnetic resonance imaging (dMRI) techniques and tractography techniques provide a representation of the anatomical connections in the white matter, as hundreds of thousands of streamlines, forming the tractogram. The literature on methods for aligning tractograms is in active development and provides methods that operate either from voxel information, e.g. fractional anisotropy, orientation distribution function, T1-weighted MRI, or directly from streamline information. In this work, we align streamlines using the linear assignment problem (LAP) and propose a method to reduce the high computational cost of aligning whole brain tractograms. As further contribution, we present a comparison among some of the freely-available linear and nonlinear tractogram alignment methods, where we show that our LAP-based method outperforms all others. In discussing the results, we show that a main limitation of all streamline-based nonlinear registration methods is the computational cost and that addressing such problem may lead to further improvement in the quality of registration.
1 Introduction
Image registration is a cornerstone of brain imaging applications. After registration of the data of two brains, homologous anatomical structures are expected to overlap better than before registration. Typically, brain images are registered to atlases, or between different subject, for different purposes, such as: conducting group-studies, guiding segmentation, or for building new atlases [9].
Diffusion magnetic resonance imaging (dMRI) techniques measure the orientation of diffusion of water molecules in biological tissues. By means of tractography techniques on dMRI data from the brain, it is possible to obtain a representation of the anatomical connections in the white matter, as hundreds of thousands of streamlines, forming the tractogram.
The linear or nonlinear transformation to register the tractogram of one subjects to that of another subject can be estimated in two main ways: from volumetric data, such as T1-weighted, fractional anisotropy (FA), or orientation distribution functions (ODFs), see [3, 4, 11], or directly from streamlines. In this last case, the literature on nonlinear methods addresses the registration of bundles, which are just a portion of the tractogram, usually because of the very high computational cost of the algorithms, see [5, 17, 18, 20, 21]. Differently, in the case of linear methods, solutions to compute an affine transformation from whole tractograms are available, see [7, 9].
Recently, a new concept has been proposed to accurately align tractograms, based on the idea of streamline correspondence [10] and formalized as a graph matching (GM) problem. There, the building block of the methodology is to compute which streamline of the first subject corresponds to which streamline of the second subject, as in a combinatorial optimization problem. The principle of streamline correspondence has also been used for the problem of bundle segmentation [6, 8, 12, 13, 19].
In this work, we present two contributions: first, we propose the use of the linear assignment problem (LAP) as computational building block to align entire brain tractograms from streamline information, extending the work of [12], which addressed only bundles and segmentation. The alignment obtained with LAP acts locally, as in a nonlinear transformation. Second, we quantitatively compare the proposed method against some methods in the state of the art for tractogram alignment, for which the implementation is freely-available. To the best of our knowledge, it is pretty infrequent to find quantitative comparisons between tractogram alignment methods, in the literature.
In order to carry out the quantitative comparison, we designed an experiment where, given the tractograms of two subjects, we computed the transformation of the first in order to match the second one. Similarly to [10], we quantified the accuracy of the whole tractogram alignment by estimating the degree of overlap between homologous bundles. The reason behind it is that, after registration, the anatomy of the white matter of the two subjects should match more accurately than before. On 90 pairs of subjects from the dMRI dataset of the Human Connectom Project [14], we compared the registration obtained with 5 different methods, using 10 main bundles as landmarks for the quantification.
The results show that the proposed method outperforms voxel-based methods and streamline-based methods in almost all cases. Moreover, as expected, linear methods are outperformed by nonlinear methods. The Symmetric Normalization (SyN) algorithm of the Advanced Normalization Tools (ANTs, see [1]), despite being designed for volumetric images, shows excellent results against streamline based methods, outperforming them in some cases. The main limitation of streamline-based methods is the high computational cost, which requires to resort to approximations.
In the following, we briefly describe the proposed method. Then, in Sect. 3, we describe the details of the experiments. In Sect. 4 we discuss the claims in the light of the results.
2 Methods
Let \(s = \{\mathbf {x}_1, \dots , \mathbf {x}_n\}\) be a streamline, i.e. a sequence of points in 3D space, i.e. \(\mathbf {x}_i = [x_i, y_i, z_i] \in \mathbb {R}^3\), \(\forall i\). Let \(T = \{s_1, \dots , s_N\}\) be a tractogram and \(b \subset T\) the set of streamlines corresponding to a white matter bundle of interest, e.g. the cortico-spinal tract (cst). With \(|\cdot |\) we indicate the number of objects in a set. Typically, |T| is in the order of \(10^5\)–\(10^6\). Several distances has been defined between streamlines, see [9] In this work, we adopt the commonly adopted mean of closest distances, \(d(s_a, s_b) = \frac{d_m(s_a, s_b) + d_m(s_b, s_a)}{2}\), where \(d_m(s_a, s_b) = \frac{1}{|s_a|} \sum _{\mathbf {x}_i \in s_a} \min _{\mathbf {x}_j \in s_b} ||\mathbf {x}_i - \mathbf {x}_j||_2\).
2.1 Streamline Correspondence
Given two tractograms, \(T_A\) and \(T_B\), the problem of aligning them can be framed as finding the correspondences between each streamline in \(T_A\) the corresponding one in \(T_B\), see [10]. If \(s^A_i \in T_A\) corresponds to \(s^B_j \in T_B\), the transformation obtained from such correspondence is the one that returns \(s^B_j\) when given \(s^A_i\). The set of correspondences will provide a good alignment of tractograms if, after the transformation, homologous anatomical structures will match. Previously, the task of finding a good set of correspondences has been formulated as graph matching [10].
2.2 Linear Assignment Problem
2.3 Large-Scale Approximation
The correspondence between entire tractograms cannot be computed even with LAPJV, because of the excessive computational cost. Here, we adopt a hierarchical two-steps procedure that exploits the geometrical structure of tractograms. In the first step, both \(T_A\) and \(T_B\) are clustered into k clusters, named \(\{\alpha _i\}_{i=1 \dots k}\) for \(T_A\) and \(\{\beta _j\}_{j=1 \dots k}\) for \(T_B\). Then, each tractogram is simplified with k streamlines, i.e. the centroids of the respective clusters. Then, LAP is computed between the two simplified tractograms, as explained above, i.e. a \(k \times k\) LAP is solved. This first step aims at finding corresponding clusters across the two tractograms, e.g. \(\alpha _i\) corresponds to \(\beta _j\). In the second step, given two corresponding clusters, the correspondence of streamlines is computed by solving the RLAP between the streamlines of the two clusters^{1}. Details are given below. In total, the two-steps procedure requires to solve 1 LAP with a \(k \times k\) cost matrix and k RLAPs each with, approximately, a \(\frac{N}{k} \times \frac{N}{k}\) cost matrix. For this reason, the resulting time complexity is reduced from \(\mathcal {O}(N^3)\) to \(\mathcal {O}\left( k^3 + \frac{N^3}{k^2}\right) \) and space complexity from \(\mathcal {O}(N^2)\) to \(\mathcal {O}(k^2 + \frac{N^2}{k})\).
In the second step of the procedure, assuming cluster \(\alpha _i \subset T_A\) to correspond to cluster \(\beta _j \subset T_B\), there are two possible scenarios: either (i) \(|\alpha _i| \le |\beta _j|\), for which finding the corresponding streamlines of \(\alpha _i\) in \(\beta _j\) is a straightforward RLAP, or (ii) \(|\alpha _i| > |\beta _j|\), for which there are not enough streamlines in \(\beta _j\) to set up a LAP or RLAP. In this last case, we propose to compute the corresponding streamlines of \(\alpha _i\) by violating the one-to-one constraint and assigning one or streamlines of \(\alpha _i\) to each of \(\beta _j\). The procedure is the following: first we solve the reverse RLAP, i.e. we compute the optimal assignment of all the streamlines in \(\beta _j\) to (some of) those in \(\alpha _i\). In this way, a subset of the streamlines in \(\alpha _i\) will obtain their corresponding ones in \(\beta _j\). We denote such subset of assigned streamlines as \(\alpha _i^a \subset \alpha _i\) and that of the remaining ones, i.e. the non assigned streamlines, as \(\alpha _i^{na} \subset \alpha _i\). Then, for each non assigned streamline \(s \in \alpha _i^{na}\), we compute its nearest neighbor in \(\alpha _i^a\). Finally, we define the corresponding streamline of \(s \in \alpha _i^{na}\) as the one corresponding to its nearest neighbor in \(\alpha _i^{a}\).
3 Experiments
We selected 30 healthy subjects at random from the publicly available Human Connectome Project (HCP) dMRI dataset [14] (90 gradients; b = 2000; voxel size = 1.25 mm isotropic). For each subject, tractograms of 400–500 thousands streamlines were obtained using the constrained spherical deconvolution (CSD) algorithm [15] and the local deterministic tracking algorithm implemented in DiPy^{2} (step size = 0.625 mm, 1 seed/voxel from the white matter). We segmented 14 major bundles from each tractogram, using the TractQuerier/white matter query language (WMQL, see [16]) and we used some of them as ground truth. In order to reduce the impact of poor segmentations obtained in some cases, we jointly selected 10 subjects and 10 bundles in order to minimize the differences in number of streamlines for each bundles across the subjects. These bundles are (both left and right): cingulum (cb), cortico-spinal tract (cst), inferior fronto-occipital fasciculus (ifof), thalamo prefrontal (thpref) and uncinate fasciculus (uf). We visually inspected the resulting bundles to avoid outliers. The experiments were then conducted on the 90 pairs of different tractograms that can be obtained from the selected 10 subjects.
3.1 Comparison
We quantified the quality of alignment between two tractograms as the degree of overlap between the voxel masks of homologous bundles, after registration, see [7, 10]. The degree of overlap was quantified as dice similarity coefficient (DSC): \(DSC = \frac{ 2 \times (|v(\hat{b}_A) \cap v(b_B)|)}{|v(\hat{b}_A)| + |v(b_B)|}\), where \(v(\hat{b}_A)\) is the voxel mask of the bundle \(b_A \in T_A\) after the alignment of the entire tractogram \(T_A\) to \(T_B\). In other words, \(v(\hat{b}_A)\) attempts to approximate the voxel mask of the homologous bundle of the target subject, \(v(b_B)\), considered as ground truth. In the comparison, we considered the following methods to align tractograms: 1) registration based on anterior and posterior commissures (AC-PC), directly provided within the HCP dataset, used as baseline. 2) Streamline linear registration (SLR, from DiPy, see [7]): in [10], SLR has shown slightly superior quality of registration with respect to other linear methods, so we considered it as a good representative of the linear methods. 3) The voxel-based nonlinear registration method of ANTs^{3}, see [1], used with default values. As reference volume, we considered the T1w images of the two subjects and the fractional anisotropy (FA) volumes. 4) Deformetrica^{4} [5], a diffeomorphic streamline-based registration method for bundles. Streamlines were modeled as varifolds and we used 7 mm and 15 mm for the varifolds and diffeomorphic kernel bandwidths respectively. 5) Correspondence between streamlines as graph matching (GM^{5}, see [10]). 6) Correspondence between streamlines as linear assignment problem (LAP) (Sect. 2). Some of the methods have too high computational cost when computed on whole tracotgrams. For this reason, tractograms were simplified for such methods following the simplification step described in Sect. 2.3, using the fast mini-batch k-means algorithm, as described in [10]. For each tractogram, we computed the approximate k-means clustering on streamlines, with \(k=1000\) and \(k=5000\). The simplified tractogram consisted of the k centroid streamlines. Note that such values of k ensure an extensive coverage of the brain, which we can assume to be enough to guarantee reasonable one-to-one assignments. We provide code and datasets of all experiments under a Free/OpenSource license here: https://github.com/FBK-NILab/WBIR2020_experiments.
3.2 Results
For each of the 10 bundles considered in this study (one per column), the table reports the degree of voxel-overlap after whole brain tractogram alignment, quantified as DSC (higher is better) and averaged over 90 pairs of subjects, across different methods - one per row. In each cell, the standard deviation of the mean is always below 0.01. In bold face are reported the highest values for each bundle, as well as those within 0.01 from them. The last column reports the computational time in minutes for aligning a pair of tractograms.
cbL | cbR | cstL | cstR | ifofL | ifofR | thprefL | thprefR | ufL | ufR | Time | |
---|---|---|---|---|---|---|---|---|---|---|---|
AC-PC | 0.42 | 0.38 | 0.41 | 0.43 | 0.32 | 0.29 | 0.38 | 0.38 | 0.21 | 0.19 | – |
SLR | 0.47 | 0.43 | 0.46 | 0.50 | 0.38 | 0.37 | 0.44 | 0.45 | 0.27 | 0.25 | 5 |
ANTs (T1w) | 0.61 | 0.56 | 0.56 | 0.59 | 0.54 | 0.51 | 0.54 | 0.55 | 0.37 | 0.35 | 30 |
ANTs (FA) | 0.62 | 0.57 | 0.56 | 0.61 | 0.57 | 0.53 | 0.55 | 0.56 | 0.38 | 0.36 | 30 |
Deform (k = 1000) | 0.46 | 0.42 | 0.44 | 0.48 | 0.41 | 0.38 | 0.44 | 0.44 | 0.28 | 0.27 | 170 |
GM (k = 1000) | 0.46 | 0.45 | 0.57 | 0.59 | 0.62 | 0.64 | 0.56 | 0.57 | 0.39 | 0.35 | 480 |
LAP (k = 1000) | 0.52 | 0.48 | 0.60 | 0.63 | 0.63 | 0.64 | 0.56 | 0.57 | 0.40 | 0.34 | 60 |
LAP (k = 5000) | 0.56 | 0.52 | 0.63 | 0.66 | 0.66 | 0.67 | 0.60 | 0.60 | 0.43 | 0.39 | 120 |
4 Discussion and Conclusions
In this work, we describe the use of the linear assignment problem (LAP) to align entire tractograms of two different subjects, by introducing approximations as computational shortcuts. The LAP acts locally, as a nonlinear registration method. In Table 1, we compare the proposed method with some linear and nonlinear methods in the literature. The results show that LAP (\(k=5000\)) outperforms other methods on almost all bundles. The only exception is the cingulum (cbL, cbR) for which ANTs provides significantly better DSC, with LAP second in the ranking.
The results also show a number of other interesting facts and confirm basic sanity checks: despite the limited quality of the ground truth provided by the WMQL, the values of DSC increase steadily from AC-PC registration, to linear registration and to nonlinear methods. 90 pairs of subjects selected as described in Sect. 3 are enough to keep the standard deviation of the means low enough (\(\le \)0.01) to clearly see differences between the methods. It is also reassuring that, for each bundle, the results are sufficiently similar across the two hemispheres, e.g. ifofL and ifofR obtain almost the same score, for all methods. Nonlinear methods outperform linear methods in all cases, with the exception of Deformetrica, most probably because we could not perform a large model selection for the user-selected parameters due to the high computational time. Furthermore, the introduced approximations might interfere too much with the underlying method, which is tailored to register bundles. ANTs provided excellent results, given the fact that it does not operate on streamline information. Most probably this occurs because the grid on which ANTs operates, i.e. the voxel grid, is much more dense than what the simplified tractograms offer. The results between graph matching (GM) and LAP are not very different, for the same level of approximation (\(k=1000\)), with LAP in advantage. This advantage can be explained by the fact that LAPJV computes the exact solution of the RLAPs, while in the case of GM the underlying algorithm, DSPFP (see [10]), provides only an approximate solution. Notably, LAP is 8 times faster than GM for this size of tractograms, see Table 1 last column, which allowed us to run LAP with \(k=5000\) in a reasonable amount of time and to obtain substantially superior scores.
All this evidence supports the hypothesis that the results are limited also by the level of approximation and that, by improving algorithms and implementations to reduce computational cost, some of the methods may reach even better results.
Footnotes
References
- 1.Avants, B.B., Epstein, C.L., Grossman, M., Gee, J.C.: Symmetric diffeomorphic image registration with cross-correlation: evaluating automated labeling of elderly and neurodegenerative brain. Med. Image Anal. 12(1), 26–41 (2008). https://doi.org/10.1016/j.media.2007.06.004CrossRefGoogle Scholar
- 2.Bijsterbosch, J., Volgenant, A.: Solving the rectangular assignment problem and applications. Ann. Oper. Res. 181, 443–462 (2010). https://doi.org/10.1007/s10479-010-0757-3MathSciNetCrossRefzbMATHGoogle Scholar
- 3.Christiaens, D., Dhollander, T., Maes, F., Sunaert, S., Suetens, P.: The effect of reorientation of the fibre orientation distribution on fibre tracking. In: MICCAI 2012 Workshop on Computational Diffusion MRI, CDMRI 2012, pp. 33–44 (2012)Google Scholar
- 4.Du, J., Goh, A., Qiu, A.: Diffeomorphic metric mapping of high angular resolu-tion diffusion imaging based on Riemannian structure of orientation distribution functions. IEEE Trans. Med. Imaging 31(5), 1021–1033 (2012)CrossRefGoogle Scholar
- 5.Durrleman, S., Fillard, P., Pennec, X., Trouvé, A., Ayache, N.: Registration, atlas estimation and variability analysis of white matter fiber bundles modeled as currents. NeuroImage 55(3), 1073–1090 (2011). https://doi.org/10.1016/j.neuroimage.2010.11.056CrossRefGoogle Scholar
- 6.Feydy, J., Roussillon, P., Trouvé, A., Gori, P.: Fast and scalable optimal transport for Brain tractograms. In: Shen, D., et al. (eds.) MICCAI 2019. LNCS, vol. 11766, pp. 636–644. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-32248-9_71CrossRefGoogle Scholar
- 7.Garyfallidis, E., Ocegueda, O., Wassermann, D., Descoteaux, M.: Robust and efficient linear registration of white-matter fascicles in the space of streamlines. NeuroImage 117, 124–140 (2015). https://doi.org/10.1016/j.neuroimage.2015.05.016CrossRefGoogle Scholar
- 8.Labra, N., et al.: Fast automatic segmentation of white matter streamlines based on a multi-subject bundle atlas. Neuroinformatics 15(1), 71–86 (2016). https://doi.org/10.1007/s12021-016-9316-7CrossRefGoogle Scholar
- 9.O’Donnell, L.J., Wells, W.M., Golby, A.J., Westin, C.-F.: Unbiased groupwise registration of white matter tractography. In: Ayache, N., Delingette, H., Golland, P., Mori, K. (eds.) MICCAI 2012. LNCS, vol. 7512, pp. 123–130. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-33454-2_16CrossRefGoogle Scholar
- 10.Olivetti, E., Sharmin, N., Avesani, P.: Alignment of tractograms as graph matching. Front. Neurosci. 10, 554 (2016)CrossRefGoogle Scholar
- 11.Raffelt, D., Tournier, J.-D., Fripp, J., Crozier, S., Connelly, A., Salvado, O.: Symmetric diffeomorphic registration of fibre orientation distributions. NeuroImage 56, 1171–1180 (2011). https://doi.org/10.1016/j.neuroimage.2011.02.014CrossRefGoogle Scholar
- 12.Sharmin, N., Olivetti, E., Avesani, P.: Alignment of tractograms as linear assignment problem. In: Fuster, A., Ghosh, A., Kaden, E., Rathi, Y., Reisert, M. (eds.) Computational Diffusion MRI. MV, pp. 109–120. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-28588-7_10CrossRefGoogle Scholar
- 13.Sharmin, N., Olivetti, E., Avesani, P.: White matter tract segmentation as multiple linear assignment problems. Front. Neurosci. 11, 754 (2018). https://doi.org/10.3389/fnins.2017.00754CrossRefGoogle Scholar
- 14.Sotiropoulos, S.N., et al.: WU-Minn HCP consortium: advances in diffusion MRI acquisition and processing in the human connectome project. NeuroImage 80, 125–143 (2013)CrossRefGoogle Scholar
- 15.Tournier, J.D., Calamante, F., Connelly, A.: Robust determination of the fibre orientation distribution in diffusion MRI: non-negativity constrained super-resolved spherical deconvolution. NeuroImage 35(4), 1459–1472 (2007). https://doi.org/10.1016/j.neuroimage.2007.02.016CrossRefGoogle Scholar
- 16.Wassermann, D., et al.: On describing human white matter anatomy: the white matter query language. In: Mori, K., Sakuma, I., Sato, Y., Barillot, C., Navab, N. (eds.) MICCAI 2013. LNCS, vol. 8149, pp. 647–654. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-40811-3_81CrossRefGoogle Scholar
- 17.Wassermann, D., et al.: White matter bundle registration and population analysis based on Gaussian processes. In: Székely, G., Hahn, H.K. (eds.) IPMI 2011. LNCS, vol. 6801, pp. 320–332. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-22092-0_27CrossRefGoogle Scholar
- 18.Waugh, J.L., et al.: A registration method for improving quantitative assessment in probabilistic diffusion tractography. NeuroImage 189, 288–306 (2019). https://doi.org/10.1016/j.neuroimage.2018.12.057CrossRefGoogle Scholar
- 19.Yoo, S.W., et al.: An example-based multi-atlas approach to automatic labeling of white matter tracts. PloS One 10(7), e0133337 (2015). https://doi.org/10.1371/journal.pone.0133337CrossRefGoogle Scholar
- 20.Ziyan, U., Sabuncu, M., Grimson, W., Westin, C.F.: Consistency clustering: a robust algorithm for group-wise registration, segmentation and automatic atlas construction in diffusion MRI. Int. J. Comput. Vis. IJCV 85(3), 279–290 (2009). https://doi.org/10.1007/s11263-009-0217-1CrossRefGoogle Scholar
- 21.Ziyan, U., Sabuncu, M.R., O’Donnell, L.J., Westin, C.-F.: Nonlinear registration of diffusion MR images based on fiber bundles. In: Ayache, N., Ourselin, S., Maeder, A. (eds.) MICCAI 2007. LNCS, vol. 4791, pp. 351–358. Springer, Heidelberg (2007). https://doi.org/10.1007/978-3-540-75757-3_43CrossRefGoogle Scholar