Abstract
This paper explicates a solution to building correspondences between molecularscale transcriptomics and tissuescale atlases. This problem arises in atlas construction and crossspecimen/technology alignment where specimens per emerging technology remain sparse and conventional image representations cannot efficiently model the high dimensions from subcellular detection of thousands of genes. We address these challenges by representing spatial transcriptomics data as generalized functions encoding position and highdimensional feature (gene, cell type) identity. We map onto lowdimensional atlas ontologies by modeling regions as homogeneous random fields with unknown transcriptomic feature distribution. We solve simultaneously for the minimizing geodesic diffeomorphism of coordinates through LDDMM and for these latent feature densities. We map tissuescale mouse brain atlases to genebased and cellbased transcriptomics data from MERFISH and BARseq technologies and to histopathology and crossspecies atlases to illustrate integration of diverse molecular and cellular datasets into a single coordinate system as a means of comparison and further atlas construction.
Similar content being viewed by others
Introduction
Since the 17th century, scientists have seen living organisms as a hierarchy of biological mechanisms at work across scales. To understand the interplay of these mechanisms, reference atlases that incorporate genetic, cellular, and connectivity measures into a single coordinate space have been constructed and which aim to summarize the mass of data across scales through a set of discrete partitions. An instance of the more general segmentation problem in computer vision, atlas construction relies on the underlying assumption of homogeneity within each region. The optimal partitioning assigns a label to each region based on this homogeneity and the presence of sharp changes at the boundaries between regions. In biology, this label frequently reflects behavior or function, as seen in two of the most common mouse brain atlases: the Allen Common Coordinate Framework (CCFv3)^{1} and the Franklin and Paxinos Atlas^{2}. Together, the common coordinate framework an atlas provides in addition to its ontology facilitates the comparison of different types or replicates of data in a single coordinate system and hones efforts of study to particular regions relevant to each unique investigation.
Hence, atlas construction and segmentation of new data with defined atlases comprise two classes of widespread problems across domains of biology. These are particularly relevant in the context of emerging datasets in spatial transcriptomics, where atlases have still yet to be constructed^{3} and where the different technologies used number quite high, thus emphasizing the need for mapping data across samples and technologies into a common coordinate framework for comparison. A central challenge in both atlas construction and segmentation, however, is in aligning in a single coordinate space data that often exists at different scales (e.g. coarse scale atlases to finescale datasets) and is fundamentally of different functional modalities (e.g. tissue regions to molecular images). These differences preclude an obvious definition of similarity for consequently optimizing alignment.
In the classical image setting, a large family of diffeomorphismbased methods^{4} has been developed within the field of Computational Anatomy (CA)^{5,6} for transforming coordinate systems at the tissue scales. The smoothness, invertibility, and nonrigid nature of diffeomorphisms are particularly relevant in medical image alignment as they reflect the mechanics/dynamics of soft tissue^{7}. Consequently, methods rooted in diffeomorphic registration come particularly from multiple labs in the magnetic resonance imaging (MRI) community, with variations developed in the contexts of both atlas construction and segmentation at these tissue scales^{7,8,9,10,11,12,13,14,15,16}. Large Deformation Diffeomorphic Metric Mapping (LDDMM)^{17}, on which many such methods are built, specifically equips diffeomorphisms with a metric that allows them to be reduced to a lowdimensional representation that conveniently identifies shape. Consequently, it has been harnessed in the setting of image classification and the study of atrophy in diseased versus control populations^{17,18,19}. While successful at aligning images of the same modality, most of these diffeomorphismbased methods focus exclusively on generating geometric transformations (e.g. diffeomorphisms). The challenge of crossing functional modalities has instead been addressed in a classical imaging setting by coupling such methods to variations in matching cost or additional transformations. These include matching based on analytical methods using crosscorrelation^{13} or localized texture features^{20}, and methods for transforming one range space to another in crossing modalities and scales based on polynomial transformations^{21}, scattering transforms^{22}, graphical models^{23}, and neural networks^{24,25,26}.
In the molecular setting, however, both the diversity and magnitude of data measured by spatial transcriptomics technologies^{27} often prohibit the representation of such data as classical continuous images discretized as regular grids, and consequently, the direct use of these image alignment methods at the molecular scales. As seen in repositories generated in the BRAIN Initiative Cell Census Network (BICCN) and archived at the Brain Image Library (BIL), these datasets are already on the order of terabytes and will only continue to increase as technologies shift from mouse to human measurements. Hence, while some methods in deep learning have been applied to align singlecell datasets, modeled as regular grid images, both to atlases at the histological scale^{28} as well as reference transcriptional atlases that are beginning to emerge^{29}, most imagebased tactics are limited in their ability to represent this highdimensional and memoryintensive data. Furthermore, many learningbased schemes in the context of classical images have relied on the use of extensive numbers (typically on the order of hundreds) of images from different subjects for extracting features useful for crossmodalities and mapping^{18,26}. In the setting of spatial transcriptomics technologies that have been developed only over the last few years, the acquisition of such large training datasets is typically prohibitive for some of these learningbased approaches.
Consequently, an independent class of methods has been developing for aligning spatially resolved transcriptional datasets at these molecular scales. With influence from imagebased methods, some of these including GPSA^{3} and PASTE^{30} focus exclusively on spotbased data in which gene expression is measured in a neighborhood of each spot for a regular array of spots, analogous to a voxel grid in an image. In contrast, imagebased technologies such as STARmap, BARseq, and MERFISH, which take point measures of individual mRNA molecules or cells, cover measurements irregularly sampled over space according to tissue architecture and dynamics. Furthermore, natural fluctuation in gene expression over time and space coupled to the dynamics of each spatial transcriptomics technology leads each tissue section, at the molecular/cellular (0.3–100 micron) scales, to have a varying number of such particles with no natural ordering of particles consistently apparent between sections. Consequently, landmarkbased methods^{31} that assume direct permutation correspondence between particles are not applicable. Methods aimed at generalizing to allow alignment within and across these additional technologies are typically rooted in different data representations, such as graphs, in the case of SLAT^{32} that aims to find correspondence between cells or groups of cells in an atlas and target or CAST^{33}, a deep graphbased neural network (GNN) algorithm which learns a graph representation of singlecell resolution omics data and subsequently aims to align datasets via these graph representations. Additionally, while many methods have showcased success at aligning different replicates within a single technology for both human and mouse samples, crosstechnology alignment has typically relied on a nonempty intersection of feature sets across these technologies^{28,32,33}, limiting extension of these methods to the type of crossmodality and crossscale mapping required for integrating tissuescale atlases and molecular scale data.
Here, we build on both of these lineages in presenting a model equally equipped at representing tissuescale atlases and molecularscale data, and an associated crossmodality registration mechanism that addresses both challenges of crossing scales and functional modalities by estimating simultaneously geometric and functional transformations to align each dataset to the other. Specifically, we harness the generalizability of image varifolds, which have emerged in molecular CA^{34}, for simultaneously modeling molecular and tissuescale data with both irregular and gridlike sampling schemes in a common framework (addressed in SLAT and CAST, for instance, by a graphbased representation in the setting of molecular and cellular data^{32,33}). A subproblem covered by the imagevarifold theory outlined in ref. ^{36} is the mapping of molecular scale data to atlas coordinate systems. As specified there, we estimate minimal energy diffeomorphic transformations as in imagebased LDDMM^{17}, but where the action of diffeomorphisms on images has been adapted to the setting, here, of image varifolds in a consistent manner^{34}. Additionally, we estimate in tandem a latent distribution over the molecular functional space for each atlas partition, without the need for the large datasets often used in deep learning approaches, but by instead relying on an assumption of spatial homogeneity in distribution across each atlas partition. We use this latent distribution to transform the functional space of each atlas (i.e., its ontology) to the molecular functional space. We refer to this method of jointly estimating diffeomorphisms and latent feature distributions, as Cross Imagevarifold LDDMM (xIVLDDMM), where the cross emphasizes its ability to map across scales and modalities^{35}. We demonstrate this methodology with two common implementations of image varifolds (triangulated meshes and point clouds). First, we introduce the methodology through a demonstration of mapping both genebased and cellbased MERFISH datasets to corresponding sections of the Allen Common Coordinate Framework (CCFv3)^{1}. Second, we quantify accuracy in mapping specifically characterized cell types in BARseq data^{36} to the CCFv3. Finally, we illustrate the method’s generalizability to additional modalities and offer diverse examples of its use across the spans of biology in atlas construction, crossreplicate and crossspecies comparison, and data segmentation.
Results
Data model and optimization problem: image varifolds and transformations for molecular scales based on varifold norms
To accommodate the high feature dimensionality and spatial irregularity of molecular datasets, as described in the Introduction, we harness the recent work that extends the theory of diffeomorphisms and imagebased LDDMM^{17} to the setting of image varifolds and estimation of correspondences between them^{34,35}. We first describe this framework of image varifolds to emphasize their capacity for modeling diverse types of tissue, cellular, and molecularscale data. Image varifolds are geometric measures over the product space, \({{\mathbb{R}}}^{2}\times {{{{{{{\mathcal{F}}}}}}}}\), therefore encompassing measures over both physical and feature spaces, \({{\mathbb{R}}}^{2}\) and \({{{{{{{\mathcal{F}}}}}}}}\), respectively. At the finest scale of capture in both classical imaging and imagebased spatial transcriptomics technologies, we might model a set of pixels or detections (e.g. mRNA reads) as a discrete set of point measures (particles) with an elementary Dirac measure, centered over physical location and feature value for each detection. At coarser (e.g. cellular) scales or in spotbased technologies, however, point measurements taken at single physical locations may capture a range of individual detections. To generalize to the range of technologies, we model both tissuescale atlases and molecular data with semidiscrete image varifolds where the physical arrangement of measurements is captured by a collection of discrete point (Dirac) measures, but where each such point measure is associated with a full distribution over feature values measured in its neighborhood. This collection of point measures is indexed by i ∈ I, with discrete measures, \({\delta }_{{x}_{i}}\), evaluating to 1 at locations \({x}_{i}\in {{\mathbb{R}}}^{2}\), and distributions over feature values modeled as weighted probability distributions: w_{i}p_{i}. Weight, w_{i}, is representative of total mass (e.g. total mRNA reads or total cells) measured at location x_{i}, thus enabling an estimate of density over physical space, and probability distribution, p_{i}, captures the proportions of each feature type within that mass.
The measure of the complete collection is denoted by the sum:
Feature spaces considered here are finite, yielding probability distributions over 10s–100s of different elements depending on the application:

MERFISH gene sections: w_{i} is the total mRNA at location x_{i}, and p_{i} is the probability distribution on gene (~700).

BARseq cell sections: w_{i} is the total cells at location x_{i} and p_{i} is the probability distribution on cell type (~30).

CCFv3 tissue sections: w_{i} = 1 for location x_{i} in foreground tissue and p_{i} is the probability distribution on ontology label (~700).
Within this established framework, we align image varifolds capturing tissuescale atlases to those capturing molecularscale data via the estimation of two types of correspondence: one between physical coordinates (φ) and the other between feature spaces (π). These correspondences act independently and in parallel on the physical and feature measure components of the imagevarifold object. Consequently, while they are both applied in the setting of optimization to evaluate the alignment of the atlas to target (top panel, Fig. 1) they can be applied individually as relevant to specific applications including data segmentation, atlas construction, and crossspecimen comparison, as depicted in the middle panel in Fig. 1.
Regarding physical correspondence, in Computational Anatomy (CA), correspondence between tissue sections is computed using coordinate transformation between the sections by solving an optimization problem characterized by the set of possible transformations to optimize the image similarity function that specifies the alignment of the sections. These transformations are modeled as affine motions and diffeomorphisms, φ, which act to generate the space of all configurations. We model physical transformations in the setting of image varifolds similarly to diffeomorphisms, optimally solving for them with LDDMM^{17}. However, while for classical images such as for MRI, LDDMM^{17} uses the action of diffeomorphisms on images I as classical functions using function composition on the right with the inverse of the diffeomorphism: φ ⋅ I(x) = I∘φ^{−1}(x) for x ∈ R^{d}, the action of diffeomorphisms on image varifolds is defined, as in^{34,35}:
for the form of the image varifold defined in (1). The determinant of the Jacobian, ∣Dφ∣, capturing the local expansion/contraction of physical space, is introduced to retain the given spatial density of the original object following physical transformation. Notably, the estimated diffeomorphism can be applied in both the forward and inverse directions, taking tissuescale atlas to molecular coordinates or vice versa, as shown in the middle panel in Fig. 1. The bottom panel depicts the determinant of the Jacobian of the diffeomorphism taking CCFv3 section to MERFISH section (B,G) with areas of local expansion in red and local contraction in blue.
To define a similarity metric between the deformed atlas and the target, we also need to carry the feature component of tissuescale atlases to the feature space of the molecular target. For this, we associate to each feature value in the atlas ontology (\(\ell \in {{{{{{{\mathcal{L}}}}}}}}\)), a distribution (denoted π_{ℓ}) over target feature values, capturing the assumption of spatial homogeneity we make within each atlas region. Importantly, these distributions are not normalized, enabling the generation of both a measure of target mass (e.g. mRNA or cells) over physical space (given by \({w}_{i}^{{\prime} }\)) and conditional probability distribution (\({p}_{i}^{{\prime} }\)) over target features (e.g. gene or cell type). Both molecular mass and conditional probability distribution are associated with each point measured in the atlas through the mixture distribution:
with w_{i}, p_{i} denoting the initial weight and probability distribution of the point measure over the atlas feature space and with
The bottom panel in Fig. 1c, h exhibits the estimated mRNA density (\({w}_{i}^{{\prime} }\)) for each location in the corresponding CCFv3 section while the middle column of the middle panel summarizes the estimated probability distribution over genes (\({p}_{i}^{{\prime} }\)) with the gene with the highest probability denoted for each location.
Finally, the estimation and application of both geometric transformations and latent feature distributions, φ, π, to the atlas image varifold take it to both the physical and feature space of the target as necessary to evaluate the similarity function in the optimization scheme (top panel, Fig. 1). In the image setting, the similarity function used is often a norm on functions, and solving the problem of minimization of the norm in the space of diffeomorphisms gives the metric theory of LDDMM for generating geodesic matching between exemplar anatomies^{37,38}. Here, the similarity function is a norm on image varifolds (see “Molecular scale varifold norm” section), capturing proximity in both physical and feature spaces of the target. xIVLDDMM jointly estimates optimal φ and π (top panel, Fig. 1) to minimize this normed difference in the space of diffeomorphisms through either simultaneous or alternating optimization algorithms using LDDMM (see “Alternating LDDMM and quadratic program algorithm for joint optimization” section), with additional regularization imposed in both settings on the estimated π to ensure, for instance, positive values (see “Variational problems” section for explicit variational problems). Note that throughout we highlight mapping examples using both meshbased^{35} and particlebased (point cloud)^{34} implementations of image varifolds, with detailed construction, similarities, and differences of each covered in “Construction of imagevarifold representation for different modalities” section. Supplementary Fig. 1 shows equivalence in these two implementations and in the corresponding mappings estimated in each case for the CCFv3 and MERFISH sections shown in Fig. 1.
Generalizability of xIVLDDMM to settings of complete and incomplete genebased and cellbased data
Here, we emphasize the generalizability of xIVLDDMM for computing mappings from tissuescale atlases to datasets spanning a range of both genebased and cellbased features. The bottom panel of Fig. 1 specifically summarizes the estimated latent distribution over genes for each atlas region with the sum over gene counts, exhibiting the estimated density of mRNA over physical space. However, as described in “Data model and optimization problem: image varifolds and transformations for molecular scales based on varifold norms” section, a full weighted probability distribution over feature values (e.g. gene type) is estimated via π for each atlas location that not only contributes to achieving geometric alignment but can be visualized independently as reflective of mean distributions in target data. Figure 2a–c, for instance, shows the relative expression of three genes, (Gfap, Trp53i11, Wipf3, respectively) over space out of a total set of 20 in the target MERFISH section shown in Fig. 1. Figure 2e–g show the corresponding expression of each of these genes in each region of the CCFv3 section, as estimated through π, with notable similarity in probability magnitude and variation over space. Figure 2d, h showcase another comparative summary of the estimated feature distributions for the CCFv3 section versus MERFISH expression in depicting at each location the most probable gene type. The assumption of spatial homogeneity in distribution is evidenced here, particularly in large areas of the CCFv3, such as the striatum, where each gene carries a single probability of expression across the entire region and Kirrel3 is uniformly the most probable gene type.
The previous result solved the mapping problem between CCFv3 and MERFISH sections based on the mRNA reads directly. Often, these raw mRNA reads are segmented into discrete cells as a mode of data reduction followed by downstream analyses clustering the cells into discrete cell types. We emphasize that there are various methods for solving the segmentation to cells and thereby dimension reduction as determined by the specific imaging technology. Some of the methods are rooted in imagebased segmentation schemes such as the Watershed algorithm, operating jointly on transcriptional data and immunofluorescence images such as DAPI stains^{39}, while others utilize learningbased methods^{40} for accommodating often a wider diversity of cell shapes and sizes. In either case, the assignment of mRNA reads to specific cells introduces a layer of functional information at the micron scale, which can now be modeled in lieu of or in tandem with the functional information at the nanometer scale (e.g. raw mRNA reads) as the feature space of a target image varifold to which we wish to map sections of an atlas.
We demonstrate the efficacy of xIVLDDMM for mapping tissuescale atlases to cellularscale data in mapping CCFv3 section Z = 675 to a section of cellsegmented MERFISH transcriptional data (courtesy of the JEFworks Lab, Johns Hopkins University) (Fig. 3). The total gene set measured is ≈500 genes, with each transcript assigned to a single cell. Transcriptional profiles per cell are clustered into 33 distinct clusters using Leiden graphbased clustering^{41} and annotated as cell types based on known marker genes. This gives a cellbased dataset analogous to the transcriptbased dataset used in the above where we now capture the spatial density of cells, and conditional probability distributions over cell types, which can be summarized via the depiction of the cell type with the highest probability at each location in space (Fig. 3a). The latent cell type distribution per each CCFv3 region, estimated in tandem with the geometric transformation taking CCFv3 section to MERFISH spatial coordinates exhibits similarity to the MERFISH target in both total cell density and cell type with highest probability predicted per CCFv3 region (Fig. 3a). Alternatively, each MERFISH section can be characterized with a feature space over genes, and a corresponding latent distribution over genes can be estimated via xIVLDDMM in mapping the same CCFv3 section to MERFISH gene target (Fig. 3b, c). Here, we subsample the entire gene space by selecting a subset of six genes (Baiap2, Slc17a6, Adora2a, Gpr151, Gabbr2, Cckar) as those most spatially varying according to Moran’s I score^{42}. We combine both cell density and gene expression by aggregating the individual mRNA transcripts into an average gene expression feature per cell and normalizing the total mRNA per cell to 1. The target and estimated probabilities for each gene out of this subset of six are shown following the estimation of an alternative geometric transformation and latent distribution, now, over genes rather than cell types. Correspondence is seen in both the absolute magnitude as well as the relative probability of each gene between those in the target MERFISH section and those resulting from the estimated feature laws, π. Two examples include the area of high Baiap2 expression adjacent to the hippocampus (probability ≈ 0.95) and the area of high Slc17a6 expression in the rhomboid nucleus (probability ≈ 0.6), both depicted in Fig. 3b. This correspondence serves to reinforce the validity in the estimated feature laws in xIVLDDMM.
Notably, the assumed biological correlation between cell type and pattern of gene expression implies that signals of variation across cell types at the scale of microns should also exist across gene types at the scale of nanometers. Consequently, we might expect similar spatial deformation of a tissuescale atlas in mapping onto the same geometric target, but with conditional feature distributions defined over either gene or cell types, with partition boundaries deforming to match regions of homogeneity that would be roughly consistent across genes and cells. Supplementary Fig. 2 shows the diffeomorphism estimated in the two cases (6 genes and cell type) in Fig. 3 together with that for a third feature space comprised of a second set of 7 different genes selected for high spatial variance using mutual information. We observe the global similarity in spatial pattern and magnitude of contraction and expansion occurring but with nuanced differences in areas where these feature spaces likely emphasize different boundaries. Hence, the manifest stability in the geometric mappings jointly estimated with the feature laws over three different feature spaces supports the stability of our method in the face of different numbers and types of features but also speaks to the stability of the biological organization across tissue, cellular, and molecular scales.
Finally, we emphasize the generalizability of our mapping methodology across particularly imagebased spatial transcriptomics technologies by showcasing the alignment of a cellsegmented BARseq^{36,43} partial coronal section to a corresponding full coronal CCFv3 section (Supplementary Fig. 3). Note that cells were typed according to the same procedure used for the full coronal section highlighted in “Quantitative and comparative evaluation of xIVLDDMM” section. The costs involved with many of these emerging technologies coupled with their use in studying particular subregions or brain circuits of interest has caused the partial measurement of tissue areas to be incredibly prevalent across both imagebased and spotbased technologies^{44}. Consequently, this generates even greater variety in the scope and shape of tissue sections measured, only further emphasizing the need for aligning such partial captures to a common scaffold, such as an atlas coordinate system, where information can be merged across the intersection of these captures. The imagevarifold representation, particularly in its semidiscrete form as used here, is amenable to representing data with regions of missing or disrupted capture as an extension of the irregularity in sampling that is assumed in the general case. To estimate mappings in these cases, variation in both the forms and sizes of kernels governing the varifold norm (see “Molecular scale varifold norm” section) can control the granularity to which matches between atlas and target should be evaluated, with coarser scales more appropriate to noisier tissue sections with higher numbers of artifacts. Additionally, the varifold normed difference in the cost function, as presented in (10) in “Variational problems” section can be appended with spatially varying weights, as used in settings of mapping digital pathology to MRI^{22} to prioritize matching amongst certain intact regions over others. We utilize this strategy to align a partial BARseq section to a corresponding full CCFv3 section where initial geometric offset (Supplementary Fig. 3A) requires accurate estimation of scale, rigid, and diffeomorphic transformations to position the partial capture within the scope of the correct CCFv3 hemisphere (Supplementary Fig. 3B).
Quantitative and comparative evaluation of xIVLDDMM
We specifically evaluated the efficacy of xIVLDDMM in mapping CCFv3 sections to corresponding coronal sections of BARseq cellsegmented and subsequently celltyped data. As described in ref. ^{37}, cells were segmented in the BARseq data using Cellpose^{45}. Gene reads were assigned to cells, and cells containing fewer than 5 unique genes and 20 total gene counts were excluded. Cells were clustered using an iterative clustering approach based on Louvain clustering to achieve a similar resolution as a subclass in recent singlecell RNAseq studies^{46}.
Given the difference in atlas and target modalities mapped with xIVLDDMM, traditional measures of accuracy, such as Dice overlap score, as are often used to evaluate image registration tools cannot be directly applied. Furthermore, landmarks common to both atlas and target modalities are not typically readily available or necessarily identifiable given the diversity in measures taken by molecular modalities^{3}. Consequently, to evaluate the accuracy, particularly in the geometric alignment achieved with xIVLDDMM, we matched corresponding atlas feature values (regions) with target feature values (e.g. cell types) and quantified the resulting distance individual cells of a given type from the atlas region in which we expect to find them.
We quantified accuracy by computing the set distance between cells of types specific to hippocampal regions (e.g. CA1, CA2, and CA3 pyramidal cells and DG granule cells) and particles of these regions in the CCFv3. We specifically defined this distance for each cell, indexed by \(c\in {{{{{{{{\mathcal{C}}}}}}}}}_{R}\), for region R, as:
where location y_{c} was the given 2D coordinate of the cell in the native BARseq coordinate system, and the set \({{{{{{{{\mathcal{A}}}}}}}}}_{R}\) refers to the set of indexed locations in the atlas section with feature value (region label) of R (e.g. with p_{a}(R) = 1, and p_{a}(f) = 0 for f ≠ R). We compared these distances to those after deformation, by replacing y_{c} with the mapped position of each cell marker to the CCFv3 section: φ^{−1}(y_{c}), via the inverse diffeomorphism estimated in our joint imagevarifoldbased method. Figure 4 depicts these measures for two separate CCFv3 sections mapped to two corresponding sections of BARseq cell data. Initial positions (Fig. 4a, c) of CCFv3 regions versus cells give distances (Fig. 4e, g) on the order of 1 mm, where notably, neighboring particles in the CCFv3 are at 10 μm, thus giving a lower bound to distance metrics we might expect for cells to be neighboring CCFv3 particles. In contrast, the positioning of CCFv3 regions versus cells following geometric transformation reflects the effects of rotations, translations, scale, and diffeomorphisms effectively estimated to bring them into correspondence (Fig. 4b, d). Kernel size governing diffeomorphism regularization and varifold norm matching were both on the order of 100 μm. Median distances for all three regions in the first section are on the order of 20−30 μm (Fig. 4f) whereas those in the second section are on the order of 5 μm for CA1 and DG and 300 μm for CA3 (Fig. 4h), which we assume is coming from the separate group of cells labeled as CA3 at the edge of CA1 (indicated by the white star) in contrast to the group overlaying the CCFv3 region, which falls within the first 10 μm bin. Note that this discrepancy likely stems in part from the approximation of each BARseq section as a strictly coronal section of the CCFv3, where the cutting plane is instead slightly offset from this coronal plane. In any case, we observe in both sections an accuracy on the order of 10−30 μm, analogous in the image setting to cells being mapped to within 1–3 pixels (at a resolution of 10 μm) to the appropriate atlas region.
We also compared our joint imagevarifoldbased approach in xIVLDDMM to a classical imagebased approach matching, estimating only a geometric transformation and matching based on foregroundbackground (Fig. 5). We mapped one of the same CCFv3 sections (Z = 837) to an image rendering of the BARseq cell section, with grayscale intensity capturing density of cells over space. Both CCFv3 and BARseq images were discretized at 50 μm resolution to estimate the mapping, with the latter computed with Gaussian smoothing, and imagebased LDDMM, as described in ref. ^{17} was used to estimate optimal diffeomorphic transformation of the atlas to the target. To compare these methods more directly, a 10 μm BARseq image was transformed with the estimated diffeomorphism to the CCFv3 space (Fig. 5a) as closer to the particlebased representation at full resolution used for estimation of the diffeomorphism in the imagevarifold based mapping (Fig. 5c). At this resolution, ~1.5 million pixels are needed to model the BARseq data as an image whereas only ~90k particles are needed in the imagevarifold representation. In addition to the computational and memory expense of image handling, we compromise our ability to delineate one cell from another, as evidenced in the blur in the image rendering (Fig. 5a) versus imagevarifold representation (Fig. 5c), with 75% of cells having a nearest neighbor less than 23 μm away and 25% having a nearest neighbor less than 3.5 μm away in the BARseq capture. Nevertheless, with a kernel of bandwidth 10 μm used to generate the 10 μm smoothed image and most regions at least on the order of 100 μm in width at their narrowest (e.g. corpus callosum), the majority of cell mass stayed primarily within the initial region it was found.
Regarding alignment, an imagebased approach can achieve global alignment of the template to the target (Fig. 5a), with initial differences in scale and shape (Fig. 4b) resolved with the estimated transformation. However, we see clear differences in alignment on the order of 50−150 μm by examination of the overlap in the different layers of the cortex and adjacent corpus callosum (Fig. 5b, d). Importantly, as described in “Data model and optimization problem: image varifolds and transformations for molecular scales based on varifold norms” section, in the joint imagevarifoldbased approach to estimating geometric transformations and latent distributions, we model each atlas region as homogeneous and stationary with respect to space. This gives an optimal alignment between the atlas and target that maximizes similarity in distribution over features across each site in a single atlas region while minimizing the energy of the geometric deformation (diffeomorphism). We would consequently expect this to skew emphasis away from the foregroundbackground boundaries that typically govern image alignment and instead highlight the underlying assumptions in the architecture of the cartoon atlas, whose boundaries were initially constructed so as to maximize the homogeneity of the region. This becomes clear in comparing the alignment of BARseq to CCFv3 section in areas of low versus high cell density where these areas of low density correctly align with layer 1 and corpus callosum (CC) as a result of the imagevarifold based approach (Fig. 5d) but not in the imagebased approach (Fig. 5b). Notably, the corpus callosum and dorsal hippocampal commisure appear with equivalently low cell density on both right and left hemispheres in the imagevarifoldbased approach (yellow arrow) whereas we see these areas partially covered by CA1 cells, particularly in the right hemisphere in the imagebased approach. Layer 1 (white arrow) is covered by high cell density around the entire circumference of the coronal section in the imagebased approach, whereas it is visible, with relatively few cells mapping to it, as expected, in the imagevarifoldbased approach. Supplementary Fig. 4 shows the relative cell density in each of the cortical layers and white matter structures within the neighborhood of the primary visual cortex, with xIVLDDMM showing more accurate alignment with higher levels (4–6×) in layers 2–6 compared with layer 1 and white matter. Imagebased LDDMM, in contrast, yields similar levels of cell density across all of these structures, with the band of high cell density in BARseq layers 2–6 covering the entirety of the cortex, thus generating overestimates (4–6×) of cell density in layer 1. Cell density within CA1 also covers the areas of the corpus callosum, leading to overestimates of density there. Finally, as the imagevarifoldbased approach jointly considers both variations in the total density of cells and relative distribution of cell type, we see within each layer of cortex delineated in the CCFv3, a majority of corresponding cell types indicating correct layerbylayer alignment (Fig. 5d). Classification of cells by nearest atlas region yields 88% of layer 2/3 cells (total 443), 86% of layer 4/5 cells (total 436), 87% of layer 5 cells (total 278), and 70% of layer 6 cells (total 608) classified correctly according to cortical layer, within a 1.5 mm^{2} section through the primary visual cortex shown in Fig. 5. In contrast, for a similar section in the result of imagebased mapping (Fig. 5b), 38% of layer 2/3 pixels (total 308), 36% of layer 4 pixels (total 269), 64% of layer 5 pixels (total 225), and 78% of layer 6 pixels (total 326) are classified correctly according to nearest cortical layer.
Crossreplicate comparison and atlas construction
As described in the Introduction, two key applications of xIVLDDMM are in crosssample comparison and atlas construction. We illustrate both of these applications in considering celltyped MERFISH sections taken from three separate mice at approximately the same coronal level (Fig. 6a–c). CCFv3 section Z = 890 out of 1320 was mapped separately to each of these targets with both geometric transformations and latent feature distributions estimated in each case. As in the classical imaging setting with LDDMM, comparison of the estimated diffeomorphisms taking the CCFv3 section to each respective target section offers one metric of similarity between both atlas and target and across targets^{37,38} (Supplementary Fig. 5A–C). Here, the CCFv3 section contracts to a similar extent (~0.5) in all three mice in areas of the midbrain around the periacqueductal gray matter. In contrast, levels of contraction/expansion across the cortical layers in the primary visual area range from 0.7 to 1.4 across the three mice, indicating differences in geometric shape. In xIVLDDMM, the jointly estimated latent feature distributions offer a second metric of similarity (Supplementary Fig. 5D–G). Differences in the predicted cell type with highest probability occur, for instance, in the perirhinal and ectorhinal areas with excitatory granule cells, excitatory pyramidal neurons, or cortical excitatory neurons predominant in the different mice (Fig. 6d–f). The sample variance for each atlas region across the three estimated cell type probability distributions, modeled as vectors, \(p\in {{\mathbb{R}}}^{C}\), for C = 33 cell types and ∣p∣ = 1, measures difference not just in the single cell type with highest probability, but amongst the entire estimated distributions over cell type for each mouse (see “Empirical distribution estimation and crossreplicate statistical comparison” section). Sample variance (Supplementary Fig. 5F, G) is computed as \(\frac{1}{N1}{\sum }_{i=1}^{N}\parallel {p}_{i}\bar{p}{\parallel }_{2}^{2}\) in \({{\mathbb{R}}}^{C}\), with N = 3 and \(\bar{p}\), the sample mean (Supplementary Fig. 5D, E). Notably, the area of the medial geniculate nucleus, while exhibiting excitatory neurons as the predominant cell type across mice, exhibits high variance in total distribution. This region is magnified in Supplementary Fig. 6A, illustrating this variance in relative distribution of excitatory neurons, astrocytes, endothelial cells, and ependymal cells.
Regarding atlas construction, the sample mean cell type distribution across those latent distributions estimated jointly with geometric transformations (Supplementary Fig. 5D, E) gives one potential construction of a cell type atlas over space. A finergrained atlas can alternatively be achieved in this setting by pulling back each target MERFISH section onto the same CCFv3 section via the inverse estimated diffeomorphism (Fig. 1). This eliminates some of the variance in geometry between targets (Supplementary Fig. 5A–C), facilitating more direct comparison of cell type distributions. Particles (analogous to foreground pixels) in the CCFv3 section serve as a scaffold for resampling mapped cells to with either a nearest neighbor assignment or dispersion of cell mass according to kernel choice (see “Empirical distribution estimation and crossreplicate statistical comparison” section). Atlases can be constructed from the resampling of each target individually, with variance across the replicates’ empirical cell type probability distributions (Fig. 6g) similar to those estimated jointly in xIVLDDMM (Supplementary Fig. 5D–G), but also together for generation of a population average (Fig. 6h). Importantly, each particle, here, is treated independently without assuming homogeneity in the cell type distribution across particles belonging to the same atlas region. This is in contrast to the distributions estimated jointly with geometric transformations (Fig. 6d–f), and consequently results in more locally varying distributions. Overall, ventral areas exhibit higher spatial variance than dorsal areas (Fig. 6i), exhibiting that in such regions, an assumption of homogeneity within these regions may not always be appropriate. The white arrow, in particular, points to an area of cortex where this spatial variance is amongst the highest, with the outermost portion of cortex predominantly composed of ependymal cells versus the innermost portion predominantly excitatory pyramidal cells (see Supplementary Fig. 6B).
The spatial variance in cell type distribution within each atlas region and the variance in distribution between replicates of a given population both influence the statistical power needed to detect biologically relevant differences in these distributions between populations. Classically, they are accounted for in mixed effects models^{47} aimed at detecting differences in group features. For instance, we have looked at differences in atrophy rate of medial temporal lobe structures between control and diseased cohorts in Alzheimer’s disease^{19,48}. Here, we observe greater spatial variance across ventral regions than dorsal regions (Fig. 6i), with particularly minimal variance in layers of cortex. Importantly, the use of nonrigid geometric transformations (diffeomorphisms) to pull back targets into CCFv3 coordinates reduces this spatial variance in distribution across cortical layers (Fig. 6j, k) compared with rigid and scaling transformations only in each of the three mice. Consequently, in settings of comparing groups of replicates under different experimental conditions, we would expect this reduction in variance to facilitate detection of significant differences in cell type distribution per CCFv3 region between them.
Here, the small sample size (n = 3) of replicates, which are all produced under control conditions, precludes any definitive statistical statement about variations in cell type distributions observed between them and within CCFv3 regions. Nevertheless, summation across all CCFv3 regions of the variance per individual cell type probability between replicates and within each region (Fig. 7) elucidates the relative contribution of each cell (sub)type probability to the variances observed in Fig. 6g, i. We observe, for instance, with both types of variance, a large contribution from the variance in ependymal cell type probabilities (Fig. 7a, b). Specifically between replicates, excitatory neurons exhibit the second largest variance in cell type probabilities, as evidenced in the area of the dentate gyrus (Fig. 7c–e). With regard to spatial variance, we see specific astrocyte subtype probabilities with large variance, exhibited particularly in the areas of CA1 (yellow arrow) and the pons (orange arrow) both medially to laterally within the regions and when considering left versus right hemispheres (Fig. 7f, g). In contrast, other astrocyte subtype probabilities (Fig. 7h) are seen to be consistent over space, in line with our assumption of homogeneity within each CCFv3 region.
Extension to additional tissue modalities for within and across species comparison
Constructed across institutions with varying combinations of molecular, chemical, genetic, and electrophysiological signals, multiple atlases per species now exist with different levels of granularity and intended applications^{1,2,49,50,51,52}. Given this plethora of atlases, each of which might define a different partitioning scheme over the same area of tissue, questions of comparison and relevance of each atlas to emerging molecular and cellular signatures naturally arise^{53}. While some atlases have been defined in the same coordinate framework—often achieved through existing methods of image registration or manual alignment^{53}—many exist in different coordinate frameworks. Together with mismatches in the number, type, and positioning of partitions, this poses a challenge not only to the evaluation of each atlas ontology’s fit to a molecular target but also to the ready comparison of atlas to atlas and the establishment of a clear metric of similarity between them. Hence, a second application of our method rests in its use to map not just atlas to the molecular dataset, but one atlas to another, with the geometric and functional correspondences (φ, π) yielded by our method serving as an anchor for crossexamination of existing ontologies both within and across species.
To map tissuescale atlas to tissuescale atlas, we model both objects as image varifolds, as described in “Data model and optimization problem: image varifolds and transformations for molecular scales based on varifold norms” section, but where both template and target are constrained to be of constant density (w_{i} = 1 for all x_{i} in foreground tissue). Both geometric transformations and latent feature distributions, (φ, π) are estimated, with the latter giving distributions over the target atlas features (labels) for each template atlas label, following the assumption of spatial homogeneity with each template atlas label presumed to map onto a single distribution of target atlas labels.
The joint estimation of geometric transformation, φ and conditional feature laws, \({({\pi }_{\ell })}_{\ell \in {{{{{{{\mathcal{L}}}}}}}}}\) in xIVLDDMM offers two modes of quantitative comparison between atlas ontologies. First, as in the classical image setting of LDDMM, the determinant of the Jacobian, ∣Dφ∣, of the estimated diffeomorphism, can be used as a metric of how similar the atlas ontologies are, reflective of how much boundaries of partitions move to maximize the overlap between homogeneous regions. However, unlike in classical image settings, the estimation of the additional family of feature laws here, π, affords a second metric of similarity with the computation of the entropy of the estimated conditional feature distributions.
The first panel in Fig. 8 shows the results of mapping one mouse atlas ontology to another with the Z section 680 in the CCFv3 mapped to the corresponding section in the Kim Lab Developmental atlas, termed Developmental Common Coordinate Framework (DevCCF)^{51}, (Fig. 8a–c and vice versa Fig. 8d–f). Allen CCFv3 and DevCCF anatomical delineations utilize two distinct ontologies based on cytoarchitecture and genoarchitecture, respectively^{1,51,54}. Fig. 8a, d depict the geometry of the section under each ontology, with the CCFv3 section hosting ≈140 independent regions and the DevCCF section ≈80. In this setting, both atlases are published in the same coordinate framework, giving φ = Id and thus, highlighting, instead, the estimated distributions over the other ontologies. Figure 8b, e depict the estimated conditional probability distributions, \({p}_{i}^{{\prime} },\,\,i\in I\), for each atlas section over the other atlas section’s ontology. The label with the highest probability in these distributions is plotted for each simplex in the mesh and which is consistent across each partition of each original atlas, given the homogeneity assumption in our model (i.e. a single π_{ℓ} for each \(\ell \in {{{{{{{\mathcal{L}}}}}}}}\)). The comparatively larger set of labels in the CCFv3 ontology results in labels being omitted from the corresponding estimated set of labels on the DevCCF ontology section Fig. 8e in the isocortex while multiple regions in the CCFv3 ontology carry the same most probable region in the DevCCF ontology. The differences in granularity with which each atlas segments the various areas of tissue are captured by the entropy of the estimated conditional feature distributions, for each simplex of the mesh (Fig. 8c, f). The entropy of the distributions estimated for the DevCCF ontology over the CCFv3 ontology Fig. 8f is on average, higher, than that of the distributions estimated for the CCFv3 ontology Fig. 8c, with probability mass distributed across ~5–7 different CCFv3 regions for each DevCCF region of the isocortex. Nevertheless, we see close to 1:1 correspondence between CCFv3 and DevCCF labels in the center section of the slice, where the entropy of the estimated distributions is near 0.
Atlas comparison can be conducted both independently and within the context of particular molecular targets. For example, here, we compare corresponding sections of the CCFv3 and DevCCF atlases^{51} via mapping each section to the same cellsegmented MERFISH section and comparing both estimated diffeomorphisms and cell type distributions (Fig. 8g–l). Cell type distributions are compared via visualization of the cell type with the highest probability within each region (Fig. 8g, j) and the overall entropy of the estimated distribution per region (Fig. 8i, l). The areas of the hippocampus (dashed circle) and striatum and amygdala (arrow) are partitioned with different levels of granularity. This leads to different optimal geometric transformations, as characterized by the determinant of the Jacobian (Fig. 8b, e), and different predicted cell type distributions (Fig. 8c, f). As discussed above, though both atlases are published as geometrically aligned^{51}, the diffeomorphism solving the variational problem, here, transforms geometrically the homogeneous regions between the atlas and target. Hence, regions of the amygdala and striatum undergo significant contraction in the optimal mapping of DevCCF but not CCFv3 to MERFISH given the partitioning of this region into fewer and thus larger presumed homogeneous regions in the DevCCF atlas. Just as entropy in estimated distribution can be used to compare atlas to atlas, directly, we examine it here comparatively between atlases as an indication of which regions in which atlas achieve more or less homogeneous cell type distributions (Fig. 8i, l). Here, the hippocampus is more finely partitioned in the Kim atlas, which yields lower entropy distributions over cell types than in those estimated for the CCFv3.
Atlas ontologies can be mapped not just within species but also across them, where both geometric transformations and estimated ontology distributions, together reflect metrics of comparison between the two. As an example, we map a coronal section, Z = 537, in the CCFv3 to a coronal section, Z = 628 in the Waxholm Rat Brain Atlas^{52}, with both sections chosen to correspond as sections through the anterior commissure (Fig. 9). Here, the CCFv3 ontology is comprised of ≈ 120 regions (Fig. 9a) with ≈30 for the Waxholm atlas section (Fig. 9d). Initial differences in size and shape exist between the two tissue sections (Fig. 9b). After scaling the volume of the mouse brain by 1.5, additional deformation, with magnitude given by the determinant of the Jacobian, ∣Dφ∣, distorts both internal and external tissue boundaries to align homogeneous regions in each atlas, such as cingulate area to cingulate area (white arrow). Estimated distributions over the Waxholm ontology labels for each region in the CCFv3 are shown in Fig. 9c, f, summarized by the maximum probability label (Fig. 9f) and measures of entropy (Fig. 9c), which highlight in gray, CCFv3 regions mapping to ≈3–4 Waxholm regions versus those in black achieving 1:1 correspondence.
Extension to additional cellularscale modalities for image segmentation
With the computational constraints imposed by modeling spatial transcriptomics datasets as classical images, this work has emphasized the mapping of tissuescale atlases to molecular datasets generated by emerging imagebased rather than the often regularized spotbased spatial transcriptomics technologies. However, as exhibited through the modeling of atlas sections as image varifolds, themselves, both as template and target (see “Extension to additional tissue modalities for within and across species comparison” section), the xIVLDDMM is equally capable of aligning tissuescale atlases to alternative molecular and cellularscale modalities. As an example, we take a DAPIstained tissue section (Fig. 10), digitized at 2.5 μm resolution, corresponding to the genebased MERFISH section shown in Fig. 1. The DAPI image (~13 million pixels) is converted to an imagevarifold particle representation by discretization of its image values into ~35 discrete bins, and selection of foreground pixels with corresponding image values in the later 30 of these bins. Approximately 250k particles are used to represent these pixels, with each particle pertaining to a set of 25 neighboring pixels (5 × 5 square). Particles each carry the distribution of bins into which the image values of these 25 pixels fall, retaining the individual values of each foreground pixel at the highest resolution in contrast to typical image downsampling, in which a pixel only captures the single mean image value of its neighbors.
Figure 10 depicts this particle representation of the DAPI image and its positioning before and after alignment with the corresponding CCFv3 section. Similar to the BARseq mapping illustrated in Fig. 5, we see areas of lower cell density (fewer foreground pixels) versus higher cell density (greater and higher intensity foreground pixels) aligning to expected CCFv3 regions (e.g. layer 1 (L1) especially between the hemispheres, and substantia inominata (SI), respectively). Interestingly, the area of the olfactory tubercle (OT) in the CCFv3 section which is initially quite larger than that in the DAPI image remains large following geometric transformation to the target space, with optimal cost minimization favoring this alignment to one in which this area would drastically contract.
Finally, while the estimated geometric transforms here could equivalently pull back the DAPI image into the space of the CCFv3 section, as done in the setting of atlas construction and crossreplicate comparison (see “Crossreplicate comparison and atlas construction” section), we show, instead, the CCFv3 section transformed to the space of the DAPI target as an illustration of how our methodology can be used in settings of image segmentation. This facilitates the extraction of measures in a particular subset of regions, that can be directly compared to other types of measures (e.g. gene expression, cell type distributions) as detected by other types of technologies and equivalently localized to these same CCFv3 regions through mappings computed with xIVLDDMM, as demonstrated here in “Generalizability of xIVLDDMM to settings of complete and incomplete genebased and cellbased data,” “Quantitative and comparative evaluation of xIVLDDMM,” and “Crossreplicate comparison and atlas construction” sections. For instance, a comparison of the mean DAPI signal (Supplementary Fig. 7) to the maximally expressed gene per CCFv3 region, as shown in Fig. 1, illustrates a positive correlation between DAPI intensity and Whrn expression, particularly in areas bordering the corpus callosum versus negative correlation with Mdga1 expression in outer cortical layers. Finally, as described in the setting of resampling image varifolds across scales^{34}, we demonstrate the feasibility of translating between particle and image representations in resampling the deformed CCFv3 particles with Gaussian smoothing onto the same 2.5 μm grid of the original DAPI image (Fig. 10b). Note that this smoothing extends the borders of the atlas section slightly beyond their respective positioning as particles, as seen in the difference in overlay laterally within L1 between image and particle representations (Fig. 10b, d, respectively).
Discussion
We have introduced, here, a universal method, xIVLDDMM, for mapping tissuescale, ‘cartoon’ atlases to molecular and cellular datasets arising in the context of emerging transcriptomics technologies. We root our method in the modeling of each object as an image varifold, as previously described^{35}, and map across scale and modality by simultaneously estimating a geometric transformation with classical deformation tools of LDDMM^{17} and latent distribution over target features for each atlas region, under an assumption of homogeneity. Offering multiple means of implementation and optimization schemes (see “Alternating LDDMM and quadratic program algorithm for joint optimization” section), we have used both point cloud and meshbased implementations of imagevarifold objects, showcasing successful estimation of geometric mappings and latent feature distributions in both cases.
As presented, xIVLDDMM fills a current need, as highlighted previously^{55}, for universal tools that can integrate the diverse types and large quantities of data emerging from the evolution of both transcriptomics and imaging technologies over the last decade. With each technology generating a slightly different perspective and a different set of animal or human samples to compare, a method that can stably handle the format of past, current, and future datasets will be paramount to integrate both new findings with the vast number of datastores currently available across institutes. The imagevarifold framework used here is general enough to model classical tissuescale imaging data (as demonstrated in our atlastoatlas mappings), digital pathology (as demonstrated in our example with the DAPIstained image) (“Extension to additional cellularscale modalities for image segmentation” section), and emerging transcriptional data from both imagebased and spotresolution technologies that might generate data complete or partial tissue captures, irregularly sampled across space. Therefore, it provides a gateway for integrating data historically curated through MRI, immunohistochemistry, and other staining and imaging procedures in addition to the emerging transcriptomics methods. Importantly, this spatial integration will enable the correlation of disease signatures at different scales, as measured through different technologies. For instance, we aim to correlate tau tangle and amyloidbeta pathology in Alzheimer’s disease to gene expression and imaging biomarkers, as vital for furthering our mechanistic understanding of the disease and developing early diagnostic strategies. Additionally, the specific estimation of diffeomorphic mappings compared with rigid+scale transformations decreases the spatial variance in feature distributions across regions, increasing the statistical power for detecting differences in feature distributions (e.g. gene expression, cell composition) not just across replicates within a single group, but between groups of replicates under different experimental conditions. For instance, we are currently looking at differences in cell type and gene distribution following neonatal binocular enucleation as measured with BARseq in four cases versus control hemispheres^{56}. As presented, xIVLDDMM fills a current need, as highlighted previously^{55}, for universal tools that can integrate the diverse types and large quantities of data emerging from the evolution of both transcriptomics and imaging technologies over the last decade. With each technology generating a slightly different perspective and a different set of animal or human samples to compare, a method that can stably handle the format of past, current, and future datasets will be paramount to integrate both new findings with the vast number of datastores currently available across institutes. The imagevarifold framework used here is general enough to model classical tissuescale imaging data (as demonstrated in our atlastoatlas mappings), digital pathology (as demonstrated in our example with the DAPIstained image) (“Extension to additional cellularscale modalities for image segmentation” section), and emerging transcriptional data from both imagebased and spotresolution technologies that might generate data complete or partial tissue captures, irregularly sampled across space. Therefore, it provides a gateway for integrating data historically curated through MRI, immunohistochemistry, and other staining and imaging procedures in addition to the emerging transcriptomics methods. Importantly, this spatial integration will enable the correlation of disease signatures at different scales, as measured through different technologies. For instance, we aim to correlate tau tangle and amyloidbeta pathology in Alzheimer’s disease to gene expression and imaging biomarkers, as vital for furthering our mechanistic understanding of the disease and developing early diagnostic strategies. Additionally, the specific estimation of diffeomorphic mappings compared with rigid+scale transformations decreases the spatial variance in feature distributions across regions, increasing the statistical power for detecting differences in feature distributions (e.g. gene expression, cell composition) not just across replicates within a single group, but between groups of replicates under different experimental conditions. For instance, we are currently looking at differences in cell type and gene distribution following neonatal binocular enucleation as measured with BARseq in four cases versus control hemispheres^{56}. As presented, xIVLDDMM fills a current need, as highlighted previously^{55}, for universal tools that can integrate the diverse types and large quantities of data emerging from the evolution of both transcriptomics and imaging technologies over the last decade. With each technology generating a slightly different perspective and a different set of animal or human samples to compare, a method that can stably handle the format of past, current, and future datasets will be paramount to integrate both new findings with the vast number of datastores currently available across institutes. The imagevarifold framework used here is general enough to model classical tissuescale imaging data (as demonstrated in our atlastoatlas mappings), digital pathology (as demonstrated in our example with the DAPIstained image) (“Extension to additional cellularscale modalities for image segmentation” section), and emerging transcriptional data from both imagebased and spotresolution technologies that might generate data complete or partial tissue captures, irregularly sampled across space. Therefore, it provides a gateway for integrating data historically curated through MRI, immunohistochemistry, and other staining and imaging procedures in addition to the emerging transcriptomics methods. Importantly, this spatial integration will enable the correlation of disease signatures at different scales, as measured through different technologies. For instance, we aim to correlate tau tangle and amyloidbeta pathology in Alzheimer’s disease to gene expression and imaging biomarkers, as vital for furthering our mechanistic understanding of the disease and developing early diagnostic strategies. Additionally, the specific estimation of diffeomorphic mappings compared with rigid+scale transformations decreases the spatial variance in feature distributions across regions, increasing the statistical power for detecting differences in feature distributions (e.g. gene expression, cell composition) not just across replicates within a single group, but between groups of replicates under different experimental conditions. For instance, we are currently looking at differences in cell type and gene distribution following neonatal binocular enucleation as measured with BARseq in four cases versus control hemispheres^{56}. As presented, xIVLDDMM fills a current need, as highlighted previously^{55}, for universal tools that can integrate the diverse types and large quantities of data emerging from the evolution of both transcriptomics and imaging technologies over the last decade. With each technology generating a slightly different perspective and a different set of animal or human samples to compare, a method that can stably handle the format of past, current, and future datasets will be paramount to integrate both new findings with the vast number of datastores currently available across institutes. The imagevarifold framework used here is general enough to model classical tissuescale imaging data (as demonstrated in our atlastoatlas mappings), digital pathology (as demonstrated in our example with the DAPIstained image) (“Extension to additional cellularscale modalities for image segmentation” section), and emerging transcriptional data from both imagebased and spotresolution technologies that might generate data complete or partial tissue captures, irregularly sampled across space. Therefore, it provides a gateway for integrating data historically curated through MRI, immunohistochemistry, and other staining and imaging procedures in addition to the emerging transcriptomics methods. Importantly, this spatial integration will enable the correlation of disease signatures at different scales, as measured through different technologies. For instance, we aim to correlate tau tangle and amyloidbeta pathology in Alzheimer’s disease to gene expression and imaging biomarkers, as vital for furthering our mechanistic understanding of the disease and developing early diagnostic strategies. Additionally, the specific estimation of diffeomorphic mappings compared with rigid+scale transformations decreases the spatial variance in feature distributions across regions, increasing the statistical power for detecting differences in feature distributions (e.g. gene expression, cell composition) not just across replicates within a single group, but between groups of replicates under different experimental conditions. For instance, we are currently looking at differences in cell type and gene distribution following neonatal binocular enucleation as measured with BARseq in four cases versus control hemispheres^{56}.
In parallel to the development and dispersion of diverse molecular datasets, there has been continued development on the side of reference atlases to reflect trends in these new measures and integrate these trends across even more samples of particular species. With regard to atlas refinement and creation, as shown in “Crossreplicate comparison and atlas construction” section and the second panel of Fig. 1, the invertibility of the estimated diffeomorphism in the setting of mapping atlas to molecular target, enables the carrying of each target into the same coordinate space of the atlas. Though we illustrate an example of atlas creation from samples of a single modality, each target to which the same atlas section is mapped can be pulled back into the same coordinate space. This effectively allows different molecular modalities to be indirectly mapped to one another, where the challenge of matching different molecular features as needed to map one modality directly to another is facilitated through the scaffold of the presumed homogeneous regions in the atlas that anchor the estimated feature distributions for alignment to each respective molecular modality. Furthermore, this enables the construction of a composite set of molecular and cellularscale data across technologies and modalities (e.g. digital pathology and spatial transcriptomics, as seen in the comparative mappings of the same CCFv3 section to MERFISH (Fig. 1) and DAPI images (Fig. 10)), that can be integrated into a multifactorial atlas with defined segmentation schemes according to homogeneous regions across these features. Our method also offers a tool for reexamining and comparing existing atlas ontologies in the context of new data^{55}, and serves as a means for developing new atlases in the future. As described in “Extension to additional tissue modalities for within and across species comparison” section, examination of the mappings achieved between different atlases and the same molecular target offers an indirect comparison between atlases in the context of a particular molecular setting. However, this comparison can also be made directly in a contextindependent setting by harnessing our method to map atlas to atlas. In the field of evolutionary biology, for instance, our method could aid in the mapping and comparison of atlases across species^{57,58} and in the field of developmental neurobiology, the available atlases of the brain at different stages of development^{49,50,51,54,59}. Finally, as shown in “Extension to additional cellularscale modalities for image segmentation” section, xIVLDDMM can be applied across molecular and cellular modalities, not just for incorporating them into atlas creation, but also as a means of segmentation in the classical image sense.
While the results presented here survey a wide variety of potential applications of xIVLDDMM to mapping atlas modalities to diverse targets, there remain uncertainties and potential modes of improvement that are the subject of current and future work. First, we have presented results mapping 2D sections of 3D atlases to corresponding 2D sections of MERFISH data. Both the Allen MERFISH and BARseq data showcased here are part of an entire set of serial sections that span the whole brain. While we have approximated these sections as strictly coronal in pairing them to respective coronal slices of the CCFv3 here, typically such datasets are generated by sectioning on an angled plane, which would be best approximated by the mapping of the entire set of sections as a 3D object to the 3D CCFv3. Consequently, we are optimizing our method to compute mappings of atlas to molecular targets in 3D, where both added dimensions and added magnitudes of data contribute to the theoretical and computational complexity of the problem. Indeed, with ≈6 billion individual transcripts measured across the span of the brain, treatment of this data as a regular lattice image would require on the order of 1000 billion voxels at 1 μm resolution, which is coarser than that needed even to resolve two molecules of mRNA. Hence, it becomes even more vital to treat such data in the particle setting, as presented here, where we capture the sparsity and irregularity of the data in modeling it effectively in its lowest dimension, as 6 billion individual particle measures. Second, though we have highlighted a range of genebased and cellbased, whole and hemibrain datasets achieved with imagebased technologies, here, the future investigation includes the use of our method to map data from additional types of technologies, such as the spotresolution SlideSeq^{44}, and in additional biological settings, where we might see further disruptions of tissue architecture (e.g. with tumors) and tissue types with varying levels of the organization and therefore varying levels to which homogeneous distributions over molecular features can be seen over physical space (e.g. in heart, breast, and lung).
Indeed, we finally emphasize that central to the model posed here is the underlying assumption that each compartment has a homogeneous distribution over molecular features that is stationary with respect to space. We make this assumption to govern the estimation of latent feature distributions for each atlas compartment to take the atlas into both the physical and feature space of the target. This is in contrast to learningbased approaches that estimate features independent of such assumptions to map atlas to target across feature (e.g. imaging) modalities but often require extensive training datasets and computational power to learn such features^{18,26}. This assumption stems from the inherent construction of atlases often to delineate regions of particular cell types or with a particular function, and thus, where we see a set of predominant cell types or gene types consistently across the region in the molecular scale data, as in Fig. 3. The successful alignment of tissuescale atlas to varying molecular scale data demonstrated through the examples shown here supports the benefit of such an assumption in jointly estimating geometric transformations and latent feature distributions. However, as manifest in Fig. 2, where the expression of Trp53i11 appears to be distributed along a decreasing gradient medial to lateral within the striatum, or explicitly through the spatial variance in empirical distribution shown for each CCFv3 region in Fig. 6, this assumption of spatial homogeneity does not always hold. The subsequent construction of empirical distributions from the pulling back of different molecular targets, as shown in Fig. 6 enables the estimation of feature distributions at a finer scale without the assumption of homogeneity and simultaneously, an evaluation of where and to what extent it holds, with notably, the use of diffeomorphic mappings achieving a further reduction in this spatial variance than rigid+scale transformations alone. Furthermore, the results presented here reflect a particular balance between expected deformation and this homogeneity assumption, imposed by the relative weighting of the separate terms in the cost function. Current work at controlling this balance further includes the addition of a term controlling the divergence of the vector field to the energy defined in the variational problem (10), which leads to solutions more robust to deformation within the interior of the tissue. Future work will also include a more rigorous evaluation of how well this homogeneity assumption holds across different tissue contexts and the effect the given balance between the two terms might have in different settings.
Methods
Construction of imagevarifold representation for different modalities
As introduced in “Data model and optimization problem: image varifolds and transformations for molecular scales based on varifold norms” section, we can represent each imagevarifold object as a point cloud (particles) or a triangulated mesh. In the first case, data is modeled as a collection of particles, each with a center \({x}_{i}\in {{\mathbb{R}}}^{2}\), and a measure over its feature space, \({w}_{i}{p}_{i}\in {{{{{{{\mathcal{M}}}}}}}}({{{{{{{\mathcal{F}}}}}}}})\). Particles may be individual mRNA detections, cell centers, or image pixels at the highest resolution, with thresholding or exclusion of particular image values done in the latter case to extract foreground pixels only. Sets of particles can be downsampled to a smaller set of discrete particles by capturing neighboring particle distributions together into one particle, as used in the implementation of the DAPI image (“Extension to additional cellularscale modalities for image segmentation” section). We refer to the resolution of the rendering to indicate the span of the neighborhood each individual particle captures with the coarser the resolution corresponding to larger and larger neighborhoods of measurements encapsulated in a single particle.
In the case of meshes, each mesh is built from a collection of vertices indexed by the set I, \({{{{{{{\bf{x}}}}}}}}={({x}_{i})}_{i\in I}\) with each \({x}_{i}\in {{\mathbb{R}}}^{2}\). Each simplex in the mesh is defined from the vertices denoted as γ(x) and is paired with a 3tuple with components that index the vertices of the simplex, (γ(x), c = (c^{1}, c^{2}, c^{3}) ∈ I^{3}) and determine the center \(m({{{{{{{\bf{x}}}}}}}})=\frac{1}{3}({x}_{{c}^{1}}+{x}_{{c}^{2}}+{x}_{{c}^{3}})\). Each triangle simplex is defined by
with positive orientation and volume \( {\gamma }_{c}({{{{{{{\boldsymbol{x}}}}}}}}) :=\frac{1}{2}\parallel ({x}_{{c}^{2}}{x}_{{c}^{1}}) \times ({x}_{{c}^{3}}{x}_{{c}^{1}})\parallel > 0\).
The total mesh τ is the collection of vertices x, and simplices and centers \({({\gamma }_{j}({{{{{{{\bf{x}}}}}}}}),{c}_{j}=({c}_{j}^{1},{c}_{j}^{2},{c}_{j}^{3}),{m}_{j}({{{{{{{\bf{x}}}}}}}}))}_{j\in J}\), with the simplices indexed by the set J, and with the resolution determining the complexity as total numbers of vertices ∣I∣ and the number of simplices ∣J∣ in the mesh. To complete the image varifold we append to the mesh the spatial density defined over the area of each simplex: \({{{{{{{\boldsymbol{\alpha }}}}}}}}={({\alpha }_{j})}_{j\in J}\) and the field of probability laws \({{{{{{{\boldsymbol{p}}}}}}}}={({p}_{j})}_{j\in J}\) on \({{{{{{{\mathcal{F}}}}}}}}\). Hence, we denote an image varifold implemented as a mesh similar to the normalized definition of (1) as:
with w_{j} = α_{j}∣γ_{j}∣ giving the assumed constant density over the area of the simplex. Notably, diffeomorphisms act directly on the vertices, with the center m_{j} of each simplex shifted according to the movement of each of its vertices. The determinant of the Jacobian is introduced to retain the same spatial density of the object before and after transformation, as in (2) but with it evaluated at the center of each simplex, and again assumed constant over the area of each simplex, approximated as the ratio in the simplex area after and before transformation:
Additionally, in the setting of meshes, it is convenient to approximate the product \( D\varphi { }_{{m}_{j}} {\gamma }_{j}\) by the new area of each simplex following transformation, ∣γ_{j}(φ_{1}(x))∣.
Meshes mapped in this work were constructed using Delauney triangulation^{60} on a grid defined over the support of the starting dataset with the size of each square dictated by the input resolution. Spatial density and conditional feature measures, α, p, were associated with the simplices of the mesh following the assignment of each individual data point (e.g. mRNA or cell read) into its single nearest simplex. Meshes were pruned of simplices that both contained fewer than 1 data point and existed outside the largest connected component of simplices containing at least one data point. In this manner, both for atlas images and transcriptomics datasets, resulting simplex meshes spanned the entire tissue foreground.
Molecular scale varifold norm
We define the space of image varifolds μ ∈ W ^{*} to have a norm \(\parallel \cdot {\parallel }_{{W}^{*}}^{2}\), and transform the atlas coordinates onto the targets to minimize the norm. The space of varifold norms is associated to a reproducing kernel Hilbert space^{34,61} (see (8) below) defined by the inner product of the space as \({\langle \mu,\nu \rangle }_{{W}^{*}}\), \(\parallel \mu {\parallel }_{{W}^{*}}^{2}={\langle \mu,\mu \rangle }_{{W}^{*}}\). To specify the imagevarifold norm for \(\mu \in {W}^{*},\parallel \cdot {\parallel }_{{W}^{*}}^{2}\), it suffices to provide the inner product between Diracs \({\langle {\delta }_{x}\otimes {\delta }_{f},{\delta }_{{x}^{{\prime} }}\otimes {\delta }_{{f}^{{\prime} }}\rangle }_{{W}^{*}}=K((x,f),({x}^{{\prime} },{f}^{{\prime} }))\). For any μ in (1) then
Throughout we use the kernel product \(K((x,f),({x}^{{\prime} },{f}^{{\prime} }))={K}_{1}(x,{x}^{{\prime} }){K}_{2}(f,{f}^{{\prime} })\) chosen as a Gaussian over physical space \({K}_{1}(x,y)=\exp (\frac{\parallel x{x}^{{\prime} }{\parallel }^{2}}{2{\sigma }^{2}})\) with \({K}_{2}(f,{f}^{{\prime} })=1\,{{{{{{{\rm{if}}}}}}}}f={f}^{{\prime} }\), 0 otherwise giving:
for the imagevarifold object, as defined in (1) in “Data model and optimization problem: image varifolds and transformations for molecular scales based on varifold norms” section.
Variational problems
As described in “Data model and optimization problem: image varifolds and transformations for molecular scales based on varifold norms” section, the mapping variational problem constructs a diffeomorphism \(\varphi :{{\mathbb{R}}}^{2}\to {{\mathbb{R}}}^{2}\) and feature laws \({({\pi }_{\ell })}_{\ell \in {{{{{{{\mathcal{L}}}}}}}}}\) on \({{{{{{{\mathcal{F}}}}}}}}\) to carry the atlas image varifold onto the target, minimizing the varifold normed difference between them, with norm defined as in “Molecular scale varifold norm” section. We follow LDDMM as described in the image case^{17}, parameterizing the diffeomorphism with the smooth timevarying velocity field, v_{t}, t ∈ [0, 1], as \({\dot{\varphi }}_{t}={v}_{t}\circ {\varphi }_{t}\). This gives the variational problem between the atlas image varifold, μ_{A}, with indexed locations, I, and the target image varifold, μ_{T}, with indexed locations, J:
with \({\mu }_{A}^{\pi }\) depicting the featuretransformed atlas image varifold, as described in “Data model and optimization problem: image varifolds and transformations for molecular scales based on varifold norms” section, as \({\sum }_{i\in I}{\delta }_{{x}_{i}}\otimes {w}_{i}^{{\prime} }{p}_{i}^{{\prime} }\) with \({w}_{i}^{{\prime} }{p}_{i}^{{\prime} }={\sum }_{\ell \in {{{{{{{\mathcal{L}}}}}}}}}{w}_{i}{p}_{i}(\ell ){\pi }_{\ell }\), the estimated distribution over target features for indexed location i in the atlas image varifold. The space of smooth timevarying velocity fields giving the flow, φ, is defined as a reproducing kernel Hilbert space, equipped with norm, \(\parallel \cdot {\parallel }_{V}^{2}\), ensuring smoothness and invertibility of φ as a diffeomorphism^{62}.
We solve (10) for optimal φ, π through either single or alternating algorithms as described in “Alternating LDDMM and quadratic program algorithm for joint optimization” section, with the opportunity to impose priors on the estimated distributions, π, appropriate to the specific setting. For instance, we typically impose positivity on all values in estimated distributions: π_{ℓ}(f) ≥ 0 for all \(f\in {{{{{{{\mathcal{F}}}}}}}}\) and \(\ell \in {{{{{{{\mathcal{L}}}}}}}}\). Given prior knowledge of the spatial density of cells or mRNA of a target, we may also specify a range of values for the resulting spatial density to take through constraints on the total measures of features estimated:
This prior can be easily incorporated as a constraint in the setting of the quadratic program used in the alternating algorithmic approach for estimating optimal distributions π (see “Alternating LDDMM and quadratic program algorithm for joint optimization” section). In the examples shown here, we typically take w^{min} to be the 5th percentile of values w_{j}, j ∈ J of the target image varifold. Additionally, in the setting of mapping tissuescale atlases to each other, as discussed in “Extension to additional tissue modalities for within and across species comparison” section, we impose the greater constraint of ensuring constant spatial density of 1, as we use for modeling both atlas and target image varifolds in this case. Finally, in other settings without prior knowledge or wish to impose any on the specific densities prescribed to each indexed location, we can add a general regularization term to (10), such as the Kullback–Liebler divergence between the normalized estimated π probability distribution and a uniform distribution across target features:
where \( {{{{{{{\mathcal{F}}}}}}}}\) gives the number of discrete feature values in the target feature space and the probability distribution, \({\bar{\pi }}_{\ell }=\frac{1}{{\sum }_{f\in {{{{{{{\mathcal{F}}}}}}}}}{\pi }_{\ell }(f)}{\pi }_{\ell }\). We use this approach in the examples with BARseq data shown and in the context of a simultaneous rather than alternating optimization algorithm.
Alternating LDDMM and quadratic program algorithm for joint optimization
For solving the variational problem of (10), optimal φ, π are jointly estimated through either a single or alternating optimization scheme. In both cases, the template and target can be initially aligned through separate estimation of rigid transformations (translation and rotation) and a single isotropic scaling applied to the template to bring the total area of the template to equal that of the target. In this setting, rigid transformations are estimated by minimizing the varifold normed difference between the rotated and translated template atlas transformed to the target with LBFGS.
Additionally, in both cases, a gradientbased optimization is performed until convergence or a specified number of iterations. We use LBFGS optimization combined with a line search using the Wolf condition. In the single scheme, geometric parameters, v_{t}, and feature parameters, π are optimized as a joint set of parameters in the gradientbased optimization. In contrast, an alternating scheme separates the estimation of geometric parameters and feature parameters with a gradientbased optimization for the former and the use of quadratic programming in the latter case.
The alternating scheme specifically follows^{35}, fixing the laws \({({\pi }_{\ell })}_{\ell \in {{{{{{{\mathcal{L}}}}}}}}}\) and optimizing over the control v(t), t ∈ [0, 1] and integrating it, with the initial condition at the identity element, φ_{0} = Id (e.g. \({\varphi }_{0}(x)=x,\forall x\in {{\mathbb{R}}}^{2}\)), to generate the diffeomorphism φ_{1}. The diffeomorphism is then fixed and quadratic programming, such as OSQP^{63}, used to estimate the feature laws. We outline this scheme explicitly below.
Algorithm 1
Initialize: \({\pi }_{\ell }(f)=\frac{1}{ {{{{{{{\mathcal{F}}}}}}}} },f\in {{{{{{{\mathcal{F}}}}}}}}\)
A: Solve for v:
1. Update and fix \({({\pi }_{\ell })}_{\ell \in {{{{{{{\mathcal{L}}}}}}}}}\).
2. Solve LDDMM, optimizing (10) with respect to vector field v_{t}, t ∈ [0, 1].
3. Solve for φ_{1}, integrating O.D.E \({\varphi }_{1}=\int\nolimits_{0}^{1}{v}_{t}\circ {\varphi }_{t}dt+Id\).
4. Flow \({\mu }_{A}^{\pi }\) according to φ_{1}, giving \({\varphi }_{1}\cdot {\mu }_{A}^{\pi }\).
B: Solve for \({({\pi }_{\ell })}_{\ell \in {{{{{{{\mathcal{L}}}}}}}}}\):
1. Fix spatial positions in the deformed template, φ(x_{i}), i ∈ I.
2. Optimize quadratic program (13) with respect to \({({\pi }_{\ell })}_{\ell \in {{{{{{{\mathcal{L}}}}}}}}}\).
Return to A
For the atlas, we define the form of the image varifold: \({\mu }_{A}={\sum }_{i\in I}{\delta }_{{x}_{i}}\otimes {w}_{i}{p}_{i}\), with, \({w}_{i}{p}_{i}\in {{{{{{{\mathcal{M}}}}}}}}({{{{{{{\mathcal{L}}}}}}}})\), a measure over the atlas feature values (partitions). The estimated feature distributions are given via π as the mixture distribution, \({w}_{i}^{{\prime} }{p}_{i}^{{\prime} }={\sum }_{\ell \in {{{{{{{\mathcal{L}}}}}}}}}{w}_{i}{p}_{i}(\ell ){\pi }_{\ell }\). We specify the target over the index set, J, as: \({\mu }_{T}={\sum }_{j\in J}{\delta }_{{x}_{j}}\otimes {w}_{j}{p}_{j}\) with each w_{j}p_{j} and estimated \({w}_{i}^{{\prime} }{p}_{i}^{{\prime} }\in {{{{{{{\mathcal{M}}}}}}}}({{{{{{{\mathcal{F}}}}}}}})\). We use this notation in defining the general form of the quadratic program, with constraint given as described in “Variational problems” section:
Remark 1
In the algorithm, under the assumption of a constant density with w_{i} = 1 for all i ∈ I, we can approximate the estimated distributions, \({w}_{i}^{{\prime} }{p}_{i}^{{\prime} },i\in I\), by the single π_{ℓ} at each location (e.g. particle or simplex) for the ℓ with the largest mass. Defining the greedy maximizer map \({\ell }^{*}(i)=\arg {\max }_{\ell \in {{{{{{{\mathcal{L}}}}}}}}}{p}_{i}(\ell )\in {{{{{{{\mathcal{L}}}}}}}}\), we can simplify the computations by using the approximation \({w}_{i}^{{\prime} }{p}_{i}^{{\prime} }\simeq {p}_{i}({\ell }^{*}(i)){\pi }_{{\ell }^{*}(i)}\). This is particularly relevant for tissuescale atlases where most particles or simplices in the imagevarifold object are interior to a single atlas region and therefore carry a conditional feature distribution, \({p}_{i}={\delta }_{{\ell }_{i}}\), with all mass attributed to a single feature value, ℓ_{i}. For these indexed locations, the above approximation becomes an equality: \({w}_{i}^{{\prime} }{p}_{i}^{{\prime} }={\pi }_{{\ell }^{*}(i)}={\pi }_{{\ell }_{i}}\).
Both single and alternating optimization schemes were utilized in the examples shown in this work and are available with respective Python git repositories: https://github.com/kstouff4/xIVLDDMMParticle v1.0.0^{64} and https://github.com/kstouff4/MeshLDDMMQP v1.0.0^{65}. The single optimization scheme is provided in the context of point cloud imagevarifold implementations, whereas the alternating scheme is provided in the context of meshbased imagevarifold implementations. The quadratic program solver used in the context of the alternating scheme is OSQP as implemented in the Python qpsolvers library. Both schemes were developed and tested on a 12 GB TITAN V GPU with CUDA 10.2 and NVIDIASMI and Driver 440.33.01, with expected runtime varying by the size of the dataset, but on the order of 2–4 h for most examples highlighted in this work.
Empirical distribution estimation and crossreplicate statistical comparison
For empirical atlas construction from single or multiple MERFISH sections (“Crossreplicate comparison and atlas construction” section), we used a 50 μm particle rendering (~20k particles) of the CCFv3 section as a scaffold. We assigned each mapped MERFISH cell to its nearest neighbor in the CCFv3 section, giving a distribution over cell types per particle in the CCFv3 section. As described in “Crossreplicate comparison and atlas construction” section, we computed variance in probability distribution within a region by modeling the probability distributions per particle, p_{i}, as vectors in \({{\mathbb{R}}}^{C}\), with the number of cell (sub)types C = 33, and ∣p_{i}∣ = 1. We computed mean cell type probability distributions per CCFv3 region, denoted \(\overline{{p}_{\ell }}\) for \(\ell \in {{{{{{{\mathcal{L}}}}}}}}\) by selecting the subset of particles, I_{ℓ} ⊂ I, for each region with at least half of their mass in that region. Under our model of spatial homogeneity per atlas region, we assume the conditional cell type probabilities per location (particle) within each atlas region are consistent. For a given cell (sub)type, c, and replicate, k, we computed spatial variance within a region, \(\ell \in {{{{{{{\mathcal{L}}}}}}}}\) as:
with N_{ℓ}, the number of particles in I_{ℓ}. Total variance per region was computed by summing (14) over cell types (Fig. 6i) and per cell type was computed by summing (14) over regions (Fig. 7b).
To assess differences in cell type distribution between replicates (Figs. 6g and 7a), we similarly constructed empirical cell type distributions for each replicate independently, giving for the same indexed particle set, I, in the CCFv3 section, three separate imagevarifold measures:
with \({w}_{i}^{k}\) for k = 0, 1, 2 denoting the cell density (cells per 50 μm^{2} area) for each of the three targets over the CCFv3 space and \({p}_{i}^{k}\) for k = 0, 1, 2. We computed spatial variance within each region for each replicate with (14) (Fig. 6j, k). We compared mean cell type probability distributions across replicates for a given cell type, c, and in a region, ℓ, as:
with total variance between replicates for a given cell type, c computed by summing (16) across regions (Fig. 7a) and total variance between replicates for a given region computed by summing (16) over cell types (Fig. 6g).
Reporting summary
Further information on research design is available in the Nature Portfolio Reporting Summary linked to this article.
Data availability
Serial MERFISH sections from the Allen Institute were produced under the BRAIN Initiative Cell Census Network (BICCN, www.biccn.org, RRID:SCR_015820) and are available at the Brain Image Library (BIL, https://www.brainimagelibrary.org/index.html) under https://doi.org/10.35077/g.610. Cellsegmented MERFISH sections with cell type annotations are available at Zenodo (https://doi.org/10.5281/zenodo.8384018) from Clifton et al.^{66}. Serial BARseq sections with celllevel data are available at Mendeley data (https://doi.org/10.17632/8bhhk7c5n9.1). The Waxholm rat brain atlas used in this study is available at https://www.nitrc.org/projects/whssdatlas. The Allen CCFv3 used in this study is available at https://download.alleninstitute.org/informaticsarchive/currentrelease/mouse_ccf/annotation/ccf_2022/. The DevCCF used in this study is available at https://kimlab.io/brainmap/DevCCF/.
Code availability
Implementations of the algorithms described here can be found at: https://github.com/kstouff4/MeshLDDMMQP (triangulated mesh implementation) v1.0.0^{65} and https://github.com/kstouff4/xIVLDDMMParticle (point cloud implementation) v1.0.0^{64}.
References
Wang, Q. et al. The allen mouse brain common coordinate framework: a 3d reference atlas. Cell 181, 936–953 (2020).
Paxinos, G. & Franklin, K. B. Paxinos and Franklin’s the Mouse Brain in Stereotaxic Coordinates (Academic Press, USA, 2019).
Jones, A., Townes, F. W., Li, D. & Engelhardt, B. E. Alignment of spatial genomics data using deep Gaussian processes. Nat. Methods 20, 1–9 (2023).
Christensen, G. E., Rabbitt, R. D. & Miller, M. I. Deformable templates using large deformation kinematics. IEEE Trans. Image Process. 5, 1435–1447 (1996).
Grenander, U. & Miller, M. I. Computational anatomy: an emerging discipline. Appl. Math. 56, 617–694 (1998).
Thompson, P. & Toga, A. A framework for computational anatomy. Comput. Vis. Sci. 5, 13–34 (2002).
Ding, Z. & Niethammer, M. Aladdin: Joint atlas building and diffeomorphic registration learning with pairwise alignment. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition 20784–20793 (IEEE, 2022).
Miller, M. I., Trouvé, A. & Younes, L. On the metrics and EulerLagrange equations of computational anatomy. Annu. Rev. Biomed. Eng. 4, 375–405 (2002).
Avants, B. & Gee, J. C. Geodesic estimation for large deformation anatomical shape averaging and interpolation. NeuroImage 23, 139–150 (2004).
Joshi, S., Davis, B., Jomier, M. & Gerig, G. Unbiased diffeomorphic atlas construction for computational anatomy. NeuroImage 23, 151–160 (2004).
Miller, M. I. Computational anatomy: shape, growth, and atrophy comparison via diffeomorphisms. NeuroImage 23, 19–33 (2004).
Ashburner, J. A fast diffeomorphic image registration algorithm. NeuroImage 38, 95–113 (2007).
Avants, B. B., Epstein, C. L., Grossman, M. & Gee, J. C. Symmetric diffeomorphic image registration with crosscorrelation: Evaluating automated labeling of elderly and neurodegenerative brain. Med. Image Anal. 12, 26–41 (2008).
Vercauteren, T., Pennec, X., Perchant, A. & Ayache, N. Diffeomorphic demons: efficient nonparametric image registration. NeuroImage 45, 61–72 (2009).
Yeo, B. T. et al. Spherical demons: fast diffeomorphic landmarkfree surface registration. IEEE Trans. Med. Imaging 29, 650–68 (2010).
Ashburner J, R. G. Symmetric diffeomorphic modeling of longitudinal structural MRI. Front. Neurosci. 5, 38074 (2013).
Beg, M. F., Miller, M. I., Trouvé, A. & Younes, L. Computing large deformation metric mappings via geodesic flows of diffeomorphisms. Int. J. Comput. Vision 61, 139–157 (2005).
Wang, J. & Zhang, M. Geosic: learning deformable geometric shapes in deep image classifiers. Adv. Neural Inf. Process. Syst. 35, 27994–28007 (2022).
Kulason, S. et al. Cortical thickness atrophy in the transentorhinal cortex in mild cognitive impairment. NeuroImage Clin. 21, 101617 (2019).
Heinrich, M. P. et al. MIND: modality independent neighbourhood descriptor for multimodal deformable registration. Med. Image Anal. 16, 1423–1435 (2012).
Tward, D. et al. Diffeomorphic registration with intensity transformation and missing data: application to 3D digital pathology of Alzheimer’s disease. Front. Neurosci. 14, 1–18 (2020).
Stouffer, K. M., Witter, M. P., Tward, D. J. & Miller, M. I. Projective diffeomorphic mapping of molecular digital pathology with tissue mri. Commun. Eng. 1, 44 (2022).
Iglesias, J. E. et al. Joint registration and synthesis using a probabilistic model for alignment of mri and histological sections. Med. Image Anal. 50, 127–144 (2018).
Yang, Q. et al. Mri crossmodality imagetoimage translation. Sci. Rep. 10, 1–18 (2020).
Islam, K. T., Wijewickrema, S. & O’Leary, S. A deep learning based framework for the registration of three dimensional multimodal medical images of the head. Sci. Rep. 11, 1–13 (2021).
Ding, W., Li, L., Zhuang, X. & Huang, L. Crossmodality multiatlas segmentation using deep neural networks. In International Conference on Medical Image Computing and ComputerAssisted Intervention 233–242 (Springer, Cham, 2020).
Tian, L., Chen, F. & Macosko, E. Z. The expanding vistas of spatial transcriptomics. Nat. Biotechnol. 41, 773–782 (2022).
Biancalani, T. et al. Deep learning and alignment of spatially resolved singlecell transcriptomes with tangram. Nat. Methods 18, 1352–1362 (2021).
Vahid, M. R. et al. Highresolution alignment of singlecell and spatial transcriptomes with cytospace. Nat. Biotechnol. 41, 1543–1548 (2023).
Zeira, R., Land, M., Strzalkowski, A. & Raphael, B. J. Alignment and integration of spatial transcriptomics data. Nat. Methods 19, 567–575 (2022).
Joshi, S. C. & Miller, M. I. Landmark matching via large deformation diffeomorphisms. IEEE Trans. Image Process. 9, 1357–1370 (2000).
Xia, C. R., Cao, Z. J., Tu, X. M. & Gao, G. Spatiallinked alignment tool (SLAT) for aligning heterogenous slices. Nat. Commun. 14, 7236 (2023).
Tang, Z. et al. Search and match across spatial omics samples at singlecell resolution. Preprint at bioRxiv https://doi.org/10.1101/2023.08.13.552987 (2023).
Miller, M., Tward, D. & Trouvé, A. Molecular computational anatomy: unifying the particle to tissue continuum via measure representations of the brain. BME Front. 2022, 9868673 (2022).
Miller, M. I., Trouvé, A. & Younes, L. Spacefeature measures on meshes for mapping spatial transcriptomics. Med. Image Anal. 93, 103068 (2024).
Chen, X., Fischer, S., Zhang, A., Gillis, J. & Zador, A. Modular cell type organization of cortical areas revealed by in situ sequencing. Preprint at bioRxiv https://doi.org/10.1101/2022.11.06.515380 (2022).
Miller, M. I., Trouvé, A. & Younes, L. Geodesic shooting for computational anatomy. J. Math. Imaging Vision 24, 209–228 (2006).
Miller, M. I., Younes, L. & Trouvé, A. Diffeomorphometry and geodesic positioning systems for human anatomy. Technology 02, 36–43 (2014).
Petukhov, V. et al. Cell segmentation in imagingbased spatial transcriptomics. Nat. Biotechnol. 40, 345–354 (2022).
Littman, R. et al. Joint cell segmentation and cell type annotation for spatial transcriptomics. Mol. Syst. Biol. 17, 10108 (2021).
Traag, V. A., Waltman, L. & Van Eck, N. J. From louvain to leiden: guaranteeing wellconnected communities. Sci. Rep. 9, 5233 (2019).
Miller, B. F., BambahMukku, D., Dulac, C., Zhuang, X. & Fan, J. Characterizing spatial gene expression heterogeneity in spatially resolved singlecell transcriptomic data with nonuniform cellular densities. Genome Res. 31, 1843–1855 (2021).
Chen, X. et al. Highthroughput mapping of longrange neuronal projection using in situ sequencing. Cell 179, 772–786 (2019).
Rodriques, S. G. et al. Slideseq: a scalable technology for measuring genomewide expression at high spatial resolution. Science 363, 1463–1467 (2019).
Stringer, C., Wang, T., Michaelos, M. & Pachitariu, M. Cellpose: a generalist algorithm for cellular segmentation. Nat. Methods 18, 100–106 (2021).
Yao, Z. et al. A taxonomy of transcriptomic cell types across the isocortex and hippocampal formation. Cell 184, 3222–3241 (2021).
Nichols, T. & Hayasaka, S. Controlling the familywise error rate in functional neuroimaging: a comparative review. Stat. Methods Med. Res. 12, 419–446 (2003).
Younes, L., Albert, M. & Miller, M. I. Inferring changepoint times of medial temporal lobe morphometric change in preclinical Alzheimer’s disease. NeuroImage Clin. 5, 178–187 (2014).
Paxinos, G., Halliday, G., Watson, C. & Kassem, M. S. Atlas of the Developing Mouse Brain (Academic Press, 2020).
Atlas, D. M. B. Allen Brain Atlas (Allen Institute for Brain Science, USA, 2006).
Kronman, F. A. et al. Developmental mouse brain common coordinate framework. Preprint at bioRxiv https://doi.org/10.1101/2023.09.14.557789 (2023).
Kleven, H., Bjerke, I. E., Clascá, F., Groenewegen, H. J., Bjaalie, J. G. & Leergaard, T. B. Waxholm space atlas of the rat brain: a 3d atlas supporting data analysis and integration. Nat. Methods 20, 1822–1829 (2023).
Chon, U., Vanselow, D. J., Cheng, K. C. & Kim, Y. Enhanced and unified anatomical labeling for a common mouse brain atlas. Nat. Commun. 10, 5067 (2019).
Thompson, C. L. et al. A highresolution spatiotemporal atlas of gene expression of the developing mouse brain. Neuron 83, 309–323 (2014).
Moses, L. & Pachter, L. Museum of spatial transcriptomics. Nat. Methods 19, 534–546 (2022).
Chen, X. et al. Wholecortex in situ sequencing reveals peripheral inputdependent cell typedefined area identity. Preprint at bioRxiv https://doi.org/10.1101/2022.11.06.515380 (2023).
Garin, C. M., Garin, M., Silenzi, L., Jaffe, R. & Constantinidis, C. Multilevel atlas comparisons reveal divergent evolution of the primate brain. Proc. Natl Acad. Sci. USA 119, 2202491119 (2022).
Beauchamp, A. et al. Wholebrain comparison of rodent and human brains using spatial transcriptomics. Elife 11, 79418 (2022).
Liwang, J. K. et al. epDevAtlas: mapping gabaergic cells and microglia in postnatal mouse brains. Preprint at bioRxiv https://doi.org/10.1101/2023.11.24.568585 (2023).
Cheng, S.W., Dey, T. K., Shewchuk, J. & Sahni, S. Delaunay Mesh Generation (CRC Press, Boca Raton, FL, 2013).
Charon, N. & Trouvé, A. The varifold representation of nonoriented shapes for diffeomorphic registration. SIAM J. Imaging Sci. 6, 2547–2580 (2013).
Dupuis, P., Grenander, U. & Miller, M. I. Variational problems on flows of diffeomorphisms for image matching. Q. Appl. Math. 56, 587–600 (1998).
Stellato, B., Banjac, G., Goulart, P., Bemporad, A. & Boyd, S. OSQP: an operator splitting solver for quadratic programs. Math. Program. Comput. 12, 637–672 (2020).
Stouffer, K., Charlier, B. & Trouvé, A.: xIVLDDMMParticle https://github.com/kstouff4/xIVLDDMMParticle (2024).
Stouffer, K. & Younes, L.: xIVLDDMMMesh https://github.com/kstouff4/MeshLDDMMQP (2024).
Clifton, K. et al. Stalign: alignment of spatial transcriptomics data using diffeomorphic metric mapping. Nat. Commun. 14, 8123 (2023).
Acknowledgements
This work was supported by the National Institutes of Health (1F30AG07773601 and T32GM13677 (K.S.); R01EB020062, R01NS102670, U19AG033655, P41EB031771, and R01MH105660 (M.M.); NIH Brain Initiative Grant U19MH114830 (H.Z.); RF1MH124605 (Y.K.)); the National Science Foundation (NSF) (16569 NeuroNex contract 1707298 (M.M.); the Computational Anatomy Science Gateway (M.M.) as part of the Extreme Science and Engineering Discovery Environment (XSEDE Towns et al., 2014), which is supported by the NSF grant ACI1548562; NSF CAREER 2047611 (J.F.); NSF 2124230 (L.Y.)); and the Kavli Neuroscience Discovery Institute supported by the Kavli Foundation (M.M.). We acknowledge Benjamin Charlier for his work on refactoring and improving the efficiency of the particlebased implementation of xIVLDDMM. We acknowledge Fae Kronman for their contribution to the development of the DevCCF atlas used in this manuscript.
Author information
Authors and Affiliations
Contributions
M.M., A.T., and L.Y. developed the mathematical theory behind the manuscript. K.S. and M.M. drafted the manuscript. K.S. and L.Y. generated codes for algorithms described in the manuscript. K.S. created all the figures in the manuscript. M.K., L.N. and H.Z. generated serial MERFISH data. M.A. and J.F. annotated cell types for cellsegmented MERFISH data. Y.K. created the DevCCF atlas analyzed here with the CCFv3. X.C. and M.R. generated and cell typed the BARseq data analyzed in this manuscript, and specified particular cell type markers for accuracy measurement. All authors contributed to the editing of the final manuscript.
Corresponding authors
Ethics declarations
Competing interests
Under a license agreement between AnatomyWorks and the Johns Hopkins University, Dr. Miller and the University are entitled to royalty distributions related to technology described in the study discussed in this. Dr. Miller is a founder of and holds equity in AntomyWorks. This arrangement has been reviewed and approved by the Johns Hopkins University in accordance with its conflict of interest policies. The remaining authors declare no conflicts of interest.
Peer review
Peer review information
Nature Communications thanks Ozgun Gokce and the other, anonymous, reviewers for their contribution to the peer review of this work. A peer review file is available.
Additional information
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary information
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Stouffer, K.M., Trouvé, A., Younes, L. et al. Crossmodality mapping using image varifolds to align tissuescale atlases to molecularscale measures with application to 2D brain sections. Nat Commun 15, 3530 (2024). https://doi.org/10.1038/s41467024478834
Received:
Accepted:
Published:
DOI: https://doi.org/10.1038/s41467024478834
 Springer Nature Limited