1 Introduction

Generating realizations of random heterogeneous composite materials is a recent field of research. It aims at enabling digital workflows such as numerical simulation and inverse design [1, 2] in order to accelerate materials engineering. Microstructure characterization and reconstruction (MCR) can be used for (i) generating multiple microstructure realizations from a single sample, (ii) reconstructing a small, periodicFootnote 1 structure from a large, aperiodic CT scan, (iii) creating 3D structures from 2D slices and (iv) exploring hypothetic materials by interpolating in the descriptor space. A multitude of approaches has been developed in the last decades that is summarized in different review articles [3,4,5]. A brief introduction is given in the following, whereby a distinction between descriptor and machine learning (ML)-based methods is made.

In ML-based reconstruction algorithms, a generative model is trained on a data set of microstructures. If the data set is sufficiently large, the trained model can be utilized to efficiently generate new realizations of the same structure. A common example for such a generative model is an autoencoder [6, 7]. Similarly, extensive research is dedicated to generative adversarial networks (GANs) [8]. These models, which usually outperform autoencoders in terms of sample quality, have been modified in multiple ways including conditional GANs [9, 10], StyleGAN [11], and gradient penalty [12]. Applications to steels [13] and various other materials [14] show high image quality. To combine the advantage of both models, they have been bundled by employing the decoder simultaneously as a generator. This improves 2D-to-3D reconstruction [15,16,17] and makes the models applicable to extremely small data sets [18]. The disadvantage of ML-based methods is generally the need for a training data set and the difficulty of estimating the model applicability after a distribution shift.

In contrast, descriptor-based methods do not require a training data set but only a single reference. For this purpose, the morphology of the reference structure is quantified using statistical, translation-invariant descriptors like volume fractions or two-point correlations [19]. Microstructure realizations are then generated from given descriptors by solving an optimization problem. The Yeong-Torquato algorithm [19, 20] is potentially the most common method in this regard. It is based on a stochastic, gradient-free optimization algorithm where the search space is restricted to structures of the same volume fractions. The constant volume fractions are ensured by only swapping two different-phase pixel locations in each iteration. Unfortunately, the deficient efficiency and scalability of this stochastic optimization algorithm necessitates subsequent works on efficiency improvements such as multigrid schemes [21,22,23,24], efficient descriptor updates [25, 26] different-phase sampling rules [27], optimized weighting of correlation functions [28], and the swap of multiple interface pixels at the same time [29]. Moreover, efficient reconstructions from Gaussian random field-based methods are used as initialization [30, 31]. Many of these methods and further ideas are discussed in detail in [4]. Successful applications include various materials, e.g., soils [32], sandstone [33], chalk [30] and batteries [26]. As an alternative, gradient-based optimization algorithms can solve the optimization problem in fewer iterations as long as the descriptors are differentiable. This differentiable microstructure characterization and reconstruction (DMCR) idea is presented in [34], extended to 3D in [35] and validated in [36]. An open-source implementation is available in MCRpy [37] and multiple approaches from the literature implement similar ideas [38,39,40]. The drastically reduced number of iterations enables the use of computationally more intensive descriptors like Gram matrices of pre-trained neural networks [41] or descriptors based on a hierarchical wavelet transform [42]. Similarly, the same descriptors can be computed with a higher fidelity [43]. With specific material classes in mind, adapted algorithms have been proposed for certain inclusion, pore or grain geometries. Examples are metallic materials [44, 45], fiber composites [46,47,48] and magnetorheological elastomers [49]. In cases where the morphology is too complex to be described by Voronoi cells, ellipsoids or cylinders, it can still be sensible to simplify the geometry for computational efficiency [50]. Finally, the optimization problem may be solved by a black-box shooting method based on arbitrary descriptors, as long as efficient generators are available [51].

It is worth noting that many microstructure reconstruction algorithms are inspired by the field of texture synthesis [52]. Texture synthesis aims at efficiently generating large images that are perceptually similar to a small reference. For example, in a computer game or an animated movie, it may be used for rendering near-regular or irregular cobblestone images onto a road without directly tiling the same image. As an example from microstructure reconstruction, the Gram matrix-based algorithms [35, 38, 40] are inspired by a well-known texture synthesis algorithm by Gatys et al. [53]. Similarly, the descriptor-based neural cellular automaton method in [54] is inspired by [55]. This is reasonable as the goals of microstructure reconstruction and texture synthesis are very similar. There are, however, some differences. First, texture synthesis results are required to appear similar to humans, whereas microstructure reconstruction results are measured by their effective properties like the stiffness, permeability and fracture resistance. Secondly, reconstructing 3D structures from 2D or 3D data is a very common target in microstructure reconstruction but is never aimed at in texture synthesis. Despite similar algorithms and ideas, these and other differences limit the congruence of the fields.

The present work is motivated by noting the similar goals of texture synthesis and microstructure reconstruction and aims at bridging the gaps between the two fields. In particular, it is noted that the Portilla-Simoncelli algorithm [56], which is very well-known in texture synthesis, shows a very promising potential in microstructure reconstruction. To the authors’ best knowledge, the capabilities of the Portilla-Simoncelli algorithm in the context of microstructure reconstruction have never been explored systematically. To be specific, the contributions of the present work lie in

  • recognizing and explaining that the Portilla-Simoncelli algorithm can be interpreted as a descriptor-based microstructure reconstruction algorithm,

  • discussing the sequential projection-based reconstruction as a fundamental alternative with respect to the classical optimization-based microstructure reconstruction procedures,

  • extending the algorithm to reconstruct 3D microstructures from 2D slices, and

  • validating this expansion by independent descriptor computations as well as the effective elastic and plastic material properties.

The algorithm is introduced in Sect. 2 where the mentioned conceptual differences with respect to typical microstructure reconstruction algorithms are discussed. This comprises a dimensionality expansion for 2D-to-3D reconstruction. Extensive numerical experiments are given in 2D and 3D in Sect. 3. Finally, a conclusion is drawn in Sect. 4.

2 Methods

As a basis, an introduction to common descriptor-based microstructure reconstruction approaches is given in Sect. 2.1. The sequential projection procedure in the Portilla-Simoncelli algorithm is outlined in Sect. 2.2 and contrasted to the minimization-based approach. In Sect. 2.3, different ideas for 2D-to-3D reconstruction are reviewed and a suitable solution for the present algorithm is presented. Finally, the implementation is outlined in Sect. 2.4.

2.1 Descriptor-based microstructure reconstruction

The idea behind descriptor-based microstructure reconstruction is to characterize the morphology of the structure \({\textbf {x}}\) in terms of translation-invariant, statistical descriptors \({\textbf {D}}_i({\textbf {x}})\), where \(i \in \left\{ 1,\ldots ,n_\text {D}\right\}\) enumerates different descriptors, which might themselves be multi-dimensional. Examples for these descriptors are volume fractions, grain size distributions, spatial correlations, etc. Then, microstructure reconstruction is formulated as the inverse process.

In some cases, like Gaussian random fields, a material realization can be directly sampled from given descriptors [57, 58]. In the general case, such a mapping is not known, so microstructure reconstruction is formulated as an optimization problem

$$\begin{aligned} {\textbf {x}}^\text {rec} = \underset{{\textbf {x}} \, \in \, \Omega }{\arg \min } \; \mathcal {L}\left( \left\{ ({\textbf {D}}_i({\textbf {x}}), \, {\textbf {D}}_{i}^{\textrm{ref}}) \right\} _{i=1}^{n_\textrm{D}} \right) \end{aligned}$$
(1)

where the space of possible microstructures \(\Omega\) is searched for the solution \({\textbf {x}}^\text {rec}\), which minimizes the cost function

$$\begin{aligned} \mathcal {L}({\textbf {x}}) = \sum _{i=1}^{n_\textrm{D}} \vert {\textbf {D}}_i({\textbf {x}}) - {\textbf {D}}_i^\textrm{ref} \vert \; . \end{aligned}$$
(2)

The cost function accumulates the error between the desired microstructure and the current reconstruction, which is measured in terms of various descriptor. Descriptor-based microstructure reconstruction algorithms differ in how exactly they parametrize the microstructure \({\textbf {x}}\) and search space \(\Omega\) as well as in how they solve the optimization problem in Eq. 1.

The present work aims at parametrizing structures by general voxel grids. This is given in 2D by

$$\begin{aligned} \Omega ^{2\text {D}} = \Big \{ {\textbf {x}} \in \mathbb R^{K \times L} \,\vert \,&\forall k \in \{1,\ldots ,K\} \; , l \in \{1,\ldots ,L\} \quad \nonumber \\&0 \le x_{kl} \le 1 \Big \} \;, \end{aligned}$$
(3)

and in 3D by

$$\begin{aligned} \Omega ^{3\text {D}} = \Big \{ {\textbf {x}} \in \mathbb R^{K \times L \times M} \,\vert \,&\forall k \in \{1,\ldots ,K\} \; , l \in \{1,\ldots ,L\} \quad \nonumber \\&m \in \{1,\ldots ,M\} \quad 0 \le x_{klm} \le 1 \Big \} \;. \end{aligned}$$
(4)

To solve the optimization problem given in Eq. 1, as discussed in section 1, an error measure is commonly defined in the descriptor space and minimized in an iterative procedure. The global minimum, which may be non-zero especially in the 2D-to-3D case, can in principle be found by the Yeong-Torquato algorithm. However, it requires millions to billions of iterations. Hence, the feasible descriptor fidelity is limited by its computational effort. This restriction is relaxed in DMCR [34], where gradient-based optimization algorithms are used. This method converges significantly faster and requires hundreds to thousands of iterations, although it might converge to a local minimum [43]. In contrast to these algorithms, in the following, a method is presented that reduces the number of iterations even further, leading to a leap forward in resource efficiency.

2.2 Sequential descriptor projection

A radically different approach to microstructure reconstruction lies in regarding the statistical descriptors as constraints

$$\begin{aligned} {\textbf {C}}_{D_i}({\textbf {x}}) = {\textbf {D}}_i({\textbf {x}}) - {\textbf {D}}_i^\textrm{ref} = {\textbf {0}} \quad \forall \; i = 1, \dots n_\textrm{D} \end{aligned}$$
(5)

to to be satisfied exactly instead of descriptors to match as well as possible. Intuitively, each descriptor constraint defines a subregion in the space of possible microstructures. As an example, the volume fraction constraint

$$\begin{aligned} C_\phi ({\textbf {x}}) = \underbrace{\dfrac{1}{KL} \sum _{k=1}^K \sum _{l=1}^L x_{kl}}_{\phi ({\textbf {x}})} - \phi ^\textrm{ref} = 0 \end{aligned}$$
(6)

is a linear constraint. Another example is the normalized total variation, which is defined in 2D as

$$\begin{aligned} C_\mathcal {V}({\textbf {x}}) = \underbrace{\dfrac{1}{KL} \sum _{k=1}^K \sum _{l=1}^L \left( \vert x_{kl} - x_{\tilde{k}l} \vert + \vert x_{kl} - x_{k\tilde{l}} \vert \right) }_{\mathcal {V}({\textbf {x}})} - \mathcal {V}^\textrm{ref} = 0 \; , \end{aligned}$$
(7)

where \(\tilde{k}\) and \(\tilde{l}\) denote the periodic next neighbors of k and l. Using the modulo operator mod, this is expressed as

$$\begin{aligned} \tilde{k} = (k + 1) \; \textrm{mod } \; K \quad \textrm{and} \quad \tilde{l} = (l + 1) \; \textrm{mod } \; L \; . \end{aligned}$$
(8)

The normalized variation can be interpreted as the amount of phase boundary per unit volume. Equation 7 defines multiple hyperplanes because different signs are possible within the absolute operator \(\vert \bullet \vert\). Equivalently, symmetry considerations of pixel permutations show that multiple solutions must be possible. The constraints \(C_\phi\) and \(C_\mathcal {V}\) are shown schematically in Fig. 1 (center) for a structure of two pixels. The intersection of all these spaces defines a subspace of valid microstructures, the feasible region

$$\begin{aligned} \Omega _\textrm{f} = \left\{ {\textbf {x}} \in \mathbb {R}^{K \times L} \, | \, \forall \, i \in \{ 1, 2, \dots , n_\textrm{D} \} \; C_{D_i}({\textbf {x}}) = 0 \right\} \; . \end{aligned}$$
(9)

Herein and in the remainder of this work, it is assumed that the maximum and minimum of the signal are also considered as descriptors with the associated constraints

$$\begin{aligned} C_{x_\textrm{max}} = \underset{k, l}{\textrm{max}} \; x_{kl} - x_\textrm{max}^\textrm{ref} = 0 \end{aligned}$$
(10)

and

$$\begin{aligned} C_{x_\textrm{min}} = \underset{k, l}{\textrm{min}} \; x_{kl} - x_\textrm{min}^\textrm{ref} = 0 \; , \end{aligned}$$
(11)

respectively. This is equivalent to restricting the search space in optimization-based reconstruction in Eq. 3. All structures in \(\Omega _\textrm{f}\) exactly fulfil all the prescribed descriptors. If too few descriptors are specified, then the region is too large, indicating that relevant microstructural features are not captured. If too many descriptors are prescribed, then there might be no intersection at all. Thus, a trade off needs to be found. In the seminal work [56], Portilla and Simoncelli (i) introduced this concept in the context of texture synthesis, (ii) proposed a suitable set of descriptors (therein called texture model) and (iii) developed a robust algorithm to solve this constraint satisfaction problem.

Fig. 1
figure 1

Schematic visualizations of the search space, the constraint sets and the loss function for a microstructure with two pixels, \(x_1\) and \(x_2\). Gradient-based descriptor optimization (left) is compared to the projection-based approach (center, right). The volume fraction and variation constraints are denoted by \(C_\phi\) and \(C_\mathcal {V}\), respectively, whereas \(C_g\) denotes a general non-linear constraint. Note that the translation-invariance of the descriptors manifests itself as a symmetry of the constraint sets with respect to certain pixel permutations. The intersection between the constraint sets defines the feasible region \(\Omega _\textrm{f}\), which is visualized by a green dot. It may be disconnected due to symmetry considerations. A random initialization \({\textbf {x}}_0\) is iteratively projected (yellow arrows) onto the constraint sets until convergence is reached. On the right, the arrow tips for the last steps are omitted because they would occlude the intersection

Full size image

The idea behind the Portilla-Simoncelli algorithm is to project any randomly initialized microstructure to the feasible region. Because multiple complex descriptors are considered, the shape of the feasible region may be too complex to derive a single projection step. The solution is an iterative procedure: For each descriptor \({\textbf {D}}_i\), where \(i \in \{1,\ldots , n_\textrm{D}\}\), a projection is derived that satisfies \({\textbf {D}}_i\) only. The issue of non-uniqueness is handled by choosing the projection that changes the structure as little as possible. These projections are carried out sequentially for all descriptors. This is shown in Fig. 1 (center), where the random initialization \({\textbf {x}}_0\) is first projected to \(C_\mathcal {V}\) and then to \(C_\phi\). In this case, the constraints are orthogonal to each other and therefore the algorithm terminates after one loop through all descriptors. In the general case, however, the projection to enforce a descriptor \({\textbf {D}}_i\) might violate previously satisfied descriptors. This is shown in Fig. 1 (right) for a generic constraint \(C_\textrm{g}\). In this case, the whole procedure is repeated in a loop until it converges to the feasible region \(\Omega _\textrm{f}\), which is defined as the intersection of all descriptor constraints. This stands in contrast to the DMCR method shown in Fig. 1 (left), where all descriptors are jointly considered in a loss function \(\mathcal {L}\), which is minimized iteratively. It is worth noting that the minima of \(\mathcal {L}\) coincide with \(\Omega _\textrm{f}\) and that \(\mathcal {L}\) quantifies the distance to \(\Omega _\textrm{f}\). Portilla and Simoncelli [56] make it very clear that they provide no proof regarding the convergence of their algorithm, but report that all their numerical experiments converged. Furthermore, it can be noted that the choice of descriptors does not only influence the existence of a solution, but also the convergence of the algorithm. The more independent the descriptors are, the fewer steps are required for convergence.

The update rules themselves

$$\begin{aligned} \Pi _{D_i}: {\textbf {x}}^n \mapsto {\textbf {x}}^{n+1} = {\textbf {x}}^n + \lambda \nabla {\textbf {D}}_i({\textbf {x}}^n) \end{aligned}$$
(12)

are defined by gradient projection. Herein, the gradientFootnote 2 of the descriptor error defines the direction for a line search in the microstructure space. The step width \(\lambda\) is chosen such that it satisfies the constraint \(C_{D_i}(\textrm{x})\). Hence, it is defined explicitly as a function of the current solution and the reference descriptor

$$\begin{aligned} \lambda = \lambda ({\textbf {x}}^n, {\textbf {D}}_i^\textrm{ref}) \; . \end{aligned}$$
(13)

This needs to be derived for each descriptor separately and is carried out in the original work by Portilla and Simoncelli [56]. While projections for simple constraints like the volume fraction can be defined easily, other descriptors like the auto-correlation are significantly more challenging to consider. For this reason, the gradient projection for the two point correlation \(\Pi _{S_2}\) and other operators are approximated in [56]. Due to the combination with sufficiently many other descriptors, the algorithm converges in practice.

The descriptors as well as the corresponding projection operators in the Portilla-Simoncelli algorithm are closely linked to the multiscale representation of the signal itself, the steerable pyramid. The steerable pyramid is a multilevel decomposition of an image into P different resolution levels and Q orientations. This is outlined in algorithm 1. A given image \({\textbf {x}}\) is first decomposed into its a high-pass residual

$$\begin{aligned} \overline{{\textbf {x}}}^\textrm{h} \leftarrow f_\textrm{h} * {\textbf {x}} \; \end{aligned}$$
(14)

and low-pass content

$$\begin{aligned} \overline{{\textbf {x}}}^0 \leftarrow f_\textrm{l} * {\textbf {x}} \; \end{aligned}$$
(15)

by convolution with a high-pass and low-pass filter \(f_\textrm{h}\) and \(f_\textrm{l}\), respectively. Therein and in the following, \(\overline{\bullet }\) indicates that the associated quantity is a part of the pyramid. Then, for each pyramid level \(p \in \{1,\ldots ,P\}\), oriented band-pass filters \(f_{\textrm{b}q}\) are applied

$$\begin{aligned} \overline{{\textbf {x}}}^{pq} \leftarrow f_{\textrm{b}q} * \overline{{\textbf {x}}}^{p-1} \quad \textrm{with} \quad q=1,\dots ,Q \; , \end{aligned}$$
(16)

where the index b stands for band-pass and q enumerates the orientations. Based on this, a down-sampled low-pass representation

$$\begin{aligned} \overline{{\textbf {x}}}^{p} \leftarrow f_\textrm{d}( f_{l} * \overline{{\textbf {x}}}^{p-1} ) \end{aligned}$$
(17)

is computed for the next higher pyramid level, where the down-sampling function \(f_\textrm{d}\) reduces the resolution of the image by a factor of two. The resulting representation comprises \(\overline{{\textbf {x}}}^\textrm{h}\), \(\overline{{\textbf {x}}}^{pq} \; \forall \; p \in \{1,\ldots ,P\}, \; q \in \{1,\ldots ,Q\}\) and \(\overline{{\textbf {x}}}^{P}\), but not \(\overline{{\textbf {x}}}^{p} \; \forall p<P\). This makes it a special case of a Laplace pyramid, in contrast to Gaussian pyramids, which are often employed in microstructure reconstruction. More details on the data structure are given in [56, 59].

Algorithm 1
figure a

Steerable pyramid decomposition.

Full size image

Based on this representation, the descriptor set comprises marginal statistics like variance, skewness and kurtosis as well as cross-correlations and central crops of the auto-correlations of the residual \(\overline{{\textbf {x}}}^\textrm{h}\), the oriented sub-bands \(\overline{{\textbf {x}}}^{pq} \; \forall \; p,q\) as well as the partially reconstructed images at each scale. The consequence of computing such descriptors on a steerable pyramid is discussed in the following, whereas the reader is referred to [56, 60, 61] for more details on the definitions as well as the associated projection operators. Because the correlations are computed on the residuals as well as the partially reconstructed images, they resemble phase and surface correlations. Hence, the insufficiency of the two-point correlation function \({\textbf {S}}_2\) in describing complex structures, which is very well-known in the field of microstructure reconstruction, is compensated by surface correlations and cross-correlations. Because all correlations are restricted to a central crop, the number of parameters is comparably small. It should be mentioned that the central cut does not discard long-range information because of multi-level pyramid. This is analogous to the multi-scale descriptors in DMCR presented in [34]. Short-range correlations, which lie within the cutoff radius on the fine scales, are computed with high accuracy. On the coarser scales, the same cutoff radius represents a longer correlation, which is therefore not omitted from the descriptor set but merely computed at a lower accuracy and higher speed.

2.3 Dimensionality expansion

Reconstructing 3D structures from 2D slices requires dimensionality expansion. To the authors’ knowledge, such an expansion has not been proposed yet for the Portilla-Simoncelli algorithm. After briefly discussing 2D-to-3D dimensionality expansions in different microstructure reconstruction frameworks, a solution for the Portilla-Simonceli algorithm is proposed.

GANs A 2D discriminator can be provided with 2D training data or 2D slices of a 3D volume. Hence, training a 3D generator from 2D data is possible for GAN-based architectures [62, 63] as well as for models which combine GANs with autoencoders [17], transformers [64] or other models.

Diffusion models Diffusion models as the new state of the art in image generation are known to be more stable than GANs and yield high-quality results, also for microstructures [65, 66]. Unfortunately, the dimensionality of the model must match the training data [67], so the GAN dimensionality extension method is not applicable here. Simply applying a 2D denoising process on slices in alternating directions creates disharmony and does not converge. As a solution, harmonizing steps were proposed very recently by Lee and Yun [68].

Autoencoders Encoder-decoder-models have been trained in the context of microstructure reconstruction and modified for 2D-to-3D applications. An interesting solution lies in regarding the third dimension as time and handling it by a video model in the latent space [69, 70]. Furthermore, a slice mismatch loss is introduced in [71], however, this approach is not truly 2D-to-3D since at least one 3D training sample is required.

MPS In multi-point statistics, dimensionality extension can be achieved by searching for patterns on each slice and plane [72] or on random slices [73].

Descriptor-based optimization If the microstructure is reconstructed by descriptor-based optimization, the dimensionality extension can be achieved by means of the descriptor computation. This is done for the Yeong-Torquato [20, 30, 74], where descriptor information in various directions and planes determines whether or not a pixel swap should be accepted. Similarly, in the DMCR [35] algorithm, descriptors are computed in all possible slices in different directions. This means that the direction of the gradient is averaged over the three dimensions before it is used in the optimization step.

Proposed solution The Portilla-Simoncelli algorithm as a sequential descriptor-based projection approach is conceptually very similar to the gradient-based descriptor optimization idea of the DMCR algorithm. Hence, a similar dimensionality expansion might be feasible. However, gradient-based projections are performed sequentially instead of the minimization of the accumulated descriptor error. In order to avoid complex interference between the sequential projections and the averaged gradient directions, a simpler alternative is presented.

The simple solution is to sequentially update individual slices for each direction, where the slice update consists of a single pass through all projection operators. This is shown in algorithm 2: For each iteration of the 2D-to-3D expansion, the outer-most loop iterates over the three dimensions. In each dimension, a loop iterates over all slices that are orthogonal to that direction. For each slice, the inner-most loop carries out all descriptor projections as if a 2D structure was reconstructed. The result from the loop over the three spatial directions are averaged and represent the 3D result of that single iteration. A stabilization inspired by a viscous regularization in partial differential equation solvers or by a momentum term in stochastic gradient descent algorithms can be introduced at this point. However, the numerical experiments of the authors suggest that no stabilization is needed. In this context, it is noted that no proof exists why this algorithm should converge to a stable state, however, this is the case in practice. Note that the same applies to the sequential projection of individual descriptors itself in the Portilla-Simoncelli algorithm.

Algorithm 2
figure b

Dimensionality expansion based on the Portilla-Simoncelli algorithm.

Full size image

2.4 Implementation

An open-source implementation of the 2D Portilla-Simoncelli algorithm is taken from GitHub [60]. This code is a Python port of the original Matlab code [61] and is mainly based on Numpy. A difference with respect to [56] lies in the pyramid-based image representation. In contrast to the Matlab code, the Heeger and Bergen steerable pyramid is implemented [59] as explained in [75]. The main difference between the two is that the former is complex, while the latter is real. Although the former is known to yield superior results, especially for relatively ordered patterns [56], the performance is similar in most cases [75]. The grayscale version of the code is used as-is for the 2D examples and modified for the 2D-to-3D dimensionality expansion. For this purpose, the code is restructured and an additional loop is introduced as given in algorithm 2. Since the individual slice updates for each given dimension are independent of each other, the algorithm is accelerated by multiprocessing. The loop over all slices is parallelized using a Pathos pool [76, 77]. This compensates for the single-core nature of the underlying Numpy code and leads to an approximate ten-fold increased speed of the reconstruction process. The reconstruction is carried out on a conventional laptop with a \(12^\textrm{th}\) Gen Intel(R) Core(TM) i7-12800 H CPU. The memory requirements are negligible and the GPU is not used.

The open-source tool MCRpy is used for evaluating the quality of the reconstructed structures. This package is developed and described in former works of the authors [37]. It provides a number of microstructure descriptors that can be computed on 2D microstructure data or on 2D slices of 3D data. The specific settings are given in Table 1.

Table 1 Parameters of microstructure characterization using MCRpy version 0.2.0. Further details on all other parameters are given in [37]
Full size table

MCRpy is also used as a reference in order to compare the Portilla-Simoncelli algorithm to descriptor-based optimization procedures. It allows to reconstruct microstructures from descriptors in an optimization-based manner. Thereby, arbitrary microstructure descriptors can be combined in a loss function to define the optimization problem, which can be solved using a number of algorithms. Therein, the descriptors, loss functions and algorithms can be interchanged in a flexible manner and may also be provided by the user as plug-in modules. In the present work, differentiable descriptors are used, enabling an efficient solution of the optimization problem by gradient-based algorithms, making the reference an example for DMCR [34]. To be specific, the two-point correlation \({\textbf {S}}_2\) [19] and the VGG-19 g matrices \({\textbf {G}}\) [38, 41] are used with a simple mean squared error loss function and the L-BFGS-B algorithm. Similar results could have been achieved with the MCRpy implementation of the Yeong-Torquato algorithm, albeit with a much higher computational cost. The settings are given in Table 2.

Table 2 Parameters of microstructure reconstruction using MCRpy version 0.2.0. Further details on all other parameters are given in [37]
Full size table

3 Numerical experiments

To provide an initial understanding of the capabilities and limitations of the method, the Portilla-Simoncelli algorithm is first applied in 2D to a number of different material classes and compared to algorithms from the literature in Sect. 3.1. Then, the 2D-to-3D reconstruction is investigated in Sect. 3.2 as well as the algorithm’s performance and scalability. Finally, the dimensionality expansion is validated in subsection 3.3 by numerical simulation and homogenization using structures where a 3D reference is available.

3.1 Baseline in 2D

As a first investigation of the capabilities and limitations of the Portilla-Simoncelli algorithm, it is compared to optimization-based microstructure reconstruction from statistical descriptors. For this purpose, Fig. 2 shows a comparison for five different material classes. Therein, the reference structures are taken from [38], where they are released under the Creative Commons license [78]. As a reference reconstruction algorithm, the DMCR implementation in MCRpy [37] is used. Similar results would have been achieved with the Yeong-Torquato algorithm. A visual investigation shows that the results are similarly good as the conventional methods for most materials. This is quantified by microstructure descriptors in Table 3. As expected, optimization-based reconstruction algorithms are best when measured by the descriptors that was used for reconstruction, i.e., reconstructions from \({\textbf {S}}_2\) excel when measured by \({\textbf {S}}_2\) and reconstructions from the Gram matrices \({\textbf {G}}\) excel when measured by \({\textbf {G}}\). The volume fraction \(\phi\) given in Eq. 6 is also listed in Table 3, however it should be mentioned that \(\phi\) is highly correlated to \({\textbf {S}}_2\). Moreover, in practice, the reconstructed, real-valued structure is often binarized with a threshold which is chosen such that the reference volume fraction is matched exactly. Clearly, this and other post-processing methods should be employed in a real-world application. However, the present work is focused on the algorithm itself and therefore presents unprocessed results directly. Comparing the other, independent descriptor in each case confirms that the descriptor errors mostly are in the same order of magnitude for the Portilla-Simoncelli algorithm as for DMCR. The copolymer deserves special attention. It can be seen that the fine, fingerprint-like structure is reproduced by the Portilla-Simoncelli algorithm, whereas it poses a serious challengeFootnote 3 for methods based on \({\textbf {S}}_2\). The quality is inferior to the Gram matrix-based reconstruction, which, however, takes significantly longer than the Portilla-Simoncelli algorithm, especially if executed on CPU. An in-depth analysis of the performance and scalability is carried out in subsection 3.2 in the context of 3D reconstruction.

Fig. 2
figure 2

Comparison of the Portilla-Simoncelli algorithm from [56] with the literature in 2D. The reference micrographs are taken from [38], where they are published under the Creative Commons license [78]

Full size image
Table 3 Mean squared descriptor errors for the reconstructed microstructures shown in Figs. 2 and 5. Descriptor errors of 3D structures are averaged over all slices
Full size table

Pointing out limitations of scientific algorithms is equally important as demonstrating their capabilities. As shown in Fig. 3, the Portilla–Simoncelli algorithm fails to reproduce long-range correlations and higher-order information. In the alloy example, the connectivity information of the grain boundary indicator constitutes high-order information which is often not captured by two-point correlation-based descriptors. However, it should be mentioned that even with higher-order descriptors, the long and thin lines of the alloy structure make it a notoriously challenging case. In contrast, the inability to reconstruct the fiber-reinforced polymer (FRP) might be more surprising to most researchers. The inclusions therein are approximately circular and already a very crude random sequential addition algorithm yields far superior results. The reason again lies in connectivity and long range information. For the PMMA in Fig. 2, the disconnected and approximately convex inclusion shapes pose no challenges to the algorithm. However, at the high resolution and volume fraction of the FRP, this seemingly trivial information becomes high-order and long-range and is therefore beyond the capabilities of the method. In summary, despite the limitations displayed in Fig. 3, a great number of materials is reconstructed accurately as shown in Fig. 2, with a competitive morphological accuracy and a superior performance.

Fig. 3
figure 3

Failure cases of the 2D Portilla-Simoncelli algorithm [56] when applied to microstructure data. Despite seeming unequal and unequally complex, the alloy (a) as well as the fiber-reinforced polymer (FRP) (c) are characterized by connectivity information, which manifests itself in higher-order correlations that are not captured by the method. The alloy is taken from [38], where it is released under the Creative Commons license [78]. The FRP is generously provided by the Institute of Lightweight Engineering and Polymer Technology at the Dresden University of Technology and described in [66]

Full size image

The competitive results of the 2D Portilla-Simoncelli algorithm in Fig. 2 naturally give rise to the following question: Is the good performance rooted purely in the projection-based approach or is the unique descriptor set of correlations in the steerable pyramid also needed? This question can be simply answered by an ablation study, where only \(\phi\) and \({\textbf {S}}_2\) as well as the signal minimum and maximum are used in a projection-based algorithm. In every iteration, \({\textbf {S}}_2\) is first accounted for and only then followed by a volume fraction correction and a final clipping. Although similar ablations are given in the original work by Portilla and Simoncelli [56], it is worth illustrating this aspect in the context of microstructures specifically. The ceramics microstructure is chosen for this ablation, because it can be reconstructed very well from \({\textbf {S}}_2\) alone as shown in Fig. 2. This requires a projection operator specifically for \({\textbf {S}}_2\). In this case, a closed solution for \(\lambda\) in the gradient projection as formulated in Eq. 12 could not be found and instead, an approximation is made. The lengthy expressions for \(\Pi _{{\textbf {S}}_2}\) are given in [56]Footnote 4 and the results are shown in Fig. 4: Although large-scale features quickly form from the initial random noise, it can be seen that the reconstruction quality is not even remotely acceptable. The plot of the temporal evolution shows that the result is not stable, as features appear and disappear after a single iteration. Moreover, even the volume fractions are strongly violated, making the result resemble the carbonate more than the ceramics, which should have been reconstructed. Because the ceramics can be reconstructed very well with DMCR based on \({\textbf {S}}_2\) only, there is no doubt that the projection-based approach using the approximated projection operators is insufficient here. Instead, the Portilla-Simoncelli descriptor set is essential to the success of the overall algorithm.

Fig. 4
figure 4

Ablation study showing temporal evolution of the microstructure using the projection method with conventional descriptors. The two-point correlation, volume fractions and minimum and maximum are corrected for iteratively in this order. Because the correlation projection is only approximate, the algorithm fails to converge and does not yield an acceptable solution

Full size image

With this ablation confirming the relevance of the Portilla-Simoncelli descriptor set, a natural question is how well the Yeong-Torquato algorithm or DMCR might work based on this descriptor. Although implementing this exceeds the scope of this work, the authors assume that classical, optimization-based microstructure reconstruction algorithms based on the Portilla-Simoncelli descriptors might work at least equally well as they currently do. However, the number of iterations of these algorithms is much higher than in the projection-based algorithm. Especially in 3D, the Yeong-Torquato algorithm and DMCR require millions to billions and hundreds to thousands of iterations, while the projection method converges after ten iterations.Footnote 5 To the authors’ best knowledge, there exists no gradient-based or gradient-free optimization algorithm for such high-dimensional search spaces which converges to comparable solutions within ten iterations. Hence, similar wallclock times can only be achieved if the cost per iteration is so much lower than with the Portilla-Simoncelli algorithm that it compensates for the increased number of iterations. In the authors’ opinion, this is currently hard to imagine. However, in view of the weaknesses regarding long-range correlations, an advantage of optimization-based reconstruction algorithms with the Portilla-Simoncelli descriptors might be the possibility to add in long-range information. While the present work proceeds to investigate the 2D-to-3D reconstruction as its central topic, this idea is a promising candidate for future research.

3.2 Performance in 3D

Figure 5 shows the results of the 2D-to-3D reconstruction proposed in algorithm 2. Thereby, the reference microstructures are identical to the 2D benchmarks given in Fig. 2. It can be seen that the structures are largely reconstructed very well. Upon a visual inspection, all materials show plausible structures that resemble the 2D references on all faces and naturally blend together at the edges. Only the copolymer constitutes an exception, as the true length of the fingerprint-like lamellar structures is not reached in 3D, although Fig. 2 shows that the 2D algorithm can in principle recover such structures. To complement this visual impression with numerical values, Table 3 shows the descriptor errors. When comparing the results from the 2D Portilla-Simoncelli algorithm to the proposed 2D-to-3D expansion, the descriptor errors are largely in the same order of magnitude. The 3D result is consistently worse by a factor of \(50 \%\) to \(100 \%\), except for the copolymer, where the previously discussed phenomenon can be clearly observed in the descriptors. It is concluded that the proposed 2D-to-3D reconstruction method can cover most, but not all morphologies that the 2D baseline can reconstruct. The reduction in accuracy is acceptable in face of the additional difficulty of matching orthogonal slices to each other.

Fig. 5
figure 5

Results of the proposed 2D-to-3D reconstruction when applied to the references shown in Fig. 2. Although the quality of the copolymer is reduced when compared to the 2D case, the remaining materials are reconstructed very well

Full size image

To deepen the understanding of the algorithm’s behavior, the temporal evolution of the solution over the iterations is investigated exemplarily by means of the carbonate structure. Figure 6 shows some intermediate solutions as well as the random initialization. It can be seen that the result of the very first iteration can already be identified as the correct material, albeit with considerable noise. The next few iterations (iterations 1 through 3 are shown here) remove this noise and sharpen the microstructural features, while leaving the overall shapes and their positions unchanged. After that, the algorithm quickly reaches a stable state, where the solution varies very little. These visual impressions can easily be quantified by observing the norm of the microstructure updates in each iteration as well as the microstructure descriptors. This is shown in Figs. 7 and 8, respectively.

Fig. 6
figure 6

Temporal evolution of the microstructure solution over the course of iterations (left to right). The result of the first iteration (b) already differs from the random initialization (a) and defines the locations of the main structural features. While the first few iterations notably reduce the noise and increase the result quality (c–d), the solution remains relatively stable after that (e)

Full size image
Fig. 7
figure 7

Convergence behavior of the 2D-to-3D reconstruction algorithm measured by the microstructure updates. The mean squared error (MSE) between each iteration and the previous result is plotted. Although the value does not decrease to numerical precision within 50 iterations, it can be seen that the changes are relatively small after the first ten iterations

Full size image
Fig. 8
figure 8

Convergence behavior of the 2D-to-3D reconstruction algorithm measured by the microstructure descriptors. The mean squared error (MSE) between each iteration and the reference is plotted

Full size image

The high performance of the algorithm requires further discussion along with its scalability. As discussed previously, a major advantage of the algorithm is its high performance, which is rooted in the very few iterations: Even in 3D, only 10 iterations are needed for obtaining high-quality solutions. For comparison, the Yeong-Torquato algorithm as the most well-known descriptor-based reconstruction algorithm requires millions or even billions of iterations, because every single iteration merely changes two pixels. The convergence plot in Fig. 7 would look quite differently, as it would randomly jump back and forth between 0 (for unaccepted swaps) and 2/(KLM) (for accepted swaps). As a compensating factor, it should also be mentioned that the computational cost of an individual iteration is significantly lower. This is because descriptors can be updated after pixel swaps instead of recomputing them from scratch, making the number of iterations insufficient for evaluating performance. Despite this factor, to the authors’ best knowledge, reconstructing complex microstructures with \(256^3\) pixels in less than ten minutes on a conventional laptop as in the present work is not feasible with the Yeong-Torquato algorithm. As mentioned in Sect. 2.1, the DMCR algorithm previously proposed by the authors reduced the number of iterations to hundreds or thousands by leveraging differentiable descriptors for efficient, gradient-based optimization algorithms. Although this significantly reduces the wallclock time, the multiple gradients required by quasi-Newton algorithms and the high memory cost of automatic differentiation necessitate expensive hardware.

The scalability of the method with the number of voxels is not trivial to quantify. It is clear that the descriptors are projected on every slice in every direction, leading to \(\mathcal {O}(K)\) evaluations, assuming \(K \approx L \approx M\). The projections in turn are partially applied in Fourier space, adding a further factor of \(\mathcal {O}(K^2 \textrm{log}^2 K)\). However, it is not straightforward to determine theoretically how the application of the projection operators scales, as some of them require iterative approximations to solutions of equation systems. As an example, the first iteration takes notably longer than the subsequent ones. For this reason, the scalability is determined experimentally as shown in Fig. 9. Assuming that caching effects can be neglected for \(K>128\), the method scales super-polynomially as can be seen in the double-logarithmic plot. As a comparison, DMCR theoretically scales polynomially in 2D and 3D as long as the correlation length is kept constant [34]. In practice, the authors could observe the polynomial scaling of DMCR in 2D, but not in 3D. An explanation might be that the slice-based descriptor computation makes the 3D algorithm memory-bound, while the 2D algorithm is compute-bound. For this reason, the comparison is made purely based on numerical experiments in the following For DMCR based on the Gram matrices, reconstruction times between 100 and 110 minutes on NVidia Quadro RTX 8000 GPUs are reported in the literature [40] for \(200^3\) voxels. The MCRpy implementation is slightly slower: On NVidia A100 GPUs, the authors measured 35 minutes, 1:50 h and 20 h for a resolution of \(64^3\), \(128^3\) and \(256^3\) voxels respectively. For the same resolutions, the 3D Portilla-Simoncelli algorithm requires less than 2, 3 and 10 min, respectively, even on significantly inferior hardware. As can be seen in Fig. 9, even microstructures with a resolution of \(512^3\) voxels are reconstructed in 75 min on a conventional laptop. However, as mentioned in Sect. 3.1, the Gram matrix-based DMCR yields superior results, allowing to trade off efficiency for accuracy. Similarly, Gaussian random field-based approaches outperform the presented algorithm in terms of computational efficiency, but are more limited in the types of morphologies they can generate [57, 79]. Thus, the present work complements the Pareto frontier of reconstruction algorithms by an additional method between direct sampling and gradient-based optimization.

Fig. 9
figure 9

Scalability of the presented 2D-to-3D reconstruction method based on the Portilla-Simoncelli algorithm and DMCR based on Gram matrices, where the latter fails in the high-resolution case due to memory limitations. The computations are carried out on a conventional laptop with a \(12^\textrm{th}\) Gen Intel(R) Core(TM) i7-12800 H CPU for the Portilla-Simoncelli algorithm and an Nvidia A100 GPU for DMCR, respectively

Full size image

3.3 2D-to-3D validation

While the 3D results in subsection 3.2 show the morphological plausibility of the reconstructed structures in terms of microstructure descriptors, a proper evaluation in terms of effective behavior is not possible without a 3D ground truth. For this reason, Fig. 10 shows three validation cases for the 2D-to-3D reconstruction where a true reference is available. The structures are taken from [36, 80], where they are released under the Creative Commons license [78]. Like in that work, the descriptor information for the reconstruction only stems from three orthogonal 2D slices that are extracted from the 3D volume, thus simulating the 2D-to-3D dimensionality expansion workflow while also accounting for anisotropy. A visual inspection shows that the first two examples, the columnar and lamellar spinodoid structures, are reconstructed very well. In contrast, the long-range correlations of the TiFe system poses similar challenges as the 2D structures shown in Fig. 3. This strong difference is not visible in the descriptor errors, which are given in Table 4. This naturally poses the question of what magnitudes of descriptor error are tolerable. Different descriptors have a different sensitivity with respect to different morphological aspects of the structure. For this reason, a very reliable way of deciding if a structure is reconstructed sufficiently well is to compute its effective properties. With a specific application at hand, it is much easier for an engineer to derive a tolerable variation in the effective material properties than a threshold for a number of statistical descriptors. Correlating these two is an important topic for future research, which, however, exceeds the scope of this work.

Fig. 10
figure 10

Microstructure reconstruction results for 2D-to-3D reconstruction using the proposed method based on the Portilla-Simoncelli algorithm. In contrast to Fig. 5, a full 3D reference is available. Three slices are extracted for computing the reference descriptors and no further information is passed to the reconstruction algorithm. Visually good results are achieved for the columnar and lamellar spinodoid structures, whereas the long-range correlations of the TiFe system exceed the capabilities of the method. All reference structures are taken from [36] where they are released under the Creative Commons license [78]

Full size image
Table 4 Mean squared descriptor errors for the reconstructed microstructures shown in Fig. 10. The descriptors are averaged over all slices
Full size table

To compare the reconstruction results with the reference solution in terms of effective mechanical properties, numerical simulations are carried out using DAMASK [81]. Using a simple isotropic elasto-plastic constitutive model for the two-phase materials, the elastic surfaceFootnote 6 and the homogenized, directional yield strength \(\overline{\sigma }_\textrm{y}\) and Young’s modulus \(\overline{E}\) are determined as described in Appendix A. The material parameters are chosen to mimic the \(\beta\)-Ti and TiFe phase for the TiFe system as well as to mimic polyvinyl chloride (PVC) and a void-like much softer phase for the spinodoids. They are given in Table 5.

The elastic surfaces [82], i.e., a 3D visualization of the Young’s modulus for each possible direction of load, for the three exemplary structures from Fig. 10 are shown in Fig. 11. The approximately isotropic characteristic of the TiFe’s elastic response (Fig. 11c) are qualitatively and quantitatively well matched in the reconstructed microstructure. The maximum relative deviation of \({0.6\,\mathrm{\%}}\) underlines this conclusion. For both spinodoid structures (Fig. 11a, Fig. 11b) the highly anisotropic behavior is qualitatively well preserved in the reconstructed microstructures. However, the maximum relative difference is with approximately \({10\,\mathrm{\%}}\) and \({8\,\mathrm{\%}}\) for the columnar and lamellar spinodoid, respectively, significantly higher than for the TiFe structures. The deviations are presumably based on slight morphological differences of the reconstructed microstructures. Especially in the case of anisotropic microstructural features, a slightly reduced connectivity or reduced length of one phase in a certain direction might decrease the stiffness perceptibly. Additionally, the reconstruction is performed to match only one slice of the original microstructure for each Cartesian direction. Assuming a statistically not perfectly homogeneous distribution of the descriptors in these directions, the reconstructions cannot perfectly match the original microstructure and so not their properties.

Fig. 11
figure 11

Comparison of elastic surfaces [82] of original and reconstructed microstructures for the structures shown in Fig. 10

Full size image

The results shown in Fig. 12, where the relative difference of the directional Young’s modulus and the directional yield strength in all three Cartesian directions are presented, seem to support the conclusions drawn from the elastic surfaces. The reconstructed TiFe structure differs only slightly with less than \({0.5\,\mathrm{\%}}\) in both properties in all three directions. The Young’s modulus of the reconstructed columnar spinoid is found to be approximately 5% to 10% smaller compared to the original one. This is possibly due to a slightly decreased interconnection of the stiffer PVC phase in the respective loading direction. In the case of the lamellar spinodoid, the Young’s modulus is found to be approximately \({5\,\mathrm{\%}}\) smaller in the stiff x- and y-direction, whereas it is found to be quantitatively well matched in z-direction. When interpreting the homogenized yield strength of the spinodoid structures, spurious noise in the reconstructed microstructure has to be considered. This noise, as slightly visible in the reconstructed microstructures in Fig. 10, might act as stress concentrator, resulting in a possible decrease of the effective yield strength. This effect could explain the deviation of more than \({-5\,\mathrm{\%}}\) in z-direction, i.e. the direction of the columnar features, for the columnar spinodoid and the deviations of approximately \({-2\,\mathrm{\%}}\) in the x- and y-direction, i.e. the direction of the lamellar features, in case of the lamellar spinodoid. Further effects, resulting from the aforementioned three slice-based reconstruction approach as well as statistical deviations, contribute to minor difference of the here considered, homogenzied properties.

Fig. 12
figure 12

Comparison of homogenized properties of the original (ori) and reconstructed (rec) microstructures from Fig. 10 in terms of their relative difference

Full size image

4 Conclusions and outlook

The Portilla-Simoncelli algorithm from the field of texture synthesis is recognized as a potent microstructure reconstruction algorithm and extended for 2D-to-3D expansion. Revisiting the role of descriptors in microstructure reconstruction, it is noted that they usually serve as a basis for an objective function \(\mathcal {L}\). Commonly, microstructure reconstruction is then formulated as an optimization problem with the aim of finding structure realizations that minimize \(\mathcal {L}\). In contrast, in the Portilla-Simoncelli algorithm, descriptors are regarded as constraints that need to be satisfied. In other words, each descriptor defines a subspace or region in the space of possible microstructures, and the intersection defines acceptable solutions to the reconstruction problem. Consequently, projection operators are derived for each descriptor that (approximately) project any given point in the microstructure space to the closest point in the corresponding descriptor subspace. Sequentially applying these projections in a loop yields an efficient and versatile reconstruction algorithm if suitable descriptors are chosen.

Based on this idea of sequential gradient projection, this work

  1. 1.

    identifies the Portilla-Simoncelli algorithm as a viable solution to the 2D microstructure reconstruction problem;

  2. 2.

    shows applications throughout a number of material classes, discussing successes and failures of the method;

  3. 3.

    provides an ablation by projection-based reconstruction from the two-point correlations, confirming the relevance of the Portilla-Simoncelli descriptor set;

  4. 4.

    develops an extension of the method for 2D-to-3D reconstruction;

  5. 5.

    applies the same to various material classes and

  6. 6.

    finally validates the 2D-to-3D reconstruction by numerical simulation and homogenization using data where a fully known 3D ground truth is available.

By requiring only ten iterations, the method is extremely efficient, which is especially relevant in the 3D case. Concretely, on a conventional laptop, volume elements of \(256^3\) and \(512^3\) voxels are reconstructed in 9 min and 1:15 h, respectively. To the authors’ best knowledge, these low memory and CPU requirements are unparalleled in the context of generic and complex material structures. Moreover, like all descriptor-based algorithms, no training phase or data set is required, since the reference descriptor can stem from a single example or from descriptor interpolation.

Naturally, the method is not a panacea and its limitations are discussed equally transparently as its capabilities. This gives clear ideas for future research efforts. As a first example, the inability of the method to reproduce long-range correlations should be addressed, possibly by means of suitable microstructure descriptors. This leads to the second limitation, namely the need for specific projection operators whenever the descriptor set is altered, reducing the algorithm’s flexibility. In view of the success of the lineal path and cluster correlation functions in microstructure reconstruction, it should be investigated if (approximate) projection operators can be defined for these descriptors. Finally, further improvements of classical descriptor-based reconstruction algorithms like the Yeong-Torquato algorithm might be achievable using the unique descriptor set developed by Portilla and Simoncelli [56].