1 Introduction

The general approach to reconstruct 3D by 2D serial sections (also termed array tomography) is long known and can be applied with various imaging techniques [2, 7, 22, 30]. This method has the common drawback that the images of the serial sections need to be aligned to the image of the adjacent slice. While this can be done manually with various programs (e.g. midas of IMOD, Fiji/ImageJ, VV, Gimp, PhotoShop), this can be very tedious labour. Although visual inspection seems easy, it often is hard to decide which transform is the “best”, one reason being the fact that adjacent images in general contain similar but not equal content due to the structure change in the 3rd dimension. This becomes of particular importance when employing registration allowing local deformations, because the natural 3D structure change is not meant to be corrected by local deformation.

Therefore, various procedures for digital automatic alignment have been investigated, which in general are based on finding a transformation that optimizes a metric (a well defined quantification in contrast to visually “best”). Many types of transformations, metrics and optimizers have been developed of which specific ones need to be chosen depending on the given data and desired results. Apart from the parameters of the transformation that get optimized during the processing, parameters of the chosen optimizer, metric and general ones such as denoising, size/shape of a mask and pyramid resolutions need to be set before the processing can start [4, 10, 32]. These parameters are referred to as meta-parameters and need to be tuned with expert knowledge in order to get an acceptable results for as many consecutive images as possible. The more serial sections the image series contains, the more difficult and time consuming this task can become. Experience shows, that for a series of a few hundred up to a few thousand realistic (i.e. non-ideal) images, the finding of suitable meta-parameters can take a few weeks, without a guarantee to succeed at all.

In order to have a better guarantee to succeed in practice, the procedure described in this paper limits the time consumption to that needed for a pure manual alignment of the whole series, while trying to use automation as much as possible.

2 Method

First Elastix [10], later SimpleElastix [16] was chosen as the framework that provides the means for automated registration. In general other implementation could be chosen, however Elastix (based on ITK [8]) already accepts initial transformations. Even if an initial transformation is provided by manual alignment, it can still enter the automatic optimization and therefore get improved quantitatively as in the default case of pure automated optimization. In other words, if the automated optimization gets trapped in a local optimum i.e. fails to find the global optimum, a manual initial transform provides a different start point for the optimization such that the global optimum is reached.

The work presented here is based on three distinct pieces of software:

  1. 1.

    The Python-implemented registration program recRegStack.py which employs SimpleElastix.Footnote 1

  2. 2.

    Extra programs and commands needed to convert gigapixel slice scans from a Carl Zeiss slide scanner (in CZI fromat) to an image series usable by recRegStack.py.

  3. 3.

    A build and invocation system to apply these to a full-size image series, with adjustments needed for the specific data at hand, using gnuplot [31] and GNU Parallel [29].Footnote 2

Fig. 1.
figure 1

Processing dependencies

Schematic flow graph to visualize the dependencies involved in the iterative process. Images are represented by squares, text-files by ovals. The parameters used by default (dfl. PF, dPF) during the registration (reg.) need to be tuned for the integral image series (2D series) in order to reduce the need for manual interactions. The first image at the start is copied unchanged. The last aligned image (ali. Img) is used as fixed image (fix. Img, fI) for registering the next image from the series (mov. Img, mI). In the case that the default parameters do not yield an acceptable result for an individual image pair, it is possible to supply a manual initial transform (man. iT, mIT) and/or provide individual registration parameters (idv. PF, iPF). Tuning the default parameters (dPF) is the most difficult (time consuming, red) task, adjusting some individual parameters less problematic (iPF, yellow) while creating a manual initial transform (mIT, green) with e.g. midas is easiest. In case some images need to be re-scanned (due to scan-artefacts, defocus, etc), the transform parameter files (tra. PF, tPF) can be used to register the new image exactly the same way (reprod.) or the registration process can be re-initiated to make use of the improved image quality.

Full size image

recRegStack.py takes an Elastix/ITK parameter file (containing the definition of various meta-parameters) into account, which allows changing the default values used by SimpleElastix. The last transformed image is used to register the following one, see Fig. 1.

For the proof of principle, midas of the IMOD package [19] was chosen for manual alignment due to its superior precision and interaction possibilities. Manually created initial transform files (mITs, Fig. 1) will then be taken into account by recRegStack.py when continuing the automated alignment. In addition, it is possible to adjust the meta-parameters for individual image pairs (iPF, Fig. 1) in case the mIT together with the global defaults (dPF, Fig. 1) do not lead to a satisfactory result.Footnote 3 This can happen for example if the fixed image (fI) and the moving image (mI) come from different section bands, possibly differing significantly in focus quality.

Fig. 2.
figure 2

Bands of serial sections of stained lung tissue on glass slides

Some exemplary thumbnails of slide scans with bands of serial sections of lung tissue stained on glass slides. Ranging from good (left, one band well aligned and no significant staining variations) to bad (right, broken bands with unobvious order, staining variations and slice loss due to folds and extending slide border). Lines indicate different glass slide.

Full size image
Fig. 3.
figure 3

Exemplary image pairs

Left column: fixed image, middle column: moving image, right column: Magenta-Green overlay of the image pair (similar to midas). Image pair rows:

1. ideal (no mIT or iPF needed).

2. dirt (mIT but no iPF needed).

3. defocus (no mIT or iPF needed).

4. folds (no mIT or iPF needed).

Metric for 3 without mIT is 34974 and with mIT 20214. (Color figure online)

Full size image

3 Application and Results

The described approach was applied to an image series of about 2600 histological serial section of lung tissue (rat, details can be found in [27]), referred to as K2-dataset. An EM UC7 microtome (Leica, Germany) was used to cut semi-thin sections with a thickness of 1 \({\upmu }\)m connected to bands of about 1 to 20 sections. These bands were placed on 177 glass slides (where possible as a single row) see Fig. 2. After staining with toluidine blue, the slides were digitalized at a magnification of 10X by an AxioScan Z1 (Zeiss, Germany) using a single-channel fluorescence camera with a very low transmission light in order to get greyscale images (in CZI format) with a dynamic range above 8-bits.

The build and invocation system for this image series can be found in http://gitlab.com/romangrothausmann/K2_fibrosis/. This git repository holds references to the raw-data (CZIs) in an annex (https://git-annex.branchable.com/), imports http://github.com/romangrothausmann/CZIto3D as a subtree for local adjustments as needed for the specific data and serves as processing protocol. recRegStack.py from http://github.com/romangrothausmann/elastix_scripts/ is invoked via a docker-image (http://www.docker.com/) containing all the needed libraries to reproducibly register the images. There is a short (down-scaled) image series for testing in tests/recRegStack/.

Fig. 4.
figure 4

Plot of metric values with markers for mIT and iPF

The point densities of mITs and iPFs are visualized with kernel density plots on the negative y-axis (unrelated to metric value, \(\sigma = 10\)). Mean of metric: \(\approx 4100\), Std. Dev.: \(\approx 9300\), some values are outside of the plot range, iPFs (52) are needed less often then mITs (621), mostly in cases of high metric values. The largest interval without any manual intervention (no mITs) is from 305 to 393 even though the default parameter file (dPF) was tuned at different locations (e.g. slice 929, 1266 and 1379). The centre xz- and yz-slice of the result stack (as shown in Fig. 5) are plotted for comparison. Distortions due to alignment drift can be seen, especially in the xz-slice up to slice 500.

Full size image

Fiji [6] was used to roughly mark the centre of each section in thumbnail images of the gigapixel scans (czi2stack/Makefile). These centres were then used to automatically extract the region of each slice as its own image (3000 x 3000 pixel, PNG) with bfconvert [23] (czi2stack/Makefile). In case of broken bands, the ordering implied by the centre marks had to be adjusted to match the physical order (czi2stack/slides/slideOrder.lst, discrepancies often only visible after a registration). This order was then used to register the consecutive slices (czi2stack/Makefile). The mask for registration and the default parameter file (dPF) were adjusted to fit the K2-dataset, applying rigid registration using a “MultiResolutionRegistration” with “AdvancedMeanSquares” metric and “AdaptiveStochasticGradientDescent” as optimizer, see czi2stack/parameterFile.txt for details. Still, 621 mITs (in average every 4th image) and 52 iPFs (in average every 50th image) were needed to align all 2607 images, see Fig. 4. Some exemplary image pairs (good, dirt, defocus, folds) are shown in Fig. 3. Since the registration reconstructs the spatial correspondence in the 3rd dimension, the resulting image stack can then be regarded as a 3D dataset of 3000 x 3000 x 2607 voxel (about 44 GB @ 16-bit), see Fig. 5.

Fig. 5.
figure 5

Volume rendering of the 3D stack

Volume rendered visualization of the reconstructed 3rd dimension of the lung tissue block (sub-extent of 1000 \(\times \) 1000 \(\times \) 2607 voxel). Tissue dark, resin semi-transparent grey (airspaces and blood vessels).

Full size image

The volume in lung samples occupied by tissue is only about 10%–20% [3, 7], so there is about 90%–80% mostly non-correlating texture in image pairs which disturbs the registration process. This is one reason why the registration of serial sections of lung tissue is challenging. A possible countermeasure is to “fill” the non-tissue space with (roughly) correlating data. This can be achieved by first auto-thresholding the image to roughly binarize tissue and non-tissue and then generating a distance map, which is implemented in the branch ot+dm and leads to image pairs as in Fig. 6. However, applying this version of recRegStack.py to the K2-dataset (K2_fibrosis@a5617581) showed, that more mITs are needed.Footnote 4 A reason for this might be that this approach is more sensitive to dirt, which ends up in the tissue segment and therefore causes significant disturbance in the distance transform, see Fig. 6. Another promising approach could be registration based on landmarks generated by SIFT [14, 15], similar to Fiji’s “Register Virtual Stack Slices” [4] but using Elastix/ITK in order to keep the features of manual intervention (mIT and iPF)Footnote 5.

Fig. 6.
figure 6

Exemplary image pair for ot+dm variant Top row: image pair as is (no mIT or iPF needed), fixed image, moving image, Magenta-Green overlay. Bottom row: ot+dm image pair for images on the left (blue-white-red LUT from -50 to 50, blue: air, blood space; red: tissue, dirt). (Color figure online)

Full size image

4 Discussion

The described proof of principle combines automated alignment with manual intervention such that ideally the automation does the whole work but also ensures that “in the worst case” at least the result of a pure manual alignment will be achieved. This comes with a need to balance the two time consuming tasks: Either tuning the meta-parameters for the automated alignment of as many images in the series as possible (total time consumption can be unlimited) or helping the automation with initial manual alignments (total time is limited).

While the current implementation with Makefiles serves as a proof of principle, it can be further improved to a more intuitive and user-friendly program. For example, by incorporating the processing done by various commands into the Python code, as well as avoiding midas and other IMOD-tools or aborting auto alignment in order to provide a manual alignment to continue with. A graphical user interface (GUI) could provide visual feedback (similar to the image viewer geeqie), incorporate the needed midas functionality and offer a “manual intervention” button. In principle, a threshold on the final metric value could be used to automatically trigger the suggestion of a manual alignment.

Another variant of recRegStack.py (branch combT_01) accumulates all former transforms and adds the newly found transform of the processed image pair as in [21]. While the two approaches should yield similar results, accumulation of a few hundred transforms can become problematic but in return can avoid larger differences in case of already similarly recorded image pairs. A third variant (branch reg2tra+prevTra) uses both approaches and chooses the result with the better metric value finally achieved.

Ideally, a (non-destructive) tomogram or some form of markers should be used for guiding 3D reconstruction of serial sections (e.g. constrain local deformation corrections and avoid continuous drift), such functionality is offered by e.g. HistoloZee [1].

recRegStack.py provides the option to ignore some images of the series in case some slices are too distorted for registration or lost during the preparation (czi2stack/lostSlieds.lst). After aligning the next usable image of the series, a reconstruction of the lost slice can be created from the adjacent slices [14].