Keywords

1 Introduction

Structured light has proven to be very useful for 3D scene acquisition. This is due to its high speed, precision and versatility. As such a wide array of related techniques have been developed in the past decades, facilitating everything from high precision metrology to real-time guidance of automation [8].

Structured light uses a calibrated camera-projector pair as shown in Fig. 1. A series of time multiplexed patterns is projected onto the scene, which can be used for matching and triangulation with the camera. This active approach makes correspondence searching much simpler than passive stereo approaches, and is applicable to scenes with poor texturing. A very important application for structured light is 3D scanning of biological materials, especially human tissue. Examples include head tracking for medical motion correction [22], vision guided surgery [18, 23], medical diagnostics [1, 4, 28] and automation in agriculture and farming [7, 21, 25]. While the progress in the field has been impressive, one must understand that many target materials are quite problematic. Indeed, they violate the inherent assumption of direct, diffuse surface reflection that most structured light methods are built on. The Fresnel equations predict that when light transitions from one media to another a portion is directly reflected and another is transmitted into the media itself. In the media the light is scattered one or multiple times until it is absorbed or retransmitted into the environment. The proportion between reflected and refracted light is determined by the specific media’s optical properties. For example only 5–7% of human skin reflectance is direct, the remainder is emitted via subsurface scattering [14]. It is therefore of paramount importance that the effect of this violation on structured light is understood and quantified.

In this study, we show that in general, a structured light scan of biological tissue deviates significantly from reference measurements, even with patterns designed specifically to reduce these effects. A large portion of the error can be predicted with a simple, stochastic linear model based on the incident ray geometry. Scans can then be corrected without the need for advanced pattern strategies or special hardware. We can effectively reduce the error in any structured light scanner applied to biological tissue by as much as factor of two or three.

Our study focuses on three types of biological tissue (fat, muscle and skin) with an emphasis on human applications. However we are using porcine materials as a substitute due to its availability and optical similarity to human tissue [26, 27]. Through empirical study we quantify the error induced in structured light by the biological material’s optical properties. This results in a linear error model based on the view geometry fitted to each method, material combination that can be used to predict and correct the scan error.

Fig. 1.
figure 1

The structured light principle: a number of patterns are projected onto the scene, and images are captured by a camera. Correspondences are determined by different encoding algorithms, and used to triangulate points on the object surface. In this example, 3-step Phase Shifting patterns are shown.

Full size image

2 Related Work

The issue of global lighting effects in the context of structured light has been recognized by many authors, e.g. in the acquisition of a human face reflectance field [6].

In order to reduce these effects, hardware modifications such as polarization have been used [2]. Recent attempts have been to design structured light encoding patterns such that they are less susceptible to global lighting effects. The underlying observation is, that with high-frequent patterns, global lighting effects can be considered constant, and invariant to a spatial shift of the pattern. This allows for efficient separation of the observed light intensities into direct and global light [20]. In Modulated phase Shifting [3], structured light patterns are modulated by means of carrier patterns, such that they become high-frequent in both spatial dimensions, thereby improving their separation power. Micro Phase Shifting [10] makes use of sinusoidal patterns in a narrow high-frequency band, promising robustness to global lighting effects and stable phase unwrapping with an optimal number of patterns. It should be noted, that the decoding process in conventional Phase Shifting methods (e.g. [13]) also implicitly performs direct/global light separation. This is true in particular for high frequency scene coding patterns. Since lower frequency phase unwrapping patterns are affected differently by global lighting effects, this can lead to gross outliers. Hence, the advantage of Micro Phase Shifting is not in higher accuracy, but rather in improved robustness (fewer outliers), and more efficient use of information in the encoding patterns.

A newer approach is unstructured light [5], in which the pattern frequency can be high in both dimensions. However the number of patterns is not ideal, and the matching procedure rather inefficient. For binary encoding methods, exclusively high or low-frequency pattern schemes can be considered robust against different global illumination effects [9].

An approach to compensate for the measurement error in isotropic semi-transparent material caused by subsurface scattering was presented in [16]. Similarly to our approach, this work empirically determines the measurement error and explains it by means of a single variable (the projected light angle), albeit only with a single verification object and structured light method. In [15], a Monte-Carlo simulation of the measurement situation was presented, which gives some insight into the error forming process.

In [11], an analytical derivation of the measurement error is given for the Phase Shifting method. This error model predicts the error to decrease with increased spatial frequency of the pattern. The model does not however take into account the loss of amplitude at higher frequency patterns, which increases noise in the measurement data. Furthermore it requires precise knowledge about the scanned material’s optical properties (extinction coefficient, phase function and index of refraction), all of which can be difficult to find or estimate.

Computer simulations of structured light scans were performed in [19] to benchmark encoding methods with respect to various parameters, and were found to have similar robustness with respect to subsurface effects.

To our knowledge, no study has thus far quantified the amount of error in scans of biological tissue, or provided a means of correcting for it.

3 Statistical Error Model

Our principle assumption is that the error is composed of a deterministic part, which once determined can be subtracted from future scans, in order to improve the accuracy. Previous work gives some hints as to which parameters to include in a statistical error model [11, 16].

Fig. 2.
figure 2

The structured light scan geometry with the parameters of our error model. The surface normal is \(\textit{n}\), view direction \(\textit{v}\), light direction \(\textit{l}\) and the projector-surface distance is d.

Full size image

Considering the scan geometry, as shown in Fig. 2, we include three variables in our error model: the view angle (given by \(\textit{n}\cdot \textit{v}\)), the light angle (given by \(\textit{n}\cdot \textit{l}\)) and the distance from projector to object, d. We then formulate the following error model:

$$\begin{aligned} y = \begin{bmatrix} 1&\textit{n}\cdot \textit{v}&\textit{n}\cdot \textit{l}&d \end{bmatrix} \begin{bmatrix} \upbeta _0 \\ \upbeta _1 \\ \upbeta _2 \\ \upbeta _3 \end{bmatrix}, \end{aligned}$$
(1)

where

  • y is the predicted error in mm,

  • \(\upbeta _{i}\) is a weight,

  • n, v, l and d are shown in Fig. 2.

We also tried including many other variables, including reflected light to view angle and coding direction to normal vector angles. These variables are inspired by the analytical error model of Holroyd [11], but did not explain sufficient variance to justify their inclusion in our model. We also fitted Holroyd’s error model directly, but our linear model provided more explanatory power.

Fig. 3.
figure 3

Structured light patterns used in our experiments. In each case, 12 patterns were used.

Full size image

4 Experiments

In order to gather data for the error quantification we scanned surfaces made of one of three porcine tissue types; fat, muscle or skin. All samples were raw and unprocessed with 8 samples of each type. The samples were placed individually in the scan volume and spanned many view and light angles. Their distance to the projector also varied from approximately \(200\, \mathrm {mm}\) to \(400\, \mathrm {mm}\). Each scan produced approximately \(5 \cdot 10^5\) data points resulting in millions for each tissue type.

In optical metrology it is common practice to prepare optically challenging surface with a spray [12]. This makes the surface optically diffuse while preserving the geometry. The method was used to acquire a ground truth surface to which each scan was compared. Specifically, after each scan the object was sprayed and covered with a thin layer of chalk. Then the reference scan was obtained. While we cannot assume that the chalk coated surfaces to be perfect, we consider them ground truth as they provide very clear contrast with virtually no global illumination. In order to verify that this procedure does not alter surface geometry, we applied two separate layers of chalk to a sample object, and compared the scan result after each layer. The mean signed distance was \(0.037 \, \mathrm {mm}\), indicating that chalk spraying the surfaces does not significantly bias the result. As can be seen in Fig. 4 the effect of chalk spraying is relatively pronounced, increasing reflectance and counteracting the pattern blurring caused by subsurface scattering.

In our experiments, we used four different structured light methods:

Fig. 4.
figure 4

Fine grained binary structured light pattern projected onto various types of tissues. The effect of subsurface scattering is clearly seen the pattern becomes blurred without chalk coating.

Full size image

  • Binary Gray coding [24]: one completely lit and one completely dark image were used to define the binary threshold individually in each camera pixel. The remaining patterns were used to encode \(2^{10} = 1024\) individual lines on the object surface.

  • N-step Phase Shifting was used with 9 shifts of a high-frequency sinusoid of frequency 1/\(76\, \mathrm {px^{-1}}\), corresponding to approximately 1/\(10\, \mathrm {mm}\) on the object surface. Three additional patterns were used for phase-unwrapping [13].

  • Micro Phase Shifting [10] using frequencies in the band \([1/80.00 - 1/70.00]\, \mathrm {px^{-1}}\). These frequencies corresponds to a spatial frequency on the object surface of approximately 1/\(10\, \mathrm {mm}\). Slightly different from [10], the specific values were determined using a derivative free non-linear pattern search.

  • Modulated Phase Shifting [3] with three shifts of a sinusoid of frequency 1/\(76\, \mathrm {px^{-1}}\) (1/\(10\, \mathrm {mm}\) on the object surface). Each of these sinusoids was modulated in the orthogonal direction using a sinusoidal carrier with the same frequency. Three additional patterns were used for phase-unwrapping.

For the sake of brevity these will henceforth be referred respectively to as; Gray, PS, Micro PS and Mod PS. The former two are standard methods of structured light and can be expected to perform very similar to many derived methods. The latter two are state-of-the-art and have been specifically designed to mitigate the effects of global illumination, as described in Sect. 2. As such we asses and compare the progress in the state-of-the-art in terms of counteracting the influence of global illumination. A pattern budgetFootnote 1 of 12 was settled on for each method as it provided a reasonable balance in acquisition time and accuracy. For all Phase Shifting methods, pattern frequency was set so that each period would be approximately \(10\, \mathrm {mm}\) on the object surface. The remaining frequencies needed in micro Phase Shifting were determined using simplex optimization as suggested in the original paper [10]. Figure 3 shows the pattern sequences used in our experiments.

For every sample, we defined a binary mask within which all possible surface points were reconstructed. This ensured that the exact same surface region was used in the evaluation of each method.

The error of each surface point was quantified by determining its signed distance to the corresponding point in the chalk sprayed reference. For Gray code scans we define the corresponding points as being the pair with the smallest absolute normal distance. With the other methods, we compared points using their position in the pixel grid.

5 Results and Discussion

The parameters obtained after fitting the error model to our data are seen in Tables 1, 2 and 3. These shows the estimated parameters as well as the RMS of data compared to the chalk coated reference before and after correction (respectively RMS\(_\text {raw}\) and RMS\(_\text {cor}\)) in units of \(\, \mathrm {mm}\). The two latter were evaluated through a process of leave-one-out k-fold cross validation with 5 partitions. In addition we have also estimated the degree of variance explained, \(R^2\), as well as the P-values for the statistical significance of our model against a constant model. All model dependencies were subject to an analysis of variance (ANOVA) [17].

Table 1. Muscle model estimate and regression quality
Full size table
Table 2. Skin model estimate and regression quality
Full size table
Table 3. Fat model estimate and regression quality
Full size table
Fig. 5.
figure 5

Signed distance (sd) between scan and reference on a single sample of muscle. Top row: before applying the linear correction model. Bottom row: after correction.

Full size image
Fig. 6.
figure 6

Signed distance (sd) between scan and reference on a single sample of skin. Top row: before applying the linear correction model. Bottom row: after correction.

Full size image
Fig. 7.
figure 7

Signed distance (sd) between scan and reference on a single sample of fat. Top row: before applying the linear correction model. Bottom row: after correction.

Full size image

In general the model provides a significant reduction in RMS for all methods with the greatest effect for muscle and skin. It is interesting to note that \(R^2\) is in general relatively low; at best 13% and at worst 0.8%. Such measure might dispute model’s validity, but the statistical test versus a constant model proves otherwise. In all cases we can conclude that our model is statistical significant within almost a 100% confidence interval, as indicated by the P-values tested against a constant model. While this might seem improbably low, bear in mind that the models was estimated using millions of points which assists in obtaining a statistical significant results. The model estimate itself is rather stable, yielding almost the same error measure for every iteration in the cross validation. This is to be expected due to the high number of training samples and the low dimensionality of the model.

It is seen that most methods have a positive intercept, meaning that regardless of measurement conditions the surface seems to be further away from the camera. The Phase Shifting methods are especially affected by this bias. This effect is further amplified under ideal scanning conditions, where view and light angle are approximately perpendicular to the measured surface. Since \(\upbeta _1\) and \(\upbeta _2\) are in most cases positive it will further add to positive surface bias. It is also interesting to note that for Phase Shifting methods distance adds a negative weight. This means that distant measurement will effectively have less of a positive bias than close ones. The worst bias can be observed in standard Phase Shifting applied to skin were error can climb to approximately \(0.75\, \mathrm {mm}\).

This positive trend can be illustrated by visualizing the per point error as a heat map upon an obtained point cloud, an interesting trend can be observed. Figures 5, 6 and 7 shows the signed error on a single sample visually before and after applying the correction model. All have a positive bias which is very strong for muscle and skin. This alludes to a general trend, subsurface scattering causes the estimated surface to lie further away from the scanner. This is intuitively correct as subsurface scattering is caused by light entering the material for a bit before it is reflected.

In all cases the application of the proper linear model reduces the error’s RMS significantly. With a relatively low reduction for fat and a high reduction for skin and muscle. Skin seems to be especially interesting for application as it has the highest error RMS and also receives the largest reduction from error prediction. The remaining unmodeled variance can probably be attributed to variance in chalk thickness, material inhomogeneity and slight vibrations in the recording environment.

6 Conclusion

Structured light is greatly affected by the optical properties of biological materials such as subsurface scattering. By comparing structured light scans of a biological object with scans of the same object covered with a thin chalk layer, we have successfully quantified the resulting error. Our study shows a general positive bias resulting in a surface that lies further away from the scanner than an identical diffuse surface. Due to this positive bias, the RMS of the error can be as high as \(0.54\, \mathrm {mm}\). We described the error by fitting a stochastic linear model based on view geometry to the obtained data. Using it, a large portion of the error can be predicted and compensated for. For instance, applying this model to Phase Shifting scans of skin reduces error RMS from \(0.54\, \mathrm {mm}\) to \(0.17\, \mathrm {mm}\).

As opposed to the solutions to global illumination proposed in [3, 10] our approach requires no specially designed pattern strategy or hardware. It can simply be applied directly to the obtained geometry. Additionally our methodology can be applied to any given structured light method and subsurface scattering material. From a pragmatic view, one must conclude that standard Phase Shifting is the superior choice for scanning biological tissue. Not because it shows the lowest error, but rather because the error can be predicted well and compensated using our method.