1 Introduction

Underwater photogrammetry has been used for three-dimensional reconstruction for many decades (Höhle 1971; Moore 1976). There are many fields of application in archaeology, marine industry, offshore energy production, fishing industry, and environmental tasks. Examples are documentation of underwater historical archaeological sites (Roman et al. 2010; Menna et al. 2018), sunken shipwreck exploration (Korduan et al. 2003), size and volume estimation of fish (Harvey et al. 2003; Dunbrack 2006; Costa et al. 2006), time dependent coral reef growth (Bythell et al. 2001; Guo et al. 2016), and inspection (Tetlow and Spours 1999; McLeod et al. 2014) of oil or gas pipelines, industrial structures, or offshore wind farm foundations.

Besides photogrammetry, several other techniques have been developed for underwater 3D surface reconstruction including laser scanning, ultrasound sensors, and time-of-flight methodology, each with their respective advantages and disadvantages.

Laser scanning systems are used for 3D measurements under water since the 1990s (Tetlow and Allwood 1994; Moore 2001; Tan et al. 2005; Massot-Campos and Olivier-Codina 2014; Bleier and Nüchter 2017), especially for industrial inspection tasks. The main advantage of this technique is the power of the laser beam which can realise a measurement distance of considerably more than ten metres. However, the necessity of merging many datasets reduces the measurement accuracy of moving systems. Several laser scanning systems are commercially available (CathXOcean 2021; Voyis 2021) for underwater usage.

Ultrasound systems provide very coarse 3D measurement results and are mainly used for rough detection over large distances (ARIS-Sonars, Chi 2019; Wang et al. 2020). Time-of-flight systems are commercially offered (3DatDepth 2021), but typically do not provide the accuracy, which is usually expected in photogrammetric measurements.

Optical 3D reconstruction based on stereo cameras and active structured illumination, which can be considered as an extension of photogrammetric measurement techniques, has been increasingly applied for underwater measurement tasks recently. This technique is very close to classical underwater photogrammetry (CUP). The main advantages of CUP are the high potential for measurement accuracy and the possibility of capturing a large area (several square metres) in one scan. The main disadvantage is the effort in preparing the scene (fixing of markers) and the limitation to static scenes. Depending on the extrinsic conditions, CUP may require substantial effort, e. g. if fixing of markers on the seabed is necessary. Another reason for greater effort may be a relative inaccessibility of the location of deployment (for example offshore) which requires greater effort for handling the calibration devices and the sensor.

Typically, greater effort in the calibration procedure leads to greater achievable measurement accuracy. This observation has been made in our own experiments with several 3D scanners for different air applications. However, other influences on the achievable quality of the calibration must be considered. Hence, a measurable correlation between calibration effort and measurement accuracy is difficult to show.

At present, structured light techniques do capture only small regions in one scan. However, there is no preparation effort to the scene, and larger fields of interest can be obtained by precise mosaicking. First experiences with underwater 3D measurement systems using structured illumination have been made in the last decade (Bruno et al. 2011; Bianco et al. 2013; Bräuer-Burchardt et al. 2016). Because of the necessarily strong illumination, the main limitation is the short measurement distance under 1 m and the small measurement field.

Whereas hardware in combination with algorithms determine the physical potential of underwater 3D scanners, correct camera modelling provides the geometric basis of potential 3D measurement accuracy. Here, several approaches have been introduced (see Sect. 2). The most accurate model for underwater 3D measurements based on triangulation of stereo camera rays, is the ray-based camera model (Grossberg and Nayar 2005; Bothe et al. 2010). However, there are some limiting reasons for a wide commercial application of this model. First, the calibration effort is considerable even in air (Bothe et al. 2010) and would be much higher for under water applications. Additionally, epipolar geometry, which is typically used for fast finding of corresponding image points, could not be applied, because this model does not include a single viewpoint (projection centre) through which all rays pass.

Underwater 3D measurement of specific objects is typically a laborious process which requires great effort and time in preparation and operation. Hence, it would be helpful to reduce preparation effort by performing a simple, but accurate and robust calibration of the 3D measurement device. It is known from photogrammetric measurements in the medium range of object distances (several metres), that classical pinhole modelling provides accurate 3D measurement results, also when refraction modelling is neglected. Our motivation was to develop a theory, which allows to use pinhole modelling also for shorter object distances. Using structured illumination systems for underwater 3D measurements, the power of light limits the maximum distance of observation.

To combine the high accuracy potential of the ray-based modelling with the advantages of the common principles and methods of pinhole modelling, we introduce a novel camera modelling concept for underwater 3D measurements. It provides fast data processing and uses structured illumination and disparity images deduced from epipolar geometry. The main idea of this advanced camera modelling is the replacement of the 2D distortion correction by variable principal distance modelling depending on the radial distance of an image point to the principal point. Hereby, a ray-based camera model with some simplification is achieved and will be described in detail in Sect. 3.

The paper is structured as follows: first, an introduction and overview to the state-of-the-art of camera modelling for underwater use is given. Recent publications are referenced concerning underwater 3D surface reconstruction with optical methods, with and without support by structured illumination. Second, advantages and limitations of several underwater camera models are discussed. The next section describes the motivation and advantages of the novel camera modelling. The mathematical theory of the modelling is derived, and the model is stated. Supporting simulations of the theory are described. Finally, an approach for realisation of an optical underwater 3D scanning device and a calibration strategy is given. Section 4 presents the experiments and results obtained by simulations and real underwater measurements obtained by laboratory setups of two devices. Outlook and conclusion complete this paper.

2 Underwater Camera Models and 3D Point Calculation—State-of-the-Art

Advanced geometric camera modelling is the basis of successful application of photogrammetric 3D measurements under water. Hence, this topic was of interest during the first beginnings of underwater photogrammetry (Höhle 1971; Moore 1976). The main difference with respect to applications in air is the camera model and the refraction of the light rays for underwater 3D measurements. The challenges according to refraction consideration can be solved using different approaches:

  • Ray-based camera modelling (Bothe et al. 2010)

  • Pinhole modelling using dome ports (Nocerino et al. 2016)

  • Explicit determination of the ray refraction (Maas 2015; Sedlazeck and Koch 2011; Jordt et al. 2016)

  • Approximation of the actual geometric situation by the pinhole model and distance-dependent distortion correction (Bräuer-Burchardt et al. 2020)

  • Usage of pinhole modelling and a 3D correction function (Bräuer-Burchardt et al. 2020)

2.1 Ray-Based Modelling

Ray-based camera modelling is the most elegant way to realise an underwater camera characterisation, because the same model can be used as was used in air. Ray-based camera modelling is valid for any possible camera description. However, ray-based modelling has some disadvantages, e. g. a greater calibration effort and no commercially available software tools for the pure ray extraction. Hence, this kind of modelling is not common. One of the main reasons is the necessarily laborious calibration procedure to obtain the modelling for every single image ray. An example of a ray-based underwater system for a laser system is given by Palomer et al. (2019).

2.2 Pinhole Modelling Using Dome Ports

The idea of using dome ports as an interface between the interior of the scanning device and the water environment outside is to keep the vision rays straight. I. e., to enforce zero refraction at the air-glass interface as well as at the glass-water interface. Because of the different refraction indices of the several media, this can be only achieved by perpendicular intersection of all vision rays with all media surfaces. The corresponding modelling and realisation can be obtained using spherical glass ports in connection with perfect placement of the projection centre of the camera into the centre point of the spheres of the glass port (see Fig. 1).

Fig. 1
figure 1

Sketch of ray course at dome ports

Full size image

Dome port applications are described e. g. by Menna et al. (2016) and Nocerino et al. (2016). The main advantage is the possibility to use the same camera model as is used in air. The disadvantages are the necessity of perfect mechanical adjustment and the lack of adequate algorithmic correction capabilities if the mechanical adjustment fails. Additionally, the original deviations of the system geometry from perfect pinhole modelling, which are usually compensated by distortion correction functions, are changed by the dome port geometry and must be additionally determined and compensated. The effort for this compensation is not easy to estimate. Since the disadvantages of this modelling predominate, only a few application systems of underwater 3D measurement use dome ports.

2.3 Explicit Refraction Modelling

When refraction of the rays at the interfaces between different media should be considered, the pinhole camera model must be modified. Here, different approaches are possible. In particular, the chosen level of simplification leads to several approaches.

The works of Telem and Filin (2010), Sedlazeck and Koch (2011), Maas (1995), and Jordt et al. (2016) provide solutions while taking the conditions of the underwater environment into consideration. All these works consider refraction at the interfaces between water and glass as well as between glass and air.

2.4 Approximation by Classical Pinhole Modelling

If plane ports are used, classical pinhole modelling for underwater photogrammetric 3D measurements is a-priori erroneous because of the faulty modelling of the vision rays. However, practical results may seem to be sufficiently accurate. To analyse the reason for this, we consider the ray paths depicted in Fig. 2. Because of different intersection points of the rays with the optical axis for different radii, one fixed projection centre leads to reconstruction errors. However, “false” rays used for reconstruction induce only a small error, which can also be compensated by a distortion correction function. A more extensive analysis is given in Sect. 3.

Fig. 2
figure 2

Sketch of ray course at plane port including refraction, not true scale

Full size image

Pinhole modelling for underwater stereo camera calibration with adapted parameters is suggested in several publications, e. g. by Shortis et al.  (2000), Fryer and Fraser (1986), Harvey and Shortis (1998), Costa et al. (2006), Wehkamp and Fischer (2014). Calibration is usually obtained using underwater images. Lavest et al. (2003) provide a solution without underwater measurements.

2.5 Extension of Pinhole Modelling by Refraction Effects

The approach of an extension of the pinhole modelling has been described by the authors in a previous publication (Bräuer-Burchardt et al. 2020). The idea is to approximate the actual course of the refracted vision rays by substituting the classical distortion correction function by variable values for the principal distance, depending on the radial distance of the image point to the principal point.

In this work, this modelling will be refined. Alternative possibilities for algorithmic simplifications are provided, together with an analysis of the expected errors.

2.6 Calibration and 3D Point Calculation

For the subsequent remarks we assume the following situation of an underwater 3D measurement task. A stereo camera pair (in the following also called sensor unit, 3D scanner, sensor, or scanner) will be used for image recording. 3D reconstruction of the observed scene will be obtained by triangulation (Luhmann et al. 2006). Here, we do not distinguish whether structured illumination is used or not, or whether the scene or the scanner or both are fixed or in motion. Structured light is mainly used for better identification of corresponding points in the stereo images.

In principle, the following tasks must be solved for obtaining 3D measurement data from stereo camera images:

  • A-priori: calibration

  • Online: finding point correspondences

  • Online: triangulation

Typically, calibration is the most onerous and time-consuming procedure of these tasks. Algorithms for finding point correspondences may be distinguished depending on the selected camera model and should be chosen accordingly. Triangulation describes the intersection process of two vision rays (from corresponding points in the two stereo cameras) and is a well-known photogrammetric task (Luhmann et al. 2006).

Underwater camera calibration techniques have been extensively described by Shortis (2015, 2019).

3 Approaches for Accurate Measurements by Extended Pinhole Modelling

As already mentioned, using the ray-based camera model for 3D reconstruction from stereo camera images would yield the most accurate modelling. Whereas algorithms for intersection of rays can be easily solved with manageable effort, performing an accurate calibration based on ray-based modelling is technically sophisticated as well as laborious, even in air (Grossberg and Nayar 2005; Bothe et al. 2010). Other models include consideration of refraction effects or, alternatively, conventional pinhole modelling with remaining measurement errors.

Explicit consideration of refraction leads to complicated modelling with uncertainties. Using classical pinhole modelling for underwater photogrammetric 3D measurements on the other hand, is a-priori erroneous because of the false modelling of the vision rays. However, this error might be relatively small and even negligible. With knowledge of the actual ray course, sufficiently accurate approximations by classical pinhole modelling can be achieved. In this section, some approximations will be described in detail.

3.1 Refraction Modelling by Variable Principal Distance

Deviations of the true ray course in a conventional camera from the pinhole camera model are typically modelled by so-called lens distortion functions (Lavest et al. 2003). These functions are 2D–2D-mappings in the image plane. However, this is only an approximation to describe the deviation of the vision rays. Vision rays do not pass through one single point (the projection centre O of the camera) but intersect the optical axis at a certain displacement from O. This displacement depends on the radial distance of an image point to the principal point p0. Consequently, radial lens distortion ∆r is due to variable principal distance c(r). Hence, correct compensation of this deviation from pinhole modelling using 2D distortion functions can achieved only for one plane (with a certain distance D0) in the object space. The relation between ∆r and c(r) (Bräuer-Burchardt et al. 2020) can be expressed by:

$$c\left( r \right) = \frac{{r \cdot c_{0} }}{{D_{0} }} \cdot \frac{{D_{0} + c_{0} }}{{c_{0} r + r + \Delta r}},$$
(1)
$$\Delta r\left( r \right) = r \cdot \left( {\frac{{c_{0} \cdot \left( {D_{0} + c_{0} - c\left( r \right)} \right)}}{{c\left( r \right) \cdot D_{0} }} - 1} \right).$$
(2)

For illustration see Fig. 3.

Fig. 3
figure 3

Distortion vs. variable principal distance in air

Full size image

Considering refraction at the interfaces, this modelling leads to a change of the principal distance for every radial value r. Following this model, which is described in more detail in a previous publication (Bräuer-Burchardt et al. 2020), refraction is taken into consideration, and the ray geometry can be reconstructed. Consequently, 3D measurement values can be obtained by ray intersection of the two stereo cameras.

As a precondition, an orthogonal adjustment of the port-glass to the optical axis of the camera is assumed. Refraction indices of air, water, and glass are known, as well as the thickness (th) of the glass (a-priori measurements) and distance (d) of the camera to the glass (design data or experimental determination).

As described previously (Bräuer-Burchardt et al. 2020), the variable principal distance cw and shift len of the projection centre depend on radial distance r of the points and are obtained by

$$c_{{{w}}} \left( r \right) = \frac{r}{{\tan \left( {\arcsin \left( {\frac{{\sin \left( {\arctan \left( {r/c\left( r \right)} \right)} \right)}}{{n_{w} }}} \right)} \right)}},$$
(3)
$$\it {\text{len}}\left( r \right) \, = {\text{len}}_{{{g}}} \left( r \right) \, + {\text{len}}_{{{w}}} \left( r \right),$$
(4)

with

$${\text{len}}_{{{w}}} \left( r \right) = d\left( {1 - \frac{r}{{c\left( r \right) \cdot \tan \left( {\arcsin \left( {\frac{{\sin \left( {\arctan \left( {r/c\left( r \right)} \right)} \right)}}{{n_{w} }}} \right)} \right)}}} \right),$$

and

$${\text{len}}_{{{g}}} \left( r \right) = {{th}}\left( {1 - \frac{c\left( r \right)}{r} \cdot \tan \left( {\arcsin \left( {\frac{{\sin \left( {\arctan \left( {r/c\left( r \right)} \right)} \right)}}{{n_{{{g}}} }}} \right)} \right)} \right).$$

.

The glass interface causes a shift of the projection centre towards the interface, whereas the water induces a shift of the projection centre to the opposite direction. Experiments have shown the potential and limitations of this modelling (Bräuer-Burchardt et al. 2020). The advantage of this kind of modelling is the possibility to perform the calibration completely in air. The disadvantages are that preconditions must fulfilled, and remaining deviations must be compensated by an additional correction function, which must be heuristically found using underwater image recordings.

Deviation from the orthogonality assumption influences the calibration parameters by a shift of the principal point, adding further decentring distortion, and by a change of the camera orientation by refraction of the principal ray (optical axis). Orthogonality of the glass interface to the optical axis means parallelism of the image plane to the glass interface. This deviation can be described by the angles ψ (pan) and τ (tilt), where ψ describes the angle between the X-axis and τ between the Y-axis of the image coordinate system to the normal of the glass interface plane. A ψ unequal to zero leads to a shift of the principal point in the X-direction and a τ unequal to zero leads to a shift of the principal point in the Y-direction:

$$\begin{gathered} \Delta x = c \cdot \tan \left( {\arcsin \left( {\frac{\sin \psi }{{n_{{{w}}} }}} \right)} \right) \hfill \\ \Delta y = c \cdot \tan \left( {\arcsin \left( {\frac{\sin \tau }{{n_{{{w}}} }}} \right)} \right). \hfill \\ \end{gathered}$$
(5)

The influence on the other rays is analogous. Because of the radially symmetric spread around the principal ray, a non-zero ψ and τ lead to further decentring. Additionally, a deviation of the principal rays leads to a change of the extrinsic orientation between the stereo cameras. The influence on the coordinates of the projection centres (ΔX, ΔY, ΔZ) with respect to the coordinate systems of the cameras is

$$\begin{gathered} \Delta X = \Delta x \cdot \frac{{D_{0} + {{th}} + c_{0} }}{{c_{0} }} \hfill \\ \Delta Y = \Delta y \cdot \frac{{D_{0} + {{th}} + c_{0} }}{{c_{0} }}. \hfill \\ \end{gathered}$$
(6)

These changes have to be transformed into the world coordinate system to get the influence on the basic length which is defined as the Euclidean distance of the projection centres of the stereo cameras.

If it cannot be enforced that ψ and τ are both equal to zero, a possible change of the coordinates of the principal points and the extrinsic orientation must be assumed. This must be checked by a suitable underwater evaluation measurement. Possible evaluation measurements are discussed in Sect. 5.

A suggestion for a calibration procedure and refinement is also given in (Bräuer-Burchardt et al. 2020).

3.2 Methodology Using Advanced Pinhole Modelling

In this section a methodology for an optimal approximation of the perfect ray-based modelling by a simplified approach and a controlled geometric design is introduced. This methodology has four principles:

  • Simplicity: usage of the common pinhole model and assumption of certain (approximately satisfied) conditions such as the perpendicular orientation of the optical axis of the camera to the glass surface of the plane port.

  • Refraction consideration: estimation of the real course of vision rays using known refraction indices, glass thickness, and distance between camera and glass surface.

  • Approximation: the simplified model description should be as close as possible to the real situation.

  • Progressive geometric design: several geometric parameters, which can be changed within a certain range, should be optimised.

Based on these principles, an approach for a novel modelling of stereo camera rigs for underwater 3D surface measurement is introduced.

3.2.1 Simplifications

The aspect of simplicity is related to certain assumptions which are likely to be fulfilled or can be approximated sufficiently exactly, as well as to available software tools for calibration or 3D point calculation. The following simplified assumptions concerning the scanner hardware and actual conditions are made:

  • The material of the plane ports is homogeneous with a constant and known refraction index

  • The distance between the projection centre and the glass surface can be determined sufficiently exactly (in the order of mm)

  • The camera’s optical axis is perpendicular to the glass surface (considerable deviations from this assumption must be explicitly regarded, see Sects. 3.1 and 5)

  • Deviations from the simplified assumptions can be partly compensated by application of distortion functions

  • Differing temperatures between water and air during the calibration process are neglected

The usage of the common pinhole model can be considered a simplification. Corresponding software solutions are available for the calibration process as well as for the reconstruction software. However, to apply this model, which is a-priori not valid for underwater application, it must be guaranteed that deviations are negligible.

3.2.2 Consideration of Refraction Effects

Before the pinhole model can be applied, an analysis of the ray course through the several media with different refraction indices is performed. We consider refraction between the media air, glass, and water and apply Snell’s law (Pedersen et al. 2018). Assuming a perpendicular orientation of the camera lens to the port, refraction leads to a transformation of the vision rays depending on the radial distance r of the image points to the principal point. The effect of the refraction is an increase in the principal distance and a shift of the intersection point of the vision ray and the optical axis (see Fig. 3). These effects depend on the thickness of the glass, the refraction indices of glass, water, and air, and on the radial distance r. Considering the parameters in the context of the pinhole model, a transformation can be obtained of the parameters principal distance c and camera position (projection centre) O by application of Eqs. (1) to (4) according to (Bräuer-Burchardt et al. 2020).

3.2.3 Optimisation of Calibration Parameters

Applying the pinhole model to the image data obtained under water, the following procedure is suggested to achieve improved 3D measurement data with minimal systematic error. First, an average projection centre must be calculated. Using Eq. (4) for the expected radial distance-dependent shift len(r) of the projection centre O(r), the average value

$$\it {\text{len}}_{0} = {{ 1/nS}}\left( {{\text{len}}\left( r \right)} \right),$$
(7)

in a predefined image region can be chosen (r < rmax; rmax may be, e.g., half the amount of the longer sensor edge). Correspondingly, O0 is obtained by shifting O by len0 in the direction of the orientation axis. The average principal distance cw0 should be determined analogously:

$$c_{w0} = {{1/nS}}c_{{{w}}} \left( r \right).$$
(8)

The calculation of a distortion function ∆rD0w(r) for a certain reference distance D0 (corresponding to a plane in the object space) can be obtained according the following equation:

$$\Delta r_{{{w}}} \left( r \right) = r\left( {\frac{{c_{0} \cdot \left( {D + {\text{len}}_{{{w}}} \left( r \right) + {{c}}_{{{{w0}}}} - c_{{{w}}} \left( r \right)} \right)}}{{{{c}}_{{{w}}} \left( r \right) \cdot (D + {\text{len}}_{0} )}} - 1} \right).$$
(9)

This distortion function describes the error of the object points for a certain object distance D according to the selected pinhole modelling. Hence, this error can be corrected by ∆rw(r) if D is known. Consequently, if D is unknown, the reconstruction error is smaller, when the variation of ∆rw(r) is small over the range of valid distances D. By considering the whole measurement volume (MV; defined as the region in the object space providing valid 3D measurement points) the minimal object distance Dmin and the maximal object distance Dmax lead to the changed distortion functions ∆rDminw(r) and ∆rDmaxw(r), according to (9). These functions determine the expected systematic measurement error ∆(X, Y, Z) according to the position (X, Y, Z) of the object point in the MV.

3.2.4 Geometric Design

The essential point of the approximation by the classical pinhole model is the extent of the expected systematic error of the 3D calculation. The calculated distortion functions give a first estimation of the quality of the parameter approximation. For example, if the magnitude of the correction terms of the distortion function changes significantly depending on the object distance, the expected accuracy of a measurement value is low.

In other words, the lower the extent of the necessary correction, the better the approximation by the pinhole model parameters. A reduction of these corrections can be achieved by variation of certain quantities that have a considerable influence on the underwater calibration parameters (e. g. shift of the projection centre).

Reduction of the expected systematic measurement error can be achieved by changing certain calibration parameters in a certain range. For example, the distance of the camera to the glass port can be varied, the refraction index of the glass can be changed by use of a different material, the glass can be made thicker, or the values len0 and cw0 can be manipulated by the user. Section 4.1 will provide some application examples.

The thickness and material of the glass can be varied subject to the necessary strength of the port window. Additionally, the distance between camera and glass can be varied in a certain range depending on the size of the housing box. To get a feeling concerning the effects of parameter variation, corresponding simulations can be performed. The input data for these simulations are

  • Air calibration data including distortion function

  • Glass refraction index ng

  • Glass thickness th

  • Distance d between camera and glass

Simulations for estimation of the expected error should be performed based on Eq. (9). The main criterion for a change of the design should be the differences of the correction coefficients over the object space in the simulations. If the differences of the necessary correction coefficients are small, the design is optimal for the suggested geometric modelling. These values of the correction coefficients are estimated by application of Eq. (9) and will be described in more detail in Sect. 4.1. When a considerable reduction of the systematic error can be achieved with acceptable effort, the design should be changed. Examples for selected simulation results are given in Sect. 4.1.

3.2.5 Approach for Air-to-Water Calibration Transformation

In this section a suggestion for a transformation of an air calibration to an underwater calibration of a stereo scanner is given. The air calibration may be performed with or without the glass of the ports. However, mounting of the glass after calibration risks a mechanical change of the orientation between the cameras.

The input of the transformation procedure is the set of calibration parameters in air including intrinsic and extrinsic parameters of both stereo cameras. In the first step, extrinsic parameters are transformed as follows. Camera positions are transformed according to Eqs. (4) and (7), whereas orientation angles stay the same. It should be noted that the insertion or omission of leng depends on the presence of the glass during air calibration. The intrinsic parameters are changed according to Eqs. (3) and (8), whereas principal points stay the same.

The distortion function is changed according to Eq. (9) using a reference distance D0 in the underwater MV. Additionally, distortion correction can be applied iteratively using Eq. (9) with the estimated true distance D instead of D0.

3.2.6 Calibration Evaluation, Revision, and Refinement

Calibration evaluation will be performed using underwater measurements of a ball-bar and a plane normal. The characteristic quantities (CQ) length deviation ldev and flatness deviation fdev (VDI/VDE 2008, Bräuer-Burchardt et al. 2016) are determined and compared to the values obtained by air measurements. When the deviation from reference values is too great, the calibration may be discarded. Rejection should be initiated by comparison of characteristic quantity values to predefined tolerance ranges.

If the calibration obtained from air-to-water translation is discarded, a new underwater calibration must be performed. This can be realised, e. g., by classical pinhole calibration under water analogously to the air calibration. Subsequently, this new calibration must be evaluated, too.

If characteristic quantities are not yet sufficiently within tolerance ranges, subsequent calibration refinement may be performed. Refinement can be achieved by several strategies, e. g., by parameter optimisation or construction and application of a 3D correction function. Calibration parameter optimisation can be achieved by a randomised algorithm using, e.g., Monte-Carlo search (Metropolis 1987) or deterministically, e. g., by the Downhill-Simplex method (Nelder and Mead 1965). Compensation of systematic 3D errors can be achieved by a 3D correction function (Bräuer-Burchardt et al. 2018), which must be generated and applied.

Refinement by Monte-Carlo search may be performed, e. g., by defining a certain target function obtained by the CQs and random variation of the calibration parameters. Refinement by 3D correction function includes the construction of an appropriate function over the measurement volume. It can be obtained by analysis of ball-bar and plane normal measurements (Bräuer-Burchardt et al. 2018).

4 Experiments and Results

This section concerning experiments and results is separated into two parts. First, simulation results are presented and discussed. The initial calibration parameters for the simulations are obtained by a stereo camera setup with structured light illumination. In the following, this setup will be denoted by UWS. It has a measurement volume of about 1000 mm × 800 mm × 800 mm, a measurement distance of 2000 mm (see Fig. 4) and is designed for deep subsea applications in water depths up to 1000 m. Additionally, measurement data of a previously introduced (Bräuer-Burchardt et al. 2016, 2020) handheld underwater 3D scanner (HUW) were involved in the experiments.

Fig. 4
figure 4

Laboratory setup of underwater scanner UWS for deep subsea application (consisting of two monochrome stereo cameras C1 and C2, projection unit P, and colour camera CC) in water basin

Full size image

Simulation experiments were performed as follows. First, data similar to an air calibration were generated, which are close to the real calibration data of the two real devices. The air calibration was translated according to Sect. 3.2.5 into an underwater calibration. Subsequently, certain parameters were manipulated, and the effects were calculated and plotted.

Experiments on real data include 3D underwater measurements of certain specimens (ball-bar and plane normal) to evaluate the quality of the actual calibration.

4.1 Simulation Results

The simulations consider certain parameters obtained from air calibration and additional underwater parameters such as refraction indices, glass thickness, and distance between camera and port. According to the approximation of the ray geometry by pinhole modelling, a prediction of the expected distortion function depending on the object distance is simulated. Additionally, certain assumptions are made, e. g., an orthogonal adjustment of the port-glass to the optical axis of the camera.

The best approximation of the pinhole model is obtained by application of Eqs. (1) to (4), and an estimation of the average shift len0 of the projection centre can be chosen as

$$\tt \it {\text{len}}_{0} = d\left( {n_{{{w}}} - {1}} \right).$$
(10)

The radial distortion function ∆r is obtained by Eq. (9). The following examples show different simulation results. Figure 5 shows distortion functions for three selected object distances (200 mm, 300 mm, and 400 mm) under assumption of the knowledge of the true parameters for a very short reference distance D0 of 300 mm. Deviations of the error function between different object distances are considerable (5.2 µm difference between D0 = 200 mm and D0 = 400 mm at radius 2 mm and 3.3 µm at radius 3 mm, and 4.2 µm at radius 4 mm). Using the same parameters and a reference distance D0 of 3000 mm, the differences between the error functions reduce significantly (0.2 µm difference between D0 = 2000 mm and D0 = 4000 mm at radius 2 mm and 0.1 µm at radius 3 mm, and 0.9 µm at radius 4 mm—no graphics shown).

Fig. 5
figure 5

Simulated distortion function depending on different (short) distances using true values for d and ng of HUW

Full size image

Using these deviations as input, 3D reconstruction errors using a stereo camera arrangement can be estimated. The error decreases as the magnification increases (factor ten in our example). This means, that the absolute systematic measurement error is approximately the same for short object distances with small measurement field as it is for long distances. Hence, the relative systematic measurement error reduces by factor of ten.

The following simulation shows, how a decrease in error at a short reference distance of 300 mm can be obtained by manipulation of the glass refraction index (1.49 for acrylic instead of 1.76 for sapphire) and a decrease of d from 60 to 24 mm (error reduction to 1.8 µm difference between D0 = 200 mm and D0 = 400 mm at radius 2 mm, 1.1 µm at radius 3 mm, and 1.5 µm at radius 4 mm). In practice, this may be impossible because of the physical dimensions of the lens. However, it impressively shows the effect of error reduction. Figure 6 shows the effects of a shortened lens-to-glass distance and use of acrylic glass.

Fig. 6
figure 6

Simulated distortion function with changed glass refraction index and shortened port distance

Full size image

Simulation results can be used to generate the parameters of the pinhole model for underwater application. Additionally, the effects of the uncertainties of the influencing quantities can be demonstrated. Disadvantages of this process are possible deviations from the assumptions. If the preconditions are not fulfilled, additional compensation functions must be defined and applied. Here, ground-truth data are necessary.

The simulations show that the change in calculated distortion is significantly larger at short distances than at longer distances. However, if certain manipulations concerning glass material and positioning of the camera regarding the port surface are possible, these changes may be reduced considerably (Fig. 6). Hence, application of pinhole modelling might be possible, too, according to the accuracy requirements.

4.2 Real Measurement Examples

Two different devices introduced in the beginning of this section were used to perform underwater 3D measurements. The first device has been described in detail previously (Bräuer-Burchardt et al. 2016, 2020).

After air calibration and transformation of the calibration parameters into underwater parameters according to the suggestions in Sect. 3.2.5, the following experiments were performed. Reference measurements of two kinds of specimen (ball-bar and plane, Fig. 7) were performed at different distances first in air and then under water. Additionally, using sensor UWS, a classical calibration process was performed under water as well. Whereas air calibration including parameter transformation took about one and a half hour, complete underwater calibration took approximately four hours.

Fig. 7
figure 7

Specimen ball-bar and plane normal for UWS

Full size image

To evaluate the quality of the measurements, the characterising quantities length deviation and flatness deviation (VDI/VDE 2008, Bräuer-Burchardt et al. 2016) were determined at several different distances. For underwater measurements, first the calculated calibration parameters were used (Air to water). A second measurement was performed on identical data using refined parameters (Water refined), either obtained from additional underwater measurements (HUW) or from additional pinhole calibration performed underwater (UWS). The following results were obtained. Tables 1 and 2 show the results of the handheld device HUW and Tables 3 and 4 of the second sensor UWS.

Table 1 Length deviation ldev for sensor HUW, all values in mm, n = 1 measurement per method and distance
Full size table
Table 2 Flatness deviation fdev for sensor HUW, all values in mm, n = 1 measurement per method and distance
Full size table
Table 3 Length deviation ldev for sensor UWS, all values in mm, n = 3 (air-to-water) and n = 4 (water refined), respectively, measurements per distance
Full size table
Table 4 Flatness deviation fdev for sensor UWS, all values in mm, n = 4 (air-to-water) and n = 3 (water refined), respectively, measurements per distance
Full size table

Using HUW, the ball-bar had a sphere centre point distance of 199.293 mm and the plane had a size of 250 mm × 250 mm. HUW has glass thickness of 3.0 mm and refraction index of glass of 1.76 (sapphire). Using UWS, the ball-bar had a sphere centre point distance of 251.204 mm and the plane had a size of 800 mm × 150 mm. Reference measurements in air were performed of the ball-bar and the plane normal without distance variation. UWS has glass thickness of 15 mm and refraction index of glass of 1.49 (acrylic).

Flatness error is defined as the difference between maximum and minimum value of a fitted plane without noise in a region on the plane normal of predefined size: 200 mm × 200 mm (HUW), and 600 mm × 100 mm (UWS), respectively. The negative value in Table 4 (Air to water, 1700) represents bending of the plane in the opposite direction with respect to all other measurements.

The examples show greater measurement errors of the underwater measurements with respect to reference measurements in air for both devices. The origins of the remaining errors might be the following:

  • Deviations from the simplified assumptions such as perpendicularity of the camera orientation to the ports

  • Errors in the initial calibrations

  • Remaining modelling errors

Refinement leads to significantly improved results in flatness as well as length measurements for both sensor devices. The main source for the relatively high errors using air-to-water calibration is probably a slight deviation from perpendicularity between optical axis and glass surfaces of the ports; this was deduced from an error source analysis (see next section).

4.3 Error Analysis

The first evaluation results of sensor UWS using the calibration obtained by air-to-water transform showed a very strong flatness deviation error of more than 7 mm. Comparison to underwater pinhole calibration showed a significant deviation of the relative positions of the projection centres of the two cameras. In numbers, the basic length difference was about 26 mm (637 mm vs. 663 mm) and the difference of the triangulation angle of the principal rays was 0.23°. When searching for the origins of these deviations, the orthogonality assumption of the principal rays with respect to the port-glass surfaces was considered. The orthogonality deviation was estimated by comparing the principal point coordinates of the air calibration with those of the water calibration. The influence of orthogonality deviation can be obtained by the principal point shift due to refraction at the interface of the three media according to Eq. (5). Having only air-to-glass and glass-to-air interfaces (sensor in housing), rays only get a parallel shift, even if orthogonality is not exactly fulfilled. In the other case (air-to-glass and glass-to-water), orthogonality deviation leads to refraction of the principal rays.

The detected deviation of 0.23° of the difference of the principal rays between air calibration and underwater calibration was exactly determined by refraction estimation of the principal rays according to Eqs. (5) and (6). This result may confirm the assumption of the origin of the error obtained by simple air-to-water calibration transform due to the suggested method. Consequently, if evaluation measurements yield the large errors such as observed, additional underwater calibration is necessary, because there is no possibility to determine the orthogonality deviation without underwater calibration.

5 Summary, Discussion, and Outlook

A methodology for simplified calibration of underwater 3D scanners based on advanced camera modelling was introduced. The methodology uses air calibration and knowledge of geometric design to get a good initial solution with small errors. Consideration of refraction and approximation of the actual geometry to pinhole modelling complete the technique. Application of the methodology was presented by simulations and real measurements. Possibilities and limitations were shown, and possible extensions of the modelling were discussed to obtain optimised results.

The introduced methodology for underwater 3D measurements using stereo camera rigs provides a simplified way to obtain underwater camera calibration. Ideally, the underwater calibration process can be completely omitted. However, the presented results show, that complete air calibration may provide acceptable, but not perfect results. Hence, refined calibration using 3D measurements under water is typically necessary. However, the presented methodology may significantly reduce the effort of a complete underwater calibration process and provides a prediction concerning the expectation of negligible or considerable 3D measurement errors.

The proposed method does not necessarily provide improved accuracy compared to previous methods. However, it has the potential to reduce the calibration effort under certain conditions. Additionally, it shows some theoretical relations which may help to perform error analysis. It also provides starting points for further improvements. Specifically, the inclusion of the plane port tilt modelling in the novel method and development of an estimation method could improve the technique considerably. If an easy way can be found to quantitatively determine the plane port tilt, then the effort of the calibration procedure under water could be considerably reduced.

Simulation results show that pinhole modelling can be used despite of refraction without considerable loss of accuracy for larger object distances (inverse magnification of 100 or more). Under certain conditions, pinhole approximation can even be used for shorter object distances.

Real measurements showed errors in length and flatness. Correction of the large flatness error was possible by compensation of a weak tilt of the glass ports of about half a degree or less. This value has been confirmed by the additional underwater pinhole calibration. The search for remaining error sources and further error reduction are the main tasks of future work.

It should be verified, whether reduced underwater calibration procedure using just one single view may be sufficient to estimate the extrinsic orientation for prediction of ψ and τ of glass assignment for supplement of the proposed air-to-water calibration procedure.

In future work we should perform more experimental measurements to recognise systematic deviations of the predicted measurement results. Then, detected systematic errors can be used to improve the calibration process and refine the calibration parameters once more.