1 Introduction

The underwater optical image is an important carrier for presenting information vital in exploring, developing, and utilizing marine resources. However, due to the scattering and absorption of light by various suspended particles, plankton, and dissolved salts in water, the quality of the image collected by underwater imaging systems is always unsatisfactory, riddled with degradation problems, such as color distortion, detail blurring or loss, and low contrast (Watson and Zielinski 2013). In practical applications, most underwater detection missions occur in turbid waters such as coasts or lake ports, and the visibility is only tens or several centimeters. The target information captured by traditional imaging equipment is almost invisible. The strong attenuation of optical sensing in complex physical scenes exacerbates the degradation issue, resulting in significantly compromised optical imaging effects. The degraded image will greatly impact the subsequent work of feature information extraction and target recognition detection (Zhong et al. 2012), as shown in Fig. 1.

Fig. 1
figure 1

Degraded underwater images. Li et al. 2019

Full size image

Severely degraded underwater images render existing post-processing methods based on image enhancement techniques ineffective. Meanwhile, specialized imaging devices designed with specific optoelectronic systems are limited in application due to their volume, weight, operation, and cost. Therefore, relying purely on imaging processing or image post-processing methods makes it difficult to achieve clear imaging in turbid water bodies. A novel and applicable approach for solving the problem of underwater image clarity effectively combines front-end optical techniques represented by underwater imaging processing and back-end computational joint imaging represented by underwater image post-processing. This idea inspires underwater computational imaging. Its method is essentially the same as that of the above-mentioned joint imaging idea. Driven by similar demands and goals, it couples optimized optical systems with signal processing techniques to achieve specific imaging functionalities. As an emerging interdisciplinary research field, computational imaging technology has rapidly developed in the past decade, aided by advancements in high-performance optical devices and signal processing techniques. It has become a hot topic with significant development potential and application prospects in optical imaging and image processing (Suo et al. 2015; Mait et al. 2018). However, application research in underwater imaging has not received sufficient attention. Most existing research focuses on extending free-space computational imaging techniques to underwater environments while lacking systematic studies on fundamental issues and critical technologies. The research framework in this area urgently requires further improvement.

To promote the rapid and comprehensive development of underwater computational imaging and more effectively overcome the inherent obstacles in underwater imaging, breaking through the limitations of traditional underwater imaging systems, this article examines the common problems in underwater optical imaging and categorically elaborates on the overview and macro deficiencies of mainstream technologies. Furthermore, it introduces the basic concepts and advantages of underwater computational imaging, integrates the current research status, outlines the mainstream trends in underwater computational imaging technology, identifies its backbone and key elements, and provides a different perspective for researchers to view the development of underwater computational imaging technology. This will contribute to better research and advancement in underwater computational imaging technology.

2 Overview of underwater computational imaging

In underwater information perception and processing represented by visual tasks, high-quality imaging is always a critical task with significant application requirements. As an emerging technology, although underwater computational imaging faces the common problem of strong medium attenuation like traditional underwater imaging technology, it has great research value and development potential in breaking through the application bottleneck due to its multi-cross attributes (Fig. 2).

Fig. 2
figure 2

Underwater computational imaging link

Full size image

2.1 Challenges in underwater imaging

The main obstacles in underwater optical sensing arise from the water medium’s absorption and scattering of light. The scattering effect of particles suspended in water is the primary factor affecting the imaging quality of underwater vision systems. The signal light reflected from the imaging target experiences scattering during transmission to the sensor, leading to blurred or lost target details, called forward scattering. Additionally, the illumination light is scattered by the water before reaching the target, consequently reducing image contrast and resulting in the sensor receiving an aerosol background, known as backward scattering. Moreover, the energy of the target signal light is exponentially attenuated with distance due to medium absorption and scattering, which severely limits the observation distance of underwater imaging systems. Furthermore, light transmission in water exhibits wavelength dependency, leading to widespread color distortion in underwater images. The combined effects of these factors significantly degrade the underwater optical imaging quality.

2.2 Related work in underwater imaging

Two approaches have been employed in researching high-quality underwater imaging. The most direct approach enhances the imaging quality by relying on hardware devices and physical components, known as underwater image processing methods. The main means include distance gating, synchronous scanning, and carrier modulation technology. This method mainly uses the unique time and space characteristics of underwater imaging to design a special photoelectric imaging system that suppresses backscattering. The 'gate technology' blocks the backscattered light and receives the target light, effectively improving the imaging alignment and increasing the imaging distance. However, these methods often rely on the equipment’s precision, have relatively high costs, and only address specific factors causing image degradation. In practical applications, issues such as blurred or lost details and color distortion persist, limiting their universality (Nie and He 2014). The underwater image post-processing method for post-processing images captured by general imaging equipment has become another mainstream idea for current underwater clear imaging because of its low equipment requirements and strong versatility. Currently, underwater image post-processing methods can be categorized into three: non-physical model methods, physical model methods, and data-driven methods. Non-physical model methods directly process underwater images using traditional image processing techniques, resulting in low computational complexity. However, due to the lack of consideration for imaging physics and water attenuation parameters, these methods often yield suboptimal results regarding detail enhancement. They may suffer from significant noise amplification issues (Watson and Zielinski 2013; Zhao et al. 2020). Physical model methods involve modeling the underwater imaging process (Jaffe 1990) and solving ill-posed problems using physical priors to invert the degradation process and achieve better processing results (Liu et al. 2020). However, the assumptions for unknown parameters may not be fully applicable or accurate in complex underwater environments (Guo et al. 2017; Li et al. 2019), limiting their application scenarios. Data-driven methods employ underwater image datasets to train machine learning models, such as deep neural networks, to obtain clear underwater images. These methods achieve better enhancement results and improve the model’s ability to fit complex underwater imaging processes by leveraging the learning from several samples, thereby overcoming the constraints of prior assumptions. However, data quality requirements are relatively high (Han et al. 2018). Importantly, these issues are further amplified in turbid water bodies, where the results obtained from general imaging devices are severely degraded. This significantly limits the performance of underwater image post-processing methods in handling such degraded sample data.

2.3 Underwater computational imaging and its characteristics

As mentioned above, current underwater imaging methods and techniques have advantages and disadvantages. It is difficult to achieve high-quality imaging in environments with strong attenuation solely through imaging or image post-processing methods. Therefore, from an application perspective, it is crucial to effectively couple optical techniques with computational processing technology to overcome the inherent obstacles of underwater light propagation, improve observation distance, and enhance image quality. Underwater computational imaging aligns considerably with this research trend and has gained popularity due to its multi-disciplinary integration advantages. Its scope is extensive. In simple terms, underwater computational imaging treats underwater imaging as an ill-posed inverse problem. It improves forward imaging physical processes (such as structured light scanning, aperture coding, high-dimensional light field, and other techniques) to obtain more dimensional information. It then couples this information with physics-based or data-driven computational reconstruction methods to achieve higher-quality imaging. Changing how information is collected, addressing the problem of missing high-dimensional information such as phase, polarization, and spectrum in traditional imaging techniques based on Fermat’s principle (Mait et al. 2018), and improving information collection ability are critical for underwater imaging, especially in turbid water (Fig. 3).

Fig. 3
figure 3

Overview of underwater computational imaging

Full size image

When solving ill-posed inverse problems using physics-based underwater computational imaging methods, it is necessary to establish a comprehensive physical model that uses high-dimensional information to accurately and comprehensively represent the actual imaging process. In turbid water environments, where dynamic changes occur and numerous highly complex and nonlinear factors exist, promptly solving and optimizing the model is challenging to precisely characterize the imaging process. Additionally, solving inverse problems requires prior knowledge of the imaging system, environment, and target categories (Barbastathis et al. 2019). These issues may limit the generalization of physics-based underwater computational imaging methods. In contrast, data-driven underwater computational imaging methods can theoretically utilize complex nonlinear deep neural network structures for fitting complex imaging processes. By expanding the dataset and incorporating regularization terms, these methods can enhance their generalization ability and obtain nonlinear transformation capabilities sufficient to represent intricate systems. Therefore, they are expected to be more suitable for complex and diverse turbid water environments.

3 Advantages in underwater computational imaging technology

Starting from the practical application demands of underwater computational imaging, this article categorizes the task attributes of existing work into three: 'underwater visibility for detection', 'precision in underwater perception', and 'stereopsis in underwater vision'. At the macro level, these three categories represent a progressive sequence. Therefore, we will introduce and analyze their current research status individually.

3.1 From 'invisible' to 'visible'

Underwater image enhancement is one of the most extensive and representative tasks within the scope of this task attribute. Based on the differences in computational reconstruction methods, we categorize this part of the study into two main classes: physics-based and data-driven approaches, as shown in Fig. 4.

Fig. 4
figure 4

Related work in underwater image enhancement. a Schechner and Karpel 2004; b Treibitz and Schechner 2008; c Spier et al. 2017; d Hu et al. 2021; e Li et al. 2018a; f Li et al. 2018b; g Zhou et al. 2019; h Lyu et al. 2019

Full size image

Physics-based underwater computational imaging methods use specific imaging techniques such as polarization, structured light scanning, and light field to acquire high-dimensional information about the imaging field, thus improving the efficiency of information acquisition. However, these methods still face challenges regarding models and priors in underwater image post-processing. In terms of imaging, Schechner and Karpel (2004) combined polarization imaging techniques for underwater imaging model inversion. Liu et al. (2018) proposed a wavelength-selective polarization imaging technique, which partially addressed the issue of traditional polarization imaging techniques’ failure in turbid water environments. Hu et al. (2021) conducted differences based on the polarization images of two optimal polarization directions and finally realized polarization differential underwater imaging with three degrees of freedom by introducing the weight coefficient of the difference term. Regarding model solving, Treibitz and Schechner (2008) focused on large field-of-view imaging and employed prior knowledge based on an artificial illumination formation model to achieve three-dimensional image reconstruction. Moreover, Spier et al. (2017) estimated water attenuation using multiple backscatter images and solved the model, enabling in-situ targetless calibration. Building upon this, Bekerman et al. (2020) stabilized attenuation estimation by analyzing the color distribution of images, providing new insights for model solving in underwater computational imaging.

Unlike physics-based underwater computational imaging methods, data-driven approaches use nonlinear deep neural network structures to fit complex and dynamic underwater imaging processes. These methods can enhance their generalization ability by expanding the dataset and adding regularization terms. They are more suitable for complex and diverse underwater environments (Zuo et al. 2020). However, there are few reports on existing relevant methods due to the limitations caused by the difficulties in underwater intermediate measurement image acquisition and special data-driven model design. Current research mainly focuses on homogeneous and stable scattering media. Specifically, Horisaki et al. (2016) first applied support vector regression methods to improve imaging quality in scattering media. Inspired by the rapid development and widespread application of deep learning, Li et al. (2018a) proposed the IDFNet deep network model, the negative Pearson correlation coefficient loss function, and PhENN for phase retrieval and lensless imaging. Subsequently, Li et al. (2018b) designed a highly scalable 'one-to-many' network model to enhance the imaging capability under different scattering media. Zhou et al. (2019) introduced an optimization method based on cosine similarity to improve the imaging performance of the model in scattering media. Furthermore, Lyu et al. (2019) proposed a hybrid neural network for reconstructing target information in scattering media.

3.2 From 'visible' to 'subtle'

Within the scope of this task attribute, underwater plankton microscopy imaging is a highly representative category of application research. Tracing the research progress in this field reveals that computational imaging has become increasingly common and critical in this specific domain (Fig. 5).

Fig. 5
figure 5

Related work in underwater plankton microscopy imaging. a Knox 1966; b Malkiel et al. 1999; c Samson et al. 2001; d Davis et al. 2005; e Sun et al. 2007; f Bochdansky et al. 2013

Full size image

Since Knox (1966) proposed the application of holographic microscopy for dynamic observation of marine plankton in 1966, which further advanced the development of holographic microscopy techniques and effectively demonstrated the value of underwater plankton imaging research. Malkiel et al. (1999) utilized scanned and reconstructed holographic images to obtain focused images of particles and developed an in-situ holographic system for recording the spatial distribution of plankton. To improve core elements in the imaging chain, Samson et al. (2001) designed an in-situ oceanic imaging system based on high-speed digital line scan cameras for studying size measurement, identification, quantification, and spatial recording of translucent and opaque particles, capable of continuous long-term imaging. To enhance efficiency in underwater plankton observation, Davis et al. (2005) designed a high-performance plankton recorder instrument (VPRII) by optimizing the field of view and depth of field selection while incorporating data visualization concepts. Sun et al. (2007) developed an underwater electronic holographic camera (eHoloCam), enabling effective freezing of high-speed moving particles and high-resolution sensing, leading to digital reconstruction. Li et al. (2007) used spectral components correlated with the reconstruction kernel to develop a fast focus detection technique based on digital holography, eliminating the need for complete image reconstruction. This technique saves computational resources and significantly improves speed, achieving high accuracy and robustness in the focus detection of plankton holograms. Bochdansky et al. (2013) simplified the core components of the camera system to adapt to different particle size ranges and environments, making it user-friendly for non-experts. They controlled the imaging rate to match the typical speed of oceanographic research vessels, designing a point-source digital inline holographic microscope capable of imaging particles ranging from 50 mm to several millimeters. It is expected to become a standard instrument on CTD rosettes (i.e., basic oceanographic instruments and sampling frames) for permanent water column profiling, applicable for future targeted particle research.

3.3 From 'subtle' to 'comprehensive'

Within the scope of this task attribute, the completeness and stereopsis of the target scene are essential for underwater imaging tasks. Recently, the demand for 3D reconstruction in underwater applications has increased. Therefore, we will discuss the research progress related to underwater computational imaging by focusing on the hot topic of underwater 3D imaging (Figs. 6 and 7).

Fig. 6
figure 6

Related work in underwater 3D imaging (1). a Jaffe 1990; b Chiang and Chen 2011; c Peng and Cosman 2017; d Lai et al. 2015; e Wang et al. 2017

Full size image
Fig. 7
figure 7

Related work in underwater 3D imaging (2). a Narasimhan et al. 2005; b Brandou et al. 2007; c Bruno et al. 2011; d Pinto and Matos 2020; e Bianco et al. 2013; f Chadebecq et al. 2020

Full size image

Building upon the Jaffe-McGlamery model (McGlamery 1980; Jaffe 1990), early researchers conducted several underwater 3D reconstruction studies, mostly in underwater image post-processing. However, the results were relatively limited by the lack of effective priors. For instance, inspired by the dark channel prior (DCP) dehazing algorithm, Chiang and Chen (2011) applied DCP to underwater scenarios and compensated for attenuation differences to address color distortion. Additionally, Peng and Cosman (2017) incorporated image blurring and light absorption priors for depth estimation in underwater imaging. Subsequent works introduced various priors to solve the model, such as color lines (Lai et al. 2015), adaptive attenuation curves (Wang et al. 2017), and blur lines (Berman et al. 2020), further improving the performance of underwater computational imaging.

With the emergence of computational imaging, researchers have made significant progress in coupling optical system design and computational reconstruction to further improve underwater 3D reconstruction. Negahdaripour et al. (2002) optimized the construction of calibration images based on the Jaffe-McGlamery model and designed a recursive 3D reconstruction technique that achieved closed-form solutions for imaging models in specific media. Narasimhan et al. (2005) comprehensively analyzed the application effect of structured lighting in underwater imaging and proposed an imaging method based on photometric stereo vision for removing scattering effects, enabling reliable 3D reconstruction in strongly scattering media. To better achieve the 3D reconstruction of small-scale underwater scenes, Brandou et al. (2007) proposed a novel image acquisition method using a stereovision system for quantitative measurement. This method can generate dense 3D models with texture mapping. To address the challenge of accurately and completely obtaining camera trajectories in underwater environments, Beall et al. (2010) used a wide baseline stereo rig to capture synchronized high-definition videos and construct high-precision sparse 3D reconstruction models, establishing a solid foundation for subsequent dense 3D reconstruction. Bruno et al. (2011) projected structured lighting patterns using a stereovision system and combined them with highly turbid underwater environments to effectively verify the feasibility of introducing active optical techniques for close-range accurate 3D reconstruction. Subsequently, Bianco et al. (2013) conducted comparative experimental studies on active and passive underwater imaging methods, further clarifying the respective advantages of the two technologies and providing important references for subsequent needs analysis. Recently, as the demand for applications increases, more studies have focused on improving the efficiency of underwater 3D reconstruction. Sarafraz and Haus (2016) proposed a non-intrusive underwater structured light-based 3D imaging method based on a single image, which enables 3D reconstruction of dynamic and static scenes. Bodenmann et al. (2017) designed a 3D mapping method based on light sectioning that can capture structure and color from images of a single camera with sufficiently high resolution. Chadebecq et al. (2020) proposed a new dual-view reconstruction method suitable for underwater imaging systems where the incident lens directly contacts the external medium to address distortion issues and improve the applicability of refractive camera models in underwater scenes. Pinto and Matos (2020) introduced an innovative hybrid imaging system (MARESye) that combines active and passive imaging information in a data-driven manner, expanding the measurement range and achieving high-density 3D estimation in dynamic underwater environments. Additionally, this system adopts gate-controlled timing as a driving method, effectively reducing the influence of several photometric issues.

4 Conclusions

Optical imaging technology plays an irreplaceable role in underwater detection. However, the inherent challenges of light transmission in strongly attenuating environments constitute a bottleneck restricting the application of underwater imaging systems. As an emerging interdisciplinary research field, underwater computational imaging technology incorporates mathematical computation into the imaging physics process. Driven by relatively concentrated demands, it focuses on optimizing imaging chain coupling design and enhancing information utilization and interpretability. To some extent, this technology achieves improved imaging resolution, refined target scales, extended detection distances, and reduced optical system volume. It demonstrates significant development potential and application prospects in underwater optical imaging research. We believe that with the continuous development and theoretical improvement of computational imaging technology in underwater applications, the underwater computational imaging system will become more comprehensive and effective. This will enable underwater computational imaging to truly become a demand-driven and goal-oriented imaging technology, better supporting the major strategic needs of the country’s marine industry and blue economy battlefield.