Introduction

Coral reefs and the vital ecosystem services they provide are increasingly threatened by human activity (Hughes et al. 2018). Establishing baselines and monitoring the effectiveness of management and restoration efforts through ecological surveys is paramount to the success of coral reef conservation. Therefore, there is a need for methodologies that can provide accurate characterisation of coral reef benthic assemblages in a timely and reproducible manner.

In the early days, classification of coral reef benthos was based on data collected manually by divers; however, image-based survey methods rapidly became more common as they allow larger areas to be surveyed using less in-water time and fewer operators, and they provide a permanent visual record of the survey site, which can be re-analysed when required (Jokiel et al. 2015). Analysis of benthic imaging datasets requires that the organisms and substrates encountered are identified, and the images annotated; however, a number of issues are associated with annotation of coral reef imaging data. First, imaging datasets can be very large, to the point that years of analysis would be required if relying on manual annotation alone (González-Rivero et al. 2014). To overcome this bottleneck, computer vision and machine learning methods have been applied to coral reef benthic datasets in an effort to automate the image annotation process (Mehta et al. 2007; Pizarro et al. 2008; Stokes and Deane 2009; Beijbom et al. 2012, 2015; Shihavuddin et al. 2013).

A second issue is that for some groups of benthic organisms, identification from photographs is problematic even when performed by experts (Ninio et al. 2003; Beijbom et al. 2015). The lowest classification accuracy is often found within algal groups, particularly between turf and crustose coralline algae (CCA) (Beijbom et al. 2012, 2015; Mahmood et al. 2016); this has led some studies to treat algae as a single group for the purposes of classification, or to group CCA with abiotic substrates (Soriano et al. 2001; Stokes and Deane 2009; Beijbom et al. 2016). From a functional perspective, however, these organisms are widely distinct. For example, CCA contribute to reef accretion (Goreau 1963; Bak 1976), produce settlement cues for coral larvae (Morse et al. 1988; Heyward and Negri 1999) and are often the main substrate-forming species in high-energy environments such as intertidal reef crests (Tracey et al. 1948). CCA are also sensitive to a number of anthropogenic stressors, including sedimentation, warming and ocean acidification (Fabricius and De’ath 2001; Anthony et al. 2008; Jokiel et al. 2008; Kuffner et al. 2008). In contrast, non-calcifying benthic algae can proliferate and outcompete corals in response to nutrient enrichment, can inhibit coral recruitment and are associated with reduced coral cover and calcification (Loya 2004; Fabricius 2005; Kuffner et al. 2006; D’Angelo and Wiedenmann 2014). The ability to accurately classify algal groups in imaging surveys is thus of relevance for monitoring and management of reef ecosystems.

A number of imaging techniques have been developed to improve the identification and automatic classification of functional benthic groups, many of which rely on knowledge of the optical properties of reef organisms and abiotic substrates (Strand et al. 1997; Gleason et al. 2007; Treibitz et al. 2015; Chennu et al. 2017). One example is multispectral or hyperspectral imaging, where reflectance images acquired in specific wavebands are used to facilitate classification (Gleason et al. 2007; Chennu et al. 2017). This approach can also be implemented from airborne remote sensing platforms (Andréfouët et al. 2004; Kutser et al. 2006; Garcia et al. 2018); however, specialised and often expensive imaging equipment is required. As an alternative approach, some studies have focused on imaging endogenous fluorescence of reef organisms to complement (or replace) reflectance imaging data (Strand et al. 1997; Mazel et al. 2003; Treibitz et al. 2015; Beijbom et al. 2016). Endogenous fluorescence is a promising target for functional imaging of coral reefs in situ because functionally distinct groups of organisms produce optically distinct sets of fluorescent pigments. Field studies have shown that targeting these pigments can enable or even improve automatic classification compared to reflectance imaging alone (Mazel et al. 2003; Beijbom et al. 2016). Hard corals and other cnidaria accumulate high concentrations of Green Fluorescent Protein (GFP)-like proteins with cyan, green or orange-red emission bands (Matz et al. 1999; Wiedenmann et al. 2000, 2002; Dove et al. 2001; Oswald et al. 2007; Alieva et al. 2008), while algae predominantly emit fluorescence in the red emission band of chlorophyll-a. Importantly, different groups of algae present distinct excitation spectra due to coupling of accessory pigments such as chlorophyll-b (green algae), chlorophyll-c (brown algae) or phycobiliproteins (phycoerythrin (PE) and phycocyanin (PC) in red macroalgae, CCA and cyanobacteria) (French and Young 1952; Myers and Kratz 1955; Falkowski and Kiefer 1985). While being linked to their genetic background, the fluorescence of many reef organism has been shown to respond to physical drivers, most prominently to light (Falkowski and Kiefer 1985; Salih et al. 2000; D’Angelo et al. 2008; Gittins et al. 2015; Bollati et al. 2017, 2020; Smith et al. 2017; Quick et al. 2018), making it a promising physiological marker of environmental change (D’Angelo et al. 2012; Zawada and Mazel 2014).

So far, efforts to include fluorescence data in benthic imaging surveys have used monochromatic or narrow-waveband blue light to excite fluorescence in multiple emission bands, separated either by interference filters on a multispectral imager (Strand et al. 1997; Mazel et al. 2003) or by the intrinsic colour sensitivity of an RGB camera (Treibitz et al. 2015; Beijbom et al. 2016). With the former method, Mazel et al. (2003) were able to separate CCA from other photosynthetic organisms using the orange emission band of PE; however, their imaging system was a highly specialised laser line scanner (Strand et al. 1997), making this method not easily accessible to many end-users in the conservation and management sector. The latter method used off-the-shelf professional photography equipment but only allowed discrimination between coral GFP and chlorophyll fluorescence (Treibitz et al. 2015; Beijbom et al. 2016).

In this laboratory-based study, we present a novel imaging method to target fluorescent pigments expressed by functionally distinct groups of benthic marine organisms. The method is entirely based on off-the-shelf components in order to ensure maximum reproducibility and accessibility to a variety of end-users. We propose that acquiring fluorescence images in two excitation (blue and green) and two emission (green and red) channels can provide spectral separation between GFP-like proteins, chlorophyll and PE, and that this information can be used to inform algorithms for automatic classification. We hypothesise that the two emission channels can discriminate between coral FPs and algal photosynthetic pigments, while the addition of two excitation channels can provide differentiation between functional groups of algae based on their dominant accessory pigment. We use a proof-of-concept configuration to image a range of coral reef organisms, and evaluate the concept in a laboratory setting under tightly controlled conditions.

Materials and Methods

Imaging system configuration

Two high-power LEDs (Luxeon Rebel in Royal Blue, peak wavelength (λ) = 450 nm, full width at half maximum (FWHM) = 40 nm, and Green, λ = 530 nm, FWHM = 60 nm. Lumileds, San Jose, CA, USA. Figure 1a) were soldered onto single-diode star printed circuit boards (Bergquist, Chanhassen, MN, USA), mounted onto a 2.5 cm anodised aluminium heat sink (Ohmite, Warrenville, IL, USA), and ran at 350 mA using a constant current driver (PowerLED, Aldermaston, UK). The LED modules were fitted with plastic connectors to enable alternate coupling of each LED into an optical fibre connected to a 66 mm microscope ring light (Leica Microsystems, Milton Keynes, UK), which was mounted onto the objective of a fluorescence microscope (MZ10 F, Leica Microsystems, Milton Keynes, UK) (Fig. S1). For reflectance images, a photonic cold light source (Leica Microsystems, Milton Keynes, UK) was used with the same fibre and ring light. The LED light sources produced a photon irradiance of 49 μmol photons m−2 s−1 (450 nm LED) and 28 μmol photons m−2 s−1 (530 nm) when the microscope was focused on the surface of a 180° quantum sensor attached to a PAR light meter (Li-Cor, Lincoln, NE, USA). Relative irradiance spectra of the LEDs were measured with a USB 4000 modular spectrometer (Ocean Optics, Dunedin, FL, USA).

Fig. 1
figure 1

Spectral properties of the MEF imaging system. a Relative spectral intensity of excitation light sources (dashed lines) and transmission spectra of emission filters (solid lines) used for microscope image collection. b The MEF image acquisition and processing pipeline. c Relative spectral sensitivity of the R, G, and B channel of the microscope camera CCD sensor

Full size image

Sample origin

Fronds of Codium sp. (Stackhouse 1797) and Solieria chordalis (C. Agardh) J. Agardh 1842 were collected from the intertidal shore in Portland, UK, frozen and maintained at − 20 °C before processing. Live corals and algae were obtained from the UK ornamental trade and cultured long term in the Coral Reef Laboratory coral reef mesocosm at the National Oceanography Centre, Southampton, UK (D’Angelo and Wiedenmann 2012), or provided by the Scripps Photobiology Group (Scripps Institution of Oceanography, UCSD) and the Birch Aquarium at Scripps (La Jolla, CA).

Algal pigment extraction

Frozen fronds were homogenised with a pestle and mortar in 0.1 M sodium phosphate buffer (pH 7.4). For extraction of Codium lipid-soluble pigments, 0.5 mL saturated MgCO3 solution was added to the mortar to prevent pheophytinisation (Strickland and Parson 1972). Pigments were extracted in 90% acetone at − 20 °C for 1 h, and debris was removed by centrifugation (3 min, 2000 rcf, 4 °C, twice). For water-soluble pigment extraction from Solieria chordalis, homogenisation was followed by 5 min sonication on ice in 30 s bursts; the extract was then clarified by centrifugation (3 min, 2000 rcf, 4 °C, twice, followed by 45 min, 20,000 rcf, 4 °C). All samples were frozen at − 20 °C before further analysis.

Coral GFP-like protein expression and purification

Fluorescent proteins amilFP497 and amilFP597 from Acropora millepora (Ehrenberg 1834) (D’Angelo et al. 2008) and EechRFP from Echinophyllia echinata (Saville-Kent 1871) (Alieva et al. 2008) were produced in a bacterial expression system as described by Wiedenmann et al. (2002). For amilFP497 and amilFP597, the clarified supernatant was used for imaging. EechRFP was purified from the clarified supernatant and desalted as described by Bollati et al. (2017); partial photoconversion was induced by placing the purified protein under a 412 nm LED (Aquaray NUV, TMC, London, UK) for 10 min (Wiedenmann et al. 2004; Bollati et al. 2017).

Fluorescence spectroscopy

Excitation/emission spectra of purified or extracted pigments were measured with a fluorescence spectrophotometer (Cary Eclipse, Agilent, Santa Clara, CA, USA) in a 10 mm quartz cuvette. For pigment characterisation, all emission spectra were collected using 450 nm excitation to match the narrow-waveband blue excitation light of our imaging system, and excitation spectra were measured across the entire emission spectrum (zero order) to match the broad detection range of our camera sensor.

Excitation/emission spectra of live corals and algae were measured by coupling the spectrophotometer with a fibre optic probe (D’Angelo et al. 2008); one excitation and one emission spectrum were collected for each specimen imaged with the microscope camera set-up. Emission/excitation wavelength used for measurement is presented in Table 1.

Table 1 Spectral properties of extracted and purified pigments and live organisms. λ = peak wavelength (nm), em = emission, ex = excitation
Full size table

Microscopic imaging

Images of purified pigments in a micro-well plate and of live organisms were taken with a CCD camera (DFC 420 C, Leica Microsystems, Milton Keynes, UK) through the fluorescence microscope (Fig. S1). Live organisms were kept in seawater, while the objective remained dry. The wheel-mounted emission filters used were a long-pass UV filter for reflectance images (cut-on 420 nm), a GFPplus filter for 450 nm images, and a Texas Red filter for 530 nm images (Fig. 1a); filter transmission spectra were measured with a UV–Vis spectrophotometer (Cary 50, Agilent, Santa Clara, CA, USA).

Extracted/purified pigments were imaged with an exposure time of 6 s, as were all live algae and the brown morph of M. foliosa; all other corals were imaged with an exposure time of 980 ms. Images for background correction were collected using the same exposure times; an empty well was imaged as background for purified pigments, while a black nonreflective material was used for live organisms.

Compact camera imaging

Macroscopic images were taken with a compact camera (TG4, Olympus, Tokyo, Japan) in aperture priority mode, mounted on a tripod; illumination was from the same ringlight system, fixed ~ 20 cm above the subject. Organisms were placed in seawater while the camera remained dry. Emission filters were held in front of the camera lens; plastic long-pass filters (Nightsea, Lexington, MA, USA) were used to image purified pigments in a 96-well plate. The wider-angle image of an assemblage of tropical benthic organism was obtained using gel emission filters (#10 Medium Yellow and #19 Fire, Rosco, Stamford, CT, USA. Fig. S2) to avoid distortion produced by the convex plastic filters. A well containing water was used for background correction of the purified pigment image, and an area of the tank free from organisms was used for background correction of the reef assemblage image.

FluorIS imaging

Images were taken with a 5D Mark-II camera (Canon, Tokyo, Japan), modified to increase far-red sensitivity and mounted on a framer as described by Treibitz et al. (2015). Illumination was provided by LED modules also mounted on the framer (Luxeon Z, Royal Blue and Cyan—λ = 505 nm, FWHM = 40 nm. Lumileds, San Jose, CA, USA). Gel emission filters (#10 Medium Yellow and #19 Fire, Rosco, Stamford, CT, USA) were held in front of the Sigma 20 mm/ 1.4 lens with filter holders. The spectral properties of these components (Fig. S3) matched those used with the microscope and compact camera, with the exception of the 530 nm LED which was replaced by a 505 nm one. All images were acquired with aperture f = 3.5, shutter speed 1/30 s and ISO800. A square of white card placed within the frame was used for background correction for both images. Organisms were imaged in seawater (with the exception of macroalgae samples, which were placed on a piece of black cardboard in air) while the camera remained dry.

Image processing

To convert 450 nm and 530 nm image pairs into R’G’B’ images, each 8-bit RGB image was first split into the 3 colour channels. The R and G channels of the 450 nm image and the R channel of the 530 nm image were background corrected by subtracting the mean pixel intensity of the corresponding background images from every pixel, then merged into a new 8-bit RGB image (the R’G’B’ image).

Pixel intensity of extracted/purified pigment images in each channel was measured using software Fiji (Schindelin et al. 2012) and compared to calculated estimates (see Supplementary Methods) using a Pearson’s product-moment correlation test.

Classification rules were manually defined after plotting the R’/G’, R’/B’ and G’/B’ ratios of purified pigments in a 3D space. The rules were selected to be exclusive, so it would not be possible for a pixel to be classified as more than one category. The background value was set to 20 for all channels and all categories. The classification algorithm was implemented in MATLAB.

Images obtained with the compact camera and the FluorIS were registered using the ImageJ plugin StackReg (Thévenaz et al. 1998) to compensate for the convex emission filter and for any small movements of the camera. FluorIS R’G’B’ images were further normalised to the highest pixel value in order to brighten the images without modifying the ratios between channels.

Results

The multi-excitation fluorescence (MEF) imaging and image processing concept

The MEF imaging system is designed to acquire two separate fluorescence images for each subject, one under blue and one under green excitation. The excitation light sources are two narrow-waveband, high-power LEDs with peak wavelength at 450 nm (blue) and 530 nm (green) (Fig. 1). The blue excitation (450 nm) images are acquired through a yellow long-pass emission filter with a cut-on wavelength of ~ 510 nm, and the green excitation (530 nm) images through a red long-pass filter with a cut-on wavelength of ~ 610 nm (Fig. 1a, b); the filters are designed to minimise bleed-through of excitation light, either direct or backscattered by the objects imaged. For the initial proof-of-concept study, we coupled the excitation lights into the illumination system of a fluorescence microscope and used the microscope CCD camera for imaging (Fig. S1). Due to the spectral sensitivity of the RGB camera sensor (Fig. 1c), the 510 nm filter excludes the majority of wavelengths that would produce signal in the B channel, as does the 610 nm filter for both B and G channels (Fig. 1a, c). All meaningful fluorescence signal from both images can thus be stored as a single three-channel image; therefore, we processed the image pairs acquired with 450 nm and 530 nm excitation as pseudocolour RGB images and defined the new colour space as R’G’B’ (Fig. 1b), where:

  • R’ = 450 nm excitation, R channel.

  • G’ = 450 nm excitation, G channel.

  • B’ = 530 nm excitation, R channel.

We included a channel-specific background correction step to account for differences in the bleed-through of each light/filter combination.

Fluorescence of purified marine pigments

To evaluate the ability of the MEF system to detect fluorescence spectra that are relevant for reef surveys, we isolated and characterised marine fluorescent pigments representative of different benthic categories. The key pigment groups were represented by acetone-extracted chlorophylls from Codium sp., aqueous phycobili protein extracts from S. chordalis, and purified recombinant proteins of major FP colour classes (amilFP497 as cyan/green cnidarian fluorescent protein (CFP/GFP), amilFP597 as red cnidarian fluorescent protein (RFP), and eechRFP as cnidarian photoconvertible red fluorescent protein (pcRFP)) (Fig. 2, Table 1).

Fig. 2
figure 2

Spectral characterisation and MEF imaging of isolated marine pigments. Excitation (Ex, emission = zero order) and emission (Em, Excitation = 450 nm) spectra with peak wavelength in nm; pigments imaged with the CCD camera under 450 nm (510 nm emission filter) and 530 nm excitation (610 nm emission filter), barplot of pixel intensity in the R’, G’, and B’ channels, and the resulting combined R’G’B’ images

Full size image

Extracted chlorophyll presented a single emission peak at 652 nm, with excitation maxima at 659 and 665 nm and lowest excitation values between 500 and 550 nm (Fig. 2, Table 1). These values are indicative of the presence of both chlorophyll-a and b, with the latter being more efficiently excited by the 450 nm measuring light thus causing an apparent hypsochromic shift in the emission spectrum compared to the excitation spectrum (Jeffrey et al. 1997). The emission spectra of the aqueous red alga extracts presented a single peak at 572 nm and excitation maxima at 497 and 564 nm, confirming that the extract contained mostly PE (French and Young 1952). Spectra of Acropora millepora FPs presented excitation/emission maxima at 479 nm/493 nm for amilFP497 and 559 nm/587 nm for amilFP597; eechRFP spectra showed contributions from a green and a red species, with peaks at 507 nm/516 nm and 571 nm/579 nm, respectively (Fig. 2, Table 1).

We then used the MEF system to image aliquots of the same purified pigments under 450 nm and 530 nm excitation and processed the images acquired into R’G’B’ format (Fig. 2). Imaging of chlorophyll extract and amilFP497 produced signal almost exclusively in the R’ and G’ channels, respectively; PE, amilFP597, and eechRFP produced signal in all three channels, with variable intensity (Fig. 2). The pixel intensity measured from R’G’B’ images showed a strong significant correlation with estimates of R’G’B’ intensity that were mathematically derived from the excitation/emission spectra (Pearson’s product-moment, r13 = 0.88, p < 0.001), indicating the suitability of the R’G’B’ format to convey spectral information (Fig. S4).

Definition of classification rules

We then explored the possibility of using the R’G’B’ image format to automatically classify benthic organisms into categories based on the dominant fluorescent pigment present. As it was not practical to obtain a dataset large enough to train and test a computer vision model with our prototype imaging system, we adopted the approach of defining a set of a priori classification rules using the R’G’B’ intensity measured for extracted/purified pigments (Fig. 2).

To define the set of rules, we first described each pigment by its R’G’B’ pixel intensity ratios R’/G’, R’/B’, and G’/B’ (Fig. 3). In this colour space, chlorophyll was strongly separated from the other pigments by its much higher R’/B’ and R’/G’ values; amilFP497 was characterised by G’/B’ >  > 1, and R’/G’ <  < 1; PE, amilFP597, and eechRFP clustered together due to similar R’/B’ values between 1 and 2, but with some separation in the R’/G’ and G’/B’ dimensions (Fig. 3). Based on these values, we arbitrarily defined regions in the R’G’B’ ratio colour space for broad classification into three categories: Chl, CFP/GFP, and RFP/PE (Table 2, Fig. 3).

Fig. 3
figure 3

R'G'B' ratios of purified pigments and of categories defined. Data points represent values measured for extracted/purified pigments (Fig. 2), polygons represent boundaries of categories. See online Supplementary Material for a 3D version of this figure

Full size image
Table 2 Rules for classification of R’G’B’ images. Thr = threshold
Full size table

The G’/B’ ratio was ignored in the definition of the Chl category, as was the R’/B’ ratio for CFP/GFP, since the respective extracted/purified pigments had only background signal in these channels (Fig. 2). Although R’G’B’ images were background corrected during initial processing, we also set a minimum intensity threshold in the relevant channels for each category in order to exclude low intensity pixels produced by bleed-through of reflected excitation light (Table 2). We defined an Other category to classify pixels with above-threshold intensity in any channel, but with channel ratios that did not match any of the three explicit categories. Pixels with below-threshold intensity in all three channels were considered to have no signal.

Classification of live organisms

We then collected spectral data and MEF images for a range of coral reef benthos, including green and red algae, CCA, and a number of coral species. We converted the 450 nm/530 nm image pairs into R’G’B’ images and applied the classification algorithm (Figs. 4, 5, S5, Table 2). Green algae had chlorophyll-like emission spectra and excitation maxima < 500 nm and correspondingly showed stronger red fluorescence under 450 nm light (Fig. 4a, Table 1). Red algae, which present the excitation/emission signature of PE as well as the emission signature of chlorophyll, appeared brighter under 530 nm light (Fig. 4b, Table 1). CCA spectra had a stronger PE emission contribution and they were strongly fluorescent in both images (Fig. 4c, Table 1). Coral spectra displayed the excitation/emission signature of the dominant FP and also showed the emission contribution of chlorophyll to a variable degree. Red Montipora foliosa and Oxypora sp., which contain an RFP and a pcRFP (Smith et al. 2017), respectively, had strong signals in both 450 nm and 530 nm images (Fig. 5a, Table 1), while A. millepora, brown M. foliosa, Porites lichen and Seriatopora hystrix appeared only brightly green under 450 nm light (Fig. 5b, Table 1). The algorithm classified green algae as Chl, cyan and green corals as CFP/GFP, and CCA and red corals as RFP/PE (Figs. 4, 5, S5). A few small areas near coral polyps or margins were also classified as Chl, and some pixels on a brown M. foliosa colony were classified as RFP/PE or Other (Fig. 5). Non-coralline red algae and some regions of rocks covered in algal films were classified as Other, with the exception of a few heteropigmented regions on red algae fronds which were classified as RFP/PE (Fig. 4).

Fig. 4
figure 4

Classification of multi-excitation fluorescence images of algae groups. Algae imaged with the CCD microscope camera under white light, 450 nm excitation (510 nm emission filter) and 530 nm excitation (610 nm emission filter), results of image classification, excitation (Ex) and emission (Em) spectra with peak wavelengths (nm). a Green algae: Caulerpa sp. (top and spectra), unidentified turf species growing on a coral skeleton (bottom). b Red algae: unidentified fleshy macroalgae (top and spectra), unidentified turf species (bottom). c CCA: unidentified species growing on a coral skeleton (top and spectra), unidentified species growing on live rock (bottom)

Full size image
Fig. 5
figure 5

Classification of multi-excitation fluorescence images of hard corals. Corals imaged with the CCD microscope camera under white light, 450 nm excitation (510 nm emission filter) and 530 nm excitation (610 nm emission filter), results of image classification, excitation (Ex) and emission (Em) spectra with peak wavelengths (nm). a Red coral: red morph of Monitpora foliosa (top), Oxypora sp. (bottom and spectra). b Green coral: (top to bottom) Acropora millepora (top spectra), brown morph of M. foliosa, Porites lichen (bottom spectra) and Seriatopora hystrix

Full size image

CMOS camera imaging

Finally, we assessed whether CMOS cameras commonly deployed in benthic surveys could be used effectively in place of the microscope CCD camera to generate R’G’B’ MEF images. Therefore, we adapted the same illumination system to image aliquots of extracted/purified pigments with a digital compact camera (Fig. 6), using emission filters of similar optical properties to the ones mounted on the microscope filter wheel (Fig. 1a, Fig. S2). With the CMOS camera set-up, PE and amilFP597 had only background-level signal in the G’ channel, making these pigments indistinguishable from extracted chlorophyll from the 450 nm image alone; the difference could, however, be easily resolved in the 530 nm image (Fig. 6). The strongest signal for amilFP497 was in the G’ channel, consistent with the CCD camera results, but eechRFP had stronger signal in R’ (Figs. 2, 6). All pigments produced a sufficient fluorescence signal to appear clearly above background.

Fig. 6
figure 6

Multi-excitation imaging of extracted/purified pigments with a compact CMOS camera. Pigments imaged under 450 nm excitation (510 nm emission filter) and 530 nm excitation (610 nm emission filter), barplots of pixel intensity in the R’, G’, and B’ channels, and the resulting combined R’G’B’ image

Full size image

We then used the same compact camera configuration at a wider angle to image an assemblage of organisms from our coral reef mesocosm (Fig. 7a). In this image, red and green corals produced a strong fluorescence signal in the R’, G’, and B’ channels; lower intensity signal was also detected in the R’ and B’ channel for red algae and CCA (Fig. 7a). However, the same green macroalgae that produced clear signal in the R’ channel when imaged with the CCD camera did not produce above-background fluorescence in any channel when imaged with a compact camera (Figs. 4, 7a).

Fig. 7
figure 7

Multi-excitation imaging of marine benthic organisms with CMOS cameras. a Tropical organisms imaged with a compact camera (Olympus TG-4). Tropical organisms b and temperate macroalgae c imaged with the FluorIS. Scenes imaged under white light, 450 nm excitation (510 nm emission filter), 530/505 nm excitation

Full size image

We also applied the MEF approach to wide-field imaging of marine organisms using the FluorIS, a modified off-the-shelf professional camera described by Treibitz et al. (2015). In this configuration the internal IR filter, which blocks most wavelengths > 650 nm, is removed in order to improve detection of the chlorophyll-a 680 nm emission band. Due to a technical issue, we replaced the 530 nm LED with a 505 nm one, keeping all remaining components of the set-up the same (Fig. S2). With this method, Discosoma striata containing cyan and green FPs produced strong signal in the G’ channel (Fig. 7b), while CCA and red macroalgae had the strongest signal in the B’ channel (Fig. 7b, c). Importantly, green macroalgae and turf algae produced above background signal in both R’ and B’, resulting in a clear visual contrast from CCA and red macroalgae in the R’G’B’ image (Fig. 7b, c).

Discussion

System design and image processing method

The MEF imaging method described in this study is a novel approach to functional imaging of benthic assemblages, which can be applied to characterise the keystone groups that support coral reef ecosystems as well as other aquatic ecosystems. The method relies on three key components: high-power colour LEDs to excite fluorescence in two spectral bands, an RGB camera detector, and simple processing of fluorescence data as pseudocolour RGB images.

The high-power LEDs used in this study provide up to 100 lumens per diode over a narrow waveband of ~ 50 nm. Narrow-waveband illumination is a prerequisite of fluorescence imaging in any environment, to avoid bleed-through of excitation light (scattered or direct) into the detector channels. In the marine environment, the excitation light source must also output enough intensity to compensate for stronger attenuation of light by water (Mazel et al. 2003) and allow a fast shutter speed to reduce the effects of water movement and ambient light contamination (Mazel 2005). Commonly, illumination for underwater fluorescence photography is achieved using off-the-shelf, wide spectrum professional strobes equipped with band-pass or short-pass filters (Mazel 2005; Treibitz et al. 2015); as a result, only a fraction of light emitted by the instrument is used to excite fluorescence, and multiple strobes are required to achieve sufficient output (Treibitz et al. 2015). Our LEDs are thus an ideal component for the development of a multi-excitation illumination system that can produce high intensities while keeping size, cost and battery requirements within practical limits. The narrow spectral width of LED light sources is also ideally suited to promote channel separation and minimise excitation light bleed-through. To further remove bleed-through contributions from the fluorescence images, we included a channel-specific background subtraction step in the image processing workflow. While this method produced good results, we note that imaging substrates with a wider range of optical properties such as highly reflective coral or sand (Hochberg et al. 2003; Enríquez et al. 2005) may accentuate the issue. Combining narrow-waveband LEDs with a short-pass excitation filter could thus further optimise the approach for field applications.

The pseudocolour RGB format provides a means to summarise all spectral information into a single frame, and results in good visual separation of the pigments in the colour space. The aim is not only to facilitate image interpretation by the user, but also to enable the application of colour or texture-based automatic classification methods developed for reflectance RGB images (Mehta et al. 2007; Stokes and Deane 2009; Beijbom et al. 2012; Mahmood et al. 2016). Our prototype imaging system was designed to eliminate or greatly reduce the need for image registration; however, we acknowledge that the need to obtain registered image pairs may pose a challenge to the application of this technique in the field (Beijbom et al. 2016). We propose the use of a custom-built framer and a post-processing registration step, an approach successfully adopted for reflectance/fluorescence image pairs obtained with two separate cameras in previous studies (Beijbom et al. 2016; Zweifler et al. 2017). Additionally, Beijbom et al. (2016) showed that the accuracy of convolutional neural network annotation improved when applied in a two-stage approach on image pairs, rather than directly on the combined image; this approach should be taken into consideration for computer vision-based analysis of MEF images.

Fluorescent pigment detection by MEF imaging

In R’G’B’ images acquired with a CCD microscope camera, chlorophylls purified from green algae appear almost exclusively red, coral GFP appears green, while coral RFPs and red algae PE appear as shades of pink and purple. The 450 nm light source has a strong overlap with the Soret absorption band of chlorophyll-a and b, therefore efficiently exciting deep-red fluorescence emission (Mackinney 1941; Falkowski and Kiefer 1985); the green gap in the chlorophyll absorption spectrum, on the other hand, results in negligible excitation by the 530 nm light. These spectral properties cause extracted chlorophyll to be clearly defined in the R’G’B’ colour space by the strong R’ signal and negligible signal in the other channels. Coral GFPs and CFPs are also strongly excited by 450 nm light (Alieva et al. 2008); while the blue part of their spectrum is blocked by the emission filter, the rest falls almost entirely within the sensitivity of the G channel of the camera sensor, resulting in clear separation of this pigment group as strong G’ signal (Treibitz et al. 2015; Beijbom et al. 2016). PE and coral RFPs have similar spectral properties; both are strongly excited by 530 nm light, which produces the signal in the B’ channel that clearly separates them from the other pigments. Orange-red fluorescence of RFPs and PE is also excited to a lesser extent by 450 nm light; however, emission is blue-shifted from chlorophyll fluorescence by about 100 nm, and as such it is expected to generate additional signal in either just R’, or both R’ and G’ channels in variable proportions depending on the spectral sensitivity of the camera sensor and on the exact emission peak of the pigment considered.

pcRFPs are a particular group of coral RFPs which emit green fluorescence upon synthesis, but are converted into a red-emitting form upon irradiation with light in the near-UV range (Ando et al. 2002; Wiedenmann et al. 2004). These proteins can produce signal in all three channels via direct excitation and emission of the unconverted green chromophore (G’), via direct excitation and emission of the converted red chromophore (B’), and via emission of the converted red chromophore upon excitation of the unconverted green chromophore (R’), which occurs through highly efficient Förster Resonance Energy Transfer (FRET) between adjacent protein subunits (Wiedenmann et al. 2004; Bollati et al. 2017). The relative proportions of R’G’B’ signal will thus depend on the degree of photoconversion of the pcRFP pool in question. The purified pcRFP used in this study had a high proportion of unconverted, green-emitting chromophores, which conferred it a different spectral signature from other RFPs and produced G’ signal when imaged with either camera. However, in shallow water environments coral pcRFPs have a higher proportion of red chromophores and emit mostly orange-red fluorescence (Bollati et al. 2017); these pigments will thus group more closely to other orange-red pigments under MEF imaging in the field.

MEF imaging and functional classification of live organisms

Due to the high efficiency of coupling between photosynthetic pigments, fluorescence emission of intact algae occurs predominantly from de-excitation of PSII chlorophyll-a (Falkowski and Kiefer 1985); however, in green algae much of the PSII excitation energy is transferred from chlorophyll-a and b molecules found in light harvesting complexes (Thornber 1975), while in red algae and cyanobacteria it is transferred from phycobiliproteins found in the phycobilisomes (French and Young 1952; Myers and Kratz 1955). Thus, spectral fluorescence excitation rather than emission is particularly suited to provide information about the accessory pigment composition of photosynthetic organisms, and consequently on their taxonomic and functional grouping. This is confirmed by the results of our classification algorithm and further supported by spectroscopic studies performed on micro- and macroalgae, which have shown that functional groups can be discriminated by comparing chlorophyll emission under blue and green excitation (Yentsch and Phinney 1985; Topinka et al. 1990).

Despite the very strong coupling between accessory pigments and PSII chlorophyll-a, in the case of PE some direct emission is still detectable in vivo albeit with a much lower quantum yield than what is observed for the pigment in isolation (French and Young 1952). In this study, the relative intensities of chlorophyll and PE emission differed between CCA and non-calcifying red macroalgae, resulting in red macroalgae being classified as a separate group by the algorithm. The concentrations of the two pigments are known to vary interspecifically within the red algae (Kim et al. 2007); these differences could thus be exploited to improve classification in this category by developing an algorithm that accounts for pigment combinations. This kind of algorithm could also allow further discrimination between CCA and red corals, which do not currently separate in the R’G’B’ space under the CCD camera due to spectral overlap between PE, RFPs and pcRFPs (Oswald et al. 2007). The rationale for this is that as well as red fluorescence, RFP-containing corals usually also emit fluorescence in the cyan or green spectral range, either through co-expression of multiple FPs (D’Angelo et al. 2008), or due to incomplete maturation of the red chromophore which is typical of both dsRed-type and pcRFP-type coral pigments (Matz et al. 1999; Baird et al. 2000; Ando et al. 2002; Wiedenmann et al. 2004); the combined signal could therefore be used to confidently separate corals from other groups. It should also be noted that confusion of CCA with hard coral is not commonly an issue for manual or automatic classification in reflectance images (Beijbom et al. 2012, 2015), and we therefore expect that machine learning algorithms will have no issue discriminating between these groups in R’G’B’ images. Integrating R’G’B’ colour with other image properties currently used by computer vision models, such as texture (Mehta et al. 2007; Stokes and Deane 2009; Beijbom et al. 2012), or combining fluorescence images with reflectance ones (Beijbom et al. 2016) might further facilitate this classification.

MEF imaging with off-the-shelf cameras

Waterproof point-and-shoot compact cameras, which mount 1/2.3′’ CMOS sensors, are one of the most accessible and portable imaging tools used in benthic surveys. When combined with our dual-LED excitation system, this type of compact camera was effective in imaging all purified pigments, which separated clearly in the R’G’B’ space. Importantly, under the CMOS camera the information provided by the 530 nm image and conveyed by the B’ channel was the sole discriminant between chlorophyll and PE/RFPs, while under the CCD camera this separation was possible based on R’ and G’ signal alone. This is due to different channel sensitivity of the two sensors, which causes PE and RFP to produce some G’ signal in the CCD camera but not in the CMOS camera. This clearly illustrates that adopting a MEF approach for in situ imaging can improve fluorescence-based classification, offering a definite advantage over methods based on a single excitation band particularly when using a CMOS camera to capture the fluorescence signal.

While extracted chlorophyll was easily detected by the CMOS compact camera in the R’ channel, red fluorescence of chlorophyll-containing organisms was low when imaged in vivo. This is due to photochemical quenching of PSII excitation, which occurs via reduction of plastoquinone in the thylakoid membranes of chloroplasts resulting in lower fluorescence yield (Falkowski and Kiefer 1985). Additionally, off-the-shelf camera sensors are usually fitted with an IR-blocking filter to mimic the sensitivity of the human eye and obtain more realistic images; the IR filter cut-on partially overlaps the PSII emission band, causing the R’ signal reduction observed with the compact CMOS camera compared to the microscope CCD camera (Treibitz et al. 2015). To improve detection of chlorophyll fluorescence in underwater images, Treibitz et al. (2015) developed the FluorIS, an off-the-shelf professional camera modified to remove the IR filter; this configuration has been extensively used in daytime wide-angle imaging of coral reef benthos, to improve automatic image annotation, and in surveys of coral recruits (Treibitz et al. 2015; Beijbom et al. 2016; Zweifler et al. 2017). Our images show that combining this approach with MEF LEDs and filters provides enhanced detection of chlorophyll emission in both R’ and B’ channels, allowing good separation of functional groups of photosynthetic organisms based on the excitation spectrum of their dominant accessory pigments. Thus, we suggest that the FluorIS is an ideal detector for use with our MEF system for field imaging of benthic assemblages on coral reefs and other aquatic ecosystems.

Beyond classification: benthic fluorescence as a physiological biomarker

Our study has shown that MEF imaging is a promising new approach to coral reef surveys, which we hope will underpin the design of a benthic imaging system based on accessible components to improve automatic classification and facilitate analysis of large datasets (Beijbom et al. 2016). The excitation and emission wavebands could be further customised to target only specific pigments or organisms by changing the LED and filter combinations, making this a flexible instrument for different survey purposes.

While this study has focused on the applications of MEF for classification into benthic groups, the approach can also be used to develop novel monitoring methods based on in situ imaging. As well as being a discriminant between groups, in fact, fluorescent pigments have potential to act as biomarkers of physiological processes within groups of organisms. For example, the relative proportions of chlorophyll and PE are known to vary intraspecifically with photoacclimation in CCA (Payri et al. 2001), a shift that may be detected from changes in the R’/B’ ratio in R’G’B’ images. In corals, the relative proportions of fluorescence signal in different spectral bands are also known to change with environmental conditions, either via differential regulation of multiple FPs, or via light-driven photoconversion. Changes in coral FP fluorescence have thus been used in laboratory settings as biomarkers of photoacclimation, defense against epibionts, growth and repair processes, heat stress, nutrient stress and bleaching (Zawada and Jaffe 2003; D’Angelo et al. 2008, 2012; Smith-Keune and Dove 2008; Bollati et al. 2020). Specifically, bleaching caused by exposure to brief episodes of heat stress or prolonged mild heat or nutrient stress can result in a reef-wide dramatic increase in coral fluorescence (Bollati et al. 2020). Clearly, our method can facilitate the automated analysis of these biomarkers in situ.

The optical properties of reef organisms may also be used to monitor interactions between corals and other taxa that have detrimental effects on reef health. For example, strains of cyanobacteria with different spectral signatures are associated with widespread coral syndromes, such as red band and black band disease (Sussman et al. 2006). Our approach could therefore be used to image the interface between healthy and diseased tissue, providing a non-invasive tool for monitoring disease progression. Similarly, spectral imaging may be used to target the interface between coral and turf algae (Barott et al. 2009); these interfaces affect coral health due to interactions between corals and the algal microbiome (Smith et al. 2006) and are important to assess how reefs recover following disturbance (Diaz-Pulido et al. 2009). Finally, while we have primarily discussed the potential applications of MEF imaging for coral reef assemblages, the method can be applied to survey a wide range of shallow water benthic ecosystems. For example, MEF imaging of PE fluorescence could be used to non-invasively identify live thalli on maerl beds, which are a highly biodiverse temperate ecosystem with priority conservation status in European legislation (Barbera et al. 2003).

Future work should focus on optimising the method for field image acquisition and explore the application of computer vision for MEF image analysis. In summary, our method has a great potential to produce high information content benthic surveys in coral reefs and other suitable marine habitats.