1 Introduction

A primary goal of the Mars 2020 Perseverance rover mission is to “determine if life ever arose on Mars” (NASA 2020; see also Farley et al. 2020). In the quest of answering this question Perseverance has been equipped with a total of 16 cameras of which seven serve primarily scientific purposes and nine have a primarily engineering function. (Maki et al. 2020). Among the seven scientific cameras are six designed to characterize the geological context of rocks within its landing site, Jezero crater. Each camera plays a unique role in Perseverance efforts to characterize Jezero’s astrobiological potential. Six of Perseverance’s cameras, distributed among four instruments, are utilized for surface imaging. Mast Camera Zoom (Mastcam-Z) has two stereo cameras mounted on the top of the rover’s mast (Bell 2021). SuperCam, also mounted on the rover mast, has a Remote Microscopic Imager (RMI) (Maurice et al. 2021). The Scanning Habitable Environments with Raman & Luminescence for Organics & Chemicals’ (SHERLOC) mounted at the end of the rover arm has two cameras, the Advanced Context Imager (ACI) and the Wide Angle Topographic Sensor for Operations and eNgineering (WATSON) (Bhartia et al. 2021). Planetary Instrument of X-ray Lithochemistry (PIXL), also mounted at the end of the rover arm, has a Micro Context Camera (MCC) (Allwood et al. 2020). The nine engineering cameras are divided into three groups: two Navigation Cameras (Navcams), six Hazard Avoidance Cameras (Hazcams) and one Cachecam. The Navcams and Hazcams ensure safe navigation of Perseverance as well as provide value information that can be used for scientific purposes such as topography maps and co-registration between the science cameras. The Cachecam is a fixed focus camera placed inside the Adaptive Caching Assembly (ACA) used for inspection and documentation of samples collected in the collection tubes.

Camera systems either rely on a passive external light source (i.e., the Sun; e.g., Mastcam-Z, SHERLOC, SuperCam, Hazcam, Navcam) or use an active light source (e.g., PIXL, SHERLOC, Cachecam). Cachecam, SHERLOC and PIXL are all equipped with light emitting diodes (LEDs). Cachecam has three white LEDs to provide a light source inside ACA enabling imaging of the samples within the sample tubes. SHERLOC has four white LEDs and two ultraviolet (UV) LEDs enabling the WATSON and ACI, respectively, to capture high resolution images of potential mineral fluorescence. The PIXL Optical Fiducial Systems (OFS) uses 24 LEDs of four colors for both highly accurate and robust navigation for positioning of PIXL (Liebe et al. 2022).

Both MastCam-Z and PIXL OFS on Perseverance collect multispectral data that enable the identification of minerals that aid the interpretation of chemical data. MastCam-Z collects multispectral images using a set of narrowband optical filters mounted on filter wheels that rotate in front of the CCDs of the two camera heads (Bell 2021). PIXL produces multispectral data by capturing images illuminated by individual LED colors (Allwood et al. 2020). Radiometric corrections are needed in order to generate multispectral data. For images taken under different lighting conditions (i.e., early morning or noon), MastCam-Z performs radiometric correction by collecting images of targets and calibration targets of known reflectance at the same time (Kinch et al. 2020). The active light source used by PIXL OFS at proximity of the target presents a challenge for developing the radiometric correction of an image under non-uniform lighting conditions with natural topography. This work represents the first such correction. We developed a novel method that considers hardware parameters, emitted light profiles, and target topography. In this manuscript, we present a step-by-step description of the radiometric correction process and show that the result has enabled robust multispectral imaging analyses. Our results are validated by comparing the spectral output to four different LabSphere© calibration targets, including some of Mastcam-Z’s geoboard standards, enabling cross comparison between the two instruments (Hayes et al. 2021, Rice et al. 2022).

The performance assessment was conducted using an OFS engineering model (EM), built as a full duplication of the OFS Flight Model (FM) of PIXL. The OFS EM is calibrated to have the same performance level as the FM (Klevang et al. 2023). Ultimately, the presented results validate the process of radiometric correction, which enables key analysis of the Martian materials from both PIXL XRF and multispectral data.

1.1 Instrument Overview

1.1.1 PIXL OFS

The PIXL OFS consists of three subsystems: the MCC, two Structured Light Illuminators (SLI), and the Flood Light Illuminators (FLI). The PIXL sensor assembly, a front view of the optics and the three subsystems are shown in Fig. 1. The three subsystems are operated in concert and are all crucial for obtaining and processing the multispectral dataset. The MCC has a Charge-Coupled Device (CCD) sensor with 752 x 580 pixels and optics designed to provide well-resolved images over the full operation envelope at a distance 10-100 mm from the instrument face to a target surface (hereafter referred to as standoff distance); optimal focus is at the same focus distance as the X-ray beam at a standoff distance of 25.5 mm. Most images presented herein are captured from this nominal standoff, though performance is also verified outside this ideal scenario. The pixel scale at 25.5 mm standoff is ∼50 \(\mu \)m/pixel.

Fig. 1
figure 1

The PIXL sensor assembly is shown in the top left image. The different front optics are shown in the top right image. The three subsystems: MCC, SLI and FLI and their controlling electronics MCCE and FLIE A and B and where they are positioned within PIXL are shown in the bottom half of the figure (Allwood et al. 2020, Klevang et al. 2023)

Full size image

For proper exposure of the targeted surfaces the MCC has two closed loop controls. The first loop autonomously adjusts the Analog to Digital Converter (ADC) for the CCD exposure, offsetting the dark level in acquired images. The second loop automatically adjusts the shutter time to regulate light emitted from the FLI in order to ensure that the intensities of the captured raw images are comparable and neither over- nor under-exposed.

The SLIs use 830 nm lasers and are used to generate topographic maps using two different patterns: a dense SLI in a 3 x 5 grid with a four degree spot separation and a sparse SLI in a 7x7 grid with a nine degree spot separation. Each point of the laser grid is known to a three-dimensional precision of ∼50 \(\mu \)m within the image frame (Pedersen et al. 2019). The dense SLI at optimal standoff distance covers an area corresponding to ∼5% of the image frame, concentrated to the region where the X-ray beam intersects the surface. The sparse SLI covers the Field Of View (FOV) of the MCC, providing the full coverage of the operational envelope.

The FLI consists of 24 LEDs evenly distributed into four color channels: UV (385 ± 4 nm), Blue (447 ± 9 nm), Green (523 ± 13 nm) and NIR (723 ± 10 nm) (Klevang et al. 2023). Wavelengths are given as (\(\lambda \ \pm \) HWHM), where \(\lambda \) is the central wavelength and HWHM is the Half Width Half Max value. Both values are given at 20 °C, corresponding to the room temperature where the EM test setup is operating. The UV, Blue, Green and NIR LEDs central wavelengths absolute movement as a function of temperature are 5.9 nm, 5.4 nm 7.9 nm and 26.7 nm respectively going from -120 °C to 70 °C. The total light emitted by the LEDs also variate as a function of temperature. The LEDs have been tested from -120 °C to 70 °C (Klevang et al. 2023) and the maximum radiated light for all channels are at -25 °C and rolls of to both sides to ∼90% of the maximum at 70 °C and -120 °C. The relative emissions between the channels are quite stable.

1.1.2 Mastcam-Z

Mastcam-Z is a high heritage instrument based on the Mast Camera (Mastcam) which is onboard the Mars Curiosity rover (Bell 2021). The Mastcam-Z camera heads consist of CCD cameras that are both focus and zoom adjustable and capable of capturing 1600x1200 pixel images. Mastcam-Z can capture images with a pixel scale of 148 \(\mu \)m/pixel at a range of 2 m with focal length 110 mm. Mastcam-Z collects multispectral data through the employment of different filters positioned in a wheel in front of each CCD. One “clear” filter (for RGB Bayer-pattern filter imaging) and seven different narrowband filters are available for each of the camera heads. One of the narrowband filters is used for direct sun imagining and the rest for surface imaging. The narrow band wavelengths used for surface imaging are 442±12 nm, 528±11 nm, 605±9 nm, 677±11 nm, 754±10 nm, 800±9 nm, 866±10 nm, 910±12 nm, 939±12 nm, 978±10 nm, 1022±19 nm and the Bayer-pattern RGB wavelengths are 480±46 nm, 544±41 nm, and 630±43 nm (Bell 2021).

Mastcam-Z images are radiometrically calibrated using pre-flight coefficients (Hayes et al. 2021) and near-simultaneous images of the Mastcam-Z calibration targets located on the rover deck (Kinch et al. 2020, Merusi et al. 2022). Prior to launch, Mastcam-Z calibration activities included imaging a number of color calibration standards, geometric targets, rock samples, and education and public outreach (E/PO) materials. Collectively, these targets are referred to as the Geology Board, or “geoboard.” Visible to near-infrared (VNIR) laboratory spectra of the geoboard targets were acquired at similar viewing geometries to those used during Mastcam-Z calibration (Lapo 2021) with Western Washington University’s TANAGER spectrogoniometer at 1 nm spectral resolution (Rice et al. 2022). The TANAGER spectra of the color calibration standards represent common reference points for the Mastcam-Z and the PIXL MCC.

2 PIXL OFS Multispectral Laboratory

To support PIXL operation on Mars, an EM test-setup of the OFS was built at the Measurement and Instrumentation division at the Technical University of Denmark (DTU) Space Department, where the flight OFS unit was both designed and manufactured. An exact replica of the PIXL OFS has been mounted on a H-820 hexapod made by Physics Instruments©, which enables maneuvers with 6 degrees of freedom and is similar to the accuracy and movement of the custom hexapod in use by PIXL (Allwood et al. 2020).

The optics of the FLI are designed and calibrated such that the uniformity of illumination and emitted power are optimal at 25.5 mm standoff distance. For the OFS EM, the X-ray optics are replaced with a dummy brass rod, which is used as the reference point for standoff distance (marked with red circle in Fig. 2C). When PIXL is performing measurements, the X-ray beam is planned to be perpendicular to the surface of the target (Fig. 2B). This is achieved by finding an average plane of the target surface using 3D points obtained from the SLIs (marked with blue in Fig. 2C) and moving the hexapod accordingly to bring the X-ray beam perpendicular to the average surface plane. The MCC (marked with an orange circle in Fig. 2C) is ∼23 mm recessed from the tip of the X-ray optic (i.e., 53 mm from a rock target) and offset ∼18 mm along the y-axis from the center of the instrument. The MCC is tilted with an angle of 18 degrees about the x-axis of the MCC (Fig. 2C), having the principal line of the MCC intersecting the X-ray approximately at the nominal standoff distance. Detailed calibration of OFS geometry is provided by (Klevang et al. 2023).

Fig. 2
figure 2

PIXL OFS test-setup at DTU Space, Measurements and Instrumentation division

Full size image

The lab unit conducted observations and tests of OFS capabilities on more than 40 samples, including a wide range of calibration targets, abraded targets, polished targets, natural targets, and 10 targets from the Mastcam-Z geoboard (Hayes et al. 2021). All targets discussed herein are listed in Table 1.

Table 1 Target overview
Full size table

3 PIXL OFS Image Processing

The PIXL OFS collects multispectral data by capturing a grayscale MCC image while the FLI is emitting light of a single color channel at a given time. This collection of four images is referred to as a color-stack. To compare the individual images in the color-stack and enable color imaging and multispectral analysis, a radiometric correction is required. Radiometric correction involves several different image processing steps, outlined in Fig. 3.

Fig. 3
figure 3

Flowchart of processing steps to radiometrically correct PIXL color-stack images. Bold yellow and blue bordered boxes correspond to input. Yellow bordered boxes represent processing done on calibration images while blue bordered boxes correspond to processing done on target images. Bold green bordered box corresponds to output

Full size image

Overall, six different processing steps are required: 1) compensation for dark level, 2) creation of a topographic map, 3) creation and application of flat fielding, 4) adjustment of different shutter times, 5) compensation for different LED current settings, and 6) compensation for LED intensity and CCD sensitivity at the specific LED wavelengths. These six specific adjustments and compensations are based on a calibrated radiometric profile of the individual color channels of the FLI, as described in (Klevang et al. 2023). In the following, the image processing steps are described with an example of applying each processing step to both a flat polished target and a natural target. A natural target in this context is a target that does not have a flat surface, for example, part of the Rainbow Rocks & Minerals target from Mastcam-Z geoboard (Hayes et al. 2021). The raw images used for the examples in this section are shown in Fig. 4. All images from stepwise processesing are included in suplementary.

Fig. 4
figure 4

Raw images from the 4 separate FLI color channels. Top row shows the flat target and bottom row shows the natural target (Table 1). These images were taken at a standoff distance of 25.5 mm

Full size image

3.1 PIXL MCC Dark Level Correction

The dark level is the pixel intensity, given as a Digital Number (DN), that specifies signal floor from the CCD, rooted from the dark current, which is thermally dependent, i.e., which pixel intensity corresponds to zero light. Operating with the FLI, the MCC autonomously adjusts the floor and ceiling levels of the Digital Analog Converter (DAC) to optimize the image histogram and dynamic range of the imaged scene. This means, prior to any scaling of the MCC images, an offset correction, bringing the dark level to zero DN, is required. In addition to the science context images, the MCC captures images using the SLI for topographic information. When operating with the SLIs the DAC levels are tracking the dark level, providing the needed DAC level information encapsulating the thermal environment at the time of context imaging. This means the dark level of the SLI images can be used to identify the dark level offset of the context images.

The dark level (\(DL\)) for science context images using the FLI is calculated using equation (1) from (Klevang et al. 2023),

$$ DL= \left ( SL I_{offset} -Im g_{offset} \right ) \boldsymbol{\cdot} \frac{1}{2} \boldsymbol{\cdot} \frac{Im g_{res}}{DA C_{res}} +14 $$
(1)

where SLIoffset is the Digital Analog Converter offset from the SLI images which represents the true dark level and is thermally compensated, Imgoffset is the DAC offset for each image in a color-stack, \(\mathit{Img}_{\mathit{res}}\) is the dynamic range of the image (8-bit), \(\mathit{DAC}_{\mathit{res}}\) is the resolution of the DAC (480). The dark level is subtracted from each of the raw images as described by equation (2), where \(\mathit{I}_{DL}\) is the dark level corrected images, \(\mathit{I}_{\mathit{raw}}\) is the raw images and (\(x\),\(y\)) is the column and row index of a pixel in the image.

$$ I_{DL} (x,y)= I_{raw} (x,y)-DL $$
(2)

The output of the dark level correction can be seen in Fig. 5.

Fig. 5
figure 5

Dark level corrected images from raw images in Fig. 4

Full size image

This dark level correction assumes the dark level global to the entire image. To capture the dark level that spatially varies in the CCD, an extra image with same camera settings, but with the LEDs turned off (a dark image) needs to be captured. This dark image is then subtracted from the original image. This approach requires one additional image for each color channel as they have individual exposure settings. However, it should be emphasized that the spatial variation of the dark level is miniscule by design with a pixel variance of 0.013 DN around the center part of the FOV. Furthermore, the spatial variation of the dark level is vanishing small compared to the topographic effects that are further elaborated in the following section, and is therefore reasonably assumed global.

3.2 PIXL SLI Topographic Map

A topographic map is necessary to generate the correct flat fielding because the profile of the light cone projected onto the targeted surface is dependent on standoff distance. The OFS obtains measurements of the standoff distance to the target surface by utilizing the SLIs. The SLIs lasers emit light in a calibrated grid and by determination of the position of the laser points in the MCC image frame the distance to laser point is determined (Pedersen et al. 2019). The generation of the topographic map and its effects are described and discussed in this section.

3.2.1 Generation of Topographic Map

A topographic map covering the whole FOV of the MCC is preferred. This is achieved by utilizing both the dense and the sparse SLIs when capturing the color-stack. An example of the coverage of the SLI-measurement using both dense and sparse SLIs and the corresponding topographic map of both targets can be seen in Fig. 6.

Fig. 6
figure 6

Illustration of the distribution of the combined dense and sparse SLI coverage. Red marks in images are the location of the SLI measurements and red dashed line is a polygon consisting of the outermost SLI measurements. White lines are topographic contours with distance scales on the right. Top left: image of flat target. Top right: topographic map of flat target. Bottom left: image of natural target. Bottom right: topographic map of natural target

Full size image

For a predominantly flat target, a simple linear interpolation between the SLI point measurements provides a sufficiently accurate model of the planar surface. However, a natural target is topographically challenging as spatial resolution from the SLI-measurements is relatively coarse. To remedy artifacts from a coarse point cloud for topographic modelling, a cubic interpolation based on Delaunay triangulation is applied. Using cubic interpolation does not necessarily provide a more correct interpretation, compared to linear interpolation, however it achieves continues transitions between the SLI beam positions in contrast to linear interpolation. The non-continuous transitions, from a linear interpolation, leave clear and distinct artifacts in images at the transitions. The Supplementary Fig. 3 (see, electronic supplementary material) illustrates the difference in the topographic map using a cubic interpolation compared to a linear interpolation.

The cubic interpolation outside the area of the SLI-measurements is not constrained, and thus a different approach is needed. Assuming the target overall is topographically uniform, a plane is fit to all the SLI-measurements and extended to cover the full MCC frame. A polygon is found that connects all the outermost SLI-measurements (illustrated with the red dashed line in Fig. 6). Each pixel outside the outer polygon is topographically positioned by finding the closest point in the outer-polygon and offsetting the found plane along the cameras z-axis such that it crosses this point, and the topographic position is extrapolated from the plane.

Combining the inner and outer interpolation might result in non-smooth edges going from inside of the outer-polygon to the outside, however, as the sparse measurements covers a large portion of the MCC FOV, this is acceptable.

SLI points localized onto a pit or on top of a high standing relief that is not representative of the surrounding topography is of concern. Due to the relatively sparse distribution of SLI-measurements, this will result in incorrect calibration images being used to create the flat fielding profile. This risk is also reversed when small peaks or holes are overlooked. This can be seen in Fig. 6, where multiple pits are present in the natural target, but none include a SLI point measurement. On the other hand, if a SLI point measurement had fallen into a pit in this scenario the first error scenario would have occurred, where the measurement would not be representative for the surrounding area. For minimal artifacts to the surroundings, these single points that deviate significantly from the overall surface plane are considered outliers and are excluded. For optimal performance on high relief terrain a higher spatial resolution of the topography should be captured, either from additional SLI measurements or utilizing virtual 3D stereo imaging.

The correction for the light intensity of the active light sources, as a function of the standoff distance, is the driving error term. Obtaining a high-resolution topographic map is of highest importance for the process of radiometric correction. If the topographic map is incorrect, the standoff distance to the individual pixel is wrong, which means that a wrong flat-fielding of the light profile is applied. Furthermore, as will be described in Sects. 3.3 and 3.3.1 are the standoff distances used for selecting the calibration images. The calibration images are scaled according to their respective shutter times which also are standoff dependent (described in Sect. 3.3 and 3.4). Finally, are the relative intensities of the calibration images used as described in Sect. 3.6 to correct for the wavelength varying LED intensity and MCC sensitivity. This essentially means using an incorrect topographic map will result in an accumulated error from wrong profile, wrong shutter time for shutter correction, and wrong relative LED intensity and MCC sensitivity correction.

3.3 Flat Fielding MCC Color-Stack Images

Flat fielding is an essential part of image radiometric correction. The purpose of flat fielding is to compensate for uneven light distribution as well as variations in the optical efficiency such as vignetting and pixel-to-pixel responsivity, to produce an image as if no optical variations occurred and as if the illumination of a target was uniform. This enables the direct comparison of each pixel, such that meaningful interpretation of a single channel is possible. The flat fielding process is applied individually to each color image to account for slight differences in the projected light cone. To create flat fielding profiles for each color channel, radiometric profiles of the radiated light are needed.

The radiometric profile of the radiated light from the FLI on the OFS EM was fully characterized at the optical test facility at the Measurement and Instrumentation division at the Technical University of Denmark (DTU). This characterization was registered by imaging an optical target that has a Lambertian and spectrally flat reflectance profile (Zenith Lite from Sphere Optics©). A series of images were captured within the full operational envelope of PIXL, i.e., 10-100 mm standoff distances. Around the nominal operational standoff distance (19 mm to 45 mm), images were captured at a resolution of 1 mm. It is in this region the quality of the multispectral images is tested in this paper. Each color band was characterized individually (Klevang et al. 2023). Prior to establishing the actual flat fielding profile, all calibration images have been adjusted for dark level, and a Gaussian filter with a sigma of 5 has been applied to remove small sized features, ∼100-200 \(\mu \)m, of the calibration target, as these are detectable by the MCC. The calibration images are scaled in accordance with the image with the longest shutter time to enable spectral comparisons (see Sect. 3.4. for further information on this process).

This process results in calibration images, that are scaled in accordance with their respective standoff distance within the individual color channels. The creation of the flat fielding profile and its application are described in the following sections.

3.3.1 Creating Flat Fielding Profile

At this stage the topographic map can be used to look up the radiometric profile of the FLI in the calibration images. Based on the actual standoff distance (and plane estimate) at which an observation is made, the light profile is extracted using a linear interpolation to make a seamless profile bridging the discrete positions at which calibration images are captured. Then, the profiles are individually normalized, such that the highest pixel intensity in each mask is 1.

An example of the resulting flat fielding profiles for the flat and rough targets are shown in Fig. 7. The flat target is essentially a plane conic section of the radiometric profile of the FLI. For the natural target with relief, the flat-fielding profile is significantly affected by the topography of the surface.

Fig. 7
figure 7

Flat fielding profiles generated from dark level corrected images in Fig. 5 using topography information in Fig. 6

Full size image

The interpolation of the radiometric profile, bridging the discretely captured calibration images, is a good approximation in cases when the corresponding pixel intensities in the two neighboring calibration images are fairly smooth. As the projected light cone is distance dependent, the interpolation will typically have a large effect in areas of the calibration images with high pixel intensity gradient as these areas are likely to change significantly between two sets of calibration images. Furthermore, artifacts from this interpolation have a greater impact on low-light regions compared to well-lit regions, as a gain from the flat-fielding is more significant in darker regions.

Errors from the interpolation can be minimized through the utilization of small step sizes between sets of calibration images. An example of the effect of having a step size of 5 mm compared to 1 mm can be seen in Fig. 8. The calibration images used for the analysis are anchor images captured at nominal standoff distance of 25.5 mm and images at 35 mm and 31 mm respectively.

Fig. 8
figure 8

Relative change between two sets of calibration images. Top row shows an example of relative change between two calibration image sets when the step size in standoff distance is 5 mm, bottom row illustrates with a step size of 1 mm. From left to right: NIR channel, green channel, blue channel and UV channel

Full size image

From Fig. 8 it can be seen that in the well illuminated center regions of the image, only relatively small changes of the radiometric profile take place between the calibration images when using 1 mm stepsize compared to using 5 mm stepsize.

3.3.2 Applying Flat Fielding Profiles

The flat fielding profile generated is applied to the dark level corrected images \(\mathit{I}_{DL}\) by dividing \(\mathit{I}_{DL}\) with the corresponding flat fielding profiles:

$$ I_{FF} \left ( x,y \right ) = \frac{I_{DL} \left ( x,y \right )}{FF_{profile} \left ( x,y \right )} \ \text{for} FF_{profile} \left ( x,y \right ) >0 $$
(3)

where \(I_{FF}\) is the flat fielding corrected image and \(FF_{\mathit{profile}}\) is the flat fielding profile. When \(FF_{\mathit{profile}}\)(\(x\),\(y\)) = 0 the output pixel \(I_{FF}\)(\(x\),\(y\)) = 0 The flat fielded images can be seen in Fig. 9.

Fig. 9
figure 9

Flat fielding corrected images using flat fielding profiles shown in Fig. 7 and applied to the dark field corrected images in Fig. 5

Full size image

The amplification from flat fielding profiles is largest in the lesser illuminated regions and, likewise, the Signal to Noise Ratio (SNR) is lower in these regions of the raw images. In other words, regions with the smallest SNR will be amplified the most by the flat fielding. To avoid artifacts in the multispectral analysis, stemming from pixels with low SNR, an upper limit of the applied flat fielding gain is enforced. Any pixels that are applied a gain larger than the upper limit are occluded for the multispectral analysis. The upper limit threshold is essentially a trade-off between a wide FOV and limiting the effect of noise in the multispectral analysis.

The MCC captures images with 8-bit resolution (0-255). The effective dynamic range is defined from the Dark Level (DL) to 255. To determine a worst-case effective dynamic range, the maximum DL among the 108 calibration images has been used. The maximum DL was 49, making the effective dynamic range 0-206 (after DL subtraction). The upper limit threshold has been set to allow for a maximum amplification of 10 during the flat fielding, which makes the effective dynamic range of the flat fielding profile 20.6-206.0 before the normalization (flat fielding profiles are not limited to integers after the interpolated). This limits the relative maximum scalation error to 4.85% if a pixel in the flat fielding profiles is offset by 1 DN due to noise (pixel is 20.6 instead of 21.6 for instance). This allows for a relatively large FOV while keeping potential noise relatively low. Any pixel that is amplified ≤10x for the flat fielding correction is considered to be well illuminated with a sufficiently high SNR for robust and accurate multispectral analysis.

In Fig. 10 the regions with a sufficiently high SNR for each color channel of the flat and the natural target are shown. The regions with a sufficiently high SNR level for the individual color channels are the white areas, and the common area with high SNR for all channels is marked with a red line. This common region can be applied as a mask to each color channel for the flat and the natural targets as shown in Fig. 11.

Fig. 10
figure 10

Illustration of regions (white areas) with high SNR and a gain mask ≤10 for each color channel. Red lines demarcate the common area of high SNR and small gain for all four channels. Top row shows the flat target, bottom row shows the natural target

Full size image
Fig. 11
figure 11

The four color channels for the flat (top) and the natural (bottom) targets with the high SNR pixel mask applied

Full size image

Besides limiting the effect of noise in the flat fielding corrected images, artifacts from the topographic changes of the target must be considered. The calibration images were collected on a flat target, which means flat fielding does not take shadowing effects into account. The relief of a natural surface may block part of the light emitted by the FLI and cast shadows. The shadows will cause the intensity within the shaded areas to not be gained sufficiently when performing the flat fielding correction, resulting in color artifacts in the radiometrically corrected images. Further, the color LEDs are not placed at the same location on the instrument front end (Klevang et al. 2023). Although a uniform mechanical layout, or placement, of the LEDs was achieved, a different shadow on the individual color channels will occur. That means, that not only are the areas darker, but the color channels are not comparable either. These shadowing artifacts are not dealt with in this algorithm. However, one can manually inspect the regions with large topographic relief and disregard the affected regions.

3.4 MCC Shutter Time Correction

Shutter time is automatically controlled by the MCC Electronics (MCCE) that support highly varying illumination conditions and take thermal effects into account. The automatic exposure control is an iterative approach that seeks to equalize the image histogram on both the upper- and lower-end of the image pixel histogram. The exposure control is activated for each independent image, meaning that the actual shutter time used depends both on the spectral response of the target and which FLI color configuration is activated (Klevang et al. 2023). To make the images cross-comparable they must be scaled according to their individual shutter times. As a baseline, the images are scaled according to the image with the longest shutter time, which is the NIR channel for both the flat and natural target. The correction due to shutter time is described by equation (4), where \(I_{ST}\) is the individual shutter time corrected images, Max_Shutter_Time is the shutter time of the image with the largest shutter time, Shutter_Time is the induvial image shutter time.

$$ I_{ST} (x,y)= I_{FF} (x,y)\boldsymbol{\cdot} \frac{Max\_Shutter\_Time}{Shutter\_Time} $$
(4)

Fig. 12 shows the image intensity with the shutter time compensation applied to each individual color channel for the flat and natural target. The exposure times of NIR, Green, Blue and UV color channel are 443 \(\mu \)s, 197 \(\mu \)s, 97 \(\mu \)s, and 348 \(\mu \)s for the flat target, and 328 \(\mu \)s, 116 \(\mu \)s, 57 \(\mu \)s, and 249 \(\mu \)s for the natural target.

Fig. 12
figure 12

Shutter time compensated images applied to the flat-fielding corrected images in Fig. 11

Full size image

The shutter time is measured in integers of \(\mu \)s which may result in a rounding error of ±0.5 \(\mu \)s. The effect of this potential rounding error becomes smaller with longer shutter times. Thus, this potential rounding error will have the largest impact on the blue channel for the flat and natural targets. Considering the worst case scenario, where the NIR shutter is rounded up by ∼0.5 \(\mu \)s and the blue channel rounded down by 0.5 \(\mu \)s, this will result in a scale error smaller than 1.3% and 2.1% for the flat and natural targets, respectively. The maximum error of this effect can be found using the following relationship, where semax is the maximum potential error from shutter rounding.

$$ se_{max} < \left ( \frac{Max\_Shutter\_Time+1}{Shutter\_Time-1} - \frac{Max\_Shutter\_Time}{Shutter\_Time} \right ) \frac{Shutter\_Time}{Max\_Shutter\_Time} $$

3.5 LED Forward Current Correction

The LED forward current needs to be considered as the radiated power from the different LED color channels is not linearly dependent on the forward current applied to the LEDs. The FLI has three different forward current settings: 140 mA, 280 mA and 500 mA. The relative scale of the radiated power for individual color channels as a function of the forward current is determined by capturing images of a white target with each of the different forward current settings and comparing their respective shutter times. Images captured with a different forward current setting are pixel wise scaled in accordance with the respective relative scale in accordance with (Klevang et al. 2023). The relative scales, that must be applied when calibration and target images are captured at different forward currents normalized to 500 mA are shown in Fig. 13. The calibration and example images used throughout this paper are captured at the same forward currents, why the output of this step is equal to Fig. 12

Fig. 13
figure 13

Normalized shutter times relative to 500 mA calibration images

Full size image

3.6 PIXL LED Intensity and MCC Sensitivity Correction

The last step of the processing pipeline accounts for the individual radiant intensities of the four color LEDs and the spectral sensitivity of the MCC CCD. Applying this last step makes the channels one-to-one comparable. The relative sensitivity of the CCD is shown together with the LED wavelengths in Fig. 14. The most straight forward approach would be to correct for the MCC sensitivity first and subsequently correct for the radiance intensity for each color channel. However, it is not possible to tell from the images if the CCD is more sensitive to a given wavelength or if the relative intensity of the LEDs at that given wavelength is larger. Therefore, the correction of the LED radiant intensity and CCD sensitivity is applied in a single step.

Fig. 14
figure 14

MCC CCD spectral response showing the relative sensitivity as a function of wavelength. Vertical lines at 385 nm, 447 nm, 523 nm and 723 nm correspond to the FLI LED wavelengths for UV, blue, green and NIR, respectively at 20 °C

Full size image

The combined relative sensitivity/intensity is found using MCC calibration images of the Zenith Lite calibration target. The spectral reflectance of the Zenith Lite target is: 96.085%, 96.127%, 96.034% and 95.588% at 385 nm, 447 nm, 525 nm and 723 nm, respectively. These absolute reflections are used to scale all calibration images such that a flat spectral response is achieved across the four wavelengths. Recall that all calibration images were scaled according to shutter time, thus at each stepwise standoff distance, were calibration data is captured, the brightest pixels for each color channel should be equal, as the spectral response is flat. This gives a relative scale between the color channels at each standoff distance. The intensity scale, as a function of distance, can be seen in Fig. 15, after correction for dark level, shutter time and absolute spectral response and LED radiant intensity. The figure shows that the spectral response of the Blue channel is largest at all these standoff distances, whereby the other channels are amplified according to this channel.

Fig. 15
figure 15

Intensity scale as a function of distance, after correcting for dark level, shutter time, CCD spectral response and LED radiant intensity

Full size image
Fig. 16
figure 16

Images from Fig. 12 corrected for LED intensity and MCC sensitivity. These images are the final product of the radiometric correction

Full size image

In regions between the calibration images a linear interpolation of the scaling factors is applied. The correction is described by Equation (5), where IInSeCorr is the intensity/sensitivity corrected image, \(I_{ST}\) is the shutter time corrected image from Equation (4), Max_Int_All_C is the maximum intensity between the four channels at a given standoff distance, Intensity_Individual_C is the individual channels maximum intensity.

$$ I_{InS e_{Corr}} (x,y)= I_{ST} (x,y)\boldsymbol{\cdot} \frac{Max\_Int\_All\_C(x,y)}{Intensity\_Individual\_C(x,y)} $$
(5)

The results after this correction step for the flat and natural target can be seen in Fig. 15. Images from all the steps of the two targets can be seen in Supplementary Fig. 1 and Fig. 2 (see, electronic supplementary material).

4 Results and Validation

The complete processing of the multispectral dataset is tested on LabSphere© calibration targets of four colors, to ensure that the radiometric correction provides a correct interpretation of the colors. The four targets used (Table 1) are from the Mastcam-Z geoboard (Hayes et al. 2021), where reference spectra from the TANAGER enables instrument to instrument comparison. Two different experiments were conducted: firstly, to demonstrate the performance of the relative spectral response at nominal standoff distance, and subsequently to demonstrate the robustness by varying the angle and distance.

4.1 Performance of Relative Spectral Response

The test was conducted with the PIXL EM OFS positioned at 25.5 mm standoff. The spectral responses of the color standards as measured by the MCC are shown in Fig. 17 together with the calibrated spectral profile obtained by TANAGER (Rice et al. 2022). To compare the two datasets, the MCC responses in four color channels are summed and averaged over the FOV shown in Fig. 19. The standard deviation is that of the average values. To compare to TANAGER data, the mean values of UV, B, and G channels are then normalized using the NIR value as an anchoring point to the TANAGER spectra. Table 2 lists the values of the mean, the mean offset from the TANAGER data, and the standard deviation of the responses.

Fig. 17
figure 17

Comparison between MCC measurements of radiometric corrected images to spectra obtained by TANAGER. Vertical bars are mean and standard deviation of radiometric corrected images. Horizontal bars are the HMHW of the different LEDs

Full size image
Table 2 Mean, mean offset and standard deviation of each color channel for each calibration target comparing the measured spectral response from the MCC with the TANAGER dataset (Rice et al. 2022)
Full size table

The variation of pixel intensity within a single color channel is a measure of the quality of the radiometric correction and the steps detailed herein. The calibration targets are close to uniform as seen with the human eye or a handheld camera, however when looking at the 50 \(\mu \)m pixel scale, texture or impurities can be seen (Fig. 19). The largest standard deviations were found to be 0.034, 0.024, 0.043 and 0.038 for the four calibration targets, which are considered to be to be very small.

The mean offset is a measure of how well the four channels are scaled to one another. In Table 2 the mean offset of NIR channel is zero as this channel has been used for scaling the spectral response. The largest mean offsets for the four calibration targets are 0.009, 0.015, 0.019 and 0.09 Green, Blue, Green, Green channels, respectively. Thus, for the four calibration targets, after radiometric correction, the largest relative offset is 1.9% and the overall mean relative offset is 0.4%, when scaled according to the NIR channel.

Target appearance is captured in Fig. 18 and Fig. 19 as observed via a standard handheld camera and the reconstructed NIR-G-B MCC images, respectively. Color-relations from the two camera systems deviate significantly due to the NIR wavelength of the MCC not corresponding to a general red channel of a standard camera. The Mastcam-Z red Bayer filter wavelength is at 630±43 nm and the MCC NIR LED is at 723±10 nm. If the spectral response of the target is different at these wavelengths (e.g., Fig. 17, blue target), the NIR will not look like standard red. The MCC green and blue LEDs align better with a standard green and blue (e.g., Mastcam-Z green and blue Bayer filter wavelengths at 544±41 nm and 480±46 nm, respectively). However, one needs to be aware of PIXLs relative narrow bands when analyzing images, as the radiometric calibrated color images might look different from a standard RGB-images. That being said, using the MCC NIR-G-B color stack is a strong tool for analyzing images, as the channels correspond well with the measured spectrum (Fig. 17) and resolves both spectral features and spatial textures of rock samples.

Fig. 18
figure 18

Calibration targets used for verification of the radiometric correction process. From left to right calibration target: Red, Green, Blue and Yellow

Full size image
Fig. 19
figure 19

Reconstructed NIR-G-B color images of LabSphere© calibration targets in Fig. 17 after radiometric correction. From left to right, calibration targets are in the same order as in Fig. 17

Full size image

4.2 Robustness of the Radiometric Correction

In addition to the validation of the spectral response, it is ensured that the radiometric correction process is consistent as the target is tilted or moved closer/further away. Additional tests were performed to test the robustness of the radiometric correction with respect to target positioning. The blue calibration target was used for this test, as it has the most dynamic range in the reflectance spectra applicable to the OFS.

Images of the blue calibration standard were captured using each color channel at 26 different positions: five different standoff distances (20.5 mm, 25.5 mm, 30.5 mm, 35.5 mm, and 40.5 mm) with the instrument placed normal to the calibration target and 21 positions with varying angles at nominal standoff distance (25.5 mm). The angles in degrees around the x-axis were: -10, -7.5, -5, -4, -3, -2, -1, 1, 2, 3, 4, 5, 7.5, 10, and around the y-axis were: 1, 2, 3,4,5 7.5, 10. The symmetric nature of the OFS system around the y-axis means we require either negative or positive angles but not both. An example of two images captured at nominal standoff distance that have been radiometrically corrected through the outline steps, can be seen in Fig. 20. The left is captured with the instrument normal to the target, and the right with the instrument rotated 10 degrees about the x-axis.

Fig. 20
figure 20

Radiometrically calibrated images of the blue calibration target. Both images are captured at nominal standoff distance of 25.5 mm. Left is captured with the instrument normal to target, right is captured when the instrument is angled at 10 degrees about x-axis

Full size image

Because radiometrically calibrated images do not necessarily have the same brightness, the mean value of the NIR channel of each image is used to provide a fix point to the scale of intensity of other channels. The resulting, mean, mean offset, and standard deviation can be seen in Fig. 21 and Table 3.

Fig. 21
figure 21

Variation of pixel intensity when changing angle and standoff distance when capturing images of a uniform target

Full size image
Table 3 Pixel intensity results of images captured from a range of different angles and standoff distances
Full size table

The largest mean deviations of the Green, Blue and UV channel means from the total respective mean are 1.4%, 2.6% and 2.3% respectively. The mean deviation of the Green, Blue and UV channel means from the total respective mean are 0.5%, 1.0% and 0.6% respectively. The largest deviations from the mean standard deviations of the NIR, Green, Blue and UV channels from all observations are 1.3%, 0.4%, 1.0% and 0.7%. These small deviation values from the channel totals demonstrate that the radiometric correction process is robust to a large variation of instrument positioning.

5 PIXL Multispectral Data Products

The radiometrically calibrated images can be combined in several ways to better understand the texture and mineralogy of rocks. The greyscale images in Fig. 14 can be hard to interpret, however, combining the channels NIR, Green and Blue to compose a color image eases interpretation. Examples of the color images of the flat and natural targets are shown in Fig. 22. Additionally, the reconstructed color images in Fig. 22 shows that our algorithm can faithfully reproduce the blue color seen visually Table 1. The difference between the blue color in the natural target and the calibration target is the material property. The blue paint in the calibration target contains materials reflecting strongly in the NIR range.

Fig. 22
figure 22

Radiometrically corrected images from the multispectral dataset. Top row shows the flat target and bottom row shows the natural target. The left column shows a color composed image using NIR, Green and Blue channels. Right column shows the UV channel separately and scaled accordingly. The red enclosure marks the region at which the flat fielding gain is below the threshold as described in Sect. 3.3

Full size image

Another analysis tool applies the ratio between two individual color channels, generating a ratio map. An example of three ratio maps of the natural target is shown in Fig. 23. The deep blue color of azurite in the natural target clearly stands out in all ratios and can easily be isolated for studies of specific regions. More complex classification methods using several ratios are also possible. These ratio maps highlight the spectral properties specific to the spectral bands, making the separation of different areas of the targets easier. This is highly useful in the segmentation and classification of different areas of a Martian rock (Liu et al. 2022; Tice et al. 2022).

Fig. 23
figure 23

Ratio maps of natural target. Left image shows ratio B/NIR. Center Image shows B/G. Right image shows ratio B/UV

Full size image

Furthermore, the ratio maps enable one-to-one comparison between different targets, as a relative measure of the spectral properties can be directly compared.

The ratios can also be interpreted and compared in a heatmap, as shown in Fig. 24, where a scatterplot of the ratios B/NIR and G/UV of the flat target are shown. Images below illustrate what different areas of the scatterplot represent.

Fig. 24
figure 24

Scatterplot of the B/NIR and G/UV ratios of all pixels in the flat target with dashed ellipses marking the regions of interest that represent distinct phases in the target. Red ellipse shows jasper (hematite bearing chert), orange ellipse shows areas containing Fe-bearing carbonate, and the green ellipse shows mostly Fe-poor chert

Full size image

6 In-Flight Environment Discussion

The novel radiometric correction method described in this manuscript has been applied and tested on calibration standards in a controlled environment at 20 °C. In order to apply this method to in-flight data captured in the Martian surface environment, thermal effects need to be considered. The center wavelength of the LEDs and radiant intensity change as a function of temperature (Klevang et al. 2023). Large shifts of the central wavelengths would be of interest for analysis purposes when comparing targets, however, the shifts of the central wavelengths are small (largest being NIR ranging from 709.8 nm to 719.9 nm) in the diurnal thermal envelope of about -70 °C to 0 °C that Perseverance experiences in the Jezero Crater. Both the shifts of the central wavelengths and the change in total radiated intensity is compensated for by the automated exposure control of the MCC, as described in Sect. 3.1 and 3.4. For instance, say an image is captured at a temperature that makes the LEDs radiate less light, this will cause the exposure control to extend the shutter time and thereby compensate for the less illumination. Similarly, if the CCD spectral response changes due to shift in temperature so will the shutter time. The thermal stability of the FLI and MCC optics was tested during the instrument’s Thermal Vacuum Tests (TVAC). There was no effect to be observed and is therefore considered negligible. The in-flight performance of the multispectral data when using the radiometric correction method described in this manuscript will be evaluated in a future collected in-flight performance manuscript for the OFS.

7 Conclusion

This paper describes a novel process of radiometric correction of raw images from the PIXL Micro Context Camera, and the associated performance level of the system. The radiometric correction process enables multispectral analysis at microscopic resolution. The process includes 1) adjusting for dark level, 2) creating a topographic map from structured light measurement, 3) creating and applying flat fielding profiles, 4) compensation for the different shutter times of the individual color channels, 5) compensation for individual LED radiated power as a function of the forward current, and 6) compensation for the quantum efficiency of the CCD and intensity of the LEDs of each color.

The processing method was verified by comparing the MCC spectral reflectance of calibrated color standards that are a part of the Mastcam-Z geoboard to TANAGERs reflectance spectra of those standards. The mean offset of all channels is 0.4% with the largest offset being 1.9% when scaled to the NIR channel. The robustness of the radiometric correction method was tested by observing the blue calibration target at different angles and standoff distances. The largest deviations from the mean standard deviations were found to be: 1.3%, 0.44%, 0.96% and 0.66% for the NIR, green, blue and UV channel respectively. The largest mean deviations of the Green, Blue and UV channel means from the total respective mean were found to be 1.4%, 2.6% and 2.3% respectively. The mean deviation of the Green, Blue and UV channel means from the total respective mean were found to be 0.5%, 1.0% and 0.6% respectively. These performance levels are considered sufficiently low to proceed with scientific intepretation.

Application of the radiometric correction to MCC images acquired of rocks (flat/polished and natural/rough) showed that we can accurately measure the spectral properties of the rock surfaces, while being robust against changes in viewing angle and distance as well as surface topography. The four band color images show that MCC multispectral imaging can be reliably used for identifying spectrally distinctive phases in the rocks and further assess variations among different rocks. The higher spatial resolution and close-proximity from PIXL multispectral analysis complements that of MastCam-Z multispectral analysis of the outcrop, which helps with the fine scale interpretation of the texture and mineralogy. This enables PIXL to analyze Martian targets using independent X-ray fluorescence and multispectral observations in a combined analysis, thereby strengthening interpretations, hypotheses, arguments, and conclusions made from PIXL data.

Furthermore, the novel radiometric correction process, presented in this paper, is directly applicable outside the scope of PIXL and can be used for other multispectral instruments with active light sources.