Multi-angle polarimetric (MAP) measurements can enable detailed characterization of aerosol microphysical and optical properties and improve atmospheric correction in ocean color remote sensing. Advanced retrieval algorithms have been developed to obtain multiple geophysical parameters in the atmosphere–ocean system. Theoretical pixel-wise retrieval uncertainties based on error propagation have been used to quantify retrieval performance and determine the quality of data products. However, standard error propagation techniques in high-dimensional retrievals may not always represent true retrieval errors well due to issues such as local minima and the nonlinear dependence of the forward model on the retrieved parameters near the solution. In this work, we analyze these theoretical uncertainty estimates and validate them using a flexible Monte Carlo approach. The Fast Multi-Angular Polarimetric Ocean coLor (FastMAPOL) retrieval algorithm, based on efficient neural network forward models, is used to conduct the retrievals and uncertainty quantification on both synthetic HARP2 (Hyper-Angular Rainbow Polarimeter 2) and AirHARP (airborne version of HARP2) datasets. In addition, for practical application of the uncertainty evaluation technique in operational data processing, we use the automatic differentiation method to calculate derivatives analytically based on the neural network models. Both the speed and accuracy associated with uncertainty quantification for MAP retrievals are addressed in this study. Pixel-wise retrieval uncertainties are further evaluated for the real AirHARP field campaign data. The uncertainty quantification methods and results can be used to evaluate the quality of data products, as well as guide MAP algorithm development for current and future satellite systems such as NASA’s Plankton, Aerosol, Cloud, ocean Ecosystem (PACE) mission.

Satellite remote sensing has revolutionized Earth observation capabilities and plays a significant role in studying atmosphere, ocean, and land systems. Remote sensing techniques have advanced rapidly to provide highly accurate geophysical property retrievals by utilizing the rich information content of observations at multiple spectral bands, viewing angles, and polarization states. Multi-angle polarimeters (MAPs) are particularly well suited to characterize aerosol microphysical properties

Joint aerosol and ocean color retrieval algorithms have been developed for a variety of spaceborne and airborne MAPs such as the Polarization and Directionality of the Earth’s Reflectances (POLDER) instruments

Uncertainty quantification is an integral part of retrieval algorithm development. The uncertainties of the retrieved products (hereafter “retrieval uncertainties”) are key to understanding retrieval performance, gauging whether the algorithm provides results of useful quality, and guiding where further efforts for improvement are best focused. In this study, we define retrieval error as the difference between the retrieval results and truth (whether synthetic data or external reference data), and we define retrieval uncertainty as the standard deviation (

However, theoretical uncertainties derived from these techniques often represent a best-case scenario as they rely on several assumptions (discussed by

In short, theoretical uncertainties provide pixel-wise estimates of performance for every parameter, while real uncertainties provide a more complete assessment of performance, but with limitations due to the availability of high-quality reference data. The two are a natural complement as ground-truth data or simulated retrievals provide an avenue to evaluate theoretical uncertainties in a statistical sense. A statistical (not one-to-one) comparison is necessary because a retrieval with associated uncertainty represents a range of plausible values of a geophysical quantity, whereas an individual reference truth has a definite value. Several approaches have been proposed to address the question of whether the distribution of observed retrieval errors is consistent with the distribution as expected from the theoretical uncertainty

In this paper, we discuss theoretical uncertainties from MAP retrievals over a coupled atmosphere and ocean system, and then we propose a flexible framework to validate these theoretical uncertainties against real uncertainties. The following topics will be addressed in this work.

This will be assessed not just for properties retrieved directly from the MAP data, but also derived properties such as aerosol optical depth (AOD), single scattering albedo (SSA), and various aspects of the derived water-leaving signals. To quantify the performance in this study, random errors are sampled from theoretical pixel-wise uncertainties using a Monte Carlo method, and results are compared with the real errors.

Uncertainty evaluation often requires Jacobian matrix and derivative calculations, which can be computationally expensive. To achieve optimal speed within the framework of this work, all Jacobian matrix and derivatives are evaluated analytically using automatic differentiation based on neural networks.

The input uncertainty model includes two main components: (a) measurement uncertainties, which are mostly characterized by instrument calibration uncertainties, and (b) forward model uncertainties, which refer to whether the forward model can sufficiently describe the measurements.

This work focuses on the first two topics. The third topic has been partially addressed using an adaptive angular screening approach, described in

The FastMAPOL algorithm

PACE will carry three instruments that are expected to advance our characterization of the atmosphere, ocean, and land states

The total measured reflectance (

Vector radiative transfer models (VRTMs) are used to simulate the reflectance and polarization over a coupled atmosphere and ocean system

The atmospheric model for the airborne measurements consists of a combination of aerosols and air molecules from surface to 2 km, an aerosol-free molecular layer (i.e., Rayleigh scattering) above that, and (for the airborne AirHARP instrument) an additional aerosol-free layer above the aircraft altitude. A total of 15 geophysical parameters, shown in Table 1, are used as inputs to the forward model. The solar and viewing geometries are represented by the solar and viewing zenith angles (

Parameters used to train the FastMAPOL forward model as described in Sect. 2.2. The minimum (min) and maximum (max) values of each parameter are also shown. The a priori uncertainties (

The aerosol size distribution is assumed as a combination of five lognormally distributed aerosol sub-modes, each with prescribed mean radii and variances; the five volume densities (

Ozone absorption is quantified by the ozone column density (

NN uncertainties

The optimal values of retrieval parameters are obtained using a maximum likelihood approach by minimizing the difference between the measurements and the forward model fit represented by a cost function

The subspace trust-region interior reflective (STIR) algorithm is employed to conduct non-linear least-square minimization of the cost function

The propagated (theoretical) pixel-wise uncertainty quantification is based upon a Bayesian approach which assumes Gaussian distributions of input uncertainty (including measurements, forward model, and a priori) and output (retrieval) uncertainty

The

Other Bayesian inference methods exist that are capable of deriving retrieval uncertainties without explicitly computing the Jacobian matrix or requiring that uncertainties be Gaussian. For example,

Verifying theoretical uncertainty estimates is necessary because real retrieval performance depends on other factors. A key factor is how well the inversions converge to the global minimum of the cost function instead of a false convergence to a local minimum. This is not captured by Eq. (

accuracy of the forward model and Jacobian matrix;

tolerance for iterative optimization, which may impact how early the iterative parameter updates stop;

the possibility that retrievals may get stuck at parameter boundaries, if not adequately treated in the inversion algorithm;

the possibility that the input uncertainty model may be insufficient, leading to inappropriate weights of different measurements in the cost function; and

false convergence from non-monotonic cost functions due to insufficient information in the measurements.

To evaluate the performance of the uncertainty quantification using error propagation, we can compare theoretical uncertainty with the uncertainties calculated by comparing the final retrieval results with reference truth values. Two useful metrics, the mean absolute error (MAE) and the root mean square error (RMSE) between the truth (

MAE is more robust to outliers than RMSE, so comparing the two can be informative as to whether the overall error distribution is close to Gaussian. MAE has also been shown to be less dependent on the number of cases considered than RMSE

Chlorophyll

Demonstration of the procedures to compare theoretical and real uncertainties.

Direct comparison of theoretical uncertainties and real errors is difficult because the former is a measure of the estimated dispersion of the retrieval in terms of a distribution of

The goal is to generate a statistical distribution of the retrieval error (defined as the difference between retrieval and truth) for both theoretical and real uncertainties and to develop proper metrics for comparison based on the distribution. Steps involved in MCEP are listed below using the example in Fig. 1.

Conduct retrievals and compute theoretical retrieval uncertainties according to the error propagation method discussed in Sect. 3.1. Here AOD is derived from the directly retrieved refractive indices and volume densities shown in Table 1, and

Generate a distribution of random theoretical errors. This is done by taking the theoretical uncertainty for each retrieval and generating a random number from a Gaussian distribution with a zero mean and a standard deviation equal to the theoretical uncertainty (i.e., individual points from Fig. 1a). This random number will be the theoretical retrieval error for the corresponding theoretical retrieval uncertainty. These sampled random errors are shown in Fig. 1c.

The real retrieval errors, shown in Fig. 1d, are calculated as the difference between the retrieval results and truth data. Figure 1c and d showed similar dependency on the AOD.

The histograms for the error data in Fig. 1c and d are compared in Fig. 1e, which shows directly comparable statistical distributions. These distributions can be analyzed using metrics such as RMSE and MAE in Eqs. (10) and (11).

Evaluate the variations of the uncertainty metrics derived from step 4: (1) generate multiple sets of random theoretical errors following step 2; (2) compute the metrics for each set of errors; and (3) compute

The MCEP method enables direct comparison of error distributions between theoretical uncertainties and real retrievals, which therefore provide additional flexibility in analyzing their statistics. For the example in Fig. 1e, the peak of real retrieval errors is

Furthermore, following step 5 in MCEP, we can analyze the uncertainties of MAE with respect to a set of random errors. MAE values for 50 sets of random theoretical errors are computed as shown in Fig. 1f. The relative standard deviation of these MAE values is about 3 % when all 1000 cases are used. The relative uncertainties increase to 7 % and 12 % when the number of cases are reduced to 200 and 50. Therefore, for discussion in the next section with a smaller number of cases considered, it is useful to understand how much the MAE varies. A similar approach can be applied to comparisons with high-quality in situ measurements. The same challenge is that the metrics such as RMSE and MAE may suffer from larger statistical variations if only a smaller number of retrieval cases are available.

To evaluate the retrieval capability of the FastMAPOL algorithm on the HARP instruments, we conducted studies on synthetic AirHARP and HARP2 data and then derived the pixel-wise retrieval uncertainties. The theoretical uncertainties are then compared with real uncertainties, and their difference is quantified using the MCEP methodology discussed in Sect. 3. The real uncertainties are derived from the retrieval results based on synthetic data which include impacts from local minima in the cost functions as summarized in Sect. 3.2; however, these synthetic data studies do not address the potential impacts of modeling errors in the forward model. To evaluate the assumption in the forward model, comparison with in situ measurements is required in future studies.

We performed radiative transfer simulations to generate 1000 synthetic sets of measurement using the coupled atmosphere–ocean VRTM

Realistic HARP-like viewing geometries are constructed as discussed in

Random noise is added to the 1000 sets of synthetic AirHARP and HARP2 measurements, and then the FastMAPOL retrieval algorithm is applied to them. The synthetic data are computed directly using the vector radiative transfer model, but the NN forward model is used in the retrieval algorithm to achieve maximum efficiency. In this way the contribution of the NN uncertainties is captured both in the simulation and the uncertainty model as shown in Eq. (

The histogram of the cost function values for the synthetic retrievals.

We apply the method discussed in Sect. 3 to compare theoretical and real uncertainties. An example of spectral AOD and

Example of AOD (solid line) and

Theoretical retrieval uncertainties estimated from error propagation plotted against the AOD at 550 nm (horizontal axis) for AOD, SSA, fine-mode volume fraction (fvf), refractive index (

For more general atmosphere and ocean conditions, Fig.

Histograms of the theoretical and real retrieval errors evaluated using the MCEP method in Sect. 3.2 for the same cases as in Fig.

Following the methodology proposed in Sect. 3.2, the statistical distributions of the retrieval errors are shown in Fig.

To quantify theoretical and real uncertainties, Fig.

The retrieval uncertainties represented by MAE averaged within several ranges of AOD at 550 nm, including [0.01, 0.1], [0.1, 0.2], [0.2, 0.3], [0.3, 0.4], and [0.4, 0.5]. The horizontal axes indicate the maximum AOD used in the corresponding AOD range. Results for both HARP2 and AirHARP are shown. Chl

The retrieval uncertainties for synthetic HARP2 and AirHARP datasets are close to each other for most retrieval cases as shown in Fig.

To understand the accuracy of the MAE as derived above for each AOD range (each with around 200 cases), we generated multiple sets of random theoretical errors following step 5 in Sect. 3.2 and compared the averaged MAE with the MAE derived from real errors as shown in Fig.

Comparing the averaged MAE derived from theoretical and real uncertainties for both HARP2 and AirHARP. The error bars indicate the

Ratios between the averaged MAEs for the real and theoretical uncertainties over five AOD intervals from Fig.

Ratio of real to theoretical retrieval MAE for the data shown in Fig.

The pixel-wise theoretical uncertainties achieve a reasonably good performance to represent real retrievals as discussed in the last two sections. Their performances on various retrieved geophysical properties are quantified by comparing with the real retrieval errors. Based on these results, in this section, we will use the theoretical uncertainties to analyze the retrieval results from AirHARP field measurements from the Aerosol Characterization from Polarimeter and Lidar (ACEPOL) field campaign conducted from October to November of 2017, where the NASA’s ER-2 aircraft carried four MAPs – AirHARP, AirMSPI, SPEX airborne, and RSP – and two lidar sensors – HSRL-2

There are a total of five AirHARP ocean scenes available in ACEPOL. Three scenes on 23 October 2017 (Scenes 1, 2, and 3) have been discussed by

Figure

Retrieved AOD at 550 nm and their uncertainties along the three lines shown in Fig.

Figure

Three AirHARP scenes on 23 October, 27 October, and 7 November 2017, which are in different flight directions but over the same region. The RBG images are shown in panels

Equivalent results for the other three scenes (3, 4, 5) are shown in Fig.

Quantifying the uncertainties associated with remote sensing retrievals is key to understanding retrieval performance and gauging the quality and utility of the retrieval results. Retrieval uncertainties depend on the spectral, angular, radiometric, and polarimetric characteristics of the instrument. Increasing dimensionality and accuracy of measurements benefits retrievals but also introduces new challenges in the inversion of geophysical properties and estimation of retrieval uncertainties.

This study discussed and applied a practical, efficient way to estimate theoretical uncertainties for aerosol and ocean data products retrieved by FastMAPOL from synthetic AirHARP and HARP2 measurements, as well as field AirHARP measurements from the ACEPOL field campaign. Theoretical retrieval uncertainties for aerosol and ocean color properties are discussed. The speed with which the uncertainties can be computed is optimized using analytical derivatives based on automatic differentiations. To validate how well the retrieval uncertainties represent real retrievals, we provided a flexible Monte Carlo error propagation (MCEP) method to compare the retrieval uncertainties from error propagation with errors from synthetic retrievals. More discussions are as follows.

Using MCEP, statistical distributions can be compared to understand their properties and develop proper metrics for comparison. The real and theoretical retrieval uncertainties for multiple retrieval parameters are compared directly by their error histograms sampled from the Monte Carlo method based on the synthetic data retrievals. The ratios of the statistical metrics such as MAE for theoretical and real errors are computed and compared. These ratios provide a tool to quantify the overall performance of the retrieval uncertainty. The ratios are mostly 1–1.5 with respect to different AOD ranges, which suggests that the FastMAPOL retrieval algorithm performs well as it approaches the optimal uncertainties predicted from error propagation. The larger ratios observed for aerosol refractive indices suggest a need to improve constraints on and/or test for proper convergence of those parameters, especially for cases with small AODs. Future studies of synthetic data with realistic statistics are needed to further evaluate the overall performance of the retrieval algorithm.

Synthetic data are only one piece of the evaluation and are limited because they use the same underlying forward model as the retrieval. Future comparison of retrieval results with in situ measurements is desirable to provide a more complete assessment. However, what is available at present for AirHARP is sparse in volume, as AirHARP data are only available for a few field campaigns and PACE has not yet launched. Notably, there is no avenue to validate all retrieved products at once. The MCEP method and others (e.g.,

The Monte Carlo method has been used widely for uncertainty quantification due to its flexibility and robustness (e.g.,

Retrieval initialization and convergence can be important.

This work provides a general framework to understand the uncertainties from the retrieval algorithm and provides a bridge from theoretical uncertainty toward future evaluation using in situ measurements. More complex input uncertainty model, such as the one including uncertainty correlations between the multi-angle measurements, can be evaluated based on this framework. Although based on synthetic and airborne measurements, the methods on uncertainty quantification are flexible and can be applied to existing and future satellite missions such as NASA’s PACE mission with advanced multi-angle polarimetric instruments.

Fast speed to compute retrieval uncertainties is useful for operational processing and analyzing satellite data. Although the error propagation method used in this study is already very efficient, it is still challenging to achieve a speed complementary to the retrievals due to the requirement to compute Jacobian matrix and multiple additional derivatives for parameters not directly retrieved as shown in Eq. (

The AirHARP and HSRL-2 data used in this study are available from the ACEPOL data portal (

MG, KK, BAF, and PWZ formulated the original concept. MG developed the algorithm and generated the scientific data. PWZ developed the radiative transfer code used in the simulations. KK, AMS, AI, YH, and OH advised on the uncertainty models. KK, PWZ, AMS, BC, and OH advised on the aerosol products. BAF, AI, and PJW advised on the ocean color products. VM and XX provided and advised on the HARP data. MG wrote the manuscript draft. All authors provided critical feedback and edited the manuscript.

At least one of the (co-)authors is a member of the editorial board of

Publisher’s note: Copernicus Publications remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

The authors would like to thank the ACEPOL teams for conducting the field campaign, thank the HARP and HSRL teams and PIs for providing the data, and thank the NASA Ocean Biology Processing Group (OBPG) system team for supporting the high-performance computing (HPC).

Meng Gao, Kirk Knobelspiesse, Bryan A. Franz, Andre M. Sayer, Amir Ibrahim, Brian Cairns, and P. Jeremy Werdell have been supported by the NASA PACE project. Peng-Wang Zhai and Yongxiang Hu have been supported by NASA (grant no. 80NSSC20M0227). The ACEPOL campaign has been supported by the NASA Radiation Sciences Program, with funding from NASA (ACE and CALIPSO missions) and SRON. Part of this work has been funded by the NWO/NSO project ACEPOL (project no. ALWGO/16-09).

This paper was edited by Piet Stammes and reviewed by Feng Xu and two anonymous referees.