The 2017 National Academy of Sciences Decadal Survey highlighted several high-priority objectives to be pursued in the decadal timeframe, and the next-generation Cloud, Convection and Precipitation (CCP) observing system is thereby contemplated. In this study, we develop a suite of hybrid Bayesian algorithms to evaluate two CCP remote sensor candidates including a W-band cloud radar and a (sub)millimeter-wave radiometer with channels in the 118–880 GHz frequency range for capability in constraining ice cloud microphysical quantities. The algorithms address active-only, passive-only, and synergistic active–passive retrievals. The hybrid Bayesian algorithms combine the Bayesian Monte Carlo integration and optimization process to retrieve quantities with uncertainty estimates. The radar-only retrievals employ the optimal estimation methodology, while the radiometer-involved retrievals employ ensemble approaches to maximize the posterior probability density function. A priori information is obtained from the Tropical Composition, Cloud and Climate Coupling (TC4) in situ data and CloudSat radar observations. End-to-end simulation experiments are conducted to evaluate the retrieval accuracies by comparing the retrieved parameters with known values. The experiment results suggest that the radiometer measurements possess high sensitivity to ice cloud particles with large water content. The radar-only retrievals demonstrate capability in reproducing ice water content profiles, but the performance in retrieving number concentration is poor. The synergistic observations enable improved pixel-level retrieval accuracies, and the improvements in ice water path retrievals are significant. The proposed retrieval algorithms could serve as alternative methods for exploring the synergistic active and passive concept, and the algorithm framework could be extended to the inclusion of other remote sensors to further assess the CCP observing system in future studies.

The 2017 National Academy of Sciences Decadal Survey

The properties of ice clouds are among the critical geophysical variables in the CCP science objectives. Ice clouds play a significant role in modulating the energy budget of the earth system by absorbing upwelling long-wave radiation emitted from the lower troposphere and reflecting incoming solar short-wave radiation

The radiative effects of ice clouds depend on the vertically integrated quantities and the vertical distribution of ice particle characteristics

Several retrieval algorithms have been developed specifically for ice cloud radiometry studies. All applicable algorithms that could be generally classified as statistical approaches and optimization approaches are under the framework of Bayes’ theorem. The statistical approaches, including the Bayesian Monte Carlo integration (MCI)

The objective of this paper is to develop candidate retrieval algorithms for synergistic radar and radiometer observations in order to quantitatively assess the capability of the next-generation ACCP observing system in constraining ice cloud geophysical variables. The algorithms for active-only, passive-only, and synergistic retrievals are developed under a hybrid Bayesian framework that combines the Bayesian MCI and optimization process to retrieve ice cloud quantities with uncertainty estimates. This paper is structured as follows: in Sect. 2 we provide an overview of the candidate ACCP remote sensors and present the simulated active and passive observations on the reference cloud scenes using the radiative transfer model. Section 3 describes the hybrid Bayesian algorithms for the radar-only, radiometer-only, and synergistic retrievals in detail, followed by Sect. 4, which describes the retrieval database using the statistics from in situ data and CloudSat radar observations. Section 5 presents the retrieval simulation experiments and a quantitative evaluation of the retrieval results. Finally, Sect. 6 presents the summary and conclusions.

The remote sensors we evaluate in this study include a W-Band (94.05 GHz) radar and a (sub)millimeter-wave radiometer both of which are candidates in the ACCP observing system. The W-band radar is nadir-looking and it is similar to the Cloud Profiling Radar (CPR) in the CloudSat satellite

This study focuses on developing synergistic retrieval algorithms for situations where the active and passive observations are coincident. Based on this purpose, although the radar and radiometer instruments have different horizontal resolution and scanning modes, both sensors are assumed to have the same nadir-looking pencil beam and the capacity of high horizontal resolution to achieve the same fields of view in the simulation experiments below. The influence of the footprint and viewing geometry will be addressed in future work once more characteristics are known.

Channel characteristics of the ACCP candidate (sub)millimeter-wave radiometer based on the information disclosed on

Simulated clear-sky brightness temperature spectrum in a tropical atmospheric scenario. All ACCP radiometer channel positions and a detailed view of the double sidebands located on either side of a central frequency are present.

The reference cloud scenes are obtained from the numerical Environment and Climate Change Canada (ECCC) model

We develop the one-dimensional forward model for both active and passive simulations based on the Atmospheric Radiative Transfer Simulator (ARTS)

Figure 2 shows the vertical distribution of IWC and NC for the selected reference cloud scenes along a latitudinal transect and the corresponding simulated W-band radar reflectivity observations. Compared to the NC, the radar reflectivity simulations show a stronger tendency to follow the variations of IWC. Figure 3 shows the ice water path (IWP) and the corresponding BT simulations for different channels of the ACCP candidate radiometer. The correlations between the IWP changes and BT depressions are evident. The channels with higher central frequencies are more sensitive to the change of the water path, especially when the IWP is around

Vertical distribution of water content (WC) and number concentration (NC) for ice and snow particles along the selected latitudinal transact and the corresponding W-band radar reflectivity simulations. The radar simulations are computed using the Atmospheric Radiative Transfer Simulator (ARTS) forward model.

Integrated ice cloud water content of the selected latitudinal transect and the corresponding brightness temperature simulations for the candidate ACCP radiometer's channels.

Scatterplot of the brightness temperature difference between simulations in the clear-sky and cloudy conditions as a function of ice water path for all ACCP radiometer channels.

We developed different hybrid Bayesian algorithms for the radar-only, radiometer-only, and synergistic retrievals. All hybrid algorithms combine Bayesian MCI with optimization processes to retrieve quantities and uncertainty estimates. Bayesian MCI introduces prior information by generating an ensemble of atmospheric cases that are distributed according to the a priori PDF, and it is highly efficient since the retrieval database is precalculated and additional forward model calculations are not required. By assuming the uncertainties for different measurement variables to be independent, the conditional PDF can be written as

The biggest challenge for the Bayesian MCI is the sparsity in the measurement space for a retrieval database with a finite number of random samples. If we increase the length of the observation vector or decrease the measurement uncertainties, the number of database cases matching the observation vector becomes smaller and the Bayesian MCI fails. When this happens, the optimization process is begun to maximize the posterior PDF.

The robust and efficient OEM method is employed as the optimization algorithm for radar-only retrievals. The fundamental assumptions of the OEM algorithm are that the forward model is moderately nonlinear and that both the prior PDF and conditional PDF are Gaussians. OEM maximizes the posterior PDF by minimizing the following cost function

The radiometer-involved retrievals that include the synergistic and radiometer-only retrievals do not apply the OEM algorithm in this study. The OEM algorithms involving BT measurements were developed in the following two studies. The first study, by

The synergistic radar and radiometer retrievals are done by extending the radar OEM algorithm to add the radiometer observations. The radar OEM algorithm provides the retrieved values as well as the associated uncertainty estimations formulated in Eq. (4). Following this step, the Cholesky decomposition is implemented on the covariance matrix to generate an ensemble of correlated random noise

In this study, an ensemble of 500 cases is generated using the Cholesky decomposition to statistically investigate the additional benefits from the BT information. The Bayesian MCI step requires a minimum number of cases (25 in the retrievals below) matching the BT observation within a specified

We employ the ensemble probability estimation (EnPE) algorithm as the optimization procedure for the radiometer-only retrievals. The EnPE algorithm was first proposed by

Flowchart of the ensemble probability estimation (EnPE) algorithm applied in the radiometer-only retrievals.

We describe the EnPE algorithm in detail here to incorporate improvements in many aspects and to make the algorithm more understandable. The EnPE algorithm stochastically explores the state space by sampling an explicit PDF estimated from promising weighted cases obtained so far from the perspective of Bayesian MCI. As the flowchart in Fig. 5 shows, the algorithm consists of two modules: the PDF estimation module numerically estimates the unknown continuous posterior PDF using the discrete cases with posterior values in the last ensemble, and the PDF sampling module synthesizes new cases according to the accumulated PDF using the resampling approach and the covariance matrix.

Starting from the situation where too few a priori database cases match the observations, the PDF estimation module artificially inflates the measurement uncertainties so that there are enough matches between the observation vector and the BT simulations from the a priori profiles, and the conditional PDF is computed by:

Following this step, the PDF sampling module reselects the samples according to their posterior value to multiply cases with high weights and eliminate cases with low weights. The weights of the selected state vectors become equivalent again. The sampling module then generates correlated random noise using the two-point correlation statistics in the covariance matrix. The covariance matrix of the retrieved quantities is computed using the posterior PDF based on Bayesian MCI:

Once a new ensemble is synthesized and the corresponding BT simulations are computed, the algorithm evaluates the new state samples based on the prior PDF and conditional PDF, and the optimization cycle starts again. As the iteration proceeds, the ensemble evolves and gradually becomes concentrated in the most likely area, compensating for the sparse distribution of the original retrieval database. The cases in the last ensemble are used to calculate the mean parameter values (retrieved values) and standard deviations (retrieved uncertainties) by Bayesian MCI. The EnPE iteration stops when a required number of cases (25 in this study) within the

We upgrade the precalculated retrieval database with the random cases distributed according to the a priori PDF. In

The method to calculate the prior PDF is consistent with the control vector transformation concept applied by Evans et al. (2012). The CDFs are used to capture the one-point statistics of the Bayesian retrievals that combine the remote sensing data and in situ microphysics by sorting different ice cloud parameters at different layers from smallest to largest in value and calculating the sum of the assigned equal probabilities up to each datum. The original ice cloud parameters are then represented by their percentile ranks, and the correlations are also preserved in the rank matrix. Following that, the percentile rank matrix is transformed into a Gaussian derivate matrix using the standard normal cumulative distribution function:

In this way, more realistic ice cloud statistics in arbitrary functional forms are added to the EnPE algorithm as regularizations to make the algorithm more applicable.

We conduct simulation experiments to assess the synergistic radar and radiometer capability in retrieving ice cloud parameters. The measurement space in the retrieval experiments consists of the noisy radar reflectivity simulations at vertical grid points and the noisy BT simulations of different radiometer channels. Independent Gaussian noise with 1.5 dBz standard deviation characterizing the radar measurement accuracy is added to the simulated radar reflectivity observations, and 4 dBz reflectivity uncertainty that accounts for estimations of the forward model uncertainty due to unknown ice hydrometeor bulk density is assumed during the radar retrieval process. The 4 dBz error estimation is based on the study of

The state space in all three retrievals consists of the IWC and NC profiles using the same vertical grids as the reference cloud scenes. The vertical resolution is 250 m. Other atmospheric parameters such as water vapor, temperature, and pressure profiles are set to the true values during the retrieval. For the radar-only and synergistic retrievals, the ice cloud parameters are transformed into lognormal distributions, which means the state variables are

The key element in implementing the Bayesian MCI is to build the retrieval database, which generally consists of two steps: creating random atmosphere and ice cloud properties that are distributed according to the prior PDF and computing the simulated radar reflectivity or BT using the forward model. In this study, we separately develop two retrieval databases for radar and radiometer retrievals using the a priori statistics from TC4 in situ measurements and CloudSat CPR observations.

The realistic ice cloud microphysical probability distributions used for building the radar retrieval database is obtained from the in situ data from instruments flown in the TC4 campaign. The in situ ice PSD are obtained from the two-dimensional stereo (2D-S) probe and the precipitation imaging probe (PIP), and the associated temperature is measured by the Meteorological Measurement System on the DC8 aircraft platform. The bimodal PSD scheme which approximates both small and large particle distribution modes by gamma functions is used to fit the in situ data, and the ice cloud parameters, including IWC, NC, and particle size, are derived. More details on the TC4 in situ analysis can be found in

Ice particle microphysical statistics defining the a priori Gaussian probability distribution derived from the TC4 in situ data.

The radar retrieval database is used to generate the initial state vector for the radar-only OEM retrieval algorithm based on the Bayesian MCI. This step helps the OEM algorithm to better satisfy the fundamental requirement for a moderately nonlinear forward model. The initial state vector generation step proceeds from top down, and the generated radar attenuation is used to correct the radar reflectivity below. The a priori Gaussian PDF listed in Table 2 is also used in the OEM algorithm as the regularization. We should note that the a priori Gaussian PDF contains single-layer constraints, but it does not describe the vertical correlations between ice cloud microphysics at different layers.

Figure 6 shows the two-dimensional histogram for the microphysical quantities and reflectivity simulations in the radar retrieval database. A fairly strong correlation between IWC and NC over the whole range is observed in the left panel. The middle panel and the right panel indicate that the radar reflectivity simulations have a strong correlation with IWC in the whole range, but its correlation with NC is much weaker.

Two-dimensional histogram for the microphysical quantities and the W-band radar reflectivity simulations derived from the random cases in the precalculated radar retrieval database.

Apart from using the TC4 in situ microphysical statistics, we also use the CloudSat observations to acquire the critical coherent vertical correlations to synthesize the random ice cloud profiles for creating the radiometer retrieval database. The data we use include CloudSat radar reflectivity, CALIPSO lidar cloud fraction, and the corresponding ECMWF profiles of temperature and relative humidity. As mentioned in Sect. 3.2.2, the active remote sensing profiles are first combined with the TC4 cloud microphysical probability distributions using the Bayesian MCI algorithm, and then the CDFs/EOFs procedures are applied to create a required number of synthetic microphysical and thermodynamic profiles (100 000 profiles in this study) using the one-point and two-point statistics that are captured from the Bayesian retrieval results. More details can be found in

Figure 7 shows the profiles of IWC, NC, temperature, and relative humidity for seven percentiles in the cumulative distributions. Layers that are identified as clear are added with random Gaussian noise to prevent discontinuity in the CDFs. The mean values for the added IWC and NC noise are

Profiles of ice water content (IWC), number concentration (NC), temperature, and relative humidity for seven percentiles in the cumulative distributions for the random atmospheric/cloud profiles in the precalculated radiometer retrieval database.

The precalculated retrieval database provides a good opportunity for estimating the degrees of freedom (DoF) for the ACCP (sub)millimeter-wave radiometer. The DoF describes the number of independent pieces of information in the radiometer measurement since some channels provide redundant information. The DoF is usually calculated as the trace of the averaging kernel matrix based on the Jacobian matrix

Figure 8 shows the DoF of the ACCP radiometer as the function of the IWP and integrated water vapor (IWV) using the measurement noise characteristics listed in Table 1. The DoF is computed only when the number of random cases in a certain IWV–IWP bin is larger than 10 to avoid noise interference. It can be seen that the DoF increases with IWP, but it decreases as the IWV becomes large. For the wet atmospheres with IWV larger than 42 kg m

The degrees of freedom (DoF) for the ACCP radiometer candidate as a function of the ice water path (IWP) and integrated water vapor (IWV). The DoF is estimated using the radiometer retrieval database that has 100 000 random ice cloud profiles. Liquid hydrometer species are not included in the retrieval database.

Comparison between the true values and the retrieval results for ice water content and number concentration profiles along the selected latitude transect. The results for radar-only, radiometer-only, and combined retrievals are presented sequentially.

In this section we present the analytical results for the radiometer-only, radar-only, and synergistic retrievals to assess the capability of the objective ACCP remote sensors in retrieving ice cloud parameters. The retrieval experiments are performed by inputting the simulated noisy radar reflectivity and BT observations into the hybrid Bayesian algorithms and then comparing the retrieved parameters with the true values to determine the retrieval accuracy.

Figure 9 shows a side-by-side comparison between the true values and the retrieval results for IWC and NC profiles along the ECCC model transect. The results for the radar-only, radiometer-only, and combined retrievals are presented sequentially. The passive-only retrieval results suggest that there is very little if any information regarding the vertical distribution of ice cloud microphysics in the radiometer measurements when considered without the radar measurements. For the active-only retrievals, the retrieved IWC profiles realistically reproduce the vertical structure of the reference cloud scenes. The retrieved values also correspond to the true values in general, even though sometimes the retrievals tend to underestimate the IWC values, especially near the top of the cloud ranging from 10 to 15 km in height. By contrast, the active-only retrievals for NC profiles perform much worse. The true NC values cover the range from 10 to over

Figure 10 shows the retrieved IWP values for the passive-only, active-only, and combined retrievals based on the hybrid Bayesian algorithms along the latitudinal transect. For the passive-only retrievals, the retrieval errors are comparable to the active-involved retrievals over the entire range. The active-only retrievals show the tendency to overestimate the IWP for thin clouds but underestimate the thick cloud IWP. The combined retrievals are based on the radar OEM results, and substantial improvements in IWP retrieval accuracy can be seen after adding the ACCP BT measurements.

Direct comparison between the retrieved ice water path (IWP) and the true values along the latitudinal transect. The passive-only, radar-only, and combined retrievals are all displayed.

Figure 11 shows the scatterplots of the retrieved parameters against the true values that are colored by density to further visualize the retrieval performance. The statistical IWC analysis below only applies to the grid points with the reference IWC larger than

The scatterplots of the retrieved parameters against the true values that are colored by density. The scatterplots for ice water content (IWC), number concentration (NC), and ice water path (IWP) are shown in different columns, and the plots for passive-only, active-only, and combined retrievals are shown in different rows.

Figure 12 displays the PDF of the logarithmic errors for different parameters under different retrieval methods. The logarithmic error is defined as:

The probability density function (PDF) of the logarithmic errors for different ice cloud parameters under different retrieval methods.

Figure 13 shows the quantitative values measuring logarithmic error distribution to compare the retrieval accuracy under different retrievals. Panels Fig. 13a and b show the mean values of the logarithmic errors and the associated IQR. The IQR is defined as the difference between the 75th and 25th percentile. The mean and IQR values were also presented in Fig. 11 in

The quantitative statistics of the logarithmic errors for the retrieved ice cloud quantities. Panels

In this study, we develop a suite of hybrid Bayesian retrieval algorithms to assess a candidate observing system representative of what is being considered for the decadal survey clouds-convection-precipitation designated observable mission to be flown later this decade. We specifically evaluate the capability of an observing system consisting of a W-band radar and a (sub)millimeter-wave radiometer in constraining the ice cloud microphysics. Our purpose is to demonstrate the value of single-instrument and synergistic retrievals of ice cloud microphysical parameters. Several new algorithms are proposed here, which could serve as alternative solutions for exploring the synergistic active and passive retrieval concepts for the actual instruments once they are known. The geophysical variables we investigate include the IWC, NC, and IWP. The hybrid Bayesian algorithms combine the Bayesian MCI and optimization processes to compute retrieval quantities and associated uncertainties. The radar-only retrievals employ the OEM optimization algorithm that uses gradient information to minimize the cost function. The OEM is initialized by a state vector that is constructed by implementing Bayesian MCI to the radar reflectivity at different grid points using the precalculated radar database. The necessary Jacobian matrix is calculated by perturbing the ice cloud microphysical quantities on different layers. The radiometer-involved retrievals employ ensemble strategies to optimize the ill-posted problem. The synergistic radar and radiometer retrievals are done by generating random cases from the radar OEM results based on the Cholesky decomposition technique. The BT simulations are then computed, and the Bayesian MCI is implemented to derive the final retrieval results. For the radiometer-only retrievals, the EnPE algorithm is applied to statistically estimate the posterior PDF using the promising weighted cases. The estimation module and the sampling module proceed iteratively to stochastically explore the state space. In addition, a new approach to implement prior constraints that enable the a priori PDF to be highly non-Gaussian is proposed in order to make the ensemble algorithm more applicable.

We conducted simulation experiments to evaluate the accuracy of retrieving ice cloud quantities for different remote sensors. The simulated noisy radar reflectivity and BT observations are input to the hybrid Bayesian algorithms, and the retrieved parameters are compared with the known values to determine the retrieval accuracies. A tropical transect of cloud profiles that are simulated using the ECCC model is selected as the reference cloud scenes. This choice ensures the independence between the atmospheric/cloud profiles for testing and the vertical profiles in the a priori database. The simulation experiments assume that both sensors have the same nadir-looking viewing angle. The influence of different footprints and viewing geometries between the active and passive remote sensors are neglected in this initial study but will be evaluated once the specific parameters of the observing system are known. Since we do not consider the forward model uncertainties from various particle habits, the retrieval errors are much smaller than the results in

1. The radiometer measurements do not have direct information about the IWC and NC vertical distribution. However, the BT measurements possess high sensitivity for large ice cloud particles with IWC values larger than

2. The radar-only retrieval demonstrates capability in retrieving IWC profiles, but it literally does not exhibit capabilities in retrieving NC vertical distribution. The radar-only retrievals for IWP have comparable accuracies to the radiometer-only retrievals.

3. The synergistic retrievals have evident improvements in retrieval accuracies compared with the radar-only retrievals. When using the median of the absolute fractional error as the quantitative parameter to evaluate the retrieval accuracies, the relative improvements after adding BT measurements for IWC, NC, and IWP are 18 % and 12 %, and 42 %, respectively.

This paper provides an end-to-end idealized simulation experiment that sacrifices precise reality in order to demonstrate nuances in the various algorithms, and several disadvantages are worth mentioning. First, there are many simplifications on the reference cloud scenes and the radiative transfer model. We only use the frozen particles in the reference cloud scenes, and the liquid clouds are ignored. The impacts from water vapor uncertainties are also neglected. Further, only one particle habit is applied and the forward model uncertainties from particle habits and PSD are not considered. These simplifications facilitate the quantitative evaluation of the proposed retrieval algorithms without complication from parameters not yet known so that the relative benefit of the observing system is considered as separate instruments or as a synergistic set. In all cases the value of synergy is demonstrated although more realistic observing systems must be considered in future work. Second, the statistical characteristics are only derived based on selected atmospheric/cloud profiles along a single latitudinal transect. Since this subset with a finite number of profiles can hardly represent the realistic spatial distribution of ice cloud microphysics that will be encountered globally, the statistics we derive may differ from the characteristics of the entire possible atmospheric conditions. Third, apart from the W-band radar and the (sub)millimeter-wave radiometer, the eventual observing system will likely include other remote sensors that would be useful for improving retrieval accuracies for ice cloud remote sensing. For instance, the eventual radar system will likely be dual-frequency and add Ku- or Ka-band to the measurements. Also, highly accurate Doppler velocity measurements will likely be observed that may allow for constraints on the ice crystal bulk density that could significantly mitigate forward model uncertainties. These problems will be investigated in future work.

The retrieval algorithm modules and the codes used to produce the results in this study are available upon request from the corresponding author.

The Environment and Climate Change Canada (ECCC) dataset used as the reference cloud scenes is available upon request from the corresponding author.

YL developed the retrieval algorithms, set up the retrieval experiments, analyzed the retrieval results, and wrote the article. GGM acted as project leader, provided guidance and insight regarding experiment configurations and results analysis.

The contact author has declared that neither they nor their co-authors have any competing interests.

Publisher’s note: Copernicus Publications remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

We thank for Pavlos Kollias at Stony Brook University for providing the atmospheric scenes from the
ECCC model that we used for evaluating the synergistic retrievals. The TC4 in situ data for this study were
collected by the members of the Stratton Park Engineering Corporation led by Paul Lawson, as well as
Andrew Heymsfield at the National Center for Atmospheric Research. The CloudSat data were obtained from
the CloudSat Data Processing Center at the Colorado State University’s Cooperative Institute for Research in
the Atmosphere (CIRA). All TC4 data are publicly and freely available in the NASA data archive at

This research has been supported by the Goddard Space Flight Center (NASA ACE and ACCP programs (grant nos. NNX15AK17G and 80NSSC19K1087)).

This paper was edited by Patrick Eriksson and reviewed by two anonymous referees.