The micropulse differential absorption lidar (MPD) was developed at Montana State University (MSU) and the National Center for Atmospheric Research (NCAR) to perform range-resolved water vapor (WV) measurements using low-power lasers and photon-counting detectors. The MPD has proven to produce accurate WV measurements up to 6 km altitude. However, the MPD's ability to produce accurate higher-altitude WV measurements is impeded by the current standard differential absorption lidar (DIAL) retrieval methods. These methods are built upon a fundamental methodology that algebraically solves for the WV using the MPD forward models and noisy observations, which exacerbates any random noise in the lidar observations.

The work in this paper introduces the adapted Poisson total variation (PTV) specifically for the MPD instrument. PTV was originally developed for a ground-based high spectral resolution lidar, and this paper reports on the adaptations that were required in order to apply PTV on MPD WV observations. The adapted PTV method, coined PTV-MPD, extends the maximum altitude of the MPD from 6 to 8 km and substantially increases the accuracy of the WV retrievals starting above 2 km. PTV-MPD achieves the improvement by simultaneously denoising the MPD noisy observations and inferring the WV by separating the random noise from the non-random WV.

An analysis with 130 radiosonde (RS) comparisons shows that the relative root-mean-square difference (RRMSE) of WV measurements between RS and PTV-MPD exceeds 100 % between 6 and 8 km, whereas the RRMSE between RS and the standard method exceeds 100 % near 3 km. In addition, we show that by employing PTV-MPD, the MPD is able to extend its useful range of WV estimates beyond that of the ARM Southern Great Plains Raman lidar (RRMSE exceeding 100 % between 3 and 4 km); the Raman lidar has a power-aperture product 500 times greater than that of the MPD.

Water vapor (WV) is one of the fundamental thermodynamic variables that defines the state of the atmosphere and influences many important processes related to weather and climate. The importance of continuously monitoring lower tropospheric WV is underscored in the National Aeronautics and Space Administration (NASA) decadal survey

To fulfill this observational need, Montana State University (MSU) and the National Center for Atmospheric Research (NCAR) have developed a micropulse differential absorption lidar (MPD) that continuously measures range-resolved WV in the lower (150 m to 6 km) atmosphere

The MPD has proven to produce accurate WV measurements up to 6 km altitude, depending on aerosol loading, clouds, and solar background. However, in high solar background conditions, MPD water vapor retrievals can be noisy as low as 2 km. The MPD's ability to produce precise measurements above 2 km and accurate higher-altitude WV measurements is impeded by the current standard WV DIAL retrieval methods. These methods are built upon a fundamental methodology that algebraically solves the WV variable through operations that exacerbate any random noise in the lidar observations

Figure

The

Advances have been made in denoising photon-counting medical images where the photon detection methodologies and forward modeling are similar to that in atmospheric lidar

Inspired by the medical image denoising methods and the Poisson total variation (PTV) method for inferring extinction from HSRL observations

The PTV-MPD method we present in this paper differs from PTV by three adaptations that are necessary for accurate WV measurements

The immediate contributions of this paper are the

adaptation of the PTV method (i.e., PTV-MPD) for the DIAL technique and

the first rigorous validation of the PTV method using in situ measurements.

This work also serves as an example of how development of advanced signal processing can provide insights into improving hardware design and trades. By leveraging advances in signal processing, lidar hardware costs might be reduced whilst maintaining or improving the retrieval precision.

We introduce the MPD forward models in Sect.

Commonly used symbols.

Geophysical variables and the lidar forward models are written as matrices. When we introduce a geophysics-related variable we immediately indicate the units of the variable using the notation [

Table

Commonly used abbreviations.

To date NCAR has five experimental MPD instruments where each instrument transmits laser pulses at a rate of 7 to 8 kHz at two wavelengths sequentially. The first wavelength is tuned on a WV absorption line near 828.2 nm, whereas the second wavelength is off an WV absorption line near 828.3 nm

The MPD instrument photon detector observes the weighted sum of the lidar equation and the laser pulse, since the laser pulse spans multiple range sampling intervals. We define the SSLE on the measurement range axis, which are denoted by

The SSLE is a function of the unknown WV

The weighted sum of the SSLE with the laser pulse is modeled by

The MPD instrument photon detectors are saturated over several range bins after each laser shot due to internal scattering in the co-axial optical configuration. Consequently, the first WV estimate starts at 125 m or 500 m depending on the hardware configuration. To model the unobserved WV from range 0 up to

The photon-counting observations at range

The instantaneous backscattered photon rates, corresponding to each laser shot, of clouds and precipitation can exceed the MPD photon detector saturation limit. Moreover, the accumulated photon counts

The saturated photon count mask

We used a sliding window standard deviation filter on the photon counts

The Poisson negative log likelihood (P-NLL) is used by the PTV-MPD method to quantify the fitting of the WV and attenuated backscatter onto the noisy observations via the MPD forward model Eq. (

A detailed discussion of the standard processing technique employed for MPD is provided in

The MPD standard method.

The noisy observations

{Low pass filter background subtracted photon counts}

{Compute differential optical depth}

{Compute the absolute water vapor}

{Low pass filter the computed water vapor}

The photon count profiles first undergo low-pass filtering to suppress the random noise

Finally, the lengths of the laser pulses are not accounted for in the standard method. Specifically, inclusion of the laser pulse in the forward model prevents the attenuated-backscatter term from directly canceling out when dividing the offline with the online channel, and a direct inversion becomes no longer possible. However, failing to account for the laser pulse length can create biases in some cases, for example at WV dry regions.

The PTV method, originally developed for photon-counting HSRL

With PTV the assumptions are that

the photon-counting noise can be accurately modeled by the Poisson PMF in Eq. (

the expected value of the photon counts can be accurately modeled with a forward model (i.e., Eq.

the geophysical variable (i.e., WV) image that we want to estimate can be accurately approximated with a two-dimensional (2D) piecewise constant (PC) function.

By making a distinction between the random noise and the geophysical variable having spatial correlation, PTV is capable of separating the noise from the geophysical variable that is being estimated. PTV achieves this separation by (1) using the P-NLL from Eq. (

The approach used by PTV is formulated as a mathematical optimization framework where we search over all candidate geophysical variables and choose the geophysical variable that minimizes the sum of the (1) P-NLL composited with the forward model and (2) the TV penalty functions. The P-NLL composite is called the “loss function” and the loss summed with the penalty functions is called the “objective function”. The optimization framework is conceptually illustrated by the equation

The (anisotropic) TV penalty function is defined as

We adopt the PTV method to infer the WV from the MPD observations, where we approximate both the WV and attenuated backscatter as 2D PC functions. In contrast, the standard method of Alg.

In order to apply PTV on MPD observations the following adaptations are required:

use of the MPD forward models of Eq. (

allowing for the simultaneous inference of the WV and attenuated backscatter.

prevention of the degradation of the WV measurement due to inaccuracies in the initial attenuated-backscatter value.

Regarding adaptation

Using the conceptual optimization framework in Eq. (

The formulation of the PTV-MPD includes tuning parameters that are computed using a cross-validation (CV) methodology; the CV involves calculating a validation error that indicates the optimality of the tuning parameters and is discussed in Sect.

Pictorial overview of Alg.

Figure

performs the necessary preparations to infer the WV

employs a CV methodology to choose the optimum tuning parameters for inferring the WV

infers the WV

The PTV-MPD loss function differs from the PTV loss function in two respects; cf. Eq. (23) in

The objective function that is minimized by PTV-MPD is defined as

In Appendix

It will be advantageous for the MPD to make WV measurements that are statistically independent of RS for the purpose of providing additional statistically independent information for NWP data assimilation. Thus, in this paper we set the initial WV to zero [g m

From the lapse rate, the WV absorption cross section is computed; see Sect.

The offline channel is less sensitive to WV absorption compared to the online channel. Hence, the initial value of the attenuated backscatter, denoted by

With the coarse-to-fine image resolution inference framework, we start with estimating a coarse-resolution WV image and use the coarse-resolution estimate as an initial WV value at a finer image resolution; this technique has proven useful to more accurately denoising and inverting images for low-SNR photon-limited application

Following the approaches delineated in

A series of values

The downsampling operator is denoted by

Let

In practice, the minimization of the objective function (

The validation and test errors are computed by basically comparing the WV

The training photon counts

The validation error of the estimates per tuning parameter is computed with the P-NLL (

The test error for the estimates

Algorithm

The alternating minimization of the objective function (

The PTV-MPD method.

(1) The noisy observations

If WV

{Split photon counts in training, validation and test counts}

{Compute the atten. backs. initial value}

{Infer the WV and atten. backs.}

{Compute the atten. backs.}

Compute test error

Cross-validation methodology in choosing optimum tuning parameters when inferring WV

(1) WV

{Choose tuning parameters via cross-validation}

{Infer WV and atten. backs. using redefined objective function}

Algorithm 4

Compute the validation error

{Choose tuning parameter that has smallest validation error}

{With chosen tuning parameters select estimates}

Coarse-to-fine image resolution framework for inferring WV

(1) WV

{Set initial values for current image resolution; downsample the initial WV estimate to the image resolution at which inference is done}

{Find minimizer of objective function (

{Solve for WV via adapted SPIRAL by minimizing Eq. (

{Solve for atten. backs. via adapted SPIRAL by minimizing Eq. (

{Set initial values for finer image resolution}

{Set final estimates of WV and atten. backs.}

In this section we quantify the accuracy of the PTV-MPD WV measurements juxtaposed with the standard method; we use the mnemonics

Information about the project is available at

The first column indicates the lidar instrument and where the instrument was located. SGP refers to the Southern Great Plains in Oklahoma, and Marshall refers to Marshall field in Boulder, Colorado. The second and third column show the corresponding laser power and telescope diameter.

Although the MPD instrument was in close proximity of the ARM site during the SGP deployment, the MPD, Raman lidar, and RS instruments all observe different atmospheric volumes at different altitudes, especially in highly convective conditions. Furthermore, the Raman lidar uses some RS observations for calibration, and therefore the instrument is not entirely independent of the RS

For the experiments we created datasets that span over specific months given (1) the high variability of WV across different months and (2) that the MPD WV measurement capability is dependent on the low-altitude WV two-way transmittance. For all the datasets presented here, the observed photon counts are binned at range and time bins of 37.5 m and 5 min, respectively. The analyzed data from SGP are not comprehensive, and we instead selected a variety of challenging cases in an effort to identify potential issues in the PTV-MPD algorithm. Specifically, much of the data targeted instances where clouds and precipitation created challenging scenes to processes.

Specific to the MPD at SGP, the WV measurements start at 500 m range due to the poor geometric overlap below this range. The next-generation MPD deployed at the Marshall field site employed optically combined telescopes with narrow and wide fields of view to improve low-altitude geometric overlap, and as a result the WV measurements start at 150 m range

The first column shows the names of the datasets used in the experiments. The second column lists the specific dates that are included in each dataset. The third and fourth columns list the number of days in each dataset and the total number of coincident RS profiles.

The first column of Table

The PTV-MPD method has the following input parameters, as indicated in Alg.

the initial WV value

the WV

the coarsest-resolution downsampling factor

In regards to the resolution downsampling factor

The mnemonics used to indicate the algorithm that was used to produce a water vapor estimate. The corresponding test errors are also shown.

In the following subsection we show individual PTV-MPD WV measurement results, and thereafter we present the WV measurement error statistics.

Figures

The

The

The

When comparing the

the

In some cases, residual noise in the

Figure

Comparing Fig.

the assumed WV value when computing the initial value of the attenuated backscatter (see Sect.

long laser pulse, denoted by operator

The higher accuracy of the

implicitly deconvolves the long laser pulse from the initial attenuated backscatter

and constrains the inference of the WV to be at a coarse image resolution, which implicitly increases the SNR of the observations.

The RS WV measurements are used as a reference to produce WV measurement error statistics of the different methods discussed in the following subsection. Next, we discuss the test errors between the

For each dataset we compute the root-mean-squared “error” (RMSE) per range and the relative RMSE per range; all the methods use the same mask to exclude the same WV pixels when computing the RMSE. The RMSE and relative RMSE were computed as follows. Let

For the all the SGP datasets,

For the all the Marshall datasets,

Figures

From Figs.

The Raman lidar outperforms the

From Figs.

Table

This table shows for which datasets the test error of the PTV-MPD method with the coarse-to-fine configuration

PTV-MPD employs the photon-counting noise model to quantitatively fit the MPD forward models on the noisy observations, while encouraging accurate spatial and temporal correlations across all WV estimate pixels via the total variation regularizer function. This holistic approach allows for accurately measuring highly structured and varying WV fields at different altitudes and varying SNRs. In comparison, the standard method employs low-pass filters to reduce the residual noise in the WV estimate, and the low-pass filter bandwidth is optimized for lower-altitude (

We also demonstrated that without careful consideration of how PTV is adapted for the MPD instrument, low-altitude biases can be introduced in the PTV-MPD WV measurements. PTV-MPD requires an initial value of the attenuated backscatter, and any inaccuracies can induce biases in the PTV-MPD WV estimates. PTV-MPD can be made more robust against such biases by inferring the WV via a coarse-to-fine image resolution framework. By inferring the attenuated backscatter and WV at a coarse image resolution, inaccuracies in the initial attenuated backscatter are reduced, and subsequent finer image resolution attenuated backscatter estimates are more accurate, which allows for more accurate WV estimates.

As of now, PTV-MPD is computationally expensive since inferring the WV requires estimating the WV with several tuning parameters, and the optimal tuning parameter is selected through a cross-validation methodology. For example, with 144 CPU cores it takes 1 to 2 h to infer the WV using the PTV-MPD coarse-to-fine image resolution framework for 24 h of data; each CPU core estimates the WV for a specific tuning parameter. We are working towards developing a methodology to infer the optimal tuning parameter from the photon-counting observations, which will reduce the number of required CPU cores to 12 or less. In addition, adapting the PTV-MPD code to use a GPU instead of a CPU might reduce the computational time by at least 10-fold

An additional benefit of decreasing the computational demands of PTV-MPD is that it would be able to process multiple days of consecutive observations and not just 24 h scenes as demonstrated in the results. A current workaround to process multiple-day scenes is to use a horizontal sliding window when inferring the WV. Specifically, the WV is inferred for consecutive overlapping 24 h periods, and the final multiple day WV image is obtained by averaging together the overlapping 24 h period WV estimates.

Future work includes the following goals.

The first is working towards quantifying the uncertainties of the PTV-MPD WV measurements using a bootstrapping methodology

The second is investigating how PTV-MPD can be made more robust against saturated photon counts.

Define the indicator function as

Here we show that for high solar background radiation (i.e., lower SNR) of the photon-counting images there can be multiple estimates of the WV

the laser pulse duration is equal to the sampling intervals, meaning that

none of the photon counts are masked out by matrix

and

In the cases where the backscattered photons are comparable to the observed background counts according to Eq. (

The PTV-MPD code will be made available through a Git repository

All of the MPD data products used in this work were produced by NCAR and are available upon request; the raw MPD data are available at

This work was a collaboration between the authors. WJM is the original developer of PTV and provided expertise in advanced signal processing. MH provided expertise in optical system modeling and DIAL observation. The PTV-MPD development was the result of a collaborative effort to mutually leverage the two disciplines.

The contact author has declared that none of the authors has any competing interests.

Publisher's note: Copernicus Publications remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

The authors would like to thank Robert A. Stillwell and Scott M. Spuler for their feedback on this paper. This material is based upon work supported by NSF AGS-1930907 and the National Center for Atmospheric Research, which is a major facility sponsored by the National Science Foundation under cooperative agreement no. 1852977. We thank Dale Hurst, Patrick Cullis, Emrys Hall, and Allen Jordan of the NOAA Global Monitoring Laboratory for providing the sounding data from the Marshall field site.

This research has been supported by the Division of Atmospheric and Geospace Sciences (grant no. NSF AGS-1930907) and the National Science Foundation (cooperative agreement no. 1852977). The MPD Net demo field campaign was supported by the Office of Biological and Environmental Research of the U.S. Department of Energy (under grant no. DOE/SC-ARM-20-002) as part of the Atmospheric Radiation Measurement (ARM) user facility, an Office of Science user facility.

This paper was edited by Hartwig Harder and reviewed by three anonymous referees.