Accurate remote sensing of atmospheric CO2 from Earth-orbiting satellites is a key component in a long-term carbon–climate observing system (Sellers et al., 2018). Airborne and spaceborne lidar can be used to remotely monitor the global CO2 and other trace-gas concentrations under conditions that are inaccessible to passive spaceborne CO2 measurement missions, such as GOSAT (Kuze et al., 2016), OCO-2 (Crisp et al., 2017), and OCO-3 (Eldering et al., 2017, 2019). Studies have shown (Kawa et al., 2018) that a polar-orbiting integrated path differential absorption (IPDA) lidar can measure XCO2 with low bias and high precision at all sun angles, seasons, and latitudes using a constant nadir-zenith illumination and observation geometry. A pulsed IPDA lidar also provides the range-resolved atmospheric backscatter profiles, so that return signals from the surface, clouds, and aerosols can be uniquely separated (Allan et al., 2019). This allows pulsed IPDA lidar to measure XCO2 to the surface, cloud tops, or both (Ramanathan et al., 2015). Because the laser pulses reflected from clouds, aerosols, and surface are separated in time, the XCO2 retrievals to the ground surface are not biased by scattering from clouds and aerosols (Mao et al., 2018).

Several types of dual-wavelength (online and offline) IPDA lidar have been demonstrated previously for measuring XCO2 from aircraft (Spiers et al., 2011; Menzies et al., 2014; Jacob et al., 2019; Dobler et al., 2013; Campbell et al., 2020; Refaat et al., 2016, 2020, 2021; Amediek et al., 2017; Zhu et al., 2019, 2020). A multi-wavelength IPDA lidar has also been reported. The retrieval algorithms used in these IPDA lidars calculate the ratios of online to offline atmosphere transmission, convert them to differential absorption optical depths (DAODs), and then solve for XCO2 from the DAOD based on atmospheric transmission models. For the multi-wavelength lidar proposed by Han et al. (2020), a series of DAODs are calculated, and a least-squares fit is used to solve for XCO2 (Han et al., 2020). The XCO2 can be solved directly from the DAOD; however, these algorithms rely on the accurate knowledge of the line shape of the CO2 absorption. They are sensitive to measurement biases due to uncertainties in spectroscopy and meteorological conditions that affect the line shape. They also require precise knowledge of the laser wavelengths, the lidar receiver optical transmission versus wavelength, and the Doppler shift of the received laser signals.

NASA Goddard Space Flight Center (GSFC) has developed an airborne multi-wavelength CO2 sounder lidar and demonstrated XCO2 measurements through a series of airborne campaigns (Abshire et al., 2010, 2013, 2014, 2018; Ramanathan et al., 2013, 2015, 2018; Mao et al., 2018). Its retrieval compares the lidar-sampled line shape with one computed from an atmosphere model to retrieve XCO2. Several other parameters, such as the Doppler shift, surface reflectance, and lidar receiver spectral response are solved simultaneously via a least-squares fit. The retrieval algorithm is similar to those used for passive trace gas measurements with modifications specifically for the lidar measurement. Although this multi-wavelength approach requires more laser power to achieve a given XCO2 measurement precision, it provides more tolerance to the uncertainties in the CO2 absorption line shape, lidar receiver response, and Doppler shift, so that the retrieved XCO2 is more robust against bias errors.

This paper describes the retrieval algorithm for the multi-wavelength CO2 Sounder lidar and provides a framework for similar IPDA lidar for other atmospheric gas measurements. Parts of the algorithm have been reported earlier (Ramanathan et al., 2013, 2015, 2018). This paper gives a complete description of the algorithm, the mathematical derivations, signal processing techniques, estimation error, and averaging kernel. An example of using the algorithm to analyze measurements from the airborne CO2 Sounder lidar is also presented.

The measurement geometry for the CO2 Sounder lidar is shown in Fig. 1. The lidar transmits laser pulses toward nadir, and its receiver telescope collects the optical signal backscattered from the atmosphere and the surface. Figure 2 shows a block diagram of the airborne CO2 Sounder lidar, which was developed as an airborne demonstrator for NASA's planned Active Sensing of CO2 Emissions over Nights, Days, & Seasons (ASCENDS) mission (Kawa et al., 2018). The laser consists of a tunable seed laser, a pulsed modulator, and a power amplifier. The seed laser module consists of two single-frequency continuous-wave (CW) diode lasers. One is the reference laser whose wavelength is locked to the center of the CO2 absorption line in a gas cell. The other diode laser (slave) is tunable and its wavelength is locked to that of the master, plus a programmable offset frequency. The offset frequency is step-tuned across the CO2 absorption line. The number of laser wavelengths in the scan and the exact wavelength of each laser pulse are digitally pre-programmed and can be adjusted via software commands (Numata et al., 2012). An electro-optical modulator is used to gate the output of the slave laser into 1 µs wide pulses. The laser pulses are then amplified by a multi-stage commercial fiber laser amplifier. The airborne lidar's laser pulse rate is 10 kHz and there are 30 wavelengths per scan, which gives a wavelength scan rate of about 300 Hz. The transmitted laser pulse energy at each wavelength is also sampled, and the results are used to normalize the received signal to correct for fluctuations in the transmitted laser energy with wavelength.

The lidar receiver detects and records the received laser pulse waveform over the entire atmosphere column traveled by the laser pulses. In the airborne lidar all signals are digitized and recorded and the lidar analysis is performed on ground after the flight. The received signals from the scattering surface are used to retrieve XCO2. The signals before the ground returns are used to obtain the atmosphere backscatter profiles as an ancillary data set. The signals recorded after the ground returns are used to estimate the solar background, the detector dark noise, and the baseline voltage offset in the detector output. The baseline offset is subtracted from the signal before calculating the ground return pulse energies. The times of flight of the laser pulse to the targeted scattering surface are used to determine the atmosphere column height over which the CO2 is measured.

An overview of the retrieval algorithm for lidar data is shown in Fig. 3. The initial processing consists of (a) processing the stored lidar data to estimate ranges to the reflecting surfaces and form a series of atmosphere transmission measurements across the CO2 absorption line; (b) generating a CO2 absorption line shape from the radiative transfer model and meteorological data at the time and location of lidar measurements; and (c) performing a least-squares fit of the modeled line shape function to the measurements to solve for XCO2 and other parameters.

The column XCO2 and several other variables are solved simultaneously from a least-squares fit of the modeled line shape to the lidar measurements. One variable is the Doppler shift in the wavelengths of the received signal, which occurs when measuring at non-nadir angles from a moving platform. Another parameter being solved for is the product of the surface reflectance and the two-way offline atmospheric transmission. For the CO2 line at 1572.33 nm and under high humidity, there is a weak isotopic water vapor absorption feature on the left wing of the CO2 absorption line. The retrieval algorithm can resolve this absorption feature to avoid causing biases in the retrieved XCO2. For our airborne lidar, there is also a small linear trend (slope) in the received laser pulse energy as a function of the wavelength. The primary cause of this trend is the residual error from modeling the uneven spectral response of the receiver optics, especially the optical bandpass filter. Since the bandpass spectral shape can change slightly with temperature and time, the retrieval also solves for this residual slope.

The least-squares fit may be formulated by expressing the lidar measurement data in matrix form, Y, a single column matrix with elements yi given by Eq. (2). The parameter to be solved for, S=sk, is expressed as a N3×1 matrix. In our case N3=5, where each element is defined as s1=rsTo2 is the product of the surface reflectance and the two-way atmosphere transmission at offline wavelength, s2=αCO2 is the scale factor for the XCO2 line shape function, s3=αwater is the scale factor for the water vapor line shape function, s4 is the linear slope of the receiver spectral response, and s5 is the Doppler shift of the received signal wavelengths.

The modeled atmospheric transmission given in Eq. (9) can be expressed as a single column matrix, FS, called a forward model, with each element equal to 10fiS=rsTA2λi,S≈s1s2TCO22λi+s5s3Twater2λi+s5×η0λi+s5,s4, where TA2λ,S is the atmosphere transmission defined in Eq. (1) but expressed as a function of both the laser wavelength and the parameters to be solved, and η0λ,s4 is the normalized receiver optical transmission as a function of the wavelength and the slope of the linear trend of the receiver spectral response. Here we also included the term for the water vapor.

A scalar-valued loss function can be defined as the sum of squared differences between the lidar measurement data and the model, as 11JY,S=Y-F(S)TWY-F(S)=∑i=1N1wi,iyλi-fiS2, where Y-F(S) is an N1×1 matrix and W is a N1×N1 diagonal matrix for weighting factors used in the fit. The weighting factors are chosen to balance the contributions from the measurements at different laser wavelengths that have different SNRs. The least-squares fit finds the parameter set that minimizes the loss function.

For small changes in XCO2 and for high SNR lidar measurements, Eq. (11) can be linearized by the first two terms of its power series expansion about initial estimates of the parameter values, S0. The function FS, also known as the forward model, can then be approximated by 12FS≈F0+∂FS∂S|S=S0S-S0, where F0=FS0=f0λ1⋮f0λN1 with fλi equal to Eq. (10) evaluated at the initial value of the parameter set.

Substituting Eq. (12) into Eq. (11) and defining ΔY=Y-F0 and ΔS=S-S0, the loss function can now be approximated as 13JY,S≈ΔY-∂FS∂S|S=S0ΔST×WΔY-∂FS∂S|S=S0ΔS. For mathematical convenience, we normalize the lidar measurements with respect to their initial estimate and define a new variable: 14Δy1λi1=Δyλif0λi. A diagonal matrix IO can be defined with each element equal to 1/f0λi. The loss function can be rewritten using the identity matrix I≡IOIO-1≡IO-1IO, as 15JY,S≈ΔY-∂FS∂S|S=S0ΔSTIOIO-1×WIO-1IOΔY-∂FS∂S|S=S0ΔS=ΔY1-∂F1S∂SΔST×W1ΔY1-∂F1S∂SΔS, where ΔY1=IOΔY, F1S=IOFS, and W1=IO-1WIO.

The use of the above normalization greatly simplifies the mathematical derivation as well as the data processing since it cancels out the exponential terms in the derivatives of FS. However, this technique can only be used when the values of the forward model fλi are not approaching zero at all sampling wavelengths.

The loss function given in Eq. (15) is of the same form as that of a linear least-squares fit with measurement data ΔY1 and weighting factor W1. The derivative of the function F1S, which is often referred to as the Jacobian, is given by 16K=∂F1S∂S|S=S0. For the CO2 Sounder lidar, each term of the Jacobian can be derived as ki,1=∂F(S)s11f0λi=1〈rsTo2〉, same for all i=1,2,…,N1; ki,2=-2ODCO2aλi, one for each laser wavelength, i=1,2,…,N1; ki,3, same as above but for water vapor; ki,4=λi-λc, with λc the center wavelength of the CO2 line shape function; ki,5≈TCO22λi+Δλ-TCO22λiΔλ⋅1〈TCO22(λi)〉, with Δλ=1 pm (or the expected average Doppler shift) and TCO22λi given by Eq. (4).

For measurement noise that is zero mean and follows a Gaussian distribution, the optimal weighting factors are given by the reciprocal of the variance of the measurement data (Bevington, 1969). In our case, the optimal weighting factors can be approximated as 17w1i,i=1varΔy1λi=f0λi2varyλi≈〈yλi〉2varyλi=SNRλi2, where 〈yλi〉 is the average value of the lidar measurement which is assumed to be close to the initial estimate f0λi. Therefore, for each wavelength the weighting factors can be approximated by the SNR of the lidar measurement at that wavelength. As mentioned earlier, the SNRs are calculated based on signal energy and background noise estimated from received pulse waveforms.

The XCO2 and other parameters can now be solved using a standard linear least-squares fitting method with the loss function Eq. (15), Jacobian Eq. (16), and weighting factors Eq. (17). The solutions can be obtained numerically using the pseudo inverse function, as 18ΔS^=GΔY1 with G a N3×N1 matrix, which is often called the gain matrix and can be computed from the pseudo inverse function pinv⋅ (Peters and Wilkinson, 1970), as 19G=pinvKTpinvW1KKTpinvW1. The pseudo inverse matrix function can be found in the MATLAB software package and in other software tools.

The covariance of the parameters can be obtained from Eq. (18), as 20covΔS^=GvarΔY1GT, with varΔY1 a diagonal matrix with each element the reciprocal of the corresponding element in Eq. (17). The covariance matrix cov(ΔS^) is in general not a diagonal matrix even though varΔY1 is a diagonal matrix.

The variances of the estimated parameters are given by the diagonal elements of cov(ΔS^). However, variance is only one of the criteria of the XCO2 retrieval. There can still be a bias in the estimated parameters if there is a mismatch between the measurements and the modeled line shape.

The total column averaging kernel can be calculated as (Borsdorff et al., 2014) 21A=GαKx, where Gα is the row of the G matrix for calculating the XCO2 scale factor and Kx is the Jacobian of the measurement with respect to the layered CO2 mixing ratios, which is an N1×N2 matrix given by 22Kx=∂F(XCO2)∂XCO2. Each term of Kx can be written according to Eqs. (2)–(4), (7), and (10), as Kxi,j=ρairHjσCO2Hj,λiΔHj. The linear least-squares fit can also be iterated by correcting for the Doppler shift of the received laser wavelengths of the modeled line shape based on the solution from the previous iteration. The Jacobian terms are recalculated about the updated linearization point in each iteration to improve the results.

The algorithm described here was used to retrieve XCO2 from measurements of our 2017 airborne lidar campaign (Mao et al., 2019). The lidar and the airborne measurements have been described in detail in Abshire et al. (2018). Table 1 lists the instrument parameters relevant to the XCO2 retrieval. Here we show a few examples of using the retrieval algorithm on a data set collected during one of the 2017 flights. We also show the retrieved XCO2 at different altitudes in comparison to XCO2 calculated from the in situ measurements made during two spiral-down maneuvers.

Figure 4 shows an example of a Level-1 data set from our 2017 airborne campaign. It shows 30 transmitted pulse waveforms and the corresponding received pulse waveforms averaged over 32 laser wavelength scans. The decrease (tilt) of laser pulse amplitudes over the pulse width interval is caused by the depletion of energy stored in the laser gain media, which does not affect the IPDA lidar measurements. The energies of the transmitted laser pulses at different wavelengths fluctuate by a few percent, which is monitored and corrected for in the signal processing. The tails in the transmitted pulse waveforms shown in Fig. 4b are caused by an artifact of the laser monitor detector, which is different from the one used in the receiver. The amplitudes and energies of the received laser pulse waveform plotted in Fig. 4c clearly show the CO2 absorption near the center of the wavelength scan. The XCO2 retrieval is carried out at 1 Hz, during which the host aircraft typically travels about 200 m.

For the least-squares fit the weighting factor for each wavelength is the square of the SNR of the lidar-detected signals at that wavelength. The SNRs are estimated from the received lidar signal as (Gagliardi and Karp, 1995)

23SNRλi=ηd〈nsλi〉Fdηd〈nsλi〉+1+τsτb〈nb〉+1+τsτb〈nd〉+1+τsτb〈na〉2〈Gd〉2. Here 〈nsλi〉 is the average number of received signal photons per pulse at wavelength λi, 〈Gd〉 is the average gain of the avalanche photodiode (APD) detector, ηd is the APD quantum efficiency, Fd is the APD gain excess noise factor, τs is the integration time for the signal pulse, τb is the integration time for the background and dark noise, 〈nb〉 and 〈nd〉 are the average number of background photons and detector dark counts integrated over the pulse interval, and 〈na〉 is the standard deviation of the preamplifier noise in terms of equivalent number of photoelectrons. The signal here refers to the number of detected signal photons, which is equal to the number of the detected photons minus the number of detected background photons.

The laser speckle noise term (Goodman, 1965, 1975) is not included in Eq. (23) since it is not a major noise source for our airborne lidar measurements at nominal flight altitude. This is because of the large number of speckle cells in the laser footprint and the numerical averaging of the 30 received laser pulses for each XCO2 retrieval. Laser speckle noise is also not expected to be a major noise source for the space version of the CO2 Sounder lidar being developed at NASA GSFC (see chap. 5 of Kawa et al., 2018) since the effects of spatial and numerical averaging are similar. The effects of errors in the meteorological data used to construct the line shape model of CO2 are also not considered in Eq. (23). We are currently conducting computer simulations to quantify the effect of meteorological data errors, and the results will be reported in a separate publication.

For the XCO2 retrieval, the average number of received signal photons is estimated from the received pulse waveform. This is obtained by first integrating the received pulse waveform from the detector in volts, dividing the result by the detector responsivity in volts per watt, and the photon energy in joules. The average number of background noise photons is estimated from the average surface reflectance, offline atmosphere transmission, nominal sunlight irradiance on the surface, and receiver optics model. All other parameter values in Eq. (23) are instrument related and can be found in Abshire et al. (2018).

Figure 5 shows the CO2 absorption line shape sampled by the lidar along with that from the forward model which assumes a constant XCO2 vertical profile of 400 parts per million (ppm). It also shows the placement of laser wavelengths across the CO2 absorption line. One laser wavelength (the second from the left) was placed at a secondary absorption feature due to deuterated water vapor (HDO). Three wavelengths were placed on the wings of the CO2 absorption line. The rest were roughly equally spaced in OD along the absorption line. The residual differences between the measurements and the model after the least-squares fit are also plotted in Fig. 5. The averaging kernel is calculated based on Eq. (21) for each fit of 1 s lidar measurement data. Figure 6 shows the normalized averaging kernel with respect to its average value over the atmosphere column height and a fourth-order polynomial fit for the data shown above.

Figure 7 shows the results of the retrieval using the algorithm described above from the airborne CO2 Sounder lidar measurements made on 21 July 2017 starting at 00:30:00 UTC for 820 s. The data consist of about a 500 s segment measured at a nearly constant aircraft altitude followed by about 300 s of measurements in a spiral descent. The last part of the flight was near Edwards Air Force Base, CA, and the surface elevation was nearly constant for the last 500 s. The ground surface over this stretch of the flight was dry desert, and the sky was visually clear at the time. The retrieved XCO2 over this period was steady with a slow downward trend. The root-mean-squared (rms) variation in the retrieved XCO2 from 2100 to 2300 s was 0.67 ppm, which includes both the fitting error and the actual XCO2 variation along the flight path. By comparison, the estimated standard deviation from the retrieval covariance matrix was about 0.35 ppm, as shown in Fig. 7e.

Figure 8 shows the retrieved XCO2 compared to that calculated from in situ measurements as the airplane flew in a spiral-down path from 12 to 4 km over Edwards Air Force Base for the flight on 21 July 2017 and for that on 8 August 2017. The in situ profiles were measured by an updated version of the AVOCET gas analyzer on board the airplane (Vay et al., 2011). The in situ XCO2 is calculated from the airplane altitude to the ground and is obtained by integrating the CO2 profile from the in situ measurements weighted by the lidar's averaging kernel. The XCO2 retrieved from the lidar measurements agrees with that calculated from the in situ measurements at all airplane altitudes above 4 km. Below 4 km, the laser beam no longer completely overlaps the field of view of the receiver, and the total CO2 absorption (line depth) becomes small. The lidar measurements are not calibrated at such a low altitude.

An algorithm to retrieve XCO2 has been developed for measurements from a pulsed multi-wavelength IPDA lidar. The retrieval algorithm uses a least-squares fit of the line shape function derived from a multi-layer atmosphere radiative transfer model based on meteorological data to the line shape sampled by the lidar measurements. In addition to XCO2, the algorithm simultaneously solves for the product of the surface reflectance and the offline atmosphere transmission, Doppler shift of the received laser signals, a secondary water vapor mixing ratio (if present), and a linear trend of the lidar receiver non-uniformity in its spectral response. Since it can accurately retrieve XCO2 as these conditions vary, this approach provides a more robust measurement of XCO2 compared to IPDA lidar that uses only online and offline wavelengths. The retrieval algorithm has been used successfully in the data processing of the NASA GSFC multi-wavelength pulsed IPDA lidar from its 2016 and 2017 airborne campaigns. The algorithm may also be used for retrievals for multi-wavelength lidars that target other atmospheric gases, such as CH4.