We review the singular value decomposition (SVD)
framework and use it for quantifying and discerning vertical information in
greenhouse gas retrievals from column integrated absorption measurements.
While the commonly used traditional Bayesian optimal estimation (OE) assumes
a prior distribution in order to regularize the inversion problem, the SVD
approach identifies principal components that can be retrieved from the
measurement without explicitly specifying a prior mean and prior covariance
matrix. We review the SVD method, explicitly recognize the use of an
uninformative prior and show it to incur no bias from the choice of the
prior. We also make the connection between the SVD method and the
pseudo-inverse, which makes it more intuitive and easy to understand. We
illustrate the use of the SVD method on an integrated path differential
absorption

In the past few decades, anthropogenic climate change has brought a renewed
interest in carbon cycle science and thus in accurate sensing of greenhouse
gases (GHGs). GHG column remote-sensing measurements are made using
satellite-based optical spectrometers such as those aboard the Greenhouse gas
Observing Satellite (GOSAT,

Although in principle, the traditional OE

Comparison of retrieval algorithms used for GHG remote
sensing based on regularization method and source of vertical information.
The approximate spectral resolution (instrument linewidth; see Sect.

The theoretical basis of the SVD method has been previously laid out in the
context of the general underdetermined inversion problem

In this work, we choose a specific greenhouse gas measurement system and study the principal components and illustrate how they provide useful, quantifiable information about the vertical distribution of the gas. In choosing to evaluate the retrieval method via the principal components, the implicit prior used is strictly uninformative and does not cause any bias in the retrieved principal components, which we explicitly show. In addition, we explore the instrument spectral resolution necessary to obtain vertical information. Finally, we illustrate the theory using numerical simulations.

This paper also attempts to make the theoretical framework of the SVD method
more accessible to readers who may not be as familiar with the matrix algebra
conventions used in books like

The SVD method works similarly to least-squares line-fitting retrieval
approaches but offers a more formal framework

The paper is organized as follows. In Sect.

A retrieval seeks to extract certain information from a measurement. Even when the number of measurement samples far exceed the number of retrieved parameters (as with column GHG absorption measurement spectra), retrieval problems may or may not be fully determined depending on the information content of the samples with respect to the retrieved parameters. In situations where the retrieval problem is fully determined, one can obtain a unique solution of the parameters of interest. When the problem is overdetermined, one can perform a least-squares fit to solve for the parameters of interest. However, for column GHG absorption measurement spectra obtained from remote sensing, the retrieval is generally underdetermined, and thus needs some kind of regularization to make it more deterministic.

The traditional Bayesian OE method

At this point it is useful to qualify what we mean by prior information. The use of prior information in some form is unavoidable in any kind of GHG remote-sensing retrieval, since it is not possible to simultaneously measure all the parameters needed for determining the GHG mixing ratio. For instance, the absorption depends on the spectroscopic parameters, which are determined from laboratory measurements, and the atmospheric pressure and temperature profile, which are typically obtained from weather models. A comprehensive quantification of uncertainty that includes errors arising from all these sources of “prior” information is well beyond the scope of this work. Rather, we will focus on how the assumption of a prior GHG distribution in the atmosphere could affect the retrieved estimate of the GHG profile.

An uninformative prior is one that fills in information necessary for a retrieval (here a GHG profile) but it tries to be as vague as possible. In this paper, our uninformative prior makes use of the principle of indifference, which assigns equal probability to all possibilities. Though the uninformative prior is used to determine the principal component basis for retrieval, so long as the validation of the retrieved parameters is also done in the principal component basis, there is no bias incurred even if the uninformative prior differs significantly from the actual GHG profile.

Although traditional OE has become the de facto standard for satellite GHG
remote sensing

Between the traditional OE retrieval and the least-squares fitting via a simple
scaling method, there exist some intermediate choices. In a recent advance,

Dimension reduction via SVD has been previously used both for satellite
retrievals

Schematic of the various terms involved in a
greenhouse gas (GHG) measurement, retrieval and end use. The singular value decomposition (SVD) method introduces a new retrieval basis space

Remote-sensing measurements of GHGs are typically assimilated into a carbon
flux inversion system or other modeling (see Fig.

We choose a nadir sounding geometry with light traveling along a perfect vertical path. Lidar instruments satisfy this condition since they are pointed nadir and have the source and detector on the same platform.

We assume perfect knowledge of the optical path with a clear atmosphere. Lidar instruments are pulsed

We assume an undistorted measure of atmospheric transmittance with negligible instrument broadening. Lidar instruments have narrow laser linewidths, which determine their instrument lineshape function, which are typically 3–4 orders of magnitude narrower than spectrometers. The laser line width is negligible compared to the molecular absorption lineshape and can be assumed to be monochromatic.

We assume negligible interference from other atmospheric species via a careful line choice. Lidar instruments typically sample a single absorption line, rather than a full absorption band. For this narrow spectral range, absorption from other species can be ignored.

We assume a sufficient number of wavelength samples. Due to complexities in generating precisely tuned laser light for wavelength samples, many
lidar GHG-sensing instruments

We divide the atmosphere into

Next we define a measurement vector

and the forward model,

Additionally, we have normalized

As with most atmospheric measurements, the retrieval problem for GHG remote
sensing cannot be expressed as a nonsingular analytic expression based on
the forward model. In the remainder of this section, we will set up the
retrieval problem analogous to

Having set up the radiative transfer problem in Eq. (

Here, we will linearize the problem around the prior mean,

The measurement

To derive an estimate of the state

To find the optimum

In the above equations, we have carefully exercised our choice in linearly
mapping the physical world to

Equation (

The singular value decomposition (SVD) approach

Before getting into the formal derivation of the principal component basis, it is useful to bring in some physical intuition. The nature of the principal components is tied to the lineshapes of the various atmospheric layers. Pressure broadening of the lineshape in the atmosphere leads to the first principal component being shaped like a “mean” lineshape and representing a sort of column average. Higher-order principal components represent higher-order moments in the atmospheric profile and, as one would expect, are more challenging to measure.

The remainder of this section formally reviews and describes the SVD
framework along the lines of

To calculate the principal component basis

The new principal component

We note that the truncation of

Finally, for completeness, we will look at transformations between the

By substituting Eq. (

In the second equation line above, we have multiplied both sides by

Equation (

In this section we will explicitly describe the SVD retrieval as an OE
retrieval with a particular uninformative prior and the replacement of the
inverse with the pseudo-inverse in computing the gain matrix

We start with the analogous traditional OE version of Eq. (

One of the strengths of the OE method is the ability to propagate errors from
the inputs to the final estimate of the state vector

Note that the SVD posterior matrix in Eq. (

Since

The averaging kernels of the

In practice, biases in GHG measurements occur due to several reasons, many of
which are out of the scope of this paper. To limit the discussion on biases
arising from solving an underdetermined problem using some form of
regularization (retrieval error, which is universal to all GHG measurements)
we make two further assumptions:

Negligible error in the knowledge of the atmospheric pressure, temperature and water vapor profile (

Negligible errors in radiative transfer equations (forward model), instrument calibration or other similar systematic effects.

Retrieval errors in the traditional OE method can arise from incorrect
assumptions about the true greenhouse gas profile distribution. For the SVD
method, we see a potential bias in retrievals in the original

We will first derive the expected bias for the OE method where the input
prior mean and covariance matrix are incorrect. We assume that the true state
process

The expected bias is defined as

Looking at Eq. (

We now derive the bias when using the SVD method. We assume a true state

As we can see in Eq. (

Fortunately, when we look at the retrievals on the

As a caveat, we note that the bias derivation above assumes that the forward
model is linear (as is the case for GHG retrievals discussed here; see
Sect.

SVD retrievals can be validated directly in the retrieval

We choose the NASA Goddard

Forward-model

The forward model can be linearized to produce the kernel matrix

For IPDA lidar instruments, one primary limitation is photon shot noise, which is a fundamental quantum noise with variance equal to the number of photons detected. Photon shot noise is the fundamental limiting factor of measurement precision when lidar instruments are laser power limited, which is often the case. Although other forms of noise, such as detector dark current noise, laser speckle noise and solar background noise, also play a role, their effect on the principal components is limited. For this reason and for simplicity, we will assume a photon shot noise limited lidar instrument.

For this example, we will assume an integrated photon count of

SVD retrieval basis:

The

The third term from the SVD or second

The vertical dipole PC carries information about the vertical distribution of

As seen earlier in Eq. (

Higher-order

For passive spectrometers that work by resolving sunlight passing through the atmosphere, the instrument spectral resolution and sampling density are directly related and often close to one another. In contrast, lidar instruments probe the atmosphere with essentially monochromatic light; i.e. the laser spectral width is much narrower than gas absorption linewidth, and have spectral resolutions orders of magnitude better than the sampling density. It is important for the reader to note that high-quality measurements for the purposes of obtaining vertical information require high spectral resolution but not necessarily high sampling density. As we shall see in this section, for a given sampling density, the measurement precision depends strongly on the instrument spectral resolution.

Retrieval uncertainty versus instrument spectral linewidth for the first two

We calculate the expected random noise in retrieving the first two PCs for a
range of instrument spectral resolutions (see Fig.

The ability of satellite-based passive spectrometers to resolve the

In this section, we will look at the retrieval performance of the SVD and
traditional OE (refers to OE retrievals with finite

Comparing SVD and OE retrievals: using simulated data

Sample retrieval for a single simulated measurement (noise instance) under a weak constraint
(four principal components for SVD, 100 % prior uncertainty for each

For the simulations, we use a

For each case, we define a “true”

Since the SVD principal components are unbiased, we make quantitative
comparisons between the two techniques in that

For the SVD approach, we set the uninformative prior

Sample retrieval for a single simulated measurement under strong constraint (one principal
component for SVD, 0.1 % prior uncertainty for each

Ensemble results for retrieved

GHG retrievals require some sort of constraint to regularize the retrieval
problem (see Sect.

SVD retrievals are constrained by the number of principal components used in
the line-fitting. While the constraint is applied in qualitatively different
ways to the two retrieval methodologies, the effect is somewhat similar
particularly for weak constraints, since the SVD method is the limiting case
of a weak prior constraint (discussed in Sect.

We choose a sample profile from an atmospheric

Ensemble results (Fig.

Sample simulated measurement of vertical dipole moment using the SVD method and
appropriate constraint.

A strong constraint puts restrictions on the state vector and prevents a
retrieval from fully minimizing the residual. Here, we set the prior
uncertainty in

Ensemble results (Fig.

Although a rather extreme constraint has been applied for the traditional OE method, the results show that there are intrinsic problems in using a constraint that is too strong. Often, such biases are subtle and less obvious, but nevertheless affect flux measurements, which are based on several thousand soundings and are sensitive to small biases. In contrast, the SVD approach is more robust.

Robust measurement of the X

Robust measurement of the

Having demonstrated the SVD method's general robustness, we now look at the
extraction of vertical information about the

Figure

Figure

The SVD framework and its use of principal components provides a
mathematical basis on which to determine what information can be extracted from GHG
column absorption measurements. Section

Although the numerical results from the SVD method have been shown for the

The primary benefits of using the SVD method with retrievals in the principal
component basis can be summarized as follows:

retrieval of higher-order terms of the greenhouse gas vertical distribution (beyond the column mean) in the atmosphere,

no bias from the use of an uninformative prior,

orthogonality of principal components leading to robust retrievals independent of the degree of constraint (number of components solved for).

The robustness of the SVD method makes it useful in situations where the
prior state is not well known or the uncertainty in the prior is not well
quantified. For instance,

The robustness of the SVD method may also make it easier to use in an
operational environment where atmospheric and surface conditions can change
the measurement precision significantly. Rather than using advanced retrieval
methods to get vertical information separately from the main retrieval of the
column mean (as in

Furthermore, when performing the retrieval in the principal component basis, the SVD method requires fewer computations than the OE method, which works in the full model basis. This has the potential to make the retrieval faster and more efficient. In addition, the reduced basis of mutually orthogonal principal components makes retrieval analysis easier. Troubleshooting systematic or forward-model errors are also simpler in the principal component basis since the basis is smaller and the prior is uninformative, allowing one to more easily see the effects (manifested as a bias) on the different components.

In practice, interference from other gas species in the atmosphere (for
instance, water vapor) and instrument systematic errors prevent the
simultaneous realization of all benefits listed in Sect.

The presence of a water vapor line at the shoulder of the

The SVD method discussed here bears some similarity to the approach used by

There are also some important differences between the two methods. In using
principal components, the SVD retrieval produces orthogonal parameters that
have uncorrelated errors and thus errors in the X

Going beyond the domain of trace gas retrievals to the broader problem of atmospheric sounding, the simulations shown in this paper underscore the importance of choosing a proper Bayesian prior and prior covariance if using the OE method. Ideally, the choice of these parameters will be from a large sampling of the true state space. In the absence of such data, the prior mean may be different from the true mean. Setting or tuning of the constraint from the Bayesian prior for the purpose of regularization of the retrieval problem runs the risk of overstating prior knowledge and thus causing a bias.

Decision tree for a suitable retrieval approach when the first principal component is a column
mean. The quality of prior information compared to the signal-to-noise ratio (SNR) determines which retrieval
method would be better suited. The SVD method (with principal components retrieved) is robust and can be applied
to a range of situations. However, in situations where the prior information is good (relative to the measurement
SNR), the OE method offers a clear advantage of a lower variance in the retrieved X

In choosing between the SVD method and the traditional OE method, one needs
to factor in the quality of the prior information (See Fig.

We have described an approach to deducing vertical information from column
GHG retrievals based on the singular value decomposition. The SVD approach
does not require an assumption of a prior distribution of the GHG profile for
regularizing the retrieval problem, and by using the principal component
basis for retrievals, the prior is rendered uninformative. Simulations
comparing the SVD method to the traditional Bayesian OE (using an informative
prior) show that the SVD method is more robust and better suited to
situations where prior knowledge of the

Intuitively, OE derives an estimate of the state using both the measurement and prior knowledge, while SVD only uses the measurement to inform its estimate. When the prior information is correct, there is no doubt that OE will have lower posterior uncertainty since OE can leverage an extra source of information to more efficiently derive its estimate. However, this efficiency comes at a potential cost when the prior is incorrect. For instance, we showed that when OE uses an incorrect prior mean, then the estimate is guaranteed to be biased. Estimates from the SVD method in the principal component basis, on the other hand, are insensitive to incorrect information coming from the prior. The choice between SVD and OE then mostly comes down to how well one understands the prior distribution of the state of interest.

In this work, we have assumed a perfect forward model and only random errors
in the measurement. This is a necessary first step check for the feasibility of
the method. However, in practice, other sources of error such as imperfect
instrument calibration, imperfect knowledge of atmospheric state and forward-model approximations play important roles. Our preliminary attempts using

Another interesting topic is extending this work to nonlinear forward
models, where the minimization of the loss function in Eq. (

Future work will also explore other aspects of the measurement problem such as determining the optimal wavelength sampling. In contrast to passive spectrometers, which can have a large number of samples, lidar instruments bear some cost for each additional sample. While in theory one needs a wavelength sample for each principal component retrieved, in practice one needs to oversample the line to help reduce systematic errors (control biases). Determining the optimal wavelength sampling to best obtain information about the vertical distribution of the GHG while keeping biases low is important in the design of space-based IPDA lidar instruments for GHG measurements.

All data shown in this work have been generated using simple model simulations as described in this work. Details of the code and data are available on request.

AKR led this work and the development of the paper and performed the simulations. HMN derived the SVD theoretical framework and its connection to OE work and co-wrote the paper. XS derived the formalism related to the lidar instrument. XS, JM, JBA, JMH and AB provided scientific input during the development of this work and key critical feedback during the writing of the paper.

The authors declare that they have no conflict of interest.

Anand K. Ramanathan, Xiaoli Sun, Jianping Mao and James B. Abshire are grateful for support from the NASA ASCENDS Mission Science definition activity. Edited by: Kimberly Strong Reviewed by: three anonymous referees