An algorithm for retrieval of vertical profiles from ground-based spectra in
the near IR is described and tested. Known as GFIT2, the algorithm is
primarily intended for CO

Since 2004 the Total Carbon Column Observing Network (TCCON) has measured
ground-based near-IR solar spectra. Their high spectral resolution and signal-to-noise ratio
(SNR) allow high-precision measurements of total overhead column abundance of
CO

Allowing the a priori CO

Column averaging kernels for simulated retrievals. “Profile
retrieval” and “profile scaling” refer to a typical retrieval using GFIT2
on the 6220 cm

It is worth noting that profile retrieval is unlikely to reduce
uncertainties in determination of

Recent work on profile retrieval development includes Kuai et al. (2013), who
retrieved CO

This paper describes the experimental implementation and early tests of a
profile retrieval algorithm for TCCON spectra. Its layout is as follows. In
Sect. 2 we describe the standard algorithm used for TCCON and briefly
discuss the history of profile retrieval and the chosen algorithm for
similar measurements. Section 3 describes implementation of the algorithm as
GFIT2. Section 4 presents tests of GFIT2, both for synthetic spectra and for
real spectra taken to coincide with overpasses by aircraft making in situ
CO

Partial column averaging kernels for the profile retrieval algorithm
of Fig. 1 (GFIT2 applied to the 6220 cm

GFIT is the algorithm adopted by the TCCON for analysis of the spectra; it was developed over many years by Geoff Toon at JPL. GFIT is also used to analyze MkIV balloon spectra (e.g., Sen et al., 1996) and was used in the Version 3 processing of ATMOS spectra (Irion et al., 2002). It is a profile scaling algorithm, employing a quasi-linear regression to derive scale factors for all important absorbers as well as other atmospheric and instrument parameters, such as continuum level and frequency shift.

GFIT is designed in such a way that its “forward model” is independent of and separable from its “inverse method”. These terms are discussed in Rodgers (2000), but briefly the forward model is an algorithm that calculates the atmospheric spectra comparable to the observed spectra, incorporating radiative transfer and molecular physics along with assumed gas distributions. The inverse method retrieves a state vector of parameters, such as molecular mixing ratio, by finding values which provide a best fit to the spectrum given other assumptions and constraints. The GFIT inverse method is a form of “optimal estimation” as described further below, which applies the Gauss–Newton method, iteratively estimating the parameters by successive approximation.

Ground-based spectra – at microwave, IR, and UV wavelengths – have been analyzed in selected applications for limited altitude profile information, for many years (Connor et al., 1995, 2007; Pougatchev et al., 1996; Schofield et al., 2004.) The physical origin of the limited profile information in ground-base spectra is somewhat varied. Most commonly, the pressure-broadened line shape is exploited; however the use of lines of varied opacity (see Fig. 1) and the use of multiple atmospheric paths (Schofield et al., 2004) are also sources of profile information.

The most common algorithm for the inverse method used in these and other studies is optimal estimation, formulated by Rodgers (1976, 2000). That algorithm will be described in detail in the following section.

Optimal estimation has been implemented as a user-selected option of inverse
method added to the version of GFIT publically released in 2012; no other
changes to the standard GFIT algorithm were made. The modified algorithm is
known as GFIT2. GFIT is designed to treat each spectral band independently.
All calculations in this paper are of the 1.61

A typical spectrum in the 6220 cm

The optimal estimation formulation of Rodgers (2000) was adapted and applied for use with the “full physics” algorithm (inverse method plus forward model) developed for the first Orbiting Carbon Observatory (OCO) satellite, which failed to reach orbit in 2009. The OCO inverse method, as it existed in 2007, is described in Connor et al. (2008) and was used as a starting point for the development of GFIT2. Much of the discussion in Sect. 2 of Connor et al. (2008) is directly applicable.

The OCO inverse method was adapted for use with GFIT and is briefly
described here. We use the notation and concepts of Rodgers (2000). The
spectrum, or measurement vector

The solution of the GFIT2 inverse method is the state vector

After each iteration, we test for convergence. To facilitate that, we
compute the change in the solution scaled by its estimated variance:

If d

Lastly, we compute the retrieval covariance matrix,

To enable use of the Rodgers algorithm, a modified GFIT code was developed which completely separates the forward model and inverse method, and allows integration of optimal estimation profile retrieval with the existing code. Conceptually, the experimental, integrated GFIT allows selection of the existing (profile scaling) or modified (profile retrieval) algorithm. This is simply accomplished by setting a parameter in an input file. The integrated algorithm has input and output files identical to the existing GFIT, plus new input and output files specific to profile retrieval, which are not required unless the modified algorithm is selected.

A critical input to the algorithm of Eq. (1) is the measurement error
covariance,

The simplest way of “de-weighting” spectral features which remain in the residuals is to increase the estimated measurement error estimate (equivalently, reduce the assumed SNR) at all frequencies, so that residual features are ignored (treated as measurement error) (e.g., Connor et al., 1995). As we will see, in practice this is somewhat effective at damping oscillations, but only at the cost of losing most of the profile information in the spectrum.

An alternative approach we have attempted to avoid profile oscillations is to
vary the assumed spectral error to reflect the real residuals obtained by
spectral fits (Rodgers and Connor, 2003). A two-stage retrieval is run, in
which stage 1 is profile scaling. The residuals from stage 1 are then used to
estimate the spectral error as a function of frequency and inserted on the
diagonal of

A third approach to the problem of systematic residuals is to estimate them empirically, and then to include them in the forward model, multiplied by a scale factor retrieved as part of the state vector (JPL, 2015). Perhaps the simplest approach is to estimate the systematic component by averaging the residuals over the entire set of spectra under study. This technique and simple variants on it will be described in Sect. 4.4 below.

The full state vector consists of the CO

A critical input is the a priori covariance matrix

The only other input parameters specific to profile retrieval concern
convergence and goodness of fit. They include the convergence parameter
defined in Sect. 3.1, the maximum acceptable

The algorithm was first tested by retrievals on synthetic spectra, where the “true” atmospheric profile is known. In these tests, the forward model (used by the algorithm) may be the same as the forward function (which includes all true physics) or may differ from it in a controlled way.

We illustrate the most basic test in Fig. 4. Here a synthetic spectrum was calculated from the profile labeled true (diamonds); then GFIT2 was run on the calculated spectrum without modification, using the a priori profile shown. (The a priori profile is selected on the basis of climatology.)

The assumed signal-to-noise ratio was 1000. The solid lines in Fig. 4 show
the retrieved profile, and the degrees of freedom for the profile are shown
in the legend. The two results shown in Fig. 4a and b are typical. The
retrieved profile has 3.3–3.5

Next we used a known instrumental limitation to test the skill of the variable-SNR technique. Namely, while the instrument nominally points at the center of the solar disk, it is common for some error to be introduced by patchy cloud cover or simply by tracking hardware problems. The effect of such pointing error is to introduce a Doppler shift due to solar rotation, making calculations of the solar Doppler shift inaccurate, and thus result in an uncompensated shift in the position of solar lines relative to telluric lines.

To assess the effect of pointing error, we assumed that an error was present
equal to 10 % of the solar diameter, which produces an error in the solar
Doppler shift of

The retrieval in Fig. 5a is qualitatively similar to the one with higher DoF
in Fig. 4a. The retrieved profile in Fig. 5b spreads the increased CO

Error in spectroscopic parameters is an important cause of systematic error in the calculated spectra, leading to systematic structures in the spectral residuals. Since the profile retrieval depends critically on the spectral line shape, spectroscopic errors will limit its performance, possibly severely. The most obvious source of spectroscopic error affecting line shape is the pressure-broadening coefficient, which simply scales the linewidth at a given pressure. It is arguably the largest source of line shape error as well. We will use synthetic spectra and simulated retrievals to evaluate its effects.

For this purpose we have multiplied the pressure-broadening coefficients in
the relevant CO

The scaling retrieval reduces the CO

Fortunately, errors in real retrievals are unlikely to be as large as in these simulations.

Much effort in recent years has gone into refining knowledge of the
spectroscopic parameters needed for modeling atmospheric CO

The extreme sensitivity of the CO

As a preliminary test, we assume that the spectroscopic error signature is given by the residuals of the scaling retrieval and add those to the calculated spectrum in the profile retrieval. The average profiles, corresponding to Fig. 6a, are shown in Fig. 6b.

The average retrieved profile is nearly identical to the scaling retrieval;
no spurious changes in profile shape are introduced. The derived

Of course, for real measurements, the signature of spectral error is not so easily derived. Later, in Sect. 4.4, we calculate the mean residual vector for large sets of real measurements and attempt profile retrievals including a scale factor applied to the mean residual vector.

Another potentially significant source of error is distortion of the measured line shape itself. For Fourier transform spectrometers (FTSs, as used by TCCON) the ILS is a convolution of contributions from the finite path difference and the finite field of view (FOV) of the FTSs. The path difference and its ILS contribution (a sinc function) are well known, but the FOV, which contributes a rectangular shape, has an uncertainty we estimate as 7 %. This causes the observed line to be broader and weaker than the atmospheric line, and it progressively has a larger effect as the line is narrower; i.e., the error due to finite aperture becomes more important at lower pressure where the intrinsic line shape is narrower (see for example Davis et al., 2001).

We illustrate this effect in Fig. 7, which is calculated for the same spectra
as Fig. 6 and so is directly comparable to Fig. 6a. The net effect of this
error is very small in the lower troposphere and grows only to

Retrievals of the spectra used for Fig. 6 but with an assumed error of 7 % in the instrument field-of-view.

Atmospheric spectra are routinely measured in Lamont, Oklahoma, at the
Southern Great Plains site of the Department of Energy Atmospheric Radiation
Measurement network. A Cessna aircraft equipped with air sampling in situ
detectors is flown there on a regular basis and produces CO

With this in mind, we have chosen several days for study. On these days, Cessna flights were made, atmospheric conditions were excellent, and many high-quality near-infrared spectra were recorded. We selected days from various times of year to allow conditions as variable as possible. The specific days chosen are 15 June 2011, 5 July 2011, 28 July 2011, 26 August 2011, 24 Decemnber 2011, 14 January 2012, and 15 January 2012.

Analysis of the data from these days immediately revealed significant errors in the solar Doppler shift. This is not only shown by simple examination of the residuals but is also formally calculated, as the difference in the frequency shift observed for solar lines (after correcting for the calculated Doppler shift) and telluric lines. GFIT does not automatically take this error into account, by recalculating the spectrum with the correct Doppler shift. However, these errors can be corrected by using the retrieved solar–telluric difference to correct the calculated Doppler shift, and then re-running the retrieval. All measured spectra and retrievals used and/or shown in this paper have been “Doppler-shift-corrected” in this way.

Another potential issue is the temperature profile used in performing each retrieval.

In clear, dry conditions, the temperature at the surface will sometimes vary
by as much as 20 K during daylight. This has an important impact on retrieval
of tropospheric CO

Note that the CO

As described in Sect. 3.2, we first attempted retrievals by setting the
SNR low enough to avoid trying to fit systematic spectral
residuals. The SNR observed to achieve this for the current dataset is
approximately 100. Two examples are shown in Fig. 9. Figure 9a may be
compared directly to Figs. 4b and 5b. A smoother version of the Cessna data
is the true profile in Figs. 4b and 5b. The a priori profile is the same.
Note that the degrees of freedom for the CO

We see that the retrieval in Fig. 9a poorly matches the Cessna profile in the
lower troposphere, although it is a small improvement on the a priori. The
profile in Fig. 9b matches CO

For present purposes the thing to note is that assuming SNR

We attempted to include as much profile information as possible while
avoiding “over-fitting” the spectral regions with the poorest residuals, by
using the variable SNR as described in Sect. 3.2. We illustrate by showing
results for the same days as in the preceding section, 28 July 2011 and
15 January 2012. We show in Fig. 10 the effective SNR on which we based the
diagonal of

Signal-to-noise ratio (SNR) derived for 28 July 2011 from the variable-SNR technique (see text).

Note that the effective SNR varies from

Examples of profiles retrieved with variable SNR are given in Fig. 11. CO

The performance of the profile retrieval algorithm with variable SNR on the measured spectra tested is clearly unsatisfactory. In an effort to understand this limitation we next closely examine the spectral residuals the algorithm produces, as described in the next section.

Past experience indicates that the oscillatory behavior of the profiles seen in Fig. 11 is most likely driven by a failure of the forward model to adequately reproduce the measured spectrum. This of course may reflect systematic error in either the model or the instrument. To isolate the spectral signature of the error, we calculated mean residuals from many spectra. We further expanded our study to examine the residuals produced by the same model instrument at a different site, namely Lauder, New Zealand.

On the 7 days examined in Sect. 4.2, a total of 5946 good-quality spectra
were recorded at Lamont. There were

At Lauder we examined every 10th spectrum recorded on 13 days selected for
clear sky and seasonal coverage. The days were 10 July 2010, 11 July 2010,
30 July 2010, 24 August 2010, 8 September 2010, 2 November 2010, 7 November 2010,
8 November 2010, 4 February 2011, 16 February 2011, 1 April 2011, 21 May 2011, and
28 September 2011. A total of 621 spectra were included; the number of spectra per day
ranged from

Figures 12–13 show expanded views of portions of the mean residuals for the
two sites, for air mass < 2 (Fig. 12) and for
2 < air mass < 4 (Fig. 13). A pair of two (upper and
lower) panels is shown for each spectral interval, 6205–6210 and
6240–6245 cm

Mean and standard deviation of residuals in selected spectral intervals, for measurements with air mass < 2.

It is immediately clear that there are systematic residuals of

While to first order all the CO

Mean and standard deviation of residuals in selected spectral intervals, for measurements with 2 < air mass < 4.

We strongly suspect that a stable profile retrieval is not possible in the presence of systematic spectral errors as suggested by the residuals of Sect. 4.3 and that these will readily produce the unsatisfactory oscillations seen in Fig. 11. This systematic spectral signature might be thought of as a “bias” of the GFIT forward model which prevents it from fitting the measured spectra precisely enough. The GFIT forward model, as with most practical atmospheric spectral line models, uses the Voigt profile as its line shape. However, it has recently become well understood that the Voigt line shape is inadequate to model atmospheric spectra at the sub-percent level. See, for example, Fig. 1 of Long et al. (2011). Unfortunately, improved line shape functions are far more complex, and, while several of them are known to improve spectral fits in the laboratory (ibid.), there is no agreement as to which of them contains the best physical description of the line formation. Since atmospheric spectra are formed in far different physical conditions than laboratory spectra, it is unclear how to modify the forward model to improve the observed spectral fits.

Pending a future clarification of the physics of line formation, we have
attempted to stabilize the algorithm by performing simple “corrections” to
the forward model to remove, as far as possible, the spectral bias. In
the first instance, we have performed retrievals on the Lamont spectra
discussed in Sect. 4.2, accounting for systematic spectral residuals as
follows. We have modified the forward model to include addition of a spectral
basis vector, multiplied by a scale factor, to the modeled spectrum. We
calculate the mean residual spectrum from a large set of the Lamont
retrievals (the set to be defined shortly). We then use those mean residuals
as the basis vector to be added to the modeled spectrum. The scale factor
which multiplies the basis vector is incorporated in the state vector, to be
retrieved for each measured spectrum. It is typically

In the first instance, we derive the mean residual from the full set of 7
days of data and use this as the basis vector in retrievals from each measured
spectrum. We show two of the daily retrieved profiles (each the average of

Figure 14a and b are directly comparable to Fig. 11a and b. The same data and algorithm are used except for the addition of the scaled mean residual as just discussed.

The comparison of Figs. 11 and 14 shows a dramatic improvement on
15 January 2012, eliminating the large oscillation of Fig. 11b. Also on
28 July 2011,
the profile of Fig. 14a, after subtracting the mean residual,
is less oscillatory than previously (Fig. 11a). On both days, the integrated
column from the profile retrieval, represented by

Deriving the residual vector from the full 7 days of measurements implicitly assumes that the residuals are independent of seasonal effects and instrumental adjustments over a long period. This is unlikely to be the case; in fact we have already noted in discussing Figs. 12–13 that the residuals depend to some degree on air mass. The mean air mass of a set of spectra will vary with season and times of day. To lessen the impact of both air mass and potential instrumental variations in the residuals, we have used monthly residuals calculated in a limited range of air mass (1–2 or 2–4) as the spectral correction vectors and run the retrievals once more. The results are shown in Fig. 15.

Unfortunately these results are a clear step backward. Figure 15a shows no
sensitivity to the enhanced lower-tropospheric CO

Our best results to date come from adding the scaled mean residuals of the set of days under study. With that in mind we will expand the discussion to include days other than the two illustrations used so far.

Figure 16a and b show results for 26 August 2010 and 14 January 2012. They were
produced in the same way as Fig. 14a and b, that is, including addition of a
scaled mean residual. The two dates are chosen to illustrate two features of
the full set of retrievals. Namely, on 26 August we see a smooth profile with
only a suggestion of oscillation, which seems (maybe fortuitously) to track
some enhanced CO

In summary these results fall into two classes. In one, the retrieved profile
is reasonably well behaved but offers little if any improvement on the
profile scaling version. In the other, the retrieved profile suffers serious
oscillations.

The algorithm behaves as expected on synthetic data. On real data, results
are usually worse than scaling, given our a priori knowledge of the CO

There are at least two directions to follow in pursuit of useful profile retrievals. One is improvements to the forward model. These could be in the form of more accurate values of spectral parameters, more appropriate models of spectral line shape, and/or knowledge of the instrument line shape. All of these areas have, however, already been the focus of intense work over an extended period of time, and breakthroughs may be slow in coming.

A second alternative is to exploit profile information from sources other
than the pressure-broadened line shape. An immediately accessible source is
spectral regions of higher and lower opacity than the spectral band
considered here. In particular, several other CO

An apparently simple alternative has been suggested, namely imposing
a priori constraints on the profile shape by experimenting with
explicit interlayer correlations in the a priori covariance matrix

Nevertheless explicit interlayer correlations may damp undesirable oscillations, and their effect should be explored. They have been used routinely in the OCO/OCO-2 retrieval algorithms (Connor et al., 2008; JPL, 2015), where experience shows that the nature and strength of correlations is key to doing this successfully. A whole series of experiments, analogous to those presented in Sect. 4 of this paper, could be envisioned to decide how best to apply correlations and to evaluate their efficacy. This would be a valuable part of a follow-on study, especially if combined with the multiband retrieval approach. Regrettably, it is beyond the scope of the present effort.

Finally, it is our intention to release GFIT2 to the community, as an option within the public version of GFIT. That would allow testing and development by a wider range of experienced researchers. So far that has proven impractical, but we hope to do so in the near future.

Part of this research was performed at the Jet Propulsion Laboratory,
California Institute of Technology, under contract with NASA. We thank
NASA's Carbon Cycle Science Investigation Program for supporting the
development of GFIT2 (NNX14AI60G). Operations of TCCON at Lamont, Oklahoma,
are made possible by NASA's OCO-2 project in collaboration with the DOE ARM
program. Cessna data from the SGP are available through the ARM archive
(