The ACOS CO 2 retrieval algorithm – Part 1 : Description and validation against synthetic observations

This work describes the NASA Atmospheric CO 2 Observations from Space (ACOS) XCO2 retrieval algorithm, and its performance on highly realistic, simulated observations. These tests, restricted to observations over land, are used to evaluate retrieval errors in the face of realistic clouds and aerosols, polarized non-Lambertian surfaces, imperfect meteorology, and uncorrelated instrument noise. We find that post-retrieval filters are essential to eliminate the poorest retrievals, which arise primarily due to imperfect cloud screening. The remaining retrievals have RMS errors of approximately 1 ppm. Modeled instrument noise, based on the Greenhouse Gases Observing SATellite (GOSAT) in-flight performance, accounts for less than half the total error in these retrievals. A small fraction of unfiltered clouds, particularly thin cirrus, lead to a small positive bias of ∼0.3 ppm. Overall, systematic errors due to imperfect characterization of clouds and aerosols dominate the error budget, while errors due to other simplifying assumptions, in particular those related to the prior meteorological fields, appear small.


Introduction
Despite decades of research, there remain significant uncertainties in many elements of the global carbon cycle and its response to anthropogenic perturbations.It is well-known that, on average, slightly more than half of annual anthropogenic CO 2 emissions are taken up by the lands and oceans (Le Quéré et al., 2009).However, details of this uptake, such as its regional-scale distribution, the dominant processes governing its interannual variability, and how it may evolve into the future are not well understood.Determining the physical mechanisms that govern carbon sources and sinks will enable significant uncertainty reduction in the projections of global climate change (e.g., Friedlingstein et al., 2006).
Ground-based and aircraft observations give an excellent picture of the global atmospheric CO 2 growth rate and even some reasonable information for hemispheric gradients, but there is an insufficient number and too sparse a spatial distribution of these observations to accurately infer carbon fluxes on regional scales.Accurate, global measurements of column CO 2 concentration, coupled with atmospheric transport models, should complement ground-based and aircraft measurements and allow for the "top-down" monitoring of regional-scale carbon sources and sinks on timescales of weeks to months (e.g., Rayner and O'Brien, 2001;Miller et al., 2007;Chevallier et al., 2007;Baker et al., 2010).
Space-based measurements of CO 2 already exist in the mid-troposphere from several thermal infrared instruments (e.g., Chédin et al., 2003;Chahine et al., 2008;Crevoisier et al., 2009;Kulawik et al., 2010), though none of these were designed with the explicit goal of monitoring CO 2 .Inversions that ingest these data have found that they provide limited information on surface-atmosphere fluxes, and only on broad spatial scales (Chevallier et al., 2009;Breon and Ciais, 2010).The SCanning Imaging Absorption SpectroMeter for Atmospheric CartograpHY (SCIAMACHY) (Schneising et al., 2011;Buchwitz et al., 2007).Unfortunately, these measurements have rather high scatter and potential large-scale artifacts that hinder their use in sourcesink estimation (Breon and Ciais, 2010).However, recent results using an improved algorithm (Reuter et al., 2010(Reuter et al., , 2011) ) show promise and compare well to ground truth, and render the outlook for SCIAMACHY CO 2 data more optimistic.
A new era of space-based CO 2 monitoring is at hand with the appearance of dedicated CO 2 instruments such as the Greenhouse gases Observing SATellite (GOSAT) (Kuze et al., 2009;Yokota et al., 2009), which successfully launched in 2009, and the second Orbiting Carbon Observatory (OCO-2) with a launch anticipated in late 2014.OCO-2 is a follow-on to the original OCO mission (Crisp et al., 2004) which failed to achieve orbit upon launch in 2009.Both OCO-2 and the Thermal And Near infrared Sensor for carbon Observations -Fourier Transform Spectrometer (TANSO-FTS) instrument aboard GOSAT infer carbon dioxide concentration via high-resolution measurements of reflected sunlight.These instruments monitor radiation in three NIR bands: one at 0.76 µm containing significant O 2 absorption (the so-called O 2 A band), one at 1.6 µm containing weak CO 2 absorption, and one near 2.1 µm containing strong CO 2 absorption.In contrast to the thermal infrared observations, these NIR measurements have a nearly uniform sensitivity to CO 2 from the surface up through the middle troposphere.The goal is to characterize the column-averaged dryair mole fraction of CO 2 , called X CO 2 , for each sounding: where u(z) is the CO 2 mole fraction with respect to dry air at altitude z, and N d (z) is the total molecular number density of dry air at altitude z.Note that definitions of X CO 2 vary slightly in the literature concerning how the vertical weighting is done, which can lead to differences in X CO 2 of tenths of a part-per-million (ppm).For instance, Connor et al. (2008) weights CO 2 concentration by pressure, and Reuter et al. (2010) weights by the total number of air molecules, rather than by the dry air component.Our definition is consistent with that of the Total Carbon Column Observing Network (TCCON) (Wunch et al., 2010).
After the launch failure of the first OCO mission, the OCO team was invited to join the GOSAT team in analyzing GOSAT observations, under the auspices of the NASA Atmospheric CO 2 Observations from Space (ACOS) task.Since 2009, the X CO 2 retrieval algorithm originally developed for OCO has been modified to allow GOSAT retrievals.An early version of the OCO X CO 2 retrieval algorithm first motivated the simultaneous use of the three NIR spectral bands identified above (Kuang et al., 2002).The algorithm development continued, and was later described in Bösch et al. (2006, hereafter B06) and Connor et al. (2008, hereafter C08).B06 described the forward model as well as results of the algorithm as applied to SCIAMACHY data over the TC-CON site in Park Falls, Wisconsin, USA.C08 described the inverse model and results of a linear error analysis.More recently, Bösch et al. (2011) examined the theoretical performance of the algorithm, but this study was limited in that no systematic errors were included.
Since these studies, the retrieval algorithm has been refined in several notable ways.Advances in the forward model have greatly enhanced the ability to fit GOSAT spectra.A novel cloud screening algorithm has ensured that scenes with thick clouds and aerosols are mostly removed.Finally, a series of post-processing filters now remove retrievals of poor or questionable quality.
In the absence of atmospheric scattering, absorption-only techniques such as Differential Optical Absorption Spectroscopy (DOAS) (e.g., Buchwitz et al., 2000, and references therein) can retrieve sufficiently accurate values of X CO 2 .However, it has been shown that with optically thin clouds or aerosols present, neglecting scattering can lead to unacceptably large retrieval errors (O'Brien and Rayner, 2002;Houweling et al., 2005;Aben et al., 2007;Butz et al., 2009).Many approaches have been devised to account for scattering affects in the retrieval of carbon dioxide (e.g., Kuang et al., 2002;Bösch et al., 2006;Connor et al., 2008;Oshchepkov et al., 2008Oshchepkov et al., , 2009;;Yoshida et al., 2011;Reuter et al., 2010).However, tests to prove their efficacy in accounting for these scattering effects are sometimes incomplete; for instance, most studies use only Lambertian surfaces, test one or two solar zenith angles, include limited types or vertical distributions of clouds or aerosols, and/or assume perfect cloud and aerosol screening below a relatively low maximum optical depth (such as 0.3).That said, several of these algorithms have recently shown relatively good agreement X CO 2 agreement as compared with simultaneous, colocated TCCON observations (Morino et al., 2011;Butz et al., 2011;Wunch et al., 2011b) or models (Oshchepkov et al., 2011).
While GOSAT contains the Cloud and Aerosol Imager (CAI) for detecting cloudy scenes, OCO-2 will not.The ACOS algorithm, while being tested on GOSAT data, will also work for OCO-2 data when available; therefore, it only uses information from the narrower OCO-2 windows within the GOSAT spectra, and does not use CAI data except for validation.The primary purpose of this article is thus twofold: first, to give a summary of the current X CO 2 retrieval algorithm used in the ACOS processing of GOSAT data; and second, to evaluate algorithm performance with a series of highly realistic, simulation-based tests that go beyond what has typically been reported in the literature.
The current operational ACOS retrieval algorithm, Build 2.8 (B2.8), has processed all GOSAT NIR soundings from 4 April 2009 to 20 April 2011, for both land and ocean targets.These data are freely available through the NASA Goddard Earth Sciences Data and Information Services Center 1 .However, the over-ocean "glint" retrievals are very preliminary, and will not be discussed further here.
Results of this algorithm as applied to GOSAT data are described in a companion paper (Crisp et al., 2012), and an extensive validation of those retrievals is described in Wunch et al. (2011b).The rest of this article is organized as follows.Section 2 summarizes the retrieval algorithm and pre-screening filter.Section 3 describes synthetic retrieval tests, including the generation of the realistic, synthetic GOSAT spectra upon which to test the algorithm, the tests themselves, and the performance of the pre-and post-processing filters used.Conclusions are presented in Sect. 4.

Retrieval algorithm
The ACOS retrieval algorithm was originally developed for the first OCO instrument.Full details of the algorithm and its implementation are given in the ACOS retrieval Algorithm Theoretical Basis Document (Crisp et al., 2010).We summarize the salient elements below.
The algorithm employs an optimal estimation approach, in which input parameters of a forward model are optimized to yield simulated spectra that best match the observed spectra, whilst simultaneously being constrained by prior information (see e.g., Rodgers, 2000).The spectra to match are each of the three OCO NIR bands: band 1 near 0.76 µm, band 2 near 1.6 µm, and band 3 near 2.1 µm.The bands and their spectral ranges are summarized in Table 1.The forward model parameters to be optimized constitute the state vector x.All channels from the three bands are aggregated into observations vector y.Mathematically, the simulation of observations y from a state vector x takes the form where F is called the forward model of the retrieval, b is a set of fixed input parameters (such as gas absorption coefficients, view angles, etc.), and contains both instrument noise and estimates of forward model errors.Because GOSAT measures each channel with two orthogonal polarizations, denoted P and S, there is a choice to make regarding the composition of the y vector.The ACOS retrievals use the quantity  (P + S)/2, rather than P and S independently or some other combination.This quantity is nearly the same as the total unpolarized intensity (O'Brien et al., 2011).This approach sacrifices potential polarization information, but has the advantage that it requires less accuracy of the polarization-sensitive components of the forward model.This will not be the case for OCO-2, which will measure light in a single polarization state.
The basic flow of the retrieval algorithm is given in Fig. 1.External data products are shown as ovals, while processing steps are shown as rectangles.The input "Level-1B Product" contains the calibrated, spectrally-resolved radiances for each of the three spectral bands, as well as pointing and geometrical information.The "Met Data" contains meteorological fields that are used to inform the prior.A pre-screening step (Sect.2.4) first removes data of bad quality (low signal, instrument problems, etc.) and scenes flagged as cloudy.Next, the filtered data are passed to the core of the algorithm, the X CO 2 retrieval.In this step, an a priori state x a is constructed based upon meteorological inputs and the observed spectra (Sect.2.1).The first-guess state vector is taken to be the prior for simplicity.An inverse model, coupled with the forward model F(x), solves for the state vector x that minimizes the χ 2 cost function: where S is the observation error covariance matrix, S a is the a priori covariance matrix, and T represents the matrix transpose.X CO 2 is calculated directly from x via Eq.( 1).The rest of this section summarizes the state vector composition and priors, the forward model, inverse model, and X CO 2 calculation, with particular emphasis placed on those elements that have changed since the original algorithm descriptions given in B06 and C08.

State vector composition and priors
The state vector contains parameters that are formally optimized during the inversion process, and as such represent physical quantities to which the spectra are sensitive.The ACOS retrieval state vector is described in Table 2, including both a priori values and errors.Carbon dioxide is retrieved as a vertical profile of dry-air mole fraction defined on a prescribed set of twenty atmospheric pressure levels.These fixed levels are spaced nearly equally in pressure, with the highest pressure at 1050 hPa to ensure all real surface pressures fall within this prescribed set (Fig. A1).The prior CO 2 profiles are derived from a forward run of the Laboratoire de Meteorologie Dynamique (LMDz) model, with fluxes optimized to match surface observations (Pickett-Heaps et al., 2011).The monthly zonal mean is calculated from the model in 10 • latitude bands, separately for land and ocean surfaces.An offset is added to the model values to force the global average surface concentration to equal the measured value from GLOBALVIEW-CO2 2 ; this offset is updated monthly to include the secular trend in CO 2 .
We impose smoothness contraints on the retrieved CO 2 profile via nonzero off-diagonal elements in the a priori covariance matrix to prevent unphysical wiggles in the posterior profile.A visual representation of this matrix is given in Fig. 2, which shows the correlation coefficient between various elements of the matrix, as well as the square root 2 Cooperative Atmospheric Data Integration Project -Carbon Dioxide, CDROM, NOAA ESRL, Boulder, Colorado, also available on Internet via anonymous FTP: ftp://ftp.cmdl.noaa.gov/ccg/co2/GLOBALVIEW/, 2011. of its diagonal elements.The diagonal values start large at the surface and decrease with altitude; they were scaled such that the total a priori uncertainty in X CO 2 is approximately 12 ppm.This structure reflects natural variability, which is largest near the surface (30-50 ppm), and decreases with elevation.Variability is smaller in the boundary layer (2-20 ppm), and smaller still (1-5 ppm) in the free troposphere  (Hansen and Travis, 1974);3 Non-spherical particles according to Baum et al. (2005a,b) where CO 2 is well-measured but still shows some variability due to vertical mixing.The variability is very small in the stratosphere (<1 ppm) where age of air models (e.g., Andrews et al., 2001) give accurate estimates of CO 2 concentration.The overall X CO 2 variability of 12 ppm, while probably larger than natural variability, was chosen to give extra weight to observations relative to the prior.The state vector contains several meteorological quantities: surface pressure, an additive offset to the prior temperature profile, and a multiplicative scale factor to the prior water vapor profile.The surface pressure is included in the state vector to partially correct for path-length modification effects and other systematic errors common to both the O 2 A-band and the CO 2 bands, similar to what is done in the TCCON retrieval (Wunch et al., 2011a).The prior surface pressure and profiles of temperature and water vapor are taken from 3-hourly European Centre for Medium-Range Weather Forecasting (ECMWF) model forecast fields, interpolated linearly in space and time to the GOSAT field-of-view (FOV).These are further interpolated to the same fixed pressure levels as for CO 2 .The surface pressure is adjusted to the elevation of the GOSAT FOV, where the elevation is taken from a 3 arc-second resolution digital elevation map (Zong, 2008).The a priori errors for temperature and water vapor are set high enough that these parameters are almost entirely determined by the measurements.For surface pressure, the typical ECMWF accuracy is 2-3 hPa, though the error can be larger in high latitude and high topography regions (Salstein et al., 2008).Thus, the 1 σ prior error was set at 4 hPa for all soundings to give the retrieval sufficient freedom to find surface pressure values that may deviate more strongly from the prior meteorology.
In the ACOS retrieval, land surfaces are assumed to be purely Lambertian.A mean albedo and slope are retrieved for each of the three GOSAT bands.These parameters are given sufficiently large prior variances such that they are essentially unconstrained.Prior values of the mean albedo are estimated directly from the level of the continuum in the observed spectrum of each band, assuming a clear-sky, absorption-free atmosphere.The prior slopes are set to zero.
To account for the altering of optical paths by atmospheric scattering, the retrieval solves for a mixture of profiles of four fixed-type atmospheric scatterers.These four types are chosen to cover a wide range of optical properties, such that by combining them appropriately, the retrieval can reproduce virtually any profile of scattering in all three spectral bands.The four scatterers chosen are water cloud, ice cloud, and two different types of aerosol.The aerosols are actually aerosol mixture types "2b" and "3b" from the aerosol climatology of Kahn et al. (2001).Type "2b" is a mixture of course and fine-mode dust, while type "3b" is a carbonaceous mixture; both mixture types contain some sulfate and sea salt.The water cloud is a Gamma distribution (Hansen and Travis, 1974) of spherical drops with an effective radius of 8 µm, while the ice cloud optical properties are taken from the model of Baum et al. (2005a,b), assuming an effective particle radius of 70 µm.The frequency variation of scattering properties is fixed for each of the scatterers.The extinction efficiency and single scattering albedo of each type in the three spectral bands are shown in Table 3.
To avoid problems of negative optical depths, which are unphysical, we fit for profiles of logarithmic extinction for each of the four scattering types, on the same predefined pressure levels as for carbon dioxide itself.The a priori profiles of cloud and aerosol optical depth are shown in Fig. 3.The two aerosol types have the same prior profile, shown as "Aerosol".All four types have equal a priori optical depths, such that the optical depth at 0.755 µm is 0.15 for all types combined.The a priori covariance matrices are diagonal, with diagonal elements assigned such that each aerosol concentration has a 1 σ uncertainty of a factor of 10.This has the implication that values that start at zero will stay at zero 3 .Thus, the retrieval is incapable of putting aerosols at pressures lower than about 300 hPa, and cannot put scatterers of any type at pressures lower than 100 hPa, both of which are physically sensible restrictions.
Three final parameters are included in the state vector that can correct errors in the instrument's spectral grid.For GOSAT, the spectral samples are equally spaced every 0.2 cm −1 across each spectral band, yielding a near-linear dispersion.However, the wavenumber scale varies from sounding to sounding due to the Doppler shift between the instrument and the FOV on the rotating earth, and also because of varying instrumental and environmental effects.Therefore, we retrieve a spectral dispersion offset for each of the three GOSAT bands.We estimate the a priori value directly from the observed O 2 A-band spectrum, using a simple method based on solar line positions as described in Taylor et al. (2012).

Forward model
The forward model takes as input the state vector described above, as well as other parameters that describe the viewing geometry and instrument details, and simulates spectra in the three NIR bands as would be produced by GOSAT or OCO-2.It also produces radiance Jacobians, which are derivatives of the radiances with respect to all the state vector parameters.These derivatives are necessary for the inverse model (Sect.2.3).The forward model was originally described in B06 and consists of a solar model, atmospheric model, surface model, radiative transfer model, and instrument model.We now describe each of these, primarily focusing on the differences from B06.

Solar model
The solar model closely follows that described by B06 and Bösch et al. (2011).It consists of a high-resolution, empirical line list for the full solar disk of over 20 000 lines that includes line frequency, line strength, and Doppler and 1/efolding width.The line list is based on a series of balloon and telescope observations, and the disk-centered version has been used extensively in the analysis of ground-based FTS spectra (Wunch et al., 2011a, and references therein).The solar continuum model is a ninth-order polynomial fit to the near-infrared part of the solar spectrum measured by the SOLSPEC instrument (Thuillier et al., 2003).Other than the variation in the sun-earth distance that modulates the overall intensity, the solar continuum is assumed to be timeinvariant; sunspots and other solar activity are currently ignored.The solar continuum is multiplied by the solar absorption spectrum to obtain the solar spectrum at high resolution; a doppler shift is also applied to transform the spectrum to a frame of reference located at a given sounding FOV.

Atmospheric model
The atmospheric model takes physical quantities such as profiles of gas concentration, clouds and aerosols, and produces profiles of optical properties to be fed into a radiative transfer module.These optical properties are computed on a highresolution spectral grid with a uniform spacing of 0.01 cm −1 .Profiles of CO 2 , H 2 O, and O 2 volume mixing ratio, specified on the twenty prescribed pressure levels of the state vector, are converted to absorption optical depth using a three-dimensional lookup table in pressure, temperature, and wavelength.
The table of gas absorption coefficients represents current state-of-the-art spectroscopic reference data for CO 2 and O 2 , including non-Voigt line shapes, speed dependence, line mixing, and collision-induced absorption (see Crisp et al., 2010, and references therein).HITRAN-2008(Rothman et al., 2009) is used for all other absorbers in the ACOS spectral ranges.Because the absorption cross-sections are nonlinear in both temperature and pressure, each atmospheric layer is subdivided into ten sublayers; cross-sections for each are calculated for the interpolated pressure and temperature at the center of each sublayer, converted to optical depth and then summed to obtain the optical depth for each (thick) atmospheric layer.This approach avoids biases due to using a coarse vertical grid.For simplicity, absorbing gas concentrations at the sublayer centers, such as for CO 2 and O 2 , are assumed to vary linearly with pressure.
The atmospheric model also includes scattering processes.Rayleigh scattering is parameterized according to the model of Bodhaine et al. (1999), in which the Rayleigh optical depth is a simple function of wavelength and atmospheric number density.Profiles of cloud and aerosol extinction per unit pressure are converted to optical depth at each high-resolution frequency of interest.The optical properties of each species are combined, and further combined with those from Rayleigh scattering and gas absorption to obtain composite profiles of optical depth, single scattering albedo, and fully-polarized scattering phase matrix.The phase matrices follow the notation of de Rooij and van der Stap (1984) and include six independent components, appropriate for spherically symmetric particles or randomly-oriented asymmetric particles (Hovenier and van der Mee, 1983).Other absorption, emission and scattering processes such as thermal emission, air glow, Raman scattering, and absorption by ozone in the Chappuis bands are generally negligible in the microwindows of interest and are currently omitted from the forward model.Fluorescence from chlorophyll in plants is ignored in ACOS B2.8, although there is evidence that this process can contribute significantly to the top-of-atmosphere (TOA) band 1 radiance (Frankenberg et al., 2011;Joiner et al., 2011), and will be incorporated in future versions of the algorithm.

Surface model
In the absence of significant atmospheric scattering, the majority of the TOA radiance comes from solar photons that either have been reflected directly by the surface or have been scattered exactly once in the atmosphere.When such single scattering prevails, the surface can be perfectly represented as a simple Lambertian albedo because only a single incoming and outgoing angle at the surface need be accounted for.However, if multiple scattering contributes significantly to the TOA radiance, there can be a nontrivial angular distribution of surface-scattered photons contributing to the TOA radiance.Because we reject soundings with significant scattering, our retrieval assumes a purely Lambertian surface, with an albedo that varies linearly with wavelength across each of the three near-infrared bands.The effect of surface polarization is ignored as we only retrieve on (P + S)/2 for GOSAT, which, as stated previously, is nearly identical to the total unpolarized intensity.In the retrieval tests described in Sect.3, we will test this assumption by using simulations that feature much more general representations of the surface reflectance.

Radiative transfer model
Once the atmospheric optical properties and surface reflectance properties have been determined, they are used to calculate the top-of-atmosphere Stokes parameters I , Q, and U on the high resolution 0.01 cm −1 wavelength grid.The Stokes parameter V , representing circularly polarized radiation, is ignored as it is generally negligible and most instruments, including GOSAT and OCO-2, are insensitive to it.The solar spectrum multiplies the high-resolution Stokes vectors calculated by the radiative transfer model, which are initially dimensionless reflectances, to give them the proper radiance units.
Because most soundings include some atmospheric scattering, a fully-polarimetric vector calculation of radiative transfer would be desirable to calculate the Stokes vector at each monochromatic wavelength.However, at 0.01 cm −1 resolution, this would lead to tens of thousands of computationally expensive radiative transfer calculations per forward model run.We therefore adopt an approximate approach called "Low Streams Interpolation" (LSI), which is described fully in O'Dell (2010) and references therein.Rather than performing full-accuracy calculations with a large number of angular streams at all monochromatic wavelengths, such calculations are only performed at a few tens of wavelengths.Very fast, low accuracy calculations are performed at all the monochromatic wavelengths; these are combined with the small number of high accuracy calculations to provide an estimate of the Stokes vector at each monochromatic point.
Monochromatic radiative transfer calculations are made using a combination of a fast single-scattering model (Nakajima and Tanaka, 1988), the LIDORT scalar multiplescattering model (Spurr et al., 2001), and a second-order-ofscattering polarization model called 2OS (Natraj and Spurr, 2007).Neglecting higher orders of scattering for Q and U is shown to lead to radiance errors on the order of 20 % or less of the expected OCO instrument noise, and X CO 2 errors typically on the order of a few tenths of a ppm or less (Natraj et al., 2008).The LSI method has radiance errors typically less than a tenth of a percent (O'Dell, 2010); errors of a similar type and magnitude were shown by Hasekamp and Butz (2008) to be somewhat less than OCO instrument noise.

Instrument model
The instrument model consists of two components: a model that computes the measured radiance for each instrument channel, and a noise model.The instrument radiance model operates on the high spectral-resolution Stokes vectors as follows: where I i is the measured radiance in the i-th channel, SRF i represents the spectral response function of the i-th channel, and λ represents wavelength in the frame of reference of the spacecraft.This is computed via a simple, non-relativistic Doppler shift, taking into account the rotation of the earth at the location of the target FOV as well as the spacecraft velocity itself.The m S coefficients are elements of the instrument Mueller matrix, and represent the polarization sensitivity of a given channel to a given Stokes component S. For GOSAT, these coefficients have been calculated explicitly, but because retrievals are done on (P + S)/2, we have m I ∼ 1 and all other components essentially zero (O'Brien et al., 2011).For OCO-2, which measures incoming light in a single polarization, this will not be true.The SRF of the TANSO-FTS bands have been provided by the Japan www.atmos-meas-tech.net/5/99/2012/Aerospace Exploration Agency (JAXA); the integration over radiance is performed over a 100 cm −1 range centered on each channel.
In the retrieval tests presented in the next section, an instrument noise model is required for two different purposes.First, for those simulations where artificial noise was added, the noise model is used to generate the properties of that artificial noise.Second, the noise model is used in all tests to construct the observation error covariance matrix S , which in principle should contain contributions from both instrument noise and forward model error.In the ACOS retrieval, S is taken to be diagonal and contains only instrument noise contributions for simplicity.In order to have highly representative GOSAT retrieval tests, we characterized the actual noise from the TANSO-FTS instrument by analyzing over a year's worth of high-gain spectra, separated by band and polarization.It was found that the TANSO-FTS instrument noise is broadly consistent with the following parameterization: where I b,p is the continuum signal level in band b and polarization p ∈ {P , S}, and A b,p and B b,p are parameters.N b,p is the corresponding noise level and is roughly constant across each band.Because we retrieve on the average of the P and S polarization channels, which as stated above is almost identical to the total intensity, it is straightforward to calculate the noise N b on the total intensity I b for each channel.This reduces to where the values of the A, B, and C coefficients are given in Table 4 for each GOSAT NIR band.These values are used to populate S in the simulation-based tests described below.However, note that for retrievals on real GOSAT soundings, the observation error in S for the ACOS retrieval is empirically determined such that the reduced χ 2 of the residuals is approximately unity in each band.This "empirical noise" approach is unique to the ACOS retrieval of real GOSAT data and is described fully in Crisp et al. (2012).Because the spectroscopy and solar models are consistent between the simulation and retrieval forward models for the idealized tests described herein, the full empirical noise approach is not required.

Inverse model & X CO 2 calculation
The inverse model used in the ACOS retrieval was previously described in C08.In brief, the modified Levenberg-Marquardt method of Fletcher (1971) is used to minimize the cost function given in Eq. ( 3).The updated state vector is calculated for each iteration as where F(x i ) is the forward model at x i , K i = ∂F(x i ) ∂x i is the corresponding Jacobian matrix, and γ is the Levenberg-Marquardt parameter.This equation is iterated until a convergence criterion has been satisfied.Details on the setting of γ and the convergence criterion are given in Crisp et al. (2010).For those retrievals that converge, several additional quantities are calculated, including X CO 2 , its a posteriori error, and its vertical averaging kernel.Because we retrieve CO 2 concentrations on discrete levels, the calculation of X CO 2 given in Eq. ( 1) is recast as where û is the retrieved profile of CO 2 concentration on fixed pressure levels and h is the pressure weighting function.The construction of h, which differs slightly from that presented in C08, is given in Appendix A, while the X CO 2 a posteriori error is derived in Appendix B. The construction of the column averaging kernel of X CO 2 is straightforward and was given in C08.

Pre-screening
Because the ACOS retrieval algorithm is computationally intensive, it is important to remove soundings that are unlikely to yield useful X CO 2 retrievals from the processing stream.Therefore, we employ a pre-screening step that consists of several filters.First, data with a solar zenith angle greater than 85 • are not processed, due to both low signal and known forward model deficiencies in these cases.For actual GOSAT observations, soundings with bad radiances as flagged in the JAXA Level 1 data are also filtered out.
Most importantly, we filter out cases with thick clouds and aerosols for which our retrieval is unlikely to yield reliable X CO 2 retrievals.This is done with the ACOS cloud screening algorithm, which uses a fast, O 2 A-band only retrieval.This simple retrieval assumes no clouds or aerosols are present; only Rayleigh scattering is included.Using a fast forward model, we fit the band 1 spectrum to the clear-sky model with five free parameters: surface pressure, an offset to the meteorological temperature profile, a spectral dispersion offset, and the surface albedo at the two band endpoints.We then define two quantities upon which to filter: P s,cld is the retrieved minus a priori surface pressure4 , and χ 2 R is the ratio of the fit chi-squared, relative to the minimum χ 2 value possible at that same signal-to-noise ratio (SNR).Scenes with | P s,cld | > 40 hPa or χ 2 R >2.3 are flagged as cloudy.The surface pressure test is two-sided as scatterers in the atmosphere occasionally lengthen (rather than shorten) the optical path.These empirically-determined thresholds are set to be relatively loose, so as to allow most clear scenes to pass while at the same time catching obviously cloudy cases.
A more complete description and validation of the cloud screening algorithm have been given in Taylor et al. (2012), which compared its performance on real GOSAT data against the standard Moderate Resolution Imaging Spectroradiometer (MODIS) cloud mask.For the over-land validation data set employed, which contained approximately 3000 soundings, the GOSAT cloud-screening algorithm was found to agree with MODIS about 78 % of the time.Due to the loose thresholds given above, it was found to be "cloudyconservative"; it falsely identified only 5 % of scenes as cloudy that MODIS identified as clear.Conversely, it passed as clear 31 % of actually cloudy scenes.Some disagreement could have been caused by pointing errors and other instrument-related problems, and hence such a comparison may give an overly pessimistic view of the algorithm.Section 3.2 examines the fidelity of the cloud screening algorithm on synthetic data, for which the true cloud and aerosol conditions are known.

ACOS CO 2 retrieval algorithm test & validation
We now describe tests of the ACOS retrieval algorithm with realistic simulations of GOSAT spectra.These tests will determine the algorithm's performance in the presence of clouds, aerosols, imperfect meteorological data, partially polarized, non-Lambertian surfaces, and known instrument noise.Given that we have exact knowledge of the true atmosphere and surface in the simulations, we are testing the impact of the general assumptions made in the retrieval concerning the above variables.While these simulations represent our best attempt at "real-world" conditions, the results will nonetheless be optimistic; real retrievals can be expected to be less accurate and precise due to physical effects not included in these tests, such as uncertainties in instrument calibration and gas absorption properties.

Generation of synthetic GOSAT data
Several thousand over-land soundings were simulated for an instrument with the characteristics of TANSO-FTS aboard GOSAT, but in purely nadir viewing geometry and in a sunsynchronous orbit matching that of CloudSat.Ten orbits were simulated for 14-16 September 2006.Figure 4 shows a map of the band 2 signal level for the simulated orbits.To avoid circular results, the simulator forward model is more general and accurate than that used in the retrieval.It has higher vertical resolution, includes more types of clouds and aerosols, more complicated surface optical properties, and has more accurate radiative transfer.However, the simulations assume the same spectroscopy and instrument properties as in the ACOS retrieval, including the Mueller matrices and the instrument line shape functions of the TANSO-FTS.
A comprehensive description of the methods used to generate the synthetic spectra by the "OCO Simulator" is given in O' Brien et al. (2009), and is only summarized here.The simulated atmospheres use meteorological information (temperature, humidity, and surface pressure) from an ECMWF model forecast at high vertical resolution, interpolated spatially and temporally to each specific sounding.Profiles of CO 2 are taken from the Parameterized Chemical Transport Model (PCTM) (Kawa et al., 2004).The surfaces in the simulator are not assumed to be Lambertian; non-Lambertian surfaces are synthesized based on the 1-km resolution MODerate Resolution Imaging Spectroradiometer (MODIS) Bidirectional Reflectance Distribution Function (BRDF) product MCD43B1 (Schaaf et al., 2002).The simulated distributions of surface reflectance of Stokes I at the primary scattering angle at the center of each of the three spectral bands is shown in Fig. 5a.Polarization is also included in the surface representation; surface polarization properties are taken from a simple model derived from POLarization and Directionality of the Earth's Reflectances (POLDER) data and based on International Geosphere/Biosphere Programme (IGBP) surface type (F.M. Breon, personal communication, 2008) simulated surface polarization fraction ranged from negligible to tens of percent, with a median polarization of about two percent.
Profiles of cloud and aerosol come from the Cloud-Aerosol Lidar with Orthogonal Polarisation (CALIOP) Level 2 cloud and aerosol layer products, version 3 (Vaughan et al., 2004;Kim et al., 2008).For each sounding, one actual CALIOP profile is pulled at random from within the 2 × 2 • grid-box in which the sounding lies.These profiles are kept on the original, high-resolution (∼2 km) vertical grid provided in the CALIOP product; the meteorological and CO 2 profiles are interpolated to this same grid.For cloud water and aerosol, the CALIPSO particles are taken to be spherical, with log-normal size distributions and optical properties calculated using Mie theory.CALIPSO ice particles are matched to ice particle distributions with accompanying optical properties as described in Baum et al. (2005a,b).For each layer with aerosol identified, the CALIPSO product identifies as essentially one of six types (clean marine, dust, polluted continental, clean continental, polluted marine, or smoke).Though each model layer contains at most one cloud or aerosol type, several types of cloud and aerosol often coexist in the same profile.The distributions of optical depth at 0.76 µm for aerosols, water clouds, and ice clouds are shown in Fig. 5b.Note that the distribution of total optical depth (not shown) cuts off at an optical depth of about ten because the CALIOP lidar cannot penetrate further than this.
Simulations were performed in two ways: first, by artificially removing the clouds and aerosols in order to simulate cloud-free soundings ("clear-sky"), and then by using the default atmospheric profiles ("all-sky").Retrievals on the clearsky simulations should be nearly perfect, and are used to test the existence of low-level systematic errors.Retrievals on the all-sky simulations illustrate how the existence of thin clouds and aerosols affect the retrieval accuracy.Finally, for some tests, instrument noise consistent with the simple GOSAT noise model given in Sect.2.2.5 was added to the synthetic spectra.
In the retrieval tests described below, errors in the retrievals of X CO 2 will be partially caused by differences between the forward models of the retrieval and simulation.The most important differences are in the surface treatment (unpolarized Lambertian vs. partially polarized full BRDF), and cloud plus aerosol treatment (4 fixed types vs. dozens of cloud types and 6 aerosol types).There are additional minor differences as well, in terms of the number of atmospheric levels (20 vs. 100+), radiative transfer (in which the simulations are more accurate), and in the treatment of Rayleigh scattering.For some specific tests, there was also the presence of instrument noise in the simulations, and differences in the prior meteorological fields.

Cloud-screening performance
In most previous studies of X CO 2 retrieval accuracy, only atmospheres with less than a certain optical depth (typically around 0.3) were tested.This assumes perfect cloud and aerosol optical depth information, which is not generally available.Therefore, in this study we process the synthetic soundings through the operational ACOS cloud screening algorithm, to eliminate soundings with obvious cloud and aerosol contamination in the same way as is done in the operational retrievals.As will be shown below, the cloud screening is imperfect and leads to significant errors in X CO 2 .
The performance of the cloud screening algorithm for the all-sky synthetic data with instrument noise added is summarized in Fig. 6.Panel a shows the distribution of cloud plus aerosol optical depth (AOD) at 0.76 µm in the synthetic data set, as well as the fraction of scenes identified as clear.Previous studies have identified 0.3 as a reasonable AOD threshold below which to attempt X CO 2 retrievals (Crisp et al., 2004).Therefore, we define "clear" as AOD ≤ 0.3, and "cloudy" as AOD > 0.3.With this definition, 26 % of all scenes are clear, and about 87 % of all scenes are classified correctly.However, about one third of scenes that pass the cloud filter are The cloud screening performance is dramatically different for low cloud versus high cloud cases.Figure 6b (c) shows the histograms of AOD for high (low) cloud or aerosol cases, in which 95 % of the AOD resides in the top 40 % (bottom 30 %) of the atmosphere.The high cloud cases have been considered by many authors to be the most problematic (e.g., It will be shown below that both thin high clouds and thicker low clouds cause problems for the X CO 2 retrieval.This problem can be partially mitigated by simply tightening the surface pressure threshold to | P s,cld | < 10 hPa, as shown in the dashed lines in Fig. 6.This reduces the rate of false positives from one in three to about one in five.This tighter threshold will be imposed as a posterior requirement in Sect.3.3.3.

Synthetic retrieval tests
We now test the ACOS algorithm performance using 6522 land-only, synthetic GOSAT soundings.To evaluate the retrieval error, one must use the averaging kernel to construct the ideal CO 2 profile u ak that can be retrieved (Rodgers and Connor, 2003): where A is the full averaging kernel matrix, u true is the true CO 2 profile, u a is the a priori CO 2 profile, and I is the identity matrix.Then the error in X CO 2 is given by where û is the retrieved CO 2 profile.In all discussions that follow, the X CO 2 retrieval error has been evaluated in this manner.
Six types of retrieval experiments were performed, which utilized three different types of simulations as well as three different retrieval configurations.These are outlined in Table 5.Some tests used clear-sky simulations without (tests 1 and 2) or with (test 3) instrument noise added to the radiances, while others used all-sky simulations with instrument noise (tests 4-6).Some tests used a clear-sky retrieval (tests 1 and 4), while others used the standard retrieval with aerosols and clouds retrieved (tests 2, 3, 5, 6).Note that Rayleigh scattering is included in all clear-sky simulations and retrievals, and the S matrix as described in Sect.2.2.5 is the same in all tests.Finally, tests 1-5 set the a priori meteorology equal to the true ECMWF meteorology, but test 6 used an alternative source of meteorological information in order to include realistic errors in the surface pressure and profiles of temperature and water vapor.

Retrievals of clear-sky simulations
We begin by analyzing the ACOS retrieval algorithm performance on simulated soundings that purposely omit cloud and aerosol.This enables us to see biases that appear when we attempt to retrieve clouds and aerosols in scenes where there are none.In retrieval test 1, these clear-sky simulations were processed by the ACOS algorithm, but with clouds and aerosols completely removed from the state vector, consistent with the clear-sky simulations.This test yields extremely good results as expected; the mean error in X CO 2 is 0.18 ppm and the standard deviation is 0.25 ppm.More detail can be seen in the black points of Fig. 7, which shows the X CO 2 error plotted versus several variables.As seen in panel b, much of this error is driven by the strong anti-correlation between X CO 2 error and surface pressure error.The slope between these two variables is −0.4 ppm hPa −1 , which is to be expected on simple theoretical grounds: all other factors being equal, errors in X CO 2 are related to surface pressure errors as follows: Performing a post-hoc correction to remove this correlation reduces the X CO 2 bias to nearly zero, and the root-meansquared (RMS) error to 0.11 ppm.The source of the surface pressure errors must be related to differences in the forward models of the simulation and retrieval, possibly associated with the different surface treatments and coupled through Rayleigh scattering, though minor differences in the radiative transfer or Rayleigh scattering parameterization itself may also be responsible.The surface-pressure corrected results can be interpreted as the theoretical accuracy limit for perfectly clear soundings that have known surface pressure and high SNR.
The blue and red points in Fig. 7 denote tests 2 and 3, respectively, which both use the operational retrieval that includes clouds and aerosol parameters in the state vector.Test 2 is run on noiseless simulations, while test 3 is run on simulations with GOSAT-like instrument noise first added to the simulated spectra.Panel a shows that a large positive bias occurs at very low values of the retrieved band 3 albedo.The same effect can also be seen versus the band 2 albedo as well as the band 2 or band 3 SNR, but is strongest with band 3 albedo.These low band 3 albedos tend to occur in regions of needleleaf evergreen forests, mixed forests containing some evergreens, and regions of permanent snow and ice cover.This bias is likely due to the retrieval exchanging surface albedo for very thin cloud or aerosol, with minimal adverse effect on the cost function.This hypothesis is supported by the fact that the mean retrieved AOD is about 0.03 rather than zero, which demonstrates that the retrieval (which Atmos.Meas.Tech., 5, 99-121, 2012 www.atmos-meas-tech.net/5/99/2012/starts with an a priori AOD of 0.15) cannot fully reach zero in general.This may be related to the logarithmic implementation employed for aerosols in the retrieval.We therefore exclude soundings with a retrieved band 3 albedo less than 0.05 in all further tests.Figure 7b shows that the relationship between X CO 2 error and P is present but less well-defined in the standard retrieval, with a reduced slope of about −0.2 ppm hPa −1 in both the noiseless (blue) and noisy (red) cases.The main effect of instrument noise is simply to add random errors of ∼0.3 to 2.5 ppm (1-σ ) depending on the scene, with a median value of roughly 0.6 ppm. Figure 7c reveals that there is another fundamental bias in the operational retrieval, related to the ratio of the signal in the weak CO 2 band relative to the signal in the O 2 A band, which we call R 21 .The distribution of R 21 is bimodal; the cluster of points around 0.2 is due to snow and ice covered surfaces.These surfaces exhibit high X CO 2 biases of ∼3 ppm on average.The general cause of this bias, like that in panel a, seems to be the trading of surface albedo for very thin cloud or aerosol, in that the bias is absent in test 1 which excludes clouds and aerosols from the state vector.However, the details of the mechanism are not yet understood and warrant further research.
Note that the overall errors are still quite low; the RMS of all soundings with a band 3 albedo >0.05 is 0.82 ppm for the noise-free simulations, and 1.04 ppm for the noisy simulations.Both are bias-free overall, having mean absolute errors less than 0.03 ppm.Subtracting a linear fit to both P and logR 21 yields RMS X CO 2 errors of only 0.28 ppm for test 2 and 0.65 ppm for test 3.After fitting out correlations with these two variables, we could find no other variable that significantly explained any of the remaining variance; the majority of the error variance was explained by these two variables and no others.

Retrievals of all-sky simulations
The previous section analyzed the algorithm performance for perfectly clear scenes.Here, we explore the algorithm's behavior for scenes with realistic types, amounts, and vertical distributions of cloud and aerosol particles.For these tests, we will also see how imperfections in the cloud screening algorithm increase errors in retrieved X CO 2 .
We begin by describing results for both the clear-sky retrieval (test 4), in which clouds and aerosols are not included in the state vectors, as well as the standard ACOS retrieval (test 5).As before, we ran 10 orbits with a total of 6522 soundings over land.In the standard retrieval, 97 % of the 1959 retrievals passing the cloud screen converged, yielding estimates of X CO 2 ; conversely, only 11 % of those flagged as cloudy converged.Thus, convergence in the standard X CO 2 retrieval is also a good indicator of cloudiness, and hence the (fast) cloud filter can also be seen as an efficient way of screening out soundings that would have failed to converge due to excessive cloud or aerosol contamination.Figure 8 shows the error in X CO 2 vs. the true AOD at 0.76 µm, for retrievals both passing (black) and failing (red) the cloud filter.The latter are shown just to illustrate their worse error statistics, even for soundings with the same AOD.Based on the results of the clear-sky retrievals, the few soundings with retrieved band 3 albedo less than 0.05 have first been removed.For points failing the cloud screen, the X CO 2 errors are large for both the clear-sky retrieval (Fig. 8a and standard retrieval (Fig. 8b).However, for points passing the cloud screen, especially those with a true AOD less than 0.3, the results are markedly better in the standard retrieval.This is in accord with previous studies, most notably that of Butz et al. (2009).In fact, for these points, the RMS error is 1.3 ppm for the standard retrieval versus 3.0 ppm for the clear-sky retrieval.
These results indicate that the standard retrieval partially corrects for the presence of clouds and aerosols, though this ability rapidly degrades with increasing AOD.This is illustrated in Fig. 8c, which shows the retrieved vs. true AOD in test 5.For AOD 0.5, there is some correlation between these two variables (r = 0.7), but above this value there is no correlation.As in tests 2 and 3, the lowest AOD the retrieval obtains for very clear cases (AOD < 0.02) is about 0.04 rather than zero.Importantly, these very clear cases exhibit no overall bias, also consistent with the clear-sky tests.This implies that any overall bias in the retrievals is related to uncorrected cloud and aerosol effects.
Considering the distinctly poorer results of the clear-sky retrieval, we now focus purely on the standard retrieval.Including soundings that fail the cloud filter, Fig. 8b shows that there are two main clusters of points that have unacceptably large errors.Soundings with true AOD 1 have positive and negative X CO 2 errors in roughly equal numbers.Many of these cases with positive errors are found to be high, thin cloud scenes.Fortunately, the majority of these are successfully screened out by the cloud filter, though some remain and may ultimately bias the retrievals high.Conversely, soundings with a negative bias in retrieved X CO 2 tend to include thicker, low clouds.These are most often water clouds, many with optical depths significantly above unity.Importantly, a substantial number of these pass the cloud screening as discussed in Sect.2.4.
The dominant atmospheric scattering mechanisms driving these biases are not yet evident.It is complicated by the fact that the algorithm simultaneously retrieves CO 2 and dry air column (via the surface pressure), so reasoning involving path-shortening vs. path lengthening effects must be broken down in terms of the different NIR bands.Biases can only occur when path shortening or lengthening effects are different between the CO 2 bands versus the O 2 A band.Future Monte-Carlo simulations may shed some light on the dominant mechanisms at work.

Post-retrieval filtering
As shown above, the errors in retrieved X CO 2 are dominated by the presence of cloud-and aerosol-laden scenes which nonetheless pass the cloud-screening algorithm.Indeed, the RMS error for test 5 retrievals is ∼4 ppm, up from ∼1 ppm for completely clear soundings (test 3).These retrievals also have an overall bias of ∼ −1 ppm, driven primarily by unscreened low clouds.The question thus arises: are there other metrics, to be used as quality control filters, that can be used to identify and remove strongly biased retrievals?Common metrics are χ 2 values of the spectral residuals, SNR, retrieved AOD, and many others.Below we identify a set of candidate filters.These filters are similar but not identical to those used for retrievals of real GOSAT observations as reported in Wunch et al. (2011b) and Crisp et al. (2012).Screening variables were selected based upon three features of candidate variables.(1) A known, clear-sky bias associated with a variable, as deduced from tests 1-3; (2) presence of an X CO 2 bias as a function of that variable; (3) scatter in X CO 2 error that strongly depends on that variable.Scatter, calculated as the standard deviation of X CO 2 error in bins of the target variable, was evaluated relative to the estimated a posteriori error so that retrievals with greater instrument noise were not automatically eliminated.
Figure 9 shows the effect of the candidate filters applied to the 1898 test 5 soundings that passed the cloud filter and for which the X CO 2 retrieval converged.The first filter (panel a) eliminates soundings with low band 3 albedo to minimize a high X CO 2 bias that occurs even for perfectly clear scenes (see Sect. 3.3.1;Fig 7a).This filter effectively removes low SNR scenes, though it is found that this filter is slightly more effective at reducing X CO 2 errors than SNR itself.
Next, we tighten the cloud-screening filter.It was shown in Sect.3.2 that tightening the cloud-screening surface pressure requirement from | P s,cld | < 40 hPa to | P s,cld | < 10 hPa may be beneficial.As show in Fig. 9b, this filter predominantly removes cloud-affected scenes and reduces the RMS X CO 2 error from 4.2 to 2.5 ppm.It is worth noting that filtering on retrieved surface pressure from the full ACOS algorithm is also effective at removing bad retrievals (Wunch et al., 2011b), but is actually slightly less sensitive than using the surface pressure retrieved by the cloud-screening algorithm.
It was found that filtering on retrieved AOD < 0.15 was effective in removing many cloud-contaminated soundings (Fig. 9c).This is sensible, as Fig. 8c demonstrated that retrieved AOD is somewhat correlated with true AOD for AOD 0.5.Implementing this requirement further reduced the RMS error to ∼1.9 ppm.
Figure 9d shows the X CO 2 error vs. the reduced χ 2 of band 2, which measures the spectral goodness of fit relative to instrument noise.Fit residuals due to instrument noise alone would display a reduced χ 2 with a mean of unity and a standard deviation of √ 2/N dof , where N dof is the number of degrees of freedom in the fit.For these retrievals, N dof is roughly equal to the number of spectral channels in each band.As this number is relatively high, the reduced χ 2 will be distributed normally.For band 2 with 601 channels, it is expected to have a 1 σ width of ∼0.06.We use a threshold of 4 σ ; this should retain greater than 99.99 % of the retrievals that fit the spectra to within instrument noise.Surprisingly, filtering on χ 2 removes only two soundings; the same is true for the other bands.In other words, applying the first three The values in the bottom-right of each panel give the percentage of soundings passing the filters applied cumulatively through that filter, as well as the mean and RMS X CO 2 error (in ppm) of those soundings.Soundings in blue (red) have a true column AOD of less than (greater than) 0.3.We stress that this display applies the filters cumulatively, though the final results are independent of order.Panel (f) shows the X CO 2 error vs. true AOD for the filtered retrievals.
filters yields a set of retrievals that have spectral fit residuals consistent with instrument noise alone.
The X CO 2 errors of retrievals passing the first three filters have an RMS value of 1.9 ppm, much better than the unfiltered value of 4.3 ppm but still highly contaminated by undetected low clouds and aerosols as indicated by the significant number of red points in Fig. 9d.As Fig. 9e shows, many of these cases occur at high values of the relative air mass5 : where µ is the cosine of the observation zenith angle and µ 0 is the cosine of solar zenith angle (SZA).For these nadir soundings, µ = 1.The simulations include low clouds at all SZAs, but the cloud filter tends to miss them at high SZAs.Therefore, one way to mitigate their effect is to screen out all soundings above a certain air mass.An air mass threshold of 2.7 (SZA ∼ 54 • ) removes most of the worst cases, at the expense of also removing some good (clear) soundings as well.This solution is unattractive in that it precludes any data at latitudes higher than 30 • in the winter hemisphere and 77 • in the summer hemisphere, and is not currently used on real data by ACOS.For independent data users, this filter should only be used until a more effective way of removing these low cloud cases is found.
Including the somewhat undesirable air mass cut, the results after filtering are extremely promising.The 63 % of retrievals that pass the filters have an RMS X CO 2 error of ∼1.04 ppm, comparable to that for completely clear scenes.The median absolute error is roughly 0.7 ppm, and 90% of the retrievals have absolute errors less than 2 ppm. Figure 9f plots the X CO 2 error vs. true AOD, similar to Fig. 8b but now including the filters.
Surprisingly, greater than 10 % of these soundings have true AOD > 0.3, but have similar error statistics as those with AOD < 0.3.This is true even for those with larger optical depths of 2-4.This suggests that the post-retrieval filters let through some soundings for which we have retrieved a lucky combination of aerosol and surface parameters that allows for a good retrieval of X CO 2 and fits the spectra to within instrument noise.The clear-sky retrieval performs terribly for these same soundings (RMS = 8.5 ppm), suggesting that it is not merely a cancellation of path-shortening and lengthening effects.The physical mechanisms at play here are unclear and warrant further research.
Finally, these filtered retrievals have an overall bias of ∼0.3 ppm.Analysis shows that this high bias is driven by thin, high cirrus clouds that pass the cloud filter and all the post filters, and have optical depths ranging from approximately 0.02 to 0.2.Indeed all soundings in Fig. 9f with an X CO 2 error greater than +3 ppm are high ice clouds.Further development is required to either better retrieve X CO 2 in these cases, or screen them out.

Errors due to imperfect meteorology
All tests thus far have been performed with perfect meteorology for the a priori values of surface pressure, temperature offset, and water vapor scale factor, as well as shapes of the temperature (T ) and water vapor (q) profiles.The first three are retrieved by the ACOS algorithm, and of these, only surface pressure is substantially constrained by the prior.However, the relative shapes of the T and q profiles are assumed by the retrieval to be correct.If they are not, as will certainly be the case to some extent for actual retrievals, errors may result.
To simulate realistic errors in these meteorological quantities, test 6 retrievals used meteorological prior data taken from the US National Centers for Environmental Prediction (NCEP) 1 reanalysis (Kalnay et al., 1996), interpolated to the sounding times and locations.Because the simulations used short-term forecasts from ECMWF that have both a higher spatial and temporal resolution than the NCEP reanalysis, the differences between these two approaches will be higher than if both operational forecasts or analyses had been used.However, some degree of correlation between NCEP and ECMWF is expected, as both assimilate similar data sets and often make similar model assumptions (Ponte and Dorandeu, 2003).For example, Salstein et al. (2008) found that RMS surface pressure differences between the NCEP and ECMWF analyses were about half that of the difference between either analysis and ground truth.
Figure 10 shows the differences in pressure, temperature at the surface and 700 hPa, and specific humidity at the surface and 700 hPa between the two data sets.The surface pressure differences are asymmetric with a tendency for NCEP to be lower than ECMWF; the median difference is −0.5 hPa, and the standard deviation is 4.4 hPa.However, these differences are largely driven by spurious vertical binning effects in the orbit simulator, and are not representative of the actual NCEP-ECMWF differences.The ∼4.4 hPa RMS difference is almost twice as large as the globally-averaged value found between ECWMF and observations (Salstein et al., 2008), and therefore represents a relatively difficult test for the retrieval.For both temperature and water vapor, the differences are larger at the surface than in the mid-troposphere, indicating different profile shapes; this is especially true for temperature.This is useful because it will directly test our implicit assumption that the a priori profiles of temperature and water vapor have the correct relative shapes.
Figure 11 shows the differences in retrieved surface pressure (panel a) and X CO 2 (panel b) due to using the imperfect NCEP meteorology as a prior (test 6) as compared to the case with perfect meteorology (test 5), for those soundings passing the filters of surface pressure; the RMS difference between the two is only 1.7 hPa, while for the prior it is almost three times larger.The mean X CO 2 difference is roughly zero and the RMS difference is 0.38 ppm, which is much smaller than errors due to the other effects (aerosol, instrument noise, etc.).Fully 70 % of the variance in this difference is due to differences in the retrieved surface pressure, which are partially driven by the unphysically large prior surface pressure differences mentioned earlier.Applying the test 5 post-processing filters to test 6 shows that the imperfect meteorology adds no additional bias and increases the random X CO 2 error by just 5 %.This modest difference justifies the simplifying assumptions made in the ACOS retrieval concerning the a priori meteorology.
Finally, we recognize that test 6 represents the most stringent test of the ACOS retrieval algorithm given in this work.The histogram of its X CO 2 errors is shown in Fig. 12a, for all retrievals (black), as well as the subset of retrievals that also passed the post-processing filters (blue).As for test 5, these retrievals have promising X CO 2 error statistics, with an overall mean of 0.3 ppm and an RMS error of 1.1 ppm. .The one-to-one line (dashed) would be expected if the a posteriori error fully explained the true errors; the best-fit line for the filtered retrievals (dotted) has a slope of 2.1.

Evaluation of the a posteriori error
As detailed in C08, the a posteriori estimate of X CO 2 error, denoted σ X CO 2 here, is generally a combination of instrument noise, smoothing error, interference errors with non-CO 2 state vector elements, and forward model errors.Both instrument noise and forward model errors should be included in the retrieval via the measurement error covariance matrix S .However, as stated previously S contains instrument noise only in the ACOS algorithm.Therefore, does the a posteriori error estimate σ X CO 2 bear any relationship to the actual error?This question is of critical importance to inversion modelers and other users who require accurate error estimates of the retrieved X CO 2 .
Figure 12b plots the RMS value of X CO 2 error vs. σ X CO 2 , again for both the unfiltered and filtered retrievals, where the RMS errors are calculated in bins of σ X CO 2 .The RMS error for all retrievals rises quickly as a function of σ X CO 2 , to error levels of 5-10 ppm.The retrievals passing the www.atmos-meas-tech.net/5/99/2012/Atmos.Meas.Tech., 5, 99-121, 2012 post-processing filters, however, have a roughly linear relationship with σ X CO 2 , with a best-fit slope of 2.1.This implies that forward model errors are responsible for about 75 % of the error variance.On the upside, it also suggests that the actual retrieval error can be modeled as a linear function of σ X CO 2 , but additional study is required to see if this relationship extends to real-world retrievals.

Conclusions and future directions
In this study we have described the details of the ACOS X CO 2 retrieval algorithm.We have used realistic simulations to evaluate systematic and random errors in retrieved X CO 2 , including the impacts of realistic cloud screening and postretrieval filters.The main conclusions regarding the ACOS retrieval algorithm can be summarized as follows.
1.The current ACOS pre-screening filter is useful but imperfect, in that some thin high cloud scenes and many thick low cloud scenes pass the filter, and these cases can cause large X CO 2 retrieval errors.
2. Approximations in the representation of cloud and aerosol properties lead to biases of 0.0-3.0ppm in retrieved X CO 2 even for perfectly clear scenes.These biases are worst over snow and ice-covered surfaces, and appear coupled with low albedo effects over these surfaces.
3. The non-aerosol related assumptions of the retrieval, such as the use of a small number of vertical layers (20), the assumption of perfectly Lambertian surfaces, imperfect a priori meteorology, and the simplified radiative transfer lead to minor X CO 2 errors of a few tenths of a ppm.
4. The use of a well-designed set of post-processing filters removes most of the poor retrievals including those contaminated by low clouds; the filtered retrievals have RMS X CO 2 errors of ∼1 ppm, relative to more than 4 ppm for the unfiltered retrievals.These retrievals have an overall bias of +0.3 ppm, largely driven by unfiltered thin cirrus.
5. The a posteriori X CO 2 error estimate in the ACOS algorithm, driven by instrument noise, interference and smoothing errors, underestimates the true X CO 2 error by more than a factor of two.The true error includes errors due to forward model assumptions which are not easily included in S .We speculate that cloud and aerosol assumptions may play a dominant role here.
These simulation-based tests, which include many realistic sources of retrieval error, are nonetheless incomplete.Spectroscopy errors, pointing errors, imperfect radiometric and spectral characterization of the instrument, and other effects are clearly present in retrievals using actual GOSAT observations.Additional real-world issues, such as forest canopy effects, partial cloudiness, cloud shadows, and plant fluorescence will further increase the retrieval errors.The results presented here therefore represent a lower limit on the ACOS X CO 2 retrieval algorithm errors; further validation efforts such as those of Wunch et al. (2011b) are required to fully assess the real-world retrieval accuracy.
Improvements in the ACOS X CO 2 retrieval algorithm are possible.It is clear that the pre-screening filter misses some clouds, including both thin cirrus and thicker, low water clouds, which lead to biased X CO 2 retrievals; the temporary solution of excluding all soundings above a certain air mass must be replaced by a method or filter that explicitly targets these cases irrespective of air mass.Also, the inclusion of clouds and aerosols in the retrieval state vector is seen to have a clear benefit in that, for scenes with very thin clouds and aerosols, the results are significantly better than if they were omitted from the state vector.However, including clouds and aerosols in the state vector also has a negative consequence: in clear or nearly-clearly scenes, they induce a rather strong (>1 ppm) bias that is primarily a function of the signal level ratio between bands 1 and 2. Refining the algorithm's aerosol treatment may help to minimize this type of bias.
Despite the issues described above, the accuracy of the current B2.8 ACOS X CO 2 retrieval, based on the synthetic tests described herein, is at or near the 1-2 ppm requirement suggested for the accurate monitoring of carbon fluxes on regional scales (Miller et al., 2007).Nevertheless, using the simulator we have identified several remaining biases in the B2.8 retrieved X CO 2 .Similar biases are observed in retrievals from the actual GOSAT spectra (Wunch et al., 2011b).Thus, the simulator provides an important vehicle for future algorithm development towards producing X CO 2 retrievals with an accuracy and precision approaching the theoretical limit as determined solely by the signal-to-noise ratio of the measured spectra.

Pressure weighting function construction
We now derive h, the pressure weighting function introduced in Eq. ( 8).Note that h was originally derived in C08, but this derivation ignored vertical variations in the gravitational acceleration as well as the presence of water vapor.Both terms slightly affect the local number density of dry air, and as such affect the calculation of X CO 2 .Let the pre-defined pressure levels, p = p 1..N , be ordered from space to surface, and truncated such that the last level, p N , is physically located below the surface, where the surface level is defined by the retrieved surface pressure p S .This is illustrated in Fig. A1. C. W. O'Dell et.al.: ACOS CO 2 retrieval algorithm not depend on x, we have k T = (h T , 0), for the CO 2 and non-CO 2 parts of the state vector, respectively, and the X CO 2 a posteriori error estimate simplifies to that given in C08.
Given that the error in x is characterized by the error covariance Ŝ, the error variance of X CO 2 , σ 2 X CO 2 , follows directly from linear error analysis: The only difference from C08 is that h replaces k in this equation.C08 break down this retrieval error into several components: measurement error (M), forward model error (F ), smoothing error (S), and interference error (I ), and assigns each component its own full error covariance matrix, ŜC , where C ∈ {M, F, S, I }.The definitions of each such matrix can be found in C08.To transform these error components from state vector space to X CO 2 space, we simply apply Eq. (B3) for each error component.The component X CO 2 error variances can then be summed to produce the total X CO 2 error variance.
instrument has Published by Copernicus Publications on behalf of the European Geosciences Union.100 C. W. O'Dell et.al.: ACOS CO 2 retrieval algorithm made near-infrared (NIR) based measurements of CO 2 since 2003

Fig. 1 .
Fig. 1.Flowchart showing the basic steps for the end-to-end ACOS Level-2 algorithm.

Fig. 2 .
Fig. 2. A priori CO 2 correlation matrix.The colors represent the error correlation between CO 2 concentrations at different pressures, arranged TOA to surface from top to bottom and right to left.The left-hand column gives square root of the diagonal elements of the covariance matrix.

Fig. 3 .
Fig. 3.A priori extinction profiles of cloud and aerosol at 0.755 µm wavelength.The two formal aerosol types each have the same a priori profile and are shown here as "Aerosol".The algorithm formally retrieves the logarithm of this quantity.

Fig. 4 .
Fig. 4. Distribution of soundings used in the simulation-based tests; the plotted variable is the band 2 signal level.There are ten orbits containing a total of 6522 soundings, spanning 14-16 September 2006.
Fig. 5. (a)Distributions of simulated reflectance at the center of each spectral band, calculated for the primary scattering angle (solar angle to observation angle).The spectral bands are defined in Table1.(b) Distributions of simulated aerosol, cloud water, and cloud ice optical depth at 0.76 µm.
Fig. 6.(a) Distribution of cloud plus aerosol optical depth (AOD) for the synthetic orbits (solid line) and fraction of scenes identified as clear by the cloud screening algorithm (dashed and dotted lines).The dashed line shows the result for the operational thresholds, while the dotted line shows the result when the | P s,cld | threshold is tightened to 10 hPa.(b) Same as panel (a), but only shows those cases where 95 % of the AOD resides in the upper 40 % of the atmosphere (high clouds).(c) Same as panel (a), but only shows those cases where 95 % of the AOD resides in the lowest 30 % of the atmosphere (low clouds).

Fig. 7 .
Fig. 7.X CO 2 error versus different quantities for retrievals performed on clear-sky simulations.Black points use noise-free simulations and a clear-sky version of the ACOS retrieval, with cloud and aerosol values fixed at zero (test 1).Blue points are for the operational retrieval on simulated spectra free of instrument noise (test 2).Red points are the same but with GOSAT-like noise added (test 3).P s is the retrieved minus true surface pressure, and R 21 is the ratio of the continuum signal level in band 2 relative to band 1.For tests 2 and 3 in panels (b) and (c), only soundings with retrieved band 3 albedo >0.05 are shown.
Fig. 8. (a) Retrieved X CO 2 error versus true AOD for test 4. Black circles denote converged retrievals which passed the cloud filter; red dots denote converged retrievals which failed the cloud filter.The corresponding mean and RMS X CO 2 errors in ppm are given in the lower left.The green, vertical dashed line denotes the original 0.3 AOD threshold for retrievals identified in Crisp et al. (2004).Note that a few soundings with AOD > 0.1 have errors larger than the plotted ordinate range.(b) Same as panel (b), but for test 5. (c) The retrieved AOD versus true AOD in test 5.The one-to-one line (dashed) is shown for reference.
Figure8shows the error in X CO 2 vs. the true AOD at 0.76 µm, for retrievals both passing (black) and failing (red) the cloud filter.The latter are shown just to illustrate their worse error statistics, even for soundings with the same AOD.Based on the results of the clear-sky retrievals, the few soundings with retrieved band 3 albedo less than 0.05 have first been removed.For points failing the cloud screen, the X CO 2 errors are large for both the clear-sky retrieval (Fig.8a)

Fig. 9 .
Fig. 9. Results from applying candidate quality-control filters to the X CO 2 retrievals from test 5. Panels (a) through (e) show the X CO 2 error plotted versus a candidate filtering variable.Dashed vertical lines with arrows show the filter threshold value for each variable, if any.The values in the bottom-right of each panel give the percentage of soundings passing the filters applied cumulatively through that filter, as well as the mean and RMS X CO 2 error (in ppm) of those soundings.Soundings in blue (red) have a true column AOD of less than (greater than) 0.3.We stress that this display applies the filters cumulatively, though the final results are independent of order.Panel (f) shows the X CO 2 error vs. true AOD for the filtered retrievals.

Fig. 10 .
Fig. 10.Differences between select NCEP and ECMWF meteorological variables; both have been spatially and temporally interpolated to the target locations and times with similar interpolation schemes.All differences are NCEP minus ECMWF.(a) Surface pressure difference.(b) Air temperature difference at the surface (black) and at 700 hPa (blue).(c) Specific humidity difference at the surface (black) and at 700 hPa (blue).

Fig. 11 .
Fig.11.Distribution of retrieved surface pressure differences (a) and X CO 2 differences (b) between tests 6 and 5, which are due to differences in the a priori meteorology.Only soundings passing the post-processing filters are included.
Fig. 12.(a) Histogram of test 6 X CO 2 errors, for both unfiltered retrievals (black, solid) and those passing the post-processing filters (blue, dashed).(b) Plot of test 6 actual RMS errors in X CO 2 vs. a posteriori error σ X CO 2, calculated by binning X CO 2 error in bins of σ X CO 2. The one-to-one line (dashed) would be expected if the a posteriori error fully explained the true errors; the best-fit line for the filtered retrievals (dotted) has a slope of 2.1.

Table 1 .
NIR spectral bands used in the ACOS retrieval.
1 Profile quantities contain 20 or fewer elements, depending on the surface pressure. 2e Fig.2and text for details. 3 Optical Depth at 0.755 µm. 4 Estimated directly from observed spectrum; see text for details.

Table 3 .
Optical properties of cloud & aerosol types in retrieval state vector.

Table 5 .
Simulations and retrieval configurations used in this work.