Starting with version B2.10, the 20-layer optical depth retrieval used for clouds and aerosols was replaced with a Gaussian-shaped vertical profile for each of the retrieved scattering particle types. As of version 8, two cloud types, two lower-atmosphere aerosol types, and one stratospheric aerosol are used. The new cloud and aerosol profile treatment is similar to that of but specifies the aerosol concentration ρaer as a function of x, the pressure relative to the surface pressure. Therefore, x ranges from zero at the top of the atmosphere to one at the surface. The functional form is simply ρaer(x)=Cexp⁡-(x-x0)22σa2, where for each aerosol type x0 is the vertical location at peak aerosol density and σa is the Gaussian 1σ profile width. Both of the latter variables are specified in units of relative pressure x. The prefactor C is defined such that the aerosol or cloud optical depth at 755 nm, hereafter OD755, equals the desired value. In the retrieval algorithm, the fitted quantities are ln⁡OD755 and peak height pr,0 for each aerosol type, with the exception of the stratospheric aerosol (described in Sect. ) for which only the optical depth is retrieved. Because it has been shown that GOSAT and OCO-2-like spectra have little sensitivity to the Gaussian profile width , this parameter is fixed in both the GOSAT and OCO-2 retrievals for all particle types. The prior profiles for each fitted type are shown in Fig. .

The change to a sigma-level pressure system was incorporated at about the same time as the shift to Gaussian aerosol profiles. Instead of fixed pressure levels, the pressure levels scale with the surface pressure: pi=aipsurf, where the ai are chosen such that the total number of pressure boundaries is 20, and the layers have roughly equal pressure widths. The top-most model level is set to 0.01 hPa.

The optical properties of the four scattering types remained unchanged from versions B2.9 to B3.4 and are described in . However, the use of two fixed aerosol types, types “2b” and “3b” from the climatology, did not accurately represent the true global variability of aerosol on the length scales and timescales probed by GOSAT and OCO-2. Beginning with build 3.5, the aerosol types were changed to be location- and time-dependent, with the prior type information coming from the aerosol climatology of the Modern-Era Retrospective analysis for Research and Applications (MERRA, ). The MERRA aerosol field is driven by the Georgia Tech/Goddard Global Ozone Chemistry Aerosol Radiation and Transport (GOCART) model , and modified by assimilating aerosol optical depth from the MODIS instruments onboard the Terra and Aqua satellites . MERRA contains five broad aerosol types: dust (DU), sea salt (SS), sulfate (SO), and black and organic carbon (BC and OC, respectively). Dust and sea salt are each tracked in five separate size bins. Organic carbon and black carbon are tracked in both hydrophobic and hydrophilic categories. In addition to the carbonaceous types, sulfate aerosol and sea salt are also hydrophilic and hence have optical properties that depend on the local relative humidity (RH).

For the aerosol prior in the ACOS retrieval, we primarily sought to specify the typical dominant aerosol types present (in terms of their contribution to the optical depth in the OCO-2 bands) in a given location at a given time of year. Monthly aerosol fields were derived from the MERRA model for the year 2010, and are used for all years in the ACOS retrieval. We aggregated the 15 MERRA types, eight of which have RH-dependent optical properties, into the five aggregated types listed above. We used typical density weightings and relative humidity values to create the optical properties for these aggregated types, as described in . Their optical properties, including extinction efficiency, single scattering albedo, and asymmetry parameter, are shown in Fig. . The organic carbon and sulfate aerosol are generally similar in their optical properties, though their single scattering albedos diverge somewhat in the CO2 bands. The sea salt, water cloud, and dust optical properties are relatively similar across the OCO-2 spectral range.

At each sounding location, the two aggregated aerosol types with the highest mean monthly values of the OD755 are selected to be retrieved by the L2FP algorithm. In previous algorithm versions, the total prior OD755 was set to 0.15, apportioned equally among four scattering types (water cloud, ice cloud, and two tropospheric aerosol types). However, it was found this was generally too high to allow a fit near OD755=0 for scenes that were almost entirely free of aerosol. This “clear-sky bias” was seen in early simulation tests . The prior OD755 is now set to 0.0125 for each cloud type, and set from the MERRA aerosol climatology for each tropospheric aerosol type as the average OD755 of that type (at a particular location and month). There is some evidence that the tropospheric aerosol priors are occasionally still too high; methods for specifying the aerosol prior are a continuing topic of investigation.

The cloud ice optical properties were updated in version 7. Before that, they were based on the band-averaged model developed by primarily for the MODIS instrument and known as the MODIS Collection 5 model. This cloud ice model considered an ensemble of size-dependent non-spherical ice crystal habits in random orientation. As ice crystal surface roughness was later shown to significantly affect scattering by ice crystals, and simulations with roughened model particles were more consistent with satellite observations of ice cloud polarized reflectances , we updated the cloud ice optical properties to correspond to the MODIS Collection 6 model, which describes scattering by severely roughened hexagonal column ice crystal aggregates . This update also fixed several minor issues in the previous cloud ice model, such as those resulting from linear interpolation of the optical properties from MODIS wavelength bands to OCO-2, and those relating to truncation of the phase function.

There have been substantial changes between the molecular cross sections used in the earliest ACOS versions and those used in B8. We continue to use in-house lookup tables of absorption coefficients (ABSCO) parameterized as a function of temperature, pressure, wavelength, and water vapor mixing ratio for each of the main absorbing gases in the OCO-2 bands: O2, CO2, and water vapor (H2O). Successive versions of these tables have been refined by incorporating new laboratory results and theoretical models for increasingly accurate absorption coefficients. The ABSCO version used in the B8 algorithm is ABSCO v5.0 ; B7 used the previous ABSCO version, v4.2.

The ABSCO v5.0 O2A band tables represent a major step forward from previous ABSCO versions. Earlier ABSCO versions integrated the highest-quality spectroscopic input from a range of studies that had focused on fitting different parameters independently. (See, for example, , and references therein.) The ABSCO v5.0 tables are based on self-consistent multispectral fits to laboratory spectra that include line mixing, speed-dependent Voigt line shape parameters, and collision-induced absorption (CIA). This self-consistency, and the use of laboratory spectra covering a range of pressures, temperatures, and measurement techniques, are key features of the approach. The O2 spectral line parameters, line mixing, and CIA used in ABSCO v5.0 are described in . Parameters for broadening of O2 by H2O are from the study by .

The impact of the latest multi-spectrum fitting update in the O2A band is shown in terms of the accuracy of the retrieved surface pressure in Fig. . Panel (a) shows the retrieved surface pressure minus the prior for ABSCO version 4.2, which was used in version 7 of the algorithm, while panel (b) shows the same for ABSCO v5.0, used in version 8. The main improvements seen are that the retrieved surface pressures in version 8 are essentially unbiased with respect to the meteorological prior over land, and that the land and ocean differences are reduced and centered closer to zero. We also note that for OCO-2, no additional line strength scaling was required in the O2A band, as has been necessary for all previous ACOS/GOSAT versions see, e.g.,. For ACOS-GOSAT B3.5 retrievals, an O2 scaling factor of 1.0125 was found to be beneficial, perhaps because of slight instrumental differences between OCO-2 and GOSAT.

The ABSCO v5.0 tables for the 1.61 and 2.06 µm CO2 bands use line parameters and line mixing models derived from self-consistent, multispectral fits by and , respectively. The parameters are derived from fits to laboratory spectra at multiple pressures and temperatures and the computation incorporates a speed-dependent Voigt line profile with nearest-neighbor line mixing. Earlier versions of the ABSCO tables were based entirely on room temperature multi-spectrum fitting, with theoretical temperature dependences of the line shape and line mixing parameters. The updated spectroscopy includes analyses of spectra recorded at temperatures from 170 to 296 K, representing a significant advance. Parameters for broadening of CO2 by H2O are from for the 4.3 µm CO2 band and extrapolated to OCO-2's CO2 bands. Validation of the ABSCO v5.0 tables using up-looking TCCON spectra is described in and . We note one important difference between the reference databases and our CO2 absorption coefficients. We found it necessary to incorporate additional absorption in the center of the 2.06 µm band. This additional absorption was parameterized in order to reduce errors in retrievals with TCCON up-looking spectra. Further details can be found in and .

Because the laboratory spectra underlying ABSCO currently are only good to roughly 1 % absolute accuracy of line intensities, the algorithm allows for overall scaling factors for each of the two CO2 bands. For the 1.61 µm band, the ABSCO v5.0 tables include a uniform scaling to bring the intensities from the multispectrum fit into line with reference intensity measurements (estimated accuracy ∼0.2 %) from . show that this pre-scaling of the ABSCO using reference laboratory measurements results in good consistency between single-band up-looking XCO2 retrievals from ground-based FTS spectra and the XCO2 values reported by TCCON (which are themselves calibrated to agree with reference airborne profiles). Reference intensity measurements are not available for the 2.06 µm band at the current time. In tests within the OCO-2 Level 2 algorithm, using OCO-2 radiances, a scaling of 1.004 for the 2.06 µm ABSCO table was found to yield the best agreement between single-band retrievals performed using this band compared with single-band retrievals performed using the 1.61 µm ABSCO table as described above.

Finally, ABSCO v5.0 tables incorporate H2O line parameters from the HITRAN 2012 compilation . We use an unofficial, modified version of the MT_CKD continuum, supplied by Eli Mlawer . This continuum version offers a compromise between previous versions of MT_CKD and measurements by , and falls relatively close to measurements by . The subsequently released MT-CKD 3.2 has been tested and shown to be a modest improvement over the unofficial version incorporated into ABSCO v5.0, with negligible changes to XCO2 but noteworthy improvements to the water column determination. Methane is not currently included in the B8 (or previous versions) of the forward model, as the impact of methane absorption was found to be negligible for XCO2 retrievals performed using the OCO-2 spectral ranges.

ACOS B3.3 introduced a new way to deal with large spectral residuals caused by imperfect spectroscopy, and solar model and instrument characterization, which were previously treated using a simple “empirical noise” parameterization . In contrast, the new approach fits scaling factors to fixed spectral residual patterns for each band.

These patterns are the EOFs that result from a singular value decomposition of spectral residuals from training retrievals. Training scenes were selected to be largely devoid of cloud and aerosol effects, such that residual patterns due to unfitted clouds and aerosols are not a large contributor to the resulting EOF patterns. The EOFs are constructed such that the residuals rs,b of each sounding s and band b can be approximately represented as a linear combination of the EOF patterns: rs,b=∑j=1Neofcj,s,bej,b, where the vectors ej,b are the EOFs for each band. For a diverse set of training retrievals, a matrix M is created for each spectral band and populated by the residuals of the spectral fits within that band. Training sets typically included more than 10 000 soundings.

Each matrix M is then decomposed into its eigenvectors using traditional singular value decomposition: M=UWVT, with the columns of U spanning an orthonormal basis of the most persistent spectral residual vectors observed in the training data set. By convention, the first eigenvector explains the largest fraction of the total variance, as indicated by descending order of singular values (the diagonal elements of W).

Application of this EOF technique substantially reduces the spectral residuals, yielding values of the relative rms of the residuals of ∼0.1 % for each band, and reduced χ2 values near unity. For GOSAT, only the first EOF was found to be necessary. For OCO-2 B7 (and B8) target-mode observations, better agreement with TCCON XCO2 was found when the largest three EOFs were employed. The first two EOFs for GOSAT, as well as the first three EOFs for OCO-2, are shown in Fig.  for each spectral band. The EOFs for each of the eight spatial footprints sampled by OCO-2 are extremely similar, though they are solved for independently due to the slightly different spectroscopic response of each. The first EOFs for OCO-2 and GOSAT are very similar for each band, indicating that common forward model errors such as spectroscopy and top-of-atmosphere solar flux, rather than instrument-specific effects, are driving the EOF patterns. The first EOF is also very similar to the mean residual pattern, and typically accounts for 50 %–60 % of the variance in the residuals. The second and third EOFs typically account for only 1 %–3 % of the variance in the residuals, with higher-order EOFs accounting for even less. For the O2A band, the second EOF appears as a Doppler shift of the first. In the weak CO2 band, the second EOF appears to be due to water vapor lines, while the third EOF appears to be a Doppler shift of the first. Higher-order EOFs often exhibit additional instrument artifacts (such as unidentified bad spectral samples) and forward model errors related to water vapor, as well as other effects that are difficult to interpret.

The EOF formulation was modified in B8 in several ways. First, the EOFs were defined in terms of radiance per unit noise rather than pure radiance, in order to be consistent with the cost function metric that is minimized during the retrieval itself. However, the structure of the EOFs in B8 was much the same as in B7. Second, improved filtering of spectral samples that are contaminated by noisy or dead detector pixels greatly reduced their impact on the EOF patterns. Finally, a manual re-ordering of the EOFs was performed for each of OCO-2's eight spatial footprints, because the standard ranking by variance would occasionally flip the EOF patterns in different footprints. This mattered because sometimes the third and fourth EOFs would change places (and the current algorithm only fits the first three EOFs). This ensured that, to the extent that the EOFs of each footprint roughly matched, the same three EOF patterns for each footprint are fit within the L2FP retrieval.

The forward model for ACOS L2FP retrievals uses one of two surface models, depending on the location of the footprint. Water surfaces are simulated as a linear combination of a Cox–Munk ocean surface and Lambertian reflector. This surface has seven parameters: wind speed, and a Lambertian albedo at a reference wavenumber with a linear spectral slope term in each of the three spectral bands. The prior wind speed is taken from the resampled meteorology (either ECMWF or GEOS5 FP-IT, as discussed in the next section), as for the other meteorological parameters. The strong CO2 band Lambertian albedo is fixed to 0.02; the other six terms are fit in an essentially unconstrained fashion. This approach leads to the fitted Lambertian albedos generally staying small and positive, the latter of which is currently required by our radiative transfer module.

Through ACOS B7, land surfaces were assumed to be purely Lambertian, with an albedo and albedo spectral slope retrieved for each band. The Lambertian surface assumes that the bidirectional reflectance distribution function (BRDF; the ratio of the radiance in the reflected direction to the irradiance from the incident direction; ) is a constant that is often specified as a scalar albedo. Since independent fits are done within each of the three OCO-2 spectral bands, this yields six state variables for land footprints.

Analysis of B7 OCO-2 target-mode observations showed that the retrieved Lambertian albedo and aerosol optical depth sometimes exhibited dependence on the sensor zenith angle for observations of the same surface location. This indicated that the true surface BRDF has dependence on the observation angles. A more physically justified approach would use a non-Lambertian model for the surface BRDF. For trace gas retrievals, a Lambertian surface assumption introduces no errors in the absence of multiple scattering between the surface and atmosphere; in this case, the retrieved albedo is interpreted as the surface reflectance at the primary scattering geometry (Sun–surface–satellite). However, over brighter surfaces with some atmospheric scattering, the assumed BRDF could in principle affect the retrieval via the interaction of the retrieved aerosol, surface pressure, and gas concentrations. Therefore, in B8 it was decided to change the surface model for land footprints to a non-Lambertian surface model. This model assumes a fixed BRDF shape and assumes the surface is azimuthally symmetric, but allows for spectral dependence of the amplitude between and within each of our three bands; full details of the BRDF model are given in Appendix . While this model does often show reduced correlation between view zenith and the retrieved BRDF amplitude, the retrieved XCO2 (as well as most other state vector parameters) shows very little change versus a version of the B8 retrieval run with a Lambertian surface. Therefore, while B8 does use a non-Lambertian BRDF parameterization, a Lambertian surface appears to work equally well. This fact may be a consequence of the strong filtering used in B8, which tends to remove soundings with multiple scattering. Future applications of the ACOS L2FP algorithm to cases with higher AOD may be more strongly impacted by the non-Lambertian BRDF.

In addition to these changes, a number of additional (mostly minor) changes have also been made to the ACOS L2FP algorithm since B2.9. In B2.10, the prior CO2 profile was changed to match that used by TCCON, which was more realistic than our previous prior formulation; as of B8, this corresponds to the GGG2014 version . Generally speaking the TCCON CO2 prior profile is relatively simple: it is a function of latitude, altitude, and date only. It includes a simple formulation of the seasonal cycle and currently assumes a fixed secular increase of 0.52 % yr-1 (or 2.08 ppm yr-1 at 400 ppm). There is no land–ocean or other meridional dependence. It requires specifying the tropopause height, and has simple formulations for the profile in the boundary layer, free troposphere, and stratosphere. A small mistake in the XCO2 averaging kernel was also fixed in B2.10; this was caused by inconsistent assumptions regarding the pressure-dependent gas absorption cross sections throughout our retrieval code, which led to an obvious “kink” in the averaging kernel that had long been visually evident see, e.g., Fig. 2 of. We use the 2016 version of the Toon solar transmittance spectrum (available at https://mark4sun.jpl.nasa.gov/toon/solar/solar_spectrum.html, last access: 4 December 2018) . Changes in the prior covariance matrix for CO2 were also considered, but rejected, as tests using alternate covariance matrices showed insignificant performance improvements.

In B3.3, solar-induced chlorophyll fluorescence (SIF) fitting over land surfaces was introduced. This change was introduced to combat a bias in XCO2 that results from not fitting for fluorescence when it is present, due to its impact on the O2A band. This problem and our fluorescence fitting scheme are described in detail in . Briefly, we fit for the mean and slope of the fluorescence at the top-of-canopy as a function of wavelength; the mean is expressed as a fraction of the continuum radiance level at a wavelength of 755 nm. The spectral dependence of the fluorescence is taken to be linear. Additional minor algorithm changes to B3.3 included reducing the prior surface pressure uncertainty from 4 to 1 hPa for GOSAT, which is likely to be a more accurate representation of the true prior surface pressure uncertainty for the majority of scenes see, e.g.,. This has the added benefit of reducing the interference error between SIF, aerosols, zero-level offset, and surface pressure. In B3.4, fitting for SIF was turned off for GOSAT medium gain observations over land, as these regions are nearly all desert with very little biological activity. For OCO-2, the SIF prior is taken from the official SIF retrieval, as described in , because SIF retrievals from individual soundings are meaningful for OCO-2 due to its relatively high SNR.

In B8, the prior height of the cirrus cloud layer relative to the surface pressure was moved slightly, from the fixed value of x=0.3 to just below the tropopause height (which is a relatively strong function of latitude). The calculation of the tropopause height itself was also refined in B8, which also improved the calculation of the prior CO2 profile. Finally, the prior meteorology was changed in B8 from ECMWF to GEOS5-FP-IT . Some statistics regarding differences in surface pressure, temperature, water vapor, and surface wind speed between the two models for retrieved GOSAT soundings are given in Table ; the corresponding difference statistics for OCO-2 soundings are nearly identical. Only soundings passing the O2A band prescreener are included. In general, the two models are very similar, with for instance 95 % of all soundings having a surface pressure difference of less than 1.5 hPa. Surface pressure probably affects our retrieved XCO2 the most, as it is used not only in the retrieval, but also in the bias correction, where differences in the prior surface pressure will lead to a first-order change in the bias-corrected XCO2. Currently then, the “noise” from the surface pressure difference between these two models would amount to roughly 0.6 hPa, or about 0.25 ppm in XCO2, which is quite a bit less than our noise-driven error (∼1.4 hPa on average) and regional biases (∼2.4 hPa on average). Retrieved surface pressure errors are discussed in more detail in Sect. .