Table 2 lists the most important retrieval specifics of six NO2 satellite data sets studied here.

In the OMNO2A v1 and v2 retrievals, the modelled spectrum is expressed in terms of reflectance (intensity), followed by a non-linear fit to the observed reflectances (intensity fit). The modelled reflectance used in OMNO2A v1 and v2 to minimise the fit residual r(λ) with the observed Robs(λ) is Rmod=I(λ)I0(λ)=P(λ)⋅exp⁡-∑k=1Nkσk(λ)⋅Ns,k⋅1+CRingIRing(λ)I0(λ)+r(λ), with I(λ) as the Earth radiance, I0(λ) as the 2005 annual average solar irradiance spectrum, and σk(λ) as the trace-gas cross sections. The Ring effect, caused by inelastic Raman scattering of incoming sunlight by N2 and O2 molecules (Grainger and Ring, 1962), is accounted for by the term inside the parenthesis on the right-hand side of Eq. (1). Here, CRing represents the Ring fitting coefficient and IRing(λ)/I0(λ) the Sun-normalised synthetic Ring spectrum. For usage in Eq. (1) σk and IRing have been convolved with the instrument slit function. This is different from many other fit models that include the Ring effect as a pseudo absorber, whereas in OMNO2A it is modelled as a source of photons influencing the backscattered contributions to the modelled reflectance. The radiance I is wavelength calibrated prior to solving the above equation, while the irradiance I0 is assumed to be well calibrated. All terms in Eq. (1) need to be given at the same wavelength grid: for OMNO2A the irradiance and the reference spectra are interpolated to the (calibrated) radiance wavelength grid. Fit parameters are the trace-gas slant columns Ns,k , the Ring effect coefficient CRing , and the coefficients αm of the DOAS polynomial P(λ)=∑αmλm of order m. Note that Eq. (1) is fully non-linear due to the way the Ring effect is included on the right-hand side. OMNO2A v2 slant column retrievals are improved relative to v1 via an optimised window used for the prior-to-fit wavelength calibration, leading to much reduced fitting errors, and via the inclusion of the absorption by the O2–O2 collision complex and by liquid water (H2Olq) (Van Geffen et al., 2015).

The OMINO2–QA4ECV retrieval performs a χ2-minimisation of the residual r(λ) using the QDOAS software (Danckaert et al., 2017) developed at BIRA-IASB, wherein the modelled spectrum is expressed in terms of optical depth, followed by a mostly linear fit to the observed optical depth (optical depth fit): Rmod*=ln⁡I(λ′)-Poff(λ′)I0(λ)=P*(λ)-∑k=1Nkσk(λ)⋅Ns,k*-σRing(λ)⋅CRing*+r*(λ), with Poff(λ) a first-order polynomial Poff(λ)=c0+c1⋅λ that describes the intensity offset correction (denoted as Ioff in Table 2), and λ′=λI+ωq⋅λI-λ0+ωs the calibrated radiance wavelength grid, with λI the input radiance wavelength grid, ωs a wavelength shift with respect to the wavelength λ0 of the centre of the fit window, and ωq a stretch (ωq>0) or squeeze (ωq<0) term. Note that the fit parameters on the left side of Eq. (2), the wavelength calibration and intensity offset correction, constitute non-linear terms of the linear fit. All terms in Eq. (2) need to be given at the same wavelength grid: for QDOAS the calibrated λ′ and the reference spectra are interpolated to the irradiance wavelength grid, calibrated before the fit using a high-resolution solar spectrum (Fraunhofer calibration). Fit parameters are the trace-gas slant columns Ns,k∗, the Ring effect coefficient CRing*, the coefficients αm* of the DOAS polynomial P*, the coefficients ci of the intensity offset polynomial Poff, and the wavelength calibration coefficients ωs and ωq. The polynomials P(λ) and P*(λ) effectively act as the high-pass filter mentioned in the description of the DOAS technique above. The coefficient ci represents the offset parameter that accounts for instrumental effects like stray light inside the spectrometer, instrumental thermal instabilities, changes in the detector's dark current, wavelength shifts between I and I0 or other remaining calibration issues in the level 1 product which are known to be sources of bias in DOAS retrievals of minor trace species. It may also account for atmospheric effects such as incomplete removal of Ring structures (De Smedt et al., 2008; Coburn et al., 2011; Peters et al., 2014; QA4ECV Deliverable 4.2 in Muller et al., 2016).

The χ2 merit function of the non-linear fit of Eq. (1) is defined by χ2=∑i=1NλrλiΔIλi/I0λi2, with Nλ the number of wavelengths λi in the fit interval and Δ(I/I0) the standard error on the measurement. In case of the mostly linear fit of Eq. (2) as performed in OMINO2–QA4ECV the residual is not weighted with the error on the measurement, so that the χ2 merit function is simply given by χ2=∑i=1Nλrλi2. The magnitude of χ2 is a measure for how good the fit is. We discuss DOAS SCD uncertainties in more detail in Sect. 4.1.2.

The OMNO2–NASA algorithm (used in NASA SP v3) uses the intensity fit (Eq. 1) as a default

When the intensity fitting approach fails (e.g. yields negative slant columns), the optical depth is modelled (Eq. 2) instead of the reflectances. If optical depth fitting also fails, then the solution from the intensity fit is provided as is.

All four OMI fitting approaches convolve high-resolution absorption cross section spectra with the OMI slit function (Dirksen et al., 2006; this pre-flight slit function is slightly modified to match the observed irradiances in OMNO2–NASA), which has proved to be stable throughout the OMI mission period (Schenkeveld et al., 2017; Sun et al., 2017). OMNO2A v1 uses a fixed slit function for all 60 rows, where in OMNO2A v2 the slit function has been updated with respect to OMNO2A v1 to better represent the across-track average (Van Geffen et al., 2015). In the OMINO2–QA4ECV and OMNO2–NASA algorithms, the cross section spectra have been convolved for each of the 60 across-track positions individually.

The GONO2A-BIRA spectral fits are performed using the QDOAS software developed at BIRA-IASB, which solves Eq. (2). The GONO2A-BIRA algorithm uses the 425–450 nm window and fits the absorption cross sections of NO2, O3, O2–O2, and H2Og. The fit also accounts for the Ring effect and includes an intensity offset, along with a third-order polynomial. The GONO2A–QA4ECV differs from the GONO2A-BIRA retrieval in the choice of a wider fitting window of 405–465 nm in the retrieval code and is largely identical to the approach taken in OMINO2–QA4ECV. Both algorithms use daily solar reference spectrum, which contrasts with the use of a fixed annual average or monthly-averaged solar reference spectra in the OMI retrievals. Previous studies indicated that SCDs retrieved from the same sensor in the 405–465 nm window are approximately 0.5 × 1015 molec. cm-2 higher than those retrieved from the 425–450 nm window (Van Geffen et al., 2015).

We compare the NO2 SCDs from the OMNO2A v2, OMNO2–NASA, and OMINO2–QA4ECV algorithms, with OMNO2A v1. Figure 1a shows average absolute NO2 SCDs as a function of latitude for all four OMI SCD products for unpolluted Pacific orbits from day 1 of January, April, July, and October 2005 up to 2015. The SCDs show lowest values in the tropics (shorter light path and lower VCDs), and higher values poleward. Averaged over all latitudes, the revised algorithms result in 12–15 % lower SCDs (1.2–1.4 × 1015 molec. cm-2) than OMNO2A v1 SCDs, in line with the reductions reported for OMNO2A v2 in Van Geffen et al. (2015). The revised OMNO2A v2, OMINO2–QA4ECV, and OMNO2–NASA SCDs are in close agreement (differences < 4 %). The OMNO2–NASA SCDs (and their uncertainties) used in this analysis correspond to the latest version (v3.1; to be released) of the new Standard Product (SP) (Krotkov et al., 2017). Over the chosen clean-sector area the v3.1 SCDs are on average higher by ∼ 0.5 × 1015 molec. cm-2 than v3.0. Differences between v3.1 and v3.0 SCD values are related to the changed approach to flagging the presumably noisy wavelength bins in the OMI radiances as well as improved solar reference spectra. The GOME-2A NO2 SCDs (Fig. 1b) are ∼ 2–3 × 1015 molec. cm-2 lower than for OMI, which is anticipated because of the diurnal increase in stratospheric NO2 (e.g. Dirksen et al., 2011) and differences in viewing geometries. The GONO2A–QA4ECV SCDs are in line with GONO2A-BIRA, with the latter showing on average slightly lower values (by < 0.5 × 1015 molec. cm-2), reflecting the similarity of the BIRA and QA4ECV algorithms. Their main differences are the choice of fitting window and that the H2Olq is not fitted in the small fitting window (for GONO2A-BIRA). Their relative difference is highest (∼ 12 %) around the Equator.

For HCHO, a comparison of SCDs is less straightforward than for NO2. First of all, daily Earth radiance spectra are used as a reference for the DOAS retrievals instead of solar irradiance spectra. The Earth radiance reference spectra are taken over a reference sector in the equatorial Pacific, where CH4 oxidation is the only significant source of HCHO. The resulting (differential) HCHO SCDs may then have values close to zero, or even be negative, indicating that a scene has a similar or smaller HCHO amount than in the reference spectrum. After the fit, a background correction is applied to the SCDs (De Smedt et al., 2015). The final differential SCDs (ΔSCDs) are the result of subtracting the mean HCHO SCD over each OMI row and by 5∘ of latitude bins within the reference sector (Ns0), from the SCDs (Ns) of the same day, ΔNs=Ns-Ns0 (QA4ECV Deliverable 4.2 in Muller et al., 2016; De Smedt et al., 2017a, 2018). This normalisation approach and the choice of daily radiance spectra results in ΔSCDs close to zero over the reference region. Selecting daily Earth radiance reference spectra helps to reduce the effects of radiance degradation for GOME-2A retrievals and the effects of stripes for OMI. The final tropospheric HCHO vertical columns (Nv) are then defined as Nv=ΔNsM+M0MNv,0,CTM, where M is the tropospheric AMF, and M0 and Nv,0,CTM are respectively the AMF and the model background column in the reference sector.

Figure 2a shows a comparison of HCHO SCDs before (light lines) and after (dark lines) background correction from the OMIHCHO–QA4ECV and OMIHCHO–BIRA algorithms. Their differential SCDs (ΔSCDs; dark green and black symbols) are highly consistent, with only a small difference of ∼ 0.7 × 1015 molec. cm-2 on average. This suggests that the improvements made in the QA4ECV OMI HCHO fitting code do not lead to substantial changes in the HCHO columns, but we will see later that there is considerable impact on the uncertainties of the fits.

We see similar behaviour for the GOME-2A HCHO SCDs provided by the GO2AHCHO–BIRA and GO2AHCHO–QA4ECV algorithms (Fig. 2b). As with OMI, averaged over all latitudes the difference between ΔSCDs is small (< 0.9 × 1015 molec. cm-2). For the retrieved SCDs, the differences are larger (up to 15 × 1015 molec. cm-2) at all latitudes, stressing the importance of the background correction.

The DOAS technique tries to minimise the differences between the observed and the modelled spectra within a nominal wavelength window (spectral points of length K). The Levenberg–Marquardt non-linear least-squares fitting procedure (M–L) is the numerical routine that performs the χ2 –merit function minimisation (Press et al., 1997) and provides the fitting parameters (of length M) (SCDs, Ns) and a covariance matrix that contains an estimate of the uncertainty in the fitting parameters (SCD uncertainty, εNs,j; “DOAS SCD uncertainty” hereafter) for a typical non-linear fit. This routine is also used by a mostly linear fit in order to find the non-linear parameters, followed by a solution (the QR decomposition of the cross sections matrix for QDOAS and the singular value decomposition for NLIN) for a typical least squares problem for the linear parameters.

The diagonal elements of the covariance matrix, C, are the variances of the fitted parameters. The uncertainty in the fitted parameter, εNs,j, is the square root of the variance: εNs,j=χ2ATAjj-1, where A is the matrix formed by the absorption cross sections, which has K×M components constructed from the M basis functions evaluated at the K abscissas xi (i.e. X1(x), …, XM(x)), and from the K measurement errors εi, using the prescription Aij=Xjxiεi. The off-diagonal elements are the covariances between the parameters. In the non-linear intensity fit approach of Eq. (1) all components of the fit are accounted for in the uncertainty estimate. In the QDOAS and NLIN fits (Eq. 2) only the linear components in the fit are accounted for: uncertainties on estimated values of the non-linear parameters (i.e. shift, squeeze and intensity offset parameters) are not taken into account in the uncertainty estimate of the SCDs, and the measurement errors are not used in the fit (εi=1) (Danckaert et al., 2017). The SCD uncertainties are then estimated using the reduced χ2 (instead of the nominal χ2), i.e. Eq. (3b) divided by the number of degrees of freedom in the fit, K-M.

Uncertainties on the retrieved SCDs thus depend on the following:

the accuracy (sensitivity) of the fitting model in capturing the ensemble of spectral features in the observed, noisy reflectance spectrum,

The DOAS SCD uncertainty may consist of two parts: a random and a systematic error component.

To evaluate the DOAS SCD uncertainty estimates and to have an independent means to intercompare the results of the different retrieval methods, we apply an alternative, statistical method. We follow the approach laid out in Wenig et al. (2001) and Boersma et al. (2007) to quantify the spatial SCD variability over pristine, unpolluted areas and assume that such estimates serve as a statistical indicator of the SCD uncertainty. The main contributors to the SCD variability are the instrument (level 1) noise, natural variability within the unpolluted area, scene reflectance (surface, clouds) and viewing geometry variability. Our objective is to provide an estimate of the random component of the SCD uncertainty by limiting the contributions from other components to the variability over the unpolluted area. We focus our analysis on the remote area within 60∘ N–60∘ S and 150–180∘ W (Pacific Ocean). Practically free of tropospheric pollution, this area is separated in 2∘ × 2∘ (longitude × latitude) “boxes”, which limits geophysical variability and provides statistically robust sampling. We assume that pixels within each box record the same NO2 or HCHO total vertical columns. Any variability emerging in the retrieved (all- or clear-sky) ensemble is then attributed to random uncertainty originating from noise in the level 1 data and imperfections in the spectral fitting model, as long as the geometric AMFs within the box show little variability

The relative AMF variability for each box was computed as follows: Mi2‾-Mi‾20.5/Mi‾, where Mi is the AMF attributed to each pixel within the box.

Boxes with relative AMF variability of more than 5 % are discarded to prevent variability in viewing geometry influencing the results. In practice, the AMF variability in most boxes does not exceed 3.5 %; i.e. SCDs in each box are observed under very similar viewing geometries. For these boxes we compute standard deviations of the SCDs as the statistical SCD uncertainties. In the DOAS fit, NO2 is fitted assuming a fixed temperature for its absorption cross section of T0=220 K, and HCHO is fitted assuming T0=298 K. In most retrieval algorithms, a post-correction on the slant columns is applied to compensate for neglecting the actual atmospheric temperature of the trace gas, but this is typically done in the later AMF step. The slant columns used in this analysis are not yet corrected for the temperature-dependency of the NO2 and HCHO absorption cross sections. For all OMI algorithms the DOAS uncertainty estimates may contain contributions from stripes. The statistical HCHO SCD uncertainties reported in the following sections concern the differential HCHO SCDs (ΔSCDs), which are known to suffer to a lesser extent from this artefact (see Sects. 2.1 and 4.3).

We now compare the OMI NO2 DOAS and statistical SCD uncertainty estimates. The algorithms show a slight decrease in statistical and DOAS NO2 SCD uncertainties with increasing latitude (Fig. 3). For OMNO2A v1, v2, and OMINO2–QA4ECV the DOAS uncertainty exceeds the statistical uncertainty. We attribute this to persistent (systematic) fitting residuals and signatures unexplained by the fitting technique. Averaged over all latitudes, the relative difference between the statistical and DOAS uncertainty reduces from ∼ 60 % for OMNO2A v1 to ∼ 20 % for OMINO2–QA4ECV. This reduction hints at an improved understanding of the spectral features, and especially at the reduction in systematic parts of the residuals in the OMINO2–QA4ECV spectral fitting method relative to OMNO2A v1, in line with findings in Van Geffen et al. (2015) and Anand et al. (2015) that OMNO2A v1 was suffering from inaccurate wavelength calibration.

Both statistical and DOAS SCD uncertainties are on average smallest for OMINO2–QA4ECV (15 and 35 % lower than OMNO2A v1), which may indicate a more physically accurate fitting model for that algorithm. The DOAS uncertainty from OMNO2–NASA shows a smoother geographical variation than the pattern of the statistical uncertainty, which shows substantial variation with latitude (Fig. 3d). The average OMNO2–NASA DOAS and statistical uncertainties are of similar magnitude, in contrast to higher DOAS than statistical uncertainties for OMNO2A v1, v2 and OMINO2–QA4ECV. The OMNO2–NASA v3.1 DOAS SCD uncertainties are on average 40 % lower than v3.0. This reduction in the DOAS SCD uncertainties stems from a correction of an error in the v3.0 algorithm. The statistical SCD uncertainties are similar between v3.1 and v3.0 (agreement within 0.02 × 1015 molec. cm-2). The DOAS and statistical uncertainties shown in Fig. 3 for the OMNO2A versions are consistent with estimates reported for OMNO2A v1 in Boersma et al. (2007) and Anand et al. (2015), and for OMNO2A v2 in Van Geffen et al. (2015).

Figure 4 shows histograms of the absolute differences between the individual SCDs and the box-mean SCD for OMNO2A v1 and v2, OMINO2–QA4ECV, and OMNO2–NASA. The histogram of SCD differences in the OMINO2–QA4ECV ensemble has the highest peak and smallest width (FWHM 1.6 × 1015 molec. cm-2) of the four algorithms. All histograms closely follow a Gaussian distribution, which is consistent with our initial assumption that random errors in the slant columns are responsible for the variability within each box, and originate mostly from measurement noise. The width (1σ) of the Gaussian function fitted to the observed distributions can be used as an alternative indicator of the overall, mission-averaged statistical uncertainty in the SCDs for the different algorithms. The mission-average uncertainty for the OMINO2–QA4ECV amounts to 0.69 × 1015 molec. cm-2, with significantly larger values for the OMNO2A and OMNO2–NASA algorithms. These findings are in agreement with the statistical uncertainty averaged over all latitudes shown as dashed lines in Fig. 3. Table 4 summarises the estimates of the statistical and DOAS uncertainties for OMNO2A v1, v2, OMINO2–QA4ECV and OMNO2–NASA SCDs for all-sky and clear-sky situations.

One question is whether SCDs for dark scenes are more uncertain than the SCDs obtained for bright scenes. The dark scenes, often associated with clear-sky conditions (cloud radiance fraction < 0.5), are of most interest for tropospheric retrievals. In the studies by Anand et al. (2015) and Marchenko et al. (2015), it was suggested that spectral fitting over (partly) cloudy scenes may result in less stable SCDs because of substantial wavelength shifts caused by the inhomogeneous illumination of the instrument slit (Voors et al., 2006). On the other hand, bright scenes have higher reflectance levels and therefore potentially higher signal-to-noise ratios, and if the wavelength calibration is sufficiently accurate in the fitting procedure, lower SCD uncertainties may be expected for such scenes. We repeated the statistical tests for the spectral fitting algorithms shown in Figs. 3 and 4, but only selected SCDs obtained under relatively cloud-free (clear-sky) conditions. For clear-sky scenes, the SCD uncertainty varies less with latitude than shown in Fig. 3 and the absolute uncertainties are higher by a factor of 1.1 compared to the all-sky SCD uncertainty estimates. This indicates that reduced signal-to-noise in the level 1 data (dark scenes) increases absolute SCD uncertainties. We recommend using the statistical estimates for clear-sky conditions in Table 4 as adequate estimates of SCD uncertainties for the above algorithms in the context of tropospheric NO2 column retrievals.

Boersma et al. (2007) reported that the uncertainty in the OMI NO2 retrievals due to spectral fitting with the OMNO2A v1 set-up is of the order of 0.7 × 1015 molec. cm-2 based on the variability seen in the de-striped SCDs over the Pacific on 7 August 2006, when the row anomaly was still confined and affected only one of OMI's rows. The larger statistical uncertainty found here for the OMNO2A v1 SCDs for the 2005–2015 time period (∼ 0.8 × 1015 molec. cm-2) is thus reasonable. The OMNO2A v2 statistical uncertainty is slightly (∼ 6 %) lower than for OMNO2A v1. Van Geffen et al. (2015) found the DOAS SCD uncertainties computed by the OMNO2A v1 and v2 spectral fits to be 1.3 × 1015 and 1.0 × 1015 molec. cm-2 respectively for Pacific Ocean orbits in 2007. The improvements to the OMNO2A v2 spectral fit reduced the DOAS slant column uncertainty by approximately 0.3 × 1015 molec. cm-2 (or 24 %). The results from our 11-year period investigated are consistent with those findings (Table 4).

Here we compare the GONO2A–QA4ECV with GONO2A-BIRA SCD uncertainties (Fig. 5 and Table 4). As with OMI, the GOME-2A NO2 DOAS uncertainties exceed the statistical ones. Averaged over all latitudes (not shown), for GONO2A-BIRA the DOAS uncertainty exceeds the statistical uncertainty by 26 %, and by 35 % for GONO2A–QA4ECV. The improvement in the GONO2A–QA4ECV spectral fitting is demonstrated by both DOAS and statistical uncertainties being on average 10 and 13 % smaller than those for the GONO2A-BIRA data set. This is confirmed by Fig. 5, which shows the highest peak and smallest width in the histogram of the SCD vs. box-mean SCD differences for GONO2A–QA4ECV (FWHM 1.3 × 1015 molec. cm-2) compared to GONO2A-BIRA (FWHM 1.5 × 1015 molec. cm-2). The deviations of the SCDs from the box-mean SCD form a normal distribution illustrative of the random nature of the noise in the GOME-2A level 1 data which drives the total SCD uncertainty. We conclude that, similarly to OMI, the improved QA4ECV fitting algorithm results in more precise fitting results for NO2.

The mission-average QA4ECV NO2 SCD uncertainties from OMI and GOME-2A are comparable in magnitude; the statistical and DOAS uncertainty for GOME-2A (0.56 × 1015 and 0.80 × 1015 molec. cm-2) are lower than for OMI (0.69 × 1015 and 0.84 × 1015 molec. cm-2). Initially, a higher spectral fit quality was expected for GOME-2A because of the instrument's higher signal-to-noise (2 × larger than OMI; see Table 1). This is indeed the case for the early years of the instruments' mission. In 2007, the GOME-2A NO2 SCD statistical uncertainty (∼ 0.45 × 1015 molec. cm-2; Fig. 11, left) was lower than for OMI (∼ 0.66 × 1015 molec. cm-2; Fig. 9c). The relatively fast degradation of the GOME-2A level 1 data has deteriorated the quality of the GOME-2A fits as diagnosed by (1) severe throughput loss (see Sect. 2.2), (2) instability of the instrument slit function due to thermal fluctuations of the GOME-2A optical bench, and (3) potential degradation of the reflectance. In contrast, OMI has shown exceptional stability, even after the occurrence and expansion of the row anomaly, and after far exceeding its designed lifespan. This explains why GOME-2A retrievals show comparable SCD uncertainties to OMI and will be discussed in detail in Sect. 4.3.

The spectral fitting of HCHO is more challenging than for NO2 because of its relatively small differential optical depth (Table 1), lower instrument signal-to-noise in the UV and stronger interferences from other absorbing species (e.g. from O3). Therefore, measurement noise and the presence of other species' absorption fingerprints in the same fitting window limit the HCHO detection. This is reflected by the larger random (and systematic) SCD uncertainties for HCHO relative to NO2. The OMIHCHO–QA4ECV SCDs have an uncertainty of ∼ 8 × 1015 molec. cm-2 (Fig. 6a), 10 times larger than OMINO2–QA4ECV (Table 4). As for NO2, QA4ECV results also show smaller OMI HCHO SCD uncertainties compared to the BIRA algorithm. The wider QA4ECV fitting window allows for a reduction in the SCD uncertainty even though bromine monoxide (BrO) is now included in the fitting procedure (and not pre-fitted). On average, the OMIHCHO–QA4ECV SCD uncertainties are 18 % smaller than those from OMIHCHO–BIRA, confirming the improvements in spectral fitting, consistent with the extensive tests and improvements for OMI HCHO fitting (QA4ECV Deliverable 4.2 in Muller et al., 2016).

The new GOME-2A fitting algorithm (GO2AHCHO–QA4ECV) did not result in a statistically significant reduction in SCD uncertainties compared to the BIRA algorithm (Fig. 6b and Table 5). On average, the HCHO statistical SCD uncertainty for GO2AHCHO–QA4ECV is 11 % higher than for GO2AHCHO–BIRA. The apparent lack of improvement is discussed in Sect. 4.3.3.

In 2017 OMI has exceeded its anticipated lifespan by 7 years. Throughout the mission, the row anomaly, stripes and the instrument's radiometric degradation all affected the SCDs and their uncertainties. In this section we discuss possible changes in stability and quality of the DOAS fits throughout the 2005–2015 period. The optical degradation in the OMI visible channel is well below 5 % over the mission so far (e.g. Boersma et al., 2011; QA4ECV Deliverable 4.2 in Muller et al., 2016; Schenkeveld et al., 2017). There are, however, clear signs of gradually increasing noise in the OMI radiances and irradiances mostly related to the long-term CCD performance (Schenkeveld et al., 2017), so we should anticipate a decrease in fitting quality over time. Figure 9 shows the evolution of the statistical and DOAS NO2 SCD uncertainties for the OMNO2A v1, OMNO2A v2, OMINO2–QA4ECV and OMNO2–NASA algorithms. For all retrievals, SCD uncertainties show a weak positive trend (also see Table 6). The statistical SCD uncertainties for OMINO2–QA4ECV increase by 0.9 % year-1 relative to the start and well below the ∼ 2 % year-1 increase for the OMNO2A and OMNO2–NASA algorithms. The OMNO2–NASA DOAS uncertainties are virtually without trend (-0.3 % year-1) in contrast with the statistical estimates. For clear-sky scenes, the rate of increase in the DOAS and statistical SCD uncertainties is somewhat higher relative to all-sky scenes for OMNO2A v1, v2 and OMINO2–QA4ECV (Table 6).

OMI shows low optical degradation and high wavelength stability over the mission lifetime. One can thus raise the question of why the SCD uncertainty increases in time, since OMI, apart from the RA, continues to perform well (Schenkeveld et al., 2017). Increases in dark current are monitored and corrected for daily, so these are unlikely to contribute to the trend. Increases in the random telegraph signal cannot be corrected for (Nico Rozemeijer, personal communication, 2017) and may contribute to a trend in SCD uncertainties. The number of pixels flagged as bad (those with off-nominal behaviour) increased to 11 %. Furthermore, stripes are apparent in trace-gas column retrievals since the beginning of the mission, and their magnitude has increased over time (Boersma et al., 2011).

In Sect. 4.1.3 we saw that the NO2 DOAS SCD uncertainty generally exceeds the statistical uncertainty reflecting persistent systematic uncertainty in the DOAS fit. Here, we investigate the amount of uncertainty in the total NO2 SCD uncertainty originating from stripes. This stripe-induced uncertainty is estimated as the root mean square of the stripe correction for rows 0–21 and 54–59 per OMINO2–QA4ECV orbit. Figure 10a shows the stripe-induced uncertainty increase from 0.33 × 1015 to 0.48 × 1015 molec. cm-2 over 2005–2015 (a 45 % increase). Hence, we subtract

The stripe-induced uncertainty, εstr, is subtracted from the total SCD uncertainty (i.e. DOAS uncertainty), εtot, by the prescription: εtot2-εstr2=εw/o, where εw/o is the SCD (DOAS) uncertainty without the contribution from stripes.

Total NO2 SCD (DOAS) uncertainties for OMINO2–QA4ECV increase by 17.5 % over 11 years. After subtracting the contribution from stripes, the SCD uncertainties increase by 9.8 % over the same time period, closer to what is expected from the radiometric degradation. Accounting for stripes reduces the systematic component to the total uncertainty by ∼ 70 %, and the DOAS and statistical uncertainty estimates are now in better agreement (within 6 %, Fig. 10b). The statistical and DOAS uncertainty now follow the same increase rate (0.9 % year-1), suggesting that stripes explain much of the discrepancy between the DOAS and statistical uncertainty estimates (Fig. 3c). The origin of the stripes is not well known but it is most likely associated with noise and instrument-related artefacts in the solar irradiance spectrum. The presence of stripes manifests when a fixed solar spectrum (2005 annual mean for OMNO2A and OMINO2–QA4ECV) is used as a reference for all years, so that the representativeness of that spectrum is reduced in years later than 2005. This is supported by the use of a daily Earth radiance spectrum as a reference rather than a fixed irradiance spectrum in OMIHCHO–QA4ECV, resulting in much weaker increases in the OMI HCHO SCD uncertainty (0.3 % year-1). Anand et al. (2015) also pointed out these (and other) benefits from using an Earth radiance reference rather than solar irradiance spectra. For future NO2 spectral fitting algorithms the choice of radiance over irradiance spectra as a reference is debatable: on the one hand the SCDs will suffer significantly less from stripes, but on the other the retrieved SCDs will no longer be absolute rather than differential. A background correction would be required to convert differential SCDs to absolute SCDs by adding an observed climatological or modelled stratospheric slant column. As a compromise, the NASA retrieval uses monthly-averaged solar data (Marchenko et al., 2015).

We now investigate the performance of the BIRA and QA4ECV DOAS fits for GOME-2A throughout 2007–2015. Both GONO2A–QA4ECV DOAS and statistical uncertainties are lower than BIRA, but they still show a substantial positive trend (Fig. 11). Starting from values of ∼ 0.4–0.6 × 1015 molec. cm-2 in 2007, the GONO2A–QA4ECV statistical and DOAS uncertainties increase by 57 and 45 % (relative to the start) by the end of 2015. This corresponds to an annual increase rate of ∼ 7 and ∼ 5 % year-1 for the statistical and DOAS uncertainty, respectively (Fig. S4 and Table S2), notably higher than what was found for OMI (Table 6). A continuous spectrally dependent throughput degradation (UV: 20 % year-1; VIS: 10 % year-1) has been observed since the GOME-2A launch in 2007. In September 2009, a second throughput test was performed (first test was in January 2009). The second test caused an additional throughput decrease of 25 % in the UV and 10 % in the visible. Despite the substantial throughput loss, the test also stabilised GOME-2A degradation. The reported linear degradation rate after the second throughput test in September 2009 fell to ∼ 3 % year-1 for the UV channel and 1 % for the visible channel. Munro et al. (2016) and Beirle et al. (2017) also reported a general long-term drift of the instrument's slit function, a key quantity for wavelength calibration and for convolution of the cross sections to the sensor's resolution. These changes are considerably weakened after the test and the slit function appears quite stable. Motivated by the continuous degradation of GOME-2A and the second throughput test in September 2009 with the positive effects reported on the quality of the level 1 data (EUMETSAT: GOME-2 Throughput Degradation ESA Final Report, 2011), we performed linear regressions for two sub-periods: before and after the second throughput test. The reduction in fitting quality for GONO2A-BIRA and GONO2A–QA4ECV appears to proceed at a much higher pace before the second throughput test (9–12 % year-1) than after (2–4 % year-1) (Table 7), consistent with the reported degradation rate for the visible channel before (11 % year-1) and after (1 % year-1) the test. The reduction in the uncertainty increase rate is even stronger for clear-sky scenes. GOME-2A HCHO SCD uncertainties show similar behaviour: before the test the uncertainty increases at a pace of 12–17 % year-1 (20 % year-1 reported for the UV), while after the test the increase rate is 1–4 % year-1 (3 % year-1 reported for the UV).

On 15 July 2013, GOME-2A pixel sizes were reduced from 80 × 40 to 40 × 40 km2. With the integration time for each detector pixel remaining the same, the SCD uncertainties between July 2013 and December 2015 have not changed relative to the period September 2009–July 2013. Table 7 summarises the trends in GOME-2A NO2 SCD uncertainties.

Figure 12 shows the evolution of the HCHO SCD uncertainties for OMIHCHO and GO2AHCHO. For OMI, the statistical uncertainty estimates show weak positive trends of 0.5 and 0.4 % year-1 for OMIHCHO–QA4ECV and OMIHCHO–BIRA, respectively, relative to the start. This confirms the remarkable stability of the OMI level 1 data and suggests that these OMI HCHO retrievals are in principle useful for the detection of trends in HCHO columns. The potential impact of spectral interferences of O3 and BrO absorption features on the HCHO fit (e.g. González et al., 2015), and conceivably on the HCHO trends, is largely mitigated by the background correction scheme. Due to the nature of this correction, only geographically localised O3 and BrO trends coincidental with high HCHO emission regions could affect the corrected HCHO columns. Such effects, if any, are unlikely to lead to pervasive, substantial biases in HCHO trend analyses.

The situation is quite different for GOME-2A. Overall, the statistical QA4ECV HCHO SCD uncertainties increased from ∼ 8 × 1015 to 14 × 1015 molec. cm-2 (2007–2015), which corresponds to ∼ 8 % year-1 relative to the start (Fig. S5 and Table S3). The effect of the throughput test in September 2009 is evident: after the test, the QA4ECV SCD uncertainties increased by only 1–2 % year-1, a clear improvement in degradation from 12 to 17 % year-1 (2 × the rate observed in GOME-2A NO2) before the test (Fig. 12 and Table 8).

Figure 12 suggests that GO2AHCHO–QA4ECV deteriorates more than GO2AHCHO–BIRA, especially after the second throughput test. This is mainly due to the fact that GO2AHCHO–QA4ECV uses a larger fitting window, and that GOME-2A radiances contain polarisation structures in this interval. To reduce polarisation-related systematic errors, pseudo cross sections have been included in the fit, which results in somewhat increased random uncertainty (and systematic uncertainty if not perfectly mitigated by the background correction) in the HCHO SCDs. Despite the increase in the random uncertainty, the SCD uncertainty increases at a slower pace, suggesting that the GOME-2A HCHO retrievals will allow the (challenging) detection of trends in HCHO columns.

The recent retrieval developments (e.g. the systematic reduction in SCDs by ∼ 1.2 × 1015 molec. cm-2 along with a 30 % reduction in fitting errors from OMNO2A v1 to v2 in Van Geffen et al., 2015) and the QA4ECV-driven improvements reported here (Figs. 1 and 3) suggest that at least part of the SCD uncertainty is systematic rather than random but also that such systematic effects can be removed. If we consider the SCD uncertainties to be completely systematic, then we should regard the DOAS SCD uncertainties as a lower limit for trends in stratospheric NO2 that can be reliably detected from stratospheric NO2 column time series. This would imply that from, for example, the QA4ECV OMI data set, one can only infer trends in stratospheric NO2 columns larger than 0.3–0.4 × 1015 molec. cm-2 decade-1 (SCD uncertainty divided by typical stratospheric AMF). In practice, however, the DOAS SCD uncertainty as we know it consists of a random (from level 1 noise) and a systematic (primarily from stripes) part, as shown in Sect. 4.3.1. The random component of the SCD uncertainty can be reduced to virtually zero by averaging over space and/or time. The differences between the total (i.e. DOAS) SCD uncertainty (with random + systematic contributions) and statistical SCD uncertainty (random component), as shown in Figs. 3 and 10 (ε2-εr2=εs2), then provide a lower limit of trend detection (from systematic uncertainty) in OMI stratospheric NO2 columns down to 0.1–0.2 × 1015 molec. cm-2 decade-1.

Uncertainty in the SCD does not directly translate into tropospheric column uncertainty as it does for stratospheric column uncertainty. The tropospheric retrieval is based on the difference between the DOAS SCDs and estimated stratospheric SCDs, as well as various factors related to the AMF evaluation. Since the stratospheric SCDs depend on the DOAS SCDs (e.g. Dirksen et al., 2011; Beirle et al., 2016), additive systematic offsets in the SCDs will largely cancel out in the tropospheric residual SCD. In Van Geffen et al. (2015), spectral fitting retrieval improvements were shown to be mostly additive, suggesting that systematic components of the SCD uncertainty are of less relevance for NO2 tropospheric column retrievals. Marchenko et al. (2015) discussed the possibility of a considerable systematic, multiplicative factor (between OMNO2A v1 and OMNO2–NASA), and such a component, if real, would be relevant for NO2 tropospheric column retrievals and their usefulness for trend detection. The instability in the SCDs because of stripes (OMI) or instrument degradation (GOME-2A) was evaluated further by testing the robustness of the tropospheric signal over a well-chosen reference area with little known pollution. We find that for OMI and GOME-2A the monthly mean tropospheric NO2 columns are stable throughout 2005 (2007 for GOME-2A)–2015 with no significant trend over a pristine region (see Fig. S6).

In the absence of a substantial systematic, multiplicative error in the NO2 SCDs, the stability of tropospheric NO2 vertical columns will therefore be dominated by instability in the AMF uncertainties. For instance, if assumptions on surface albedo or a priori NO2 profile shape grow increasingly inaccurate over time (because of e.g. urbanisation, increasing aerosol haze, change in vegetation), this will lead to growing systematic uncertainties in tropospheric AMFs (Lamsal et al., 2015). Such systematic or structural uncertainties may increase to up to 30–40 % in rapidly changing regions such as parts of India and China (Lorente et al., 2017).