Several UV/vis satellite instruments provide column measurements of atmospheric NO2. Table  summarizes the characteristics of the instruments and provides references to the data products used in this study, from which the total NO2 SCD, the stratospheric AMF, and the cloud fraction/cloud top height are taken as input for STREAM. Below we provide details on the satellite characteristics and the data sets used in this study, starting with OMI (as STREAM was optimized for OMI within TROPOMI verification) and GOME-2, followed by older instruments with particular challenges such as poor spatial coverage (SCIAMACHY) or resolution (GOME).

For comparisons, and for the calculation of synthetic total columns for performance tests of STREAM, we make use of stratospheric NO2 as provided by the ECHAM5/MESSy Atmospheric Chemistry (EMAC) model, which is a modular global climate and chemistry simulation system .

We use the results from simulation RC1SD-base-10a of the ESCiMo (Earth System Chemistry integrated Modelling) project as detailed by . Here, only basic information on this specific simulation is summarized.

The model results were obtained with ECHAM5 version 5.3.02 and MESSy version 2.51 at T42L90MA resolution, i.e., with a spherical truncation of T42, corresponding to a quadratic Gaussian grid of approx. 2.8∘ by 2.8∘ in latitude and longitude, and 90 vertical hybrid pressure levels up to 0.01 hPa. The dynamics of the general circulation model was nudged by Newtonian relaxation towards ERA-Interim reanalysis data .

Simulation RC1SD-base-10a was selected from among the various ESCiMo simulations for several reasons: a.

it has been nudged to reproduce the “observed” synoptic situations;

In conclusion, this simulation represents the state-of-the-art in terms of numerical simulation of the atmospheric chemistry. Moreover, the applied nudging technique allows a direct comparison with observational data, since the simulated meteorological situation corresponds to the observed.

Specifically for this study, the submodel SORBIT was used to extract NO2 mixing ratios along the sun-synchronous orbit of the Aura satellite, thus matching the local time of OMI observations. Stratospheric VCDs were calculated by vertical integration of the modeled NO2 mixing ratios between the tropopause height (as diagnosed according to the WMO definition based on lapse rate equatorwards of 30∘ north/south and as iso-surface of 3.5 PVU potential vorticity poleward of 30∘ latitude) and the top of the atmosphere.

In this study, we make use of the modeled stratospheric columns for two purposes.

We perform a simple model-based STS for comparison. To remove systematic biases between satellite measurements and EMAC, a latitude dependent offset is determined in the Pacific and corrected for globally, similar as in and . We refer to this EMAC-based STS as STSEMAC and applied it to OMI data (Sect. ).

We test the performance of STREAM on synthetic VCDs, which allows a quantitative comparison of the estimated TR to the a priori “truth”. The input to STREAM, i.e., synthetic total columns of NO2, should realistically represent (a) stratospheric chemistry and dynamics, (b) tropospheric emissions, transport, and chemistry, (c) cloud fields, and (d) the satellite sampling. For these purposes, we construct synthetic NO2 column densities V∗ based on a.

stratospheric VCDs from EMAC

Stratospheric columns are taken from EMAC rather than TM4, as the latter does not represent a free model run of stratospheric chemistry and dynamics but uses the satellite measurements for assimilation.

Figure  displays the OMI daily mean VCD V∗ (top) as well as the respective stratospheric field from RSM (second row) and STREAM (third row) for 1 January (left) and 1 July 2005 (right), respectively. The overall latitudinal as well as longitudinal dependencies are clearly reflected in the stratospheric fields, while small-scale stratospheric features are lost by the spatial convolution. Figures  and  display the resulting TRs, respectively, for both daily (top) and monthly (bottom) means. Figure  summarizes the daily and monthly statistical properties of TR, i.e., the median as well as 10th/90th and 25th/75th percentiles (light/dark bars) for different regions (see Fig. S8 for an illustrative sketch of the meaning of the percentile bars, as well as the definition of regions).

Overall, spatial patterns of TR are similar for RSM and STREAM, in particular the enhanced values reflecting tropospheric pollution over, e.g., the USA, central Africa, or China. However, RSM reveals several artifacts of both enhanced as well as systematically negative TR as a consequence of the simple assumption of zonal invariability of stratospheric NO2. For instance, on 1 January 2005, VCDs over northern Canada are lowered due to the polar vortex (Fig.  top left). Consequently, the simple RSM results in negative TRSM∗ down to -0.7 CDU (Fig. ). In contrast, TRSM∗ over northeastern Russia is quite high (> 0.5 CDU). This pattern is slightly reduced but still present in the monthly mean (see the statistics of TRSM∗ for high latitudes in Fig. ).

This artifact is largely reduced by STREAM (Fig.  top left). The spread of T∗ at high latitudes is more than 3 times lower than that of TRSM∗ (Fig. ). Also for July, systematic structures showing up in TRSM∗ (in polar regions, but also in the Indian ocean at 30–60∘ S) are largely reduced in STREAM.

Over the Pacific, TRSM∗ is, by construction, 0 on average. T∗ is systematically higher by about 0.1 CDU (Fig. ). This results from the emphasis of clouded pixels used for STREAM, which directly reflect the stratospheric rather than the total VCD. This additional advantage of STREAM over RSM is further discussed below (Sect. ).

As both RSM and STREAM generally assume stratospheric patterns of NO2 to be smooth, i.e., do not resolve longitudinal variations at all (RSM) or on scales <σφ (STREAM), the small-scale variations in daily total VCDs (Fig.  top) are transferred to the TR, resulting in “patchy” daily TRs ranging from about -0.1 up to +0.4 CDU in remote regions (10th–90th percentiles). In the monthly means, however, these patchy structures have mostly vanished (both for RSM and STREAM), as the spatial patterns of different days at variable locations cancel each other out. The remaining systematic patterns in monthly mean T∗ have generally larger spatial scales and are within 0 up to +0.25 CDU in remote regions.

On 1 July, a band of enhanced V∗ shows up around 20–30∘ S, where (a) V∗ is higher in the Indian Ocean compared to the Pacific and (b) the structure of enhanced V∗ is “tilted” in the Pacific (see Fig.  top right); i.e., the RSM assumption of zonal invariance is not fulfilled. Consequently, the RSM results in extended horizontal structures (“stripes”) of low/high-biased T∗ over South America and the Indian Ocean, respectively, ranging from -0.5 up to almost 1.0 CDU (Fig.  top right). Again, temporal averaging reduces the amplitude, but systematic patterns of about ±0.4 CDU remain in the monthly mean (Fig.  bottom right). As STREAM also assumes a weak variation of Vstrat with longitude, in particular at low latitudes, the artifacts in T∗ are very similar to those of TRSM∗ at 20–30∘ S. Note that this artifact is particularly strong in July 2005 (as compared to 2010; see Sect. ).

In Sect. , TRs from STREAM are investigated for other satellite instruments and compared to other STS algorithms, and the advantages and limitations of STREAM are discussed further.

STREAM determines the stratospheric NO2 VCD Vstrat based on weighting factors as described in Sect. . The resulting TRs thus depend on the weighting factor definition and convolution settings. We performed runs of STREAM with one-by-one modifications of each parameter and compared the results to the baseline setting. Overall, the effects of a priori settings on T∗ have been found to be rather small (of the order of 0.1 CDU), and the STREAM  results are thus robust with respect to the parameters chosen in v0.92.

Below, we summarize the main findings of the performed sensitivity studies. Figures and details are provided in the Supplement.

In order to estimate the uncertainties of the STREAM stratospheric estimate (and thus tropospheric residues), we apply the algorithm to synthetic input data, as defined in Sect. , for which the “true” stratospheric fields and TR are known. Again, a simple RSM is applied as well for comparison.

Figure  displays the statistics of the error of T∗, i.e., the difference ΔT∗ of estimated and a priori TR, which equals the difference between the true and the estimated stratospheric VCD, for different regions. The spatial patterns of ΔT∗ are shown in the Supplement (Fig. S20).

Over the Pacific, RSM results in TR biased low by 0.1 CDU. With STREAM, the bias is reduced (0.05 CDU) but not completely removed. On 1 January 2005, ΔT∗ shows a variability of almost 0.4 CDU (10th to 90th percentile) for both algorithms. This is mainly caused by the small-scale structures of stratospheric NO2 in EMAC over the Pacific, in particular at southern latitudes (see Fig. S7), which are resolved by neither STREAM nor RSM. The respective spatial variability of the monthly mean, however, is much lower (about 0.1 CDU).

Again, the simple RSM results in large biases and high variability of ΔT∗ at high latitudes, which are largely reduced by STREAM.

Overall, the agreement of a priori and estimated T∗ from STREAM is very good, in particular for monthly means. Remaining systematic biases are about -0.1 CDU over potentially polluted regions; i.e., resulting TRs are slightly underestimated, as expected due to the general approach of using total column measurements as proxy for the stratospheric estimation.

The application of STREAM to synthetic data thus provides a valuable estimate of the algorithm's accuracy. One might think that using the synthetic data for optimizing the definition of weighting factors is the next step forward. However, we refrain from doing so due to some contradictory results for different instruments. Concretely, the remaining bias in TR for synthetic data of about 0.1 CDU could be further reduced by increasing wcld. This, however, has adverse effects on SCIAMACHY and GOME results (see Sects. 5.3 and 5.4).

As shown in the previous section, STREAM as applied to OMI data generally shows a good performance (Figs.  and ). The systematic artifacts of a simple RSM, such as the large variability of T∗ at high latitudes, are largely removed by STREAM. In addition, the application of wcld emphasizes cloudy observations which directly reflect the stratospheric column. Mean T∗ over the Pacific is thus not 0 anymore as in RSM, and an additional correction for the tropospheric background is not required in STREAM.

The sensitivity of STREAM on a priori parameters has been found to be small. Remaining monthly mean TRs in clean regions and their variability are of the order of 0.1 CDU.

Below, we compare the OMI results for 2005 to other algorithms, i.e., the operational DOMINO (Sect. ) and NASA (Sect. ) data products as well as a simple model-based correction using EMAC (Sect. ). Figure  summarizes the statistics of regional T∗ from the different algorithms. Note that only coincident measurements where all four data products exist are included in Fig.  in order to allow for a meaningful comparison; in particular, high latitudes in hemispheric winter are skipped, as DOMINO data are not provided for SZA > 80∘. Thus, the statistics for STREAM slightly differ from those shown in Fig. .

STREAM has been applied to GOME-2 (Metop-A) data for the year 2010. The resulting daily and monthly mean maps are shown in Fig. S27. Again, statistical properties are summarized in Fig. .

The overall performance of STREAM, i.e., median and variability range of TR, is generally similar to that found of OMI. However, while OMI TRs are about 0.1 CDU over the Pacific, lower values (0.05 CDU) are found for GOME-2. This might be related to differences of cloud statistics due to pixel size, in particular a lower number of fully clouded pixels for GOME-2, as well as differences in local time, cloud products, or systematic spectral interferences caused by clouds in either retrieval algorithm.

On 1 July 2010, GOME-2 is operated in narrow swath mode, causing poor global coverage. This, however, does not affect STREAM performance.

On 15 January 2010, STREAM results in extraordinarily high TR over the ocean (Fig. S28), which turned out to be caused by a solar eclipse (Espenak and Anderson, 2008). Removing the affected orbit results in normal performance for this day. We recommend that screening of solar eclipses be done automatically (as done for OMI) before running any STS algorithm.

We have applied STREAM to SCIAMACHY VCDs from the MPI-C NO2 retrieval . The resulting daily and monthly mean maps for 2010 are shown in Fig. S31. Regional statistics are provided in Fig. , compared again to the simple RSM and, additionally, to the results of LNM.

Though SCIAMACHY provides poorer daily spatial coverage, STREAM overall still works well. Again, a clear reduction of the variability of T∗ is found at high latitudes as compared to RSM. Over the Pacific, mean TR from STREAM is higher than for the RSM (=0) but, similar to GOME-2, not as high as for OMI. Again, this could be related to the low number of cloudy satellite pixels and spectral interferences, affecting the DOAS analysis, related to clouds. Overall, regional statistics of T∗ are similar to OMI or GOME-2. However, a systematic latitudinal dependency of T∗ remains, showing positive values in hemispheric summer and negative values in hemispheric winter. This results from the latitudinal dependencies of V∗ being different for clouded and cloud-free observations, as shown in Fig. S33, for reasons not yet understood.

GOME was the first instrument of the investigated series of UV/vis spectrometers suited for DOAS analyses of tropospheric trace gases. The comparably small number of GOME pixels and the large across-track footprint (320 km) required a modification of STREAM: we have switched the resolution of the global grid used for weighted convolution from 1 to 5∘, i.e., wider than the GOME across-track width at moderate latitudes. Thereby it is ensured that a grid pixel usually contains multiple satellite pixels and that also adjacent grid pixels are not empty (as would be the case for an 1∘ grid), which is the prerequisite for the calculation of wTR.

We have applied STREAM to GOME data as provided by TEMIS. The resulting TRs are again compared to the simple RSM. Figure  displays the regional TR statistics for GOME in January and July 1999. The respective maps are provided in the Supplement (Fig. S34).

Overall, STREAM yields reasonable results for GOME as well. However, some systematic biases are observed:

over the Pacific, TRs from STREAM were found to be negative, which can only be explained when the measured columns for cloudy pixels are higher than for cloud-free pixels;

This explanation would be consistent with previous findings: while base the STS on cloudy pixels, switched the Heidelberg STS to cloud-free pixels after noticing that GOME columns are higher instead of lower over clouds. relate this to the contribution of lightning NOx. However, as (a) the impact of lightning NOx on satellite observations is generally small and (b) lightning activity over the remote Pacific used for the RSM is very weak, we rather suspect that a different effect is responsible for this finding, most probably related to the specific instrumental features of GOME , in particular the dichroic mirrors causing polarization dependent spectral structures. It might thus be worth re-checking the DOAS analysis of NO2 for GOME for spectral interferences related to clouds. A second possible effect, which might in particular contribute to the large discrepancy over polluted regions, is that cloud properties are averaged over the large GOME ground pixel; i.e., in an extreme case, low and high cloud layers, which would both be skipped in wcld if resolved by the satellite pixel, might yield, on average, an effective cloud height with a high wcld. Any tropospheric pollution within (or directly above) the low cloud layer would then bias high the stratospheric estimate and bias low the TR.

STREAM was successfully applied to various satellite measurements with a wide range of spatial resolution and coverage. STREAM is an MRSM and does not need any model input. It can thus be considered as a complementary approach to data assimilation, as chosen for the TROPOMI prototype algorithm.

As (M)RSMs usually estimate the stratospheric column based on total column measurements over clean regions, they generally miss the (small) tropospheric background of the order of some 0.1 CDU. Several (M)RSMs explicitly correct for this effect based on a priori tropospheric background columns . In case of STREAM, however, cloudy pixels, which allow a direct measurement of the actual stratospheric column (except for the small tropospheric column above the cloud), are emphasized. Thus, an additional tropospheric background correction should be unnecessary. Accordingly, in case of OMI, TRs from STREAM are about 0.1 CDU over clean regions, similar as for TRs from DOMINO and NASA. This is close to the a priori value chosen by but below the values given in (about 0.15–0.3 CDU, assuming a tropospheric AMF of 2) and (0.1 up to > 0.6 CDU

Note, however, that the high values are only reported at higher latitudes in winter, when the ratio Astrat/Atrop is almost 2 (Fig. S1); thus the large discrepancy is at least partly resolved when the TR is transferred in a TVCD via Eq. ().

In case of other satellite instruments, however, the TR over the Pacific was found to be lower (GOME-2 and SCIAMACHY) or even negative (GOME-1). The latter can only be explained by cloudy measurements being systematically higher than cloud-free measurements. Further investigations are needed to infer this discrepancy between OMI and GOME-1/2/SCIAMACHY and find how it is related to differences in the cloud products and/or the spectral analysis of NO2.

STREAM assumes stratospheric NO2 fields having low zonal variability, in particular at low latitudes. This is reflected by the choice of a wide convolution kernel at the equator. STREAM is thus not capable of resolving diurnal small-scale patterns caused by stratospheric dynamics. These patterns, however, largely cancel out in monthly means.

Whenever actual stratospheric fields do not match the a priori assumption of zonal smoothness, e.g., in case of “tilted” structures or actual large-scale zonal gradients like differences in the stratospheric column over Pacific and Indian ocean, the TR resulting from STREAM can show artificial “stripes”. Further investigations might lead to additional sophisticated algorithm steps to remove these artifacts. However, it has to be taken care that the benefit really outbalances the drawbacks (added complexity) and that no other artifacts/biases are introduced.

The dependencies of TR on STREAM parameter settings have been found to be low (≲ 0.1 CDU). The application of STREAM on synthetic data results in deviations to the a priori truth of the same order. These deviations are systematic, i.e., the stratospheric patterns estimated by STREAM are slightly biased high, which can be expected, as they are based on total column measurements, which are always higher than the stratospheric column.

Overall, STREAM uncertainty is well within the general uncertainties of STS (see next section). Note that systematic changes of the NO2 columns of the same order of 0.1 CDU can also result from changes of the settings for the DOAS analysis, like fit interval, inclusion of additional absorbers in the analysis, or the treatment of rotational and vibrational Raman scattering, creating overall biases as well as spatial patterns, e.g., over oligotrophic oceans (E. Peters, personal communication, 2016).

The uncertainty of STS can often not be directly quantified, as the “true” stratospheric 4-D concentration fields are not known. One approach to assess the STS performance is the usage of synthetic data, as in Sect. . In addition, the TR can be used to evaluate the plausibility of the stratospheric estimate and to derive realistic uncertainties:

Negative TRs are nonphysical. Thus, the occurrence of on average negative T∗ (exceeding the values/frequencies explainable by noise) clearly indicates a positive bias of the estimated stratosphere.

STREAM was developed as STS algorithm for NO2. However, several other trace gas satellite retrievals face problems which are similar to STS from an algorithmic point of view, i.e., that a small-scale tropospheric signal has to be separated from a smooth background (e.g., caused by stratospheric columns or, in particular in case of trace gases with low optical depth, shortcomings of the spectral analysis, introducing artificial dependencies on, for example, SZA or ozone columns). Thus, the concept of weighted convolution could be used within the satellite retrievals of, for example, SO2, BrO, HCHO, or CHOCHO, with appropriately chosen and optimized weighting factors.

Additional images, tables, and text are provided in the Supplement. All references to tables and figures in the Supplement are indicated by a prefix “S”. For readability, the Supplement is structured analogously to the paper; i.e., additional material to Sect. 2.3 can be found in Sect. S2.3 of the Supplement.