The standard operational TROPOMI NO2 data product is described in the Algorithm Theoretical Basis Document ATBD;. The Product User Manual PUM; and the Product ReadMe File PRF; describe usage of the data and the data product versions. The most recent version of these documents can be found on http://www.tropomi.eu/data-products/nitrogen-dioxide/ (last access: 7 March 2022) and on https://sentinel.esa.int/web/sentinel/technical-guides/sentinel-5p/ (last access: 7 March 2022) and includes information on earlier versions.

The NO2 data product is made by the so-called NLL2DP (“Netherlands level-2 data processor”) that provides the TROPOMI data products for which KNMI and SRON are responsible, and if either of these product algorithms is updated, the processor version is updated for all these products. In this paper product version numbers are given with two digits; in practice a third digit may be used to account for minor bug fixes.

The following is an overview of the NO2 data versions and the version of the level-1b (ir)radiance spectra (Sect. ) used as input for the retrieval as well as the diagnostic dataset (DDS; Sect. ) versions; further details are given in the ATBD and PRF.

NO2-v1.2 with level-1b v1.0 is used as of 30 April 2018, the start of the publicly released data; this version replaced all older versions, which are therefore not discussed.

Note that near-real time (NRT) data are not considered here; validation of both the offline (OFFL) and NRT data has shown that results of the two processing chains do not differ significantly .

The first step in the data processing chain is the selection of the spectral index range [ib:ie] that comprises the wavelength window [λb:λe] needed for the wavelength calibration and DOAS retrieval steps; for the NO2 SCD retrieval, λb=405 nm and λe=465 nm. The selection is done at the nominal wavelength grid assigned to the level-1b (ir)radiance spectra. For a given spectral index, i, the radiance wavelength varies across the detector rows, as illustrated in Fig. ; this is the so-called spectral smile. Consequently, each detector row has its own [ib:ie]. With λb=405 nm, for example, the level-1b v2.0 radiance nominal wavelength gives ib=36 for the rows along the swath edge and ib=24 for the central rows. Some of the rows around changes in the ib and/or ie have in v1.2–v1.4 a slightly higher SCD error estimate than neighbouring rows. This difference, which is less than 1 µmolm-2 (6×1013 molec cm-2), is reduced by two small corrections in the spectral index selection, with little to no effect on other rows.

Spectral pixels flagged in the level-1b v1.0 data as suffering from saturation or transients (or other errors) are skipped from the measurement before the spectra are used in the data processing.

Level-1b v1.0 spectra have no flagging for spectral pixels suffering from blooming (cf. Sect. ), and hence there may be many problematic spectral pixels around spectral pixels that are flagged as suffering from saturation. These spectral pixels suffering from blooming have radiance levels very different from what is expected, leading to outliers (spikes) in the DOAS fit residual, which is the difference between the measured and DOAS-modelled reflectance. Similarly, level-1b data are flagged for transients caused by charged particles hitting the detector, but not all such events constitute transients and perhaps not all transient events are captured, thus leading to possible outliers in the residual.

Since the NO2 v1.2–v1.4 processor does not have an algorithm that removes the spectral pixels that show such an outlier (“outlier removal”) from the DOAS fit, the maximum fraction of the spectral pixels within the NO2 fit window (405–465 nm, which covers 304 or 305 spectral pixels) allowed with saturation flag without skipping the ground pixels was necessarily low (Table ). With the introduction of an outlier removal routine in NO2 v2.1 (as announced by ; see also , Appendix F) and the fact that level-1b v2.0 flags the spectral pixels suffering from blooming the same way as saturated pixels (Sect. ), a larger fraction of the spectral pixels is allowed to be flagged as saturated (third column in Table ). In case of outliers in the residual of a given ground pixel caused by charged particles hitting the detector, it appears that the number of spectral pixels showing outliers is usually small (less than five), while in the case of saturation/blooming the number of outliers may be much higher. If the number of outliers is really high, the outlier removal routine may not work well, because it is applied only once , and the maximum number of allowed outliers is for the operational processing set to 10 (Table ).

Figure  shows a map of the difference “v2.1 minus v1.2” (referring to the settings listed in Table ) in the NO2 SCD error estimate (top panel) and SCD value (bottom panel) from the DOAS fit for an orbit over the South Atlantic Anomaly (SAA) using the level-1b v2.0 (ir)radiance spectra as input. There are a few along-track lines visible in Fig. : these are rows for which most v1.2 pixels have exactly one outlier at the beginning of the NO2 fit window while the corresponding v2.1 pixels have not; i.e. these lines are thus related to details of the wavelength assignment (Sect. ) in the pre-v2.1 processor. (As a reference, Fig.  shows maps of the SCD error without and with outlier removal.)

Figure  shows for the same orbit along-track averages over all 633 scanlines that have a nadir latitude within 40∘ south and north of the Equator. For the southern area outlier removal with the v2.1 settings (blue line) clearly leads to a lower SCD error (panel a) than the v1.2 settings (red line), both made using the level-1b v2.0 spectra. With level-1b v1.0 spectra the SCD error (dotted grey line) is slightly higher than with the level-1b v2.0 spectra, partly because the latter has improved flagging (Sect. ). For the northern area outlier removal has little effect on the SCD error: there is no clear difference between the green line (v1.2 settings) and the black line (v2.1 settings), again both made using the level-1b v2.0 spectra. The red and green lines, with the v1.2 settings, show a few peaks due to some strong outliers in the residuals which are not removed. The “jumps” visible at rows 21–22 and 429–430 are caused by changes in the onboard across-track binning of data. For the southern (northern) area the outlier removal removes along a given row outliers every 2–3 (5–10) scanlines.

The SCD depends strongly on the along-track and across-track variation in the solar and viewing zenith angles. To ease evaluation of the SCD, consider what could be called the geometric column density (GCD), defined as Nvgeo=Ns/Mgeo with Mgeo the geometric AMF, which depends only on the viewing angles. Outlier removal does not change the NO2 SCD itself significantly: along-track averages of the GCD (Fig. b) differ by <0.4 µmolm-2, except for pixels where (strong) outliers are removed; the average GCDs for the southern and northern areas are ∼50 and ∼40 µmolm-2 (∼3.0 and ∼2.4×1015 molec cm-2), respectively.

The NO2 v2.1 processor has been used with level-1b v1.0 spectra to provide dedicated NO2 data files for lightning NOx studies that look at the production of NO2 above bright storm clouds, where saturation/blooming may be a big issue. To this end special configuration settings, listed in the fourth column of Table  as “v2.1_test”, are used: more outliers are accepted, but the number of spectral pixels flagged for saturation is limited somewhat because level-1b v1.0 spectra lack flagging for blooming. With these special settings, more ground pixels can be used – but with great care – for such lightning NOx studies .

Figure  shows the relative changes in the main SCD retrieval results for the DDS-2 winter 2019 (top) and DDS-3 autumn 2020 (bottom) periods based on averages over the tropical latitude (TL) region used by for the evaluation of the SCD uncertainties: all scanlines with sub-satellite latitude points – corresponding to the nadir-viewing detector rows – within a 30∘ latitude range that moves along with the seasons, in an attempt to filter out most of the seasonality in the NO2 columns. Table  lists relative and absolute changes averaged over all orbits of each of the DDS-2 and DDS-3 periods.

The main SCD retrieval results shown here are the SCD value and the associated error following from the DOAS fit as well as the RMSE: the root-mean-square of the so-called fit residual, i.e. of the difference between the modelled and measured reflectance, which serves as a measure of the quality of the fit. Another such measure is the magnitude χ2, the chi-squared merit function that is minimised in the DOAS fit, which takes into account the uncertainty in the measured reflectance (the RMSE does not). Definitions and other details of the DOAS fit approach are given by and can also be found in the ATBD .

The averages in Fig.  and Table  are not an exact measure but are a good indicator of the combined impact on SCD retrieval results of the above-mentioned improvements and of the use of level-1b v2.0 spectra combined. Based on the evaluation of only 12 test orbits with the v1.2 NO2 retrieval, estimated that the update of the level-1b (ir)radiance spectra has a small impact on the NO2 SCD value, SCD error and RMSE of on average +2 %, -1 % and -6 %, respectively.

Figure  shows that the impact of the new outlier removal over the SAA, discussed in Sect. , leads to a strong decrease in the SCD error and RMSE around 30–60∘ W, in particular in the winter period (panel a), when the TL region lies just south of the Equator and thus covers a large part of the SAA; in the autumn period (panel b), the TL region lies around the Equator and overlaps less with the SAA, so that for this period the TL region average changes in the error terms are smaller. This SAA “dip” in the SCD error and RMSE leads to a somewhat larger standard deviation of the winter 2019 overall results listed in Table .

What stands out from comparing the two panels in Fig.  and the numbers given in Table  is that the change in the SCD error is very different for DDS-2 and DDS-3: in DDS-2 there is a small increase, while in DDS-3 there is a stronger decrease in the SCD error. The reason for this difference is an unfortunate bug introduced in the v2.1 processor used for DDS-2 that was repaired again in v2.2 used for DDS-3: in v2.1 there is a mistake in the calculation of the noise on the reflectance (from the noise on the (ir)radiance spectra), and this reflectance noise determines in part (i.e. scales) the magnitude of the SCD error, as well as the χ2 of the DOAS fit.

Averaging the SCD error changes in the overlapping 21 orbits of the 4 April 2019 test data shows a clear decrease of about 0.33 µmolm-2 or about 3.5 % from DDS-2 to DDS-3. Using this to correct the DDS-2 SCD error differences to the DDS-3 level leads to the numbers in the fifth and eighth columns of Table : after the correction all test data periods show more or less the same decrease in the SCD error, with some variation between the periods, which is likely caused by differences in atmospheric circumstances and remaining seasonal effects, despite the use of a moving TL region for the averaging.

The RMSE is not directly affected by the reflectance noise, and the numbers given in Table  do not show a clear difference between DDS-2 and DDS-3 (averaged over the 4 April 2019 period, the RMSE of DDS-3 is 0.2 % lower than that of DDS-3), indicating that the quality of the NO2 SCD fit has not been affected by the unfortunate bug. All DDS-2 periods have comparable RMSE decreases, with possibly a somewhat larger decrease in the autumn 2020 DDS-3 period, which may be due to a small change in the level-1b irradiance degradation correction but may also be due to atmospheric circumstances.

The SCD values themselves show an increase of 3 %–4 % for DDS-2 and about 2.5 % for DDS-3, while averaged over the 4 April 2019 test data the DDS-3 SCD values are 1.1 % lower than those of DDS-2. Again, the difference between DDS-2 and DDS-3 may be due to the small change in the irradiance degradation correction (reflectances have changed by less than 0.5 %) and/or to atmospheric circumstances.

In summary, the v2.1–v2.2 DDS data, compared to the v1.2–v1.3 OFFL data, show an improved DOAS fit quality, a reduced SCD error, and a small increase in the SCD values (Table ).

The SCD increase shows in Fig. a some east–west variation, while in Fig. b there is hardly any such variation. On the whole, it appears that the SCD increase is more or less uniform across the world, with few or no hotspots. Due to the physics of the subsequent NO2 data assimilation, the more or less homogeneous SCD increase leads to an increase in the stratospheric NO2 vertical column (Nvstrat).

The data assimilation is set up in such a way that the total column is made consistent with the TROPOMI observations over regions with small levels of air pollution (oceans, remote land regions), basically by adjusting the stratospheric column because of the minor contribution of the troposphere in those locations. A uniform increase in the TROPOMI total column will therefore lead to a similar increase in the stratospheric vertical column, while the tropospheric columns will be hardly affected.

Table  lists the relative and absolute differences of the Nvstrat averaged over the TL region using the orbits of the VCD period given in Table . For the DDS-3 period the increase in Nvstrat is somewhat less than for the four DDS-2 periods, like with the SCD values and for the same reasons.

Figure  shows as example maps over Europe of selected datasets gridded on 0.8∘×0.4∘ and averaged over the VCD part of the spring 2019 period (cf. Table ) using all pixels with a valid retrieval (qa_value >0.50). The top row shows on the left the GCD of v2.1, with clear hotspots over polluted areas, while the “DDS minus OFFL” difference on the right is fairly homogeneous. The second row shows on the left the stratospheric VCD of v2.1 and on the right the difference between the v2.1 and v1.3 data, which features a more or less homogeneous increase in the stratospheric VCD, although over some strongly polluted areas, such as Moscow and St. Petersburg, the stratospheric VCD change can be larger. As a reference, the bottom row shows the corresponding tropospheric VCD v2.1 value (left) and difference (right), with the latter also influenced by changes in the AMF discussed in Sect. .

Figure  shows a scatter plot of the v2.1 and v1.3 spring 2019 gridded average Nvstrat for all valid pixels with latitude <85∘, which shows that the worldwide average increase in Nvstrat for this period is about 0.50 µmolm-2 (0.30×1014 molec cm-2). For the other four test periods, the average increase ranges from 0.60 to 1.55 µmolm-2, all with slopes between 0.99 and 1.02. The other three DDS-2 periods have scatter similar to that seen in Fig. , while the autumn 2020 DDS-3 period has more scatter but still with high correlation (r=0.981). Scatter plots of the gridded GCD (not shown) reveal linear fits with slopes of 1.00 and offsets ranging from 0.73 to 1.04 µmolm-2, with little scatter for the four DDS-2 periods and more scatter for the DDS-3 period, all with high correlations (r>0.98). Note that linear fits mentioned in Sects.  and  are all performed with an orthogonal distance regression (ODR), i.e. taking into account that both datasets have uncertainties, rather than only the data along the y axis, while in Sect.  a different linear regression approach is used.

According to the NO2 SCDs of OMI and TROPOMI agree quite well, with TROPOMI a few percent higher than OMI, as a result of small differences in the DOAS retrieval details, and with OMI showing more scatter than TROPOMI due to its lower spatial resolution. The above-described minor changes in the TROPOMI SCD values imply that the same conclusion still holds. Linear fits in scatter plots of worldwide average gridded GCD v2.1 and OMI/QA4ECV data (not shown) for the five test VCD periods have slopes ranging from 0.98 to 1.03 and offsets between 1.27 and 2.57 µmolm-2, with high correlation (r>0.94). A more detailed slant column comparison with OMI, based on regional averages, can be seen in Fig.  (Sect. ).

The FRESCO+ algorithm retrieves cloud information from the O2 A-band around 758 nm (cloud fraction and cloud pressure) as well as scene parameters assuming clear sky (scene albedo and scene pressure) and was developed for the GOME-2 instrument. Due to the high spectral resolution of TROPOMI compared to GOME-2, the fact that TROPOMI has a spectral smile (cf. Sect. ), and because of TROPOMI's row-dependent instrument spectral response function (ISRF, known also as the slit function) with spectral shifts caused by inhomogeneous slit illumination, the FRESCO+ algorithm needed to be re-written, and the corresponding lookup tables needed to be generated once more. The resulting implementation is called FRESCO-S (short for FRESCO-Sentinel), and its cloud pressure data are used for the v1.2–v1.3 NO2 data product. Studies showed that the FRESCO-S cloud pressure is too high for some scenes, in particular for scenes with low cloud fractions and/or a considerable aerosol load (which the FRESCO algorithm sees as an effective cloud), in which case the cloud pressure is close to the surface pressure (cf. ). The consequence of this is that the tropospheric NO2 VCD is too low for these scenes, as shown also in validation comparisons (see the Introduction).

FRESCO+ makes use of the wavelength ranges 758–759, 760–761 and 765–766 nm. For the FRESCO-S implementation the first window, representing the continuum, was shifted a little to 757–758 nm. As a further improvement of the cloud retrieval, nicknamed FRESCO-wide, the third window is extended to 765–770 nm in order to include more of the weaker and narrower O2 absorption lines. This extension mainly impacts the lower clouds, generally decreasing the cloud pressure in the order of 50 hPa, and is relevant for all instruments where FRESCO has been applied. For high clouds the FRESCO versions deliver very similar cloud heights on average. Further details are given in the ATBD .

FRESCO-wide, used as of NO2 v1.4, provides a more realistic estimate of the cloud pressure for scenes with low cloud fractions: the cloud pressure is lower, i.e. the cloud is higher up, as a result of which the tropospheric AMFs decrease, which in turn leads to higher tropospheric NO2 VCDs. To a large extent, this closes the gap between the TROPOMI and validation data, though for certain cases a difference between the two datasets remains.

The FRESCO-wide approach is also used for the cloud pressure in v2.1 (DDS-2) and v2.2 (DDS-3 and its public data release) NO2 data but with the cloud data retrieved from the improved level-1b v2.0 spectra. Figure  shows the frequency distribution of the cloud pressure, cp, of a single orbit, considering only ground pixels identified as snow-/ice-free land and ocean by the snow/ice flag (cf. Sect. ), in two cloud fraction regimes: small (fc≤0.1; panel a) and medium (0.3≤fc≤0.6; panel b), with fc the cloud fraction in the NO2 fit window. This example shows that the FRESCO-wide implementation in NO2 v1.4 indeed leads to lower cp; see also the recent study by . The use of level-1b v2.0 in NO2 v2.2 has a smaller impact than the implementation of the FRESCO-wide approach for the lowest cloud fractions (panel a) and appears to lead to somewhat higher cloud pressures in that range. For the medium (panel b) and high (not shown) range cloud fractions, no change is visible due to the switch to level-1b v2.0 spectra.

Inspection of the frequency distribution of the NO2 cloud fraction (Fig. ) reveals an increase in the number of fully cloud-free pixels in v2.2 at the expense of the number of pixels with small but non-zero cloud fractions due to the use of v2.0 level-1b spectra, while the improvement of the FRESCO cloud pressure in v1.4 has no visible impact on the cloud fraction distributions.

The cloud fraction used in the AMF and VCD steps of the processing is not taken from the FRESCO data but is directly calculated in the NO2 fit window at 440 nm. This NO2 cloud fraction is determined from the surface and cloud albedo (see Sect. ), the surface pressure and the cloud pressure from FRESCO using a look-up table (LUT) that was made with a radiative transfer model (RTM), in the same manner as FRESCO determines the cloud fraction in combination with the top-of-atmosphere reflectance RTOA at 440 nm .

The procedure to determine RTOA has changed. In pre-v2.1 data it is determined by averaging the measured reflectance Rmeas over 1 nm around 440 nm. As of v2.1 it is determined as follows: RTOA=P(λ=440)⋅(1+Cring), where P(λ) is the DOAS fit polynomial and Cring is the Ring effect fit parameter, i.e. RTOA is determined from the modelled reflectance Rmod without the terms for the absorption by the trace gasses (for definitions and details of the DOAS fit, see ). This correction makes RTOA consistent with the fact that the cloud LUT was constructed without trace gas absorption. Because trace gas absorption is in general a negative contribution to Rmod, the v2.1 approach leads to somewhat higher reflectances and hence to somewhat higher cloud fractions. This impact, however, is so small that it does not really show up among the other changes that affect the NO2 cloud fraction.

It is important to have information on the presence of snow or ice in a given satellite ground pixel, so that if necessary the climatological surface albedo can be adjusted or the AMF calculation can switch from using the cloud fraction and cloud pressure to the use of the effective scene pressure and effective scene albedo (assuming fc=0), because the cloud algorithm has difficulty distinguishing clouds above snow/ice (cf. ).

To this end the v1.2–v1.4 processing uses the daily snow/ice cover database from NISE . The NISE data, however, appear to suffer from a number of problems: they have a rather coarse spatial resolution, for a given day they are based on an average over a few days, and they have problems determining snow/ice content around coastlines. The latter is in particular problematic at high latitudes where snow/ice coverage may be important, while short-term snow episodes occurring at mid latitudes may be missed due to the averaging over multiple days.

As of v2.1 the snow/ice information is taken from the daily ECMWF meteorological data, which solves the issues with NISE, thus improving the reliability of the NO2 data. The ECMWF snow data are derived from synoptic data and from the Interactive Multi-sensor Snow and Ice Mapping System (IMS); show that the IMS has better agreement with in situ observations over North America and that NISE misses a significant number of snow-covered pixels.

Figure  shows an example over Canada of the two snow/ice flag datasets, where the NISE flag numbering is used, except that the ice-free ocean has been given the colour orange (value 175) instead of red for its flag value 255, so as to distinguish it from the problematic NISE flags 252–254. The cloud fraction and cloud pressure are used for the AMF calculation for pixels flagged as ocean (255), snow-free land (000) or a percentage sea-ice flag smaller than 006 for the ECMWF data (in case of NISE data this was smaller than 002, since NISE has the tendency to underestimate snow/ice cover); in case of other flags the scene parameters are used.

Another issue solved with the switch to the ECMWF snow/ice data is that the NISE data over shallow water areas that may run dry during low tide can be wrong. Over the western part of the Waddenzee in the Netherlands, for example, NISE gives on 1 January 2019 3 % sea ice, whereas this area cannot possibly have any sea-ice: the ECMWF data correctly identify pixels as ocean (flag value zero). Because of this corrected identification, the NO2 surface albedo is adjusted from the value of 0.62 in the climatology to a more realistic 0.04. Such adjustments are made only in for cases without any snow or ice reported.

The surface albedo in the NO2 fit window, used in e.g. the computation of the cloud fraction and the AMF , is taken from the 5-year version of the OMI Lambertian-equivalent reflectivity (LER) climatology at 440 nm, which is given on a grid of 0.5∘×0.5∘, where the “mode LER” is used.

The cloud fraction, fc, is determined in the NO2 fit window at 440 nm following the same approach FRESCO uses for the cloud retrieval in the O2 A-band, with an assumed cloud albedo Ac=0.8. On physical grounds fc lies within the range [0:1]. If the actual surface albedo, As, is lower than expected from the climatology, the cloud retrieval leads to fc<0. Up to v1.4 this was clipped to zero, whereas as of v2.1 the As is adjusted (decreased) to match fc=0 and thus ensure radiative closure. Similarly, in case of very bright clouds the cloud retrieval leads to fc>1, which is no longer clipped, but instead the cloud albedo, Ac, is adjusted (increased) to ensure radiative closure with fc=1. (For details, see Appendix C.) This approach of adjusting the surface or cloud albedo to keep the cloud fraction within [0:1] was implemented in the FRESCO cloud retrieval in processor v1.3, leading to more realistic cloud pressures (see the PRF; ). With the same implementation in use for the NO2 cloud fraction, the treatment is consistent.

Figure  shows as an example a map of the difference “v2.1 minus v1.2” in the NO2 surface albedo for a part of an orbit. A lower surface albedo leads to a smaller AMF and thus to a higher tropospheric NO2 VCD. Figure  shows for the full orbit the relationship between the NO2 tropospheric VCD of v2.1 (vertical axis) and v1.2 (horizontal axis), considering only ground pixels for which the cloud retrieval gives a cloud fraction fc<0.001 (i.e. effectively zero) and thus for which the surface albedo may have been reduced in v2.1: the tropospheric VCD increases by about 15 %, as the linear fit in Fig.  shows. The increase is larger for higher VCDs: for tropospheric VCDs <100 µmolm-2 (6×1015 molec cm-2) the increase is about 10 % (linear fit: y=1.091x-0.670). This increase is partly related to the use of level-1b v2.0 (ir)radiance spectra as input in the v2.1 processing; a test processing of the orbit with v2.2 (not shown) reveals that level-1b v2.0 spectra give a tropospheric VCD that is about 5 % higher than level-1b v1.0 spectra (linear fit over all positive VCDs: y=1.053x+1.233, correlation coefficient: r=0.999). A similar increase is found when looking at the ground pixels for which the cloud radiance fraction 0.2<wc<0.5 (y=1.059x-1.135, r=0.982).

The impact of processor changes on the tropospheric VCD data is dominated by the update of the FRESCO cloud retrieval, mentioned in Sect. , as of v1.4: the other updates in the NO2 processor and the inclusion of the level-1b v2.0 spectra come on top of that. Unfortunately none of the DDS periods covers v1.4 data, which means that a differentiation of the results before and after the FRESCO-wide update is not possible without also including the other updates in the NO2 algorithm.

The bottom row of Fig.  shows example maps of the tropospheric VCD (Nvtrop) of v2.1 and the difference between DDS and OFFL data, based on all ground pixels with valid retrieval, i.e. including cloudy pixels. For a good comparison of Nvtrop data versions it is, however, better to consider only (nearly) cloud-free ground pixels with wc<0.50 (qa_value >0.75), which amounts to cloud fractions of about 0.2 and less. With this filtering and only 7 or 10 d of data for the average gridded data, the results are somewhat more noisy than those presented in Sect. . The bottom right panel of Fig.  shows some land–sea contrasts in the tropospheric VCD difference: sources of large NO2 concentrations are on land, and the difference between the two retrievals, which is chiefly caused by cloud pressure differences, scales with the NO2 column value.

Figure  shows a scatter plot of the v2.1 and v1.3 winter 2019 gridded average Nvtrop for all ground pixels with wc<0.50 and latitude ≤85∘. The period is chosen as an example because it clearly shows that a linear fit through all the data is dominated by high tropospheric columns: limiting the linear fit to Nvtrop≤100 µmolm-2 (6×1015 molec cm-2) gives a slope of 1.105 rather than 1.410. Table  lists the results for the linear fits of the five DDS periods, as in Fig. , for all Nvtrop and for Nvtrop≤100 µmolm-2.

From Fig.  and Table  it is clear that the average Nvtrop increases with the improvements in the algorithm. With this increase, the TROPOMI data lie closer to the OMI/QA4ECV tropospheric VCD, as shown in Fig.  and the last three columns of Table . To further investigate the changes in the TROPOMI data, Fig.  shows comparisons of averages over selected regions (defined in Table ) of the gridded tropospheric VCD (red solid lines) and the GCD (green dashed lines), where the TROPOMI averages are divided by the OMI/QA4ECV averages. Clearly, TROPOMI v2.x gives higher tropospheric VCDs than v1.x, in particular for the winter periods in polluted areas (upper panels in Fig. ). In most cases the TROPOMI v2.x tropospheric VCD lies closer to OMI than TROPOMI v1.2.

It should be noted here that the OMI/QA4ECV processing does not apply the albedo adjustment discussed in Sect.  to the OMI data, which means that for these cases the difference with TROPOMI data may now be underestimated. This issue does not affect the improvement of TROPOMI data with regard to ground-based measurements. (The forthcoming collection-4 OMI NO2 reprocessing will contain the albedo adjustment algorithm of TROPOMI.)

The ground-based validation of the stratospheric NO2 column data reveals an improvement in the bias, from a median difference over all co-located pairs of -0.2 Pmolec cm-2 (identical to the bias reported in , and amounting to about -6 %) for the operational OFFL data to -0.1 Pmolec cm-2 (-3 %) for the updated DDS data, which is in line with the slight increase in the TROPOMI stratospheric column mentioned in Sect. . Typically, stratospheric columns show a seasonal variation between 2 and 3 Pmolec cm-2 for nearly equatorial sites and between 1 and 6 Pmolec cm-2 for sites at very high latitudes. The updated processing does not change significantly the correlation (Pearson R) and the dispersion (half of the central 68 interpercentile, shorthand 0.5 IP68) of the difference between ground-based (“GB”) and S5P stratospheric column data and only slightly the results of a linear regression (see Fig. a), as expected from the reduced bias.

Investigating results at individual ground stations (Fig. b) shows improvements in bias (reduction of the absolute value of the median difference) at six out of nine stations, almost no change for one, and increases for the last two stations. The large bias at the Ny-Ålesund station, at about 79∘ N, is under investigation; other high-latitude stations, for which there unfortunately were no co-locations in the DDS periods, do not show such a large bias.

The comparison of TROPOMI to MAX-DOAS tropospheric NO2 column data reveals an improvement in both the bias and the dispersion. The former improves from a median difference over all co-located pairs of -1.4 Pmolec cm-2 (or about -32 % and similar to the bias reported in ) for the OFFL to -0.9 Pmolec cm-2 (-23 %) for the DDS data. The dispersion of the difference improves from 3.3 to 2.4 Pmolec cm-2. Figure a demonstrates that the linear regression also improves somewhat, with a slight increase in the slope, as expected from the improvement in the derived (multiplicative) bias.

Looking at the change in bias at each individual station (Fig. b), the DDS data show lower biases than the OFFL data at all but 2 of the 16 stations. However, results at these two outlying sites cannot be considered meaningful: they represent relatively clean background conditions with small tropospheric column values, with already very small biases in the OFFL data. It is interesting to note that the improvement does not scale with the tropospheric column value: the most polluted site does not benefit from a larger improvement.

Similar to the tropospheric column validation, the comparison of TROPOMI to PGN total column NO2 data reveals an improvement in both the bias and the dispersion. The former improves from a median difference over all co-located pairs of -0.8 Pmolec cm-2 (or about -12 %) for the OFFL data to -0.3 Pmolec cm-2 (-5 %) for the DDS data. The difference dispersion improves slightly from 2.5 to 2.3 Pmolec cm-2.

Figure a shows that the linear regression also improves somewhat, with a clear increase in the slope, as expected from the improvement in the derived (multiplicative) bias.

Looking at the bias per station (Fig. b), the situation is more complex to describe than for the tropospheric column. At relatively clean sites with small tropospheric column values, for which the OFFL data already presented slight positive biases with regard to the PGN measurements, the increased columns in the DDS data lead to even larger positive biases. The increased DDS total columns improve the bias only of those sites for which the OFFL data underestimated the PGN columns. This is also broadly true for the right-hand half of the graph, i.e. for sites with larger total columns due to a significant tropospheric contribution, in line with the findings for the MAX-DOAS comparisons (where the network is heavily biased towards sites with significant tropospheric columns).

In summary, ground-based validation of the updated DDS NO2 vertical column data, in comparison to the validation of the corresponding operational OFFL data, confirms the improvement (reduction) in the bias of the stratospheric column expected from the increase in the stratospheric column observed in the DDS data. Results presented here are consistent with what was found by in a recent study.

For the tropospheric and total NO2 columns, the dispersion is lower with the DDS data, but whether the bias improves depends on the range of tropospheric column values: at sites with large tropospheric columns, affected by strong negative biases in the OFFL data, the increased tropospheric (and total) columns imply a clear improvement. At clean background sites, however, the increased columns of the DDS data actually worsen the already positive bias of the OFFL data, a finding which is somewhat at odds with the ZSL-DOAS comparisons for the stratospheric columns, where an originally negative bias is reduced. This apparent inconsistency between direct-Sun and zenith-sky measurements was already observed in and work is ongoing to elucidate and address this, including a reprocessing of the PGN data (upcoming v1.8) with more appropriate absorption cross sections for clean sites where the total column resides mostly in the stratosphere.

Note that the two most polluted measurement sites, which show in the tropospheric (Fig. b: Vallejo) and total (Fig. b: Unam) column bias a behaviour very different from the other polluted sites, are both located on the Mexican plateau, a situation very different from the other measurement sites.

Processor version v2.2, used for DDS-3 and operational since 1 July 2021, includes in the NO2 processing chain an implementation of the O2–O2 cloud data retrieval algorithm used for OMI and described by , which is based on a DOAS SCD retrieval of the absorption feature of the O2–O2 collision complex in the wavelength window [460:490] nm. The results of this cloud retrieval are included in the standard NO2 data file but are not further used yet: they are still under evaluation for fine-tuning the algorithm settings.

The NO2 cloud (radiance) fraction is currently derived using the FRESCO cloud pressure, as mentioned in Sect. . Using the O2–O2 cloud pressure instead would mean that the cloud pressure is determined (a) from almost the same wavelengths as NO2 and (b) from measurements by the same detector, thus eliminating the small spatial mismatch between ground pixels of NO2 (band 4) and FRESCO (band 6). In addition, it seems that for certain atmospheric circumstances the O2–O2 cloud pressure may be more realistic than the FRESCO cloud pressure. included O2–O2 cloud data from OMI in their comparison of FRESCO-S and FRESCO-wide cloud pressures with those of VIIRS (the Visible/Infrared Imager/Radiometer Suite aboard the SUOMI National Polar-orbiting Partnership (SNPP) satellite): O2–O2 cloud pressures are systematically higher at low cloud fractions. Evaluation of the O2–O2 cloud data product quality is ongoing, which may lead to selection rules in the NO2 processor to choose between the two cloud pressures.

Fixes have been included in data processor version v2.3, operational as of 14 November 2021, of minor bugs related to the output of some detailed data not used by most data users (notably wavelength calibration parameters and NO2 DOAS polynomial coefficients) that were accidentally introduced in v2.2 with the inclusion of the O2–O2 cloud retrieval and which do not affect the v2.2 SCD and VCD values or quality.

The improvements in the level-1b v2 spectra (cf. Sect. ; ) include a correction for the degradation of the irradiance but not for the radiance, because at the time of delivery of the initial level-1b v2.0 CKD the accumulated degradation in the radiance was still too small to reliably determine a degradation correction for it. With a stronger effect and more radiance data available, it has become possible to determine a degradation correction, and updated CKD have been determined.

With this update a new test dataset, DDS-4, was made in autumn 2021. For NO2 the results are as expected: almost no effect on the SCD values, while the SCD error and RMSE are reduced a little due to the degradation correction – given the lack of sufficient overlap days between DDS-4 and DDS-2/3, more precise statements cannot be made as yet. Evaluation of DDS-4 was favourable for all TROPOMI data products, and hence the radiance degradation correction will be included in the operational processor, and these updated level-1b data will have version number v2.1.

Level-1b v2.1 will also include a further improvement of the flagging of transients. Analysis of the NO2 SCD retrieval results of two test orbits, over the SAA and the Pacific Ocean, reveals that this update affects only a very small number of ground pixels (<0.1 % for the SAA orbit and even less over the Pacific Ocean), for which the SCD error and RMSE of the fit may decrease.

NO2-v2.4, due for activation in the operational stream in mid-2022 and due to be used for a full mission reprocessing later in 2022, will make use of level-1b v2.1 spectra.

As mentioned in Sect. , the surface albedo in the NO2 fit window is taken from the 5-year version of the OMI LER climatology , which is given on a grid of 0.5∘×0.5∘ and measured at almost the same overpass time as TROPOMI is measuring. The OMI LER, however, does not contain NIR wavelengths, and for the FRESCO cloud retrieval the GOME-2 LER is used, which is given on a grid of 0.25∘×0.25∘ and measured at mid-morning rather than early afternoon. These climatologies are not optimal for TROPOMI, in particular in view of the spatial resolution. Furthermore, the LER approach assumes isotropic reflection of light, while in reality there is a viewing angle dependency in the reflected light (see e.g. ).

For this reason a dedicated TROPOMI surface albedo climatology has been developed, based on TROPOMI measurements, which contains both a traditional LER as well as a directionally dependent LER (DLER), similar to the one developed recently from GOME-2 measurements by , at a grid of 0.125∘×0.125∘. The initial release of this dataset is based on level-1b v1.0 spectra and as such it will be used in both the FRESCO and NO2 v2.4 operational processing and planned mission reprocessing. At a later stage, after the mission reprocessing, an update of the TROPOMI climatology will be made using level-1b v2.0 spectra. Whether and if so when that updated DLER will be implemented in the FRESCO and NO2 processing is as yet undecided.