The SCIAMACHY HDO/H2O data set and its retrieval algorithm were first described in . further improved the original data set and incorporated updated water spectroscopy. The retrieval is based on nadir short-wave infrared spectra in the 2.3 µm range (channel 8 of SCIAMACHY). In a microwindow ranging from 4212 to 4248 cm (2354 to 2374 nm) the total vertical columns of the species H2O, HDO, H218O, CO and CH4 are fitted simultaneously. The retrieval approach follows and uses an iterative maximum a posteriori (IMAP) algorithm, similar to the SCIAMACHY CH4 retrievals . No atmospheric scattering or clouds are taken into account, which means that we need to filter for clouds using a posteriori filter criteria (described below). Since the final product consists of the ratio of the retrieved total columns of HDO and H2O, certain light-path modifications due to scattering and remaining instrumental effects that modify the HDO and H2O columns in similar ways are expected to cancel out . SCIAMACHY observes backscattered sunlight that has passed through the atmosphere at least twice, which results in a high sensitivity to water vapour

in the lower troposphere where the concentration is highest. The spatial footprint of a single measurement is 120×30 km, with a local overpass time of 10:00 a.m. at the equator (mean local solar time).

As mentioned earlier, the values for δD are calculated a posteriori from the retrieved HDO and H2O columns. Although the output of the retrieval consists of total HDO and H2O columns, the algorithm assumes five retrieval layers (equidistant in pressure), of which only the bottom layer is used to fit the partial column density. Measured variations in the HDO and H2O concentrations higher in the atmosphere (where their concentration is low) will be ascribed to the lowest layer. This so-called “bottom-scaling” approach assures that the sensitivity to real changes in atmospheric HDO and H2O are comparable in the lower troposphere. We will show this in more detail in terms of the total column averaging kernels in Sect.  below, where we also show the sensitivity to variations in the choice of retrieval layers. Differences between the HDO and H2O retrieval sensitivities at higher layers, however, as well as cross-dependencies between HDO and H2O (e.g. how actual atmospheric H2O affects retrieved HDO and vice versa) could introduce issues in the derived δD if not properly corrected for by use of the full averaging kernels . Unfortunately, we did not save the full averaging kernels (including the cross-correlations) due to storage limitations. It is therefore possible that the derived δD not only reflects actual atmospheric δD but partly also actual atmospheric H2O. Besides these cross-dependency effects, which act on a layer-by-layer basis, we also expect some effects from taking column-averaged values. For example, we could observe some variability in column-averaged δD, even if the vertical distribution of δD were to remain constant, due to variability in the vertical distribution of H2O. These caveats have to be considered throughout this work (and will be addressed in a future study). These caveats also apply to the TCCON data, which will be described in Sect. , but not to the MUSICA data (which have been corrected for cross-dependencies using the full averaging kernels).

The new HDO/H2O product, from here on referred to as IMAP v2.0, has been updated with respect to the original product (IMAP v1.0, including the updates described in ). First of all, we have used consolidated level 1b files from the Instrument Processing Facility version 7.04-W. Compared to the previous version 7.03-U, the updated files incorporate improved auxiliary files, fixing incorrect in-flight calibration information. The auxiliary information is used as input in the nadc_tools instrument calibration software package, developed at SRON

http://www.sron.nl/~richardh/SciaDC/

The output format of the HDO/H2O product has been changed to the more user-friendly netCDF4 format and the filter criteria have been updated. Measurements that satisfy the following criteria are deemed suitable for further study and have received a quality flag (QF) of 1:

VCDH2O/VCDH2O,apriori>0.7

The retrieval algorithm uses variable a priori information from ECMWF for the profiles of H2O. The a priori HDO profile is derived from the H2O profile by assuming a depletion of δD=-100 ‰ at the lowest layer, increasing to δD=-500 ‰ at the highest layer. Even though the prior information is variable, the top panel of Fig.  shows that the prior column-averaged δD is roughly constant at -150‰. The small variations in δD can be explained by small variations in the scale height of H2O. A larger scale height, for example, gives more weight to the higher, and therefore more depleted, HDO layer, resulting in a slightly lower prior column-averaged δD. This also illustrates the issue mentioned in Sect.  of how variations in the vertical distribution of H2O can affect column-averaged δD even though the δD profile remains the same. The bottom panel of Fig.  shows how the prior δD has been changed by the retrieval. The retrieval of H2O and HDO introduces strong variations in the a posteriori calculated δD, which are uncorrelated to the prior information. This shows that the variations in a posteriori δD contain new information and are not simply the result of variations in the a priori H2O in combination with the fixed depletion profile. Additionally, we have tested that the a posteriori calculated δD is not very sensitive to the choice of a priori δD depletion profile by assuming a constant HDO profile with zero depletion. This resulted in some additional scatter in the a posteriori δD but without introducing any bias with respect to the standard retrieval (not shown).

Figure  (top panel) shows the impact in terms of total column averaging kernels of a bottom-scaling retrieval relative to a profile-scaling retrieval. A profile-scaling retrieval keeps the shape of the a priori H2O and HDO profiles constant by fitting a single scaling factor that applies to all five layers. This typically results in an H2O column averaging kernel being >1.0 in the bottom layer, meaning that the retrieval is too sensitive to real atmospheric H2O variations in this layer. The bottom-scaling retrieval, however, ascribes all measured H2O variations to the bottom layer and is therefore perfectly sensitive to real atmospheric H2O variations in this layer (i.e. the averaging kernel is 1.0 in this layer). Since HDO is a much weaker absorber than H2O, the HDO column-averaging kernels are practically equal for both scaling approaches and show almost no variation with altitude.

The bottom panel of Fig.  shows the impact of a profile-scaling retrieval relative to a bottom-scaling retrieval in terms of the a-posteriori-derived δD. Overall, the retrieval is quite robust against this change in the layering setup: the highest density of data points is found on the one-to-one line. The few outliers with very large shifts in δD are scenes where both retrieval setups retrieve H2O columns lower than the a priori column (not shown). The >1.0 averaging kernel for the lowest layer of the profile-scaling setup explains why this approach leads to an even lower H2O column compared to bottom-scaling (while the HDO columns remain similar). This translates into a positive shift in the ratio HDO/H2O. This is another example of how differences in the retrieval sensitivity of HDO and H2O can impact the derived δD. By using the bottom-scaling approach in our retrieval algorithm we limit this impact, since the averaging kernels for HDO and H2O are very similar in the bottom layer where most of the water vapour resides.

An offset in the humidity total column, i.e. the total column that would be measured under perfectly dry conditions, will lead to a humidity-dependent shift in δD (since a constant offset has a relatively larger impact on smaller total columns). Multiplicative correction factors for the H2O and HDO columns, however, will lead to a constant shift in δD, which is easier to correct. Correcting for an offset in the H2O and HDO columns will have a similar impact on the data set as applying a noise filter (namely larger shifts in δD for drier areas), except that it will not introduce sampling biases, since all measurements are corrected and none are rejected.

Since the existence of humidity-dependent biases can potentially hamper the correct use of the δD data set, we have tested if the SCIAMACHY δD retrievals are affected by offsets in the H2O and HDO columns and what the impact of a correction for these offsets would be on the validation of δD. found offsets in the H2O and HDO total columns from GOSAT, significantly affecting the δD retrievals. Similar to their method for estimating the offset above Antarctica, we estimate the offset by selecting all 2003–2007 data in a box above Greenland (60–90∘ latitude, -15–-60 ∘ longitude). Like Antarctica, Greenland is a high-altitude region at high latitude, which results in very dry total columns. In Fig.  we show a two-dimensional density distribution of the retrieved H2O and HDO total columns as a function of the a priori H2O total column from ECMWF. The HDO total column has already been divided by the VSMOW abundance of 3.1153×10-4. The assumption is that both the H2O and HDO columns should be zero at an a priori H2O column of zero (even though at non-zero columns the ratio between the H2O and HDO columns can be variable). The data have been filtered according to the QF=1 criteria mentioned above except for the criterion VCDH2O/VCDH2O,apriori>0.7, as this would lead to a bias against the driest H2O columns while the corresponding HDO column would not be affected. We performed a linear regression between 0 and 0.8×1022 molec cm-2 and define the offset as the Y-intercept of this regression. This gives us H2O and HDO offsets of 6.8×1020 and -5.7×1020 molec cm-2 respectively. Subtracting these offsets from the H2O and HDO columns before we calculate δD gives an offset-corrected δD, of which the world map is shown in the second panel of Fig. . The third panel shows the difference with the uncorrected world map. It clearly shows that, as expected, the offset correction has the largest impact on the driest regions, such as the Atacama desert and the Tibetan plateau. In these regions the offset correction can induce shifts in the 2003–2007 averaged δD of up to ∼ 120 ‰, while the impact at the more temperate regions is ∼ 40 ‰. As a robustness test we have also determined the offsets using data above Antarctica. This resulted in slightly larger offsets (9.5×1020 molec cm-2 for H2O and -2.4×1020 molec cm-2 for HDO) but no significant difference in the impact on δD.

It is difficult to validate the impact of the offset correction for the driest areas, as either no FTS stations exist in these regions or the stations have a very sparse sampling throughout the year and limited SCIAMACHY coverage due to their high latitudes and resulting high SZAs (such as for the dry high (ant-)arctic stations at Eureka, Ny-Ålesund and Arrival Heights). The corrected values of δD for these areas, however, are more consistent with the simulations from isotope-enabled general circulation models such as IsoGSM and LMDZ , suggesting that the world map of the offset-corrected data set is more realistic. Since it is expected that the offset correction has a relatively larger impact on the driest months in the regions where ground-based FTS data do exist, we consider both the non-corrected and the offset-corrected SCIAMACHY data in the remainder of this validation study, and we test if the offset correction leads to a better data set. In Sect.  we come to a recommendation on which data set to use.

The TCCON network consists of about 20 operational ground-based FTS stations that use direct solar spectra in the near-infrared to measure the column-averaged abundances of various atmospheric constituents, including H2O and HDO. We have used version GGG2012 of the TCCON data from the stations at Ny-Ålesund (Spitsbergen), Bremen (Germany), Park Falls (USA), Pasadena (Jet Propulsion Laboratory or JPL, USA), Darwin (Australia) and Lauder (New Zealand) (also see Table ). The TCCON measurements and their calibrations using aircraft profiles are described in detail by and, more specific to δD, by . A characterization of a posteriori calculated δD from TCCON-like measurements is presented by . The HDO and H2O columns are independently retrieved in the near-infrared wavelength region by scaling a priori profiles of the volume mixing ratio (VMR). The a priori profiles are taken from re-analysis data from the National Centers for Environmental Prediction (NCEP). The HDO a priori profile is inferred from the H2O profile with an H2O dependent fractionation (-40 ‰ at 1% H2O VMR, decreasing to -600 ‰). Since the a priori profiles are highly variable and close to the true atmospheric profiles, the H2O and HDO retrieval products will be strongly correlated with the a priori. Since δD is not retrieved directly but calculated a posteriori, the TCCON retrievals suffer from the same caveats as mentioned in Sect.  for the SCIAMACHY retrievals regarding the averaging kernels of H2O, HDO and their cross-dependencies.

Fifteen spectral windows are used for H2O and six for HDO, of which two are partly overlapping with the spectral window used for the SCIAMACHY retrievals. The spectroscopic parameters used by TCCON are different from the ones used by the SCIAMACHY retrievals. For the SCIAMACHY HDO/H2O retrievals the H2O line list compiled by was used, while for TCCON a compilation of parameters from various sources was used, where the selection was based on the performance of the spectral fit at each absorption line .

have calibrated the TCCON data using aircraft profiles, from which a correction factor of 1.031 was determined for the total column H2O. All TCCON H2O columns were already divided by this factor. Since no correction factor was applied to HDO (this could not be determined from the aircraft measurements), the H2O correction would introduce a bias in the ratio HDO/H2O. We have therefore first removed the H2O correction by multiplying the H2O columns by a factor 1.031, before deriving δD according to Eq. (). This implicitly assumes that the (spectroscopic) errors that the factor corrects for are approximately similar for HDO and H2O, so that their effect cancels once the ratio is taken. We have tested that not removing the TCCON H2O correction would lead to an increase in the SCIAMACHY δD bias of ∼30 ‰, as expected.

The average measurement precision of the a posteriori calculated δD varies between 12 and 20 ‰ for the six TCCON stations used in this study.

The ground-based component of the project MUSICA consists of several FTS stations operating within the NDACC network. From here on we refer to this data set simply as “MUSICA”. MUSICA uses high-resolution direct-solar FTS spectra in the mid-infrared to retrieve column-averaged abundances and profiles of H2O, HDO and also δD. Ten spectral microwindows are used for H2O and HDO, and the spectroscopic parameters come from a combination of HITRAN 2008 and . Contrary to the a posteriori δD calculation of TCCON, MUSICA performs a direct retrieval of δD. The a priori profile information for H2O is based on radiosonde measurements at different locations. The a priori δD profile is based on the measurements by . For every ground station the a priori information is held constant, so all the observed variability in H2O and δD is due to the measured spectra. The retrieval algorithm is described in detail by . have performed a first empirical validation of the H2O and δD products using aircraft profiles, from which a bias in δD was estimated of about 30 ‰ close to the surface, increasing to about 70 ‰ above 5 km altitude. This would translate into a high bias in the column-averaged δD of roughly 35 ‰. The measurement precision is estimated to be around 10 ‰.

We have used the 2012 version of the MUSICA data (column-averaged values for H2O and δD only) from the stations at Kiruna (Sweden), Bremen (Germany), Jungfraujoch (Switzerland), Izaña (Tenerife, Spain), Wollongong (Australia) and Lauder (New Zealand) (also see Table ). Since the MUSICA project uses data from the already existing NDACC network, time series for some stations go back to the mid-1990s, while the earliest TCCON data are from 2004. The MUSICA data therefore have more temporal overlap with the 2003–2007 SCIAMACHY data. The measurement frequency, however, is lower for MUSICA, which means that per overpassing SCIAMACHY measurement the number of co-located FTS measurements is also significantly lower.

Figures  and show the entire 2003–2007 time series of δD measurements from TCCON and MUSICA respectively (dark grey dots and magenta curve) together with all the spatially co-located SCIAMACHY δD measurements (light grey dots and blue curve). To clearly show all the available data per station, no temporal constraints have been applied to the data points in these figures. A few stations show very sparse sampling throughout the year, either due to their high latitude and resulting high SZAs (e.g. Ny-Ålesund and Kiruna) or due to their location close to oceans which have a very low surface albedo (Lauder). The MUSICA stations Jungfraujoch and Izaña are high-altitude stations and will therefore measure a lower column-averaged δD as a result of δD decreasing with altitude (the so-called altitude effect). The co-located SCIAMACHY measurements for these high-altitude stations consist mostly of reflected sunlight from nearby lower surfaces, which results in the observed strong positive bias in δD.

For every station we have determined various statistics, which are printed at the top and bottom of the figures and will be explained below. These statistics are based on the spatially and temporally co-located data points from the algorithm as described in Sect. . A summary of the statistics is given in Table , including the values before and after correcting for the offset.

As mentioned above, the averaged bias with respect to MUSICA is larger than with respect to TCCON, while for the two individual stations that participate in both networks the bias with respect to MUSICA is either consistent with TCCON, or even smaller. Since the MUSICA and TCCON stations are spread across different geographical locations, this could be the result of a selection effect in combination with a latitudinal bias.

In Fig.  we show the bias at all the FTS stations (except the two high-altitude stations) as a function of latitude. For the two stations participating in both networks (Bremen and Lauder), the weighted average of their biases is used. The figure shows evidence for a latitudinal dependent bias both for the offset-corrected and uncorrected data set. The bias at the higher latitudes can be up to 30 to 60 ‰ larger than at the lower latitudes. Although the offset correction leads to a larger shift in δD at higher latitudes, this gradient is not sufficient to remove the latitudinal dependency of the bias.

Three of the four stations at low to moderate latitudes (-40 to +50∘) are TCCON stations with relatively small error bars. The only MUSICA station in this latitudinal range is Wollongong which has the largest error bar on the bias. The other three MUSICA stations are at higher latitudes where the SCIAMACHY δD bias is larger. This explains why the weighted mean bias over all stations is smallest with respect to TCCON, even though the bias at Lauder and Bremen is either comparable or smaller with respect to MUSICA. It also shows that, in order to accurately determine a latitudinal bias of SCIAMACHY δD using the combination of both TCCON and MUSICA networks, more information on the differences between TCCON and MUSICA δD is necessary, including more elaborate cross-validations and their latitudinal dependencies.

We show monthly means for all stations in Figs.  and . Since we now focus on the mean seasonality, we have relaxed the temporal co-location constraint of matching every FTS measurement within ±2 h to a SCIAMACHY measurement. Instead, we selected all available FTS measurements between 2003 and 2012, which were taken in a ±2 h window around the mean SCIAMACHY overpass time. The mean overpass time for a specific ground station was estimated from all available 2003–2007 SCIAMACHY measurements within a 500 km radius of the station. This results in much more available data and therefore more accurate monthly means, while still sampling the same window in the diel humidity cycle. For SCIAMACHY all available measurements between 2003 and 2007 within a 500 km radius of the FTS station were used, and we required a minimum of 10 measurements for any month to be used as a monthly mean.

Figures  and show that the SCIAMACHY δD measurements reproduce the shape of the mean seasonalities quite well for most ground stations. This is also visible in the correlation coefficients, which are printed in the figures for both the offset-corrected (rcorr) and uncorrected (runcorr) SCIAMACHY data.

As a reference, we have also plotted the seasonalities of the a priori δD as dashed curves. In contrast to MUSICA (which uses a single fixed a priori assumption per station), the TCCON a priori is variable and highly correlated with the posterior δD seasonality. The SCIAMACHY a priori δD, however, is fairly constant at about -150 ‰. We can therefore conclude that the correlation between the SCIAMACHY and FTS δD seasonalities is a result of the retrieval process and not of the a priori assumptions.

For Ny-Ålesund and Lauder (and to a lesser extent Kiruna) the similarity in δD seasonality is not good and the correlation coefficient is even negative. These are the stations with the poorest sampling of SCIAMACHY measurements throughout the year, either due to the high latitude (Ny-Ålesund, Kiruna) or isolated position surrounded by oceans (Lauder), making their seasonalities incomplete.

Due to the averaging, the correlation coefficients for the monthly means are much higher than the correlation coefficients for the single co-located observations (as were shown in Figs.  and and Table ). The offset correction, however, generally leads to lower correlation coefficients. This seems to be caused by overcorrecting some of the driest months with the largest uncertainties, such as winter for Park Falls and spring for Ny-Ålesund and Kiruna. The seasonalities for those stations match better without the offset correction. So even though the offset correction leads to an overall smaller bias in δD and more realistic global patterns for the driest areas (such as the Tibetan plateau, see Fig. ), it slightly deteriorates the shape of local seasonalities.

Interestingly, the shape of the seasonality for the high-altitude Izaña station is reproduced quite well (except for the negative shift due to the altitude effect), even though many SCIAMACHY observations for that station are measured above the Sahara. In contrast, the seasonality measured from the top of the Jungfraujoch mountain is much steeper than observed by SCIAMACHY, possibly related to the higher altitude of Jungfraujoch (compared to Izaña) or differences between Central Europe and the subtropics in the dominating circulation patterns that control moisture transport at high altitudes.

Finally, it has been suggested that SCIAMACHY might overestimate seasonalities in δD due to a selectional bias in the observations of high altitude scenes due to fractional cloud cover . We do not see any evidence for such overestimation of the seasonalities from Figs.  and , although we do acknowledge that fractional cloud cover might play a role in the observed overall low bias in δD, since small fractions of clouds will reflect signal of higher altitude atmospheric layers (depleted in δD) into the light path.

To study if δD derived from the SCIAMACHY measurements can also add information with respect to humidity retrievals alone, we show the monthly mean δD as a function of humidity in Figs.  and . As for the monthly means, all available years were used for the FTS data (within ±2 h of the mean SCIAMACHY overpass time), while for SCIAMACHY we used the years 2003–2007 (offset corrected). For both FTS and SCIAMACHY we only use months with at least 10 measurements. Figure  shows a close-up for the TCCON stations Park Falls and Darwin, including the SCIAMACHY and TCCON a priori assumptions (blue and magenta dashed curves). Figure  shows a close-up for Bremen and Lauder, which participate in both TCCON and MUSICA networks.

Following , by numbering and connecting the months in sequence, the transition of δD and humidity throughout the year becomes visible. We find significant differences between the transition from winter to summer and summer to winter for most FTS stations. The TCCON station Darwin is the clearest example (bottom panel in Fig. ), showing lower δD values for the transition from January to July than for the transition from July to January, even though the humidities during these transitions are the same. All FTS stations seem to suggest the same pattern of higher δD in spring and lower δD in fall, and this is also observed by SCIAMACHY above the stations of Bremen, Park Falls, Darwin, Jungfraujoch, Izaña and Wollongong. For reasons unknown, JPL is the only station for which SCIAMACHY observes a reversed pattern compared to FTS of lower δD values from April to June than from September to November. Nevertheless, the general behaviour of asymmetrical spring and fall transitions is captured fairly well. These asymmetries between the seasons point to differences in the distance or temperature of the source of the water vapour and show that the SCIAMACHY δD measurements add additional information to the measurements of humidity alone.

The FTS stations of Jungfraujoch and Izaña show much more depleted δD and humidities than SCIAMACHY due to their high altitude, but the SCIAMACHY observations seem to correctly connect to their Rayleigh curves, including the decreasing slope towards higher humidities. The SCIAMACHY observations above the stations Ny-Ålesund, Kiruna and Lauder show incomplete seasonal cycles due to their geographical locations and are therefore less useful to study the seasonal asymmetries. Also, the first δD observations above these stations in local spring are all overestimated compared to observations later in the season (also visible in the monthly means in Figs.  and ). This leads to the negative correlation coefficients in the monthly means and the wrong orientation (too low or even negative slopes) of the δD vs. humidity curves. It is possible that these overestimations are caused by sampling biases due to the high latitudes. Within the 500 km co-location radius, the lower latitudes will generally have higher humidities with higher δD. Early in the season these lower latitudes will preferentially be sampled by SCIAMACHY due to their lower SZAs and therefore stronger signals. Such a sampling bias could also explain why SCIAMACHY overestimates the lowest humidities (and δD) for Park Falls. SCIAMACHY's underestimation of the very high humidities measured at Darwin, however, might result from a sampling bias against very cloudy and humid conditions in local summer. While the FTS instruments can measure under such conditions in between the clouds, SCIAMACHY needs less cloud-contaminated (and thus less humid) conditions for a measurement to successfully pass the filter criteria.

It is important to note that the seasonal asymmetry is not present yet in the a priori information. Figures  and show that the prior SCIAMACHY δD vs. humidity curve is constant at about -150 ‰. Although the prior TCCON δD varies throughout the season depending on humidity, there is no clear difference between the spring and fall transition yet. This shows that the asymmetry of both SCIAMACHY and TCCON is a result of the retrieval and that their posterior correlation has been improved compared to their a priori correlation. The same holds for MUSICA, as the a priori total column δD is constant for every station and the a priori total column humidity varies only slightly with surface pressure. Figure  shows that the posterior δD vs. humidity curves are very similar for TCCON and MUSICA, even though their prior information is completely different. The TCCON curves are correlated with the prior information, but especially for Lauder it can be seen that the retrieval has added information, enhancing the seasonal asymmetry and bringing it very close to the asymmetry as seen by MUSICA. For example, the spring transition in Lauder (months 8–12) shows lower δD than the fall transition in the a priori, while this pattern is reversed by the retrieval.

A simultaneous validation in two dimensions (δD and humidity) introduces the complication of possible biases in two dimensions as well. For the SCIAMACHY humidity measurements we have used the total column H2O retrievals. Since these are not ratio retrievals, certain instrumental artefacts and light path modifications (such as scattering from the clouds remaining after filtering) might not cancel out. As a very simple humidity validation exercise we have compared the retrieved H2O columns with the H2O columns from ECMWF in Fig. . The figure shows a high correlation (r=0.96) but also somewhat underestimated columns (the linear regression has a slope of 0.87). The H2O profiles from ECMWF are also used as a priori information for the H2O retrievals. This explains why in the δD vs. humidity diagrams the retrieved SCIAMACHY humidities remain close to their a priori. A similar high correlation between prior and posterior H2O holds for TCCON but not for MUSICA, as the latter uses a priori H2O information based on a constant mixing ratio. For SCIAMACHY and TCCON, the slope in the δD vs. humidity diagram could therefore easily be affected by the choice of the H2O prior. The retrieval of HDO, however, can add significant information independently of its prior assumptions (this was also shown by Fig.  in Sect. ). The seasonal asymmetry introduced by this information is therefore a more robust metric of hydrological changes than the slope of the curve itself.