We address the application of differential optical absorption spectroscopy (DOAS) of scattered light observations in the presence of strong absorbers (in particular ozone), for which the absorption optical depth is a non-linear function of the trace gas concentration. This is the case because Beer–Lambert law generally does not hold for scattered light measurements due to many light paths contributing to the measurement. While in many cases linear approximation can be made, for scenarios with strong absorptions non-linear effects cannot always be neglected. This is especially the case for observation geometries, for which the light contributing to the measurement is crossing the atmosphere under spatially well-separated paths differing strongly in length and location, like in limb geometry. In these cases, often full retrieval algorithms are applied to address the non-linearities, requiring iterative forward modelling of absorption spectra involving time-consuming wavelength-by-wavelength radiative transfer modelling. In this study, we propose to describe the non-linear effects by additional sensitivity parameters that can be used e.g. to build up a lookup table. Together with widely used box air mass factors (effective light paths) describing the linear response to the increase in the trace gas amount, the higher-order sensitivity parameters eliminate the need for repeating the radiative transfer modelling when modifying the absorption scenario even in the presence of a strong absorption background. While the higher-order absorption structures can be described as separate fit parameters in the spectral analysis (so-called DOAS fit), in practice their quantitative evaluation requires good measurement quality (typically better than that available from current measurements). Therefore, we introduce an iterative retrieval algorithm correcting for the higher-order absorption structures not yet considered in the DOAS fit as well as the absorption dependence on temperature and scattering processes.

Differential optical absorption spectroscopy (DOAS)

However, to be not only more time efficient (by performing RTM calculations at
only one or a limited number of wavelengths) but also more independent from
a priori constraints, one might wish to quantitatively characterize the
influence on the retrieval caused by strong absorption. One possibility would
be to introduce additional functionals quantifying the non-linearity effects,
besides the already well-known conventional box AMFs. The
box AMFs quantify the linear influence of the trace gas absorption
coefficient on the logarithm of the measured intensity; they describe the
effective light paths through a given atmospheric region (also called box or
voxel) normalized by the box spatial extent. Here, we parametrize the
contribution to OD by non-linearity effects by introducing higher-order
box AMFs or higher-order effective light paths. The

The consideration of the higher-order contributions allows us to implement a two-step algorithm (separating the spectral retrieval part from the spatial
retrieval part) even for strong absorption scenarios

The article is structured as follows: in Sect.

The intensity (in our study the Sun normalized spectral radiance) as observed
by a detector depends on the paths the photons from the Sun took in the
atmosphere until reaching the detector, each with a certain probability to be
observed by the detector. The intensity is basically the mean of the
probability of the light paths after entering the atmosphere per element of
the area, element of time, and element of wavelength to enter an element of
the detector's aperture solid angle. The path integral formulation of the
radiative transfer equation

The weight

One can modify the weight

Since the Beer–Lambert law does not hold for a multiple light path ensemble, only its approximation can be applied for scattered light observations. Standard DOAS applications use its approximation up to the first order. However, for strong absorption scenarios the first-order approximation is usually not applicable, which motivates us to investigate the use of higher-order terms.

Expanding the OD

Here,

A sketch of light paths (with corresponding weights) crossing two
boxes (layers) in limb geometry for the illustration of the variables in
Eqs. (

Normalized by the box dimensions (usually by the height) a dimensionless
functional is obtained: the well-known box AMF

However, the higher-order OD terms of Eq. (

Figure

In this case the contributions to the observed intensity by the second-order
terms like in Eq. (

The third-order effective light paths are

They are weighted products of the individual light paths through three boxes and quantify the dependence of the sensitivity between three locations in atmosphere; i.e. they account for all light paths which cross the given three boxes.

In general, any order effective light paths can be defined, if the Taylor
series expansion of Eq. (

Note that similar to the effective light paths

In analogy with Eq. (

In the next subsections the validity of different approximations (linear or higher-order terms) in DOAS is assessed.

In order to get a DOAS-like equation

In the following we skip explicitly mentioning the dependence of wavelength for better readability.

The wavelength dependence of the scattering term

Since the Taylor series expansion is performed at the absorber concentration
of 0, the SCD in the formula represents the effective light path for an
atmosphere without absorption. Here it is important to note that for strong
absorption scenarios the retrieved SCDs as well as ODs are
systematically underestimated by this linear model derived at a zero
absorption background, because longer light paths contributing to the
measurement become less probable. This leads to the fact that a fit by a
linear model will distribute the higher-order absorption structures (with a
mostly negative contribution by the second-order terms, see also the
discussion later in Sect.

In order to correct for both the systematic underestimation of the SCDs and
to minimize the systematic biases due to higher-order absorption structures,
the Taylor series approach in

Thus the OD terms for absorber

where

This modification for a strong absorber (ozone) in the UV largely decreased the fit residual structures. However, at the same time a systematic bias for weak absorbers still remained (see Figs. 12 and 13 therein) because the method does not account for cross-correlative terms (see also Sect. 5.1 where we evaluate the method in sensitivity studies).

Besides the significant improvements, the method relays on a priori considerations when performing radiative transfer simulations for the spatial evaluation. Effective light paths are evaluated at a background a priori scenario at an empirically selected wavelength (where the discrepancy with the true profile as determined in sensitivity studies is minimum).

Another possibility to address the problem of scattered light observations
with a linear method is the so-called AMF modified DOAS

Instead of limiting the DOAS fit to the linear approximation (standard DOAS)
or a qualitative consideration of the variation of the SCD due to scattering
and second-order self-correlative absorption effects

Like for the approximation of the wavelength dependence of

Here

Considering only the

The retrieved SCD coefficients are quantitatively associated with the trace
gas concentration by

The equation allows to investigate the relationship between the SCD and the
spatial distribution of an absorber at any wavelength. Additionally, it is
worth noting that the SCDs correspond to an atmosphere without absorption,
so theoretically it is possible to calculate the effective light paths

From these two equations it can be basically concluded that additional information for the spatial evaluation can be derived by fitting higher-order terms, because different-order effective light paths have different spatial distributions, as discussed later. Note, however, that in this paper the content of this additional information is not explicitly investigated because (a) a non-linear inversion algorithm would be needed and (b) the high correlations between the derived SCDs of different orders, especially when they correspond to the same trace gas. For accurate fit results, in general a very good measurement quality (e.g. signal-to-noise ratio) would be needed and, depending on the trace gases of interest and the measurement quality, fit components with a significant contribution to the fit results should be considered.

In order to account for the effects caused by strong absorption scenarios,
let us consider the higher-order Taylor series terms of the expansion in
Eq. (

The second-order Taylor series term for any two absorbers

Assuming that the cross section is invariant in space (i.e. neglecting its
variation due to temperature and pressure), the single trace gas term for any
of the trace gases is

Including the squared cross section in the DOAS fit

The second-order Taylor series expansion term for the interdependence between
two absorbers

Note the similar structure with Eq. (

As an example, the Taylor series terms of different orders for a typical
scenario with ozone (strong absorber, vertical column

The lines in Fig.

The second-order effective light paths (Eq.

Let us analyse the difference between the second-order effective light paths
and the products of the corresponding (first-order) effective light paths (as
in Eqs.

For illustration, first-order effective light paths and their product between
different boxes are compared with second-order effective light paths in
Fig.

Continuing analysing the Taylor series expansion of
Eq. (

From Fig.

In this section we discuss the applicability of fundamental concepts of DOAS for scattered light observations. We show that these fundamental concepts are not valid in a strict sense.

While the basic consequence of these findings is a possible bias of the results of the retrieval algorithms, in practice these aspects are specifically relevant only for scenarios with strong non-linearities due to absorption. For most of the current DOAS applications these considerations are not relevant since they are performed in the (almost) linear range.

The fundamental assumption by the standard DOAS is that it is possible to
distinguish between absorption features of different trace gases; i.e. the
total atmospheric absorption

In order to improve the fit results sometimes differential cross sections are applied to better differentiate between narrow and broadband spectral features, but it is always assumed that each trace gas has its own “fingerprint” completely originating from the contribution of this absorber.

Looking at the higher-order terms discussed before, it becomes clear that
this assumption (Eq.

It can in particular be shown that the definitions for optical quantities (OD, SCD, and AMF) used in DOAS are affected. In the following subsections we discuss two commonly used definitions: the classical definition and the light path integral definition. They are introduced in the following subsections.

The classical definition is used by e.g.

The OD

This means that the OD of two absorbers (or any number of absorbers) is not equal to the
sum of the individual ODs of these absorbers, since the cross-correlation
between different absorbers is not considered. Consequently also the
definitions of the AMF

Difference between the sum of the ODs of the considered absorbers,
each calculated by the classical formula, and the OD calculated for the whole
absorption (red crosses), in comparison with the sum of the inter-correlative
terms (blue line: sum of second-order terms only; green line: sum of second- and
third-order terms). Top panel: the result in the UV for the limb observation
scenario with TH

Figure

The result in Fig.

The classical OD thus consists of the absorption terms attributed to the
absorber

Applying Eq. (

Alternatively, according to the LPI definition, the SCD
is defined as concentration integrated along the effective light path

The corresponding definition of the OD would be the integration of the
absorption coefficient along the effective light path:

Please note that

Also for the LPI definition the optical quantities can be expressed in terms
of the Taylor series expansion. Expanding effective light paths in
Eq. (

Please notice that the summation over

While the application of the LPI definition has been limited for retrievals
of weak absorbers

In summary, both the classical and LPI definitions of the OD are not
applicable to describe the total OD as the sum of the ODs of individual
absorbers. In order to obtain an agreement between the sum of the
approximations of the ODs

Therefore, the approximation of the OD in Eq. (

The effective light paths defined earlier (Eqs.

Effective light paths for limb geometry (TH

If the Taylor series expansion of Eq. (

By performing the Taylor series expansion at a zero absorption background,
effective light paths at an arbitrary absorption background can be expressed
as a function of higher-order effective light paths (simulated for the zero
background scenario) and the trace gas distributions. Assuming spatially
constant cross sections, the first-order effective light path (as in
Eq.

Here, trace gas concentrations

Such a parametrization is practical, since it can be used for any wavelength
chosen for the procedure described in Sect.

Figure

If the true cross-section

DOAS evaluation of a simulated limb spectrum at TH

In a similar way, i.e. by considering the cross-section ratio, one can also include modifications in all other formulas containing products of cross section and concentration.

In comparison to standard DOAS, Taylor series modified DOAS

The OD for two absorbers considering terms up to the second order can be
written as

Standard DOAS accounts only for the terms 1 and 2. Taylor series DOAS as
discussed in

Fit parameters included in the DOAS evaluation of synthetic spectra in the spectral range 520–570 nm.

As an example, a DOAS evaluation for a simulated limb spectrum at
TH

Mean and standard deviation of the fitted OD of the first-order
NO

To illustrate the consequences if the cross-correlative term between ozone
NO

Due to the high cross-correlation between the different-order terms, the retrieval quality, among others, depends on the way this correlation is accounted for. There are several possibilities, including the following.

Results of the Taylor series DOAS evaluation either with or without
the cross-correlative ozone and NO

A quasi-full retrieval approach: the absorption OD is calculated for
each considered spectral point according to the higher-order Taylor series
expansion, and the forward model is linearized at a certain a priori
absorption scenario first. Then an iterative least squares fitting technique
is applied, improving the knowledge about the trace gas distributions and
adjusting the forward model according to the improved knowledge using the
effective light paths of different orders. This method must be the most
accurate but also the most time consuming as the calculations are performed
for every spectral point and the inversion is performed for all spectral
points and measurement geometries together. However, it will be still faster than
the classical full retrieval approach

A quasi-AMF modified DOAS: other than the method by

A two-step approach: in the first step, a DOAS fit is performed including all relevant higher-order terms, especially the inter-correlative terms. In the second step (spatial retrieval) a retrieval procedure is performed at a selected wavelength (e.g. a wavelength in the centre of the spectral region or where the differential absorption structures are largest or the absorption effects smallest; with a compromise between all these aspects). Further, there are two possibilities: (a) to use only the first-order terms which describe the absorption as there would be no influence of absorption by the strong absorber (the absorption is sorted out by the cross-correlative term) or (b) to perform a non-linear spatial retrieval including the cross-correlative terms of the absorber of interest and the strong absorber. The possible correlation between the different-order terms of the same absorber, however, needs to be approached iteratively and at the end a balance between (a) and (b) should be found. This method would be faster than the first two, because the effective light path calculations are limited to one or few spectral points. At the same time, the algorithm would be less accurate due to the correlations between the first-order and second-order terms in the DOAS fit, or, in other words, it requires better measurement quality.

While the implementation of the algorithms mentioned in
Sect.

First, a DOAS fit is performed for the simulated spectra similarly as it is
done by the non-iterative algorithm described in

Settings of the iterative two-step algorithm for simulation studies.

Effective light paths of the first- and second-order pre-calculated for zero
absorption are used to calculate first-order effective light paths for the
profiles retrieved in the previous iteration (for the initial iteration for
the a priori data) at the selected wavelength according to Eq. (

The profile is retrieved according to the Gauss–Newton optimal estimation
scheme

In further iterations, absorption spectra based on the first- and second-order
absorption terms according to Eq. (

We apply the iterative two-step algorithm to absorption spectra simulated at
different THs to investigate its ability to reproduce the true profiles used
for the simulated spectra and compare it with the linear algorithm where
(1) no iteration for spectral correction is applied and (2) a linear forward
relationship between the a priori and OD is assumed. The simulation is
performed for the same scenario (limb observation in the subarctic atmosphere
in spring) as for previous studies; i.e. while the spatial retrieval is
performed similarly as the initial step of the iterative algorithm applying
Eq. (

Residual and fitted ODs of the absorbers obtained by the iterative
two-step approach in the UV. The spectra are simulated for the same scenario
(limb observation in subarctic atmosphere in spring) as for previous figures.
Left panel: initial DOAS fit for a spectrum at TH

Figures

Same as Fig.

Odd plots (from left for ozone, NO

The obtained vertical concentration profiles are plotted together with the
true profiles in Fig.

In the VIS spectral range, much larger ozone absorption occurs, increasing
gradually with wavelength. Thus it is not possible to select a wavelength
with low ozone absorption for the spatial evaluation. Therefore an
overestimation of the retrieved profiles (see the two plots on the left in
the bottom panel) by the linear retrieval of up to 16 % is observed
because the effective light path integral formalism for the OD used in the
retrieval set-up overestimates the higher-order contributions to the
absorption OD (see Sect.

From left: ozone, NO

For weak absorbers like NO

The BrO retrieval in the UV (right two plots on the top) shows similar
findings as the NO

Same as upper panel of Fig.

As stressed in Sect.

Same as Fig.

We investigate the stability of the iterative retrieval if the spatial
retrieval is performed at different wavelengths. Figure

Same as Fig.

Same as Fig.

Figure

For strongly non-linear problems, and if the a priori is too far away from
the true profiles, the results during the iterative retrieval might converge
towards profiles that differ from the truth. Applying the standard DOAS fit
(i.e. skipping higher-order terms), much larger discrepancies are found even
after the iterative process (Figs.

Settings of the iterative two-step algorithm for the application to SCIAMACHY limb observations in the UV.

In the following we apply the two-step iterative algorithm introduced in
Sect. 5.3 for the retrieval of trace gas profiles from SCIAMACHY limb
measurements. We investigate the response to different retrieval settings
(like the cross-section dependence on temperature or the consideration of
the cross-correlative terms). We also compare the retrieved profiles with
correlated balloon measurements provided by

The SCIAMACHY instrument

Comparison of the current retrieval algorithm implementation for BrO
with the algorithm in

Continued.

The retrieval of vertical profiles from SCIAMACHY measurements is performed
in a similar way as for the studies with simulated spectra above. The
retrieval scheme is based on the linear two-step retrieval algorithm
developed in our group

use of the more recent temperature-dependent ozone cross sections from

the measured spectra are gridded to a TH grid of 3 km;

spectra at the same TH (across track) are averaged;

a Sun spectrum (A0 spectrum in the SCIAMACHY data set) is used as a reference spectrum to improve the signal to noise ratio in the fit – in order to correct for possible offsets in the retrieved quantities caused by unexplained spectral structures in the Sun spectrum, the absorber ODs are corrected (by subtraction) by the ODs retrieved at high TH;

the spectral shift is linearized and the shift between the reference and
the analysed spectra is fitted as additional fit parameter according to the
method in

to account for the spectrally varying polarization sensitivity of the SCIAMACHY instrument eta and zeta spectra are orthogonalized and the spectra corresponding to the largest eigenvalues are considered in the fit.

The settings for the spatial evaluation and the calculation of the correction spectra are similar to for the sensitivity studies, but in addition cross sections at all available temperatures are considered for the calculation of correction spectra and effective light paths.

The most important modifications in the spatial retrieval step include the
following (see Table

The vertical profiles are now obtained on a 1 km retrieval grid.

Instead of the SCD, the OD is used for the inversion allowing the consideration of the temperature dependence of the cross sections.

Weighting functions (i.e. the product of the effective light paths and the
cross sections) are calculated at 342 nm (outside the strong ozone
absorption band) while in

Effective light paths are approximated by a broadband function from the
first- and second-order effective light paths obtained at zero absorption and
at selected wavelengths. The absorption contribution is approximated up to
the second order, while in

Although the modifications are discussed in the scope of the iterative
algorithm, the retrieval with the linear approach used in the comparison plots
in the next subsection also includes all respective modifications while
omitting the iterative process. Consequently the results are not the same as
presented in

Before discussing the retrieved profiles by the new iterative algorithm (next
subsection) we provide evidence for higher-order absorption structures in the
measured spectrum. Figure

Difference between residuals of the standard DOAS fit and the fit
with higher-order ozone terms included according to

We retrieve vertical concentration profiles by the iterative algorithm and
the linear algorithm and analyse the effect of considering the temperature
dependence of the cross sections or the second-order cross-correlative terms.
We compare the vertical profiles retrieved from SCIAMACHY measurements with
collocated LPMA/DOAS balloon observations of direct sunlight

Comparison of SCIAMACHY BrO profiles retrieved by the linear
approach (green), iterative approach (red), iterative approach without
considering the temperature dependence of the cross sections (blue), and
iterative approach without considering the cross-correlative terms in the
calculation of the correction spectra (cyan) with collocated balloon
observations. In order to match the SZA of the SCIAMACHY measurement the
balloon results are photochemically corrected

Same as Fig.

An agreement between SCIAMACHY and the balloon profiles (within the error
bars) can be seen for most of the profiles and altitudes, which fulfil the
collocation criterion (indicated by the yellow shading). The profiles
obtained by the linear retrieval are systematically lower than presented in
the retrieval comparison by

For scattered light observations the Beer–Lambert law generally does not hold due to multiple light paths contributing to the measurement. In the presence of strong absorption and a strong wavelength dependence of the light scattering probability (characteristic for complex viewing geometries like satellite limb measurements) the commonly used linear approximation between the trace gas concentrations and the logarithm of the measured intensity does not hold anymore. The non-linearity is especially strong when large wavelength intervals are used in the spectral analysis. Therefore algorithms based on the DOAS technique are especially affected. In essence, basic quantities like the OD or the SCD cannot be unequivocally defined anymore. Even with modifications, such as including additional fit terms accounting for the effects of strong absorbers, this problem cannot be solved completely because of remaining cross-correlative structures between weak and strong absorbers or remaining structures due to the broadband variability on wavelength. While we show that, in principle, the higher-order cross-correlative structures can be considered in the DOAS fit as additional fit parameters, to obtain meaningful results a good signal-to-noise ratio is required to minimize the cross-correlation between the fitted parameters.

To avoid the need to apply a full retrieval algorithm requiring time-consuming, iterative wavelength-by-wavelength forward modelling of absorption spectra, we introduce an iterative two-step retrieval algorithm separating the spectroscopic and spatial retrieval steps. From the profiles retrieved in a previous step, corrections for the measured spectra are calculated. After they are applied to the measured spectra, the next iteration is performed. Usually, convergence is found after 2–3 iterations. The correction spectra are calculated based on the a priori profiles or the profiles derived in the respective iteration. The calculation is based on RTM calculations at only a few selected wavelengths for an atmosphere with zero absorption, which strongly reduces the calculation time. Also higher-order absorption effects and the temperature dependence of the cross sections is considered. The quantitative definition of higher-order absorption effects can be obtained by expanding the radiative transfer equation into higher-order Taylor series. Higher-order effective light paths are calculated as weighted mean products of the light paths through several boxes. By approximating the wavelength dependence of the first- and higher-order effective light paths derived from pre-calculated RTM simulations at several wavelengths by a broadband function, we calculate absorption spectra and first-order effective light paths for any atmospheric trace gas composition and for any wavelength within the fit range, thus eliminating the need for RTM simulations during the retrieval process.

We find a strong influence by second-order absorption structures for
retrievals from measurements in limb geometry, which often exceed the
contributions by minor absorbers (ODs

By applying the iterative two-step algorithm for simulated measurements, the
residual structures caused by absorption are reduced: sensitivity studies
reveal good agreement within 1 % for retrieved ozone profiles with
respect to the truth. This good agreement is obtained for most of the
altitudes for retrievals both in the UV and the VIS spectral range even
if the effective light paths up to just the second-order are considered. The
agreement practically does not depend on the selected wavelength for the
spatial evaluation. Also for the minor absorbers NO

The pre-calculation of higher-order sensitivity parameters by RTM allows to
enhance the performance speed of iterative retrieval algorithms. The
consideration of only second-order absorption terms in the simulation of the
correction spectra in the iterative two-step approach accounts for most of
the non-linear absorption effects, reducing the systematic residual structures
to the order of

Although our studies are limited to the limb geometry which is in many aspects the most sophisticated one, MAX-DOAS and even nadir geometry under extreme conditions (high SZA and/or high pollution) can benefit from the approach, but this needs to be investigated in separate studies. The obtainable improvement depends on the precision at which one is measuring (uncertainties of the spectral retrieval) and performing the retrieval (e.g. uncertainty of the radiative transfer simulations).

We use the 3-D full spherical Monte Carlo RTM McArtim

Information provided by McArtim about the simulated light trajectories
include, among others, the coordinates of the scattering/absorption/ground
reflection/box boundary crossing events, information about the event type,
and,
in case of scattering, the weights characterizing the probability that the
trajectory from the scattering position will reach the Sun (in a
probabilistic sense representing the weights used in
Eq.

According to the dependent sampling formalism used in McArtim, each
trajectory contains as many light paths as radiative transfer events involved
(technical events like box boundary crossing are not counted as an event in
this sense). Therefore, considering Eqs. (

Considering an absorption cross section invariant in space (an assumption
used in DOAS), a third-order impact of the absorption of one trace gas
(compare Eq.

The third-order cross-correlative term between two trace gases

Since there are three combinations of

The third-order cross-correlative term between three trace gases

There are six combinations of

It can be seen that the expressions are third-order products of cross sections and these cross-section products can in principle be considered in a DOAS fit.

We want especially to thank Tim Deutschmann for providing the RTM McArtim and his knowledge about radiative transfer. We thank also ESA and DLR for providing the SCIAMACHY level 1 data. J. Puķīte is funded by the Deutsche Forschungsgemeinschaft (PU518/1-1). The article processing charges for this open-access publication were covered by the Max Planck Society.Edited by: Jochen Stutz