Global satellite observations allow for the timely detection and monitoring of SO2 emitted from volcanic eruptions, even in remote regions, where no ground-based instruments are installed (see, e.g., ). Satellite measurements of UV earthshine spectra in the wavelength range between 305 and 335 nm provide the highest sensitivity to SO2 in the Earth's atmosphere. Volcanic eruptions can inject large amounts of sulfur dioxide (SO2) into the atmosphere, where it is either subject to wet and dry deposition, as well as oxidization within a few days in the troposphere (see, e.g., or ), or oxidization over a period of several weeks to sulfate aerosols in the stratosphere (see, e.g., and ). Sulfate aerosols can affect the Earth's radiative forcing and have an impact on clouds (see, e.g., and ).

Based on UV earthshine measurements, SO2 vertical column densities (VCDs) can be retrieved easily using techniques such as differential optical absorption spectroscopy DOAS; see, e.g.,, a principal component analysis PCA; see, e.g.,, or the Krueger–Kerr algorithm, which was applied to retrieve SO2 from NASA TOMS see, e.g.,. These methods are fast enough for near-real-time (NRT) retrievals. Nevertheless, all algorithms retrieve only the slant column – to calculate the VCD, a conversion factor (called the air mass factor, AMF) has to be applied, which includes explicit or implicit assumptions about the vertical distribution of SO2 in order to determine the effective light path. Note that AMFs are calculated by means of multiple-scattering radiative transfer models assuming known vertical distributions of SO2 and O3, as well as viewing, surface, and cloud properties.

Unfortunately, the vertical distribution (in terms of the plume layer height) of SO2 is usually unknown at the time of the measurement as it is not easy to extract from the spectral signature (see and ): the SO2 loading (the VCD) has a direct effect on the optical depth, whereas the layer height (LH) has an indirect effect on the optical depth as it influences the number of photons passing through the SO2 layer, the UV wavelengths interacting with the SO2 layer, and the layer optical depth due to the temperature dependency of the SO2 absorption cross sections .

Even if there are ground-based or aircraft measurements of the SO2 layer height (LH), the data are generally difficult to use for validation, as, e.g., for strong eruptions (a volcanic explosivity index, VEI, of greater than or equal to 3), volcanic plumes are typically transported over long distances and the number of collocations is small. Thus, for volcanic SO2 measurements, the vertical distribution of SO2 is a key parameter limiting the product accuracy. The usual approach for operational SO2 retrievals is to assume several different a priori SO2 vertical distributions and provide VCDs for each (see, e.g., ) along with an averaging kernel (AK), so that the user can calculate the VCD for an arbitrary SO2 vertical distribution. However, in the upper troposphere and above, the vertical SO2 distribution has little impact on the VCD.

To date, SO2 layer height retrievals have used computationally demanding direct fitting inversion methods, which are not suitable for NRT applications. For the retrievals based on satellite UV measurements, and developed an “Extended Iterative Spectral Fitting” (EISF) algorithm for the Ozone Monitoring Instrument (OMI) aboard the NASA Aura satellite, and introduced an optimal estimation (OE) scheme for the GOME-2 instrument (Global Ozone Monitoring Experiment-2) aboard the EUMETSAT/ESA MetOp satellite fleet. For strong volcanic eruptions (VEI ≥ 3), the accuracy of the retrieved SO2 LHs using this approach is in the 0.5–1 km range, whereas for small SO2 absorption it is around 2 km (see ).

Satellite infrared sounders also offer the opportunity to measure both SO2 VCDs and LHs (see ) using optimal estimation algorithms for the Infrared Atmospheric Sounding Interferometer (IASI) aboard the EUMETSAT/ESA MetOp satellite fleet. While infrared sounders have an inferior sensitivity to lower tropospheric SO2 compared with UV instruments, the layer height retrievals tend to have a better accuracy, and perform well even for low column amounts (up to the single-DU level; see, e.g., or ); however, the horizontal resolution (12 km) is rather coarse.

In contrast, the TROPOMI instrument onboard the Sentinel-5 Precursor satellite (S5P) launched on 13 October 2017 has a much higher spatial resolution of 7×3.5 km2, and will operate at an even smaller resolution of 5.5×3.5 km2 beginning mid-August 2019. This allows us to observe and study SO2 plumes at an unprecedented level of detail. Data turnover from TROPOMI is very large, and this consideration will require the development of new retrieval schemes for the fast and accurate retrieval of SO2 layer heights in an operational environment.

To this end, we have developed an algorithm called “Full-Physics Inverse Learning Machine” (hereafter referred to as FP_ILM) for the retrieval of the SO2 LH based on satellite UV earthshine spectra. The FP_ILM algorithm has been used for the retrieval of ozone profile shapes , the retrieval of surface properties accounting for bidirectional reflectance distribution function (BRDF) effects , and the retrieval of SO2 LH from GOME-2 . The algorithm creates a mapping between the spectral radiance and SO2 LHs using machine learning methods. The time-consuming training phase of the algorithm using radiative transfer model calculations is performed off-line, and only the inversion operator has to be applied to satellite measurements – this makes the algorithm extremely fast; thus, it can be used in near-real-time processing environments. In this second paper on the FP_ILM SO2 LH, we describe some improvements to the original algorithm from , and we apply it to a number of volcanic eruptions observed by TROPOMI during the operational phase of the mission (started April 2018).

The paper is organized as follows. In Sect.  we describe the improved FP_ILM SO2 LH algorithm. The sensitivity of retrieved SO2 LHs to a number of different parameters is discussed in Sect. . In Sect. , the FP_ILM is applied to S5P data to retrieve SO2 LHs for selected volcanic eruptions. Section  describes how the algorithm could be integrated in the operational TROPOMI SO2 VCD retrieval algorithm. We summarize the paper in Sect. .

Conceptually, the FP_ILM consists of a training phase, in which the inversion operator is obtained using synthetic data generated with an appropriate radiative transfer (RT) model, and an operational phase, in which the inversion operator is applied to real satellite measurements. The main advantage of the FP_ILM over classical direct fitting approaches is that the time-consuming training phase involving complex RT modeling is performed off-line; the inverse operator itself is robust and computationally simple and, therefore, extremely fast. In our previous paper (see ), we first introduced the FP_ILM algorithm and applied it to GOME-2 observations of a number of volcanic eruptions. We used a combination of principal component analysis (PCA) and principal component regression (PCR) methods to train the inversion operator in order to retrieve the SO2 LH. For the current paper we have improved the FP_ILM algorithm with the use of a neural network (NN) approach, as outlined below.

During the training phase, the LInearized Discrete Ordinate Radiative Transfer model (LIDORT) with inelastic rotational Raman scattering (RRS) implementation is deployed to compute simulated reflectance spectra in the 310–335 nm wavelength range. These spectra depend upon the following n=8 input parameters: the SO2 VCD and LH, the surface albedo, the surface height, the O3 VCD, the solar zenith angle (SZA), the viewing zenith angle (VZA), and the relative azimuth angle (RAA). Table  provides a summary of the final parameter space after optimization (see below) used for the training of the final retrieval operator. Note that O3 has to be included due to the strong spectral interference between SO2 and O3 in the spectral range considered. Ozone profiles are classified according to the total column amount of O3, and the month and latitude zones as specified in the TOMS Version 8 O3 profile climatology . The SO2 plume profile is taken to have a Gaussian shape, characterized by the total SO2 VCD loading and centered at a peak-concentration layer height zp, along with a half-width fixed to 2.5 km. In the following, the retrieval of the SO2 layer height refers to the retrieval of the peak-concentration height zp.

Simulations were carried out on a pressure/temperature/height grid from the US standard atmosphere, with a finer-grid vertical height resolution of 0.25 km below 15 km in order to properly resolve the Gaussian SO2 plume shape. In total, some 131 072 simulated reflectance spectra have been calculated on a selective parameter grid established by means of a smart sampling technique developed by . Further details on the smart sampling technique applied to the SO2 LH retrieval can be found in .

The use of the LIDORT-RRS RT model is necessary, as it enables us to account for the effect of Raman scattering in the atmosphere: solar irradiances exhibit strong Fraunhofer structures in this part of the UV spectral range, and earthshine spectra are characterized by the “filling-in” of Fraunhofer-solar and telluric-absorber features due to “inelastic” (wavelength-redistributed) rotational-Raman scattering by air molecules. For further details, see and .

The simulated high-resolution reflectance spectra are convolved with the TROPOMI instrument spectral response function (ISRF) v3.0.0 (released 1 April 2018) (available at: http://www.tropomi.eu/data-products/isrf-dataset/, last access: 14 October 2019). Note that the TROPOMI instrument comprises 450 rows, which are in principle single detectors with their own ISRFs. The signal-to-noise ratio (SNR) is about 1000 in our UV wavelength range. Thus, to account for instrumental noise in the training phase, uncorrelated Gaussian noise with a fixed SNR of 1000 is added to the simulated spectral data.

To extract the information about the layer height and to reduce the dimensionality of the spectral dataset, a PCA is applied to the simulated spectra. By characterizing the set of simulated measurements with fewer parameters in this fashion, a simpler, more stable, and computationally efficient inversion scheme can be realized. It was found that using 10 principal components (PCs) was sufficient to retrieve information about the SO2 LH. These 10 PCs accounted for 99.994 % of the spectral variance. The inclusion of additional PCs beyond 10 did not result in any improvements to the LH retrievals, as higher-order PCs are increasingly affected by noise. Figure  shows the ratio between the variance of the PCA-derived dataset and the total variance of the complete spectra dataset (the “explained” variance ratio) as a function of the number of PCs included in the PCA.

The 10 PCs, along with the information about the O3 VCD, the SZA/VZA/RAA angles, the surface pressure, and albedo of each training data point, are then used as input to train a feed-forward artificial neural network (NN) including regression (in our case a “Multilayer Perceptron Regression” – MLPR), with the corresponding SO2 LH as the output layer. Note here that the SO2 VCD is not part of the training, as it depends directly on the SO2 layer height.

In general, NNs can be used for establishing a non-linear mapping between a dataset of numeric inputs and a set of numeric outputs. A NN consists of interconnected neurons (or nodes) that implement a simple, non-linear function (a sigmoid function in our case) of the inputs. Neural networking is a powerful tool for determining non-linear dependencies between datasets in remote sensing, as shown by . It consists of an input layer (representing the abovementioned input parameters), at least one so-called hidden layer, and an output layer (representing the expected output – in our case the SO2 LH). Each neuron in the hidden layer(s) transforms the values from the previous layer(s) with a weighted linear summation, followed by a non-linear activation function, which is the sigmoid function in our case. The output layer receives the values from the last hidden layer and transforms them into output values.

The training of the MLPR is an iterative process that tries to minimize the so-called loss function (also known as the cost function), which is a measure of how well a model predicts the expected outcome; at each time step, the partial derivatives of the loss function with respect to the model parameters are computed in order to update the parameters. We note here that MLPR uses the “mean square error” loss function. A regularization term is added to the loss function that shrinks model parameters to prevent over-fitting: By building a complex neural network, it is quite easy to perfectly fit the training dataset. However, when this model is evaluated on new data (here the satellite measurements), it performs very poorly. Thus, the regularization modifies the loss function by adding additional terms that penalize large weight vectors and preferring diffuse weight vectors.

After carrying out a PCA and MLPR parameter optimization using closed-loop retrievals to minimize differences between the retrieved and simulated layer heights, the final configuration for the neural network settled on the use of two hidden layers, with 32 nodes in the first layer and 10 nodes in the second.

In the operational phase, the first step is to use the principal component scores acquired during the training phase to transform a given TROPOMI spectral measurement dataset to one with a lower dimension. Once this is done, the neural network inverse function is then applied to retrieve the SO2 LH. In order to avoid the training of the NN for each of the 450 TROPOMI detector rows (with their own ISRF), we only trained the network for every 50th detector row and interpolated the retrieved SO2 LH results to the actual row where it was detected.

We note that the FP_ILM algorithm only needs to be re-trained when large changes in the ISRF or SNR occur (see Sect. ).

In this section, we study the dependency of the layer height retrieval on different parameters. We discriminate between direct dependencies (i.e., those parameters affecting the reflectance spectra) and indirect dependencies (i.e., affecting the training data and inversion algorithm) of the retrieved layer height:

Direct dependencies are defined as the viewing geometry, surface properties, ISRF, noise level (SNR), instrumental stray light, and O3 VCD.

First, we train the FP_ILM operator using 90 % of the training dataset (see Table ), and then apply the trained operator to the remaining 10 % training spectra, for which we know the exact SO2 LH. Figure  (red dots) shows the SO2 LH difference as a function of solar/viewing geometry, SO2 VCD, O3 VCD, albedo, and surface pressure, respectively. The figure clearly shows a number of marked dependencies in the retrieved layer height, with notably high differences with respect to the real SO2 LH for low and high layer heights, for low SO2 VCD as well as for high SZA.

Regarding the SZA dependency, a cutoff limit of 75∘ is usually set in operational SO2 retrievals, because at high SZAs the light path becomes very long and the noise level increases. Accordingly, we limit the SZA of the spectra used in the training to SZA values less than or equal to 75∘ in the following.

Clearly, for small SO2 VCDs, the information content on the LH in the spectral signature is very low. It follows that the inclusion of spectra with low SO2 content in the training will have a negative effect on the entire neural network. In principle, for lower SO2 VCD loadings, more PCs could be included, but at some point the noise level signature will exceed that of the actual SO2 absorption.

We performed several tests in which we limited the training dataset by varying the allowed input parameter ranges. We found an optimal parameter range that allows for the retrieval of a broad range of SO2 LHs, even for low SO2 VCDs. In the training phase, we only use spectra with SO2 VCDs greater than or equal to 20 DU, surface albedo values less than or equal to 0.5, and SZAs less than or equal to 75∘ to train the final inversion operator for TROPOMI. The albedo limit is set to 0.5, as large albedo values will induce large variations in the spectra (multiple reflections from the surface). These additional variations correlate with the variations due to perturbations in the SO2 VCD and LH. Therefore, to make the algorithm more stable, we restrict the albedo range to the physically relevant cases. We further limited the training values of the SO2 LH to the range between 0 and 25 km. Although higher LHs from strong volcanic eruptions can occur, the use of a broad training data range also has an influence on the accuracy of the retrieval. To a limited extent, however, the FP_ILM is also able to extrapolate to an untrained parameter range, although with significantly less accuracy.

Using the optimized training dataset (see Table  for a summary of the parameter space), Fig.  (blue triangles) shows that the error on the retrieved SO2 LH is less than 2 km and the dependency on the SO2 VCD is reduced. Figure  shows the retrieved layer height as a function of the SO2 VCD. As mentioned above, for low SO2 VCDs, high-altitude layer heights cannot be retrieved – there is always a bias towards low layer heights. Only for SO2 loadings in excess of 20 DU, do we retrieve layer heights with an uncertainty of less than 2 km.

To investigate the dependency of the retrieved SO2 LHs as a function of the SNR, we used two different noise levels (a SNR of 500 and 1500). Figure  clearly shows that the SNR only has a minor effect on the accuracy of the retrieved layer height, with only slightly higher accuracy for increased SNR. This is to be expected, as we used the first 10 PC scores and, thus, basically removed all noise features.

Concerning the dependency on the ISRF, we note that in the operational retrieval of SO2 LHs from TROPOMI data, each detector row is effectively a single instrument with its own ISRF. The accuracy of the retrieved SO2 LH can vary across detector rows. Figure  shows the SO2 LH results when applied to the Sinabung eruption (see Sect. ). Black dots show the LHs for each 50th row, whereas the red cross shows the interpolated LH for the measurement row. Clearly, the retrieved LH is slightly different in each row (within 1 km) due to the different ISRF used for training the NN. Note that we determine the LH for a set of fixed detector rows (for which we have trained the FP_ILM separately), and then interpolate the layer height to the actual row. In this way we avoid jumps in the retrieved layer height between adjacent detector rows.

Instrumental stray light can introduce spectral features that may lead to bias in the retrieved SO2 LH. However, according to the in-band stray light of TROPOMI after correction is as low as 0.5 % and, thus, can be neglected. Furthermore, no evidence for out-of-band-stray light was found (Quintus Kleipool, personal communication, 2019).

During major volcanic eruptions (VEI ≥ 3) with very high SO2 loadings, the O3 VCD retrievals may be inaccurate due to SO2 interference. However, this effect is negligible for weak eruptions. For strong eruptions (exceeding about 50 DU of SO2) the error on the O3 VCD may reach a few percent (see ), which means that the error on the SO2 LH is still negligible.

Reflectance spectra from TROPOMI are determined from the operational L1 solar irradiance and earthshine radiance data (solar irradiance is measured on a daily basis). To correct for Doppler shifts between earthshine and irradiance spectra, we apply the wavelength calibration information from the operational L2 SO2 product to first register the solar spectrum and then we use the fitted shift and squeeze parameters from the DOAS retrieval to calibrate the earthshine spectra. From this calibrated L1 data, we then calculate reflectances in the 310–335 nm wavelength range.

In the following subsections, we have applied the FP_ILM operator to three major volcanic eruptions measured by TROPOMI. We chose eruptions with a peak SO2 VCD exceeding our 20 DU threshold criterion and with an extended SO2 plume that allows us to compare the FP_ILM results with independent retrievals from other satellite data sources. In addition to L1 reflectances, we use information on SO2 VCD, O3 VCD, surface, and viewing conditions from the operational SO2 L2 product (see ).

For validation of our results, we performed comparisons with independent MetOp/IASI SO2 LH data (see for details), as well as SO2 profile data from the Microwave Limb Sounder (MLS) on the NASA/Aura satellite (see ). The MetOp platforms have different orbits from that of S5P, with widely different overpass times; thus, direct satellite comparisons with IASI data will only give a qualitative validation on the accuracy of retrieved SO2 LHs from TROPOMI measurements. The afternoon Aura/MLS overpass is nearly coincident with TROPOMI. The MLS can provide some information on the SO2 LH, albeit with limited spatial coverage and vertical resolution, as it is only sensitive above an altitude of about 147 hPa (i.e., above around 13 km). We also checked on the availability of an overpass of the Cloud-Aerosol Lidar and Infrared Pathfinder Satellite Observation (CALIPSO, ) satellite with the Cloud-Aerosol Lidar with Orthogonal Polarization (CALIOP) instrument. Note that CALIPSO measures only ash and aerosol absorption profiles. In this regard, we note that SO2 plumes and ash or aerosols are not necessarily colocated, as gas and ash can separate in volcanic clouds. At the time of writing, IASI and MLS data are unfortunately the only sources for independent SO2 LH satellite validation.

In this section we describe the manner in which the algorithm could be implemented in the operational S5P/TROPOMI ground segment.

As the same input parameters are used for retrieving operational SO2 VCD, i.e., O3 VCD, viewing parameters, and surface conditions as well as radiance and irradiance data, the FP_ILM algorithm can be easily integrated within the operational UPAS processor used for generating the SO2 products. In the operational TROPOMI environment, cloud properties and the O3 VCD are retrieved before the SO2 algorithm is started. Thus, all of the required input parameters are already available when the FP_ILM algorithm is triggered in the case of a volcanic eruption. Only the wavelength calibration and calculation of the reflectance spectrum has to be performed prior to retrieving the SO2 LH, as this step differs from the operational SO2 VCD retrieval.

To process a satellite pixel containing volcanic SO2, the FP_ILM algorithm should be triggered by the operational enhanced SO2 detection flag (see for a reference) and using a threshold (a priori) SO2 VCD of at least 10 DU. The resulting SO2 LH for this pixel should be stored in the final L2 SO2 product and can be also used to calculate an optimized SO2 VCD for this LH.

The SO2 LH retrieval only takes about 2 ms per TROPOMI spectrum; hence, even for an extended volcanic plume, the entire LH retrieval can be performed in a matter of seconds. This is important for operational retrieval environments with strong time constraints. Currently, the entire processing of a single TROPOMI pixel (i.e., cloud parameters, O3 and SO2 VCD retrieval) in the operational ground segment takes about 90 ms, which translates to about 24 min for an entire S5P orbit. Thus, the additional time required for the retrieval of the SO2 LH for selected pixels is not significant.

We have developed a new algorithm for the fast and accurate retrieval of SO2 layer heights from UV earthshine observations of volcanic SO2 eruptions by the TROPOMI sensor onboard the Sentinel-5 Precursor platform. The SO2 LH retrieval has two phases: (1) a computationally expensive off-line training phase in which the retrieval inverse operator is obtained using the FP_ILM (Full-Physics Learning Machine) algorithm, and (2) a fast operational phase, in which the FP_ILM inverse operator is applied to measured UV reflectance spectra. The FP_ILM combines a principal component analysis with a neural network regression using the UV reflectance, O3 total column, viewing geometries, and surface properties as input. Based on an optimized training dataset created with smart sampling techniques, the principal component scores calculated for reflectance spectra in the 310–335 nm wavelength range are used along with the other parameters to train a feed-forward artificial neural network. For S5P/TROPOMI measurement data, an initial dimensionality reduction of the reflectance spectra is performed by applying the PCA-derived principal component scores before retrieving SO2 LH with the trained neural network.

The FP_ILM can be used for NRT applications with strict time constraints. S5P/TROPOMI, with its high spatial and spectral resolution, provides a huge amount of data and the computationally intensive direct fitting approaches to SO2 LH retrieval developed so far are not applicable. In contrast, the FP_ILM operator performs SO2 LH retrieval within about 2 ms per TROPOMI spectrum. Hence, even the retrieval on extended volcanic plumes can be performed in a matter of a few seconds, allowing for the determination of the SO2 layer height with an accuracy better than 2 km for SO2 total column densities larger than 20 DU.

In this paper, we deployed an independent simulated reflectance spectra dataset to investigate the accuracy of SO2 LH retrievals and their dependencies on a number of different factors and parameters both direct and indirect. In particular, it was found that the retrieved SO2 LH is strongly dependent on the SO2 VCD for low VCDs (VCDs ≤ 20 DU) as well as for high SZAs (SZAs ≥ 75∘). For high VCDs and low SZA, according to closed-loop tests, the SO2 LH can be retrieved with an accuracy of better than 2 km. We also investigated the dependencies on ISRF and SNR, both of which turned out to be relatively slight effects.

Broad-band spectral scattering and absorption due to sulfate aerosols or volcanic ash plumes will certainly affect SO2 LH retrievals. Although not considered in the present work, we will address this important issue in a forthcoming paper in this series; aerosols will be explicitly accounted for in the training of the FP_ILM. Nevertheless, we should note that SO2 and ash are likely to be colocated only for fresh volcanic plumes. For mature plumes, mass differences will ensure that ash and SO2 plumes are not located at similar altitudes, and the corresponding plumes are thus subject to different wind-direction dispersal.

We applied the FP_ILM to a number of volcanic eruptions (VEI ≥ 3) observed recently by S5P/TROPOMI. Our SO2 LH results were compared to SO2 LHs retrieved from IR measurements from two MetOp/IASI satellites, as well as the NASA Aura/MLS SO2 profile and CALIOP/CALIPSO ash and aerosol measurements. Unfortunately the orbits of the MetOp satellites and S5P have widely different overpass times, allowing for only qualitative comparisons. Despite this, there is generally very good agreement between the IASI and TROPOMI results. Very good agreement with MLS data was found, which provides SO2 profiles with limited spatial coverage and vertical resolution, however with overpass times close to the S5P overpass. Nevertheless, further verification work is certainly needed. For the Sinabung eruption in February 2018 and the Raikoke eruption in 2019, our results also showed excellent agreement with CALIOP/CALIPSO LIDAR data measuring the ash plume.