Algorithms that estimate properties of atmospheric species from satellite measurements of top-of-atmosphere (TOA) radiance (including spectral signatures of gases) in planetary atmospheres typically employ an inverse method based on least squares. When retrieving terrestrial properties, this approach requires spectrally resolved measurements of the TOA Earth radiance, solar irradiance and a forward model as the minimal base ingredients with which the state vector parameters can be retrieved (which are also model parameters). The goal of the least squares approach is to minimize a cost function, which aims to reduce discrepancies between the forward model and the measurement by iteratively manipulating state vector parameters. Upon minimization, the iterative scheme converges to a solution that, in principle, best describes the forward model's representation of the measurement.

Many atmospheric retrieval algorithms employ a weighted least squares estimation (WLSE) method modified to include a priori information on the state vector. An example of such an inverse method set-up is optimal estimation OE;, which is an attractive method particularly because of its efficacy in providing a posteriori error statistics on the retrieved parameter. The KNMI aerosol layer height (ALH) retrieval algorithm uses an inverse method based on OE and exploits the spectral structure of the near-infrared spectrum of the top-of-atmosphere radiance between 758 and 770 nm, where photons travelling through the Earth's atmosphere predominantly get absorbed by molecular oxygen. Oxygen is a well-mixed gas and the spectral structure of its absorption lines is pressure-dependent . The further the light in the oxygen A band passes through the atmosphere, the more it gets absorbed until it interacts with scattering species (such as clouds and aerosols) and scatters back to the TOA. It is this feature of the oxygen A band that has made it an attractive wavelength region for retrieving aerosol information . The algorithm is operational for the TROPOspheric Monitoring Instrument (TROPOMI) on board the Sentinel-5 Precursor (S5P) mission and is also a part of the Sentinel-4 (S4) and Sentinel-5 (S5) missions under the Copernicus satellite programme of the European Union.

Due to the large spectral variability in absorption within the oxygen A band, the measured TOA radiance and the measurement noise have a high dynamic range. The minimization of the propagation of measurement noise to the final retrieval solution should be a critical component of any retrieval algorithm. In WLSE, this is accomplished by the inverse measurement error covariance matrix, which ranks the measurement on each detector pixel using the information available on the measurement noise. Due to the extent of the dynamic range of the measurement noise in the oxygen A band, this ranking matrix becomes a primary controlling entity; if the measurement noise is very large, the inverse noise variance is very low, which results in a lower rank to the measured signal from that specific detector pixel.

Since the measured signal is scene dependent, the spectral rank of each detector pixel is also scene dependent. This has special consequences over bright surfaces, where the dynamic range of the measured signal is much larger than over dark surfaces. Due to this, photons at wavelengths where the oxygen A band has a lower absorption cross section are less absorbed (subsequently travelling further into the atmosphere) and have a much larger representation in the WLSE method. A consequence of this, reported by , is that the retrieved ALH values are inaccurate for measurements over land.

In order to account for unknown instrument and model errors, multiply the measurement error from L1b by two for their GOME-2 case studies and by 10 in SCIAMACHY case studies for retrieving ALH over ocean and land. They observe that increasing the measurement noise results in an increase in the number of retrieval convergences without significantly decreasing the accuracy of the retrieved ALH for the already converged solutions. The method utilized by does not change the shape of the noise spectrum since it is multiplied by a constant. This paper investigates a vector-based weighing scheme (we call it the dynamic scaling method, as opposed to the formal approach which is unscaled OE), which dynamically varies from scene to scene and changes the shape of the noise spectrum itself. The objective of the dynamic scaling method is to influence the inverse measurement error covariance matrix in its ranking of the instrument's detector pixels in its spectral dimension in order to maximize sensitivity to the ALH. The study discussed in this paper is a part of a series of papers discussing the ALH retrieval algorithm developed at the KNMI .

The retrieval algorithm is described in Sect , which provides a description of the forward model and the formalism of OE. The incompatibility of retrieving aerosol properties from oxygen A band measurements with the formal design of the measurement error covariance matrix are briefly discussed in the same section (Sect. ) before a full description of the proposed method in Sect  and a demonstration in a synthetic environment in Sect  are given. This method is applied to real data in Sect . The Russian wildfires in August 2010, which were discussed by , are revisited to compare the two approaches. The data are derived from the GOME-2A (Global Ozone Monitoring Experiment on board the MetOp-A platform of the European Organisation for the Exploitation of Meteorological Satellites, EUMETSAT) instrument and validated with a co-located CALIPSO (Cloud-Aerosol Lidar and Infrared Pathfinder Satellite Observation of the National Aeronautics and Space Administration, NASA) overpass. The dynamic scaling method is further applied to the Portugal fire plumes over western Europe on 17 October 2017 using data from the GOME-2B instrument, with validation from the ground-based European Meteorological Services Network EUMETNET, ceilometer network in the Netherlands and Germany, along with radiosonde measurements of the relative humidity profile and the back trajectory of the aerosol plumes. This demonstration is followed by the conclusion in Sect. .

The algorithm is comprised of a forward model and an inverse method. The forward model uses a radiative transfer model described by to calculate the top-of-atmosphere (TOA) Earth radiance (I) in the oxygen A band. This is done by propagating incoming solar irradiance (E0) in the oxygen A band through the Earth's atmosphere, which is described by an atmospheric model. Finally, this model is fitted to the measured spectrum to retrieve the primary unknown ALH while fitting the aerosol optical thickness (AOT). For more details, the reader may refer to .

The dynamic scaling method identifies favourable spectral points for ALH retrieval by first identifying spectral points that are the least favourable. The noise is increased at these unfavourable points while keeping the noise at the other points unchanged. These favourable and unfavourable spectral points are identified using a class of vectors known as modifying vectors (with the symbol M and length equal to the number of spectral points).

To identify the unfavourable spectral points at which the measurement noise is to be modified, a modifying vector MAs/zaer is proposed as

MAs/zaer(λi)=KAs(λi)Kzaer(λi)[hPa], where KAs(λi) is the derivative of the TOA reflectance with respect to surface reflectance at the ith index of the spectral point on the detector, and Kzaer(λi) is the same for zaer. In principle, the ratio of KAs and Kzaer is used as an identification tool since our primary retrieval parameter is zaer, which reduces its information as As increases. This opposing nature is discussed by (Figs. 3 and 4 in their paper), where they show an anti-correlation in the sensitivity of τ and zaer in the atmospheric path contribution and surface contribution to the TOA reflectance. A large value in MAs/zaer(λi) represents spectral points in the measurement with more sensitivity to As than to zaer. The motivation for choosing derivatives as the means for modification is also partly motivated from the fact that they are scene-dependent parameters, which makes each modification unique to the scene.

Spectral points with a MAs/zaer(λi) higher than a specific threshold value should have a limited representation in the estimation – these are the unfavourable spectral points. We define this threshold as the modifying threshold (T), which is the 20th percentile value of MAs/zaer. The threshold value set in our method has been chosen in a way to avoid scaling the deeper parts of the R and P branches in the A band. The choice of thresholding remains configurable to the user of this method, based on their requirements – in our case we have chosen to use a static rule for deciding the value of T, but this could also be made dynamic. An example of the shape of MAs/zaer is provided in Fig.  (top row).

The reason for increasing the noise at specific unfavourable spectral points is to increase the value of Sϵ at these points. With a higher Sϵ value, the Sϵ-1 value will be lower, and hence that spectral point will have a lower weight in the estimation. In principle, this is equivalent to artificially increasing noise of measurements that contain less sensitivity to ALH. To do this, the modified SNR (denoted by SNRM) is defined as

SNRM(λi)=SNR(λi)if MAs/zaer(λi)<TSNR(λi)/MAs/τ(λi)otherwise, where MAs/τ(λi) (belonging to the class of modifying vectors) is defined as the ratio between the derivative of the TOA reflectance with respect to the surface (KAs(λi)) and with respect to AOT (τ) at 760 nm (Kτ(λi))

MAs/τ(λi)=KAs(λi)Kτ(λi)[-].

The choice of modifying the SNR based on MAs/τ arises from the fact that the amount of contribution by the aerosol layer to the TOA reflectance depends on its optical thickness. In this case, we are interested in how much this contribution fares against the contribution from the surface. Information on both of these contributions can be inferred from the ratio of KAs and Kτ, which have comparatively similar shapes. If the measurement of a spectral pixel i is more sensitive to As, MAs/τ(λi) will be larger, and hence the noise at i will be increased, following Eq. ().

To run a retrieval using the dynamic scaling method, the derivatives of the reflectance with respect to As, zaer and τ at 760 nm are calculated first, followed by the modification of SNR according to Eq. (). The state vector parameters τ and zaer are then estimated using SNRM. Users of this method may choose to scale the measurement error covariance matrix at each iteration, since the derivatives change at each iteration. Nevertheless, we have chosen to do it semi-statically since the measurement error covariance matrix is a static matrix throughout every iteration.

Examples of modifying vectors and SNRM are provided in Fig.  (bottom row), which shows the robustness of the method in scaling the SNR for different surfaces. The spectra generated in the figure represents two scenes with identical atmospheric parameters, solar and satellite geometries, but different As. MAs/zaer, T and MAs/τ for different surfaces are different – this is important, since overscaling the SNR can force the retrieval to rank the measurements of photons travelling from the upper parts of the atmosphere higher while ignoring those from the lower parts of the atmosphere. This is why the modifying vector MAs/τ is chosen as a dynamically scene-dependent parameter (according to Eq. ), such that the scaling is large when As is large (Fig. , mid row). In the next section, the dynamic scaling method is demonstrated and compared to the formal approach (which is the unscaled OE method) for synthetically generated spectra.

To demonstrate the dynamic scaling method, synthetic spectra are generated for randomly varying values in zaer, τ, solar-satellite geometry (θ, θ0 and ϕ-ϕ0), and As, while keeping other parameters constant. Noise is not added to the synthetic spectra. This method of randomly generating model parameters for generating synthetic spectra gives a broad picture of the method's behaviour. Table  provides a brief overview of the input model parameters chosen to generate these spectra. An error is introduced in the forward model during retrieval, and the bias in zaer (defined as retrieved – true) is used to assess the retrieval. The a priori zaer and τ are set at 825 hPa and true τ, respectively. While there are many possible sources of errors, this paper presents two kinds of errors: (a) error in the thickness of the aerosol layer, and (b) error in the surface albedo database. A reason for limiting the retrieval experiment scope to these two errors in the atmospheric part of the forward model is due to the fact that they are one of the more common contributors to retrieval biases. In real cases, aerosol layers may not be concentrated in a single layer of 50 hPa thickness, and the true surface albedo may vary significantly (to the order of 10 % relative errors) from a monthly database of Lambertian equivalent reflectivity (LER) values depending on many parameters. In total, 2000 synthetic spectra are generated for each synthetic experiment and the parameters zaer and τ are estimated using both the formal approach and the dynamic scaling method to be compared side by side. The results from analysing biases in retrieved zaer are plotted in Fig. . Although the dynamic scaling method is specifically designed for land, retrievals over surfaces with a low As (less than 0.1) are also included.

The GOME-2 instrument is a part of an operational mission by the European Organization for the Exploitation of Meteorological Satellites (EUMETSAT) to monitor trace gases and aerosols in the atmosphere. It is a spectrometer with an across-track scanning mirror that projects the TOA Earth radiance and solar irradiance through a prism on a grating to get information in the ultraviolet, visible and the near-infrared regions of the electromagnetic spectrum. In the oxygen A band, the spectral sampling interval is typically about 0.20 nm and the FWHM is 0.50 nm . The GOME-2 instrument is designed to have a footprint size of 80km×40km in the oxygen A band. The instrument also measures the linear polarization of Earth radiance, which is important for correcting the measured signal to calculate the reflectance accurately.

In this section, measured spectra from the GOME-2A instrument on board the Metop-A satellite over Russian wildfires on 8 August 2010 (Fig. a) and the Portuguese fire plume are used with the GOME-2B instrument on board the MetOp-B satellite on 17 October 2017 over western Europe (Fig. b). The formal OE method is compared to the dynamic scaling method by using space-based and ground-based validation data. The noise spectrum is derived from the GOME-2 level 1-b product, which is a combination of the systematic and random error components of the measurements .

Auxiliary information required for these retrievals are meteorological data, surface albedo, and a priori values for the optimal estimation (Table ). The meteorological data required are temperature–pressure profiles and the surface pressure, derived from the ERA-Interim database from . These meteorological parameters are available on regular space (1∘×1∘ spatial resolution) and time grids, and require interpolation to the satellite pixel's coordinates and time of record. This interpolation is done using the nearest-neighbour algorithm. The surface albedo database is derived from version 2.1, which has a resolution of 0.25∘×0.25∘, derived from the GOME-2A instrument. The surface LER is chosen as the median of all LER database pixels intersecting the GOME-2 instrument pixel at wavelengths 758 and 772 nm with linear interpolation used for calculating LER values at intermediate wavelengths. Algorithm settings are detailed in Table . The test cases chosen in this paper are relatively cloud-free but not fully.

For validation, atmospheric lidar data from satellite and ground-based instruments are chosen. For the 2010 Russian wildfires, the lidar-attenuated backscatter at 1064 nm from the CALIOP instrument (Cloud-Aerosol LIdar with Orthogonal Polarization) are used on board NASA's CALIPSO (Cloud-Aerosol Lidar and Infrared Pathfinder Satellite Observations) mission. These data have a very good representation of the scattering ability of clouds and aerosols in the atmosphere at a vertical resolution of 60 m and a horizontal resolution of 5 km. For the 2010 Russian wildfires, the CALIPSO overpass is at 10:45 UTC. All GOME-2A pixels co-located within a 100 km vicinity of a CALIOP profile are considered for validation. For the 17 October 2017 Portugal fire plume over western Europe, ground-based ceilometer data are used for validation (Table ). These ceilometers are a part of the ALC (Automated Lidars and Ceilometers) network of the E-PROFILE observation programme in the framework of the European Meteorological Services Network (EUMETNET). The parameter used for validation is the uncalibrated raw backscatter profile, since the paper focuses on qualitatively assessing the aerosol height retrievals with the lidar backscatter profiles. Lidar profiles within an hour of the satellite instrument overpass time are averaged into a single averaged profile in order to reduce noise. These lidars have a vertical range of approximately 15 m and record data at a very high temporal resolution, nominally every 6 s . Although CALIOP data is available for the plume over western Europe for October 2017, CALIPSO does not have as good a co-location (both spatially and temporally) in comparison to the ceilometers.

Inversion algorithms that retrieve aerosol properties from spectral measurements in the oxygen A band (between 758 and 770 nm) can face a lot of trouble over land. This is primarily because of the location of oxygen A band beyond the red edge, a wavelength region that diminishes the ability of vegetation to absorb solar radiation as wavelength increases. This is especially the case when retrieving aerosol layer height using optimal estimation and radiative transfer models, as observed from and .

The optimal estimation framework, an application of the weighted least squares technique, is designed to rank data points (in this case, spectral points in the measured TOA radiance and solar irradiance) higher when the SNR is higher, in order to reduce the influence of measurement error in the final retrieved solution. In the oxygen A band, these spectral points coincide with weak oxygen absorption cross sections, since low absorption equates to a high number of photons that can traverse through the atmospheric medium. Over oceans, due to its low albedo the number of photons that travel back from the surface are few. The signal recorded by satellites from an ocean scene hence predominantly arises from scattering and absorption by atmospheric species (in this case, aerosols). Over land, however, the number of photons that travel back from the surface increase dramatically. Due to this, the optimal estimation framework ranks spectral points representing photons that have travelled back from the surface higher than those from aerosol layers. This is the primary error source when it comes to biases in aerosol retrievals from oxygen A band measurements over land.

This paper introduces the dynamic scaling method, which is designed to retrieve ALH over bright surfaces from oxygen A band measurements. The core principle of this proposed improvement is the wavelength-dependent modification of the measurement error covariance matrix by the subsequent wavelength-dependent modification of the signal-to-noise ratio of the measured spectrum, in order to reduce its preference towards photons that interact with the surface. The modification uses the scene-dependent Jacobian matrix, which makes it robust. The dynamic scaling method is compared with a formal optimal estimation approach by retrieving ALH and AOT from synthetically generated spectra with randomly varied model parameters and model errors (that is, the forward models for simulation and retrieval have different model parameters). The results from the synthetic experiments generally favour the dynamic scaling method, which shows a significant improvement in the accuracy of retrieved ALH in the presence of errors in the assumed aerosol geometric thickness and the surface albedo (up to 10 % relative errors) in the model.

The dynamic scaling method is also demonstrated for real spectra by using GOME-2A and GOME-2B oxygen A band measurements of two separate wildfire incidences in Europe, one being the 8 August 2010 Russian wildfires and the other being the more recent 17 October 2017 Portugal wildfires. In the case of the 2010 Russian wildfires, the formal optimal estimation retrieval approach produces few convergences and misses out the primary biomass burning aerosol plume (as observed from a MODIS Terra image). The fitted AOT are unrealistically high and spatially inconsistent with the aerosol plume observed by MODIS Terra. Co-located CALIOP lidar profiles show that the retrieved ALH is biased low in the atmosphere, closer to the surface. The dynamic scaling method, on the other hand, increases the number of converged pixels by 60 % in comparison to the formal approach. The fitted AOT is still too high, but the spatial distribution of the AOT compared to same observed in the MODIS Terra image is consistent. The retrieved ALHs are also more realistic, as they are positioned close to the centroid of the CALIOP-backscatter-profile-describing aerosols. For the Portugal wildfire plume on 17 October 2017 over western Europe, the dynamic scaling method does not increase the number of convergences significantly. The dynamic scaling method retrieves ALHs that are only slightly higher and fits AOTs at values are slightly lower in comparison to those from the formal approach. The retrieved heights from both methods are compared to lidar profiles from the EUMETNET ACL network of ceilometers. The comparison shows that both methods retrieved heights that are within the profiles that could be associated with aerosol layers. Analysing a radiosonde profile of the relative humidity and calculated back trajectories, it is observed that the ceilometer profiles miss higher aerosol layers due to attenuation of the signal at lower atmospheric levels. This explains why the retrieved heights from the dynamic scaling method are slightly higher than the same from the formal approach.

In general, the dynamic scaling method improves the number of converged pixels. Between the two discussed cases, the dynamic scaling method provides a better improvement in the 2010 Russian wildfires case. This is primarily because the method is scene dependent. An important driver that determines the improvement of retrievals is the level at which the surface influences the TOA reflectance, which is jointly influenced by two parameters – the surface albedo and the AOT. The average surface albedo of the scene for the 2010 Russian wildfires was observed to be brighter than the same for the 2017 Portugal wildfires. This is a possible explanation for the differences in the performance of the dynamic scaling method for the two cases.

The fitted AOT is systematically lower for the dynamic scaling method in comparison to the formal approach. A part of this can be attributed to the reduction in the influence of spectral points in the measurement with a larger influence from the surface albedo. While this is expected, the method does not necessarily make the fitted AOT more realistic. It may well be the influence of assumptions in aerosol properties such as aerosol single-scattering albedo and the phase function. It could, however, also be that the method does not fully remove the influence of surface in the measured top-of-atmosphere reflectance signal. In any case, the dynamic scaling method improves the representation of the fitted AOT of the MODIS Terra observed smoke plume.

The dynamic scaling method is designed to modify the signal-to-noise ratio to an extent that is necessary and sufficient to reduce the influence that photons travelling from the surface back to the detector have on the weighted least squares estimate of aerosol properties. Using the Jacobian to dictate the preference of weight least squares for spectral points in the measurement makes the dynamic scaling method a robust, generally applicable retrieval set-up. Results from this paper are applicable to other algorithms using weighted least squares techniques to retrieve atmospheric properties from measurements of top-of-atmosphere reflectance in the oxygen A band over bright surfaces.