Using neural networks for near-real-time aerosol retrievals from OMPS Limb Profiler measurements

Among aerosol characterization methods, limb scattering measurements provide both near-global coverage and information about how aerosol is vertically distributed through the atmosphere. Near-real-time retrievals of aerosols produced by volcanic eruptions are particularly important for aviation safety, but the radiative transfer modeling of scattering processes performed by traditional retrieval methods can be too computationally expensive for near-real-time applications without simplifying assumptions, depending on the instrument and available computational resources. Here we present a near-real-time approach based on neural networks (NNs) for aerosol retrievals from the Ozone Mapping and Profiler Suite's Limb Profiler (OMPS LP) instrument aboard the Suomi National Polar-orbiting Partnership satellite. We find it is at least 60 times faster than the current operational code and on average achieves agreement within 20 % at most altitudes and latitudes with sensitivity and non-negligible aerosol abundances. We also apply our trained NNs to measurements of the recent Shiveluch and Ruang eruptions from NOAA-21's OMPS LP and find results consistent with the operational retrieval algorithm, indicating our methodology generalizes to future iterations of the same instrument without retraining the NNs.

1 Introduction

Stratospheric aerosols play a key role in the atmospheric radiation budget. Negative radiative forcing due to background sulfate aerosols can slow the rate of increase in surface temperature (Solomon et al., 2011), and the increased aerosol load from eruptions like El Chichón, Pinatubo, and Hunga can lead to periods of increased outgoing shortwave radiation and cooler temperatures in the lower troposphere (Santer et al., 2014; Schoeberl et al., 2023). Major wildfires can trigger pyrocumulonimbus (pyroCb) events and cause stratospheric warming (Christian et al., 2019; Das et al., 2021). Characterizing these stratospheric aerosols therefore constrains climate and chemical modeling.

Stratospheric aerosols can be characterized through in situ, ground-based, and space-based measurements (see Sect. 4 of Kremser et al., 2016 for a detailed review). In situ (e.g., Junge et al., 1961; Hofmann et al., 1975; Jonsson et al., 1995; Brock et al., 2000; Hermann and Wiedensohler, 2001; Borrmann et al., 2010) and ground-based lidar (e.g., Fiocco and Grams, 1964; Poole and McCormick, 1988; Osborn et al., 1995; Barnes and Hofmann, 1997; Jäger, 2005; Chouza et al., 2020) measurements can provide high vertical resolution at the expense of limited spatial coverage. Space-based measurements, on the other hand, offer much broader spatial coverage. Solar occultation measurements (e.g., McCormick et al., 1982; Chu et al., 1989; Hervig et al., 1996; Hayashida et al., 2000; Lumpe et al., 2002; Thomason and Taha, 2003; McElroy et al., 2007) allow for the highest precision and accuracy among space-based methods and provide information about the vertical distribution of aerosols, but are only possible during sunrise/sunset and thus have limited spatial and temporal coverage. Space-based nadir lidar measurements (e.g., Vernier et al., 2011; Kar et al., 2019) provide better spatial and temporal coverage, but the measurements typically have low signal-to-noise ratios for stratospheric aerosol, limiting their precision. Additionally, they measure polarization and aerosol backscatter coefficient rather than aerosol extinction, and they have more significant power requirements compared to other space-based methods. Limb-scattering (LS) measurements (e.g., Llewellyn et al., 2004; Rault and Taha, 2007; Taha et al., 2011; Loughman et al., 2018; von Savigny et al., 2015) serve as a middle ground among space-based methods, providing near-global coverage of the vertical distribution of aerosols with lower cost and power requirements compared to other measurement techniques.

The Ozone Mapping and Profiler Suite (OMPS) Limb Profiler (LP) instrument (Flynn et al., 2006) was launched aboard the Suomi National Polar-orbiting Partnership (SNPP) satellite in 2011 and subsequently aboard the NOAA-21 satellite in 2022. OMPS LP measures LS radiances between 290 and 1000 nm with a 1 km vertical sampling through three slits; the lines of sight for the left and right slits are separated by approximately 4.25° with respect to the center slit. On SNPP, measurements are made of the sunlit limb 180 times per orbit for 14–15 orbits per day.

Taha et al. (2022) introduced version 2.1 of the OMPS LP aerosol retrieval algorithm, which improved on the convergence criteria used in version 2.0 (Taha et al., 2021). It uses the Gauss–Seidel LS radiative transfer model (Loughman et al., 2004) to retrieve the aerosol extinction coefficient up to 38.5 km at wavelengths of 510, 600, 675, 745, 869, and 997 nm. Each wavelength is retrieved independently. For more details, see the aforementioned references.

Near-real-time (NRT) characterization of aerosols is particularly important for informing aviation flight paths in the vicinity of a recent volcanic eruption, as well as providing advance notice to coordinate ground-based or in situ follow-up measurements. Aerosol retrieval algorithms like that of Taha et al. (2022) provide parallelizable, deterministic, physics-based results, but their runtime can be prohibitive to NRT applications unless compromises are made, such as retrieving at a reduced number of wavelengths and/or slits, depending on the available computational resources. In the case of the NASA Atmospheric Composition Processing System used to produce the NASA OMPS LP aerosol product, the available computational resources result in just over 2 h to process one SNPP orbit (and more than double that for NOAA-21 orbits), not including the time to downlink the data and process it into the Level 1 Gridded (L1G) radiance product. At present, this does not meet NASA's prevailing NRT definition of within 3 h of the observations. While these runtimes can be reduced by newer computing hardware, processing speed improvements can be offset by updates to the realism of the radiative transfer and retrieval models (e.g., tomographic retrievals as in Zawada et al., 2018).

Deep learning, a subfield of machine learning that focuses on neural networks (NNs; Goodfellow et al., 2016), offers a potential solution to this problem. Given a set of corresponding input–output pairs, NNs can learn to approximate complex, nonlinear relationships between inputs and outputs via a sequence of weighted transformations, without knowledge about the underlying physical processes that connect the inputs and outputs. This can be particularly useful to approximate a computationally expensive algorithm (e.g., Baydin et al., 2019; Cranmer et al., 2020; Munk et al., 2020; Kasim et al., 2021; Himes et al., 2022), such as aerosol retrievals from limb-scattered radiances. The trained NN model introduces some error in the outputs, but provided that the error is small enough for the desired applications, the resulting NN can be used in place of the computationally expensive algorithm. Werner et al. (2023) recently demonstrated this for NRT products from the Aura Microwave Limb Sounder; they found that NNs trained on the level 2 products generally outperformed the optimal estimation-based NRT product.

Here we present an NRT data product for retrieved aerosol extinction and related parameters from OMPS LP measurements based on NNs. In Sect. 2 we detail our methodology. We present comparisons with the existing aerosol data product and discuss our results in Sect. 3. Finally, we present our conclusions in Sect. 4.

2 Methods

In this study, we seek to train NN models that can accurately retrieve the aerosol extinction coefficient from SNPP OMPS LP measurements.

2.1 Data selection

Much of the OMPS LP record measures low stratospheric aerosol loading; elevated stratospheric aerosol from major volcanic and wildfire events can last for months to years, depending on the amount of aerosol produced. We utilize measurements during specific periods between October 2013 and December 2022 (Table 1). The periods were determined from Fig. 10 of Peterson et al. (2021) to capture a quiescent baseline as well as major volcanic and wildfire events that have occurred during SNPP's lifetime. We randomly select 10 % of these dates, totaling 1.24 million profiles spread throughout 241 d, to form the data set that will be used for our machine learning approach.

Table 1 Date ranges of OMPS LP data considered for the machine learning data set.

Notes. We select 10 % of the above dates for our machine learning data set, totaling 241 d, which is subsequently split into training, validation, and test sets (see Sect. 2.2). We specifically exclude the dates where OMPS LP initially measured the plume from the Hunga eruption from the training data so they can be used as test cases.

Utilizing these data, we prepare the input–output pairs to use during NN training. For each case, the inputs are comprised of

• the version 2.6 OMPS LP gridded sun-normalized measured radiances (Jaross, 2023) between 0.5 and 40.5 km, with altitude normalization at 38.5 km;

• atmospheric pressure and temperature from the NASA Global Earth Observing System Forward Processing for Instrument Teams product (GEOS FP-IT; Lucchesi, 2015);

• single-scattering angle (calculated from quantities available in the L1G radiance data product);

• solar zenith angle; and

• spacecraft-centered solar azimuth angle.

These parameters were selected given their use in the operational aerosol retrieval algorithm and availability in the L1G version 2.6 data product and the associated ancillary files. Each case within the data set is comprised of the above listed inputs paired with the corresponding aerosol extinction coefficient reported in the OMPS LP aerosol retrieval version 2.1 data product (V2.1; Taha et al., 2022). For the radiances and aerosol extinction coefficients, we consider wavelengths of 510, 600, 675, 745, 869, and 997 nm, as in the V2.1 product. At altitudes where the aerosol extinction coefficient was not retrieved, we assume a value of 10⁻⁸ km⁻¹, which is effectively treated as zero aerosol extinction in V2.1. This replacement is required as the NNs cannot handle missing data. During post-processing, any values predicted by the NN to be less than 10⁻⁸ km⁻¹ are replaced with a fill value of −999, for consistency with the V2.1 data product. To determine the altitude of clouds and aerosols, we utilize an updated version of the detection algorithm of V2.1, since it is already sufficiently fast and does not require further speed improvements from ML.

When preparing the input–output pairs, slit position is not included as a parameter or dimension. The resulting trained NN can therefore be applied to measurements from any of the three slits, thereby minimizing the up-front computational costs of training the NN and enabling retrievals in situations where only one or two slits have valid measurements.

We highlight that the current OMPS LP aerosol retrieval algorithm (V2.1) uses the version 2.5 radiances, while here we use the version 2.6 radiances as input because it is the only version of radiances processed in NRT. This complicates the relationship between inputs and outputs, as the version 2.6 radiances use different tangent height and stray light corrections. These correction methods introduce differences in the retrieved aerosol extinction coefficient, and so our input–output pairs do not have a perfect one-to-one relation. However, at the time of this work, no NASA OMPS LP aerosol retrieval version uses the version 2.6 radiance data as input, and retroactively processing the OMPS LP record using that version is computationally prohibitive. Since the differences in correction methods can be viewed as different transformation functions applied to the same underlying data, our methodology ignores them and assumes that the NN will learn to perform the transformation from version 2.6 radiances to V2.1 aerosol extinction coefficients. To confirm the validity of this assumption, we processed a limited number of retrievals on L1G version 2.6 and trained NNs as described below (see the end of Sect. 3.1 for a discussion of these results).

2.2 Neural network training and evaluation

The aforementioned 241 d of data are split into training, validation, and test sets. We enforce that the days which contain extrema for each input/output parameter must be within the training set. The remaining days are randomly split among the training, validation, and test subsets, in a proportion of approximately 70 %, 20 %, and 10 %, respectively. To pre-process these data and train the NNs, we use the open-source Python package MARGE (Himes et al., 2022), which is built on TensorFlow (Abadi et al., 2016). We normalize these data by taking the base-10 logarithm of the radiances, pressure, and aerosol extinction, then scaling each input and output parameter to be within the closed interval $[-\mathrm{1},\mathrm{1}]$ based on their training set extrema.

To determine NN architectures which are well suited to solve the problem of interest, we perform a Bayesian hyperparameter optimization. We consider models with two to five convolutional and/or fully connected hidden layers. The convolutional layers are allowed to have 2^x feature maps with $\mathrm{6}\le x\le \mathrm{8}$, while the fully connected layers are allowed to have 2^y nodes with $\mathrm{6}\le y\le \mathrm{10}$. These parameters are related to the overall complexity that can be captured by the NN. For each hidden layer, we consider rectified linear unit (ReLU), exponential linear unit (ELU), leaky ReLU (LReLU), sigmoid, and hyperbolic tangent (tanh) activation functions, which introduce nonlinearities into the model such that it can approximate complex behaviors. For ELU and leaky ReLU activation functions, we allow their free parameter to vary between 0.01 and 0.6. For all models, the input and output layers have linear activations, and the output layer is a fully connected layer with a number of nodes based on the target outputs (6 wavelengths × 41 altitudes). During the Bayesian optimization, each considered model is optimized according to the validation mean-squared-error (MSE) loss and trained for 60 epochs (number of iterations over the data set; relates to model convergence) using the Adam optimizer (controls how the NN weights and biases are updated), a batch size of 256 (number of samples considered in each training iteration; relates to variance in the gradient and thereby how the model learns), and a cyclical learning rate (scaling factor for magnitude of model updates; cycling reduces number of epochs needed to train model to a certain performance, e.g., Smith, 2015; Himes et al., 2022) that linearly cycles between 10⁻⁵ and 10⁻³ over 12 epochs.

Based on this optimization, we selected an architecture with hidden layers of Conv3D(256)–LReLU(0.0733)–FC(1024)–tanh–FC(512)–ELU(0.2944)–FC(1024)–ReLU, where Conv3D(m) denotes a three-dimensional convolutional layer with m feature maps, LReLU(s) denotes a leaky ReLU activation with slope parameter s, FC(n) denotes a fully connected layer with n nodes, tanh denotes a hyperbolic tangent activation, and ELU(α) denotes an ELU activation with scaling parameter α. We note that while different architectures result in variations in model performance, many different architectures can achieve similar performance; the training data, its pre-processing, and the training methodology had a more significant effect on the resulting model accuracy.

Using the selected architecture, we train two separate NNs, as we found poor performance at 510, 600, and 675 nm in the southern hemisphere, where OMPS LP observes backscattered radiation, when using a single NN. For each NN, we determine a cyclical learning rate range based on the range test described in Himes et al. (2022). Unlike in Himes et al. (2022), here we utilize real data during NN training; our problem requires that the minimum learning rate must be increased above that determined via the range test described in Himes et al. (2022) to avoid overfitting the validation data. We train one NN using a cyclical learning rate varying between $\mathrm{3}\times {\mathrm{10}}^{-\mathrm{6}}$ and 10⁻³, and an MSE loss; it is used for the northern hemisphere at 675 and 745 nm and for all latitudes at 869 and 997 nm. We train the second NN with a cyclical learning rate over the range of 10⁻⁵–$\mathrm{2}\times {\mathrm{10}}^{-\mathrm{3}}$ and a custom loss function which minimizes the maximum MSE across wavelength channels; it is used for all latitudes at 510 and 600 nm and in the southern hemisphere for 675 and 745 nm. We allow each NN to train until engaging early stopping after a patience of 60 epochs. To retrieve on one orbit predictions are made with both models, and then the aforementioned relevant subsets of predictions from each model are combined.

To thoroughly evaluate the NN performance, we compare the retrieved extinction coefficient and the stratospheric aerosol optical depth with the current OMPS LP aerosol data product over January 2019–December 2023 and compute a mean percent error with respect to altitude and latitude. We also test the NN predictions for the 2022 Hunga eruption to ensure the models can be applied to extreme cases beyond those seen in the training set, as well as the 2023 Shiveluch and 2024 Ruang eruptions to ensure the models are accurate when applied outside the range of years considered in the training data set. We also utilize the 2024 Ruang eruptions to demonstrate how visualizations from our NRT product can accurately track the transport of a volcanic plume. Finally, given that OMPS LP is planned to launch on multiple additional satellites over the next decade, we briefly consider the NN performance when applied to selected orbits from NOAA-21's OMPS LP that measured the recent Shiveluch and Ruang eruptions to investigate the applicability of our SNPP NRT approach to OMPS LP measurements in general.

3 Results and discussion

3.1 Comparison with OMPS LP Version 2.1 aerosol data product

Figure 1 summarizes the difference in zonal means between the NRT and V2.1 aerosol data products over the period January 2019–December 2023. This period is marked by a series of volcanic eruptions and massive wildfires that injected large amounts of aerosols into the stratosphere in both hemispheres. Errors are generally within 20 % in regions with non-negligible aerosol extinction. The notable exceptions to this are lower altitudes in the southern hemisphere at shorter wavelengths and in the troposphere, where the percent errors increase to >50 %. The former regime generally has a small aerosol scattering index, which causes the retrieval algorithm to struggle to provide consistent results (Taha et al., 2021); consequently, the NNs also struggle in this regime. The large differences in the troposphere are mostly caused by cloud interference, which may cause inaccurate retrievals for both algorithms.

Figure 1 Plot of the mean differences in percent between OMPS LP NRT and V2.1 aerosol extinction profiles for the center slit at (a) 510, (b) 600, (c) 675, (d) 745, (e) 869, and (f) 997 nm, zonally averaged at 5° latitudes for the period between January 2019 and December 2023. The dashed line is the zonal mean GEOS FP-IT tropopause altitude. Contours are shown for ±20 % errors. Errors $<-\mathrm{50}\mathit{\%}$ are shown in white; this occurs where aerosol extinction is typically negligible (upper altitudes) and where the aerosol retrieval algorithm struggles to provide consistent results (lower altitudes in the southern hemisphere at shorter wavelengths; Taha et al., 2021).

Figure 2 (a) OMPS LP V2.1 daily stratospheric aerosol optical depth (sAOD) at 997 nm for the center slit, calculated in 5° latitude bands by integrating the aerosol extinction from the tropopause altitude to 37.5 km. (b) The same but for NRT. (c) The daily zonal differences between (a) and (b) in percent. Black markers indicate volcanic eruption or pyroCb smoke plumes.

Figure 3 (a) OMPS LP V2.1 daily 20.5 km retrieved aerosol extinction at 997 nm for the center slit, calculated in 5° latitude bands. (b) The same but for NRT. (c) The daily zonal differences between (a) and (b) in percent.

Figures 2 and 3 show the stratospheric aerosol optical depth (sAOD) and aerosol extinction at 20.5 km, respectively, at 997 nm between January 2019 and December 2023 for the V2.1 and NRT aerosol data products. The NRT results generally agree with V2.1, capturing increases in aerosol loading due to both eruptions and wildfires during this period. For example, aerosol due to the Raikoke eruption and northern hemisphere wildfires in 2019 are clearly visible in Fig. 2; the NRT product correctly infers the decrease in sAOD over time as well as the transport southward into the mid-latitudes. Similarly, the increased sAOD throughout 2022 associated with the Hunga eruption agrees both in the initial months after the eruption and later in the year when the aerosol was transported toward the southern pole. Most errors are within 20 %, consistent with Fig. 1. Some periods have larger percent errors, which are mostly associated with small sAOD and extinction values, such as the period leading up to the Ulawun eruption in 2019. We similarly see larger percent errors during the southern polar vortex, which is attributable to the small aerosol extinction within the vortex; outside the vortex, we find that biases are generally consistent with the temporal mean.

The Hunga eruption is the only event in this five-year period where the NRT product consistently underpredicts the extinction at 20.5 km at 997 nm (Fig. 3) compared to V2.1. The NRT results show a bias up to −50 % between May–September 2022 and November 2022–January 2023; the latter period corresponds to transport into the northern hemisphere. This is likely attributable to the extreme nature of the event. Despite this, the NRT predictions for Hunga are qualitatively consistent with V2.1, and errors are generally within 20 %.

Considering results for individual orbits, the NRT results generally agree with V2.1, capturing similar structures and magnitudes across the orbit. Figure 4 compares the V2.1 and NRT results at 997 nm for the OMPS LP orbit on 16 January 2022 that first measured Hunga's plume. This orbit features aerosol extinctions that are more extreme than those included in the training data set. Though it slightly overestimates the extinction below the plume and below the cloud tops in the southern hemisphere, the extinction values near the top of the plume are consistent and the plume height agrees well. Figure 5 shows the V2.1 and NRT results at 997 nm for the center slit of the OMPS LP orbit that measured Shiveluch shortly after the eruption on 11 April 2023. While a relatively small eruption, Shiveluch highlights the importance of NRT aerosol retrievals: numerous flights to and from Alaska were canceled due to volcanic ash at airplane cruising altitudes. Despite being outside the temporal range of data that trained the NNs, the NRT approach agrees with V2.1.

Figure 4 (a) OMPS LP V2.1 retrieved aerosol extinction at 997 nm for the center slit of orbit 52958 on 16 January 2022, measured shortly after the Hunga eruption. (b) The same but for NRT. The plume is visible around 20° S latitude, reaching roughly 30 km in altitude.

Another example is the Ruang volcano in North Sulawesi, Indonesia that erupted on 16 April 2024 and continued intermittently through 23 April. The largest emissions occurred on 17 April, and plumes reportedly reached up to almost 20 km and generally drifted west of the island (Global Volcanism Program, 2024). Late on 29 April, Ruang began erupting again, with OMPS LP measuring the plume reaching up to 23.5 km. Figure 6 shows the average retrieved extinction coefficient between 19.5 and 21.5 km at 997 nm for the operational and NN-based algorithms for all orbits on 1 May. Aerosol from the first eruption can be seen stretching from eastern Africa to the eastern Pacific near the coast of South America, while aerosol from the second eruption can be seen over Borneo and the Indian Ocean. We also processed the orbits between 16 April and 13 May to produce a time-lapse animation of the average retrieved aerosol extinction between 19.5 and 21.5 km; see the video supplement (Himes et al., 2024b). This visualization represents one use case of our NRT product, enabling the tracking of volcanic plumes and (in combination with wind fields) predicting their future transport, e.g., to coordinate balloon-borne measurements or inform aviation flight paths.

Figure 5 (a) OMPS LP V2.1 retrieved aerosol extinction at 997 nm for the center slit of orbit 59342 on 11 April 2023, measured shortly after Shiveluch's eruption. (b) The same but for NRT. The plume appears above the tropopause at approximately 57° N latitude.

When analyzing a given orbit on our processing system using both algorithms, the NRT approach requires ∼ 2 min, achieving a $\sim \mathrm{60}\times$ speedup in terms of wall-clock time compared to V2.1 while utilizing fewer computational resources. This speedup metric does not include the ∼ 13 h spent training each model on an NVIDIA V100 graphics processing unit; aside from requiring specialized hardware, this total runtime is comparable to the time required to process 1 d of data with the standard physics-based algorithm. The NN-based NRT approach significantly reduces the delay between obtaining and processing the data, which is particularly important for aviation safety in the wake of a volcanic eruption. Much of the NRT processing time is spent loading the NN models into memory; typically a total of ∼ 20 s is spent predicting the aerosol extinctions using just eight cores on our processing system. Thus, processing a range of orbits where the NN models can remain loaded in memory can substantially increase this speedup factor (e.g., processing 10 orbits would result in approximately 400× less computational time).

Figure 6 (a) V2.1 average retrieved aerosol extinction coefficient between 19.5 and 21.5 km at 997 nm for 1 May 2024, which shows aerosol from Ruang's recent eruptions stretching from Borneo to the eastern Pacific Ocean. (b) The same but using the NRT algorithm. Individual points are colored by the mean extinction coefficient value at each measurement location.

When developing this data product, we originally attempted to train a single NN to retrieve aerosol extinction for the entire orbit by minimizing the MSE. However, we found that this led to the NN only performing accurately in the northern hemisphere and at longer wavelengths, with strong biases in the southern hemisphere and shorter wavelengths. This is likely due to a combination of increased Rayleigh scattering at shorter wavelengths and OMPS LP's Sun-synchronous orbit and viewing direction. As OMPS LP ascends from the south pole towards the north pole, the single-scattering angle changes throughout the orbit, resulting in measurements of backscattering in the southern hemisphere and forward-scattering in the northern hemisphere. This leads to a weaker aerosol scattering signal in the southern hemisphere, which can be on the order of or smaller than the Rayleigh scattering signal. To address this, we considered custom loss functions; we found that separately calculating the MSE per wavelength channel and minimizing the maximum MSE across wavelengths significantly improved performance in the southern hemisphere and at shorter wavelengths, but the northern hemisphere and longer wavelengths generally performed worse compared to just minimizing the MSE. Consequently, we chose to use a combination of those models to achieve acceptable performance in both hemispheres at all wavelengths.

We also considered a more traditional surrogate modeling approach by applying the aerosol retrieval code to the L1G version 2.6 radiances for the same 241 selected days and training the NNs on these data; it achieved similar performance to that presented here (not shown). This confirms our approach's implicit assumption that the NNs can learn to handle the minor differences in corrections applied to the radiances between versions 2.5 and 2.6, circumventing the need to reprocess the aerosol retrievals after future updates to the radiance data product.

3.2 Limitations

Our NRT approach is subject to two types of limitations: those inherent in the operational retrieval code that produced the training data, and those specific to the NN approach used here. The operational retrieval code generally struggles to provide consistent results at high altitudes where aerosol abundances are small, and in the southern hemisphere due to the aforementioned weak aerosol backscattering signal and increased scattering at shorter wavelengths. It also struggles when aerosol reaches the normalization altitude of 38.5 km, which happened with Hunga (Taha et al., 2022). Readers interested in more details on these limitations are directed to Taha et al. (2021, particularly Fig. 7) and Taha et al. (2022). For high altitudes and in the southern hemisphere, the NRT algorithm does not always yield results consistent with V2.1; it tends to predict small aerosol values. In the case where aerosol reaches the normalization altitude, the NRT algorithm produces artifacts in the retrieved extinction not seen in V2.1, but this situation can be identified by the reported enhanced aerosol altitude occurring at or above the normalization altitude.

The most significant NN-specific limitations are related to the input data. Our methodology relies on a diverse data set that samples low, moderate, and extreme aerosol loading in order to train NNs to correctly interpret the radiances under these conditions. These trained NNs are only applicable to OMPS LP; they cannot be reliably applied to data from any other instrument. As future improvements are made to the gridded radiances data product, these NNs may need to be retrained. Similarly, future improvements to the aerosol retrieval code may necessitate reprocessing the selected dates used to train the NNs, then retraining the NNs to predict those data. However, these situations may require only retraining the weights associated with the relevant layer (transfer learning), reducing the computational cost of training updated models. Future work should explore this possibility in detail.

More broadly, these NNs are limited by the extrema encountered in the training data. This limits the applicability of this methodology to recently launched instruments due to a lack of diverse data records that sample multiple major eruptions and wildfire events in both hemispheres. While the Hunga case presented above demonstrates that the NNs can be applicable to more extreme radiances and extinctions than those encountered during training, this does not necessarily hold true for all anomalous events, nor for the other input parameters.

Figure 7 (a) NOAA-21 OMPS LP retrieved aerosol extinction at 869 nm for the right slit of orbit 2153 on 11 April 2023, measured shortly after Shiveluch's eruption. (b) The same but using the SNPP NRT algorithm. The plume appears above the tropopause at approximately 57° N latitude.

Figure 8 (a) NOAA-21 OMPS LP retrieved aerosol extinction at 997 nm for the left slit of orbit 7618 on 30 April 2024, which measured the plume from Ruang's second major eruption that month over Borneo. (b) The same but using the SNPP NRT algorithm. Both algorithms found that the plume reached a height of approximately 23 km, with similar magnitudes for the extinction throughout the plume as well as background aerosol throughout the orbit.

3.3 Application to NOAA-21 OMPS LP

Our methodology is challenged when applied to a new instrument due to the short data record that has likely not measured sufficiently diverse conditions to train generalized NNs. NOAA-21 was launched in November 2022; the OMPS LP instrument onboard has been operational since February 2023 and thus represents an ideal test case to determine whether the SNPP NRT model can be reliably applied to NOAA-21 data, thereby circumventing the data record limitation and enabling our method to be applied to new instruments in situations where our NRT approach has already been applied to an earlier version of the same instrument.

When applying the NN models trained using SNPP retrievals to select NOAA-21 orbits, we find that they agree qualitatively and quantitatively with the operational results, with biases that are broadly in agreement with Fig. 1. For more detailed comparisons, we present two cases. Figure 7 shows the operational and NN-based retrieval at 869 nm for the NOAA-21 orbit that initially measured the plume from the Shiveluch eruption. The NN retrieves aerosol extinction profiles that are consistent with the operational retrieval algorithm, both for background stratospheric aerosol as well as Shiveluch's plume. The Ruang eruptions have been the most significant events measured by NOAA-21's OMPS LP to date and thus represent an ideal test case to ensure our SNPP-trained NNs are applicable to major events measured by NOAA-21. Figure 8 shows the retrieved aerosol extinction coefficient at 997 nm for the operational and NN algorithms for the NOAA-21 orbit that measured the plume on 30 April from Ruang's eruption on the prior day. The inferred plume height (∼23 km) and retrieved extinction values agree well throughout the orbit.

Despite the calibration of the NOAA-21 data not being finalized yet, these results suggest that the methodology presented here can produce NNs that are applicable to other OMPS LP instruments beyond SNPP without the need to retrain the models. For cases where there are future iterations of the same instrument, our methodology can circumvent the limiting requirement of a long, diverse record to train generalized NNs by utilizing measurements from the older instrument.

4 Conclusions

We presented a near-real-time (NRT) aerosol retrieval product for OMPS LP based on neural networks (NNs). We found good agreement with OMPS LP's operational version 2.1 (V2.1) aerosol retrieval product, achieving zonal-mean errors that are generally within 20 % at altitudes with sensitivity and non-negligible aerosol extinction. The NRT product exhibits the smallest biases in zonal means at 997 nm, with 869 nm providing similar accuracy but with slightly higher biases at low altitudes in the southern hemisphere. Given that Taha et al. (2021) recommend focusing on 869 and 997 nm for scientific studies using the operational product, we similarly recommend that users focus on these wavelengths when using the NRT product, particularly 997 nm given its more consistent performance.

The NRT approach is $\sim \mathrm{60}\times$ faster than V2.1, requiring roughly 2 min to retrieve the extinction for an entire orbit. More quickly providing information on volcanic plume height after an eruption can be especially useful for groups like the Volcanic Ash Advisory Centers to inform aviation flight paths, as well as providing advance notice to coordinate ground-based or in situ follow-up measurements just after large eruption or pyroCb events. With around 97 % of the runtime spent loading the NNs into memory, this speedup factor can grow by more than an order of magnitude when processing a range of orbits. We found it necessary to train separate NNs for the southern hemisphere / short wavelengths and northern hemisphere / long wavelengths in order to achieve more consistent performance at all wavelengths and latitudes. If a single NN could reliably perform in both hemispheres at all wavelengths, the factor speedup would double; future work should explore approaches to achieve this. Additionally, future work should consider how including ozone profiles among the NNs' inputs impacts the resulting accuracy, particularly at the shorter wavelengths considered in this study, which are sensitive to ozone.

Our approach has two types of limitations: those related to the aerosol retrieval code, and those related to the NN approach. The limitations associated with the aerosol retrieval code inherently cannot be circumvented by our methodology; the NNs similarly struggle in situations where the retrieval code struggles to provide a consistent result or where it cannot retrieve the extinction (e.g., when the aerosol plume reaches the normalization altitude; Taha et al., 2022). The NN-specific limitations are associated with the training data. In general, these NNs cannot be reliably applied to other instruments or different versions of radiances/extinctions without retraining; future work should explore applications of transfer learning to reduce the burden of updating the models as new versions of radiances/extinctions are developed. While applying NNs to data beyond those encountered during training can produce poor results, we found sufficient accuracy when applied to the Hunga eruption, which was more extreme than any events included in the training data set. The NRT approach also accurately identified the plumes for the recent Shiveluch and Ruang eruptions, which occurred outside the temporal range covered by the training data. Our results indicate that the NNs generalized for the problem and can be applied to future events measured by OMPS LP.

We also found that the NNs trained on SNPP data are consistent with the operational retrieval algorithm when applied to data from NOAA-21's OMPS LP without retraining. With multiple successors to SNPP and NOAA-21 planned over the next decade, future work should explore this in more detail, as it offers a way to circumvent the methodology's requirement of an extensive data record to apply our NRT approach.