We developed, trained, and evaluated a ML model to simultaneously predict a variety of TROPESS CO variables, primarily using observed CrIS radiances and geolocation data as inputs. This setup is illustrated in the simplified diagram in Fig. b, where we drastically limit the input and output variables to aid visibility. In this example the model uses three features (F1–3) as input: CrIS radiances at 2181.88 cm-1, the sensor viewing angle, and the surface altitude, respectively. These features are matrices, where each element f1–3s corresponds to one of the N samples, indexed as s=1,…,N. The ML model maps these features to a set of output labels (L1–3), which in this simplified example are the CO total column concentrations, the total column retrieval error, and the column AK at ≈511 hPa. Like the features, these labels are matrices that contain elements l1–3s for each individual sample. Again, s=1,…,N denotes the individual sample (i.e. CrIS column).

The ML model developed in this study is a feedforward artificial neural network (ANN), which maps the input to the output through several hidden layers, each consisting of a large number of interconnected neurons. A simplified schematic of an example ANN, with two hidden layers containing 7 and 5 neurons, respectively, is shown in Fig. c. This diagram also illustrates how the geolocated features are transformed into two-dimensional input and output matrices and how they connect to the individual neurons.

The exact model structure and hyperparameters (i.e. model settings) are determined through the procedures described in . By applying k fold cross-validation across a range of potential model setups, the ideal hyperparameters were found to be two hidden layers with 1506 neurons per layer, “Rectified Linear Unit” activation functions after each hidden layer, an L2 weight decay parameter of 5.00×10-34, and the “Adaptive Moment Estimation” optimizer with a learning rate of 1×10-5. The loss function minimized during training is the mean squared error. For each training iteration, batches of samples are passed through the model in a forward pass to compute predictions, followed by a backward pass in which model weights are updated via backpropagation. Each mini-batch contains 8192 samples. Further details on these parameters and their impact are provided in , , and .

Model training was carried out using the “Keras” library for Python (version 2.10.0; ), with “TensorFlow” (version 2.10.0) as the backend . Of the available CrIS radiances and TROPESS retrievals over April 2023–January 2025, 98% of randomly selected samples were used as training data. After each training iteration, the model's performance was evaluated for an independent validation dataset comprised of 1% of the available data (approximately 185 000 samples). After several thousand iterations, the model weights corresponding to the best performance scores on the validation set were saved.

The specific features used for the CO model include radiances from all 2224 spectral channels, the FOV index, the latitude and longitude of each sample, UTC time, a day/night flag, the sensor viewing angle, the day of the year, and the TROPESS subcolumn a priori values. This yields an input matrix containing 2235 variables. Note that the surface altitude was included for models predicting retrievals and diagnostics for other TROPESS species. We tested reduced channel sets focused on CO-sensitive regions but found that using the full spectrum provided slightly improved performance, likely due to additional information on atmospheric state variables (e.g. temperature and humidity). The predicted labels of the CO model consist of the subcolumn concentrations, column AKs, and subcolumn retrieval errors, resulting in an output matrix containing 24 variables.

Prior to training, these inputs and outputs were filtered to remove invalid samples using a set of basic quality filters, including non-finite values, fill values, extreme outliers, failed retrieval quality flags, and target values outside the valid retrieval range. In addition, extreme outliers in the label distributions were masked using percentile-based tail cutoffs, and both input features and output labels were standardized before training.

Model training was performed on a high-performance computing cluster and took ≈10 d to converge to a solution for the >12000000 model weights.

Model performance is evaluated using an independent test dataset, which consists of the remaining 1% of randomly sampled data that were not included in the training or validation process. Ideally, (i) the model should reliably predict CO concentration retrievals and OE diagnostics for these data points, even though the ML algorithm was not trained on them, and (ii) performance metrics should be similar to those derived from the training and validation datasets. It is important to note that the objective of the ML model is not to generalize beyond the statistical characteristics of the OE retrieval, but to emulate the OE solution and provide full spatial coverage for the same observing system. As such, the random split into training, validation, and test datasets ensures that the model is evaluated on samples that were not explicitly used during training, while still reflecting the same underlying distribution of atmospheric states and observational conditions. Given the strong spatial and temporal correlations inherent in satellite observations, this approach is appropriate for assessing the model's ability to reproduce OE retrievals across the full range of conditions encountered in the dataset. In operational use, the model is continuously retrained with newly available OE retrievals, ensuring consistency with evolving atmospheric variability.

Figure a presents a joint histogram of total column CO from the ML and OE algorithms for over 180 000 samples in the independent test dataset. Yellow colors represent regions with the highest density of data points, while blue colors correspond to areas with very few samples. The good agreement between the ML and OE results is evident, as most observations are narrowly clustered around the 1:1 line (indicated by the gray, dashed line). Five performance metrics are provided in the panel: Pearson's product-moment correlation coefficient (r), the root-mean-square deviation (RMSD), the median deviation between the predicted and retrieved CO (50p, i.e. the bias), and the 1st and 99th percentiles of the deviation. Notably, for the total column concentrations, we find r>0.99, RMSD=3.11×1016 moleculescm-2, a median difference of -7.27×1014 moleculescm-2, and maximum absolute differences for the majority of samples of <1.00×1017 moleculescm-2.

Similar comparisons for tropospheric column concentrations, total AK at 162 hPa, and DoF are shown in Fig. c, e, and g. Again, the distributions closely follows the 1:1 line, with similarly high correlations (r>0.99). The lowest correlation occurs for the column AK at the lowest atmospheric level (not shown), where r=0.98. These performance metrics are almost identical to those obtained for the validation dataset, where the comparison of predicted and retrieved total column CO concentrations yields r>0.99, RMSD=3.14×1016 moleculescm-2, a median difference of 6.57×1014 moleculescm-2, and maximum absolute differences for the majority of samples of <1.00×1017 moleculescm-2.

Performance metrics are consistent across training, validation, and test datasets, with nearly identical correlation coefficients (|Δr|<0.0006), normalized RMSD values (|ΔRMSD|<0.21%, apart from the last two column AK levels deep in the stratosphere where values of effectively 0), and biases (|Δbias|<0.12%) for all predicted variables. This indicates that there is no evidence of overfitting and that the model exhibits stable behavior across the available datasets.

We note that the use of a random split does not enforce strict independence between training, validation, and test datasets, as atmospheric states exhibit strong spatial and temporal correlations. In this study, the purpose of the split is therefore not to assess fully independent generalization, but to evaluate how well the model reproduces OE retrieval behavior across the distribution of atmospheric states and viewing geometries sampled by the instrument.

To ensure that the split remains representative, we verified that the distributions of key variables are statistically consistent across training, validation, and test datasets using Kolmogorov–Smirnov tests. Combined with the consistent performance metrics reported above, this indicates that the model generalizes well within the sampled observational distribution. This behavior is consistent with the intended hybrid OE–ML application, in which the model is designed to operate on the same observational distribution as OE.

To complement the regression analysis, we employ Bland–Altman plots to further assess the agreement between the OE and predicted results. This approach highlights systematic differences and potential heteroscedasticity that may not be apparent in standard correlation-based evaluations, especially for non-Gaussian or magnitude-dependent variability, and has been shown to provide a more robust framework for intercomparison of geophysical datasets e.g.. Panels b, d, f, and h show the Bland–Altman distributions, with the paired mean of predicted and OE values on the x axis and the difference between the two datasets on the y axis, along with three horizontal dashed lines. The central line denotes the mean difference (bias), while the outer lines show the 95% limits of agreement (mean ±1.96 standard deviations), indicating the interval that contains approximately 95% of the differences under the assumption of normally distributed residuals.

The distributions show no evidence of magnitude-dependent bias or systematic slope. The proportion of data points within the 95% limits of agreement (95%, 95%, 96%, and 95%, respectively) aligns closely with expectations for normally distributed residuals, and 79%, 80%, 86%, and 78% of points fall within ±1 standard deviation. This indicates well-behaved error distributions with no pronounced evidence of heavy tails or heteroscedastic spread, suggesting that the model adequately captures variability across the full dynamic range. This is particularly relevant for machine learning retrievals, which can otherwise struggle in the presence of heteroscedastic relationships between observables and geophysical variables e.g., and indicates that such effects are not evident in our ML predictions. Minor increases in spread near the peak of the distributions reflect regions of highest data density and do not indicate systematic bias or magnitude-dependent variability.

The close agreement between predicted and OE-derived diagnostics indicates that the ML model not only reproduces the retrieved state, but also captures the associated sensitivity and uncertainty characteristics of the OE solution. This consistency is critical for downstream applications, such as data assimilation and model evaluation, where the proper interpretation of retrievals depends on the availability of reliable AKs and error estimates.

Figure a presents a representative example scene of total column CO from the TROPESS OE retrieval on 10 June 2023. A large area of enhanced CO concentrations (>3.00×1018 moleculescm-2) is evident over Western Canada, associated with the unprecedented wildfire season that year . These fires produced large smoke plumes that affected portions of Canada and the US for several weeks, before spreading across the Northern Hemisphere. Notably, enhanced CO concentrations (>2.50×1018 moleculescm-2) are also recorded over Eastern Canada, the entire Eastern US, and parts of the Atlantic Ocean.

The associated OE DoF are shown in Fig. b. Areas of moderate to high CO concentrations generally coincide with regions of elevated DoF. Smaller DoF<0.6 are observed over Greenland, the Atlantic Ocean, and over isolated regions over the continental US. These reduced DoF are indicative of lower retrieval sensitivity and are likely due to the interference of clouds or poor thermal contrast.

Figure c and d show the ML predictions for total CO concentrations and DoF, respectively. These results are derived for each CrIS L1B sample. The increased spatial resolution is particularly noticeable over the oceans, but even over land the ML results capture much finer spatial features, while faithfully reproducing the CO enhancements and DoF from the OE retrieval. Divergence maps in Fig. e and f illustrate the differences between predicted and retrieved results. The median differences are <0.1% for both variables, and for the majority of samples (i.e. within the 5th and 95th percentiles), ML predictions are within ±2.40% for total CO concentrations and within ±4.12% for DoF. Overall, the difference between ML and OE total column CO concentrations exceeds the retrieval error for only 14 of the 5308 samples in the scene (0.26%). Similarly, excellent agreement is observed for the retrieval errors (not shown), with a majority of ML predictions within ±6.11% and a median difference of 0.04%.

The close agreement between the ML predictions and OE retrieval shown in Fig.  primarily reflects interpolation within the sampled observational distribution, as this day is part of the training period, rather than fully independent spatiotemporal generalization. To further assess the performance of the ML model beyond the training period, Fig.  shows a global scene for 8 June 2025, which lies outside the training range (April 2023–January 2025). This experiment represents a limited temporal extrapolation test rather than a comprehensive assessment of long-term model stability. The results demonstrate that the model retains strong predictive skill under these conditions, indicating that it can generalize to unseen temporal states to a certain extent. However, we emphasize that such standalone predictive capability is not the primary objective of the framework. The model is designed to operate as part of a hybrid OE–ML system, benefiting from periodic retraining and remaining closely tied to the evolving distribution of OE retrievals.

Similar to Fig. 3, the ML predictions reproduce the spatial structure of total column CO with high fidelity, including regions of enhanced concentrations associated with wildfire activity over North America. Differences between ML predictions and colocated OE retrievals remain small and spatially unstructured. The median difference is <0.13%, and the majority of samples (again, within the 5th and 95th percentiles) has ML CO concentrations within ±3.00% of the OE results.

In addition to column concentrations, Fig. 4 demonstrates that the ML model accurately reproduces key retrieval diagnostics. The column AK exhibit a maximum at 383 hPa and is shown in panels b, d, and f, showing strong agreement in both spatial structure and magnitude. The median difference is <0.67%, and the majority of ML predictions lie within ±8.50% of the OE results. The global mean AK profiles (Fig. 4g) are nearly identical between OE and ML, with differences well within the natural variability of the OE retrieval. Median differences over all vertical levels are within 0.001 % and 90% of predictions are within 0.028 of the true AK value at any level. This translates to 1.5% and 10%, respectively, at levels where the AK is noticeably different from zero, i.e. in the troposphere above the surface.

Likewise, differences in retrieval errors (Fig. 4h) and degrees of freedom (Fig. 4i) are centered near zero and exhibit narrow distributions, with full-width-at-half-maximum values of 0.02×1018 molec.cm-2 and 0.032, respectively. This behavior indicates that the ML-predicted diagnostics are statistically consistent with those derived from OE, within the intrinsic variability of the retrieval.

Together, these results demonstrate that the ML framework not only reproduces the retrieved state, but also captures the associated sensitivity and uncertainty characteristics of the OE solution for unseen atmospheric conditions, while resolving finer spatial structures beyond the native OE sampling. These characteristics suggest that the predicted diagnostics retain the key properties required for downstream applications such as data assimilation, although a full assessment within an assimilation framework is beyond the scope of this study.

A key question is whether the ML product captures physically meaningful CO variability below the nominal 0.80° TROPESS retrieval resolution, or whether it primarily behaves as a spatial interpolation of the OE field. While interpolation can reconstruct smooth fields between observations, it cannot introduce new information at unresolved spatial scales.

To investigate this, we employ two complementary approaches: (i) comparing the ML-predicted CO fields with linearly interpolated TROPESS CO retrievals, and (ii) analyzing the spatial power spectral densities EI(k) to identify scale-dependent variability, particularly in the sub-0.80° domain. Together, these approaches allow us to assess whether interpolation is sufficient to capture the underlying structure, or whether additional variability persists at smaller scales that requires the higher-resolution information provided by the ML model.

Figure a shows the interpolated OE CO retrievals. This field appears significantly smoother than the ML predictions (Fig. c), especially for the region of enhanced CO over Western Canada and the Northeastern US. The difference between the interpolated and predicted CO concentrations is illustrated in Fig. b, where blue and red colors indicate underestimation and overestimation by the interpolation, respectively. Deviations are centered around 0.50% but can exceed ±30%, especially in areas of enhanced CO. Maximum differences increase further, to ±48% and ±58%, when using nearest-neighbor or cubic spline interpolation, respectively.

While these differences highlight limitations of interpolation in regions of enhanced CO, a more general assessment of spatial variability across scales requires a spectral analysis. We therefore calculate power spectral densities EI(k), which describe how variance in a spatial signal is distributed across different wavenumbers (k). Since this analysis requires data on a regular grid, the ML-predicted CO concentrations are first interpolated onto a grid with constant spacing. As in the previous section, we focus on the total column CO field over North America on 10 June 2023, gridded at a resolution of 0.80°/6≈0.133° in both latitude and longitude. Because the CrIS L1B radiances and corresponding ML predictions are provided on a similar but irregular grid, nearest-neighbor interpolation is used to retain most of the native variability; for comparison, linear and cubic spline interpolations are also evaluated.

Many geophysical fields exhibit scale-invariant behavior over a large range of wavenumbers, with EI(k) following a power law: 1EI(k)∼k-β. Sudden changes in the slope β, so-called scale breaks, indicate changes in the physical processes governing variability. Such breaks have been reported in cloud-reflected radiances e.g., paleotemperature records e.g., and climate variability e.g.. We compute EI(k) as the squared amplitude of the Fourier-transformed CO predictions in both latitudinal and longitudinal directions.

Figure c and d present EI(k) averaged over all grid points in latitude and longitude, respectively. A scale break is observed at k≈-1.70, corresponding to spatial scales of ≈3.0–3.5°, in both directions. Linear fits before and after the break, shown in blue and orange, were computed using the octave binning method reported in , which mitigates noise and limits energy accumulation at small scales. The binned EI(k) values are plotted as black dots. At the scale break, the slope in latitude (longitude) flattens from β≈1.77 (1.77) to β≈0.49 (0.23), indicating a relative enhancement of small-scale CO variability. This transition is consistent with the de-correlation length scale of CO (not shown) and indicates a transition to enhanced small-scale variability. A detailed attribution of the underlying processes is beyond the scope of this study. Notably, no secondary break is observed at smaller scales, and no steeper decline in EI(k) is found below the operational TROPESS retrieval resolution, which would imply reduced variability and a smoother distribution. Instead, the CO field remains highly variable down to the Nyquist limit of 2⋅0.80°/6≈0.267° (derived from the effective sampling resolution).

Importantly, the observed scale break is neither an artifact of the retrieval nor dependent on the interpolation scheme. Applying the same analysis to CO-sensitive radiances in the spectral microwindow used in the OE retrieval (gray line in Fig. c and d) yields similar results: comparable scale breaks at k≈-2 and consistent β values. Changing the interpolation scheme from nearest neighbor to linear or cubic spline has minimal effect on the location of the break, though β values flatten slightly, reflecting increased variability across all scales. These minimal changes are not surprising, since the ML data are available at a very high spatial resolution (albeit on an irregular spatial grid).

For comparison, the power spectral density of linearly interpolated OE fields exhibits a similar large-scale slope (β≈1.70) but a much steeper decay of variability toward smaller spatial scales (β≈3.28), reflecting the smoothing inherent in interpolation. While the apparent scale break shifts slightly, its exact location is sensitive to the fitting procedure and binning choices and is therefore not interpreted further.

In summary, the results in this section demonstrate that significant variability in total CO concentrations exists at scales below ≈4°, and importantly, below the nominal 0.80° resolution of the TROPESS retrievals. While interpolated OE fields exhibit a strong suppression of variability at these smaller scales, the ML product retains substantial structure down to the Nyquist limit. This indicates that the ML model captures physically meaningful sub-resolution variability that is not recovered by interpolation alone.

A key advantage of applying machine learning models in inference mode is their computational efficiency . As expected, the ML model is able to process a full day of CrIS radiance observations with remarkable speed. For 10 June 2023, the OE algorithm generated 44 192 CO column retrievals in ≈160 min. In contrast, the ML model predicted CO concentrations and associated diagnostics for 2 916 000 columns in just ≈6 min.

This performance difference is even more striking when considering the computational resources used. The OE algorithm was run on 60 compute nodes utilizing a total of 480 CPU cores, while the ML model required only a single compute node with 8 CPU cores. Additionally, the prediction success rate was higher: 98.4% for the ML model (based on a conservative outlier flag) compared to 90.69% for the OE retrievals.

The superior computational performance of the ML model ensures that every individual CrIS sample can be processed efficiently, enabling predictions for any species included in the TROPESS retrieval framework. Moreover, this efficiency opens the door to near-real-time applications and provides a practical means to use the ML outputs to help constrain or enhance OE retrievals (see the discussion in Sect. 5).

Note, however, that the OE algorithm produces a vertical profile and an associated AK matrix whereas TROPESS-HYREF currently only predicts the derived column.