The benefit of hyperspectral infrared sounders to weather forecasting has been improved with the representation of inter-channel correlations in the observation error model. A further step would be to assimilate these observations in all-sky conditions. However, in cloudy skies, observation errors exhibit much stronger inter-channel correlations, as well as much larger variances, compared to clear-sky conditions. An observation error model is developed to represent these effects, building from the symmetric error models developed for all-sky microwave assimilation. The combination of variational quality control with correlated errors is also introduced. The new error model is tested in all-sky assimilation of seven water vapour sounding channels from the Infrared Atmospheric Sounding Interferometer (IASI). However, its initial formulation degrades both tropospheric and stratospheric analyses. To explain this, the

The author's copyright for this publication is transferred to ECMWF.

Geophysical quantities are inferred from indirect observations (such as satellite radiances) using techniques ultimately derived from Bayes' theorem. This requires a representation of the error in the prior state and in the observations. Especially in meteorological data assimilation, accurate modelling of the prior or “background” error is critical

Recently many numerical weather prediction (NWP) centres have started to represent observation error with more sophistication. For the assimilation of hyperspectral infrared (IR) sounder radiances in clear-sky conditions, the representation of inter-channel error correlations has improved the skill of operational weather forecasts

The first problem of observation error modelling is to estimate the observation error covariances

The estimation of situation-dependent error variances for all-sky microwave assimilation has followed a different approach

Since all-sky microwave assimilation has been most successful where

The current work will follow common practice in all-sky assimilation by fitting an observation error model directly to the covariances of the background departures, assuming that background error is relatively small. This is justified both by the previous success of this approach and by the lack of suitable alternatives. First the

One other option for estimating observation error is the

This study has been performed as part of wider developments towards the assimilation of IR radiances in all-sky conditions at ECMWF, to be reported by

ECMWF operates an NWP system with the aim of predicting weather globally for the medium range and beyond (day 3 onwards). Initial conditions for the forecast are produced by 4-D variational data assimilation

The forecast model is run at TCo1279 horizontal resolution (around 8–9 km) and with 137 terrain-following vertical levels. Cloud water, cloud ice, and rain and snow precipitation are prognostic variables. Both the large-scale and convective moist processes are parameterized. The convective precipitation is not included in the prognostic precipitation variables, which is not an issue for the IR but requires special treatment in the all-sky microwave, which is strongly sensitive to convective precipitation

The data assimilation uses an incremental formulation

The experiments in this study are run at reduced horizontal resolution: forecasts and data assimilation outer loops are at TCo399, around 25 km, and inner loops reach a maximum of TL255, around 80 km. The early delivery assimilation window is dropped, so that forecasts out to 240 h are run from the main 12 h assimilation window. This is the standard configuration for testing new developments at ECMWF and in most cases its results have been representative of those in the full operational configuration. Experimentation is carried out for two periods of 3 months: from 1 June to 31 August 2017 and from 1 December 2017 to 28 February 2018; results from the two periods are combined so as to give up to around 360 forecast samples. A control experiment has been run that includes the full observing system but without the seven IASI water vapour channels, and then a series of experiments (to be introduced later) add these channels with various configurations of observation error and variational quality control (VarQC). Cycle 45r1 of the ECMWF system has been used in most of the work presented here: this is a version that went operational in June 2018.

Full details of the all-sky IR configuration are given by

Details of the seven mid- and upper-tropospheric water vapour channels to be assimilated in all-sky conditions.

ECMWF assimilates

For the operational clear-sky assimilation, a globally constant observation error covariance matrix is used, which includes correlations between all the different channels of one observation

Hyperspectral infrared observations are also assimilated from the Atmospheric Infrared Sounder (AIRS) and Cross-track Infrared Sounder (CrIS) using similar configurations to IASI. Further, the mid- and upper-tropospheric water vapour channels are assimilated from five geostationary imagers around the Equator. The presence of all of these data (plus its equivalent from many microwave sounders) means that changes in forecast quality coming from different usage of IASI water vapour data will not be large, but, as will be explained below, there is still enough sensitivity in the fits of the short-range forecast to other observations that it is possible to clearly measure the impact of the work described here.

In the all-sky IR experiments, it is only the usage of the seven mid- and upper-tropospheric water vapour sounding channels that has been changed. The error correlations between these channels and the others are set to zero, so that the seven channels are treated in many respects as an independent instrument. The remaining coupling to the other channels is through the skin-temperature sink variable and through the thinning that includes a selection of the smallest window channel background departure. However, this is not expected to have much effect due to the removal, through a screening test, of situations where the seven chosen channels have sensitivity to the surface. The principal changes for all-sky assimilation are (i) to stop rejecting cloud-affected observations (but to retain rejection of aerosol-affected scenes); (ii) to use the cloud-capable version of the RTTOV observation operator described above and (iii) to use the situation-dependent all-sky error covariance matrix developed in the current study.

Some of the more detailed processing is retained from the clear-sky assimilation, such as the 100 km box thinning and the same bias correction model. But most other data selection and quality control processes have been changed to implement all-sky assimilation. Background quality control is now applied to the whole block of seven channels, so that either all channels are kept, or all are rejected. This means that the eigenvectors of the observation error covariance matrix remain fixed, allowing the eigenvalues to be scaled in a controlled way as described later. Instead of checking the size of the normalized background departures, it is the size of the normalized eigendepartures (Sect.

To find the best estimate of the state,

Some of the key concepts of observation error covariance matrices can be understood through the second term on the right-hand side of Eq. (

The gradient of the observation term with respect to the state can also be written as a summation including independent observation errors:

If observation errors are correlated, none of the above simplifications can be made because of the off-diagonal terms in

Just as there is an equivalence between the departures and the eigendepartures, there is also what we can call an eigenjacobian that gives the sensitivities of each eigenvector to the state. Now, the gradient of the observation cost function can be computed from the following summation:

Finally, these equations have so far been written with one giant observation error covariance matrix, but this is not how current data assimilation systems work. When the assumption is made that observations are uncorrelated in space, but errors are correlated across the channels of one instrument, the matrix becomes block diagonal. Then the same maths can be applied to the sub-matrices of

Details of the error covariance matrices examined here

The all-sky error covariance matrix has been in development for a while, so a number of different versions will be examined here (Table

Error correlation matrices

Figure

Error standard deviations

Figure

Eigenvectors of possible observation error covariance matrices for the seven assimilated IASI upper-tropospheric water vapour channels. Generally, the black lines (clear-sky error matrices) are almost totally overlaid, and similarly the coloured lines (all-sky error matrices) are also often overlaid.

The eigenvector decomposition

Square root of the eigenvalues associated with the eigenvectors in Fig.

Figure

In the Introduction it was argued that the

Figures

Temperature parts of channel Jacobians (

The sensitivity of the observed radiances to geophysical quantities is described by the Jacobian matrix of partial differentials of the observation operator. For each channel

Humidity parts of channel Jacobians (

Figure

Temperature sensitivity of the Jo cost function by eigenvector (

The role of the eigenvalues of the error covariance matrix in amplifying sensitivities to higher-order vertical oscillations can be seen in Fig.

Temperature parts of channel Jacobians (

The effect of cloud is explored in Fig.

Temperature parts of eigenjacobians (

It is also possible to compare the eigenjacobians of the clear-sky and all-sky error covariance matrices, as shown in Fig.

Mean normalized background departure,

When assimilating data with a diagonal observation error representation, it is the background departures that are routinely analysed for Gaussianity (desirable) and bias (undesirable). For example, Fig.

Mean mean normalized background eigendepartures,

When using a correlated observation error representation, it also makes sense to explore the Gaussianity and bias of the eigendepartures. Figure

The mean eigendepartures in Fig.

Probability density functions of normalized background eigendepartures (thin line) and a standard Gaussian (dashed line). For eigenvector 1, normalized background departures with all-sky error scaling make the sample more Gaussian (thick line). Based on the sample of departures from 1 to 20 June 2017, as in Fig.

Making the assumption that background errors are small relative to observation errors, an appropriate observation error model should generate normalized background eigendepartures with a PDF similar to a standard Gaussian. Figure

The best choice of cloud proxy variable for all-sky IR assimilation is still a matter of research.

Standard deviation of normalized background eigendeparture of eigenvector 1, computed in 1 K bins of the cloud proxy variable

Figure

Standard deviations were computed from each population, and then this distribution could be approximately fitted by the piecewise linear function:

To generate a situation-dependent error covariance matrix,

Error standard deviations for the seven selected IASI upper-tropospheric water vapour channels, ordered by ascending altitude of weighting function, using the adaptive all-sky error covariance matrix with scaling factors

Error correlation matrices for the 7 selected IASI upper-tropospheric water vapour channels, ordered by ascending altitude of weighting function, showing the form of adaptive error covariance matrices for

Figures

The scaled observation error covariance matrix was tested in a total of 6 months of full-cycling data assimilation experiments using the ECMWF all-sky IR assimilation framework at cycle 45r1 that is summarized in Sect.

Standard deviations of

Initial results are shown in Fig.

Figure

Other experiments in Fig.

A final test “all-sky diagonal” explores whether error correlations are necessary at all, here by using the diagonal of the all-sky error covariance matrix (clearly this requires the adaptive error scaling to be switched off). This simple approach results in much better analysis fits to ATMS, but it generates similarly underwhelming improvements in the background fits. The initial conclusion is that something in the error covariance matrix must be causing problems in the quality of the analysis.

As in Fig.

As in Fig.

Figure

Statistics of the quality of the minimization from the last inner-loop iteration of 4D-Var (median condition number and mean number of iterations) across all data assimilation cycles in the experiments.

It has been shown that using the raw correlated error model, all-sky IASI WV assimilation degrades the analysis. Particularly strange is the degradation in stratospheric and lower tropospheric temperatures, when the IASI WV observations are mainly sensitive to the mid-troposphere and upper troposphere. A first possible issue would be the conditioning of the observation error matrix. In the Met Office system described by

Table

Poor conditioning cannot directly explain the degradation in the quality of the analysis and forecast, unless the minimization stops before full convergence has been achieved. This is not thought likely as the hard iteration limit is 50 in the current experiments, so even at 35 iterations the minimization has stopped due to satisfying the standard convergence criterion. Hence, something more than just the conditioning is required to explain the degradations coming from the non-adjusted error covariance matrices.

Section

Figure

An equally worrying aspect of Fig.

Mean change in zonal mean temperature analysis between experiments and the control. Cross-hatching indicates 95 % confidence with a Šidák correction for 20 independent tests.

Temperature increment (solid line) required to correct normalized background eigendepartures of [0, 0, 0,

Additional temperature increments generated by all-sky IR assimilation on top of the otherwise full observing system, at 21:00 Z on 31 May 2017, at the beginning of the experiments where the background is identical across all experiments.

A final possibility that could generate increments in the stratosphere would be if, under certain conditions, the eigenjacobians could directly generate sensitivities in the stratosphere. The physics of the situation can potentially generate this: for example, over a cold tropical cloud top at around 185 K, emission from the stratosphere can increase brightness temperatures by several Kelvin

Relative size, in percent, of stratospheric temperature sensitivity in a tropical profile with and without optically thick cloud at the tropopause (sum of temperature Jacobian in the stratosphere divided by the total sum of temperature Jacobian). Results given either for the regular Jacobian or for eigenjacobians.

Figure

This study describes the first observation error covariance matrix to have both inter-channel error correlations and an adaptive scaling. It has been developed to support the all-sky assimilation of seven IASI infrared water vapour sounding channels at ECMWF that is described in more detail by

The error covariance model starts by computing the covariance of a global sample of all-sky background departures. This follows standard practice in all-sky assimilation where the observation errors are assumed to be much larger than the background errors, due to large errors of representation, so that the covariance of the background departures is a reasonable approximation to the observation error itself. To make the error covariance model adaptive, the leading eigenvalue is scaled as a function of the symmetric cloud proxy variable. This takes advantage of the fact that most of the cloud signal projects onto the leading eigenvector, which is true for the seven water vapour channels examined here but would not necessarily be true for more diverse sets of channels.

Temperature parts of the Jacobians

Another novelty of this work is the application of VarQC alongside a correlated observation error matrix (mostly described in the Appendix). This follows the proposal of

Experiments have been run over a total of 6 months with a control that uses the full observing system minus the seven IASI WV channels. The focus has not been on the quality of the all-sky IR assimilation itself (which is reported in the other study) but on the different possible configurations of the error covariance model and other associated settings. The results have been summarized based on analysis and background fits to ATMS temperature and humidity sounding channels and some diagnostics of the assimilation system. However, they are backed up by many other observation fits, not shown here.

The baseline all-sky assimilation has the problem that it degrades analysis fits to ATMS in both humidity channels (sensitive to mid-troposphere and upper troposphere) and temperature channels sensitive to the troposphere and stratosphere. Although in the background forecast it improves humidity fits (indicating that it does provide some benefits) degradations in the temperatures persist into the forecast, particularly in the stratosphere. These degradations appear to come from additional gravity waves and tropical waves created by the all-sky assimilation. Further experiments were made and compared to the baseline as follows.

Turning off the adaptive scaling makes the analysis fits worse and increases the condition number of 4D-Var. This is likely due to the underweighting of clear-sky observations and overweighting of cloudy observations. Hence, the adaptive scaling is worthwhile, but there is no evidence that it improves the quality of forecasts compared to a globally constant matrix. A possible explanation is that all seven eigenvectors of the error covariance matrix contribute equally to the observation cost function and thus (filtering due to background errors aside) all have equal weights in the assimilation. So although the scaling makes a big difference to the error covariances in brightness temperatures, it only affects a small part of the observations' influence on the assimilation.

Using diagonal errors (without adaptive scaling) prevents the degradations in the analysis and in the temperature forecast and provides similar benefit to humidity forecasts. This showed that the problems in the baseline configuration must come from the correlated part of the observation error model.

Applying an additional unilateral inflation to the error variances was not helpful in reducing them, rather it reduced the weight of the observations so far that all-sky assimilation provided no benefit at all to the forecast.

Using variational quality control (VarQC) is also useful to avoid further degrading analysis fits. However, it has little effect on the subsequent forecast, which is in contrast to earlier results from microwave data

Another step beyond previous work has been the introduction of the eigendeparture and eigenjacobian as useful diagnostics of a data assimilation system that uses inter-channel correlated observation errors. The eigenjacobians show the sensitivity of the eigendepartures to the state, indicating that the leading eigenvector responds to a broad vertical average of temperature and moisture, and trailing eigenvectors see increasingly higher harmonics of this but with increasingly little direct sensitivity because they are computed from the differences between channels. However, when weighted by the square of the eigenvalues (the equivalent of observation error when using a diagonal error matrix), the sensitivities to the vertically broad features are reduced and to the high periodic vertical features are enhanced. Along with examination of the monthly mean eigendepartures, this helps establish a much clearer picture of how correlated error covariance matrices change the sensitivities of the data assimilation system, complementing earlier work

To explain the problems encountered with the trailing eigenvectors, this study has described four potential complications with representing inter-channel observation error correlations. Already well-known is the issue of ill-conditioning. As explained by

Three new potential issues with error covariance matrices have been outlined that could help explain the worse short-range temperature forecasts.

A further possible issue would be the stability of the error covariance matrices, whether because of insufficient sampling

Whatever the exact explanation for the problem, it could be addressed by adjustments to the trailing eigenvalues that are also useful for improving conditioning. A likely hypothesis is the amplified sensitivity to gravity waves, which is backed up by the observed large increase in the prevalence of stratospheric gravity waves and tropical waves in the increments generated by the all-sky assimilation with the baseline error covariance matrix. However, as with eigenvalue adjustment purely for conditioning purposes, the solution has come at the cost of losing useful information that might project onto the trailing eigenvectors. Adjustment to 0.37 showed slightly better results in some places than to 1.0, hinting that over-inflating the trailing eigenvalues may lose useful information. None of the three newly hypothesized issues are problems with the error covariance matrices themselves, but rather with the ability of the current data assimilation framework to make use of the full spectrum of information contained in the observations.

All-sky radiance assimilation is a particularly difficult case for a correlated observation error model. First, inaccuracies in cloud modelling mean biases are common. Second, assimilating data across all both clear and cloudy states makes for highly heterogeneous (heteroskedastic) error covariances. A solution might be to compute error covariance matrices for a number of different subpopulations. For example, a series of error covariance matrices could be computed from departures that have been binned according to a cloud proxy variable. This approach is being tested for the all-sky microwave assimilation at ECMWF. There is also scope for improving the initial observation error model described here: perhaps the second eigenvector needs cloud scaling as well. Maybe there could be a more detailed exploration of error scaling and different methods for diagnosing observation error

It is also possible that clear-sky assimilation using correlated observation errors could be similarly affected by the amplification of sensitivities to bias, gravity waves and unexpected parts of the atmosphere that current assimilation can struggle to deal with. Problems may not have been apparent so far due to the routine inflation and adjustment of eigenvalues

This paper summarizes around 160 TB output from experiments using the ECMWF forecasting system. Permanently storing this data would be a major practical problem and the necessity of doing so would need to be established.

VarQC has never been applied to operational hyperspectral IR assimilation at ECMWF, so

If VarQC is applied to the eigendepartures, it means the model for gross error is also flat across a range of eigendepartures, rather than of brightness temperature departures. Traditional kinds of gross error, such as a bad measurement in a single satellite channel, will project onto all eigenvectors, and hence an independent top-hat error model in eigendepartures is incorrect. However, VarQC is not used in all-sky assimilation for dealing with traditional gross error so much as to down-weight situations where the analysis struggles to match the observations, particularly those associated with cloud and precipitation errors

To apply VarQC for IASI, settings of

AG planned and executed the work and wrote it up.

The authors declare that they have no conflict of interest.

Niels Bormann, Peter Weston and Stephen English are thanked for very helpful discussions and reviews of the original manuscript. The four reviewers are thanked for their helpful comments that improved the paper in many ways and brought up the covariance stability issue.

This paper was edited by Isaac Moradi and reviewed by two anonymous referees.