Correlated observation error models for assimilating all-sky infrared radiances

The benefit of hyperspectral infrared sounders to weather forecasting has been improved with the representation of inter-channel correlations in the observation error model. A further step would be to assimilate these observations in all-sky conditions. However, in cloudy skies, observation errors exhibit much stronger inter-channel correlations, as well as much larger variances, compared to clearsky conditions. An observation error model is developed to represent these effects, building from the symmetric error models developed for all-sky microwave assimilation. The combination of variational quality control with correlated errors is also introduced. The new error model is tested in all-sky assimilation of seven water vapour sounding channels from the Infrared Atmospheric Sounding Interferometer (IASI). However, its initial formulation degrades both tropospheric and stratospheric analyses. To explain this, the eigendeparture and eigenjacobian are introduced as a way of understanding the effect of correlated observation errors in data assimilation. The trailing eigenvalues can be problematic because they strongly amplify high-order harmonic combinations of the water vapour channels, which could have at least three consequences. First, if there are small interchannel biases, these can be greatly amplified. Second, the trailing eigenjacobians map onto features resembling gravity waves that the data assimilation may not be able to handle. Finally, these harmonic combinations can amplify trace sensitivities, for example, revealing a strong upper stratospheric sensitivity over high cloud in what are usually midto uppertropospheric water vapour channels. A likely explanation is the sensitivity to gravity wave features that are present in the observations but hard for the data assimilation to handle. After reducing the sensitivity to the trailing eigenjacobians, the new error covariance matrix gives good results in all-sky infrared assimilation. Copyright statement. The author’s copyright for this publication is transferred to ECMWF.

A very well-written manuscript with no obvious flaws which presents a clear picture of some experiments towards infrared all-sky radiance assimilation. The need to inflate the trailing eigenvalues and the justification for doing so is a significant finding and results in a consistency with much of the other work being done and what has been diagnosed from them for similar activities. I find the manuscript is ready for publication after corrections of any wording or clarity flaws which may be uncovered, but none were found by this reviewer.
These positive and helpful comments are much appreciated 2.1 I was very interested in one particular aspect of the paper. On page 30, beginning about line 20 when the trailing eigenvalues are adjusted so they are no smaller than 1 or 0.37. What is the resulting observation error in brightness temperature space for the clear-sky conditions as compared to the current "clear-sky" technique? Does this result in high errors for these same clear-sky scenes when the all-sky technique is applied? It could be a very appropriate thing to do, and could even be indicating additional uncertainty should be added due to non-detection of partially cloud filled pixels.
This is an interesting point. The effect of eigenvalue adjustments on the clear-sky errors in the brightness temperature basis has been described in words but not visually, so a figure will be added, which would also support further discussion. As identified by the reviewer, the eigenvalue adjustment broadly increases observation error variances relative to those used in the clear-sky approach, but this is a broad change, not targeted at potentially cloudy areas, and at maximum the increase is around 20%. Bormann et al. (2016) tested a range of inflation factors for the error variances on the brightness temperature basis; they found that further error inflation beyond 1.75 (relative to their Desroziers estimate) did not improve the results. Hence to test the reviewer's hypothesis it would be interesting, although not feasible for the current manuscript, to try a more targeted error inflation by further inflating just the trailing eigenvalues of the clear-sky error covariance matrix in the clear-sky approach. However Campbell et al. (2017), in a clear-sky framework, found better convergence and fewer iterations to convergence using a "additive" adjustment to all eigenvalues, rather than the "Ky Fan" approach boosting only the trailing eigenvalues. This might hint that further trailing eigenvalue adjustment would not be that beneficial for clear-sky assimilation.
A likely more relevant aspect to improving the clear-sky approach is that many observations that are "clear" from the point of view of a "clear-sky" assimilation scheme are actually at locations where the model has cloud, often even deep convection. Though the clear-sky framework represents clear-sky radiative transfer in the observation operator, areas where the model is cloudy will have a high relative humidity, and the observation equivalent may still be in substantial error compared to the observation that almost certainly comes from a clear area (the cloud detection schemes for hyperspectral instruments seem to be very efficient). In the all-sky framework, these "obs clear -model cloudy" situations will get an additional boost to their observation error; however this effect is already present in the "baseline" all-sky experiment and is not further affected by eigenvalue adjustment.
Manuscript change: Add a new figure showing the effect of eigenvalue adjustment on the observation error in fully clear and fully cloudy situations. The possibility that error inflation in "clear" areas may provide part of the benefit of trailing eigenvalue adjustment will be acknowledged; however the clear-sky effect is likely a secondary explanation for the results.

2.2
Lastly, a small note the figures which use 2D line plots use very fine lines. This makes it particularly difficult to often discern between colors particularly the blue and black. Thicker lines though causing some overlap would make these much easier to discriminate.
This was also mentioned by reviewer 1.
Manuscript change: I will experiment with thicker lines as advised.
2.2 Very last, very pithy comment. The label "all-sky diag" in figure 16 and 17 one could go ahead and spell out "diagonal" fully as there seems to be plenty of space for this in the figure label.
Manuscript change: "Diagonal" will now be fully spelt out where it appears on the plots, and also in the text when the experiment is referred to by name.