Using machine learning to model uncertainty for water vapor atmospheric motion vectors

Teixeira, Joaquim V.; Nguyen, Hai; Posselt, Derek J.; Su, Hui; Wu, Longtao

doi:https://doi.org/10.5194/amt-14-1941-2021

Articles | Volume 14, issue 3

https://doi.org/10.5194/amt-14-1941-2021

© Author(s) 2021. This work is distributed under
the Creative Commons Attribution 4.0 License.

https://doi.org/10.5194/amt-14-1941-2021

© Author(s) 2021. This work is distributed under
the Creative Commons Attribution 4.0 License.

Articles | Volume 14, issue 3

Research article

|

09 Mar 2021

Research article |

| 09 Mar 2021

Using machine learning to model uncertainty for water vapor atmospheric motion vectors

Joaquim V. Teixeira, Hai Nguyen, Derek J. Posselt, Hui Su, and Longtao Wu

Download

Final revised paper (published on 09 Mar 2021)
Preprint (discussion started on 23 Apr 2020)

Interactive discussion

Status: closed

AC: Author comment | RC: Referee comment | SC: Short comment | EC: Editor comment

- Printer-friendly version

- Supplement

RC1: 'Review of the manuscript amt-2020-95', Anonymous Referee #1, 06 May 2020
- AC1: 'Uncertainty Quantification for Atmospheric Motion Vectors with Machine Learning; Author Response to Referee Comment 1.', Joaquim Teixeira, 11 Aug 2020
RC2: 'Review of “Uncertainty Quantification for Atmospheric Motion Vectors with Machine Learning” by Teixeira et al.', Anonymous Referee #2, 25 Jun 2020
- AC2: 'Uncertainty Quantification for Atmospheric Motion Vectors with Machine Learning; Author Response to Referee Comment 2.', Joaquim Teixeira, 11 Aug 2020
- AC3: 'Uncertainty Quantification for Atmospheric Motion Vectors with Machine Learning; Author Response to Referee Comment 2.', Joaquim Teixeira, 11 Aug 2020

Peer-review completion

AR: Author's response | RR: Referee report | ED: Editor decision

AR by Joaquim Teixeira on behalf of the Authors (13 Aug 2020) Author's response Manuscript

ED: Referee Nomination & Report Request started (17 Aug 2020) by Ad Stoffelen

RR by Anonymous Referee #2 (28 Aug 2020)

Suggestions for revision or reasons for rejection

Review of the first revision of Teixeira et al. 2020

I thank the authors for their reviews and the replies to earlier comments. In this revised version, the conceptual design and the particular implementation of the error estimation algorithm is unchanged to the previous version, but the text has been clarified in places and a further characterisation of the regimes identified in the clustering algorithm has been added. The aim of the paper continues to be to provide a “proof-of-concept” for an algorithm to supply error estimates (random and systematic) for data assimilation.

While the manuscript has improved in several areas, I feel that some of my earlier more substantive comments have not been sufficiently addressed. These include comments on the usefulness of the algorithm in practical situations (earlier general points 1 and partially 3), the robustness of the performance (particularly of the clustering; general point 1, specific points 6 and 10), and the evaluation of the performance (specific points 16 and 17). In some of their replies to the comments the authors acknowledge limitations and issues of their algorithm, yet this is not reflected in the revised manuscript (especially general point 3, specific points 6 and 10). I therefore continue to be unconvinced that this algorithm is useful for the intended application.

For the paper to be acceptable for publication, I think these points need to be more thoroughly addressed. In my view this means that the results are either more critically evaluated and the limitations more clearly outlined, or the algorithm has to be refined and made more robust so that applicability to real-life applications is indeed ensured. In my view either of this requires another major revision. I do not think that the present results serve as a “proof-of-concept” of a useful algorithm that could be applied to practical situations with real data, as the authors claim. I could accept if the paper was written from the perspective of introducing a novel conceptual framework which has been explored in an initial implementation, which shows some skill in assigning regime-specific observation errors, but has revealed a range of issues that would need to be addressed for this to be a viable and useful algorithm for real applications (and which may not be possible to address). A clear path of how these issues could be addressed should be provided, including an outline how the algorithm could be derived without the truth being available. The latter requires recognition and discussion of the issues that will be encountered when substituting the truth with proxy data beyond the superficial suggestion that has been added in the conclusion section. These proxy data (the authors suggest reanalysis data or collocated reference observations) will introduce their own regime-specific errors, and there is no mechanism apparent in the present implementation of the algorithm that would be able to separate the errors in the proxy data from those in the tracked winds. The clustering may identify regime-differences in errors in the proxy data (or the collocation errors), rather than in the AMVs. In addition, real-life applications would encounter errors in the humidity retrievals used for the tracking, and these will affect the characteristics of the errors in the tracked winds as well as the performance of the algorithm.

Some additional specific points:

1) Abstract: “… that is purely data driven and requires few tuning parameters”: I think this statement reflects more a limitation of the present study than a feature of the conceptual algorithm. There are significant tuning parameters (e.g., number of clusters, choice of predictors), but they are not explored in this study. I would argue that the method still requires substantial tuning of these parameters to be robust and useful compared to other methods, and some of the responses of the authors appear to agree with this. So I suggest to rephrase this statement to avoid giving the impression that this is a positive feature of the algorithm and that it can be run “out of the box”.

2) Abstract, last sentence “… and it is shown to adequately capture the error features of the tracked wind.”: I don’t think it is clear what “adequate” means in this context, and I think it would be useful to be frank that the truth was available for training. I suggest to rephrase “…, and it is shown to capture some of the regime-dependent error features of the tracked wind in a setting where the truth is available for training the algorithm.”

3) Reply to my earlier point 11: Thanks for providing the information on the significance of the standard deviation and bias statistics for the clusters included in the reply. This should be included in the revised manuscript, as it would help to substantiate the claim that the clusters identify statistically significantly different error behaviours.

Hide

RR by Anonymous Referee #1 (04 Sep 2020)

ED: Reconsider after major revisions (09 Sep 2020) by Ad Stoffelen

AR by Joaquim Teixeira on behalf of the Authors (06 Nov 2020) Author's response Manuscript

ED: Referee Nomination & Report Request started (09 Nov 2020) by Ad Stoffelen

RR by Anonymous Referee #2 (16 Nov 2020)

Suggestions for revision or reasons for rejection

Thanks for the second revision of this paper. It features a substantially extended discussion of the applicability of the algorithm for applications outside a simulation framework, as well as the limitations of the initial implementation. These largely address my previous concerns.

I am, however, surprised by the paragraph L426-430 in the discussion, which lists only two main issues for “large scale applications” (I read this as “applications to real satellite-derived AMVs” - I suggest to rephrase this to make it clear what is meant): 1) uncertainties in the humidity field, and the 2) representativeness/variability of the training data. I would argue that the first and foremost problem is that the truth for both the wind and the humidity field is not available in applications with real data. In the present study the Nature Run serves as truth (by design), and hence the errors of the AMVs are completely known from the differences between AMVs and Nature-Run winds. These “true errors” are essential input to train the algorithm from the first 1.5 months of data. As soon as the Nature Run is replaced with real data (for the humidity field, but more importantly the wind field), the “true errors” are not available anymore, and other errors will be introduced (e.g., collocation and representativeness errors if other observations are used as proxy of the true wind, or analysis and representativeness errors if reanalysis data is used as proxy for the truth). As far as I can see, the algorithm has no knowledge that would allow it to separate the errors in the “reference” wind data from the errors in the AMVs alone. This aspect is related, but rather different from the one presently listed as 2nd issue. I think it should be mentioned separately here, and it may deserve some more explicit discussion. The following paragraphs are touching on this aspect (esp. L496-501), but the text offers little in terms of addressing this. It seems to me a fundamental question that is left unanswered in the paper, ie how to train the machine learning algorithm in the case of real-data applications, so that it is able to separate different sources of error in the training data.

I suggest the authors address this aspect in a minor revision of the conclusions/discussion section.

Hide

ED: Publish subject to minor revisions (review by editor) (18 Nov 2020) by Ad Stoffelen

AR by Joaquim Teixeira on behalf of the Authors (01 Dec 2020) Author's response Manuscript

ED: Publish as is (06 Dec 2020) by Ad Stoffelen

AR by Joaquim Teixeira on behalf of the Authors (12 Dec 2020) Manuscript

Short summary

Wind-tracking algorithms produce atmospheric motion vectors (AMVs) by tracking satellite observations. Accurately characterizing the uncertainties in AMVs is essential in assimilating them into data assimilation models. We develop a machine-learning-based approach for error characterization which involves Gaussian mixture model clustering and random forest using a simulation dataset of water vapor, AMVs, and true winds. We show that our method improves on existing AMV error characterizations.