Singular Vector Decomposition (SVD) of satellite datasets: relation between cloud properties and climate indices

Carboni, Elisa; Thomas, Gareth E.; Siddans, Richard; Kerridge, Brian

doi:https://doi.org/10.5194/amt-2023-232

Preprints

https://doi.org/10.5194/amt-2023-232

Preprints

22 Nov 2023

| 22 Nov 2023

Status: this preprint was under review for the journal AMT but the revision was not accepted.

Singular Vector Decomposition (SVD) of satellite datasets: relation between cloud properties and climate indices

Elisa Carboni, Gareth E. Thomas, Richard Siddans, and Brian Kerridge

Abstract. We describe a technique using singular vector decomposition (SVD), that can identify the spatial patterns that best describe the temporal variability of a global satellite dataset. These patterns, and their temporal evolution, are then correlated with established climate indices. We apply this technique to datasets of cloud properties over three decades, derived from five visible/IR imagers ((A)ATSR, SLSTR-A/-B and MODIS and jointly from the IR and microwave sounders on MetOp (IASI, MHS,AMSU-A), but it can be more generically used to extract the pattern of variability of any regular gridded dataset such as different parameters from satellite products and models. The leading singular vector for these three independent global data sets, on both cloud fraction and cloud-top height, from these polar orbiting satellites covering different time periods, is found to be strongly correlated with the ENSO index. The SVD approach could potentially offer a new tool for using global satellite observations in assessing global climate model (GCM) performance.

Received: 03 Nov 2023 – Discussion started: 22 Nov 2023

Publisher's note: Copernicus Publications remains neutral with regard to jurisdictional claims made in the text, published maps, institutional affiliations, or any other geographical representation in this preprint. The responsibility to include appropriate place names lies with the authors.

Download & links

Elisa Carboni, Gareth E. Thomas, Richard Siddans, and Brian Kerridge

Status: closed

RC1:
'Comment on amt-2023-232', Anonymous Referee #1, 14 Dec 2023
This scientific paper presents the application of a statistical (or algebraic) technique that identifies that portion of data in a time series that is useful in interpreting the dynamics of the atmospheric system through correlations with climate indices. Specifically, the technique presented is one of the possible belonging to the group of the Eigen techniques, the Singular Value Decomposition (SVD), and the time series are those of cloud properties derived from satellite measurements.

After reading the manuscript, I find myself in the uncomfortable position of rejecting the work and having to suggest that the authors rethink the logic of the study from the ground up. This work is not only incomplete, it is also wrong in its logical premises, the type of data used, the technique presented and fail novelty. I set out my points below.
Is this scientific journal the most appropriate for such work? I don't think so. This manuscript presents no new algorithm, no validation of the data, no error analysis of the data. In truth, there is only the application of a decomposition technique of a historical series and correlations with climate indices. The authors themselves indicate in the abstract as an application to assess the performance of global climate models. AMT is not the appropriate venue for this type of message. Perhaps ACP (I have my doubts), but even better would be ClimDyn or JCLIM.

With reference to the technique presented (SVD), the choice is unsatisfactory.

A characteristic hurdle in climate research is the size of the phase space. In a real system such as the climate it is basically infinite and in GCM is quasi-infinite. Clearly, the observations are per se limited and there is the need to single out those significant first-order components that best describe the underlying dynamics of the system. In the pursue of which, the direction in the phase space assumes a relevant role (which by the way justifies the difference between statistics and the isotropic statistical mechanics). Long story short: to achieve this goal we have already a myriad of Eigen Techniques, better suited to this purpose. Alone the EOFs, the rotated EOFs, or climate networks (e.g. Ludescher et al 2014, Donges et al. 2011, 2015). There are more out there that are more sophisticated and appropriate approaches. Why is this the case? Because in all the techniques I have mentioned it is possible to embed constraints to analyze what really counts: the variability. The argument that SVD is preferable to other techniques because it is simpler is true at the expense of depth and accuracy of analysis. With all due respect to colleagues, this strikes me as more of a task for a freshman in a master of science course than for established researchers aiming at novel results. Moreover, nowadays, thanks to the open science paradigm, there are many public repositories where implementations of the respective techniques can be conveniently downloaded. Thus, the objection of having to code everything from scratch no longer exists.

Ludescher, J., Gozolchiani, A., Bogachev, M. I., Bunde, A., Havlin, S., & Schellnhuber, H. J. (2014). Very early warning of next El Niño. Proceedings of the National Academy of Sciences, 111(6), 2064-2066.

Donges, J. F., Schultz, H. C., Marwan, N., Zou, Y., & Kurths, J. (2011). Investigating the topology of interacting networks: Theory and application to coupled climate subnetworks. The European Physical Journal B, 84, 635-651.

Donges, J. F., Petrova, I., Loew, A., Marwan, N., & Kurths, J. (2015). How complex climate networks complement eigen techniques for the statistical analysis of climatological data. Climate dynamics, 45, 2407-2424.

The choice of data. The authors cannot use L3 monthly averages for their analysis, rather they must use L2 in conjunction with the standard deviation (or variance) and respective error metrics and ask how the aggregation of the L2 time series impacts the goodness of fit. This is because each sensor has a different spatio-temporal sampling and the patterns emerging from the analysis are affected by these differences. In other words, the authors must first try to answer the question of what correlation length is required for different sensors to represent the same cloud field. No data set can ever represent reality. So every data set is exposed to the same shortcomings. Before looking for a quasi-orthogonal basis to link to a climate index, one must make the observational data sets as homogeneous as possible among each other. And introduce error metrics.

The manuscript is not consistently elaborated because all climate indices are stated in a table (superfluous at this point), but only ENSO is mentioned in the manuscript. From the title chosen by the authors, the manuscript suggests a (laudable and ambitious) generalisation, but this is nowhere to be found.

Figures 2 and 3. I invite the authors to fill in the time gap between 2012 and 2017. Without coverage of these dates, it is not even remotely conceivable for me to scrutinise the results as the trend and eventual statistical significance (which is missing by the way). There are no prerequisites because the data presented are lacking.

As a final point, I suggest that the authors, for the next draft of the paper, reserve some of their time and effort for interpreting the results they find. The correlation of CTH with ENSO, for example, is straightforward and immediately understandable on the basis of basic arguments of fluid- and thermo-dynamics. The matter becomes more interesting if one creates Hovmoller graphs of correlations between certain cloud properties and climate indices. The evolution of these teleconnections may reveal as yet unknown aspects. For example, the timing of the start of the monsoon season or exchanges of energy and momentum between low latitudes and the poles. Be that as it may, it is not an easy task, precisely because the authors want to tackle it empirically, on the basis of data and not models. But precisely for this reason, from my point of view, the highest possible precision must be requested in the formulation of the problem.
Citation: https://doi.org/10.5194/amt-2023-232-RC1
- AC1: 'Reply on RC1', Elisa Carboni, 24 May 2024
  
  The comment was uploaded in the form of a supplement: https://amt.copernicus.org/preprints/amt-2023-232/amt-2023-232-AC1-supplement.pdf
  
  Citation: https://doi.org/10.5194/amt-2023-232-AC1
RC2:
'Comment on amt-2023-232', Anonymous Referee #2, 16 Apr 2024
General Comments:
This paper demonstrates an application of the Singular Value Decomposition (SVD) statistical technique to correlate cloud properties observed in satellite data against climate indices.
SVD is commonly-used in the atmospheric sciences as a method to understand the spatio-temporal variability of geophysical data. The authors effectively demonstrate how one can use SVD to compare the modes of variability in a long observational satellite record against climate indices.
Because the authors both a. emphasize the applicability of the technique to any gridded satellite dataset (not just atmospheric fields), and b. position this methodology as primarily novel for climate model validation, this paper may be better suited for a journal focused more generally on climate rather than atmospheric measurements.
That said, I think it would be appropriate for publication in Atmospheric Measurement Techniques with revisions addressing the specific comments below.
Specific Comments:
The study is framed as a technique that can be effectively generalized to any gridded dataset, but only provides examples of cloud observations. An additional example would be helpful to support the generalization.

The authors report strong correlations in the results but omit statistical significance.

Spend time exploring what the authors themselves state as one of the novel aspects of using this technique: understanding the underlying causes of the variability.

Did the authors try any other decomposition methods? How does using SVD compare to other methods such as using EOFs or other approaches? A clear statement for why this technique was chosen and what its limitations are should be included.

Technical Corrections:
Paragraph 45: typo: de-seansonalised

Paragraph 145: typo: de-sesonalized

Paragraph 185: type: series off time-lags

Figs 3, 4, 5: consider centering the maps on the Pacific Ocean instead of the Atlantic to better show the ENSO correlation
Citation: https://doi.org/10.5194/amt-2023-232-RC2
- AC2: 'Reply on RC2', Elisa Carboni, 24 May 2024
  
  The comment was uploaded in the form of a supplement: https://amt.copernicus.org/preprints/amt-2023-232/amt-2023-232-AC2-supplement.pdf
  
  Citation: https://doi.org/10.5194/amt-2023-232-AC2

Status: closed

RC1:
'Comment on amt-2023-232', Anonymous Referee #1, 14 Dec 2023
This scientific paper presents the application of a statistical (or algebraic) technique that identifies that portion of data in a time series that is useful in interpreting the dynamics of the atmospheric system through correlations with climate indices. Specifically, the technique presented is one of the possible belonging to the group of the Eigen techniques, the Singular Value Decomposition (SVD), and the time series are those of cloud properties derived from satellite measurements.

After reading the manuscript, I find myself in the uncomfortable position of rejecting the work and having to suggest that the authors rethink the logic of the study from the ground up. This work is not only incomplete, it is also wrong in its logical premises, the type of data used, the technique presented and fail novelty. I set out my points below.
Is this scientific journal the most appropriate for such work? I don't think so. This manuscript presents no new algorithm, no validation of the data, no error analysis of the data. In truth, there is only the application of a decomposition technique of a historical series and correlations with climate indices. The authors themselves indicate in the abstract as an application to assess the performance of global climate models. AMT is not the appropriate venue for this type of message. Perhaps ACP (I have my doubts), but even better would be ClimDyn or JCLIM.

With reference to the technique presented (SVD), the choice is unsatisfactory.

A characteristic hurdle in climate research is the size of the phase space. In a real system such as the climate it is basically infinite and in GCM is quasi-infinite. Clearly, the observations are per se limited and there is the need to single out those significant first-order components that best describe the underlying dynamics of the system. In the pursue of which, the direction in the phase space assumes a relevant role (which by the way justifies the difference between statistics and the isotropic statistical mechanics). Long story short: to achieve this goal we have already a myriad of Eigen Techniques, better suited to this purpose. Alone the EOFs, the rotated EOFs, or climate networks (e.g. Ludescher et al 2014, Donges et al. 2011, 2015). There are more out there that are more sophisticated and appropriate approaches. Why is this the case? Because in all the techniques I have mentioned it is possible to embed constraints to analyze what really counts: the variability. The argument that SVD is preferable to other techniques because it is simpler is true at the expense of depth and accuracy of analysis. With all due respect to colleagues, this strikes me as more of a task for a freshman in a master of science course than for established researchers aiming at novel results. Moreover, nowadays, thanks to the open science paradigm, there are many public repositories where implementations of the respective techniques can be conveniently downloaded. Thus, the objection of having to code everything from scratch no longer exists.

Ludescher, J., Gozolchiani, A., Bogachev, M. I., Bunde, A., Havlin, S., & Schellnhuber, H. J. (2014). Very early warning of next El Niño. Proceedings of the National Academy of Sciences, 111(6), 2064-2066.

Donges, J. F., Schultz, H. C., Marwan, N., Zou, Y., & Kurths, J. (2011). Investigating the topology of interacting networks: Theory and application to coupled climate subnetworks. The European Physical Journal B, 84, 635-651.

Donges, J. F., Petrova, I., Loew, A., Marwan, N., & Kurths, J. (2015). How complex climate networks complement eigen techniques for the statistical analysis of climatological data. Climate dynamics, 45, 2407-2424.

The choice of data. The authors cannot use L3 monthly averages for their analysis, rather they must use L2 in conjunction with the standard deviation (or variance) and respective error metrics and ask how the aggregation of the L2 time series impacts the goodness of fit. This is because each sensor has a different spatio-temporal sampling and the patterns emerging from the analysis are affected by these differences. In other words, the authors must first try to answer the question of what correlation length is required for different sensors to represent the same cloud field. No data set can ever represent reality. So every data set is exposed to the same shortcomings. Before looking for a quasi-orthogonal basis to link to a climate index, one must make the observational data sets as homogeneous as possible among each other. And introduce error metrics.

The manuscript is not consistently elaborated because all climate indices are stated in a table (superfluous at this point), but only ENSO is mentioned in the manuscript. From the title chosen by the authors, the manuscript suggests a (laudable and ambitious) generalisation, but this is nowhere to be found.

Figures 2 and 3. I invite the authors to fill in the time gap between 2012 and 2017. Without coverage of these dates, it is not even remotely conceivable for me to scrutinise the results as the trend and eventual statistical significance (which is missing by the way). There are no prerequisites because the data presented are lacking.

As a final point, I suggest that the authors, for the next draft of the paper, reserve some of their time and effort for interpreting the results they find. The correlation of CTH with ENSO, for example, is straightforward and immediately understandable on the basis of basic arguments of fluid- and thermo-dynamics. The matter becomes more interesting if one creates Hovmoller graphs of correlations between certain cloud properties and climate indices. The evolution of these teleconnections may reveal as yet unknown aspects. For example, the timing of the start of the monsoon season or exchanges of energy and momentum between low latitudes and the poles. Be that as it may, it is not an easy task, precisely because the authors want to tackle it empirically, on the basis of data and not models. But precisely for this reason, from my point of view, the highest possible precision must be requested in the formulation of the problem.
Citation: https://doi.org/10.5194/amt-2023-232-RC1
- AC1: 'Reply on RC1', Elisa Carboni, 24 May 2024
  
  The comment was uploaded in the form of a supplement: https://amt.copernicus.org/preprints/amt-2023-232/amt-2023-232-AC1-supplement.pdf
  
  Citation: https://doi.org/10.5194/amt-2023-232-AC1
RC2:
'Comment on amt-2023-232', Anonymous Referee #2, 16 Apr 2024
General Comments:
This paper demonstrates an application of the Singular Value Decomposition (SVD) statistical technique to correlate cloud properties observed in satellite data against climate indices.
SVD is commonly-used in the atmospheric sciences as a method to understand the spatio-temporal variability of geophysical data. The authors effectively demonstrate how one can use SVD to compare the modes of variability in a long observational satellite record against climate indices.
Because the authors both a. emphasize the applicability of the technique to any gridded satellite dataset (not just atmospheric fields), and b. position this methodology as primarily novel for climate model validation, this paper may be better suited for a journal focused more generally on climate rather than atmospheric measurements.
That said, I think it would be appropriate for publication in Atmospheric Measurement Techniques with revisions addressing the specific comments below.
Specific Comments:
The study is framed as a technique that can be effectively generalized to any gridded dataset, but only provides examples of cloud observations. An additional example would be helpful to support the generalization.

The authors report strong correlations in the results but omit statistical significance.

Spend time exploring what the authors themselves state as one of the novel aspects of using this technique: understanding the underlying causes of the variability.

Did the authors try any other decomposition methods? How does using SVD compare to other methods such as using EOFs or other approaches? A clear statement for why this technique was chosen and what its limitations are should be included.

Technical Corrections:
Paragraph 45: typo: de-seansonalised

Paragraph 145: typo: de-sesonalized

Paragraph 185: type: series off time-lags

Figs 3, 4, 5: consider centering the maps on the Pacific Ocean instead of the Atlantic to better show the ENSO correlation
Citation: https://doi.org/10.5194/amt-2023-232-RC2
- AC2: 'Reply on RC2', Elisa Carboni, 24 May 2024
  
  The comment was uploaded in the form of a supplement: https://amt.copernicus.org/preprints/amt-2023-232/amt-2023-232-AC2-supplement.pdf
  
  Citation: https://doi.org/10.5194/amt-2023-232-AC2

Elisa Carboni, Gareth E. Thomas, Richard Siddans, and Brian Kerridge

Viewed

Total article views: 534 (including HTML, PDF, and XML)

HTML	PDF	XML	Total	BibTeX	EndNote
383	113	38	534	36	34

HTML: 383
PDF: 113
XML: 38
Total: 534
BibTeX: 36
EndNote: 34

Views and downloads (calculated since 22 Nov 2023)

Month	HTML	PDF	XML	Total
Nov 2023	38	23	5	66
Dec 2023	57	12	2	71
Jan 2024	24	13	2	39
Feb 2024	26	5	0	31
Mar 2024	29	15	2	46
Apr 2024	42	7	8	57
May 2024	37	9	10	56
Jun 2024	44	7	2	53
Jul 2024	28	3	5	36
Aug 2024	21	6	2	29
Sep 2024	17	2	0	19
Oct 2024	11	7	0	18
Nov 2024	9	4	0	13

Cumulative views and downloads (calculated since 22 Nov 2023)

Month	HTML	PDF	XML	Total
Nov 2023	38	23	5	66
Dec 2023	57	12	2	71
Jan 2024	24	13	2	39
Feb 2024	26	5	0	31
Mar 2024	29	15	2	46
Apr 2024	42	7	8	57
May 2024	37	9	10	56
Jun 2024	44	7	2	53
Jul 2024	28	3	5	36
Aug 2024	21	6	2	29
Sep 2024	17	2	0	19
Oct 2024	11	7	0	18
Nov 2024	9	4	0	13

Viewed (geographical distribution)

Total article views: 524 (including HTML, PDF, and XML) Thereof 524 with geography defined and 0 with unknown origin.

Country	#	Views	%

Latest update: 20 Nov 2024

Short summary

We analyzed different satellite datasets of cloud properties with a new approach to quantify and interpret their interannual variability based on singular vector decomposition (SVD). The spatial pattern and its temporal evolution are strikingly similar for all the satellite datasets and follow the El Nino Southern Oscillation. The SVD approach reported here has potential for application to satellite data sets and to evaluate consistency between models and observations.


Total:	0
HTML:	0
PDF:	0
XML:	0