Dear authors,

Thank you for your revised submission. Based on my own reading and one further review, I find that the paper is almost ready for acceptance but would benefit from a small number of clarifications - please see the attached reviewer comments.

Best wishes,

Andrew

Dear authors,

As communicated by email, very shortly after issuing the previous decision letter I received the second review of your revision. This is an unusual situation timing-wise and the first time it has happened in 10 years as an Associate Editor with the journal. I and the journal office feel the best course of action for scientific transparency is to update this decision letter to include the contents of the second review. I am sorry for the disruption this causes after receiving my first decision letter earlier this week. Please know that I gave this matter much thought before proceeding.

I have included comments from both reviewers below:

Referee #1

In my opinion, this paper is not yet suitable for publication. A stated objective, the data analysis methods, and the conclusions seem to be flawed. In my opinion, the authors have not adequately addressed the reviewers’ concerns, and in revision, have managed to add additional confusion that further reduces the quality of the manuscript. The results described in the manuscript indicate that the statistical comparisons between MODIS and CALIOP are not very good, yet it is claimed without providing evidence that MODIS provides a reliable cirrus mask when compared to CALIPSO. A significant shortcoming is that the classification schemes for the two sensors are not well described, and the data filtering and matching procedure may be inadequate for conducting a fair comparison between the two sensors during daytime and nighttime, particularly with the inclusion of very thin stratospheric clouds only detected by CALIPSO. The presentation of the results is confusing and not well explained. I am concerned that the results presented here may misrepresent the accuracy and utility of the operational MODIS cloud products. If further consideration is to be given for the publication of this manuscript in ACP, I highly recommend that it be sent to someone from the operational MODIS cloud team for their opinion as they would be able to better interpret the MODIS results presented here.

Major concerns
1. As stated in the abstract, the objective of this paper is “to determine if a MODIS product exists that detects cirrus with the same accuracy as CALIOP”. This objective seems off base since several publications have shown that the CALIOP active sensor is more sensitive to cirrus than the MODIS passive sensor. If MODIS is less sensitive, then obviously, it will be less capable of detecting some cirrus. Therefore, the authors should revise the objective stated in the abstract. Something like that stated in the introduction on line 73 would be more reasonable, i.e. “Our objective is to determine how well the MODIS products can be used to identify cirrus clouds compared to CALIPSO.” Another objective stated on lines 74-75 does not seem to be addressed in the paper (that I could find), i.e. “we aim to assess whether MODIS cloud detection tests used to generate MYD35 operational data can be re-used for a time-effective masking of cirrus.” Beside the fact that the meaning of ‘time-effective’ is unclear, there is no evidence presented in the manuscript that the temporal consistency of the MODIS products was evaluated. Therefore, this objective should either be removed or supported with data.

2. Unfortunately, the authors did not adequately clarify and defend their definition of ‘cirrus’ as used in this study, nor how consistent that definition is for the two data products being compared, as requested by the reviewer(s).

The evaluation of the 6 spectral tests for cirrus may be of modest interest to algorithm developers as they provide performance metrics against CALIOP in a relative sense. However, these tests are not meant to stand alone for cirrus detection. From a practical standpoint, the ATC test which combines the results of all six tests could be a more useful gauge as to how well the MODIS data product can be used to identify cirrus overall, provided that the population of data being tested is evaluated in context with the total population of cloudy pixels. Unfortunately, from what I can tell, this isn’t done. It isn’t clear if the ATC collection of six tests encompass all possible cirrus pixels determined by the mask or if there are other information contained in the MODIS data products that could lead to a different population. The reader should not have to guess at this. This is important because if there are other cloudy pixels as determined by the mask for which there are other indicators (that these pixels may be cirrus (e.g. cloud phase and height), then the statistical comparisons don’t have much meaning and could even be misleading regarding the accuracy and utility of using the MODIS data products to discern cirrus. Is there a population of cloudy pixels for which it is unknown whether these could be cirrus or not which? Compared to the cirrus screening used here, would the population be the same if all cloudy pixels were included as determined by the mask that are also determined to be ice phase, either anywhere in the vertical column or above some height level? Are such other tests not possible due to failure rates in the cloud optical property and/or height algorithms?

The parameter called ROP (rate of observations performed) is not meaningful to me as it is defined for a specific test to be the fraction of observations evaluated in the test to the total observations. The problem is that it isn’t explained what population the total observations represents? Is it meant to be all cloudy pixels, or all cloudy pixels evaluated with the six tests, or something else?


Furthermore, regarding the CALIPSO data, it isn’t clear why the optically thinnest clouds that are impossible for MODIS to detect, particularly those in the stratosphere, are included in these comparisons. Stratospheric ice clouds are important for atmospheric chemistry, but MODIS is not a suitable sensor to study stratospheric clouds that it cannot detect. The CALIPSO products themselves are also not consistent between day and night due to inconsistencies in the lidar sensitivity. So why are the optically thinnest clouds included, particularly those in the stratosphere? The authors need to discuss this and justify the rationale for including stratospheric clouds detected by CALIPSO. At the very least, the statistical comparisons should be conducted, or stratified, using data with and without the thinnest CALIOP clouds. It doesn’t seem that this has been done (more on ths regarding figure 10 below). This would provide perhaps a fairer comparison and more informative performance metrics, but certainly a more informative comparison across daytime and nighttime where the CALIPSO sensitivities are much different.

3. In the discussion section, it is stated that this study proved that MODIS ready-to-use cloud mask product can be used for producing a reliable cirrus mask, however, it is totally unclear how this conclusion is arrived at. By what metrics levels is the mask deemed to be reliable and how are those levels of ‘reliability’ determined? The ‘goodness of fit’ parameters shown in table 3 for example are not particularly impressive, especially for nighttime as pointed out in the manuscript. Whatever potential the paper has up to this point really becomes confusing and seems to fall apart near the end when figures 8-10 are introduced.

Regarding figure 8: This shows a remarkable inconsistency (factor of 4 difference) between the daytime and nighttime cirrus coverage as determined from MODIS that I can’t understand. Is this day/night difference representative of the difference in high cloudiness as determined from MODIS in other studies or is this a result of the cirrus screening procedure adopted in this study? In other words, are the operational MODIS products really this inconsistent with respect to the ability to identify high clouds consistently during daytime and nighttime?

Regarding figure 9: This figure shows a comparison between the MODIS and CALIOP cirrus cloud cover for daytime and nighttime. First, it isn’t clear what the individual points represent as this is not explained in the text. Are these annual regional means? Second, the daytime comparison is awful (MODIS considerably oveestimates cirrus cover compared to CALIOP), while the nighttime comparison is much better, which seems to contradict the discussion regarding the goodness of fit analyses that imply much more significant issues at night than during daytime. It’s acknowledged on line 466 that “MODIS will inevitably miss a significant portion of cirrus clouds due to its lower sensitivity. This comparison offers valuable insights into the practical efficiency of the MODIS instrument.” Yet, there is no attempt to explain the large daytime overestimates (false alarms) in cirrus cover from MODIS. It's impossible to know whether these are MODIS errors or the result of something related to the obscure definitions and confusing analysis methods undertaken in this study.

Regarding Fig 10: This figure shows the detection accuracies as a function of COT. The authors don’t clarify which sensor the COT is from. One might assume this is from MODIS since the CALIPSO signal saturates near a value of 4 (any values beyond that, if they exist, would have no meaning). However, the MODIS cloud mask misses many of the thinnest clouds that CALIPSO can detect (as shown in this study!) and the MODIS cloud optical property algorithm has a somewhat high failure rate for the thinnest of clouds that are detected. Therefore, if the results in fig 10 are with respect to MODIS COT, it isn’t clear how representative the values are at the low end of COT when matched with the MODIS pixel populations used to compute the statistical comparisons agains CALIPSO. Did this population all have corresponding successful COT retrievals? Or, is CALIPSO COT used in Fig 10? We don’t know! Also, it’s stated that “The most noticeable changes occur at COT values close to 10, though these may be influenced by the sample size, as the occurrence of cirrus clouds with a COT near 10 is limited or may represent a misclassification by CALIOP.” It’s difficult to know if this is an interesting finding or not since there is little discussion or attempt to explain it. I would like to know how the higher COT’s could be associated with CALIOP misclassifications? How can that be? It’s impossible to understand without a better explanation of the classification schemes adopted here for the two sensors.

Minor concerns:

The title doesn’t make sense to me. Cirrus detection is done with a cloud mask algorithm rather than cloud mask data. The data are the result of applying the algorithm. I suggest that you consider modifying the title. Here are two suggestions:
Comparison of Operational MODIS Cirrus Cloud Detections with CALIPSO data
Evaluation of the Operational MODIS Cloud Mask for Detecting Cirrus Clouds

Line 8-9: I suggest rephrasing to the following: “Our objective was to determine how well the operational cloud mask from the MODIS Science Team can be used to infer the presence of cirrus clouds relative to data products derived from the highly sensitive CALIOP instrument by the CALIPSO Science Team.”

Line 28-31: Suggest the following: “Globally, it’s been estimated that cirrus clouds have a net warming effect of 35.5 Wm-2 (Campbell et al., 2016; Kärcher, 2018;Lolli et al., 2017; Oreopoulos et al., 2017) in part because they trap and reduce outgoing longwave radiation more efficiently than they reflect solar radiation back to space.”
The following are probably not the most appropriate original citations for these phenomena but do provide examples. Therefore, it is appropriate to add citations for the original findings or cite in the following way:
Line 61: (e.g. Kortaba and Nguyen Huu…)
Line 67: (e.g. Heidinger and Pavolonis…)

Section 3 and later: Consider using references to the 5-degree areas as ‘regions’ rather than ‘pixels’

Line 85: change ‘in the range of’ to ‘at least’

Line 107: clarify what a ‘middle threshold’ is or remove

Line 163: change ‘other’ to ‘passive’

Line 296: It isn’t clear what P.P. means. Please clarify.

Line 297: Please briefly describe finding from Kortoba and Nguyen-Huu with regards to what you mean by ‘sampling frequency’ and how it affects the estimate of cirrus cloud fraction. Are you referring to occasional missing time periods in the CALIPSO record? If so, maybe it is more clear to say “can vary significantly due to occasional gaps in data availability due to instrument or spacecraft issues.”

Line 300: should read ’…detected at nighttime are 2-3 times higher those detected during daytime’

Line 315-320: This argument does not make sense to me. CALIOP is more sensitive at night, which means it should detect more thin clouds (higher cloud cover) which would lower the average COT, relative to daytime. It seems more likely to me that your analysis that indicates higher nighttime COT from CALIOP is either due to a real diurnal change in the nighttime cirrus COT, a retrieval algorithm artifact, an artifact of your screening method, or some combination of all of these.

Line 411: What do you mean by ‘reliable’? By what measure? For what applications are these measures deemed to be reliable and how is that determined? These questions should be answered if you are going to make such a definitive and broad statement. I suggest that you back off a bit and simply focus on summarizing the statistical findings.

Line 444-445: This statement seems overstated also and is more likely an assumption. What evidence have you shown that supports the contention that the detection accuracies you find are high enough to accurately monitor climate quality long-term changes in cirrus clouds?



Referee #2

General comments:

The Introduction is now more clear, more focused, and much improved. The simple example added to the discussion of bootstrap sampling is helpful and will make it clear to the community why bootstrapping was used to compute the performance metrics. Overall, the manuscript is much improved but a few additional changes are necessary to be ready for publication.

Specific comments:

Line 168 – the criteria for classification as Category 6 (pressure at cloud top less than 440 mb and nonopaque) should be mentioned here so the reader understands how this class is selected.

Lines 174-175 – When the CAD algorithm gives a CAD score near zero, the algorithm finds the probability of aerosol and the probability of cloud are nearly equal. This is most often because the detected ‘layer’ did not match the characteristics of either an aerosol or a cloud, often because the detection algorithm triggered on a noise spike or other signal artifact and not on an actual aerosol or cloud layer.

Line 288 – I found the use of “pixel” here to be confusing. I think this refers to a 5-degree lat-lon grid cell.

Lines 301-305. Sassen (2009) used an earlier version of the cloud product and also used different criteria for selecting and screening cloud layer data. Both of these likely contributed to differences when compared with the later results. Higher cirrus occurrence at night is primarily due to better sensitivity due to a lack of solar background (see Winker et al. 2024). The true diurnal difference in cirrus occurrence is complicated, as convective clouds have different diurnal cycles depending on geographic region. The day-night difference in background noise likely produces an artificial diurnal difference which outweighs the true diurnal differences.

Lines 317-319 – Yes, lidar systems are more sensitive to optically thicker clouds, but they also have much greater sensitivity at night due to a lack of solar background and higher signal-to-noise ratio. Whether higher frequency of cirrus detection at night is (partly) due to increased nighttime optical depth is open to debate.

Line 325 – I am still confused by “parameters that precluded the use of the test”, used in line 325 and the caption of Table 2. What does ‘parameters’ refer to? To me, a parameter is something like reflectance or radiance. The authors provided a clear response to my previous comment on this: By the phrase "precluded the use of the test," we meant that the specific indicators in question reach values that, in our judgment, make it impossible to use these tests directly for identifying Cirrus cloud masks. Given that the tests indicated by numbers in bold do not help in identifying cirrus for the cloud mask, do the bolded numbers in Table 3 give us a threshold value of the metric (ROP, POD, FAR, etc) where the test is not useful below (or above) that threshold? A little more explanation is necessary.

Line 471 – Figure 10 shows results as a function of cloud optical depth, up to an optical depth of 10. Winker et al. (2024) points out that CALIOP retrievals of cirrus with optical depth become very uncertain when the optical depth is greater than 2 or 3. Optical depth uncertainty can grow to much larger than 100%. The authors should consider whether this large uncertainty at large optical depths might impact the results shown. Technical corrections: There are several instances of ‘p.p.’ which I think should be ‘%’

Line 181 – polar orbits with 16-day revisit cycle

Line 182 – CALIPSO followed the Aqua spacecraft

Line 188 – only the 5 km product Line 258 – The balancing of the sample …

Line 261 – ‘more accurate results’, ‘more insightful results’, rather than ‘more reliable’?

Line 417 – indicates a very high level …

Line 419 – high values of POD are observed ?


Reference: Winker, D., X. Cai, M. Vaughan, A. Garnier, B. McGill, M. Avery and B. Getzewich, 2024: “A Level 3 monthly gridded ice cloud dataset derived from 12 years of CALIOP measurements”, Earth Syst. Sci. Data, 16, 2831–2855, https://doi.org/10.5194/essd-16-2831-2024.

Please let me know if you have any questions or concerns about the above.

Best wishes,

Andrew

Dear authors,

Thank you for your revision. I have read and thought about it in detail. I appreciate the additional work you have put in, though feel that some more revisions are necessary before it can be accepted. After acceptance, I would also suggest a general copy-editing by the journal.

1. I am not sure that I agree with the way the ISCCP classifications are framed here. The paper talks about these as a “detection” of cirrus but they are a post-hoc classification of an already-detected (and retrieved) cloud as a cirrus “type” based on retrieved quantities, not based on direct measurements (like the MODIS cloud tests are). So it doesn’t in my view make sense to talk about the fidelity of ISCCP classifications as a cirrus detection. Despite this, the abstract notes that all tests were applied to “MODIS radiances” (even though the ISCCP classifications are not). I am also not certain about the scientific motivation for using the ISCCP classifications in this way. As a reviewer notes, the failure rate of MODIS optical properties retrievals for optically-thin clouds mean that the proportion of pixels that could be used to calculate a “cirrus fractional coverage” or similar is somewhat reduced compared to the number of pixels that cloud masking tests are actually applied to. There is some literature which could be discussed on this, but in my mind a high failure rate means that using retrieval outputs as a classification and proxy "detection" does not quite make sense? I also have concerns about the applicability of the “cirrostratus” ISCCP classification (which may often be opaque to CALIOP due to the moderate optical depth up to 23) to “cirrus” as seen by CALIOP (and defined here, by my understanding, to require a transparent layer). These seem mutually incompatible. The “0 to 3.6” ISCCP cirrus bin seems more comparable to me, but if there is some COT threshold then this should probably be linked to CALIOP’s saturation point given that is chosen as the reference. Overall the ISCCP aspect does not feel well-connected to the rest of the analysis to me. I suggest thinking more deeply about what you are trying to convey with ISCCP classifications here and revising the framing of that discussion (and perhaps the analysis) accordingly. I think that some of this is what the reviewer was getting at. I am listing this in the system as "major revisions" to give you some more time/space to respond to this (though it might be that it does not require a large amount of work - so please do not interpret the phrase "major revisions" too deeply).

2. Please add a reference for Cohen’s kappa. Additionally, I suggest replacing the word “kappa” with the Greek letter throughout (rather than spelling the letter).

3. Please check acronyms are defined at first use in abstract and body text (e.g. CALIOP is out of order in the Abstract, but please check throughout).

4. Line 387: “A distribution resembling BT1.38 exhibited test ATC, but with an upward shift of about 10 percentage points” does not make sense, can you rephrase? I think this is a statement about distributions from different tests looking similar, but it was not clear.

Please let me know if you have any questions.

Best wishes,

Andrew

Dear authors,

Thank you for your further revisions of this manuscript. From my reading of your response and revised paper, I am pleased to accept this version for publication in AMT. I appreciate these extra revisions, which I feel have helped with the framing and clarity of some aspects of the paper.

Best wishes,

Andrew