Review of “Exploring the effect of training set size and number of categories on ice crystal classification through a contrastive semi-supervised learning algorithm”

This study proposes a new algorithm for ice crystal image classification: the contrastive semi-supervised learning algorithm (CSSL). Morphology is a major characteristic of atmospheric ice crystals, and its systematic retrieval can provide insight into the history of ice formation in clouds. Over the last five years, machine learning algorithms have been trained and deployed to successfully perform this task through the use of convolutional neural networks (CNNs). The corresponding models were either pre-trained (semi-supervised) or trained from scratch (fully supervised), but they were always trained and evaluated on hand-labeled data, which varies across studies. The CSSL algorithm aims to reduce the amount of hand-labeled data required and thereby limit the subjective nature of data collection and labelling. The CSSL algorithm is trained and applied for the first time to ice crystal images obtained in three recent research campaigns using a holographic imager. It is then compared to fully supervised CNNs.

Two main results are highlighted in this study:

1) with a relatively small decrease in accuracy, most of the manual hand-labeling can be avoided, and the accuracy loss diminishes with the number of classes (this is true for both fully supervised and semi-supervised algorithms);

2) the generalization ability of the models generated with the CSSL algorithm is on average slightly superior to that of the purely supervised model.

The idea is promising, and the potential to generate classification tools with high generalization capabilities is extremely valuable. If successful, this approach could shift the bottleneck of building classification tools from time-consuming hand-labeling to a more data-driven process, constituting a geometric growth in the number of images that can be used for training morphological recognition tools. The experiments are well-designed to test this hypothesis. However, the results are not entirely clear, as the non-pretrained, purely supervised baseline model achieves very similar performance as the models obtained with the CSSL algorithm. The conclusion is nuanced and raises relevant questions about the size of the training set and category distributions.

This study can be published with some minor revisions and the need to publish the data and code (see below).

Public code and data:

-An associated GitHub repository or making the code and data public is highly encouraged. This article uses holographic imager data and could inspire researchers working with other image types, such as CCD imagers, optical array probes, or even 2D scattering probes, for which a wealth of hand-labeled datasets and trained algorithms already exist. The experiments could thus be easily reproduced with other data types and campaign datasets to validate the general conclusions on the CSSL algorithm.

Minor comments:

- Some improvements can be made in the presentation of each model in the tables. Initially, the semi-supervised models are not straightforward to identify in Tables 4 and 5, which may carry over into further reading of the study. The two unsupervised models are listed in Table 4, while the two semi-supervised models are labeled as supervised models (which is technically true). The fact that both tables do not directly correspond to the experiments, due to the inclusion of the “Unsup” models and the classification of CSSL-generated models as “Sup,” may be confusing for readers unfamiliar with the employed technique.

- Large error bars are found in Figures 4, 9, and 10 for small training sets. Additionally, the baseline model (fully supervised, with varying training set sizes) shows virtually the same performance as the two CSSL generated models. For the sake of transparency and setting realistic expectations for the paper, these limitations and the relative success of the experiments could be made more apparent in the abstract.

- Typos remain in the text. For example, in Section 3, the end of line 116, “useful fatures,” should be corrected to “useful features.”

Review AMT-2024-31610:

"Exploring the effect of training set size and number of categories on ice crystal classification through a contrastive semi-supervised learning algorithm"

This manuscript presents a new method for ice crystal classification using a contrastive semi-supervised learning algorithm (CSSL). The work addresses a significant challenge in using deep learning approaches: the time-intensive process of manually labeling images for morphological classification. The authors demonstrate that their CSSL approach can achieve comparable results to fully supervised methods while requiring substantially less labeled data.

The main findings are:

1) The practical application of semi-supervised learning to reduce the manual labeling burden

2) A comprehensive comparison across different training set sizes and category numbers

3) The evaluation of model performance across three research campaigns

Technical notes:

The computational requirements and training times for both CSSL and baseline models should be discussed, as these are relevant for practical implementation. The community would highly benefit if the code and data for the CSSL algorithm were made publicly available.

The manuscript is generally well-written and structured. The figures effectively support the main arguments, but some improvements in visualisation could enhance clarity:

Major comments:

- Figure 1 would benefit from additional scale bars

- Could you discuss the potential for transfer learning to other imaging systems that capture lower-quality crystal images (e.g., those with coarser resolution) such as:

    - VIZZZ:  https://amt.copernicus.org/articles/17/899/2024/ or 

    - PIP/2DVD: https://amt.copernicus.org/articles/15/5141/2022/ (only binary black-and-white images)

Minor corrections:

- Line 90: Typo: ".in" -> ". In"

- Line 116: Typo: "fatures" -> "features"

- Line 178: Typo: "accurracy" -> "accuracy"

- Line 203: Missing reference citation

- Line 245: Unclear sentence structure needs revision

Additional comment:

While rerunning the analysis with fewer convolutional layers would be beyond the scope of this review, it would be valuable if you could elaborate on the choice of network architecture and its implications. In particular, the use of 49 convolutional layers raises questions about computational efficiency versus model performance. Could a shallower network potentially achieve similar results with reduced computational overhead? 

This work represents another step forward towards better ice crystal classification. After addressing the suggested revisions, the manuscript can be published. The potential impact of this work may also extend beyond ice crystal classification to other areas where manual labeling data sets for deep learning is currently a bottleneck.

Recommendation: Accept with minor revisions