the Creative Commons Attribution 4.0 License.
the Creative Commons Attribution 4.0 License.
A Comparative Evaluation of Snowflake Particle Size and Shape Estimation Techniques used by the Precipitation Imaging Package (PIP), Multi-Angle Snowflake Camera (MASC), and Two-Dimensional Video Disdrometer (2DVD)
- 1Mesoscale Atmospheric Processes Laboratory, NASA Goddard Space Flight Center, Greenbelt, MD, USA
- 2NASA Postdoctoral Program—Universities Space Research Association, Columbia, MD, USA
- 3Joint Center for Earth Systems Technology, University of Maryland, Baltimore County, Baltimore, MD, USA
- 4Climate and Space Sciences and Engineering Department, University of Michigan, Ann Arbor, MI, USA
- 1Mesoscale Atmospheric Processes Laboratory, NASA Goddard Space Flight Center, Greenbelt, MD, USA
- 2NASA Postdoctoral Program—Universities Space Research Association, Columbia, MD, USA
- 3Joint Center for Earth Systems Technology, University of Maryland, Baltimore County, Baltimore, MD, USA
- 4Climate and Space Sciences and Engineering Department, University of Michigan, Ann Arbor, MI, USA
Abstract. Measurements of snowflake particle size and shape are important for studying the snow microphysics. While a number of instruments exist that are designed to measure these important parameters, this study focuses on the measurement techniques of three digital video disdrometers: the Precipitation Imaging Package (PIP), the Multi-Angle Snowflake Camera (MASC) and the Two-Dimensional Video Disdrometer (2DVD). To gain a better understanding of the relative strengths and weaknesses of these instruments and to provide a foundation upon which comparisons can be made between studies using data from different instruments, we perform a comparative analysis of the measurement algorithms employed by each of the three instruments by applying the algorithms to snowflake images captured by PIP during the ICEP-POP 2018 field campaign.
Our analysis primarily focuses on the measurements of area, equivalent diameter, and aspect ratio. Our findings indicate that area and equi-area diameter measurements using the 2DVD camera setup should be the most accurate, followed by MASC, which is slightly more accurate than PIP. In terms of the precision of the area and equi-area diameter measurements, however, MASC is considerably more precise than PIP or 2DVD, which provide similar precision once the effects of the PIP image compression algorithm are taken into account. Both PIP and MASC use shape-fitting algorithms to measure aspect ratio. While our analysis of the MASC aspect ratio suggests the measurements are reliable, our findings indicate that both the ellipse and rectangle aspect ratios produced by PIP under-performed considerably due to the shortcomings of the PIP shape-fitting techniques. That said, we also demonstrate that reliable measurements of aspect ratio can be retrieved from PIP by reprocessing the PIP images using either the MASC shape-fitting technique or a tensor-based ellipse-fitting technique. Because of differences in instrument design, 2DVD produces measurements of particle horizontal and vertical extent rather than length and width. Furthermore, the 2DVD measurements of particle horizontal extent can be contaminated by horizontal particle motion. Our findings indicate that, although the correction technique used to remove the horizontal motion contamination performs remarkably well with snowflakes despite being designed for use with rain drops, the 2DVD measurements of particle horizontal extent are potentially unreliable.
Charles Nelson Helms et al.
Status: final response (author comments only)
-
RC1: 'Comment on amt-2021-427', Anonymous Referee #1, 30 Jan 2022
amt-2021-427: Helms et al. A Comparative Evaluation of Snowflake Particle Size and Shape Estimation Techniques used by the Precipitation Imaging Package (PIP), Multi-Angle Snowflake Camera (MASC), and Two-Dimensional Video Disdrometer (2DVD)
In short: This study compares the different algorithms behind the measurement techniques of three digital video disdrometers: the Precipitation Imaging Package (PIP), the Multi-Angle Snowflake Camera (MASC), and the Two-Dimensional Video Disdrometer (2DVD) in observing snowflakes. The focus is on defining the uncertainties in the defined area influencing the equivalent diameter, and the aspect ratio. The authors quantify the motion blurring, in the case of PIP also the image compression, the shape-fitting measurements, and in the case of 2DVD, the estimate of the bounding box measurement when particle horizontal motion needs to be adjusted with an unskewing algorithm.
The topic is interesting and relevant for surface observations and the development of retrieval methods in global monitoring of snowfall. The study has novelty in the way it examines the measurement algorithms internal to the instruments, which typically are not transparent to the end-user of the data. The theory is clearly outlined with illustrative examples, and the conclusions are well-supported and valid. The manuscript is well written and provides a clear storyline, however, at least in my opinion, it leaves the reader questioning what is the magnitude/importance of these studied uncertainties in respect to the other uncertainties e.g. particles out of focus or only a fraction of particles observed, wind effects to particle fall velocity, miss-matching of particles, partially illuminated measurement space or limited observations of particles from only one plane projection, these are referred in several publications prior to this one. I would like to see more discussion on this topic and references to other related studies. My recommendation is to publish the paper after addressing this concern and some small remarks mentioned below.
Minor comments:
Line 11: “… PIP or 2DVD which provide similar precision once the effects of the PIP image compression algorithm are taken into account.” This sentence is somehow unclearly connected to the previous statement in lines 10-11 and is not clear for a reader who just reads the abstract. Please rephase.
Line 41: “There are numerous examples of studies which rely heavily on either of these measures of particle size.” The statement “numerous examples” follows only by two references, leaving for example relevant fields such as the snow model or satellite retrieval development unmentioned. I would like to see a broader scan of the research field, just mentioning applications and example references would be enough.
Line 64: “separate from the snowflake size,” This is unclear. I don’t understand how these mentioned studies are concerning only aspect ratio separate from size. Please rephrase.
Line 87: Altitude of the site?
Line 89-91: I would like to see more data of this event to support the assumptions of aggregation and lump graupels, e.g., time series of temperature, PSD, and mean fall velocity.
Line 336-8: “Even with a very fast fall speed of 4 m/s, the overestimation of the equivalent diameter for very large circular particles (diameter ~ 10 mm) is approximately two orders of magnitude or smaller than the actual equivalent diameter.” Two orders of magnitude? This is not clear to me.
Line 339: “perfectly circular”. Why assumed the particle to be circular, though written in lines 68-69 “aspect ratio is frequently prescribed, often with a mean value of 0.6 assumed (e.g., Matrosov et al., 2005)” and then without quantification stated that for the oblate particles “the relative (and absolute) effects of motion blurring on the area and equivalent diameter measurements will also grow”. Please justify and elaborate.
Figure 4: Could you add the number of analyzed particles and a density plot would add information instead of a scatter plot.
Figure 6. Just to add more information about the particle habit, could the approx. fall velocity be added to the corner of the image? The colored fitted shapes, could the line be slightly thinner or the image larger, it is now hard to see the lines in respect to the shaded image, they are all on top of each other.
Lines 404-406. As PIP is only seeing a plane projection of the particle, but here the particle is referred to as an ellipsoid, it is confusing whether in this perimeter stretching factor analysis, the computations are performed in 3D with ellipsoids and is it then assumed the same axes ratio in both directions or is it performed in the 2D projection. Could you please clarify this?
Paragraphs 408 – 445: I understood that this section provides explanations why the ellipse-fit in PIP has an arbitrary upper threshold close to 0.6, and why with the rectangular fit in PIP, there is a gap in aspect ratio between 0.9 and 1.0. However, it was not always clear, which “gap” the authors were pointing at. I would suggest that you would refer in the text (when addressing for the first time) to the image, where the “gap” is shown. E.g. in lines 436-438, I assume here the authors are referring to Figure 5b?
Figure 9. Same as Figure 4. It would be nice to see the number of analyzed particles and then rather a density plot than a scatter plot.
Lines 526-527: “and, as a result, the maximum dimension and aspect ratio measurements are unreliable; however, the PIP variables other than the ellipse and rectangle dimensions appear to be reliable” I assume here it is referred that the PIP-fitted maximum dimension of an ellipse is unreliable and not that the observed maximum dimension is unreliable. Please clarify.
Lines 528-530: “As the present study has demonstrated, the PIP imagery can be reprocessed and reliable measurements of maximum dimension (the previous comment) and aspect ratio can be made via the application of an alternative ellipse-fitting algorithm, such as the MASC or tensor-based algorithms.” In the manuscript, it was described that the AVI file contains only the first 2000 frames from the 10 - minute section, and with 380 frames per second, this translates to 5.3 seconds of data. It is unclear that can an end-user reprocess the whole data volume or just the sample frames in the AVI-files? Could this be elaborated?
- AC1: 'Reply on RC1', Charles N. Helms, 25 Apr 2022
-
RC2: 'Comment on amt-2021-427', Anonymous Referee #2, 03 Feb 2022
This is an important contribution to understanding how various optical imagers observe properties of snowfall. Overall, the manuscript is understandable. While I have no qualms with the included analysis, there does appear to be some low hanging fruit that would greatly improve the impact of the paper. These and other comments are highlighted below.
Specific comments:
Horizontal motion should/needs to be addressed. Sampling/flow issues aside (and this is extremely important but beyond the scope of the article), the open design of PIP will lead to more translational motion vs. the MASC. Based on the field campaign data, there should be surface wind data you could use that would help guide upper-values for analyses like Figure 3. Further, I would like to see how this relates to the PIP vs. 2DVD analysis (e.g. Figure 9).
The normal caveats apply to studies based on one case during a campaign. Is there the opportunity to conduct this analysis on other cases? How many were available during ICE-POP 2018? More reasoning is needed. While this case provided lots of variety, it is important to demonstrate how varied results are for other types of cases as perhaps you could demonstrate for certain types of cases, the discrepancies between instruments is either amplified or diminished.
The manuscript could be more concise by merging the Data and Instruments sections. For example, the first paragraph in section 3.1 is essentially duplicative with the material under data. I would remove this, then make the sections on the instruments as sections 2.x.
Technical comments:
Line 200: is vs. will be performed?
Paragraph (255-268): This paragraph could be cleaned up. Examples include multiple ‘For simplicity’ phrase and you could omit ‘it should be noted’. I would lead off with the 2nd sentence to remind the reader, than discuss the number of factors that aren’t addressed rather than revisiting the ‘number of factors’ phrase.
Line 275: Agreed, but you should probably provide a citation for this statement. The larger concern is potential horizontal motion
Line 304: Extra ‘both’ in this sentence.
Line 321: How about: Motion blur of the top (bottom)- edge pixels occurs when the particle leaves (enters) those pixels during the image exposure period.
Figure 3: The sentence starting with ‘Calculations… and Specifically…’ is repetitive with the body of the text and does not describe the visual properties of the figure. I would omit for brevity or restate in text instead of the caption.
Figure 4: best fit lines? I would omit the last sentence in the figure caption as this is already included in the text right after Line 355.
Figure 7: Once again, some of the caption is discussed in text (sentences starting with ‘This’ and the first ‘The’.
Line 447: ‘Because the 2DVD’
- AC2: 'Reply on RC2', Charles N. Helms, 25 Apr 2022
-
RC3: 'Comment on amt-2021-427', Anonymous Referee #3, 03 Feb 2022
The sudy is an interesting evaluation of processing algorithms to derive two characteristic dimensions, length and width, of snow particles from 2-dimensional images.
Algorithms from three instruments, PIP, MASC, and 2DVD, and their resulting dimensions are evaluated. The conclusions allow future users of these instruments to chose a suitable algorithm.
Only PIP data are used. Using emulated data for testing other algorithms, which are used with MASC or 2DVD data. This provides a fair comparison of the algorithms only. However, this method cannot compare the actual qualities of the PIP, MASC, or 2DVD measurements related to specific instrumental issues. In particular, incase of MASC and 2DVD, the method cannot evaluate if the PIP-derived emulated or the actual measurements (if available) would more accurately represent the particle's dimensions.
The study clearly describes the chosen method and recognizes its limitations.
I suggest publication of the study after some shortcomings have been addressed.Major issues:
1) All conclusions are vaguely formulated, some are only speculative.
In general the analysis is not sufficiently quantative in comparing the resulting derived measurements from the various algorithms. Similarly, the Abstract uses many "should" and leaves doubts about the usefulness of the conclusions.
Quantify and better describe things like "spread", "agreement", "reasonable estimates", and "greatly underestimates".
Examples:
Improve discussion around Fig.4. In these scatter plots it is difficult to see the differences tated in the discussion of for example 4a) vs. 4b) or 4d) vs. 4e). As suggested by Referee 1, density plots can be more useful. In addition, some quantitative statistical measures will be useful (e.g. to better argue that "MASC-fitted and tensor-fitted ellipses tend to produce fairly similar particle dimensions", L. 394, and that "MASC-fitted ...ellipses tend to provide more reasonable estimates of particle dimensions...", L 396).
L. 371-372: reformulate this sentence, using a certain aspect ratio will not increase the spread in dimensions.
Fig.9: a) not needed, instead argue/discuss that the two heights are the same.; b) quantify the range (and maybe distribution as histogram) of ratios between 2DVD width and PIP width (I guess they vary between 0.8 and 20 and will show two modes on a histogram); c) quantify similarly and then compare to b). That will allow for less vague descriptions than "can sometimes underestimate the bounding box width" (L. 473) or "are surprisingly accurate" (L. 477).2) The ellipse-fitting algorithm (PIP, MASC, or tensor variant) could also be applied to 2DVD measurements. This can be tested within this study on the emulated 2DVD measurements, adding a valuable aspect to 2DVD measurements. Then, the conclusion about the limited usefulness of the 2DVD measurements can be revisited (L. 519-521.)
3) Sect 5.1, L256-258:
"For this study, however, we will examine the theoretical accuracy and precision of the MASC, PIP, and 2DVD area and equivalent diameter measurements in terms of motion blur as determined from the pixel resolution, exposure time (i.e., shutter speed), and particle fall speed."
Motion blur is not related to precision. Overall, I find the discussion around "precison" unnecessary and not well introduced (only later on in Sect 5, L347, it is mentioned what precision or "precise measurement" refers to. The effects of this theoretically higher precision are, however, not discussed in this study. If the authors consider this to be an important aspect of their study, then I would recommend to evaluate the consequences by using the same algorithm with differing pixel resolutions. As MASC measurements are only emulated here, the actual effects of the higher precision remain unclear (and are not part of this study and instrumemnt specific and related to questions such as if the increased precision is accompanied by a corresponding better optical resolution and accuracy).4) Sect 5.1 and Figures 2 and 3:
The discussion of motion blur and its effects on accuracy seems to be wrong. It is correct that, considering a vertical particle motion during expeosure, the blurring affects both the upper and lower edge. However, the particle extension is not increased (blurred) upwards and downwards. At the start of the exposure, the particle has an upper and a lower edge. Both these edges are moving (blurring) downwards, i.e. blurring will not add extra pixel(s) above, only below. By incorretly assuming added pixels above and below, the authors seem to overestimate blurring by a factor of two.5) The arteficial "cap" is not explained satisfactorily.
Instead, the value of the cap is translated in a certain perimeter stretching factor. However, the authors do not try to explain, why no smaller perimeter stretching factors exist. Assuming that (L. 442-443) only few particles have a smaller perimeter stretching factor seems wrong, I guess (from looking at Fig. 4.g) that there is not a single particle with smaller perimeter stretching factor.
The reason for this cap is likely to be found in effects of pixelation affecting the perimeter by artificially extending it, more noticeably for smaller particles (~0.5mm) than for larger ones.
Having said this, I need to remark that it should be discussed how the perimeter is determined.
L. 410-413: The pixelation effects should be considered.
Fig.7: Reformulating the discussion around the artificial cap may result in that Fig 7 is not needed. E.g., currently the whole discussion about it in L. 418-445 is difficult to understand and doesn't explain the cause of the cap.
Reconsider the usefulness of Fig.8.6) Similarily, the apparent gap between aspect ratios around 0.9 and 1.0 is not explained properly (L. 375-376). It seems to stem from the fact that there is a minimum perimeter stretching factor that is above 1 in case a rectangle(square is fit instead of an ellipse/circle. There is no gap, but all particles with smaller perimeter stretching factor are simply "piling up at the aspect ratio of 1.0.
7) Using ellipses or rectangles that best fit the particle can be used to describe shape, they are, however, not sufficient as complete measurements of the particle's shape. The limitations of the evaluated algorithms could be highlighted better.
Other minor or technical issues:
Terminology:
Inconsistent use of terminology:
E.g. "tensor method" only used twice (L179-180 "hereaftyer referred to as the tensor method" and L242 "referred to here as the tensor method"), elsewhere "tensor-fitted ellipse" or "tensor-fitted ellipse method" or "tensor-fitted ellipse measurement"
Or: Inconsistent use of "resolution", not always used correctly. L100 "resolution" refers to the size on the particle that corresponds to one pixel. This is later more adequately referred to "pixel resolution" (e.g. L.269) or "pixel size" (L. 314).Maximum dimension is not used in this study. The term "maximum dimension" is, however, used three times in the Conclusions. The authors likely wanted to refer to an ellipse- or rectangle-fitted dimension.
Sect 3.3:
Make it clear that the viewing planes are horizontal and that they are separated vertically by 6 mm (or 7?).
Discuss how the "piecing together" of the single line scans is carried out and what errors or accuracies are to be expected. Is the sentence in L. 517-519 ("highly accurate") true?
Provide information on pixels and pixel resolution (as done for PIP and MASC in 3.1 and 3.2).L. 199-200 reformulate "made" (measurements are doen or carried out), e.g.
"... before the MASC measurements are emulated by using the same ..."L.205: "a five pixel particle" is ambiguous as the PIP measured particle image and the emulated MASC image have different pixel resolutions. Use something like "a five PIP-pixel particle".
L. 214: "product of the particle fall speed and the camera observation frequency" seems wrong, should it be v/f?
Sect. 4.3: Specify that the tensor elements are mean values of the quantities (e.g. square of Delta y) for all particle pixels (or otherwise explain better eq. 1).
L. 377-378: Repeated use of "expected" and unclear when the increase in aspect ratio (or the period of lump graupel) is.
L. 389-390: "lack of a warm nose" and its implications should be explained if that is relevant for the discussion.
L. 399-401. While it seems intuitively obvious what the sentence tries to explain, it needs to be refromulated for correctness and clarity.
L. 402-403: remove last part of sentence ("note, extending the short...") to improve clarity.
L.473-476: reconsider the explanation, it seems that the example particle should move to the left while moving down to be compressed horizontally.
L.247 correct spelling: "eingenvalues"
L. 304 delete duplicate "both"
- AC3: 'Reply on RC3', Charles N. Helms, 25 Apr 2022
Charles Nelson Helms et al.
Charles Nelson Helms et al.
Viewed
| HTML | XML | Total | BibTeX | EndNote | |
|---|---|---|---|---|---|
| 431 | 109 | 20 | 560 | 7 | 7 |
- HTML: 431
- PDF: 109
- XML: 20
- Total: 560
- BibTeX: 7
- EndNote: 7
Viewed (geographical distribution)
| Country | # | Views | % |
|---|
| Total: | 0 |
| HTML: | 0 |
| PDF: | 0 |
| XML: | 0 |
- 1