the Creative Commons Attribution 4.0 License.
the Creative Commons Attribution 4.0 License.
Drone-based photogrammetry combined with deep-learning to estimate hail size distributions and melting of hail on the ground
Martin Lainer
Killian P. Brennan
Alessandro Hering
Jérôme Kopp
Samuel Monhart
Daniel Wolfensberger
Urs Germann
Abstract. Hail is a major threat associated with severe thunderstorms and an estimation of the hail size is important for issuing warnings to the public. Operational radar products exist that estimate the size of the expected hail. For the verification of such products, ground based observations are necessary. Automatic hail sensors, as for example within the Swiss hail network, record the kinetic energy of hailstones and can estimate with this the hail diameters. However, due to the small size of the observational area of these sensors (0.2 m2) the estimation of the hail size distribution (HSD) can have large uncertainties. To overcome this issue, we combine drone-based aerial photogrammetry with a state-of-the-art custom trained deep-learning object detection model to identify hailstones in the images and estimate the HSD in a final step. This approach is applied to photogrammetric image data of hail on the ground from a supercell storm, that crossed central Switzerland from southwest to northeast in the afternoon of June 20, 2021. The hail swath of this intense right-moving supercell was intercepted a few minutes after the passage at a soccer field near Entlebuch (Canton Lucerne, Switzerland) and aerial images of the hail on the ground were taken by a commercial DJI drone, equipped with a 50 megapixels full frame camera system. The average ground sampling distance (GSD) that could be reached was 1.5 mm per pixel, which is set by the mounted camera objective with a focal length of 35 mm and a flight altitude of 12 m above ground. A 2D orthomosaic model of the survey area (750 m2) is created based on 116 captured images during the first drone mapping flight. Hail is then detected by using a region-based Convolutional Neural Network (Mask R-CNN). We first characterize the hail sizes based on the individual hail segmentation masks resulting from the model detections and investigate the performance by using manual hail annotations by experts to generate validation and test data sets. The final HSD, composed of 18209 hailstones, is compared with nearby automatic hail sensor observations, the operational weather radar based hail product MESHS (Maximum Expected Severe Hail Size) and some crowdsourced hail reports. Based on the retrieved drone hail data set, a statistical assessment of sampling errors of hail sensors is carried out. Furthermore, five repetitions of the drone-based photogrammetry mission within about 18 min give the unique opportunity to investigate the hail melting process on the ground for this specific supercell hailstorm and location.
- Preprint
(31116 KB) - Metadata XML
- BibTeX
- EndNote
Martin Lainer et al.
Status: final response (author comments only)
-
RC1: 'Comment on amt-2023-89', Anonymous Referee #1, 07 Aug 2023
Review of AMT-2023-89
“Drone-based photogrammetry combined with deep-learning to estimate hail size distributions and melting of hail on the ground” by M. Lainer et al.
Summary: The authors report on drone-based measurements of hail sizes after an event in Switzerland that produced hail up to ~4 cm in maximum dimension. They describe a new, deep-learning based technique to automatically identify and size hailstones from the drone imagery. The technique is an improvement/extension (or at least a twist) on the methods reported in Soderholm et al. (2020, AMT). A particularly valuable contribution of this paper is the multiple drone missions to observe hailstone melting rates. The content is highly relevant to the hail community and is a timely contribution.
Unfortunately, I was disappointed to find at least 3 examples of plagiarism (see below) from websites in the description of some of the methods. I did not check for further instances, because the journal should have some ability to do so. In my view, plagiarism is a serious offense, and thus I recommend rejection at this time. However, I do find the research to be useful, and I do hope that the authors can rewrite the plagiarized portions of the manuscript in their own words, and address the other comments and suggestions below.
Major Comments:
1. I found at least 3 examples, based on where the style/tone of the writing abruptly changed. The first is on Lines 154-156. The text from the manuscript is as follows:
“An orthomosaic is a photogrammetrically orthorectified image product that has been mosaicked from an image collection, correcting for geometric distortion and color matching the image data to create a seamless mosaic data set.”
and from the ArcGIS website (https://pro.arcgis.com/en/pro-app/latest/help/data/imagery/generate-an-orthomosaics-using-the-orthomosaic-wizard.htm#:~:text=An%20orthomosaic%20is%20a%20photogrammetrically,produce%20a%20seamless%20mosaic%20dataset):
“An orthomosaic is a photogrammetrically orthorectified image product mosaicked from an image collection, where the geometric distortion has been corrected and the imagery has been color balanced to produce a seamless mosaic dataset.”
The second is on Lines 168-169. The text from the manuscript:
“The library serves as a processing pipeline for reconstructing camera poses and 3-dimensional scenes from multiple images. Here we make use of some basic modules for SfM: Feature detection, feature matching, minimal solvers.”
is largely taken from the github page for this software (https://github.com/mapillary/OpenSfM/blob/main/README.md):
“OpenSfM is a Structure from Motion library written in Python. The library serves as a processing pipeline for reconstructing camera poses and 3D scenes from multiple images. It consists of basic modules for Structure from Motion (feature detection/matching, minimal solvers) with a focus on building a robust and scalable reconstruction pipeline.”
The third is from Lines 192-193. The text from the manuscript:
“Object detection is a technology related to computer vision and image processing that tries to detect instances of semantic objects of a certain class (e.g. cats, dogs, cars, buildings, etc.) in digital images and videos.”
is taken from the following website (https://www.credly.com/skills/image-processing-object-detection#:~:text=Object%20detection%20is%20a%20computer,in%20digital%20images%20and%20videos.):
“Object detection is a computer technology related to computer vision and image processing that deals with detecting instances of semantic objects of a certain class (such as humans, buildings, or cars) in digital images and videos.”
2. Section 2.1: I appreciate the detailed information and experiences in this section, but it comes across as a little bit “preachy” or reads like pontification. Please take a look at this section and try to trim it down to what is necessary and germane for the main story about the new technique.
3. How would this motion blur affect the hail size results? This should at least be mentioned here, even if the answer is “not at all” so you do not leave the readers wondering.
4. Lines 339-340: The authors should make a note here that the aspect ratios reported are probably not the same as measured in other studies (Knight 1986, Shedd et al. 2021), which are the measured maximum and minimum axes of the hailstones. What the drone sees are the projected maximum and minimum axes, based on whichever way the hailstone happens to be laying on the field. If hailstones are perfectly oblate spheroids, you would always capture the maximum dimension but not always the minimum dimension. Because hailstones tend to be ellipsoidal or irregular, this means your axis ratios probably do not correspond to the true stone axis ratios.
Minor comments/Typos/Grammar issues:
1. Line 24: “asses” should be “assess”
2. Line 29: I think “alps” should be capitalized? Same in Line 60?
3. Line 39: probably more accurate or clearer to say “maximum dimension” instead of “diameter” (the latter connotes a sphere or circle)
4. Line 52: no comma after “known”
5. Line 53: A more comprehensive and more recent study is by Shedd et al. (2021, JAS) that looks at hailstone shapes; consider comparing the Knight (1986) results to those of Shedd et al. here.
6. Line 54: Soderholm et al. (2020, AMT) also report on the axis ratios, correct?
7. Line 56: “decent” is a bit informal, is there a way to quantify what this means?
8. Line 61: what is the lowercase s? Is this South? If so, it might be clearer to spell it out. Update, it happens again in Line 62, so I don’t know what this means. Please spell it out.
9. Line 74: “respectively” is used incorrectly here, should read as “at a distance of 770 m and 1470 m, respectively, to the NNE of…”
10. Figure 1, caption: the description in the caption regarding EMLs and “loaded gun” belongs in the text. However, “loaded gun” is a bit colloquial, consider using other terminology. Check on the convention for how to portray units (i.e., m/s or m s^-1, etc.) for AMT. Finally, explain or provide a legend for what the colors mean in the hodograph, and indicate the units (m/s or kts?) for the rings on the hodograph.
11. Figure 2: consider enlarging the dots for the hail reports, they are all very small and hard to see.
12. Section 2: the first 3 or 4 lines are probably not needed, since they are just telling readers what is coming up. How about just start with the material? Similarly, the second sentence of subsection 2.1 can be removed, it is useless for the narrative of the paper.
13. Line 106: no comma needed after “found”
14. Line 113: “Aside the” should be “Aside from the”
15. Lines 119-121 are not needed – it is pretty obvious that any field experiment would require good forecasting ahead of time! Just start with “During days with conditions favorable for supercells,” or something like that.
16. Line 130 and elsewhere, “hereby” is not the correct word to use here. Please revise.
17. Line 136: “hail core punch” is too colloquial, please revise.
18. Line 141: “analyses” should be “analysis”
19. Lines 142-147: Even though these are important points for storm chasers, I don’t think these are appropriate for the manuscript because they aren’t relevant for reporting on the technique and results. Please remove.
20. Line 206: no comma after “mentioned” (And, if you’re writing it in the paper, it seems worthy of mentioning. Best practice is to not include text like “It is worth mentioning” etc. and just cut to the chase with the important points).
21. Line 231: “tow” should be “two”
22. Line 241: the brackets usage for quotes here needs to be changed to conform to AMT’s convention/guidelines. This occurs throughout the manuscript.
23. Lines 354-355: This is repeating the finding from the first sentence in the paragraph; combine these two sentences into one and keep them together in the text (otherwise the logic is jumping around).
24. Lines 356-367: There are several one-sentence paragraphs here; simply combine them into a coherent paragraph with connecting sentences or words.
25. Line 377: Does the changing shapes of the larger hailstone agree with the cartoon drawn in Shedd et al. (2021)? In other words, is there evidence that protuberances or lobes melt more rapidly than the rest of the stone, tending to “smooth” the stones?
26. Line 387: No comma needed after “range” (or after “bins” on the next line)
27. Line 389: But, certainly, physics tells us that there should be some dependence on size, right? One can refer to Rasmussen and Heymsfield (1987, JAS), for example.
28. Line 394: This is important information that could be included earlier in the text, near the description of the event!
Citation: https://doi.org/10.5194/amt-2023-89-RC1 - AC2: 'Reply on RC1', Martin Lainer, 19 Oct 2023
-
RC2: 'Comment on amt-2023-89', Anonymous Referee #2, 31 Aug 2023
Review of Drone-based photogrammetry combined with deep-learning to estimate hail size distributions and melting of hail on ground
Lainer et al. 2023 AMT
This study present a significant advancement on the initial drone-based technique developed by Soderholm (2020), in particular the derivation on melting rates to estimate the original HSD and simulation of hail pad measurements. This is really great work! The authors also improve the underlying methodology and explore the parameter tuning in much more details.
It was unfortunate to see the very serious findings of plagiarism from R1. Further, I believe the manuscript needs to be reviewed by a scientific english language editor first before resubmission. The corrections to make to the language were too numerous that I did not document them and instead tried focused on the science content. The text also uses many superfluous words and sentences with repetition that could be removed, significantly reducing the length. I’d encourage the authors to take these actions on board and prepare a resubmission.
Comments/corrections:
Line 24: "asses"
Line 38-39: Crowd sourced data might not even provide the largest diameter, just the size at some unknown percentile
Line 59: The short sentence about chasing needs some more context - who and why?
Line 59: How was the supercell track generated in figure 2a?
Line 62: Please add in the values for storm motion to provide some context for comparing against other events.
Paragraph ending line 75: I think some work is needed to improve the flow between this paragraph and the next perhaps by moving the limitations of hail impact sensors here as motivation for aerial surveys
Sentence starting line 78: Some duplication in this sentence around "melting", which is mentioned twice
Line 81: Please keep units consistent, either mm or cm
Line 91: The use of "chain" could be improved with "methodology chain"
Start of section 2: I find it's often clearer for the reader to be more direct, e.g., "
Here we first go into the challenging part" to "First we discuss the"
Lines 119-128: I don't think this paragraph is necessary to support this paper. The methodology of storm chasing is quite specific to the region and the individual.
Lines 142-147: I feel this is more a reflection on the storm chasing approach that is specific to the authors experiences. It's not necessary to support this paper.
Line 150: Earlier it was started that a 50 MP full frame camera was used. Also MP needs to be expanded.
Line 180: That's an extremely high ISO! Were there any issues with noise or over exposure?
Line 183: I'm not sure what the convention is for AMT with notation for number (either comma for full stop). Might be worth checking. Also this needs to be made consistent throughout the text as there’s many numbers written within any separator.
Line 203: ResNet should be expanded in the previous paragraph where it is first introduced
Line 221: The sentence starting "The idea behind..." can be merged with the next sentence to make the text less verbose
Line 231: "tow"
Line 244: It might be worth explaining to the reader how the term epoch is used for deep learning
Lines 259-260: I feel this statement about the methodology is already covered in 2.3.1 - "Because we want the test data set to be locked down until we are confident enough about our trained model, we do another division and split a validation set out of the train set. In this scenario we end up with three data sets." Further, "train" should be "training"
Line 268-269: How extensive was this manual QC to remove non-hail objects? It might have been worthwhile including some tiles with these uncommon non-hail objects in the training.
Paragraph 270: I think this could be shortened significantly by outlining the parameter space of F1 with a reference.
line 287: "quadruple variation of the learning rate" could be improved with "four different learning rate values tested"
Lines 310: "Right tail" should be "Upper tail". I'd also be more specific than saying "smaller devices"
Lines 312: "logarithmic view" is not needed in the main text, this belongs in the figure caption.
Line 313: check use of 'maximal' I think maximum is more suitable.
line 343: I'm unclear what the author is trying to assert with "We note that the HSD is considered at the scale of a single hail cell."
line 354-355: repeated from the start of the paragraph. This is some really nice work too!
line 363-364: Can you use the time series information from these disdrometers to separate the two hail events for HS3?
lines 376: please avoid repeating information from the caption "Those hailstones shrink from initially 33 mm to 21 mm, respectively 25.5 mm, during the course of 1119 s."
lines 385: I feel a more effective plot for the analysis of melting rates would be to use initial size bins, fit a linear fit to each size bin and plot the slope. This would directly show the melting rate for different sizes.
line 388: This analysis of 48 hailstones doesn't seem necessary as you can't confirm a robust result and isn't completely described (where is the hail from, what sizes, etc).
Section 3.2: I'm curious how this experiment would go considering only severe hail sizes (e.g., above 20 mm). But there might not be enough information from the hail disdrometers for a comparison.
Section 4 paragraph 1: I don't think this opening paragraph is needed as this information is discussed again later. Further, it doesn't flow well into the second paragraph.
line 395-400: I would clarify that dry growth produces high densities of microscope air bubbles. Wet growth definitely soaks, but it also accretes on top of existing outer later too.
line 401: "pure" isn't needed here. Also "In a first step" should be "In a first attempt". Please also update "Second step" in line 408.
Line 419: I'm unclear how "Also a cropped hailstone binary mask can still lead to the correct major axis length." I would argue that it would lead to a negative bias.
Line 422: I wonder if soaking during melting is the main driver of changes in brightness
Line 428: Ryzhkov et al. 2013 uses simulations of melting hail to estimate changes in polarimetric radar information (which is later used to develop a retrieval). So this isn't a radar study of melting hail.
Paragraph starting on 421: This paragraph feels incomplete. I'd suggest removing it if the authors can't link this into the results.
Line 473: "different ages" should be "the duration".
Table 4: The information about the different comparison points should be in the text, not the caption.
Figure 1: Please reduce the number of wind barbs so it's readable! Can you please also annotate the hodograph with the levels or indicate what they are in the figure caption.
Figure 2: Please check all the text and annotations in this figure can be read at 100% zoom when rendered. The font size is also not consistent across the subplots. The white box and magenta cross in (b) are not visible at 100% zoom and I also can't find HS1 in subplot (c). Finally, I'd suggest not repeating the same information in both the caption and the main text; for example "corresponding to the MeteoSwiss app categories: smaller than coffee bean, coffee bean, 1 CHF coin, 5 CHF coin and tennis ball), are given." is repeated in both.
Figure 3: Please just describe colours as their proper names (e.g., dark red, light grey and green). Also, there overlap in (d) is significant and I can't really get much from it. Can you just show the center locations perhaps?
Figure 4: Is this really a spaghetti plot? I would describe this as line plots.
Figure 8: HSD should be expanded in the title. and # replaced with "hail count"
Figure 9: Font sizes is not consistent for percentiles and I think the colours change? I could extend these lines such that the text sits on top of the highest bars (with no overlap).
Figure 10: (a) Q25 and Q75 lines should ideally be different colours. Issue with percentile font sizes again. X-axis labels on (c) are also a bit too close with that font size. Caption: "from virtually and random placed hail sensors" reads better as "from simulated hail sensors at random locations"
Figure 11: Can you please add the time since first capture above the columns of images? This will be useful to info the reader about the duration since first capture. The final sentence of the caption could be improved with "During the 1119 s these hailstones shrink about 12 mm (upper row) and 7.5 mm (lower row) in their major axis length."
Figure 12: I don't think the log view adds much value to this analysis because the sample size in the upper tail is so small. The main message is carried well by plot (a). I'd suggest replacing the use of "map" with "flight" or "survey". I'd also suggest changing "secure" to "capture". Finally, how many hailstones are in this sample during flight 1?
General comment:
I suspect that hail is most likely to fall such that the major and intermediate axes are visible from drone imagery. The minimum axis is most likely orientated to the vertical as the centre of mass is lowest to the ground at this most, and therefore has a (likely) high stability. This should be considered when discussing aspect ratio as a function of the major and minor axis.
Citation: https://doi.org/10.5194/amt-2023-89-RC2 - AC1: 'Reply on RC2', Martin Lainer, 19 Oct 2023
Martin Lainer et al.
Martin Lainer et al.
Viewed
| HTML | XML | Total | BibTeX | EndNote | |
|---|---|---|---|---|---|
| 306 | 97 | 26 | 429 | 13 | 11 |
- HTML: 306
- PDF: 97
- XML: 26
- Total: 429
- BibTeX: 13
- EndNote: 11
Viewed (geographical distribution)
| Country | # | Views | % |
|---|
| Total: | 0 |
| HTML: | 0 |
| PDF: | 0 |
| XML: | 0 |
- 1