the Creative Commons Attribution 4.0 License.
the Creative Commons Attribution 4.0 License.
Spectral replacement using machine learning methods for continuous mapping of Geostationary Environment Monitoring Spectrometer (GEMS)
- Department of Climate and Energy Systems Engineering, Ewha Womans University, Seoul, 03760, Republic of Korea
- Department of Climate and Energy Systems Engineering, Ewha Womans University, Seoul, 03760, Republic of Korea
Abstract. Earth radiance in the form of hyperspectral data contains useful information on atmospheric constituents and aerosol properties. The Geostationary Environment Monitoring Spectrometer (GEMS) is an environmental sensor measuring such hyperspectral data in the ultraviolet and visible (UV/VIS) spectral range over the Asia-Pacific region. After successful completion of the in orbit test of GEMS in October 2020, bad pixels are found as a remaining calibration issue to be updated with follow-up treatment. Currently, one-dimensional interpolation in the spatial direction is performed in operation to replace the erroneous pixels of GEMS, which causes high interpolation error for a wider defect area on a detector array. To resolve the issue, this study suggests machine learning methods with artificial neural network (ANN) and multivariate linear regression (Linear) to fill in a spectral gap of defective spectra. The machine learning models are trained with normal measurements to emulate spectral relations between input and output radiances in a spectrum. For efficient training, dimensionality reduction for the input radiances is applied with principal component analysis (PCA) prior to the training process. The results show that the defect area at the wavelengths of strong absorption lines is better replaced with PCA-ANN with the error of 5 %, while PCA-Linear is better for reproducing radiances having strong correlation with input radiances. The shorter the spectral range of output radiances is, the smaller the prediction error is with PCA-Linear (0.5–5 %). Spectral and spatial discontinuity caused by real bad pixels can be significantly improved with the trained machine learning models especially for wide defect areas. This study verifies that spectral relations of radiances in the UV/VIS spectrum are successfully reproduced with a simple machine learning model, which has high potential to be investigated further for enhancing measurement quality of environmental satellite measurements.
Yeeun Lee et al.
Status: final response (author comments only)
-
RC1: 'Comment on amt-2022-37', Glen Jaross, 25 Feb 2022
I have difficulty understanding the goals of this paper. There seems to be a mismatch between what the authors are trying to achieve and the investigation they describe in the paper. The authors describe a problem on GEMS whereby “bad” pixels, i.e. missing pixel radiances, cause problems in the ensemble of measured radiances. These problems are not fully described. The authors refer to a previous approach to deal with the missing pixels involving radiance interpolation across the missing areas. That approach does not yield good results according to their description. There is no mention of what criteria are used to assess the results.
But what is the real problem that the authors are trying to solve? In an instrument such as GEMS, designed to measure trace gas composition of the atmosphere through hyperspectral measurements, the goal is probably to produce trace gas products without spatial gaps caused by missing radiances. Does anyone really care if the missing radiance fields are filled with more realistic values if they do not result in more accurate representations of the trace gas fields? The authors do not discuss the issue of trace gas retrievals or other products derived from their predicted radiances. Is there any improvement at all in those products?
Furthermore, the authors fail to discuss a more fundamental question. Is replacement of missing radiances with predicted radiances of any value? These are not measurements, yet they may be provided to the retrieval algorithms as though they are measurements. Is this what the producers of GEMS atmospheric products really want? Surely those producers do not want to report an atmospheric parameter as a measurement when in fact it is merely a prediction based on previous measurements. If their goal is to merely remove spatial gaps in maps of the trace gases, this can be accomplished more effectively and efficiently by manipulating the gas concentration values directly, i.e. spatial smoothing techniques.
The investigation the authors describe is not without merit. From an academic perspective the question of how well ML techniques can describe Earth backscattered radiances is an interesting one. If the authors approached this paper from that perspective they might provide a useful contribution to our ability to characterize the atmosphere through numerical techniques. But in doing so they must take a more rigorous approach to evaluating their radiance predictions.
The first thing the authors should do is to forget about the GEMS pixel defects. These are of no use in evaluating the efficacy of the technique since the true radiances remain unknown for these regions. Instead, choose regions of the detector where there are good measurements and treat them as missing for the purpose of deriving errors. There can be a variety of region shapes and sizes, including ones that look very similar to Defects 1, 2, and 3.
The authors must also devise evaluation criteria that are more robust and quantitative than “these spectra look realistic.” Since the goal for GEMS radiances is to derive atmospheric products such as trace gases, perhaps these trace gas retrievals can be used as the metric. Merely stating that predicted radiances agree on average with measured radiances to within X% ignores the subtle spectroscopic sensitivity of trace gases such as NO2, where the exact relationship between wavelengths is of utmost importance.
I hope the authors can revise this investigation so that the results are more meaningful. It has the potential to be an interesting paper, but in its current form it represents incomplete work.
- AC1: 'Reply on RC1', Yeeun Lee, 06 May 2022
-
RC2: 'Comment on amt-2022-37', Anonymous Referee #2, 10 Mar 2022
<General comments>
This study proposes methods for replacing GEMS radiances measured at bad pixels. The manuscript is well written, with an informative description of GEMS measurements. However, I believe the demonstration of the method validity can be improved. In particular, the qualitative discussion about the performance of reproduction, described with Figs. 11-12, needs to be improved and more objective. Also, in addition to the prediction errors presented in Figs. 7–9, it is recommended to show how well the proposed methods reproduce known good spectra (i.e., actual measurements). For example, panels can be added in Figs. 11-12 to show examples of reproduction for known good spectra. (The message would be more straightforward if measured spectra were overlaid with the reproduced data.) Besides, how can the spectral sampling of input/output (0.1 nm) be finer than the original GEMS data (0.2 nm)? More detailed descriptions about this are recommended. Overall, I suggest this manuscript be reconsidered after major revisions.
<Specific comments>
- Line 78: Please give the full names of the gaseous species (i.e., O3, SO2, NO2, and HCHO).
- Line 82: The authors refer to each of ~700 east-west pixels as a “scan,” but probably this term is not accurate. Isn’t the whole ~700 pixels considered to be in one scan? Also, can GEMS cover the entire field of regard by one scan? It seems that is what the authors are implying.
- Line 84: Do the CCD pixel numbers presented here represent those for only photoactive pixels?
- Line 89: The general description of the bad pixel detection method is informative. But how about presenting how long the GEMS integration time is (by adding another sentence)?
- Line 99: This sentence sounds as if the results of 1-D interpolation were presented earlier, which is not true. How about rephrasing this sentence, using a verb like “imply” instead of “indicate”?
- Line 104: The subject affected by the defective pixels is the quality of ozone retrieval, not the ozone properties themselves.
- Line 148: How can the spectral interval of input and output (0.1 nm) be narrower than that of original GEMS measurements (0.2 nm)? How are the GEMS measurement spectra sampled onto the finer grids? Please give more details here.
- Line 149: Did you investigate how much the results changed when trained without SZA and VZA? Please describe the impacts of including these variables.
- Figure 5: The caption and the color bar title do not correspond. Which wavelength was used between 310 and 354 nm?
- Line 264: How can we tell if spectra look “reasonable”? This statement is vague. Please consider changing Figs. 11-12 to include any reference (know, good, measured) spectra for the reconstructed parts.
- Line 269: I believe the term “noise” itself implies randomness, which would not necessarily be canceled in the normalized radiance. Please consider replacing the term with another, e.g., error, bias, artifact, etc.
- Please consider re-writing the units in the figures as W cm–3 sr–1.
- Please consider minor English corrections below.
- Lines 42, 49, 50, 100: affect to -> affect ?
- Lines 109, 148, 184, 185, 199, 214, 221, 238, 241, 243, 250, 254, 258, 268: Defect -> Defects
- Line 225: Fig. -> Figs.
- Line 242: N-S -> North-South
- Line 276: A period (.) missing between sentences
- AC2: 'Reply on RC2', Yeeun Lee, 06 May 2022
Yeeun Lee et al.
Yeeun Lee et al.
Viewed
| HTML | XML | Total | BibTeX | EndNote | |
|---|---|---|---|---|---|
| 284 | 91 | 17 | 392 | 7 | 7 |
- HTML: 284
- PDF: 91
- XML: 17
- Total: 392
- BibTeX: 7
- EndNote: 7
Viewed (geographical distribution)
| Country | # | Views | % |
|---|
| Total: | 0 |
| HTML: | 0 |
| PDF: | 0 |
| XML: | 0 |
- 1