the Creative Commons Attribution 4.0 License.
the Creative Commons Attribution 4.0 License.
Development of the MEaSUREs blue band water vapor algorithm – Towards a long-term data record
Gonzalo González Abad
Chris Chan Miller
Hyeong-Ahn Kwon
Caroline R. Nowlan
Zolal Ayazpour
Heesung Chong
Xiong Liu
Kelly Chance
Ewan O'Sullivan
Robert Spurr
Robert J. Hargreaves
Abstract. We report the development of an algorithm for the retrieval of Total Column Water Vapor (TCWV) from blue spectra obtained by satellite instruments such as the Ozone Monitoring Instrument (OMI). The algorithm is implemented in an automatic processing pipeline and will be used to generate a long-term data record as part of a MEaSUREs project. TCWV is calculated as the ratio between the Slant Column Density (SCD) and Air Mass Factor (AMF). Both these factors are improved upon previous work by incorporating more constraints or physical processes. For the SCD, we have optimized the retrieval window to 432–466 nm, performed a temperature correction, and employed a new stripe-removal post-processing routine. The use of OMI Collection 4 spectra reduces the fitting uncertainty by ~9 % with respect to Collection 3. For the AMF, we perform on-line radiative transfer using VLIDORT. Over land surfaces, we use bi-directional reflectances based on MODIS products. Over the oceans, we consider surface roughness and water-leaving radiance, and we find that water-leaving radiance is important for avoiding large TCWV biases over the oceans.
Under relatively clear conditions, the MEaSUREs data are well correlated with the reference datasets, having correlation coefficients of r ~0.9. Over the oceans, MEaSUREs-AMSR_E has an overall mean (median) of ~ 1 mm (0.6 mm) with a standard deviation of σ ~6.5 mm, though large systematic differences in certain regions are also found. Over land surfaces, MEaSUREs-GPS has an overall mean (median) of -0.7 mm (-0.8 mm) with σ ~5.7 mm. Even a small amount of cloud can introduce large bias and scatter; thus, without further correction, strict data filtering criteria are required. However, the MEaSUREs TCWV data can be corrected through machine learning. In this regard, under all-sky conditions, the mean bias of MEaSUREs reduces from 4.5 mm (without correction) to -0.3 mm (with correction using LightGBM models), and the standard deviation decreases from 11.8 mm to 3.8 mm. We also examined the representation error of the GPS stations using the dense GEONET data. The within-pixel variance of TCWV varies with grid size following a power law dependence. At 0.25°×0.25° resolution, the derived representation error is about 1.4 mm.
- Preprint
(10124 KB) - Metadata XML
- BibTeX
- EndNote
Huiqun Wang et al.
Status: open (extended)
-
RC1: 'Comment on amt-2023-66', Anonymous Referee #1, 01 Aug 2023
reply
General comments
In recent years, there has been a steady improvement in TCWV retrievals in the visible blue spectral region, aiming to exploit long-term satellite measurement series for climatological research. These advances have included improvements in spectral analysis as well as in the conversion of SCDs to VCDs, for example, by using flexible a priori profiles instead of monthly/climate average profiles for AMF calculation.
This study by Wang et al. describes an update to a TCWV retrieval algorithm in the visible blue spectral region for OMI measurements. The update includes a temperature correction of the SCD, new fit settings or a new fit window, destriping algorithms, and an optimized AMF calculation. Furthermore, the retrieval is compared with satellite and in situ reference data sets. Likewise, the retrieval output is further improved through the use of a machine learning algorithm.
Overall, the study shows many interesting and innovative improvements compared to the work of the same authors on predecessors of the algorithm. However, before the study can be published, I have various major issues that need to be addressed.
Major issues
- Temperature correction
The manuscript explains that for the temperature correction, the temperature profile is weighted by the box AMF profile or by the profile of scattering-weights. However, this approach lacks coherence and may leads to inaccurate temperature values, because the estimated temperature does not align with the actual distribution of the gas in the atmosphere. For instance, if a gas only occurs in the boundary layer, the temperature in the upper troposphere is irrelevant. Instead, what should be done is to apply a weight to the temperature profile using the (a priori) gas profile or the product of a priori profile and the box-AMF profile. Furthermore, it is also unclear to what extent this correction is actually feasible due to the cross-correlations between the cross-sections, which means the cross-section is influenced differently at various temperatures.
- Fit window optimization
The optimization of the fit window relies on spectral analyses from OMI collection Version 3. However, this approach renders the analysis outdated, considering that Version 4 is expected to include significant updates, particularly in the irradiance (resulting in fewer flagged spectral pixels). Consequently, the entire section, along with the conclusions drawn from it, needs to be revisited using Version 4. Additionally, the analysis is currently based solely on one orbit, which is inadequate.
- Row anomaly
Since mid-2007, the OMI measurements have been affected by the so-called “row anomaly” (Schenkeveld et al, 2017). However, the paper does not make any mention of this issue, and there is no discussion on how it impacts the retrieval, both quantitatively and qualitatively If the aim is to utilize this retrieval for the long-term measurement series of OMI, more attention must be given to addressing this matter thoroughly.
- Destriping
The issues related to the fit window optimization and the row anomaly have direct implications for the destriping algorithm. As a result, the destriping process must be revised to accommodate these factors appropriately.
- AMF calculation
Borger et al. (2020) and Chan et al. (2022) have clearly demonstrated that employing a static water vapor profile as an a priori for the AMF calculation is notably less effective compared to using a flexible a priori in retrievals. Given this evidence, it raises a question as to why this study continues to employ the static water vapor profile in its AMF calculation.
- Validation study
In the validation study, only early periods are examined which remain unaffected by both the row anomaly and the significant degradation of OMI. As a result, it is imperative to include more recent periods, considering these two factors in the analysis. Moreover, for the comparison over land, the absence of a global dataset limits the analysis to specific regions only. To address this, additional datasets (such as reanalysis or reference data) should be incorporated for a more comprehensive assessment.
Furthermore, the authors missed an opportunity by not utilizing the ML-optimized TCWV dataset in the validation study. Instead, the "standard" retrieval employs very strict filter criteria. This raises questions about the usefulness of the retrieval, as it leads to the exclusion of significant portions of the OMI orbits. This is particularly concerning for long-term analyses and clear-sky biases.
- OMI Level 1 collection 3 vs collection 4
Here and there it becomes challenging to discern for which L1 version the assessments were conducted. It would greatly benefit the reader if only one version (preferably Collection 4) is consistently used throughout the study. Additionally, it is recommended to include a paragraph or section in the manuscript that outlines the differences between Collection 3 and Collection 4 to provide clarity on the chosen version and its implications.
Minor/technical issues
Section 2: I think it would be helpful to write down the equation to make the differences to DOAS clear. How are the terms accounting for changes in the ISRF from Beirle et al. (2017) implemented in this scheme?
L93: It should be mentioned that the H2O cross-section is still subject to large uncertainties in this spectral range and many researcher utilize different versions of HITRAN (e.g. Borger et al., 2020; Chan et al., 2022).
L95: In this context, it should be clarified whether HW1e refers to the full width at half maximum (FWHM) of the spectral line.
L100: Which updates have been implemented that affect the visible blue spectral range? Please clarify.
L138: “(TCWV < 5mm)”: For spectral analysis, the SCD is crucial, not the VCD.
L144f: In addition to the mentioned references, it would be relevant to include Borger et al. (2020) and Borger et al. (2023) for TROPOMI and OMI, respectively. Regarding Garane et al. (2023), since it is a validation study, it could be replaced with Chan et al. (2022).
L150ff: Wouldn’t it be simpler to use SCD/σ(SCD) instead of two separate variables? How do you determine the “fraction of valid retrievals” and what are the criteria for valid retrievals? I don’t really understand the “common mode amplitude”. Can you provide a formula?
L177ff: The discussion should also include the fitting uncertainty or SCD uncertainty as it is relevant to the analysis.
L218ff: Why is an Earthshine spectrum not used for destriping the OMI orbit? For instance, it is frequently employed in NO2 or HCHO retrievals (Anand et al., 2015; De Smedt et al., 2018). Moreover, Borger et al. (2023) successfully utilized it for their OMI TCWV retrieval.
L229ff: The stripes also change since OMI has a new irradiance every 15 orbits.
L245ff: The calculation of the correction vector for each row typically involves the use of how many pixels? Doesn't the filtering eventually lead to a bias towards cloudy pixels or brighter scenes due to their lower fit RMS? How are rows affected by the row anomaly handled in the analysis?
L263: What is the purpose of the "reflection extension"?
L271f: Instead of using "optically thin assumption," it might be more appropriate to use "weak absorber assumption”.
L275: Borger et al. (2020) should be included as a reference.
L284: Please add a reference for IPA (e.g., Cahalan et al., 1994).
L308ff: It would be beneficial to consider using the cloud and albedo information from the L2 NO2 product (Lamsal et al., 2021) in the analysis. This product takes into account BRDF effects and offers a consistent approach to handling these parameters.
L316: How do you address the different spatial resolutions between the OMI pixel and the MODIS product in the analysis?
L324f: Are the MERRA-2 winds monthly means?
L330: Instead of using "cloud scattering," it may be more accurate to refer to "cloud shielding" in this context.
L346f: NO2 is primarily concentrated in the boundary layer and, as a result, exhibits a profile shape completely different from the water vapor profile. A comprehensive error analysis of AMFs in TCWV retrievals can be found in Borger et al. (2020).
L349: What is the accuracy of the OMCLDO2 product?
L376: What is the reason for imposing an upper limit for the AMF, and how was this limit determined?
L376ff: With respect to the complete OMI orbit, what proportion of OMI pixels is utilized in the TCWV product?
L390ff: It should be noted that Chan et al. (2020) and Garane et al. (2023) employ different cloud filters compared to this study. Additionally, Borger et al. (2020) should also be mentioned here. Based on the outcomes, one might question whether the retrieval from Wang et al. (2019) performs better than the update presented in this study.
L460ff: This suggests that the presented retrieval might not be as robust as desired. It would be highly insightful to investigate whether using a flexible a priori profile, as demonstrated in Borger et al. (2020) and Chan et al. (2020, 2022), could potentially address this issue.
L444: The regions with high deviations seem to coincide with regions experiencing frequent cloud cover, such as the SPCZ. Alternatively, these deviations could also be indicative of inaccuracies in the a priori profile.
L466: Do you use monthly mean or daily profiles of temperature and water vapor for the GBM? To ensure the GBM model's generalization, it might be preferable to omit latitude and longitude information.
Section 4.2.1: It would be valuable to include a comparison of the semivariograms of the two datasets, as suggested by Souri et al. (2022).
L579: “clear” --> “clear-sky”
Table 1 "Shape parameter k": Is this referring to the asymmetry parameter? Also, why is the term for the parameter "w" not included?
Figure 7: It would be helpful to zoom into an orbit to enhance the visibility of the destriping effectiveness.
Figure 13: Additionally, including a map showing the distribution of deviations would aid in identifying systematic error sources, such as coastlines and mountains.
Figure A9: the color where there are no points, possibly by using a mincnt option, to improve the clarity of the figure.
Figure A10: The stock image contains colors similar to the colorbar, particularly over the oceans. It is recommended to remove the stock image to avoid confusion.
References
Anand, J. S., Monks, P. S., & Leigh, R. J. (2015). An improved retrieval of tropospheric NO2 from space over polluted regions using an Earth radiance reference. Atmospheric Measurement Techniques, 8(3), 1519-1535.
Beirle, S., Lampel, J., Lerot, C., Sihler, H., & Wagner, T. (2017). Parameterizing the instrumental spectral response function and its changes by a super-Gaussian and its derivatives. Atmospheric Measurement Techniques, 10(2), 581-598.
Borger, C., Beirle, S., Dörner, S., Sihler, H., & Wagner, T. (2020). Total column water vapour retrieval from S-5P/TROPOMI in the visible blue spectral range. Atmospheric Measurement Techniques, 13(5), 2751-2783.
Borger, C., Beirle, S., & Wagner, T. (2023). A 16-year global climate data record of total column water vapour generated from OMI observations in the visible blue spectral range. Earth System Science Data, 15(7), 3023-3049.
Cahalan, R. F., Ridgway, W., Wiscombe, W. J., Bell, T. L., & Snider, J. B. (1994). The albedo of fractal stratocumulus clouds. Journal of the Atmospheric Sciences, 51(16), 2434-2455.
Chan, K. L., Valks, P., Slijkhuis, S., Köhler, C., & Loyola, D. (2020). Total column water vapor retrieval for Global Ozone Monitoring Experience-2 (GOME-2) visible blue observations. Atmospheric Measurement Techniques, 13(8), 4169-4193.
Chan, K. L., Xu, J., Slijkhuis, S., Valks, P., & Loyola, D. (2022). TROPOspheric Monitoring Instrument observations of total column water vapour: Algorithm and validation. Science of The Total Environment, 821, 153232.
De Smedt, I., Theys, N., Yu, H., Danckaert, T., Lerot, C., Compernolle, S., ... & Veefkind, P. (2018). Algorithm theoretical baseline for formaldehyde retrievals from S5P TROPOMI and from the QA4ECV project. Atmospheric Measurement Techniques, 11(4), 2395-2426.
Lamsal, L. N., Krotkov, N. A., Vasilkov, A., Marchenko, S., Qin, W., Yang, E. S., ... & Bucsela, E. (2021). Ozone Monitoring Instrument (OMI) Aura nitrogen dioxide standard product version 4.0 with improved surface and cloud treatments. Atmospheric Measurement Techniques, 14(1), 455-479.
Schenkeveld, V. M., Jaross, G., Marchenko, S., Haffner, D., Kleipool, Q. L., Rozemeijer, N. C., ... & Levelt, P. F. (2017). In-flight performance of the Ozone Monitoring Instrument. Atmospheric measurement techniques, 10(5), 1957-1986.
Souri, A. H., Chance, K., Sun, K., Liu, X., & Johnson, M. S. (2022). Dealing with spatial heterogeneity in pointwise-to-gridded-data comparisons. Atmospheric Measurement Techniques, 15(1), 41-59.
Citation: https://doi.org/10.5194/amt-2023-66-RC1
Huiqun Wang et al.
Data sets
MEaSUREs blue band total column water vapor sample data for the Ozone Monitoring Instrument H. Wang, G. González Abad, C. Chan Miller, H. Kwon, C. R. Nowlan, Z. Ayazpour, H. Chong, X. Liu, K. Chance, E. O'Sullivan, K. Sun, R. Spurr, and R. J. Hargreaves https://doi.org/10.5281/zenodo.7795648
Huiqun Wang et al.
Viewed
HTML | XML | Total | BibTeX | EndNote | |
---|---|---|---|---|---|
182 | 45 | 9 | 236 | 7 | 6 |
- HTML: 182
- PDF: 45
- XML: 9
- Total: 236
- BibTeX: 7
- EndNote: 6
Viewed (geographical distribution)
Country | # | Views | % |
---|
Total: | 0 |
HTML: | 0 |
PDF: | 0 |
XML: | 0 |
- 1