the Creative Commons Attribution 4.0 License.
the Creative Commons Attribution 4.0 License.
| 27 Jun 2023
Development of the MEaSUREs blue band water vapor algorithm – Towards a long-term data record
Abstract. We report the development of an algorithm for the retrieval of Total Column Water Vapor (TCWV) from blue spectra obtained by satellite instruments such as the Ozone Monitoring Instrument (OMI). The algorithm is implemented in an automatic processing pipeline and will be used to generate a long-term data record as part of a MEaSUREs project. TCWV is calculated as the ratio between the Slant Column Density (SCD) and Air Mass Factor (AMF). Both these factors are improved upon previous work by incorporating more constraints or physical processes. For the SCD, we have optimized the retrieval window to 432–466 nm, performed a temperature correction, and employed a new stripe-removal post-processing routine. The use of OMI Collection 4 spectra reduces the fitting uncertainty by ~9 % with respect to Collection 3. For the AMF, we perform on-line radiative transfer using VLIDORT. Over land surfaces, we use bi-directional reflectances based on MODIS products. Over the oceans, we consider surface roughness and water-leaving radiance, and we find that water-leaving radiance is important for avoiding large TCWV biases over the oceans.
Under relatively clear conditions, the MEaSUREs data are well correlated with the reference datasets, having correlation coefficients of r ~0.9. Over the oceans, MEaSUREs-AMSR_E has an overall mean (median) of ~ 1 mm (0.6 mm) with a standard deviation of σ ~6.5 mm, though large systematic differences in certain regions are also found. Over land surfaces, MEaSUREs-GPS has an overall mean (median) of -0.7 mm (-0.8 mm) with σ ~5.7 mm. Even a small amount of cloud can introduce large bias and scatter; thus, without further correction, strict data filtering criteria are required. However, the MEaSUREs TCWV data can be corrected through machine learning. In this regard, under all-sky conditions, the mean bias of MEaSUREs reduces from 4.5 mm (without correction) to -0.3 mm (with correction using LightGBM models), and the standard deviation decreases from 11.8 mm to 3.8 mm. We also examined the representation error of the GPS stations using the dense GEONET data. The within-pixel variance of TCWV varies with grid size following a power law dependence. At 0.25°×0.25° resolution, the derived representation error is about 1.4 mm.
This preprint has been withdrawn.
-
Withdrawal notice
This preprint has been withdrawn.
-
Preprint
(10124 KB)
Interactive discussion
Status: closed
-
RC1: 'Comment on amt-2023-66', Anonymous Referee #1, 01 Aug 2023
General comments
In recent years, there has been a steady improvement in TCWV retrievals in the visible blue spectral region, aiming to exploit long-term satellite measurement series for climatological research. These advances have included improvements in spectral analysis as well as in the conversion of SCDs to VCDs, for example, by using flexible a priori profiles instead of monthly/climate average profiles for AMF calculation.
This study by Wang et al. describes an update to a TCWV retrieval algorithm in the visible blue spectral region for OMI measurements. The update includes a temperature correction of the SCD, new fit settings or a new fit window, destriping algorithms, and an optimized AMF calculation. Furthermore, the retrieval is compared with satellite and in situ reference data sets. Likewise, the retrieval output is further improved through the use of a machine learning algorithm.
Overall, the study shows many interesting and innovative improvements compared to the work of the same authors on predecessors of the algorithm. However, before the study can be published, I have various major issues that need to be addressed.
Major issues
- Temperature correction
The manuscript explains that for the temperature correction, the temperature profile is weighted by the box AMF profile or by the profile of scattering-weights. However, this approach lacks coherence and may leads to inaccurate temperature values, because the estimated temperature does not align with the actual distribution of the gas in the atmosphere. For instance, if a gas only occurs in the boundary layer, the temperature in the upper troposphere is irrelevant. Instead, what should be done is to apply a weight to the temperature profile using the (a priori) gas profile or the product of a priori profile and the box-AMF profile. Furthermore, it is also unclear to what extent this correction is actually feasible due to the cross-correlations between the cross-sections, which means the cross-section is influenced differently at various temperatures.
- Fit window optimization
The optimization of the fit window relies on spectral analyses from OMI collection Version 3. However, this approach renders the analysis outdated, considering that Version 4 is expected to include significant updates, particularly in the irradiance (resulting in fewer flagged spectral pixels). Consequently, the entire section, along with the conclusions drawn from it, needs to be revisited using Version 4. Additionally, the analysis is currently based solely on one orbit, which is inadequate.
- Row anomaly
Since mid-2007, the OMI measurements have been affected by the so-called “row anomaly” (Schenkeveld et al, 2017). However, the paper does not make any mention of this issue, and there is no discussion on how it impacts the retrieval, both quantitatively and qualitatively If the aim is to utilize this retrieval for the long-term measurement series of OMI, more attention must be given to addressing this matter thoroughly.
- Destriping
The issues related to the fit window optimization and the row anomaly have direct implications for the destriping algorithm. As a result, the destriping process must be revised to accommodate these factors appropriately.
- AMF calculation
Borger et al. (2020) and Chan et al. (2022) have clearly demonstrated that employing a static water vapor profile as an a priori for the AMF calculation is notably less effective compared to using a flexible a priori in retrievals. Given this evidence, it raises a question as to why this study continues to employ the static water vapor profile in its AMF calculation.
- Validation study
In the validation study, only early periods are examined which remain unaffected by both the row anomaly and the significant degradation of OMI. As a result, it is imperative to include more recent periods, considering these two factors in the analysis. Moreover, for the comparison over land, the absence of a global dataset limits the analysis to specific regions only. To address this, additional datasets (such as reanalysis or reference data) should be incorporated for a more comprehensive assessment.
Furthermore, the authors missed an opportunity by not utilizing the ML-optimized TCWV dataset in the validation study. Instead, the "standard" retrieval employs very strict filter criteria. This raises questions about the usefulness of the retrieval, as it leads to the exclusion of significant portions of the OMI orbits. This is particularly concerning for long-term analyses and clear-sky biases.
- OMI Level 1 collection 3 vs collection 4
Here and there it becomes challenging to discern for which L1 version the assessments were conducted. It would greatly benefit the reader if only one version (preferably Collection 4) is consistently used throughout the study. Additionally, it is recommended to include a paragraph or section in the manuscript that outlines the differences between Collection 3 and Collection 4 to provide clarity on the chosen version and its implications.
Minor/technical issues
Section 2: I think it would be helpful to write down the equation to make the differences to DOAS clear. How are the terms accounting for changes in the ISRF from Beirle et al. (2017) implemented in this scheme?
L93: It should be mentioned that the H2O cross-section is still subject to large uncertainties in this spectral range and many researcher utilize different versions of HITRAN (e.g. Borger et al., 2020; Chan et al., 2022).
L95: In this context, it should be clarified whether HW1e refers to the full width at half maximum (FWHM) of the spectral line.
L100: Which updates have been implemented that affect the visible blue spectral range? Please clarify.
L138: “(TCWV < 5mm)”: For spectral analysis, the SCD is crucial, not the VCD.
L144f: In addition to the mentioned references, it would be relevant to include Borger et al. (2020) and Borger et al. (2023) for TROPOMI and OMI, respectively. Regarding Garane et al. (2023), since it is a validation study, it could be replaced with Chan et al. (2022).
L150ff: Wouldn’t it be simpler to use SCD/σ(SCD) instead of two separate variables? How do you determine the “fraction of valid retrievals” and what are the criteria for valid retrievals? I don’t really understand the “common mode amplitude”. Can you provide a formula?
L177ff: The discussion should also include the fitting uncertainty or SCD uncertainty as it is relevant to the analysis.
L218ff: Why is an Earthshine spectrum not used for destriping the OMI orbit? For instance, it is frequently employed in NO2 or HCHO retrievals (Anand et al., 2015; De Smedt et al., 2018). Moreover, Borger et al. (2023) successfully utilized it for their OMI TCWV retrieval.
L229ff: The stripes also change since OMI has a new irradiance every 15 orbits.
L245ff: The calculation of the correction vector for each row typically involves the use of how many pixels? Doesn't the filtering eventually lead to a bias towards cloudy pixels or brighter scenes due to their lower fit RMS? How are rows affected by the row anomaly handled in the analysis?
L263: What is the purpose of the "reflection extension"?
L271f: Instead of using "optically thin assumption," it might be more appropriate to use "weak absorber assumption”.
L275: Borger et al. (2020) should be included as a reference.
L284: Please add a reference for IPA (e.g., Cahalan et al., 1994).
L308ff: It would be beneficial to consider using the cloud and albedo information from the L2 NO2 product (Lamsal et al., 2021) in the analysis. This product takes into account BRDF effects and offers a consistent approach to handling these parameters.
L316: How do you address the different spatial resolutions between the OMI pixel and the MODIS product in the analysis?
L324f: Are the MERRA-2 winds monthly means?
L330: Instead of using "cloud scattering," it may be more accurate to refer to "cloud shielding" in this context.
L346f: NO2 is primarily concentrated in the boundary layer and, as a result, exhibits a profile shape completely different from the water vapor profile. A comprehensive error analysis of AMFs in TCWV retrievals can be found in Borger et al. (2020).
L349: What is the accuracy of the OMCLDO2 product?
L376: What is the reason for imposing an upper limit for the AMF, and how was this limit determined?
L376ff: With respect to the complete OMI orbit, what proportion of OMI pixels is utilized in the TCWV product?
L390ff: It should be noted that Chan et al. (2020) and Garane et al. (2023) employ different cloud filters compared to this study. Additionally, Borger et al. (2020) should also be mentioned here. Based on the outcomes, one might question whether the retrieval from Wang et al. (2019) performs better than the update presented in this study.
L460ff: This suggests that the presented retrieval might not be as robust as desired. It would be highly insightful to investigate whether using a flexible a priori profile, as demonstrated in Borger et al. (2020) and Chan et al. (2020, 2022), could potentially address this issue.
L444: The regions with high deviations seem to coincide with regions experiencing frequent cloud cover, such as the SPCZ. Alternatively, these deviations could also be indicative of inaccuracies in the a priori profile.
L466: Do you use monthly mean or daily profiles of temperature and water vapor for the GBM? To ensure the GBM model's generalization, it might be preferable to omit latitude and longitude information.
Section 4.2.1: It would be valuable to include a comparison of the semivariograms of the two datasets, as suggested by Souri et al. (2022).
L579: “clear” --> “clear-sky”
Table 1 "Shape parameter k": Is this referring to the asymmetry parameter? Also, why is the term for the parameter "w" not included?
Figure 7: It would be helpful to zoom into an orbit to enhance the visibility of the destriping effectiveness.
Figure 13: Additionally, including a map showing the distribution of deviations would aid in identifying systematic error sources, such as coastlines and mountains.
Figure A9: the color where there are no points, possibly by using a mincnt option, to improve the clarity of the figure.
Figure A10: The stock image contains colors similar to the colorbar, particularly over the oceans. It is recommended to remove the stock image to avoid confusion.
References
Anand, J. S., Monks, P. S., & Leigh, R. J. (2015). An improved retrieval of tropospheric NO2 from space over polluted regions using an Earth radiance reference. Atmospheric Measurement Techniques, 8(3), 1519-1535.
Beirle, S., Lampel, J., Lerot, C., Sihler, H., & Wagner, T. (2017). Parameterizing the instrumental spectral response function and its changes by a super-Gaussian and its derivatives. Atmospheric Measurement Techniques, 10(2), 581-598.
Borger, C., Beirle, S., Dörner, S., Sihler, H., & Wagner, T. (2020). Total column water vapour retrieval from S-5P/TROPOMI in the visible blue spectral range. Atmospheric Measurement Techniques, 13(5), 2751-2783.
Borger, C., Beirle, S., & Wagner, T. (2023). A 16-year global climate data record of total column water vapour generated from OMI observations in the visible blue spectral range. Earth System Science Data, 15(7), 3023-3049.
Cahalan, R. F., Ridgway, W., Wiscombe, W. J., Bell, T. L., & Snider, J. B. (1994). The albedo of fractal stratocumulus clouds. Journal of the Atmospheric Sciences, 51(16), 2434-2455.
Chan, K. L., Valks, P., Slijkhuis, S., Köhler, C., & Loyola, D. (2020). Total column water vapor retrieval for Global Ozone Monitoring Experience-2 (GOME-2) visible blue observations. Atmospheric Measurement Techniques, 13(8), 4169-4193.
Chan, K. L., Xu, J., Slijkhuis, S., Valks, P., & Loyola, D. (2022). TROPOspheric Monitoring Instrument observations of total column water vapour: Algorithm and validation. Science of The Total Environment, 821, 153232.
De Smedt, I., Theys, N., Yu, H., Danckaert, T., Lerot, C., Compernolle, S., ... & Veefkind, P. (2018). Algorithm theoretical baseline for formaldehyde retrievals from S5P TROPOMI and from the QA4ECV project. Atmospheric Measurement Techniques, 11(4), 2395-2426.
Lamsal, L. N., Krotkov, N. A., Vasilkov, A., Marchenko, S., Qin, W., Yang, E. S., ... & Bucsela, E. (2021). Ozone Monitoring Instrument (OMI) Aura nitrogen dioxide standard product version 4.0 with improved surface and cloud treatments. Atmospheric Measurement Techniques, 14(1), 455-479.
Schenkeveld, V. M., Jaross, G., Marchenko, S., Haffner, D., Kleipool, Q. L., Rozemeijer, N. C., ... & Levelt, P. F. (2017). In-flight performance of the Ozone Monitoring Instrument. Atmospheric measurement techniques, 10(5), 1957-1986.
Souri, A. H., Chance, K., Sun, K., Liu, X., & Johnson, M. S. (2022). Dealing with spatial heterogeneity in pointwise-to-gridded-data comparisons. Atmospheric Measurement Techniques, 15(1), 41-59.
Citation: https://doi.org/10.5194/amt-2023-66-RC1 -
RC2: 'Comment on amt-2023-66', Anonymous Referee #2, 27 Sep 2023
This paper presents an algorithm capable of calculating the water vapor column by exploiting the absorption of the said molecule in the wavelength range of the UV spectral band measured by OMI. Intermediate details such as the fit for deriving the oblique column density and calculating the air mass factor are introduced.
The text is very clearly written and needs no corrections. The logic on which the article is based is robust, and the most important ingredients for reproducing the algorithm are adequately explained.
Two aspects make me lean toward suggesting major revisions.
- The article mixes two different versions of OMI radiances and is not exhaustive in validating the algorithm. While this aspect is not problematic for a developer, it becomes problematic for a data user.
I fully support the first referee's opinion regarding the inconsistency of presenting results for both collection 3 and 4 in one manuscript. I suppose that focusing only on collection 3 makes the presentation of the work unnecessary and not timely anymore, while focusing only on collection 4 makes the work in itself obsolete. Whichever way one looks at the topic, a harmonization effort is needed, in my opinion. - The part of the paper describing the bias correction seems like a foreign body to the text. In the sense that it is very interesting but it is not developed to such an extent that it becomes an integral part of the article. Since by now the algorithms in the community are converging toward a standard approach, bias correction has the potential to shed light on the errors introduced by clouds (for example). This aspect is not discussed at all, except by citing external work. But then I wonder why present section 4.1.1 without then elaborating on its potential.
I suggest major revisions because these are major conceptual points to work on, on which results make into the paper. The technical details are more than fine.
Specific commentsabstract l 29: the acronym AMSR_E is not broken out. Perhaps it is worth just to mention AMSR_E as independent spaceborn record, just not to excessively clog the abstract. Just a minor style correction. Up to the authors.
P3 L86 and ff: I find problematic mixing OMI collection 3 and 4. I wonder whether the ingestion of collection 3 is appropriate for several reasons. Will be the product based on collection 3 maintained and updated even after the processing of collection 4? If this is not the case, then this paper could already be obsolete in its premises.
If collection 4 will need different settings in the algorithm, then it does not seem consistent to present results based on both collections in this paper, because the reader can not keep trace of the relative error sources between the two versions. It might well be that collection 3 is a more challenging test bed, but then the results hold true only if the algorithm will not be updated. And we know that any collection update could require an adaption of the algorithm.
P7, Section 1.3: In general, I find the logic in the choice of spectral window for fit convincing. However, I agree with reviewer #1's observation about the inadequacy of having analysed only one orbit.
P12 L279-285: I have a problem with this paragraph and especially with the logic and scientific explanation of the results. While I can understand how the spectral fit does not appreciably change in the presence of clouds, given their spectral neutrality, I wonder why it should increase the SNR with CF > 0.8. Such a high CF value actually screens the light path and, consequently, limits the sensitivity of SCD retrieval. I would also appreciate seeing a stratification of RMS as a function of cloud height. Clearly the authors take advantage of an increase in the statistics due to the inclusion of cloudy pixels (L 281-282), but this is an almost forced choice due to the analysis of few orbits.
P14 L 362 and ff: on-line use of VLIDORT and the a priori water vapor profile from GEOS-chem.
I find the justification of the usage of the GEOS-chem climatological profile ("For practical reasons, ... " L344) not really scientific. We know well how labour intensive would be to adopt another approach, but at least a more quantitative justification is desirable. Also, I wonder why the online use VLIDORT, predicated to dynamically accomodate BRDFs, topography, aerosols and other factors (P15 L367) cannot then be applied at time-resolved actual profiles. At line 375 it is said that GOES-chem is the source for profiles, but it is not clear to me whether they are climatological or actual. Can the authors be more specific on this point?P17 L 441 and ff: I have two remarks on the treatment of uncertainty.
1) "Pixel-wise AMF uncertainty is under development for the MEaSUREs pipeline and will be included in a future public release."
Given the paragaphs above this sentence, I wonder why the authors cannot include AMF uncertainty in this paper. All the ingredients are there and a quadrature of the relative error sources is already a commendable diagnostic to include.
2) "While error propagation provides one way of evaluating the uncertainty, it usually does not include model error."
But models errors can be included in the propagation chain. Validation is not mutually exclusive to that. One cannot avoid the first, carrying out the second or vice versa. I do really not follow the authors here.P18 L484-485: "There are more than 2 million data points used for each panel."
2 million points out of how many in total? Basically, what is the relative data yield for the filtering you have enforced? Please, provide some statistics in a table or similar, so that the data user knows what to expect and how to use it.Figure 11, right column: the GPS data are here over ocean, over land or both? Please specify, because it is not readily clear, provided that Fig.11 is discussed as part of Section 4.1, titled "Comparison for the Ocean". Please, in the caption or in the main text, specify how many stations are used, so that one can judge the representativeness of the comparison.
P22 Section 4.1.1
I find this section quite interesting. Especially the application of a ML model to bias correct the original dataset. But it is surprising how little attention is given to the results. Figure A-9 deserves, in my opinion, a dedicated spot in the main text and not in the appendix. Moreover, please add bias maps to Figure A-9 (ML - AMSR and Measures - AMSR) so that we appreciate the correction.
I want to be more specific on this point. What I expect from the ML bias correction is patterns that are closely related to determined cloud fields described by recurrent and common properties. So that the authors can extrapolate the impact that the cloud forward model, described in Veefkind et al 2016 for generating the OMCLDO2 dataset, has on TCWV.
Rephrasing even more, I would appreciate the authors' response to the question of whether the cloud representation in OMCLDO2 is adequate, whether it can be improved (and in what way), and whether ML gives guidance in this regard. Clouds are actually NOT Lambertian objects and do not have a fixed albedo of 0.8 (which roughly corresponds to an optical thickness of 30). Obviously this is a simplification, in vogue for a long time and adopted by many research groups, due to the need to simplify water vapor calculations but perhaps it is time to look forward. Can your ML approach give guidance in this regard? Can clouds finally be considered real light scattering objects?Technical typos
Figure 3-b, title: corrlation -> correlationCitation: https://doi.org/10.5194/amt-2023-66-RC2 - The article mixes two different versions of OMI radiances and is not exhaustive in validating the algorithm. While this aspect is not problematic for a developer, it becomes problematic for a data user.
Interactive discussion
Status: closed
-
RC1: 'Comment on amt-2023-66', Anonymous Referee #1, 01 Aug 2023
General comments
In recent years, there has been a steady improvement in TCWV retrievals in the visible blue spectral region, aiming to exploit long-term satellite measurement series for climatological research. These advances have included improvements in spectral analysis as well as in the conversion of SCDs to VCDs, for example, by using flexible a priori profiles instead of monthly/climate average profiles for AMF calculation.
This study by Wang et al. describes an update to a TCWV retrieval algorithm in the visible blue spectral region for OMI measurements. The update includes a temperature correction of the SCD, new fit settings or a new fit window, destriping algorithms, and an optimized AMF calculation. Furthermore, the retrieval is compared with satellite and in situ reference data sets. Likewise, the retrieval output is further improved through the use of a machine learning algorithm.
Overall, the study shows many interesting and innovative improvements compared to the work of the same authors on predecessors of the algorithm. However, before the study can be published, I have various major issues that need to be addressed.
Major issues
- Temperature correction
The manuscript explains that for the temperature correction, the temperature profile is weighted by the box AMF profile or by the profile of scattering-weights. However, this approach lacks coherence and may leads to inaccurate temperature values, because the estimated temperature does not align with the actual distribution of the gas in the atmosphere. For instance, if a gas only occurs in the boundary layer, the temperature in the upper troposphere is irrelevant. Instead, what should be done is to apply a weight to the temperature profile using the (a priori) gas profile or the product of a priori profile and the box-AMF profile. Furthermore, it is also unclear to what extent this correction is actually feasible due to the cross-correlations between the cross-sections, which means the cross-section is influenced differently at various temperatures.
- Fit window optimization
The optimization of the fit window relies on spectral analyses from OMI collection Version 3. However, this approach renders the analysis outdated, considering that Version 4 is expected to include significant updates, particularly in the irradiance (resulting in fewer flagged spectral pixels). Consequently, the entire section, along with the conclusions drawn from it, needs to be revisited using Version 4. Additionally, the analysis is currently based solely on one orbit, which is inadequate.
- Row anomaly
Since mid-2007, the OMI measurements have been affected by the so-called “row anomaly” (Schenkeveld et al, 2017). However, the paper does not make any mention of this issue, and there is no discussion on how it impacts the retrieval, both quantitatively and qualitatively If the aim is to utilize this retrieval for the long-term measurement series of OMI, more attention must be given to addressing this matter thoroughly.
- Destriping
The issues related to the fit window optimization and the row anomaly have direct implications for the destriping algorithm. As a result, the destriping process must be revised to accommodate these factors appropriately.
- AMF calculation
Borger et al. (2020) and Chan et al. (2022) have clearly demonstrated that employing a static water vapor profile as an a priori for the AMF calculation is notably less effective compared to using a flexible a priori in retrievals. Given this evidence, it raises a question as to why this study continues to employ the static water vapor profile in its AMF calculation.
- Validation study
In the validation study, only early periods are examined which remain unaffected by both the row anomaly and the significant degradation of OMI. As a result, it is imperative to include more recent periods, considering these two factors in the analysis. Moreover, for the comparison over land, the absence of a global dataset limits the analysis to specific regions only. To address this, additional datasets (such as reanalysis or reference data) should be incorporated for a more comprehensive assessment.
Furthermore, the authors missed an opportunity by not utilizing the ML-optimized TCWV dataset in the validation study. Instead, the "standard" retrieval employs very strict filter criteria. This raises questions about the usefulness of the retrieval, as it leads to the exclusion of significant portions of the OMI orbits. This is particularly concerning for long-term analyses and clear-sky biases.
- OMI Level 1 collection 3 vs collection 4
Here and there it becomes challenging to discern for which L1 version the assessments were conducted. It would greatly benefit the reader if only one version (preferably Collection 4) is consistently used throughout the study. Additionally, it is recommended to include a paragraph or section in the manuscript that outlines the differences between Collection 3 and Collection 4 to provide clarity on the chosen version and its implications.
Minor/technical issues
Section 2: I think it would be helpful to write down the equation to make the differences to DOAS clear. How are the terms accounting for changes in the ISRF from Beirle et al. (2017) implemented in this scheme?
L93: It should be mentioned that the H2O cross-section is still subject to large uncertainties in this spectral range and many researcher utilize different versions of HITRAN (e.g. Borger et al., 2020; Chan et al., 2022).
L95: In this context, it should be clarified whether HW1e refers to the full width at half maximum (FWHM) of the spectral line.
L100: Which updates have been implemented that affect the visible blue spectral range? Please clarify.
L138: “(TCWV < 5mm)”: For spectral analysis, the SCD is crucial, not the VCD.
L144f: In addition to the mentioned references, it would be relevant to include Borger et al. (2020) and Borger et al. (2023) for TROPOMI and OMI, respectively. Regarding Garane et al. (2023), since it is a validation study, it could be replaced with Chan et al. (2022).
L150ff: Wouldn’t it be simpler to use SCD/σ(SCD) instead of two separate variables? How do you determine the “fraction of valid retrievals” and what are the criteria for valid retrievals? I don’t really understand the “common mode amplitude”. Can you provide a formula?
L177ff: The discussion should also include the fitting uncertainty or SCD uncertainty as it is relevant to the analysis.
L218ff: Why is an Earthshine spectrum not used for destriping the OMI orbit? For instance, it is frequently employed in NO2 or HCHO retrievals (Anand et al., 2015; De Smedt et al., 2018). Moreover, Borger et al. (2023) successfully utilized it for their OMI TCWV retrieval.
L229ff: The stripes also change since OMI has a new irradiance every 15 orbits.
L245ff: The calculation of the correction vector for each row typically involves the use of how many pixels? Doesn't the filtering eventually lead to a bias towards cloudy pixels or brighter scenes due to their lower fit RMS? How are rows affected by the row anomaly handled in the analysis?
L263: What is the purpose of the "reflection extension"?
L271f: Instead of using "optically thin assumption," it might be more appropriate to use "weak absorber assumption”.
L275: Borger et al. (2020) should be included as a reference.
L284: Please add a reference for IPA (e.g., Cahalan et al., 1994).
L308ff: It would be beneficial to consider using the cloud and albedo information from the L2 NO2 product (Lamsal et al., 2021) in the analysis. This product takes into account BRDF effects and offers a consistent approach to handling these parameters.
L316: How do you address the different spatial resolutions between the OMI pixel and the MODIS product in the analysis?
L324f: Are the MERRA-2 winds monthly means?
L330: Instead of using "cloud scattering," it may be more accurate to refer to "cloud shielding" in this context.
L346f: NO2 is primarily concentrated in the boundary layer and, as a result, exhibits a profile shape completely different from the water vapor profile. A comprehensive error analysis of AMFs in TCWV retrievals can be found in Borger et al. (2020).
L349: What is the accuracy of the OMCLDO2 product?
L376: What is the reason for imposing an upper limit for the AMF, and how was this limit determined?
L376ff: With respect to the complete OMI orbit, what proportion of OMI pixels is utilized in the TCWV product?
L390ff: It should be noted that Chan et al. (2020) and Garane et al. (2023) employ different cloud filters compared to this study. Additionally, Borger et al. (2020) should also be mentioned here. Based on the outcomes, one might question whether the retrieval from Wang et al. (2019) performs better than the update presented in this study.
L460ff: This suggests that the presented retrieval might not be as robust as desired. It would be highly insightful to investigate whether using a flexible a priori profile, as demonstrated in Borger et al. (2020) and Chan et al. (2020, 2022), could potentially address this issue.
L444: The regions with high deviations seem to coincide with regions experiencing frequent cloud cover, such as the SPCZ. Alternatively, these deviations could also be indicative of inaccuracies in the a priori profile.
L466: Do you use monthly mean or daily profiles of temperature and water vapor for the GBM? To ensure the GBM model's generalization, it might be preferable to omit latitude and longitude information.
Section 4.2.1: It would be valuable to include a comparison of the semivariograms of the two datasets, as suggested by Souri et al. (2022).
L579: “clear” --> “clear-sky”
Table 1 "Shape parameter k": Is this referring to the asymmetry parameter? Also, why is the term for the parameter "w" not included?
Figure 7: It would be helpful to zoom into an orbit to enhance the visibility of the destriping effectiveness.
Figure 13: Additionally, including a map showing the distribution of deviations would aid in identifying systematic error sources, such as coastlines and mountains.
Figure A9: the color where there are no points, possibly by using a mincnt option, to improve the clarity of the figure.
Figure A10: The stock image contains colors similar to the colorbar, particularly over the oceans. It is recommended to remove the stock image to avoid confusion.
References
Anand, J. S., Monks, P. S., & Leigh, R. J. (2015). An improved retrieval of tropospheric NO2 from space over polluted regions using an Earth radiance reference. Atmospheric Measurement Techniques, 8(3), 1519-1535.
Beirle, S., Lampel, J., Lerot, C., Sihler, H., & Wagner, T. (2017). Parameterizing the instrumental spectral response function and its changes by a super-Gaussian and its derivatives. Atmospheric Measurement Techniques, 10(2), 581-598.
Borger, C., Beirle, S., Dörner, S., Sihler, H., & Wagner, T. (2020). Total column water vapour retrieval from S-5P/TROPOMI in the visible blue spectral range. Atmospheric Measurement Techniques, 13(5), 2751-2783.
Borger, C., Beirle, S., & Wagner, T. (2023). A 16-year global climate data record of total column water vapour generated from OMI observations in the visible blue spectral range. Earth System Science Data, 15(7), 3023-3049.
Cahalan, R. F., Ridgway, W., Wiscombe, W. J., Bell, T. L., & Snider, J. B. (1994). The albedo of fractal stratocumulus clouds. Journal of the Atmospheric Sciences, 51(16), 2434-2455.
Chan, K. L., Valks, P., Slijkhuis, S., Köhler, C., & Loyola, D. (2020). Total column water vapor retrieval for Global Ozone Monitoring Experience-2 (GOME-2) visible blue observations. Atmospheric Measurement Techniques, 13(8), 4169-4193.
Chan, K. L., Xu, J., Slijkhuis, S., Valks, P., & Loyola, D. (2022). TROPOspheric Monitoring Instrument observations of total column water vapour: Algorithm and validation. Science of The Total Environment, 821, 153232.
De Smedt, I., Theys, N., Yu, H., Danckaert, T., Lerot, C., Compernolle, S., ... & Veefkind, P. (2018). Algorithm theoretical baseline for formaldehyde retrievals from S5P TROPOMI and from the QA4ECV project. Atmospheric Measurement Techniques, 11(4), 2395-2426.
Lamsal, L. N., Krotkov, N. A., Vasilkov, A., Marchenko, S., Qin, W., Yang, E. S., ... & Bucsela, E. (2021). Ozone Monitoring Instrument (OMI) Aura nitrogen dioxide standard product version 4.0 with improved surface and cloud treatments. Atmospheric Measurement Techniques, 14(1), 455-479.
Schenkeveld, V. M., Jaross, G., Marchenko, S., Haffner, D., Kleipool, Q. L., Rozemeijer, N. C., ... & Levelt, P. F. (2017). In-flight performance of the Ozone Monitoring Instrument. Atmospheric measurement techniques, 10(5), 1957-1986.
Souri, A. H., Chance, K., Sun, K., Liu, X., & Johnson, M. S. (2022). Dealing with spatial heterogeneity in pointwise-to-gridded-data comparisons. Atmospheric Measurement Techniques, 15(1), 41-59.
Citation: https://doi.org/10.5194/amt-2023-66-RC1 -
RC2: 'Comment on amt-2023-66', Anonymous Referee #2, 27 Sep 2023
This paper presents an algorithm capable of calculating the water vapor column by exploiting the absorption of the said molecule in the wavelength range of the UV spectral band measured by OMI. Intermediate details such as the fit for deriving the oblique column density and calculating the air mass factor are introduced.
The text is very clearly written and needs no corrections. The logic on which the article is based is robust, and the most important ingredients for reproducing the algorithm are adequately explained.
Two aspects make me lean toward suggesting major revisions.
- The article mixes two different versions of OMI radiances and is not exhaustive in validating the algorithm. While this aspect is not problematic for a developer, it becomes problematic for a data user.
I fully support the first referee's opinion regarding the inconsistency of presenting results for both collection 3 and 4 in one manuscript. I suppose that focusing only on collection 3 makes the presentation of the work unnecessary and not timely anymore, while focusing only on collection 4 makes the work in itself obsolete. Whichever way one looks at the topic, a harmonization effort is needed, in my opinion. - The part of the paper describing the bias correction seems like a foreign body to the text. In the sense that it is very interesting but it is not developed to such an extent that it becomes an integral part of the article. Since by now the algorithms in the community are converging toward a standard approach, bias correction has the potential to shed light on the errors introduced by clouds (for example). This aspect is not discussed at all, except by citing external work. But then I wonder why present section 4.1.1 without then elaborating on its potential.
I suggest major revisions because these are major conceptual points to work on, on which results make into the paper. The technical details are more than fine.
Specific commentsabstract l 29: the acronym AMSR_E is not broken out. Perhaps it is worth just to mention AMSR_E as independent spaceborn record, just not to excessively clog the abstract. Just a minor style correction. Up to the authors.
P3 L86 and ff: I find problematic mixing OMI collection 3 and 4. I wonder whether the ingestion of collection 3 is appropriate for several reasons. Will be the product based on collection 3 maintained and updated even after the processing of collection 4? If this is not the case, then this paper could already be obsolete in its premises.
If collection 4 will need different settings in the algorithm, then it does not seem consistent to present results based on both collections in this paper, because the reader can not keep trace of the relative error sources between the two versions. It might well be that collection 3 is a more challenging test bed, but then the results hold true only if the algorithm will not be updated. And we know that any collection update could require an adaption of the algorithm.
P7, Section 1.3: In general, I find the logic in the choice of spectral window for fit convincing. However, I agree with reviewer #1's observation about the inadequacy of having analysed only one orbit.
P12 L279-285: I have a problem with this paragraph and especially with the logic and scientific explanation of the results. While I can understand how the spectral fit does not appreciably change in the presence of clouds, given their spectral neutrality, I wonder why it should increase the SNR with CF > 0.8. Such a high CF value actually screens the light path and, consequently, limits the sensitivity of SCD retrieval. I would also appreciate seeing a stratification of RMS as a function of cloud height. Clearly the authors take advantage of an increase in the statistics due to the inclusion of cloudy pixels (L 281-282), but this is an almost forced choice due to the analysis of few orbits.
P14 L 362 and ff: on-line use of VLIDORT and the a priori water vapor profile from GEOS-chem.
I find the justification of the usage of the GEOS-chem climatological profile ("For practical reasons, ... " L344) not really scientific. We know well how labour intensive would be to adopt another approach, but at least a more quantitative justification is desirable. Also, I wonder why the online use VLIDORT, predicated to dynamically accomodate BRDFs, topography, aerosols and other factors (P15 L367) cannot then be applied at time-resolved actual profiles. At line 375 it is said that GOES-chem is the source for profiles, but it is not clear to me whether they are climatological or actual. Can the authors be more specific on this point?P17 L 441 and ff: I have two remarks on the treatment of uncertainty.
1) "Pixel-wise AMF uncertainty is under development for the MEaSUREs pipeline and will be included in a future public release."
Given the paragaphs above this sentence, I wonder why the authors cannot include AMF uncertainty in this paper. All the ingredients are there and a quadrature of the relative error sources is already a commendable diagnostic to include.
2) "While error propagation provides one way of evaluating the uncertainty, it usually does not include model error."
But models errors can be included in the propagation chain. Validation is not mutually exclusive to that. One cannot avoid the first, carrying out the second or vice versa. I do really not follow the authors here.P18 L484-485: "There are more than 2 million data points used for each panel."
2 million points out of how many in total? Basically, what is the relative data yield for the filtering you have enforced? Please, provide some statistics in a table or similar, so that the data user knows what to expect and how to use it.Figure 11, right column: the GPS data are here over ocean, over land or both? Please specify, because it is not readily clear, provided that Fig.11 is discussed as part of Section 4.1, titled "Comparison for the Ocean". Please, in the caption or in the main text, specify how many stations are used, so that one can judge the representativeness of the comparison.
P22 Section 4.1.1
I find this section quite interesting. Especially the application of a ML model to bias correct the original dataset. But it is surprising how little attention is given to the results. Figure A-9 deserves, in my opinion, a dedicated spot in the main text and not in the appendix. Moreover, please add bias maps to Figure A-9 (ML - AMSR and Measures - AMSR) so that we appreciate the correction.
I want to be more specific on this point. What I expect from the ML bias correction is patterns that are closely related to determined cloud fields described by recurrent and common properties. So that the authors can extrapolate the impact that the cloud forward model, described in Veefkind et al 2016 for generating the OMCLDO2 dataset, has on TCWV.
Rephrasing even more, I would appreciate the authors' response to the question of whether the cloud representation in OMCLDO2 is adequate, whether it can be improved (and in what way), and whether ML gives guidance in this regard. Clouds are actually NOT Lambertian objects and do not have a fixed albedo of 0.8 (which roughly corresponds to an optical thickness of 30). Obviously this is a simplification, in vogue for a long time and adopted by many research groups, due to the need to simplify water vapor calculations but perhaps it is time to look forward. Can your ML approach give guidance in this regard? Can clouds finally be considered real light scattering objects?Technical typos
Figure 3-b, title: corrlation -> correlationCitation: https://doi.org/10.5194/amt-2023-66-RC2 - The article mixes two different versions of OMI radiances and is not exhaustive in validating the algorithm. While this aspect is not problematic for a developer, it becomes problematic for a data user.
Data sets
MEaSUREs blue band total column water vapor sample data for the Ozone Monitoring Instrument H. Wang, G. González Abad, C. Chan Miller, H. Kwon, C. R. Nowlan, Z. Ayazpour, H. Chong, X. Liu, K. Chance, E. O'Sullivan, K. Sun, R. Spurr, and R. J. Hargreaves https://doi.org/10.5281/zenodo.7795648
Viewed
| HTML | XML | Total | BibTeX | EndNote | |
|---|---|---|---|---|---|
| 528 | 144 | 55 | 727 | 54 | 53 |
- HTML: 528
- PDF: 144
- XML: 55
- Total: 727
- BibTeX: 54
- EndNote: 53
Viewed (geographical distribution)
| Country | # | Views | % |
|---|
| Total: | 0 |
| HTML: | 0 |
| PDF: | 0 |
| XML: | 0 |
- 1