We use several data sets from the OMI and Moderate resolution Imaging Spectroradiometer (MODIS) instruments flying on the NASA Aqua and Terra satellites. OMI is a spectrometer that acquires Earth and solar spectra at UV–vis wavelengths from 270 to 500 nm with a spectral resolution of approximately 0.5 nm. The OMI ground footprint varies; near the nadir, it is approximately 12 km along the satellite track and 24 km across the 2600 km track. The footprint size increases towards the swath edge. We use TOA radiance and solar irradiance in the OMI vis channel to retrieve cloud parameters and NO2 amounts.

The MODIS-derived BRDF kernel coefficients from the 16-day MCD43GF data set are used to compute GLERs over land for the OMI swath . The kernel coefficients are provided for snow-free land and permanent ice at a high spatial resolution. Over transient snow-covered regions, we retain the standard climatological LER of that was routinely used for the previous cloud retrievals.

The operational OMI O2-O2 SCD retrieval uses the differential optical absorption spectroscopy (DOAS) approach, simultaneously retrieving SCDs of O2-O2 and O3 in the 460–490 nm wavelength interval and using single-temperature O2-O2 cross sections and a first-degree polynomial approximating the wavelength dependence of reflectances in the fitting window.

Here, we generally follow the approach developed by for the NO2 SCD estimates. Instead of simultaneous retrieval of coefficients of multiple parameters as takes place in the classical DOAS formalism, we divide the problem into a series of sequential steps (Fig. ).

Step 1 involves the removal of interfering trace-gas absorption. The spectral range chosen for the O2-O2 SCD retrievals is affected by relatively strong O3 absorption that, in most cases, distorts the O2-O2 profiles (Fig. ). The same applies to NO2 absorption over polluted regions (e.g., the Beijing area; see Fig. ) and, to a far lesser extent, the mainly equatorial regions over the open-water Pacific, where the H2O absorption may distort the flanks of the broad O2-O2 profiles (Fig. ). Note the clear presence of the ozone feature around λ∼462 nm, as well as the large distortion of the O2-O2 profile caused by the broad ozone absorption around λ∼482 nm. In this particular example, the only easily recognizable Ring spectrum feature is seen at λ∼486.5 nm. The gradual ∼13 % change in reflectances between 450 and 500 nm comes from a combination of comparable strength signals: the ozone absorption and the Rayleigh-scattering component. For better guidance, in Figs. – we show scaled absorption spectra of the main trace gases that may contribute to the general appearance of a reflectance spectrum. We keep the same plotting style for Figs. –, though noting that in each particular example one may see quite a different impact from the same absorption constituent. For example, while the NO2-related signal barely registers in Fig.  (practically unpolluted region), the heavily contaminated Beijing area (Fig. ) shows a clear presence of the NO2 absorption at λ∼457–466 nm (three features), λ∼475 and 480 nm (these two are superposed on the broad O2-O2 absorption), and the well-defined NO2 absorption at λ∼489 nm that rivals the strength of the retrieved O2-O2 feature. In this particular case of the heavily NO2-polluted region, the omnipresent O3 absorption plays a far less important role compared to the spectrum shown in Fig. . In addition, though to a far lesser extent (when compared to ozone, Ring, and NO2), the H2O absorption may distort the flanks of the broad O2-O2 profiles, mainly in the equatorial regions over the open-water Pacific.

The spectral domain chosen for the O2-O2 retrieval is not optimal for simultaneous O3, NO2, or H2O estimates. Optimal fitting windows are the ∼290–340, 400–465, and 435–450 nm intervals for O3, NO2, and H2O, respectively. Hence, to minimize the biases that may be introduced by the sub-optimal DOAS estimates of the interfering trace-gas species, we use the SCDs provided by independent OMI products: NO2 and H2O from OMNO2SCD and O3 from OMDOAO3 , and we remove the corresponding absorption features from the observed radiances. We find that, as expected, at large SZAs the corrections based on the UV O3 SCD retrievals result in large spectral residuals pointing to systematic underestimates of O3 absorption strength. This stems from the notion that the relatively (to the visual range) higher Rayleigh optical depth effectively masks the lower-atmosphere O3 absorption. Hence, at SZA>80∘ we adjust the UV SCDs by a constant 1.25 coefficient. This helps to reduce the spectral residuals related to the underestimated O3 absorption to a manageable (on average <0.1 %) level.

Step 2 closely follows the approach from , comprising the simultaneous, iterative wavelength adjustment and Ring spectrum removal. At each FOV (row, 60 in total) the reflectances are produced from the individual, prefiltered earthshine radiances (from Step 1) normalized by the monthly-averaged OMI irradiances. These irradiances are iteratively adjusted to accommodate slight relative (radiances vs. irradiances) wavelength shifts. For estimates of the line-filling factors (i.e., the Raman scattering amplitudes), we use an appropriate combination of the air and water Raman scattering spectra . We split the retrieval region into two “micro-windows”, 451–469 and 483–496 nm, and iteratively evaluate the wavelength shifts and the Raman-spectrum amplitudes in each window. For the final removal of the Raman scattering patterns, we use an average of the two micro-window estimates. The individual micro-window wavelength shifts are used for wavelength adjustments of irradiances in each micro-window, interpolating these estimates in the 469–483 nm domain occupied by the main O2-O2 absorption.

Step 3 involves normalizing the O2-O2 profile in preparation for SCD evaluations. We deem this step to be the most important procedure; it may change the outcome by as much as ∼20 % in extreme cases such as the open-water scenes (Fig. ) and Sahara desert (Fig. ) representing two extremes and the remaining cases falling in between. Figures a and a show the observed reflectances before (black lines) and after (blue lines) the removal of trace-gas absorption and the Raman line-filling patterns, and the red lines follow the adopted continuum fits. Note the profound difference among the wavelength dependencies of the reflectances in these extreme cases. While the cloud-free, open-water case (Fig. ) is predominantly Rayleigh controlled, leading to a steep decline in reflectances, the much brighter Sahara surface controls the appearance of the radiances at long OMI wavelengths, leading to the gradual increase in reflectances (Fig. ). Figures b and b show normalized O2-O2 profiles. In a case-by-case study of the presumably cloud-free areas, we have found that various combinations of linear functions fitting the flanks of the O2-O2 profile lead to gross underestimates (mainly over open-water areas) or overestimates (deserts and semideserts) of the retrieved scene pressures that are directly linked to the biases in the SCD evaluations. Hence, we have implemented a more flexible approach, defining two broad categories of the surface reflectances and applying different fitting approaches to each of them.

The reflectances from Step 2 are averaged in 2 nm intervals, providing a set of estimates at λ=463 and 495 nm that are partitioned into two general categories. The first broad category comprises all the relatively cloud-free low-reflectance scenes, with r(463)r(495)>1.05 and r(463)<0.25. The second class includes the remaining scenes. For both categories, the fitting starts from applying the third-degree polynomial to the 459–466 and 484–494 nm regions, identifying and eliminating large (∼5σ, i.e., ±0.5 %) deviations and then repeating the procedure, ultimately normalizing the reflectances in the 450–500 nm range by the fit. The normalized reflectances in the 459–465 and 484–490 nm intervals are refitted with a first-degree polynomial and, again, all reflectances in the 450–500 nm range are renormalized by this fit. This concludes the fitting for the second category of scenes. However, at this point the fitting proceeds for the first class of the relatively cloud-free, low-reflectance scenes. Yet again, the normalized radiances in the 465–470 and 482–487 nm intervals are iteratively (rejecting the large ±1 % deviations) refitted with a second-degree polynomial; then this fit is applied exclusively to the region occupied by the O2-O2 profile, 465–491 nm. The line edges are further refined by applying piecewise fits (first- or second-degree polynomials) to the relatively narrow windows, 459–465 and 486–491 nm, thus concluding this rather involved procedure for the first category of the low-reflectance scenes.

Step 4, the SCD retrieval, follows the approach described in . Here, preliminary SCD values are obtained from two algorithms, the Nelder–Mead minimization method and the least-squares Levenberg–Marquardt fit , taking the latter as a default and fitting the normalized (Step 3) O2-O2 profile in the 465–487 nm interval. These evaluations are repeated for each temperature-dependent O2-O2 cross section; there are five of them measured by . Each cross section is fitted to the data, providing an individual root mean square (RMS) value of the fitting residuals. These five RMS values are approximated by a parabolic function. The minimum of the function is used to construct via linear interpolation a synthetic O2-O2 profile that is removed from the normalized reflectances, thus leaving us with the residuals that are presumably dominated by instrumental noise. The noise is reduced by an iterative procedure similar to one described in . The final SCD evaluation is performed over a slightly broadened wavelength range, 463–488 nm.

As implemented, the algorithm relies on optimal SCD retrievals of the O3, NO2, and H2O trace gases, as well as preliminary cloud-fraction estimates. The latter is used exclusively over deep-water areas during the wavelength calibration and the Raman scattering removal. If needed, such cloud fractions can be substituted for appropriately adjusted reflectances, thus vying for self-sufficiency. The use of independent O3, NO2, and H2O SCDs is an essential part of the algorithm that, especially for the scenes with heavy O3 and NO2 loads, leads to more accurate O2-O2 SCDs. The use of the trace-gas SCDs does not create any paradox when the NO2 values are used in order to retrieve cloud properties that should be incorporated into the NO2 estimates. Note that in the implemented algorithm we use the NO2 SCD estimates that can be obtained without any relevance to cloud properties. These cloud properties are used later, during the conversion of the NO2 slant columns to the NO2 vertical columns. Opting for a complete self-reliance of the cloud algorithm, one may substitute the required O3, NO2, and H2O SCDs for SCD estimates provided by the appropriate trace-gas climatologies.

The O2-O2 cloud algorithm described here is based on the O2-O2 absorption band at 477 nm. This algorithm is broadly similar to the operational O2-O2 cloud algorithm developed at the Royal Netherlands Meteorological Institute (KNMI) known as OMCLDO2 . However, our approach differs in a number of aspects.

First, we use normalized radiance at 466 nm to compute f with Eq. () in a separate step. This wavelength was selected because it is not significantly affected by gaseous absorption and rotational Raman scattering and it is still sufficiently close to the O2-O2 absorption band center at 477 nm. f is calculated using linear interpolation of lookup tables (LUTs) of Ig and Ic. The tables were generated for 23 different surface and cloud pressures, 20 surface reflectivities, 30 SZAs, 20 VZAs, and 20 relative azimuth angles. Nodes and their locations were selected on the basis of the analysis of interpolation errors. A threshold for acceptable interpolation error was set at 0.2 %. It should be noted that aerosols are implicitly accounted for in the determination of f, as they are treated (like clouds) as particulate scatters.

Our algorithm retrieves cloud OCP from the OMI-derived oxygen dimer SCD at 477 nm. The OCP, here also denoted as Pc, is estimated using the MLER method to compute the appropriate air mass factors (AMFs) . To solve for OCP, we invert the following equation: SCD=AMFg(Ps,Rg)VCD(Ps)(1-fr)+AMFc(Pc,Rc)VCD(Pc)fr, where VCD is the vertical column density of O2-O2 (VCD=SCD/AMF), AMFg and AMFc are the precomputed (at 477 nm) clear-sky (subscript g) and overcast (cloudy, subscript c) subpixel AMFs, Ps is the surface pressure, and fr is the cloud radiance fraction (CRF) given by fr=fIc/Im. Equation () is similar to that frequently used for retrieval of trace-gas VCDs with the MLER model provided Pc and Rc are known (see, e.g., ). Here we use this equation for retrieval of Pc assuming that the O2-O2 VCD is known. The CRF is calculated at 466 nm. CRF defines a fraction of TOA radiance reflected by the cloud. It should be noted that CRF is wavelength dependent (see discussion in Sect. ). The CRF retrievals at different wavelengths are included in our output. Lookup tables of the TOA radiances and AMFs were generated using VLIDORT. Temperature profiles needed for estimation of VCD and AMF are taken from the NASA GMAO GEOS-5 global data assimilation system .

In addition to OCP, we retrieve the so-called scene pressure, Psc. The scene pressure is derived from Eq. () assuming that fr=1 and Rc is equal to the scene LER, Rsc: SCD=AMFc(Psc,Rsc)VCD(Psc). Rsc is determined from the measured TOA radiance using Eq. () for a known surface pressure. In the absence of clouds and aerosols, the Psc should be equal to Ps. Psc is therefore an important diagnostic tool for evaluation of the performance of cloud pressure algorithms.

To evaluate our cloud algorithm we have compared the retrieved values of f and Pc with those from the operational OMCLDO2 version 2 . For this comparison, the cloud products are retrieved for the climatological surface LER identical to that used in OMCLDO2 v2. Figure  shows scatter plots of effective cloud fractions (ECFs) calculated with our algorithm versus those calculated with OMCLDO2 for land and ocean for a selected date, 13 November 2006. Different dates show very similar trends. The scatter of data around the 1:1 line is somewhat higher for low values of fs. The mean differences between the two data sets do not exceed 0.01 for all values of f. The standard deviation of the f differences is within 0.01 for ocean and 0.03 for land. Differences in values of f are probably due to contrasting approaches used in the two algorithms. We retrieve the ECF at 466 nm independently from the OCP retrieval, whereas OMCLDO2 retrieves the ECF and OCP simultaneously at 477 nm.

Figure  shows scatter plots of OCPs calculated with our algorithm versus those calculated with OMCLDO2 v2 for the same day. There is a bias between OMCLDO2 and our algorithm: our OCP retrievals are higher than those from OMCLDO2 by about 50 hPa on average. The standard deviation of the OCP differences ranges from about 100 hPa for OCP<400 to 150 hPa for lower OCPs. Higher OCP retrievals from our algorithm compared to OMCLDO2 can be related to slightly higher O2–O2 SCD estimates and also to differences in ECF, which affect the OCP retrievals.

Figure  shows a comparison of Ps and Psc from our algorithm and OMCLDO2 v2 along cross-track position 20 of the OMI orbit 4415. A similar comparison of the OMCLDO2 v2 Psc and with Ps for this cross-track position was carried out in . We added our scene pressure and cloud fraction for this cross-track position to this comparison. First, Psc retrievals from both algorithms agree very well with the surface pressures for the high-reflectivity scenes in Greenland (OMI scan lines along the orbit with numbers iTimes=1300–1400). For mostly cloud-free conditions over the ocean (iTimes∼400 and iTimes ∼270), the scene pressures retrieved from both algorithms are higher than the surface pressures. These differences for those oceanic regions are slightly lower for our algorithm than for OMCLDO2. Over desert and semidesert areas, Psc retrieved with our algorithm over mostly cloud-free conditions (iTimes=720–900 and iTimes∼570) is close to the surface pressure while OMCLDO2 significantly overestimates the scene pressure. We attribute the better performance of our algorithm over deserts and semideserts to the special adjustment of the spectral fitting procedure for those areas (see Sect.  and Fig. ). Figure  shows maps of differences between the scene pressure and the surface pressure retrieved from our algorithm and OMCLDO2 v2. The differences are shown for mostly clear scenes with f<0.25. A comparison of the maps shows that the performance of our algorithm over land is better than the performance of OMCLDO2 v2. This is clearly seen for the Sahara, the Arabian Peninsula, Australia, etc. Over the ocean, our algorithm performs slightly better only for certain areas in the Southern Ocean, e.g., for three locations at ∼55∘ S, ∼60∘ W, ∼20∘ W, and ∼100∘ E. Over other areas of the ocean, the performance of the two algorithms is similar.

Figure  shows scatter plots of f calculated with GLER versus those calculated with climatological LER (ClimLER) in the form of 2-D histograms. The color scale on the 2-D histograms represents a number of data points. As expected, the scatter and systematic deviation of data around the 1:1 line diminish with increasing f. Over land, f calculated with GLER is mostly higher than f calculated with the climatological LER (Fig. a) particularly for low f. This is explained by differences between GLERs and climatological LERs , which have the most pronounced impact on f for low cloudiness. The GLER values are mostly lower than the climatological LERs because the former are derived from atmospherically corrected MODIS radiances while the latter are affected by residual aerosols. Moreover, climatological LERs can be contaminated by clouds owing to substantially larger sizes of OMI pixels compared with those of MODIS. As it follows from Eq. (), lower surface LER leads to lower clear-sky radiance, thus increasing f. For the most important NO2 retrieval range of f<0.25, f retrieved with GLER is higher than that retrieved with climatological LER by approximately 0.02 on average. The standard deviation of the GLER–ClimLER f differences varies between 0.03 and 0.05 depending on the f value.

Over ocean, the GLERs are higher than the climatological LERs in the areas affected by sun glint and areas observed at large viewing zenith angles . Therefore, f retrieved with GLER in these areas is lower than that retrieved with the climatological LER (Fig. b). For other oceanic regions, the GLERs are slightly lower than the climatological LERs, resulting in slightly higher values of f. For the range of f<0.25, which is frequently used in tropospheric trace-gas retrievals, the mean difference between f retrieved with GLER and that retrieved with climatological LER is approximately 0.02 when averaged globally. The standard deviation of the f differences varies within 0.02–0.03 depending on the f value. Even though the f differences are small on average, they can be as large as 0.05–0.07 for individual pixels – this is quite significant for the low f range. It should be noted that the fraction of negative f retrievals is lower when using GLER compared with the climatological LER. This is clear evidence of the improvement of f retrievals with GLER.

Figure  is a geographic map of differences between f calculated with GLER and climatological LER for 13 November 2006. Over the ocean, the most prominent features are sun glint areas where the negative f differences (GLER–ClimLER) are at maximum. This is because the climatological LERs are derived from minimum values of LERs from a long time series (up to 5 years) of observations over a given area; that is why the impact on observations affected by sun glint is somewhat mitigated. Over land, the f differences are mostly positive due to aerosol and possible cloud contamination of the climatological LERs (note that the relatively large OMI footprints lead to predominance of cloudy scenes).

One indicator of the cloud algorithm performance is the cross-track dependence of the retrieved f and OCP averaged over a given latitude bin. Ideally, this dependence should be fairly flat if the retrieved parameters are averaged globally. Figure  compares the cross-track dependencies of f retrieved with GLER and ClimLER for 13 November 2006. The comparison is carried out for 30∘ latitude bins for land and ocean separately and specifically for the low f range 0.05<f<0.25. Over land, the cross-track dependence of f is reasonably flat. f retrieved with GLER is mostly lower than that retrieved with the climatological LERs because GLER is generally lower than the climatological LER. Over the ocean, the most noticeable feature of the cross-track dependence is related to the sun glint area in the 30–0∘ S latitude bin. Even though the use of GLER smooths out the irregularity in the cross-track dependence of f, it slightly overcorrects this irregularity owing to overestimation of GLER in sun glint areas. We have not yet determined the exact cause of this GLER overestimation. However, our preliminary radiative transfer simulations show that the presence of nonabsorbent aerosol with low optical thickness (τ<0.2) can reduce the sun glint GLER. It should be noted that the cross-track dependence of f for cloudy scenes with f>0.25 is much flatter than that for scenes with f<0.25.

Figure  shows 2-D histogram plots of OCPs calculated with GLER and climatological LER. Over land, the GLER-retrieved OCPs are slightly higher than those retrieved with climatological LER (Fig. a). Over ocean, the data are closer to the 1:1 line (Fig. b) but the overall scatter is somewhat higher than that over land. Data for which the GLER-retrieved OCPs are lower than those retrieved with climatological LER are mostly from sun glint areas. There is a cluster of data for which both GLER-retrieved OCPs and those retrieved with climatological LER are higher than the surface pressure (1013 hPa). These data are retrieved over virtually clear-sky conditions, and there are two possible reasons for this effect. First, OCP retrievals higher than the surface pressure are evidence of enhanced O2-O2 absorption. This can be caused by scattering, low-altitude aerosols in which enhanced photon path length prevails over other aerosol effects. A second possible reason is due to a remaining deficiency in our spectral fitting procedure, which somehow overestimates SCDs over the clear-sky ocean. A very small fraction of the OCP retrievals with values around 100 hPa are likely due to an artifact caused by the LUT extrapolation over the minimum pressure node of 100 hPa.

Geographic distribution of the differences between OCPs calculated with GLER and climatological LER is shown in Fig. . Over ocean, most areas with negative differences (OCP(GLER) – OCP(ClimLER)) correspond to sun glint.

Figure  shows cross-track dependencies of OCPs retrieved with GLER and climatological LER for 13 November 2006. Again, the range 0.05<f<0.25 is used for the analysis. Over the ocean areas affected by sun glint, OCPs retrieved with GLER are significantly lower than those retrieved with climatological LER. The underestimation of OCP is mainly related to underestimation of O2-O2 SCDs in those areas. This issue requires further investigation. It should be noted that the cross-track dependence of OCP for cloudier scenes with f>0.25 is much flatter than that for lower cloudy scenes with f<0.25. The cross-track dependence of OCPs significantly deviates from the uniform dependence for pixels near the edges of the OMI swath. The OCPs at the edges are substantially higher than those in the near-nadir parts of the swath. This behavior of the OCP cross-track dependence is observed for both land and ocean and is not affected by the use of different surface reflectivities. This increase in OCP retrievals at the swath edges is not understood. It should be noted that similar behavior of the OCP cross-track dependence is also seen in the operational OMCLDO2 v2 retrievals for low f.