Detection of non-linear effects in satellite UV/Vis reflectance spectra: Application to the Ozone Monitoring Instrument

Non-linear effects, such as from saturation, stray light, or obstruction of light, negatively impact satellite measured ultraviolet and visible Earthshine radiance spectra and downstream retrievals of atmospheric and surface properties derived 10 from these spectra. In addition, excessive noise such as from cosmic ray impacts, prevalent within the South Atlantic Anomaly, can also degrade satellite radiance measurements. Saturation specifically pertains to observations of very bright surfaces such as sun glint over water surfaces or thick clouds. Related residual electronic cross-talk or blooming effects may occur in spatial pixels adjacent to a saturated area. Obstruction of light can occur within the zones of solar eclipses as well as from material located outside of the satellite instrument. The latter may also produce unintended scattered light into a 15 satellite instrument. When these effects cannot be corrected to an acceptable level for science quality retrievals, it is desirable to flag the affected pixels. Here, we introduce a new detection method that is based on the correlation, r, between the observed Earthshine radiance and solar irradiance spectra over a 10 nm-spectral range; our Decorrelation Index (DI for brevity) is simply defined as DI=1-r. DI increases with non-linear effects or excessive noise in either radiances (the most likely cause in OMI data) or irradiances. DI is relatively straight-forward to use and interpret and can be computed for 20 different wavelength intervals. We developed a set of DIs for two spectral channels of the Ozone Monitoring Instrument (OMI), a hyperspectral pushbroom imaging spectrometer. For each OMI spatial measurement, we define 14 wavelengthdependent DIs within the OMI visible channel (350-498 nm) and 6 DIs in its ultraviolet 2 (UV2) channel (310-370 nm). As defined, DIs reflect a continuous range of deviations of observed spectra from the reference irradiance spectrum that are complementary to the binary Saturation Possibility Warning (SPW) flags currently provided for each individual 25 spectral/spatial pixels in the OMI radiance data set. Smaller values of DI are also caused by a number of geophysical factors; this allows one to obtain interesting physical results on the global distribution of spectral variations.

and Deland, 2014). OMI's near real-time (NRT) data are disseminated within 3 hours of sensing (Krotkov et al., 2015), 35 while the very-fast-delivery (VFD) data are made available within 20 minutes of overpass, contributing to medium-range weather and air quality forecasts, as well as to detection and tracking of volcanic plumes (Hassinen et al., 2008;Levelt et al., 2018). OMI measurements also provides estimates of tropospheric ozone columns (e.g., Sellitto et al., 2011;Ziemke et al., 2017). Several sensors which are similar to OMI are currently in orbit, including the Tropospheric Monitoring Instrument Non-linear effects, such as saturation and blooming, can degrade Earthshine radiance measurements from passive solar backscatter UV/Vis satellite spectra and thus impact retrievals of atmospheric constituents. Saturation occurs when bright 45 light causes the number of electrons in a sensor pixel to exceed either the maximum charge capacity of an individual CCD photodiode, or the maximum charge transfer capacity of the sensor. Once the excessive electrons from the saturated pixel flow into neighboring pixels, saturation leads to CCD blooming, frequently rendering such data useless.
Hereafter we refer to the spatial domain (30 or 60 simultaneously acquired scenes) as rows, and the spectral domain as columns. Per OMI design, during the CCD readout the excessive charge spreads between spatial pixels more easily, and 50 therefore the blooming effect is predominantly observed between different rows.. Retrievals of atmospheric gases or aerosols can be compromised when observing very bright surfaces such as sun glint in low wind speed conditions (Cox and Munk, 1964;Kay et al., 2009;Butz et al., 2013;Feng et al., 2016), as well as over scenes predominantly covered by optically thick clouds. For example, the GOME2 (launched on 2007) UV band 1 detectors experienced saturation due to reflectance from clouds; this effect was predicted in the GOME-2 error assessment study (Siddans et al., 2002). Saturation caused by Sun 55 glint routinely occurs in the visible imagery of the MODerate Resolution Imaging Spectroradiometer (MODIS) flying on NASA's Aqua and Terra satellites. MODIS data show a gradual increase of saturated data towards the red and NIR bands, reaching around 1500 pixels, or ~ 0.03% of pixels, in a granule at 869 nm (Singh and Shanmugam, 2014). The Orbiting Carbon Observatory-2 (OCO-2) and similar greenhouse gas monitoring instruments occasionally point directly at the sunglint. The OCO-2 in-orbit checkout activities revealed an unexpectedly high signal from Lake Maracaibo, Venezuela on 60 August 7, 2014. This signal saturated all 3 channels and was attributed to an oil slick on a wave-free lake. After this event, known as the Lake Maracaibo Saturation Incident, an automated saturation warning algorithm was incorporated into the OCO-2 processing to identify such events (Crisp et al., 2017). Solar glint from ocean and clouds, as well as "saturation tails" or blooming effects are also seen in many images from the Earth Polychromatic Imaging Camera on the Deep Space Climate Observatory (EPIC/DSCOVR) (Varnai et al., 2019). A blooming effect occurs when electrons from a highly illuminated 65 pixel of the charge-coupled device (CCD) matrix jump to a neighboring pixel, causing distortion of its signal. TROPOMI also experiences detector saturation and blooming problems, typically caused by bright tropical clouds seen in bands 4 (400-areas are not detected by the TROPOMI L0-1b processor. A flagging algorithm is under development Ludewig et al., 2019). 70 A set of 16 operational flags, called the Saturation_Possibility_Warning (SPW) flags are currently included in the OMI level 1b data set. SPWs are designed to flag OMI pixels with 16 various radiation anomalies (e.g., saturation, stray light, nonlinearity). These flags are defined for each OMI wavelength: 751 wavelengths of the Vis spectrum and 557 wavelengths of the UV2 spectrum (GDPS, 2006). All of the 16 SPW flags are binary; a pixel with any degree of abnormality (e.g., 75 saturation) at a given wavelength is marked as bad.
Here, we describe a new approach to identify potentially erroneous OMI data based on the correlation r between the observed back-scattered Earthshine spectrum and a reference solar spectrum computed over limited spectral regions.
Earthshine spectra differ from the solar spectra due to Rayleigh, rotational-Raman, aerosol and surface scattering as well as 80 absorption of radiation by ozone and other atmospheric components. Most of these factors, with the exception of strong ozone absorption in the UV, amount to secondary effects on the correlation coefficient between the solar and Earthshine spectra within a limited spectral window. The degree of correlation under normal conditions depends mainly on the strength (depth) of solar Fraunhofer features and is typically close to unity. In this case, the correlation coefficient is sensitive to any additional systematic additive deviation in the Earthshine spectra and rapidly decreases in saturated (solar glint or bright 85 clouds) or other anomalous conditions such as cosmic ray hits on the detector. We apply our approach to OMI data and analyze individual cases and global distributions of flagged data. While these effects have been known for some time and dealt with by instrument teams by various methods, the prevalence of the different effects globally for a particular instrument has rarely been documented. This work provides a detailed analysis of non-linear effects specifically as they affect OMI, as well as a general and straight-forward approach that may be applied to similar instruments (TROPOMI, OMPS, GOME-2, 90 etc.) to identify and filter out suspect or erroneous data.

Ozone Monitoring Instrument (OMI) description
The Aura satellite that hosts OMI is in a polar Sun-synchronous orbit orbit with a local equator crossing time of 13:45.. OMI is a nadir-looking, push-broom UV/Vis grating spectrometer (Levelt et al., 2018). The light entering the telescope is 95 depolarized using a scrambler and then split into two channels: the UV (wavelength range 264-383 nm) and the Vis (wavelength range 349-504 nm) (Schenkeveld et al., 2017). The UV channel is further divided into the two sub-channels, UV1 (264-311 nm, 0.63 nm resolution and 0.21 nm sampling) and UV2 (307-383 nm range, 0.42 nm resolution with 0.14 nm sampling). Measurements are collected on two-dimensional charge-coupled device (CCD) sensors used for the UV and Vis channels. Spectral information is dispersed along one dimension of each CCD and spatial is imaged on the other. Each 100 https://doi.org/10.5194/amt-2020-327 Preprint. Discussion started: 25 August 2020 c Author(s) 2020. CC BY 4.0 License. channel has a devoted frame-transfer CCD detector with 6e5 electrons/pixel full-well capacity. To avoid blooming and ellipsoid effects, the pixel filling should be kept below 3e5 electrons (Dobber et al., 2006). OMI also measures the solar irradiance once per day through the solar port. Here, we use the UV2 sub-channel and Vis channel only; in the UV1 channel, strong, variable ozone absorption renders our approach impractical.

105
In the global mode, each orbit spans the pole-to-pole sunlit portion, typically comprising 1644 along-orbit exposures, referred to as iTimes hereafter. The 114 • viewing angle of the telescope corresponds to a 2600 km wide swath on the Earth's surface and consists of 60 simultaneously acquired rows or ground pixels across the track. In this mode, the OMI pixel size is 13 × 24 km 2 at nadir. The in-flight performance of OMI is discussed in Schenkeveld et al. (2017). The radiometric degradation of the OMI radiances since launch ranges from ~2 % in the UV channel to ~0.5 % in the Vis channel, which is 110 much lower than any similar sensor (Levelt et al., 2018). The one major disadvantage of OMI is the so-called row anomaly (Schenkeveld et al., 2017), which is presumably caused by a partial detachment of insulation material exterior to the instrument and produces a number of non-linear effects on sun-normalized radiances. The row anomaly is discussed in detail in Section 3.4.

The Decorrelation Index (DI) 115
We introduce a new parameter, the decorrelation index (DI), and defined as , where the correlation coefficient is derived for radiances and irradiances at each spectral region: for OMI, 14 regions of ~10 nm (51 wavelengths for each spectral region) on the Vis channel and 6 regions of ~10 nm (69 wavelengths for each region) on the UV2 channel. For the standard solar spectrum or reference irradiance, we take an average of all solar spectra obtained by OMI in 2005. We consider the atmospheric spectra of the UV2 and Vis channels separately. Each atmospheric spectrum is re-gridded via linear 120 interpolation to match the wavelengths of the averaged irradiance spectrum. An exact match between the radiance and irradiance spectral features gives DI = 0, whereas when the features in the radiance and irradiance spectra deviate, the DI approaches 1 to 2, where values greater than 1 indicate that irradiance and radiance spectra exhibit anti-correlation. Hence, cases of DI > 0 may indicate distortions of atmospheric spectra.

125
In this initial version of the OMI DI, we use the spectral range 309.9-370.0 nm for UV2 and 349.9-498.4 nm for Vis.
Overlapping of these ranges is useful for assessing the calibration between the UV2 and VIS channels. For solar zenith angles (SZA) > 90 o , the radiance level drops, noise begins to dominate, and the DI grows rapidly. Therefore, for the cases of SZA > 90 o , we do not compute the DI. The DI is sensitive to the degree of distortion of the reflectance spectrum, regardless of the cause of the distortion (saturation, crosstalk, noise etc), so that it detects distortions other than saturation. For example, 130 the DI may detect electronic cross-talk (or blooming) effects in pixels adjacent to the saturated area. In a number of cases, the decorrelation index proves to be either more or less sensitive than the current SPW (Saturation_Possibility_Warning) flags reported in the OMI PixelQualityFlags filed of the Level 1b data, as shown in the next section. The DI provides a range https://doi.org/10.5194/amt-2020-327 Preprint. Discussion started: 25 August 2020 c Author(s) 2020. CC BY 4.0 License. of values that describes the deviation of observed spectra from the reference irradiance spectrum, while the SPW flag is a binary value. The DI therefore allows flexibility in setting thresholds for different applications. The DI value for a given 135 spectral interval depends strongly on the number of Fraunhofer lines as well as presence of strong ozone absorption features within the wavelength range. Therefore, the DI values corresponding to likely damaged spectra vary somewhat for each spectral region. For example, the 14 DI divisions of the Vis spectrum generally fall into two distinct groups; for the first group, the value of DI above 0.01-0.03 is a sign of a significant distortion of the spectrum, while for the second group a typical distortion threshold value is larger (~0.1-0.4). 140

Results
To study the DI, we first concentrate on scenes that are most likely to contain saturation and blooming effects: sun glint areas 155 with relatively calm water surfaces and contiguous bands of deep convective clouds. Next, we examine the global distribution of the DI, which reveals other effects that damage observed spectra. We then investigate the impact of the row anomaly on the DI.

Saturation over clouds
A typical problematic cluster of bright clouds in the Pacific Ocean is shown in Fig.1a, where two zones are highlighted, a 160 small northern zone (denoted A) and a large southern zone (marked as B). Figure 1c shows the number of wavelengths for a given pixel marked with the SPW flag as saturated. Figure 1b -1d shows the corresponding DI values for the Vis interval 414-424 nm. The DI indicates that the spectra in zone A are weakly affected, and in zone B they are badly damaged. Figure   1c shows the number of wavelengths for a given pixel marked with the SPW flag as saturated.

165
Figures 2 and 3 illustrate the properties of the DI that characterize the quality of a given part of the spectrum using a single parameter. Figure 2 shows an example of a spectrum with slight distortions that are captured by the SPW flags, but nevertheless has low values of the DI. Small deviations of the DI from 0 can result from geophysical effects, for example, an increased amount of ozone, and minor damage to the spectrum, as shown in Fig. 2. Those users who have strict requirements for the quality of the spectra should use the SPW flag in this case, which detects minor damage to the 170 spectrum. Figure 3 shows the Vis spectrum for a pixel in zone B (indicated by an arrow in Fig. 1)  Raman scattering also known as the Ring effect (e.g., Joiner et al., 1995). Both zones in Fig.1 have high values of 180 reflectance; for zone A, reflectance is between 0.95 and 1.0 (Fig. 2), while for zone B, reflectance is between 1.0 and 1.1 (Fig. 3). In some viewing directions the reflectance can exceed unity due to anisotropic angular distribution of the TOA radiance.    Fig. 1(c,d). The red and black lines are the radiance and irradiance, respectively (the magnitude of the irradiances is shifted to line up with the radiance. Radiance is reported in photons sr -1 nm -1 cm -2 , irradiancein photons nm -1 cm -2 . Both radiance and irradiance    Fig.1(c,d). The radiance was lowered by a few percent for better comparison.

Saturation over lakes and ocean 205
The South American lake Salar de Uyuni is used for calibration of many satellite sensors (Lamparelli et al., 2003;Fricker et al., 2005). Salar de Uyuni is dry for most months of the year, but during the rainy season, it is filled with shallow water with strong direct reflectance from the sun. This may cause saturation of OMI's detectors. The lake, covered with shallow water, generated strong solar glint, for example, on orbit 7987 (January 14, 2006). Figure 4a shows this shallow lake on January 14, 2006 as observed by the Aqua MODIS sensor. The SPW flags (Fig. 4c) and DIs (Fig. 4b,d) for this case show that the lake 210 generates two bright spots: southern and northern. The solar glint from the northern spot is so bright that the signal extends to nearby pixels (iTimes 823-825, Rows # 11-14). The resulting spectral distortions, called blooming, are caused by artificially increased radiance and are detected by the DI (see also Cao et  220 Figure 5 shows the Vis spectrum for a pixel where a strong distortion due to a blooming effect produced amplification in the radiance spectrum (iTimes -824, Row -15). Most of the reflectance values for these pixels are in the normal range (0.3-0.6).
However, perturbations, in the form of peaks in the reflectance due to blooming, often exceed unity. The DI highlights the affected parts of the spectrum.   Fig. 4(c,d) showing solar glint from Lake Salar de Uyuni. Figure 6 shows the spectrum for a pixel where the spectral distortion due to the blooming effect causes peaks in the radiance 230 and reflectance (iTimes -824, Row -11). High DI values are seen for a number of corrupted parts of the spectrum where the SPW flags are zero. Users who make special demands on the quality of the spectra can use spectra whose quality is confirmed by both parameters.       The interpretation of low DI for normal spectra (for example, spectra with DI > 0.1 for Vis 445.3-455.7 nm) is quite complicated as low DI values depend on many factors. Figure 9 shows the spatial distribution of the number of spectra with 270 DI > 0.1 in the 445.3-455.7 nm region compared with ocean reflectance at 443 nm. There is obvious spatial correlation between the spectra the DI identifies and ocean reflectance: larger numbers of such spectra correspond to ocean areas with higher reflectance. This is particularly pronounced in the southern Pacific Gyre whose waters exhibit extremely low bioproductivity and thus are very bright in the blue region (Tedetti et al., 2007). The strong spectral dependence of waterleaving reflectance in the blue region in these extremely clear waters results in lower correlation with the solar spectrum. 275

Orbital and Global Distribution
This may be attributed in part to vibrational Raman scattering that is prevalent in clear ocean waters (Vasilkov et al., 2002;Westberry et al., 2013). Additionally, the Pacific Gyre area is characterized by low cloudiness and low aerosol loadings.
Therefore, Rayleigh and Raman scattering in the atmosphere contributes significantly to the top-of-atmosphere radiance. The The distribution of bright clouds with DI > 0.25 also shows a strong propensity for the geometrical conditions of solar glint ( Fig.10a). This is consistent with EPIC/DSCOVR' data showing solar glint from clouds that contain oriented ice plates (Varnai et al., 2019). These strongly saturated (or damaged) spectra with DI > 0.60 number about 2500 (~0.0005%) or ~7 spectra/day. Slightly affected spectra (0.25 < DI < 0.6) occur at a rate of ~0.002% or ~33 spectra/day. 290

Row Anomaly
The row anomaly (RA) renders a significant portion of the OMI rows as unusable. The anomaly was clearly detected in two rows in June 2007. In May 2008 the row anomaly spread to several other rows on the sensor. The row anomaly has continued to develop since then, with particularly swift changes around January 2009 and early fall of 2011. Currently about 33% of the UV2 rows are affected in the southern hemisphere parts of the OMI orbit. This increases to ∼57 % in the 295 northern hemisphere. These estimates are comparable in the Vis channels (Schenkeveld et al., 2017). Figure 11 similarly shows DI distributions in the space row by iTimes, but for the overlapping region of the UV-2 and Vis detectors. This shows that the row anomaly is well detected by the DI in both detectors though with more impact on the Vis detector. The row anomaly is a complex phenomenon that may result in artificially low or high values of radiances, depending on the interplay of two major factors: the blocking of the incoming Earth shine and the solar light scattering. The former generally increases 300 Dis. At the same time, the additive component related to the scattered (most likely, by the instrument's thermal blanket) solar light is known to predominantly affect the radiances in the northern part of OMI orbit. This leads to the significant decrease of Dis in the RA-affected areas seen in Fig. 11. Figure 11 shows that the number of spectra with DI > 0.01 generally increases from polar to equatorial regions. This may be explained by an increase in the contribution from the oceans that undergoes vibrational Raman scattering (see Fig. 9). But in the area of rows 10-30 and iTimes 650-1000, there is a region 305 with lower DI values. This spot is clearly associated with the growth of bright sun glare from the sea and cloud that reduces the amount of Raman scattered light from the atmosphere and ocean and thus reduces the DI. We expect that the value of the DI in these regions depends on the average slopes and wave heights in a given month as well as on wind speed. This behavior contrasts with the increased DI values at the center of these regions in the Vis (Fig. 10). This is because an increase in the fraction of solar radiation decreases the DI, but when the solar radiation exceeds the limits of saturation, the spectra 310 will begin to be damaged and a local zone with a high DI (see Fig. 10) will appear in the center of the zone with reduced DI. https://doi.org/10.5194/amt-2020-327 Preprint. Discussion started: 25 August 2020 c Author(s) 2020. CC BY 4.0 License.

Discussion and Conclusions
We are convinced of the completeness of the PixelQualityFlags (PQF) designed to characterize each wavelength of the OMI spectrum (SPW flag is just one of the 16 bits in the PQF). The DI, developed on the basis of the correlations between observed and solar spectra, can serve as a simple but effective and complementary method for detecting and discarding 320 anomalous UV and Vis satellite spectra, for example associated with saturation, blooming, excessive noise, low reflectivity (as in solar eclipse), or the OMI row anomaly. The DI summarizes all changes in the spectrum in one parameter and https://doi.org/10.5194/amt-2020-327 Preprint. Discussion started: 25 August 2020 c Author(s) 2020. CC BY 4.0 License.
eliminates the need to examine all the available flags for a given pixel. An important motive for introducing such an index is the convenience of handling it. For example, to infer enhanced information of the quality of spectra in the Vis region, we introduce 14 scalar-valued DIs for regions of the spectrum. For comparison, there are 751 binary saturation flags per 325 spectrum in the level 1b. Similarly, we use 6 DIs for the UV2 spectrum; much less than the 577 flags assigned in the level 1b. Interpreting a large number of flags can be difficult. The DI product gives an indication of spectral quality based on overall correlation that is easier to interpret. The continuous nature of the DI allows data users to assign lower confidence to regions of the spectra that may not be completely saturated as detected by an electronic saturation algorithm. DI values vary for spectra that do not experience any anomalies. These variations of the DI may carry information that can be used for other 330 purposes. For instance, the DI can be used to search for areas of clear ocean water, in which the spectra are not abnormal, but experience significant deviations from the solar spectrum due to geophysical reasons.
The DI can be a useful tool for analyzing spectra obtained from other current and future space-borne sensors that may suffer from saturation and blooming such as TROPOMI (launched in 2017) or the similar Environmental trace gases Monitoring 335 Instrument (EMI) on the GaoFen-5 satellite (Chen, 2016)  and/or smaller field of view (FOV) than OMI. For such instruments, this may lead to an increase in the effects of sun glint.
Studies utilizing the DI with current instruments may benefit the design of future instruments by identifying how often and under what conditions spectra are impacted by non-linear effects.

Data availability
The Decorrelation Index data for OMI Collection 3 data will be available at GES DISC. The OMI Level 1b data used for 345 calculations of the DI are available at https://aura.gesdisc.eosdis.nasa.gov/data/Aura_OMI_Level1/. MODIS data are available at https://worldview.earthdata.nasa.gov/.

Competing interests
The authors declare that they have no conflicts of interest.

Authors contributions 350
NG developed a computer code, analyzed the DI results and wrote the manuscript. ZF supported the development and implementation of the algorithms and comparison DI results with the ocean reflectance (Fig.9). DH proposed a concept of DI and wrote the manuscript. SM proposed a concept of DI and supported the development and implementation of the algorithms. JJ set the task of developing an effective method for determining solar glints, supported the development of the algorithm, and wrote the manuscript. AV supported the development of the algorithm, and wrote the manuscript (chapters 355 3.1-3.4).

Financial support
This work was supported by the NASA Aura project (OMI core team) managed by Ken Jucks.