Aerosol extinction is only retrieved where aerosol is detected by the CALIOP feature finder. Level 2 atmospheric samples classified as “clear-air” (i.e., no feature is detected) are assumed in the level 3 algorithm to have aerosol extinction equal to 0 km-1, denoted by σclear (specifically, extinction σ for clear-air samples are assigned σ≡σclear; the triple bar denotes the assignment). However, because layer detection is based on vertically resolved backscatter, diffuse aerosol layers which span a large altitude range can remain undetected, particularly if they have significant absorption. Solar background noise further impacts feature detection (Winker et al., 2013). Assuming σclear=0 km-1 thereby provides a lower bound on the true aerosol extinction. In reality, aerosol is present virtually everywhere throughout the troposphere (e.g., Kim et al., 2017), though concentrations can be very low in regions of the free troposphere not affected by continental transport. Clarke and Kapustin (2002), for example, show background aerosol extinction levels of 10-4 to 10-3 km-1 in remote parts of the Pacific basin, implying a missing AOD ranging from 10-3 to 10-2 in the cleanest regions (assuming well-mixed aerosols in a 10 km deep column).

Several researchers have recently sought to characterize the optical depths of the aerosol layers undetected by CALIOP using collocated observations (Kacenelenbogen et al., 2011; Sheridan et al., 2012; Rogers et al., 2014; Thorsen and Fu, 2015; Toth et al., 2018) or independent retrievals (Winker et al., 2013; Kim et al., 2017). Exactly how these undetected layers affect the level 3 mean extinction is difficult to estimate given that the resulting underestimate depends on the magnitude of missing extinction and the frequency of non-detection. Answering this question is a topic for forthcoming level 3 aerosol product validation.

The CALIOP feature finder sometimes leaves a gap between the base of the lowest aerosol layer and the surface, even in cases where the aerosol layer extends to the surface. An example over the Pacific Ocean is shown in Fig. 3a, circled in red. In this region, the dominant aerosol source is the ocean itself and the marine boundary layer is well-mixed, so it is reasonable to expect aerosol to exist down to the surface. However, aerosol is not identified in range bins near the surface. The level 2 aerosol base extension algorithm is designed to compensate for situations like this by extending aerosol layer bases downward to capture more of the surface-attached layer (Vaughan et al., 2010). However, gaps of apparent clear-air between the surface and aerosol layer base can remain for two reasons. First, base extension is only executed if the integrated attenuated backscatter signal between the original layer base and the surface is positive. In Fig. 3b the backscatter signal adjacent to the surface is strongly negative due to the negative signal anomaly (discussed in Sect. 5.4) so these aerosol layer bases are not extended. Second, the base extension algorithm only extends layer bases to 90 m above the local surface in order to prevent the surface signal from contaminating the aerosol profile. These “clear-air” gaps would cause a low-bias in the level 3 mean aerosol extinction profile near the surface if they were assigned σclear=0 km-1.

To avoid a low bias in near-surface mean aerosol extinction, the level 3 algorithm ignores all clear-air samples below the lowest aerosol layer in each column having a base below 250 m. The underlying assumption is that the atmosphere is well mixed below 250 m. Turbulent mixing within the daytime boundary layer tends to homogenize aerosol loading, and the planetary boundary layer is generally much deeper than 250 m for marine and continental conditions (e.g., McGrath-Spangler and Denning, 2013; Luo et al., 2014). Note that the beta version of the level 3 product used 2.46 km rather than 250 m as the threshold (Winker et al., 2013). Ignoring the range bins in near-surface gaps gives more weight to range bins where aerosol was detected, preventing a low-biased level 3 average. Figure 4 shows the effect on a level 3 mean aerosol extinction profile over the Arabian Sea. The extinction of the range bin nearest to the surface is increased, making the drop-off in extinction less severe. Consequently, global mean level 3 AOD is increased by a small amount, roughly 1 %.

Mean aerosol extinction is calculated from all quality-screened level 2 aerosol extinction coefficients (σ) and clear-air samples within each latitude, longitude, altitude grid cell using Eq. (1). σ‾=∑i=1Naerσaer,i+∑j=1Nclearσclear,jNaer+Nclear Here, σ‾ is the monthly mean aerosol extinction coefficient, σaer,i is the set of aerosol extinction coefficients accepted by quality screening, σclear,j is the set of clear-air aerosol extinction coefficients retained after accounting for near-surface aerosol (Sect. 4.2), Naer is the total number of aerosol extinction samples accepted, and Nclear is the number of clear-air samples in the grid cell. Under the assumption that σclear=0 km-1 and the definition Navg=Naer+Nclear, Eq. (1) reduces to Eq. (2). σ‾=∑i=1Naerσaer,iNavg Profiles of σ‾, Naer and Navg are reported in the level 3 product with the science data set (SDS) names Extinction_532_Mean, Samples_Aerosol_Detected_Accepted, and Samples_Averaged, respectively. Multi-month averages of aerosol extinction can be calculated from σ‾ by weighting each month by Navg.

Mean aerosol extinction is reported for all aerosol species combined and reported separately for dust, polluted dust, and smoke. When computing σ‾ for a single-species, σaer for all other aerosol species is assumed to equal 0 km-1. This is consistent with the CALIPSO aerosol typing paradigm where aerosol layers are assigned a single type. In reality, different aerosol types can be mixed within the same layer, but the CALIPSO aerosol typing algorithm is unable to determine when different species are mixed or by what proportions. Therefore, assigning σaer≡0 km-1 for other species is equivalent to assuming that only one aerosol type is present in the detected layer. By contrast, the beta version of the level 3 product ignored other species rather than setting their extinction to 0 km-1. This caused extinction to be biased high where multiple aerosol subtypes exist at the same altitude, as demonstrated by Amiridis et al. (2013) (their Fig. 7 and accompanying discussion). Assigning σaer≡0 km-1 for other species avoids these biases and maintains consistency with the CALIPSO aerosol typing paradigm.

AOD is the standard parameter used by spaceborne passive sensors and sun photometers to quantify total column aerosol loading in cloud-free sky conditions. Temporal averaging is accomplished by averaging a set of AOD measurements/retrievals over the time period of interest. However, computing temporally averaged AOD for CALIOP retrievals, requires a different approach because the averaging set consists of σ profiles rather than total column AOD measurements. In the level 3 product, monthly mean AOD is computed by first averaging the set of quality-screened σ profiles for the month and then vertically integrating the mean extinction profile σ‾, i.e., average-then-integrate. The alternate method is to first integrate each of the quality-screened σ profiles and then average the set of AODs, i.e., integrate-then-average. These two methods do not produce the same results. Figure 5 shows that the monthly mean AOD for these two methods is very different for both the all-sky and cloud-free sky conditions; mean AOD is often smaller when integrate-then-average is used. This is because the σ profiles in the averaging set do not uniformly sample the same geometric depth of the atmosphere after cloud-clearing and quality screening.

As a simplified example, the integrate-then-average AOD will be artificially small for two columns where one σ profile extends to the surface and the other profile stops at 10 km due to an opaque cloud. The first profile will have a larger AOD because it observed aerosol down to the surface, whereas the second profile will have a smaller AOD because aerosol observations are terminated at 10 km. These two σ profiles do not measure the same geometric depth and the subsequent mean AOD is biased low. This example readily illustrates the mean AOD differences for the all-sky condition where clouds exist in the averaging set (Fig. 5a). Further, the cloud-free sky condition also exhibits lower mean AOD for integrate-then-average even though the observations are unencumbered by clouds (Fig. 5b). In this case, the geometric depth still differs between the two methods because σ samples are rejected from various range bins by quality screening. The net effect yields σ profiles with disparate geometric depths for both the cloud-free and all-sky sky conditions. In short, mean AOD will always be biased low when computed by the integrate-then-average method. Hence, level 3 mean AOD is computed by averaging then integrating. This is an important consideration for computing AOD from space-based profiling instruments.

Level 2 aerosol layers detected at 80 km horizontal resolution that are not in contact with another aerosol layer are assumed to be noise-induced misclassifications and are rejected.

In low SNR regions such as beneath optically dense clouds, some detected features may actually be artifacts due to noise rather than legitimate aerosol. These noise artifacts are usually detected at 80 km horizontal averaging resolution. This filter reduces the occurrence of noise misclassified as aerosol.

In scenes with significant overlying attenuation, features may be detected at 80 km resolution after more strongly scattering features have been detected and removed. However, if these layers are isolated and not in contact with other aerosol layers, it is possible they represent detection artifacts rather than actual aerosol layers. These weakly scattering layers contribute little to monthly mean AOD, but would affect the spatial distribution of level 3 aerosol occurrence if accepted. For this reason, the level 3 algorithm rejects isolated aerosol layers detected at 80 km resolution.

For the 10-year evaluation period, the isolated 80 km aerosol layer filter rejected 3.8 % (5.9 %) of samples at night (day). Daytime rejection is higher because solar noise reduces SNR, making coarser averaging necessary for layer detection relative to night. The largest frequency of rejection is over the poles and over Greenland (Fig. 8b). Aerosol is most often rejected at altitudes where deep convective clouds are expected (Mace and Wrenn, 2013): above 8 km at the equator and lower towards the poles (Fig. 9b). During the day, larger rejection frequencies occur at lower altitudes: ∼6 km over the equator and near 4 km towards the poles (Fig. S4b). In this case, legitimate aerosol may be rejected because weakly scattering aerosol layers are not always detected due to the reduced SNR. Therefore, it becomes less likely for an aerosol layer detected at 80 km resolution to be in contact with another, and the possibility of rejection is higher. This phenomenon is exacerbated by high albedo surfaces, which induce noise through the profile, limiting the fidelity of feature detection.

Another source of error that can bias level 3 aerosol statistics is clouds misclassified as aerosol. Two filters are employed to reject layers suspected of being misclassified clouds. The first filter uses the CAD score, a built-in level 2 quality flag with a strong empirical foundation. The second filter uses a spatial proximity test to reduce the impact of the tenuous edges of cirrus clouds that are misclassified as aerosol.

Two metrics reported in the level 2 aerosol profile product are used to assess the quality of extinction retrievals: the extinction QC flag and the extinction uncertainty. The extinction QC flag summarizes the final state of the extinction retrieval solution, while the extinction uncertainty provides an estimate of systematic and random errors. Note that these filters do not remove negative extinction values. Though unphysical, negative extinction values can result from signal noise and must be retained to prevent biasing σ‾ high.

All level 2 atmospheric samples (aerosol, cloud, and clear-air) are ignored within 60 m of the local surface to avoid aerosol extinction affected by the negative signal anomaly.

The final quality filter addresses an intermittent phenomenon referred to as the “negative signal anomaly” (NSA). This signal artifact occurs when the level 1B attenuated backscatter becomes strongly negative preceding a strongly scattering target such as the surface. The NSA is intermittent, but tends to occur in sequences of adjacent profiles within latitude bands that vary seasonally. If these negative spikes are treated as part of a surface-attached aerosol layer, they can produce large negative aerosol extinction values just above the surface. An example of the NSA is evident in the attenuated backscatter signal shown in Fig. 18a. The retrieved σ value from this data can be strongly negative, or worse, it can bias the signal low and yet still remain positive. Figure 18b shows three aerosol extinction profiles retrieved from the attenuated backscatter in Fig. 18a along separate 5 km segments containing the NSA. While the strongly negative values adjacent to the surface are readily apparent for the extinction profiles in this example, positive σ values that are biased low are not as easy to detect. This can occur because σ is retrieved after averaging 15 level 1B attenuated backscatter profiles to 5 km horizontal resolution. If only some of the level 1B profiles are affected by the NSA, the average backscatter can still be positive, yet biased low.

In order to prevent near-surface σ affected by the NSA from biasing σ‾, all atmospheric samples within 60 m of the local surface are ignored. This approach was adopted because it is difficult to know when the NSA has influenced σ at the surface. An example of the impact of this NSA mitigation is shown in Fig. 19. AOD increases by roughly 5–10 % in level 3 profiles affected by the NSA (based on values >1 within the red boxes) because strongly negative near-surface σ is rejected. Conversely, the NSA in this example is also present along the equator, and yet excluding these σ values does not increase AOD, illustrating the difficulty of predicting the influence of the NSA on retrieved extinction. AOD also decreases by roughly 5 % on average in unaffected regions, a consequence of this conservative strategy. Note that recently released version 4 level 1B and level 2 data products have mitigation procedures in place to remove the effect of the NSA on σ by excluding affected level 1B backscatter (Vaughan et al., 2018). Future versions of the level 3 aerosol product using version 4 data should no longer require the mitigation strategy described here.

The quality filters implemented in the level 3 aerosol algorithm influence σ‾ profiles and AOD to varying degrees. This section quantitatively compares the influence of the quality filters on σ‾ and AOD when applied independently in order to identify the most influential filters. Though the rejection frequency of different filters varies regionally and seasonally, a globally averaged annual summary is sufficient to establish their relative ranking.

Figure 20 summarizes the impact of quality filters on the global σ‾ profile, the frequency of rejection, and filter aggressiveness. The aggressiveness metric Agr(z) indicates the effectiveness of sample rejection on changing σ‾, with larger values indicating the filter is more aggressive at changing σ‾ than other filters. It is computed as the change in σ‾ per reduction in number of aerosol samples due to filtering (Eq. B1). For context, Fig. 20b shows the number of unfiltered aerosol samples, which decreases with increasing altitude.

The filters rejecting the highest frequency of samples below 2 km are the CAD and extinction QC filters (Fig. 20c). Above 4 km, the cirrus fringe and isolated 80 km layer filters dominate the aerosol sample rejection. Based on the Agr(z) metric in Fig. 20d, the extinction QC filter is the most aggressive in changing σ‾ at low altitudes (<4 km) despite the CAD filter rejecting a higher frequency of samples below 2 km. This demonstrates that σ rejected by the extinction QC filter is often quite large relative to σ rejected by the CAD filter. Above ∼6 km, the cirrus fringe filter is by far the most aggressive at changing σ‾. A similar conclusion is expected for small (∼1 km) perturbations of the 4 km altitude threshold for this filter. Daytime σ‾ at these high altitudes is influenced by both the cirrus fringe and isolated 80 km layer filters to a similar degree (Fig. S6d).

As a global summary of the impacts of quality filtering on σ‾ and AOD, Table 3 compares four metrics for each filter applied independently: quality filtered AOD, percent change in AOD (ΔAOD), change in extinction scale height (Δz63, Eq. B3), and sample-weighted mean filter aggressiveness (Agr, Eq. B3). While mean AOD and its percent change ΔAOD characterize the full-column impact on strongly scattering aerosol, Δz63 is an indicator of impacts on the vertical distribution. Positive values indicate that the altitude containing 63 % of total AOD has moved upward after quality filtering.

The most influential quality filters are the extinction QC and extinction uncertainty filters, respectively. The extinction QC filter is responsible for the largest reductions in AOD: -19 % (global ocean) to -28 % (global land). This accounts for all but 3–5 % of total AOD reductions due to all filters together. The extinction uncertainty filter is the second-most impactful filter in terms of AOD reduction, but with reductions 2–3 times smaller than the extinction QC filter. These same two filters are also responsible for increasing Δz63 over land by 180–240 m (with the extinction QC filter causing the larger increase). The altitude containing the bulk of AOD increases because these filters are more aggressive in the lowest altitudes (Fig. 20d), leaving higher-altitude aerosol to contribute more to the total AOD. For the remaining filters, Δz63 is zero or decreases by 60 m (over land and ocean) because these filters act upon layers at higher altitudes. Reducing σ‾ at higher altitudes allows lower-altitude aerosol to contribute more to total AOD. In terms of filter aggressiveness Agr, the extinction QC filter is the most aggressive of all filters with the cirrus fringe filter being the second most aggressive, albeit at higher altitudes (Fig. 20d).

This section examines changes in σ‾ and AOD due to quality filtering in twelve key regions (Fig. 21). These regions are roughly the same as those defined by Yu et al. (2010) and adapted by Koffi et al. (2016) to characterize dust, marine, biomass burning, and industrial aerosols for global aerosol model comparisons with CALIOP retrievals.

Regional σ‾ profiles are shown in Fig. 22 before filtering and after applying all quality filters. Median surface elevations are shown to indicate altitudes where the number of samples averaged begins to decrease (often rapidly) relative to higher altitudes, thereby increasing the uncertainty in the mean values being compared. Impacts of applying just the extinction QC filter are shown as the dashed green line. The σ‾ profile differences are very slight between the extinction QC filter-only and all-filters cases. Most differences occur at the surface where the extinction uncertainty filter has rejected suspiciously large extinction values; e.g., the Central Africa (CAF) region. In order to compare changes in σ‾ and AOD quantitatively within these regions, Fig. 23 presents the same change metrics defined in the Sect. 6.1. Numerical values for these metrics, their seasonal counterparts, and AODs are tabulated in Table S1 and shown in Fig. S7.

Regions experiencing the highest fractional AOD reductions also tend to have lower AOD both before and after filtering relative to other regions. For instance, the Eastern United States (EUS) and the North Atlantic Ocean (NAT) have the lowest annual AOD relative to other regions, yet the AOD reductions are among the highest: ΔAOD=-34 and -33 %, respectively (Fig. 23a). South America (SAM) experiences the highest annual AOD reduction, but this occurs during the March–May (MAM) season when mean AOD is lower than in the September–November (SON) season when biomass burning becomes prevalent (AOD=0.09 vs. 0.30, respectively; Table S1). A possible explanation is that the default Sp for smoke is closer to the true value during the biomass burning season, requiring fewer Sp reductions, compared to default Sp values used during the non-burning DJF–MAM seasons. Mean AOD is reduced by around 24 % for regions with high AOD: India (IND), Eastern China (ECN), North and Central Africa (NAF, CAF). Mean AOD over Western China (WCN), the region with the highest AOD, is reduced by 34 %.

The change in extinction scale height Δz63 is positive for most regions (Fig. 23b), indicating that the altitude containing the bulk of the mean AOD is higher after quality filtering. For most regions, z63 exhibits small increases of 60 m. Larger increases of 180–240 m occur over land where large σ values near the surface are rejected. For example, the largest change occurs in SAM, where the maximum unfiltered extinction value at the surface was reduced substantially by quality filtering (Fig. 22). As the majority of the AOD is no longer contained within the large near-surface peak, z63 is higher after quality filtering. The CAF and Southern Africa (SAF) regions also experience an z63 increase of 120–180 m for this same reason.

Aerosol sample-weighted quality filter aggressiveness Agr (Fig. 23c) is largest in WCN, particularly during DJF (Fig. S7c), where AOD and the frequency of extQC=2 solutions are relatively high (Fig. 8e). Since the extinction QC filter impacts a high proportion of the aerosol retrievals in this region, the filter has a strong impact on the mean aerosol extinction profile. Quality filtering is also relatively more aggressive in the NAT and Northwest Pacific (NWP) regions where aerosol loading is typically low. When aerosol loading is low, rejecting just a small number of aerosol samples may have a large impact on σ‾ because there are not many aerosol samples to begin with. Despite the substantial change to the SAF σ‾ profile below the median surface elevation (Fig. 22), Agr is small relative to other regions because the metric is weighted by aerosol sample number, and most aerosol in that region is elevated above 1 km. Quality filtering is least aggressive over NAF.