Cloud misclassification is a serious problem in the retrieval of aerosol optical depth (AOD), which might considerably bias the AOD results. On the one hand, residual cloud contamination leads to AOD overestimation, whereas the removal of high-AOD pixels (due to their misclassification as clouds) leads to underestimation. To remove cloud-contaminated areas in AOD retrieved from reflectances measured with the (Advanced) Along Track Scanning Radiometers (ATSR-2 and AATSR), using the ATSR dual-view algorithm (ADV) over land or the ATSR single-view algorithm (ASV) over ocean, a cloud post-processing (CPP) scheme has been developed at the Finnish Meteorological Institute (FMI) as described in Kolmonen et al. (2016). The application of this scheme results in the removal of cloud-contaminated areas, providing spatially smoother AOD maps and favourable comparison with AOD obtained from the ground-based reference measurements from the AERONET sun photometer network. However, closer inspection shows that the CPP also removes areas with elevated AOD not due to cloud contamination, as shown in this paper. We present an improved CPP scheme which better discriminates between cloud-free and cloud-contaminated areas. The CPP thresholds have been further evaluated and adjusted according to the findings. The thresholds for the detection of high-AOD regions (> 60 % of the retrieved pixels should be high-AOD (> 0.6) pixels), and cloud contamination criteria for low-AOD regions have been accepted as the default for AOD global post-processing in the improved CPP. Retaining elevated AOD while effectively removing cloud-contaminated pixels affects the resulting global and regional mean AOD values as well as coverage. Effects of the CPP scheme on both spatial and temporal variation for the period 2002–2012 are discussed. With the improved CPP, the AOD coverage increases by 10–15 % with respect to the existing scheme. The validation versus AERONET shows an improvement of the correlation coefficient from 0.84 to 0.86 for the global data set for the period 2002–2012. The global aggregated AOD over land for the period 2003–2011 is 0.163 with the improved CPP compared to 0.144 with the existing scheme. The aggregated AOD over ocean and globally (land and ocean together) is 0.164 with the improved CPP scheme (compared to 0.152 and 0.150 with the existing scheme, for ocean and globally respectively). Effects of the improved CPP scheme on the 10-year time series are illustrated and seasonal and temporal changes are discussed. The improved CPP method introduced here is applicable to other aerosol retrieval algorithms. However, the thresholds for detecting the high-AOD regions, which were developed for AATSR, might have to be adjusted to the actual features of the instruments.
The retrieval of aerosol properties from radiance measured at the top of the atmosphere (TOA) using space-borne instruments is highly sensitive to the presence of clouds. Aerosol retrieval is only performed for cloud-free areas, which implies that a very strict cloud detection scheme has to be applied to remove all cloud-contaminated pixels from the retrieval area. If pixels in the retrieval area contain undetected clouds, the aerosol optical depth (AOD) will be too high, while, if the cloud detection is too strict, i.e. pixels are removed which do not contain clouds, aerosol pixels are wrongly discarded. Thus, effective cloud screening for aerosol retrieval is important and requires sophisticated algorithms and multispectral visible and infrared radiance data (e.g. Remer et al., 2005; Frey et al., 2008; Grandey et al., 2013; Backer, 2013; Shi et al., 2014).
No perfect method for cloud detection in satellite data exists and cloud
contamination is considered one of the major problems in aerosol retrieval
results. Cloud contamination imposes a positive but unknown bias in the AOD
values which may vary with time and thus hamper the use of these data for
trend analysis and other studies. Shi et al. (2014) showed that, on
average, thin cirrus cloud contamination introduces a possible
Cloud misclassification occurs because of the large variety in cloud and underlying surface properties. It comes from, e.g. the thin cloud pixels (underestimation over dark surfaces, overestimation over a high albedo surface), small cumulus pixels, clear pixels over bright surface (e.g. Wang et al., 2013). Optically dense aerosol features, such as desert dust plumes, volcanic ash, biomass burning aerosol and industrial pollution, can be misidentified as clouds (Martins et al., 2002; Kahn et al., 2007; Stap et al., 2015).
False-colour RGB images (composite of 0.555, 0.659 and
0.865
Stereo view effect in nadir/forward cloud screening, for the overpass on 29 July 2008, ca. 10:00 UTC (as in Fig. 1). Clouds detected with at least one of four tests in both views, in only nadir and in only forward views are coloured with light blue, magenta and green respectively. To better show the stereo effect, cloud screening over ocean was performed here for both nadir and forward views.
To eliminate residual clouds from the retrieved AOD fields, a cloud post-processing method has been developed to recognise and discard undetected clouds in AOD retrieved from the AATSR radiances with the ATSR dual-view (ADV) algorithm for aerosol retrieval over land and the ATSR single-view (ASV) aerosol retrieval algorithm for application over ocean (Kolmonen et al., 2016). The ATSR has been designed to measure sea surface temperature and, therefore, the cloud detection scheme designed for use with this instrument has been optimised for application over open ocean and does not perform well over land (Závody et al., 2000; Birks, 2007a). Therefore, an improved cloud detection scheme has been developed for application to ADV/ASV (Roblez González, 2003; Kolmonen et al., 2016), but the retrieved AOD is still affected by residual cloud contamination. To remove this, a cloud post-processing (CPP) algorithm has been designed at the Finnish Meteorological Institute (FMI) which partially solves the problem, as illustrated by Kolmonen et al. (2016), and results in smoother AOD maps and improved validation results when compared to AOD data from the AERONET, which is a federated network of sun photometer instruments (Holben et al., 1998). Based on the CPP test, if a certain retrieved pixel is found to be cloud-contaminated, as recognised from the AOD spatial distribution, no retrieval products are provided for that pixel. However, as shown in this paper, it also appears that areas with elevated AOD may inadvertently be removed by the existing CPP method (Kolmonen et al., 2016), resulting in screening of high-AOD events. To avoid this, a method has been developed to detect high-AOD regions and prevent their elimination from the retrieval results, while still effectively removing cloud-contaminated pixels as in the existing CCP scheme. This method is applicable to other aerosol retrieval algorithms. However, the thresholds for detecting the high-AOD regions, which were developed for AATSR, might have to be adapted to the actual features of instruments (e.g. width of the swath).
The paper is structured as follows. The most important features of the AATSR instrument, the ADV/ASV retrieval algorithms and the ADV/ASV cloud tests are described in Sect. 2.1. Cloud-screening results and retrieved AOD are illustrated with some examples in Sect. 2.2. Post-processing methods, both the existing and the improved versions, are described in Sect. 3. Test results are illustrated with examples (Sect. 4.1). The AOD results are evaluated in Sect. 4.2. The effects of the two schemes are discussed in Sect. 4.3 as regards the effects of the improved CPP on the spatial AOD distributions globally and over eastern China. Time series over different regions for 2002–2012 are presented in Sect. 4.4. The conclusions are summarised in Sect. 5.
The ATSR dual-view (ADV) algorithm has been developed for the ATSR
instruments (ATSR-2 on board ERS-2, 1995–2003, and AATSR on board ENVISAT,
2002–2012) for the retrieval of aerosol properties. The ATSRs are dual-view instruments, with one view near-nadir and the other one at a 55
Over land, the ADV uses the two ATSR views simultaneously to eliminate the
contribution of land surface reflectance to the TOA radiation (Veefkind et
al., 1998; Kolmonen et al., 2016). AOD retrieval is based on the assumption
that the ratio of the surface reflectance for the nadir and forward ATSR
view (
Cloud detection in ADV/ASV is done with the AATSR Level 1B pixels with a
nominal resolution of 1 gross cloud test (T1) thin cirrus test (T2) TOA reflectance test (T3) reflectance ratio test (T4).
Example of a retrieval scene over western Europe, for the overpass
on 29 July 2008, ca. 10:00 UTC (as in Fig. 1): all L2 AOD pixels
retrieved
The gross cloud (T1) and thin cirrus (T2) tests are similar to the AATSR ESA
standard cloud tests six and seven (Birks, 2007b). ADV/ASV TOA reflectance
(T3) and reflectance ratio (T4) tests are based on the work by Koelemeijer
et al. (2001) and Robles González (2003). Robles González (2003)
also developed a method to automate the cloud-screening process. To this
end, the AATSR orbits are divided into scenes of 512
The default retrieval is performed on retrieval areas of
0.1
Example of a retrieval scene over China, for the overpass
on 18 August 2010: false-colour RGB image for nadir view
(left), L2 AOD retrieved with ADV/ASV
Example of a retrieval scene over China, for the overpass on
18 August 2010, 01:00 UTC: all L2 AOD pixels, the subdivision of the
partial track in 5
Figure 1 shows the false-colour RGB images (composite of 0.555,
0.659 and 0.865
The cloud-screening results are not smooth, since the cloud tests are
applied to scenes of 512
For the imagery provided by a dual-view instrument one should remember the
so-called stereo effects (Virtanen et al., 2014) which often occur at cloud
edges. In (A)ATSR Level 1B data the forward and nadir views are nominally
collocated at a standard surface ellipsoid corresponding to sea level. In
the case of a cloud located above a given surface pixel, the forward view might
look under the cloud when approaching the cloud edge, while the nadir view
sees a fully cloud pixel. These effects on cloud screening are illustrated
in Fig. 2, where the cloud detection by the forward, the nadir, and both
views are indicated in colour. The stereo effect is clearly visible at the
edges of the cloud over the North Sea, e.g. between 53 and
55
ADV AOD vs. AERONET AOD validation results (blue,
Example of the visual inspection of the AOD spatial
distribution for the overpass on 29th July 2008, ca.
10:00 UTC (as in Figs. 1–3) for AODstd < 0.2
The AOD retrieved with ADV/ASV for cloud-free pixels (as recognised with the four ADV/ASV cloud tests) for the same test scene used in Figs. 1 and 2 is shown in Fig. 3. Most of the retrieved pixels show AOD values of 0.1–0.4 (Fig. 3a). However, near cloud edges or over areas with scattered clouds, such as over parts of the UK and France, some clouds are missed by the ADV/ASV cloud tests, resulting in cloud-contaminated retrieval areas where the AOD is unreasonably high compared with the surrounding areas. As a result, the application of the retrieved AOD for climate or air quality studies over such areas would yield overestimated values. To avoid cloud contamination due to residual clouds, CPP method has been developed as explained in Sect. 3.
A cloud post-processing method has been developed for application to L2 AOD
data (0.555
Number of pixels left (compared to number of ADV/ASV-retrieved pixels) after the application of the existing CPP and the improved CPP for different cases, see Fig. 7. For orbit details, in ATS_TOA_1PUUPAyyyymmdd_hhmiss_000065272091_00403_nnnnn_7095.N1, yyyy is year, mm is month, d is day, h is hour, min is minute, s is seconds, nnnnn is the ENVISAT orbit number.
Example of retrieval scenes with high AOD. Upper row: over China
on 3 August 2010 (case 1) and 24 July 2010 (case 2), lower row: Saharan dust
outbreak on 12 March 2006 (case 3) and Siberian biomass burning episode on 1
August 2010 (case 4). For each of these cases we show
An example of the application of this CPP method for the test scene over
western Europe is presented in Fig. 3b. The method recognises AOD areas with
high AOD due to cloud contamination over the UK and France (Fig. 3a) and
discards them (Fig. 3b). However, when applied over areas with higher AOD
the method may also remove areas which are not cloud contaminated, as
illustrated for eastern China (Fig. 4). The false-colour RGB image for the
nadir view (Fig. 4, left) shows that most of the AATSR track is cloud-free
over that area. Figure 4a shows the AOD retrieved using ADV/ASV before
application of the ExCPP method. Figure 4b shows that the cloud-contaminated
pixels south of 25
Globally, with this existing CPP method, about 15 % of the pixels are
discarded as possibly cloud-contaminated. Validation of the remaining AOD
with AERONET data (see Sect. 4.2 for more details about the AERONET) shows
the improvement of the ADV-retrieved AOD with respect to those before CPP,
i.e. the correlation coefficient before CPP is
The reason the ExCPP method fails for high-AOD regions is that in those
areas the AOD spatial variation in the 3
Scatter plots of AOD retrieved with ADV vs. AERONET AOD for
all pixels retrieved (left column), for pixels left after application of the
existing CPP (ExCPP, middle column) and for pixels left after applying the
improved CPP (ImCPP, right column) for the whole world (upper row), Europe
(middle row) and eastern China (lower row) for the period of 2002–2012. The
black broken line is the identity line;
The test for recognising high-AOD regions is applied to parts of AATSR tracks
extending 5
Global aggregated AOD retrieved with ADV/ASV for the
period 2003–2011 after application of the ExCPP
For the example over China (Figs. 4, 5a), the cumulative AOD distribution
functions for each area are shown in Fig. 5b. Using the thresholds discussed
above, the areas 1 and 2, which contained less than 40 % of low
(< 0.6) AOD pixels (23 and 34 % respectively) were therefore
classified as high-AOD areas for which all ADV/ASV-retrieved pixels are
accepted. Hence, the high-AOD values are all retained. The areas 3 and 4 are
classified as low-AOD regions, and Npix and AODstd tests are applied. The
results of the application of the ExCPP (with standard deviation 0.1) versus
the ImCPP tests (including the increased standard deviation to 0.2, see
AODstd threshold correction in Sect. 3.2.2) are illustrated in Fig. 3 for an
area over Europe with a relatively low AOD and in Fig. 4 for an area with a
high AOD over China. Over Europe (Fig. 3), the high-AOD region detection gave
negative results (not detected). However, the improved CPP results in an
increase of valid pixels as a result of the higher AODstd threshold. With
ImCPP (Fig. 3c), more pixels are retained over the UK
(
The effectiveness of the
improved CPP method depends on the threshold set in the procedure to detect
high-AOD regions. The thresholds used above (0.6 AOD, 40 % of pixels) performed well in
95 % of the cases. For the other 5 % of the cases, mostly for small
(less than 1000 km
As was discussed in Sect. 5.1, the implementation of the thresholds for the
number of pixels (Npix) and AOD standard deviation (AODstd) in 3
We examined the Npix and AODstd thresholds combination that gives the optimum
combination of AOD validation results and coverage. The ADV/AERONET AOD
correlation coefficient and the percentage of pixels accepted from the number
of all pixels retrieved with ADV/ASV were plotted together with the resulted
AOD yearly mean values obtained with ExCPP (dashed line) and ImCPP (solid
line) for various Npix and AODstd combinations (Fig. 6). Global results are
presented in Fig. 6a and those over China in Fig. 6b. The results clearly
show the effect of retaining high-AOD regions on both the coverage and the
AOD values. With the AODstd
AOD over China for the period 2003–2011 retrieved using ADV and
after application of the ExCPP
AATSR ADV AOD over land seasonal time series for the period 2002–2012 after application of the existing (dashed lines) and the improved CPP (solid lines) for different areas: Europe (light blue, upper panel), China (dark blue, upper panel) and Amazon (green, lower panel). Global AOD over land (black lines) is plotted on both panels, for comparison. Seasons are marked with coloured circles (see legend). Note the different AOD scales in each plot. For areas definition see Sect. 4.4.
The results in Fig. 6 show that the value of AODstd is a more selective threshold than that of Npix. Keeping Npix at 3 (see below), the increase of the AODstd threshold from 0.1 to 0.2 results in an increase in the AOD coverage (ca. 5 % more pixels are accepted globally and ca. 10 % over China) with similar validation results. The number of validation points (not shown here) does not increase much, since AERONET measurements are performed for clear sky conditions and AOD testing near the cloud edges is not clearly seen in the validation results (note that validation was not possible over China with ExCPP, when high-AOD cases were discarded). A further increase of the AODstd threshold to 0.3 results in better coverage (3 % more pixels are accepted globally) with the same AOD validation results. However, visual inspection of the results from application of AODstd threshold values of 0.2 and 0.3 to the western European test case (Fig. 7) shows that with AODstd < 0.2 most of the cloud-contaminated pixels are rejected (Fig. 7a), whereas with AODstd < 0.3 cloud-contaminated pixels are accepted in the northern edge of the AOD pattern over the North Sea, eastern England and central Europe (Fig. 7b). Therefore, the AODstd threshold of 0.2 was selected as an optimum.
A decrease of Npix from three to two, which potentially gives better coverage, has not been admitted, since it resulted in accepting more cloud contaminated pixels near the cloud edges (visual inspection, not shown here). Hence, for ImCPP, the ExCPP Npix threshold (Npix > 3) was retained. In summary, taking into account three main criteria, such as validation results, coverage and visual inspection, the combination of Npix > 3 and AODstd < 0.2 thresholds have been chosen for cloud contamination detection in low-AOD conditions globally.
To demonstrate how the ImCPP performs in different environments, we show (a) the AOD retrieved with ADV/ASV and AOD after post-processing using (b) the existing and (c) improved CPP schemes in Fig. 8 for anthropogenic emissions (case 1 and 2), Saharan dust outbreaks (case 3) and biomass burning in Russia (case 4). For the orbit details, see Table 1. Figure 8 (3a–b, 4a–b) shows that in most of the cases the ADV/ASV cloud detection tests (see Sect. 4) do not screen the high-AOD areas. However, many areas with high AOD were removed by the ExCPP (Figs. 8, 1a–b, 2a–b, Table 1). For the anthropogenic pollution cases, the number of the pixels left after the ExCPP, compared to those originally retrieved with ADV/ASV, was 71.6 and 55.8 % for the cases 1 and 2 respectively (Figs. 8, 1b and 2b). As discussed more extensively for the test cases in Figs. 3 and 4, the ExCPP removes most of the high-AOD areas. After the application of the ImCPP the AOD coverage is considerably higher with 93.4 and 89.5 % for the anthropogenic pollution cases (Figs. 8, 1c and 2c) as well as for the dust (82.3 %, Figs. 8, 3c) and the biomass burning (96.1 %, Figs. 8, 4c) events. Thus, the high-AOD area detection (Sect. 4.2) followed by CPP for the low-AOD areas effectively removes cloud-contaminated areas.
The effect of post-processing on the resulting AOD was evaluated by comparison with independent reference values available from ground-based sun photometer measurements from AERONET (Holben et al., 1998). The AERONET sun photometers, deployed at several hundred locations globally, measure solar irradiance at multiple wavelengths to provide AOD with an uncertainty of 0.01–0.02 (Eck et al., 1999). The cloud-screening algorithm developed for the AERONET has been comprehensively tested on experimental data obtained in different geographical and optical conditions (Smirnov et al., 2000). AERONET (quality assured) Level 2.0 AOD has been used for the ADV/ASV AOD validation, despite the fact that this algorithm (Smirnov et al., 2000) could have screened some high-AOD events.
Comparisons of the ADV/ASV-retrieved AOD and AOD results after application of
either ExCPP or ImCPP with AERONET Level 2 AOD for the period 2002–2012 are
shown in Fig. 9 globally, over Europe and over China. For each case, we
report numbers of collocated ADV/AERONET pairs used in the validation (
CPP has “cleaned” ADV/ASV AOD in two ways. Pixels with high ADV/ASV-retrieved AOD which could be cloud contaminated have been removed. However, as explained above, the ExCPP also inadvertently removes part of the “good” high-AOD pixels. With the ImCPP, which is less strict, about 15–20 % more pixels (from 73 to 91 % globally, from 75 to 92 % for Europe and from 69 to 93 % for eastern China) have been accepted for validation as cloud-free pixels. Here, the ImCPP shows slightly better values compared to the ExCPP, correlation with the AERONET AOD globally (0.86 vs. 0.84) and slightly worse correlation for China (0.89 vs. 0.91), while the correlation is similar (0.81) for Europe with both CPP schemes. Thus, with the ImCPP scheme we obtain better AOD coverage with better global quality.
The binned AOD mean and standard deviation are also indicated in Fig. 9. The bin approach clearly shows in which AOD range the retrieval results are in good agreement with the reference AOD provided by the AERONET. The averaged magenta circles (Fig. 9) show a good agreement between the ADV-retrieved and AERONET AOD for AOD < 1. For AOD > 1, ADV AOD is biased (overestimation in the AOD range 1.5–2 globally, as example), which partly might be explained by the sparse observations. However, for most of the bins with AOD > 1, the AOD error bar is within the standard deviations.
The improvement in the CPP (as the increase in AOD coverage) results in
changes in the spatial and temporal distribution of AOD (Fig. 10). With the
ImCPP, high-AOD episodes are recognised more often and thus are not screened
out as cloud (compare Fig. 8b and c), contributing to the averaged AOD
values. Over land, the yearly AOD value averaged over the whole period of
2003–2011 is higher by
As an example, the spatial distribution of the AOD over China for the period 2003–2011 yearly and for each season is presented in Fig. 11, as calculated with the ExCPP (Fig. 11a) and with the ImCPP (Fig. 11b). With the ImCPP, the multi-annual mean AOD over China becomes 0.480 compared to 0.386 with the ExCPP. Seasonal changes in the AOD values after application of the two CPP methods are evident. The highest correction to the seasonal AOD over China occurs in the summer, when the mean AOD is 0.641, compared to the previous value of 0.467 (AOD value increase is 0.17, or ca. 37 %). In the winter, when the AOD retrieval is limited by low solar zenith angle and snow on the ground, changes in AOD related to the ImCPP are not significant. Differences due to the improvements in CPP method are evident, especially for China, but the global values are also affected.
As discussed above, the ImCPP results in higher AOD values and obviously
these are reflected in the time series. The seasonal AOD time series over
land over different areas (Fig. 12) retrieved from the AATSR data using ADV
have been presented and discussed in Kolmonen et al. (2016) where the ExCPP
was applied. In the current section, we present the AATSR ADV AOD seasonal
time series over land averaged globally, as well as over selected areas:
polluted China (25–40
In areas with relatively low AOD, such as Europe (Fig. 12a), which are not usually affected by high-AOD episodes, the difference between the seasonal AOD values calculated with the existing and the improved CPP is negligible. China (Fig. 11a) and India (not shown here), which are among the most polluted countries in Asia, contribute substantially to the higher AOD. In China, the application of the improved CPP results in an increase of the AOD value by 0.22 in the summer of 2006, which is 51 % of the value obtained with the existing CPP.
In other regions, the high-AOD episodes have seasonal behaviour. Thus, as a result of the application of the improved CPP, a strong change in the AOD values is observed during the biomass burning seasons due to the retention of the high-AOD events, as we expected. In the Amazon (Fig. 12b), the AOD values are higher by 30–40 % in September–November. In Africa (not shown here), an AOD increase by 20–30 % is observed in March–August. In India, during the monsoon season, the application of the ImCPP results in an increase of the AOD value by 0.26 in the summer of 2008, which is 82 % of the value obtained with the ExCPP.
The existing CPP scheme (Kolmonen et al., 2016) applied to AATSR-retrieved AOD using ADV/ASV resulted in the inadvertent loss of valid pixels, especially over areas with high AOD. Therefore, the scheme has been modified such that high-AOD areas are recognised and excluded from the post-processing. In addition, the post-processing selection criteria have been adjusted.
The main difference between the existing and improved CPP schemes is the
increase in AOD coverage (globally, 10–15 % more of the retrieved pixels
are accepted with the ImCPP) and improved comparison with the AERONET AOD.
This comparison shows that 91 % of the ADV/ASV-retrieved points are
accepted with the ImCPP, giving a better correlation coefficient
After the application of the ImCPP the AOD values for the period of 2003–2011 are higher by 0.019 with respect to the old scheme (0.163 vs. 0.144) over land, by 0.012 (0.164 vs. 0.152) over ocean and by 0.014 (0.164 vs. 0.150) globally. However, the strongest effect was on areas with a generally high AOD such as those with strong anthropogenic pollution and those affected by desert dust transport or biomass burning (e.g. China, India, Africa, South America). In the summer, the average AOD for the period 2003–2011 over China was higher by 0.174 (0.641 vs. 0.467). Likewise, over India the summer 2008 AOD value increased by almost a factor of 2 (from 0.32 to 0.58).
While having a considerable improvement in both the AOD coverage and quality,
the improved CPP method has its limitations related mostly to the threshold
in high-AOD regions. The limiting value of more than 60 % of pixels
retrieved with AOD > 0.6 has been chosen for the detection of
high-AOD regions after examining ca. 150 high-AOD cases (ca. 500 tested
areas) and this worked well in 95 % of them. For the other 5 % of the
cases, mostly for small (less than 1000 km
AATSR ADV aerosol data are available on ICARE (
The authors declare that they have no conflict of interest.
The research presented in this paper was carried out in the framework of the Marco Polo project (EU FP7 SPACE Grant agreement no. 606953), ESA-ESRIN project AO/1-6207/09/I-LG (Aerosol_cci), with further support by the Centre of Excellence in Atmospheric Science funded by the Finnish Academy of Sciences Excellence (project no. 272041).
We thank the editor Marloes Penning de Vries for her valuable comments and suggestions, which considerably improved the manuscript. Edited by: M. Penning de Vries Reviewed by: three anonymous referees