The Greenhouse Gases Observing Satellite (GOSAT) was launched in 2009 to measure global atmospheric CO2 and CH4 concentrations. Preparations are continuing for the launch of its successor, GOSAT-2, in the fiscal year 2018. The mission objectives of GOSAT-2 are as follows: to continue and improve the satellite measurements of major greenhouse gases performed by GOSAT, to monitor the effects of climate change and human activities on the carbon cycle, and to contribute to climate science and climate change related policies (NIES GOSAT-2 Project, 2014). These policies include Reducing Emissions from Deforestation and Forest Degradation and the role of conservation; sustainable management of forests and enhancement of forest carbon stocks in developing countries (REDD+); and the Joint Crediting Mechanism (JCM), which was proposed by the Japanese government to facilitate the diffusion of leading low-carbon technologies, products, systems, services, and infrastructure in developing countries (Ministry of the Environment, Japan, 2015). Monthly regional CO2 fluxes are estimated from the column-averaged dry-air mole fractions of CO2 (XCO2) retrieved from spectral observations made by GOSAT (Maksyutov et al., 2013). The results are publicly available as the L4A CO2 product (Maksyutov et al., 2014). The expected role of the CO2 fluxes estimated from the GOSAT data is the system for measurement, reporting and verification (MRV) of CO2 fluxes estimated from forest inventory data. Currently, the uncertainty of the L4A CO2 product is about 0.9 Gt-C region-1 yr-1 in the Amazon (L4A CO2 product V02.03 in region ln 09-12, 2009–2012). Thus, the total net CO2 flux from deforestation for the period 2000–2010 in tropical America was estimated to be 0.56 Gt-C yr-1 (Baccini et al., 2012). It is required to reduce the uncertainty of the L4A CO2 product by a factor of 16, assuming that the MRV for REDD+ and JCM needs an accuracy of 10 %.

GOSAT is equipped with two sensors: the Thermal And Near infrared Sensor for carbon Observations (TANSO)-Fourier transform spectrometer (FTS) and TANSO-Cloud and Aerosol Imager (CAI) (Table 1). The presence of clouds in the instantaneous field of view of the FTS leads to incorrect estimates of greenhouse gas concentrations (Uchino et al., 2012). To solve this problem, the FTS data suspected to have cloud contamination must be identified by the Cloud and Aerosol Unbiased Decision Intellectual Algorithm used with CAI (CLAUDIA1–CAI) (Ishida and Nakajima, 2009) and rejected. The cloud information is publicly available as the CAI L2 cloud flag product. However, CAI does not have a thermal infrared band. In general, cirrus cloud is identified by using multiple thermal infrared bands, which include water vapor absorption bands (Ishida et al., 2011a). Meanwhile, the FTS has a 2 µm band that contains many strong water vapor absorption bands. Moreover, the CAI L2 cloud flag product may not be sensitive enough to detect clouds of subpixel size in ocean observations. To cope with these difficulties, the FTS data suspected to have cloud contamination are identified by two additional tests: the 2 µm band test and the CAI coherent test (Yoshida et al., 2010). Conversely, overestimation of clouds reduces the amount of the FTS data that can be used to estimate greenhouse gas concentrations. This is a serious problem in tropical rainforest regions, such as the Amazon, where there is a small amount of suitable FTS data (approximately 3 % of the number of observations) because of cloud cover (Figs. 1 and 2). For this reason we need to optimize thresholds between cloudy and clear sky because there are tradeoffs in maximizing cloud detection accuracy while minimizing false detection. To solve the problem, a new cloud discrimination algorithm (CLAUDIA3) using a support vector machine (SVM) (Vapnik and Lerner, 1963) was developed (Ishida et al., 2018). CLAUDIA3 can automatically identify the optimized thresholds using clear-sky training data, although CLAUDIA1 requires setting various thresholds by radiative transfer calculation results and fine tuning in some methods. Verification was also performed by comparing it with the MODIS cloud mask algorithm (Ackerman et al., 2010) and ceilometer data provided by the Atmospheric Radiation Measurement Climate Research Facility (Mather and Voyles, 2013) in Ishida et al. (2018). Furthermore the impact of different support vector generation procedures on cloud discrimination using CLAUDIA3 has also been evaluated in a previous study (Oishi et al., 2017).

The accuracy of CLAUDIA1–CAI was evaluated by comparing it with the MODIS/Aqua cloud mask data product (MYD35) (Ackerman et al., 2010) because the MODIS cloud mask algorithm uses a larger number of bands for cloud discrimination than CLAUDIA1–CAI, and CLAUDIA1 was developed based on the MODIS cloud mask algorithm (Taylor et al., 2012; Ishida et al, 2011b). However, these comparisons cannot identify common weak points in the algorithms and another verification method is required. Although the use of visual inspection of clouds as a standard is not practical for screening a full satellite data set, it has the advantage of allowing for locally optimized thresholds, while CLAUDIA1 and -3 use common global thresholds. Thus, the accuracy of visual inspection is better than that of these algorithms in most regions, with the exception of snow- and ice-covered surfaces, where there is not enough spectral contrast to distinguish cloud. In other words, visual inspection results can be used as truth data for accuracy evaluation of CLAUDIA1 and -3. For this reason visual inspection can be used as the truth metric for the verification exercise. Therefore, the accuracy of CLAUDIA1–CAI has also been evaluated by visual inspection in tropical rainforests (Oishi et al., 2014). In this study, we deal with the application of the CLAUDIA3 to GOSAT CAI data. Then, we compare CLAUDIA1–CAI and CLAUDIA3–CAI for various land cover types and evaluate their accuracy by comparing both against visual inspection (400 × 400 pixels) of the same CAI images in tropical rainforests.

In this study, “accuracy” is defined as the ratio of the number of pixels for which the standard image and output from the cloud discrimination algorithm agree with the total number of pixels in the input image. “Overlook” is defined as the ratio of the number of pixels judged clear in the output and cloudy in the standard image to the number of pixels that were judged cloudy in the standard image. “Overestimate” is defined as the ratio of the number of pixels judged cloudy in the output and clear in the standard image to the number of pixels judged clear in the standard image. These definitions are written as follows. Accuracy=Bothcloudy+BothclearTotalnumberofpixels, Overlook=CleardespitecloudyBothcloudy+cleardespitecloudy,Overestimate=CloudydespiteclearBothclear+cloudydespiteclear.

Comparative results for CLAUDIA1–CAI and CLAUDIA3–CAI for various land cover types indicated that CLAUDIA3–CAI had a tendency to identify bright surface and optically thin clouds; however, CLAUDIA3–CAI had a tendency to misjudge the edges of clouds compared with CLAUDIA1–CAI. There are tradeoffs in maximizing accuracy while minimizing overlook and overestimate. Thus, it is sufficient to change the integrated-CCL threshold according to the purpose. Furthermore, CLAUDIA3–CAI misjudged vegetation areas as clouds in Japan. It is necessary to add clear training data of Japanese vegetation areas for CLAUDIA3.

The averaged accuracy of CLAUDIA3 used with GOSAT CAI data (CLAUDIA3–CAI) was approximately 89.5 % in tropical rainforests, which was greater than that of CLAUDIA1–CAI (85.9 %) for the test cases presented here. This is mainly because, in contrast to CLAUDIA1–CAI, CLAUDIA3–CAI can detect optically thin clouds and the edges of clouds, which prevents cloud-contaminated FTS-2 data from being processed as cloud-free FTS-2 data in the greenhouse gas concentration calculations. However, CLAUDIA3–CAI tends to overestimate the surroundings of clouds, which are judged to be cloudy despite being clear. Thus, CLAUDIA3–CAI is not expected to increase the amount of the FTS-2 data that can be used to estimate greenhouse gas concentrations in tropical rainforests. Conversely, CLAUDIA3–CAI may be able to detect optically thin clouds that cannot be detected by visual inspection.

CLAUDIA3–CAI misjudged muddy rivers and boundaries between land and water as cloudy in the same manner as CLAUDIA1–CAI. This has three possible causes: (1) insufficient training data on muddy rivers so the differences in the spectral reflectance properties of muddy water and other water cannot be distinguished; (2) deviation of the positions in each CAI band owing to the band-to-band registration error; and (3) insufficient resolution of the surface albedo data. The surface albedo data were generated at 1/8∘ resolution by separating the land and water regions. If the border pixels between land and water regions were mixed pixels, the albedo data of 1/8∘ areas that include the mixed pixels would be included. To decrease this effect, higher-resolution surface albedo data are needed. For boundaries between land and water, the resolution of surface albedo data is being investigated because it may be the main problem: the misjudged regions and grid pattern of albedo data match. CLAUDIA3–CAI is more sensitive to differences between land and water than CLAUDIA1–CAI because there is a large difference in the structure of support vectors between land and water. However, generating higher-resolution surface albedo data from CAI L1B data for 10 recurrent cycles cannot completely remove clouds in the minimum reflectance calculation. To solve this, initially we need to confirm whether 500 m resolution albedo data should be used. If necessary, we will develop a new method for generating surface albedo data. For example, simple cloud discrimination could be added to calculate the minimum reflectance, and if it is a cloud-contaminated pixel then the pixel is replaced by a minimum reflectance pixel, which is calculated from the same month over several years.

Although we used MODIS data as training images to generate support vectors in this study, the MODIS data and CAI data depend on observation conditions. In future work, we will use CAI data as training images to perform cloud discrimination for CAI data. Furthermore, we will verify CLAUDIA3–CAI by using global CAI data with an alternative method. For instance, it can be compared with satellite lidar data, such as CALIPSO, because it is impossible to perform a visual inspection of global data and visual inspection is also itself not perfect. Addressing these points will make CLAUDIA3–CAI more reliable for GOSAT-2 CAI-2 cloud discrimination.