Rainfall data for 2010 to 2014 were obtained from the South African Weather Service (SAWS). The data were recorded at 229 automatic rainfall stations and 91 automatic weather stations (Fig. ). They were complemented by 22 stations from SASSCAL WeatherNet (www.sasscalweathernet.org/) located in southern Namibia and Botswana. For 2014, data from an additional 15 stations in South Africa operated by the IDESSA project (An Integrative Decision Support System for Sustainable Rangeland Management in southern African Savannas, www.idessa.org/) were available. The data passed general provider-dependent quality checks before they were used in this study. This includes filtering data beyond common data ranges or carrying out situational checks for consistency with related parameters (e.g. air humidity) from SASSCAL. SAWS payed attention to rainfall values > 10 mm within 5 min and deleted those values if they were unreliable. Data from all providers were then included in an on-demand processing database system where they were automatically cross-checked for reliability by filtering values < 0 and > 500 mm of rainfall per hour. All station data that provided subhourly information were aggregated to a temporal resolution of 1 h within the database. Though the station data are not randomly distributed in the model domain, they cover the entire aridity gradient, from sites with very low (< 200 mm) precipitation to sites in areas with highest (∼ 1500 mm) yearly precipitation sums.

MSG SEVIRI scans the full disk every 15 min with a spatial resolution of 3×3 km at subsatellite point (3.5×3.5 km in southern Africa). Reflected and emitted radiances are measured by 12 channels, three channels at visible (VIS) and very near-infrared wavelengths (NIR, between 0.6 and 1.6 µm), eight channels ranging from near-infrared to thermal infrared wavelengths (IR, between 3.9 and 14 µm) and one high-resolution VIS channel with a spatial resolution of 1×1 km, which was not considered in this study.

The rainfall retrieval technique presented here works under the assumption that VIS, NIR and IR channels of MSG SEVIRI provide proxies for microphysical cloud properties, which are, in turn, related to rainfall. VIS and NIR channels have been shown to be related to cloud optical depth and cloud water path where the NIR channel is further related to cloud particle size . The IR channels have been shown to provide information about the cloud top temperature which was used as a proxy for cloud height . The cloud droplet effective radius as well as liquid water path at night was approximated using IR differences .

MSG SEVIRI Level 1.5 data were preprocessed to radiance values according to and brightness temperatures according to using a processing scheme based on a custom raster processing extension of the eXtensible and fleXible Java library (see https://github.com/umr-dbs/xxl) which enables parallel raster processing on CPUs and GPUs using OpenCL.

A cloud mask was used to exclude all pixels that were not cloudy in the respective SEVIRI scenes. For 2010 to 2012, the CM SAF CMa Cloudmask product was applied. Due to the availability of the CM SAF CMa cloud mask data set, which was currently limited to the years 2004 to 2012, we used the cloud mask information of the CLAAS-2 data record for the years 2013 and 2014, which is the second edition of the SEVIRI-based cloud property data record provided by the EUMETSAT Satellite Application Facility on Climate Monitoring (CM SAF; see also for further information on CLAAS). All pixels that were classified as cloud contaminated or cloud filled were interpreted as cloudy. Pixels that were classified as cloud-free were excluded from further analysis.

The modelling methodology follows the study of who used the spectral channels of MSG SEVIRI to train a random forest model that is able to spatially estimate rainfall areas and rainfall rates over Germany. Based on this study, have shown that neural networks outperform the initially used random forest algorithm. In these previous studies on the rainfall retrieval, the radar-based RADOLAN product was used as ground truths to train the model. The high data quality and spatially explicit information allowed the model to be optimised without too much confusion caused by uncertainties in the training data. However, the goal of the retrieval was that it can be applied to areas where spatially explicit data for rainfall are not available, as it is the case in southern Africa.

Cloud masked MSG data from 2010 to 2014 were extracted at the locations of the weather stations. To match the temporal resolution of all available rain gauge data, the extracted data were aggregated to hourly values. This was done by taking the median value of the four scenes available every hour. However, only if all four scenes were masked as cloudy, the corresponding hourly values for a respective station were used for further analysis. The extracted and aggregated MSG data were then matched with the corresponding rain gauge information under consideration of the time shift between MSG data (UTC) and rain gauge data (UTC+2).

The spectral channels as well as the channel differences ΔT6.2–10.8, ΔT7.3–12.1, ΔT8.7–10.8, ΔT10.8–12.1, ΔT3.9–7.3, ΔT3.9–10.8 and the sun zenith were used as predictor variables during the daytime, in accordance to and previous studies on MSG-based delineation of cloud properties (see Sect. ). Thus, the predictor variables contain the SEVIRI channels as well as channel combinations. Although this partially duplicates information, the channel combinations allow for highlighting patterns that might not be apparent in the individual channels. As additional potential predictors, tested different cloud texture parameters and have shown that the chosen spectral channels and differences are sufficient as predictors.

Since neural networks require that the predictor variables are standardised, all predictors were centred and scaled by dividing the values of the mean-centred variables by their standard deviations. Since the VIS and NIR channels of MSG are not available at night-time, the data set was split into a daytime data set (data points with a solar zenith angle < 70∘) and a night-time data set (data points with a solar zenith angle > 70∘) and were considered in separate models. Though two different models might lead to rough transitions between daytime and night-time estimates, accurate estimates were in the foreground of this study, leading to the decision of separate models according to data availability. The response variables (rainfall yes/no and rainfall quantities) were taken from the rain gauge measurements.

The years 2010 to 2012 were used for model training. The year 2013 was used for validation. The retrieval process was in two steps and consisted of (i) the identification of precipitating cloud areas and (ii)  the assignment of rainfall quantities. All 2010 to 2012 data from the rain gauges that are masked as cloudy by the cloud mask products were used for training the rainfall area model. All recorded rainfall events were used for training the rainfall quantities model. The resulting training data set comprised 917 774 (daytime) and 1 409 072 (night-time) samples for the rainfall area training and 69 703 (daytime) and 129 325 (night-time) samples for training of rainfall quantities from 26 243 individual MSG scenes.

A single-hidden-layer feed-forward neural network was applied as a machine learning algorithm. The spectral channels of MSG SEVIRI as well as the channel differences served as input nodes (predictor variables). The neural network was then applied to learn the relations between these spectral information and rainfall areas or rainfall quantities, respectively. In this context, a sophisticated preselection of input variables is not required, as the network is able to deal with correlated and even uninformative predictors unless their number is very high , which was not the case in this study. For the technical realisation, all steps of the model training were performed using the R environment for statistical computing . The neural network implementation from the “nnet” package in R was used in conjunction with the “caret” package , which provides enhanced functionalities for model training, estimation and validation.

Neural networks require two hyperparameters to be tuned to avoid under- or overfitting of the data: the number of neurons in the hidden layer, as well as the weight decay. The neurons in the hidden layer represent non-linear combinations of the input data and their number influences the performance of the model . Weight decay penalises large weights and controls the generalisation of the outcome . The number of neurons as well as the weight decay were tuned using a stratified 10-fold cross-validation. Thus, the training samples were randomly partitioned into 10 equally sized folds with respect to the distribution of the response variable (i.e. raining cloud pixels, rainfall rate). Thus, every fold is a subset (1/10) of the training samples and has the same distribution of the response variable as the total set of training samples. Models were then fitted by repeatedly leaving out one of the folds. The performance of a model was then determined by making predictions on the held back fold. The performance metrics from the held back iterations were averaged to the overall model performance for the respective set of tuning values. For the rainfall areas classification models, the distance to a “perfect” model, based on receiver operating characteristics (ROC) analysis (see for its application in rainfall retrievals) was used as decisive performance metric. For the rainfall quantity regression models, the root mean square error (RMSE) was used. The number of hidden units were tuned for each value between two and the number of predictor variables. Weight decay was tuned between 0 and 0.1 with increments of 0.02 . To train rainfall areas, the threshold that separates rainy from non-rainy clouds according to the estimated probabilities was used as an additional tuning parameter. The optimal threshold was expected to be considerably smaller than 0.5 since the number of non-rainy samples was higher than the number of rainy samples. Therefore, the range of tested thresholds was 0 to 0.1 with increments of 0.01, and 0.4 to 1 with increments of 0.1. See for further details of the threshold tuning methodology.

The optimal values for the hyperparameters that were revealed in the tuning study (Table ) were adopted for the final model fitting. In this step, the model is fit to all training data using the optimal hyperparameters.

Final models were applied to all hourly MSG SEVIRI scenes from 2010–2014 for the southern African extent to obtain spatio-temporal estimates of rainfall. Therefore, the clouded areas of a scene were first classified into rainy or non-rainy using the respective model. The rainfall quantities were then estimated for the estimated rainfall areas. To ensure consistency within one scene, the choice of the model being applied (either the daytime or night-time model) was made according to the mean solar zenith angle of the respective scene. If the mean solar zenith angle was < 70∘, rainfall for the entire scene was estimated using the daytime model. For scenes with a mean solar zenith angle > 70∘, the night-time model was applied.