Remote Sensing of solar surface radiation - A reﬂection of concepts, applications and input data based on experience with the effective cloud albedo

. Accurate solar surface irradiance data (SSI) is a prerequisite for efﬁcient planning and operation of solar energy systems. Respective data are also essential for climate monitoring and analysis. Satellite-based SSI has grown in importance over the last few decades. However, a retrieval method is needed to relate the measured radiances at the satellite to the solar surface irradiance. In a widespread classical approach, these radiances are used directly to derive the effective cloud albedo (CAL) as basis for the estimation of the solar surface irradiance. This approach has been already introduced and discussed in the early 5 1980s. Various approaches are brieﬂy discussed and analyzed, including an overview of open questions and opportunities for improvement. Special emphasis is placed on the reﬂection of fundamental physical laws and atmospheric measurement techniques. In addition, atmospheric input data and key applications are brieﬂy discussed. It is concluded that the well established observational-based CAL approach is still an excellent choice for the retrieval of the cloud transmission. The coupling with Look-Up-Table based clear sky models enables the estimation of solar surface irradiance with high accuracy and homogeneity. 10 This could explain why, despite its age, the direct CAL approach is still used by key players in energy meteorology and the climate community. For the clear sky input data it is recommended to use ECMWF forecast and reanalysis data.

An outstanding socio-economic application area is energy meteorology, which is linked to a huge commercial market associated with extensive and many-sided user demands on solar radiation data. The goal of energy meteorology is to provide 30 specific information for efficient and sustainable use of renewables. Climatological data of solar surface irradiance are necessary for an efficient planning and monitoring of solar energy systems, e.g. the estimation of energy yield for Photovoltaic (PV) (Huld et al., 2012), monitoring and analysis of operational PV-systems (Hammer et al., 2003) and PV Performance Studies Huld and Amillo, 2015). Near real time data and forecasts of solar surface irradiance are needed for the efficient integration of solar energy into the electric grid. Solar energy is fluctuating, hence forecasts are needed to optimise the 35 levelling of conventional power plants (e.g. increase/decrease of power supply) and storage systems, to minimise the need for spontaneous trading of electricity on an in-balanced market (Hammer et al., 2003(Hammer et al., , 2015Urbich et al., 2019) and to avoid grid overloads. Supported by the European Commission (EC) goal to aim for a secure, independent, sustainable and environmental friendly energy supply (EC1, 1996;EC2, 2013) the renewable energies contribute nowadays significantly to the energy production. The share of PV of the total electricity sold in 2018 was 7.1 % in Italy (Jäger-Waldau, 2019) and 9.3 % 2020 Germany 40 in 2020 (Wirth, 2021). Even in less sunny regions (e.g. Denmark, UK) the share is of about 3 %. PV electricity generation capacity accounted for 39 % of the new installed capacity in 2018 (Europe), but worldwide Asia and the Pacific region had the highest share of new installed PV power capacity in 2018. Nevertheless, also the United States had reached a cumulative PV capacity of almost 62.7 GW by the end of 2018 (Jäger-Waldau, 2019). Almost all players predict a further increase of PV. As a consequence, the demand for accurate climatological data, near real time data and forecasts of SSI data are expected to further 45 increase.
For all mentioned applications accurate SSI data with a large geographical coverage and a high spatial and temporal resolution is either needed or of great benefit. SSI data gained from satellite observations meet these requirements and are therefore widely used in many application areas. There are typically two types of satellite systems available for the retrieval of SSI.
Geostationary weather satellites with a fixed location above the equator and polar orbiting satellites that fly in low orbit around 50 the Earth. flying in low orbit around the Earth. Geostationary satellites provide images in high spatial and temporal resolution, but are blind near the poles. This gap can be filled by polar orbiting satellites or by data from Numerical Weather Prediction (NWP). Geostationary satellites like METEOSAT (METEOrological SATellite, Schmetz et al. (2002), GOES (Geostationary Operational Environmental Satellite GOES (2019)) or HIMAWARI (Sunflower, JMA (2017)) provide satellite data with a spatial resolution of about 500m to 3 km (sub-satellite point) and a temporal resolution of about a few minutes to 15 minutes, 55 depending on the mode of operation and spectral channel. E.g. the visible 600 nm channel of HIMAWARI has a spatial resolution of 500 nm and a temporal resolution of 10 minutes, with even higher temporal resolutions in rapid scan mode. These satellites cover the complete geostationary ring and are quite useful for the estimation of solar irradiance at any point outside the polar region.
The atmosphere modifies the amount and distribution of the TOA solar irradiance by scattering and absorption. All retrieval methods rely on the same basic equations, which is based on the law of energy conservation, expressed here as (1) T is hereby the Transmittance, which defines the proportion of radiation which passes through a medium. R is the proportion 130 which is reflected and A the proportion that is absorbed by the medium, respectively. The sum of transmitted radiation plus the absorbed radiation plus the reflected radiation is 1. This means radiation that is not absorbed or scattered back by the atmosphere reaches the surface, leading to the following basic equation, see Müller (2012b) for further details.
with 135 SSI clear = T clear * SSI ext * cos(SZA) SSI is the solar surface irradiance, SSI clear is the clear sky irradiance, T cloud is the cloud transmission, SSI ext is the extraterrestrial irradiance, often approximated as solar constant and cos(sza) is the cosine of the solar zenith angle. Various algorithms and methods have been developed to estimate broadband surface solar radiation data sets based on equation 2 e.g., (Möser and Raschke, 1984;Cano et al., 1986;Bishop and Rossow, 1991;Pinker and Laszlo, 1992;Pinker et al., 1995;Darnell 140 et al., 1992;Rigollier et al., 2004;Mueller et al., 2009;Dürr and Zelenka, 2009;Posselt et al., 2011b;Müller et al., 2015). In the following the individual components of equation 3 are discussed concerning applied methods and needed input information.
Please note that all quantities in the above equations are fluxes and not radiances.

Extraterrestrial irradiance
The extraterrestrial irradiance SSI ext results from the radiation emmitted by the sun and the distance between Earth and sun 145 and is given by equation 4.
Here, SSI ext,const is the so called solar constant, a long term mean of the extraterrestrial irradiance for a mean distance between the Earth and the Sun (d cor = 1). Nowadays accurate values of SSI ext can be gained from satellite measurements (Kato et al., 2013;Harries et al., 2005). The University Corporation for Atmospheric Research (UCAR) proposes a value of 150 1361 W/m 2 for SSI ext,const (UCARteam).
SSI ext is valid for the whole globe and results from the nearly constant flux of energy emitted by the sun (Carroll and Ostlie, 2017). However, the term solar constant is somehow misleading, as there are variations in the solar irradiance at the top of atmosphere due to sun activity. The changes of the solar irradiance due to variations in sun activity are usually within the range of 2-3 W/m 2 and are therefore often neglected (Müller, 2012b).

155
In addition, variations in extraterrestrial irradiance are caused by the distance between Earth and Sun. Induced by the elliptical orbit of the Earth around the sun, the Earth-Sun distance varies sinusoidal throughout the year in the order of +/-3.3 %, leading to an respective increase and decrease of solar irradiance at the top of atmosphere. This effect is considered in equation 4 by the factor d cor and can be gained from standard astronomical equations (Iqbal, 1983).

Solar zenith angle 160
The Earth is approximately a sphere, hence, close to the equator the incoming solar irradiance covers a smaller region than at higher latitudes. Here, the same amount of incoming photons are spread over a larger region, hence the solar irradiance is lower per unit area. In mathematical terms this effect is expressed by the cosine of the solar zenith angle. The solar altitude is defined as 90 degree minus the solar zenith angle. As can be seen in equation 3, the solar zenith angle is a dominant factor for solar surface irradiance. The local noon solar zenith angle and its diurnal and seasonal variation depend on the latitude, 165 longitude and time of the year. These variations originate from the rotation of the Earth around it´s axis in combination with the rotation of the Earth around the Sun on an elliptical sphere. The Earth's rotation axis is tilted at 23.5 degree with respect to the ecliptic axis. This tilt is called the obliquity of Earth's axis and is responsible for winter and summertime. The higher the latitude the lower the local noon solar zenith angle, which decreases from summertime to wintertime. The sun earth geometry is well defined and the SZA can be calculated with standard astronomical equations (Iqbal, 1983) using latitude, longitude and 170 time as input. The calculation of the SZA is therefore straightforward.   (Pincus and Evans, 2009). An exemplary application of SHDOM within the scope of remote sensing of SSI is discussed in (Girodo, 2003;Girodo et al., 2006). However, concerning satellites an explicit usage of RTM for each pixel and time step would be too slow for the generation of long time series (climate data records) or near real time applications. Thus, instead of explicit RTM runs pre-calculated look-up-tables (LUTs) are usually used. A Look Up Table (LUT) contains the pre-calculated results of radiative 180 transfer calculations performed for many different states of the atmosphere, surface albedo, and solar zenith angles. Figure 1 illustrates the principle of the LUT approach.
Nevertheless, classical LUT approaches have still a significant disadvantage. First, the computation speed is still hampered by the need to interpolate within large multi-dimensional arrays. Second, and more importantly, many RTMs must be performed to generate the LUTs, resulting in large and cumbersome binary matrices that raise the recalculation threshold. However 185 recalcuation , whiis necessary from time to time to take advantage of new developments. Further, big and cumbersome LUTs increase also the chance for hidden bugs. E.g. after 4 years of development the CM SAF classical LUTs had a bug , probably induced by a switch in the indices somehow. Thus, 4 years of development were more or less worthless. The LUTs (clear sky, cloudy sky) contained over 100 000 RTM calculations each. As a consequence of the mentioned handicaps and the burden of a recalculation of classical LUTs a novel approach was developed, referred to as the "eigenvector hybrid" LUT approach 190 ) . This method takes benefit of the symmetries of atmospheric absorption processes and separates them from the linear independent processes of aerosol scattering. A further reduction is achieved by the use of the Modified Lambert Function (Mueller et al., 2004;Ineichen, 2008), which requires only two SZA points (0 and 60 degree) to cover the complete  (Mueller et al., , 2012a. The RTM calculations were performed with libRadtran 195 (Mayer and Kylling, 2005) using the DISORT solver (Stammes et al., 1988). The respective retrieval procedure enables also the calculation of direct irradiance, resulting from a adaptation of the Skartveit et al. (1998) model for diffuse irradiance.
Overall, this work implements several of the requirements discussed in  for the improvement of solar surface irradiance retrieval, RTMs are not only needed for the calculation of LUTs but are also essential for the development of parameterizations and a better understanding of the processes in the Earth atmosphere.

Clear sky transmittance
The clear sky irradiance can be derived by multiplication of the extraterrestrial irradiance SSI ext with the cosine of the solar zenith angle and the clear sky transmittance T clear The term clear sky radiation is commonly used, but nevertheless misleading. It is the radiation received in cloud free skies, 205 which can still be quite turbid (low visibilities, high aerosol load). The incoming irradiance is scattered and absorbed by gases and particles in the atmosphere. Scattering and absorption lead not only to attenuation of light but also modify the spectra of the extraterrestrial irradiance. Scattering leads to a continuous modification of the spectra whereas absorption occurs in specific spectral bands, the so called absorption bands. In these bands the absorption is strong and the radiation is significantly reduced. E.g. O 3 is a strong absorber in the UltraViolet (UV), hence almost all of the UV-A radiation reaching the top of 210 atmosphere is absorbed in the atmosphere, which is great luck for us, because it causes skin cancer. Further, water vapour has strong absorption bands in the InfraRed (IR), but also in the microwave, thus acting as strong greenhouse gas. The absorption bands result from the transition of radiation into periodic vibrations of molecules or atoms (IR) or change in electrons state (UV, VIS), whereby the energy of the radiation is used to reach a higher energy level. The effect of the atmospheric scattering is apparent every day. The blue sky results from Rayleigh scattering at air molecules (particles smaller than wavelength λ).

215
The air molecules scatter the incoming irradiance with a wavelength dependency of λ 4 . This means that the scattering effect decreases in the power of 4 with increasing wavelength. As a result, blue light is scattered and reflected much more by the sky than other colours leading to a blue illumination of the sky (blue sky). In contrast, cloud droplets and aerosols are leading to Mie scattering, which has a wavelength dependency of λ −1 (particles are about the same size as wavelength and assumed to be spheres). Thus there is only a small dependency on the wavelength leading to grey sky. Scattering forces the light to deviate 220 from the direct path and is induced by the interaction of the photons with the respective media as a consequence of oscillating electric fields or by consideration of the wave-particle dualism of quantum physics as particle to particle collisions. More details can be found in Jackson (1998). The great majority of air molecules (N 2 , O 2 , CO 2 , methane, noble and inert gases) are well mixed and uniformly distributed and do not affect the spatial and temporal distribution of solar surface irradiance. Thus, their effect is rather static in space and time and they pose no problem for the estimation of SSI as a static transmission can 225 be assumed (Müller, 2012b). In contrast, ozone, water vapour and tropospheric aerosols are not well mixed. Their distribution is driven by atmospheric dynamics. They show significant temporal and spatial gradients and patterns on different scales.
Thus, the spatial and temporal variation of clear sky irradiance is dominated by these parameters. Yet, the effect of ozone on the broadband solar irradiance is rather small and is therefore neglected in some methods. Sources of information for these variables are discussed in detail in section 7.3. Figure 2 shows the irradiance at the top of atmosphere and the surface on an 230 inclined plane at 37 degree tilt (toward the equator) for a US standard atmospere and aerosol load and a SZA of 48.18 degree.
The respective data was downloaded from NREL (NREL2, 1998), where more details on the atmsopheric components are given.
Different methods are applied to consider the effects of the clear sky components on the solar surface irradiance. A set of models use semi-empirical functions to derive the clear sky irradiance, e.g. Beyer et al., 1996;Cano 235 et al., 1986;Rigollier et al., 2004;Bird and Hulstrom, 1981;Sengupta and Peter, 2003;Perez et al., 2002). More complex methods are based on look up tables derived from radiative transfer modelling e.g. (Gupta et al., 2001;Mueller et al., 2004Mueller et al., , 2009 or some kind of hybrids of the two. However, as a satellite retrieval of SSI clear is not needed for a proper and accurate estimation of SSI (Mueller et al., 2012a(Mueller et al., , 2004Müller et al., 2015), the futher discussion of the methods focus on the retrieval of the cloud transmittance. The main processes leading to the attenuation of irradiance are shown in the associated spectral range.

Cloud transmission
Overall, clouds are the dominant atmospheric component for the spatial and temporal variation of solar surface irradiance.
Thus, accurate information of the cloud transmittance is crucial for the accurate retrieval of solar surface irradiance. Satellites measure the reflection of the surface and atmosphere. In order to derive the cloud transmission the reflection of the surface and clear sky atmosphere has to be separated from that of the clouds. The scattering effect of clouds is by far dominant and 245 absorption can be neglected in good approximation. Thus, radiation that is not reflected by clouds is transmitted trough the clouds. Therefore cloud transmission equals in good approximation one minus cloud reflection, see equation 1.
Here, T cloud is the cloud transmission and R cloud the cloud reflection. It is important to mention here that the above equation is valid for fluxes. Satellites however, are observing radiances, i.e. photons reaching the satellite from a specific direction, see 250 figure 3 for illustration. The reflection of the Earth's atmosphere is usually not isotropic (equal share of radiation in all directions) but depends on the specific direction and viewing angle. Therefore angular distribution models (ADM), which results from Bidirectional Reflectance Distribution Function (BRDF) have to be applied to transfer radiances to fluxes (Nicodemus et al., 1977) . See Loeb et al. (2005) for further details about BRDF and ADM.
Further, the upward flux depends on the solar zenith angle as the Surface Albedo (SAL) changes with changing SZA 255 (Briegleb and Ramanathan, 1982). Further, the retrieval of SAL is characterised by severe uncertainties (Shuai et al., 2020;Carrer et al., 2010Carrer et al., , 2018Fell et al., 2021). This adds further difficulties to the accurate definition of ADMs (BRDFs) as the problem is somehow ill-posed. Is the change in the fluxes induced by a different viewing geometry for the same SAL or by the change in SAL induced by SZA, change in vegetation, different pixel size, calibration issues (ageing of channels, change of spectral response function), change of satellite instruments, and others. BRDF and ADM are also spectrally dependent, hence 260 differ from wavelength band to wavelength band. Thus, they can not be easily transferred from one satellite instrument to another. This poses the question which path is the best to follow, because there are mainly two different paths to get the cloud transmission. We follow the suggestion of Skartveit and Olseth to name them direct and indirect paths. For the indirect path fluxes and thus ADMs and accurate SAL information are needed. For the direct path this information is not needed but implicitly considered by the satellite observations within the retrieval of the cloud transmission. The two paths are briefly discussed 265 in the following sections.

The indirect paths: Methods using flux quantities
The cloud transmission can be defined by the micro-physical parameters cloud optical depth (COD) and the effective radii (r ef f ) of cloud droplets. Thus, one way forward is to retrieve cloud optical depth and effective radii, following an approach discussed in Nakajima and King (1990). Once COD and r ef f are retrieved the cloud transmission can then be calculated  cloud physical properties retrieved with a LUT approach for the estimation of surface solar irradiance. The respective retrieval of COD and r ef f was developed by the Royal Netherlands Meteorological Institute (KNMI) and is applied to retrieve cloud optical thickness, liquid water path and particle size from satellite imagers within the CM-SAF framework (Roebeling et al., 2005). A similar approach was used by ISCCP Zhang et al., 1995) for the generation of CDRs.

275
As mentioned in the method section, several approaches for the retrieval of solar surface irradiance are using the cloud optical depth and the effective radii for the estimation of the cloud transmission. However, for the retrieval of COD and r ef f accurate surface albedo (surface reflection) and ADMBRDF simulations are needed in order to separate the reflection by clouds from that of the surface and to transfer radiances to fluxes. Further, the clouds are usually assumed to be plane-parallel, which is a rough assumption. Finally, the problem is somehow ill-posed as the retrieval of COD depends on r ef f and vise versa, and on 280 the ratio between ice and water particles in mixed phase clouds, see Nakajima and King (1990) for details. However this ratio is usually unknown. For some of these effects a quantitative analysis is given by Kato et al. (2006) Another indirect approach takes benefit of the linear relation between Transmission (T) and the Top of Atmosphere (TOA) albedo, as discussed in (Li et al., 1993). For a given surface albedo and clear sky state the cloud transmission changes linearly with increasing COD. Thus, assuming a standard plane-parallel cloud, the relation between transmission and TOA albedo can 285 be pre-computed for a variety of CODs, surface albedos and clear sky parameters. The precomputed RTM results are saved in a Look-Up- Table (LUT). The observed TOA albedo is then used as variable to retrieve the cloud transmission by interpolation within the LUT. This approach has been the basis for the first operational version referred to as prototype of the CM-SAF scheme, described and validated in Mueller et al. (2009) and Hollmann et al. (2006). This prototype followed the concepts and ideas of Pinker and Laszlo (1992) and Pinker et al. (1995) . Also, the OSI SAF retrieval procedure for SSI is using TOA Albedo as input (Marsouin, 2019;Gautier et al., 1980). However, also in this approach clouds are assumed to be plane-parallel, ADMsBRDFs are needed to convert radiances to fluxes and accurate SAL data are necessary.
In brief, using the indirect approach means to be further away from the observations, to introduce further error sources by weak assumptions and to use simulations instead of observations directly. All in all, there are additional variables and functions needed, which are partly based on ill-posed retrievals.

4 The direct path
For the indirect paths (flux based methods) accurate SAL and ADMsBRDFs are needed. Further the clouds are assumed to be plane-parallel. These induces uncertainties into the estimation of the cloud transmission and motivates the use of direct path.
The direct path relates the measured reflection (radiances) directly to the cloud transmission without the need of any radiative transfer modelling, ADM or SAL, as they are implicitly considered by observations. In the following the method applied at 300 CM SAF and DWD is described in more detail. It is based on the concepts of (Möser, 1983;Möser and Raschke, 1984;Cano et al., 1986;Diekmann et al., 1988) and originates from the Heliosat code developed at the University of Oldenburg (Beyer et al., 1996;Hammer, 2000;Hammer et al., 2003). Only observed radiances are needed in order to retrieve the Effective Cloud Albedo (Mueller et al., 2011), which is also referred to as cloud index or effective cloud coverage. The effective cloud albedo (CAL) is defined as the normalised relation between the all sky and clear sky reflection derived from satellite images in the 305 solar spectral range. CAL is linearly related to the cloud transmission for CAL values lower than 0.8. A clear sky model is used afterwards to calculate the solar surface irradiance based on the retrieved cloud transmission according to equation 2. This approach is still the basis for several state of the art retrieval schemes, described in more detail in (Dürr and Zelenka, 2009;Posselt et al., 2011b;Mueller et al., 2012a;Müller et al., 2015;Castelli et al., 2014;Pfeifroth et al., 2019b).
In order to derive the effective cloud albedo it is important to consider the effect of the solar zenith angle on the extraterrestrial 310 irradiance. Furthermore, the dark offset of the instrument has to be subtracted from the observed reflections. Thus, the observed reflections are normalised by application of equation 7 in a first processing step.
Here, D is the observed digital count (including the dark offset). D 0 is the dark offset, the instruments value in the dark, which is provided by EUMETSAT. The sun-earth distance variation is considered by the factor f , finally, the cosine of the solar 315 zenith angle corrects the different illumination conditions at the top of atmosphere introduced by different solar altitudes. The resulting ρ is referred to as normalised reflection. The effective cloud albedo is then derived from the normalised reflection ρ, the clear sky reflection ρ cs and the calibration value ρ cal by equation 8 Here, ρ is the observed normalised reflection for each pixel and time. ρ cs is the clear sky reflection, which is derived for 320 every pixel and time slot separately. In principle this is done by applying statistics over the lowest reflections observed during a typical time span of 3-4 weeks, e.g the average of the lowest reflections within an epsilon band, which can be defined by the user and is typically of about 10 % of ρ cal . Also percentile functions can be used, but based on tests at DWD this is expected to lead to a lower performance. Further details on the method to derive ρ cs are given in Hammer (2000); Müller et al. (2015) and Mueller and Trentmann (2015). The need to calculate ρ cs for every time slot of the day results from the strong dependency of the clear sky albedo (which is in turn usually dominated by the surface albedo) on the solar zenith angle (Briegleb and Ramanathan, 1982). ρ cal is a measure for the reflections observed for the highest cloud albedos. It is therefore often also referred to the "maximum" reflection ρ max . It is typically determined by the 95-98 percentile of all reflection values at local noon in a target region, characterised by high frequency of cloud occurrence for each month (Mueller et al., 2012a;Müller et al., 2015). In this manner changes in the reflections induced by the satellite are accounted for (Posselt et al., 2011a). Further,
For CAL values below 0.8 the relation between CAL and T cloud is simply given by Thus, For very thick clouds SSI does not decrease linearly any more with CAL as a result of lateral scattered light and a modification of the relation 10 is needed, e.g. as following.
The term Heliosat is widely used in the energy meteorology community for methods using the relation given in 8. Yet, this naming is somehow misleading. In some Heliosat methods the observational approach for ρ cs is partly replaced, e.g Heliosat-2

345
(2021)). In order to avoid misunderstandings we introduce the term CALSAT for the described method and use it instead of Heliosat throughout the manuscript. The CALSAT approach is applied routinely in real time at the University of Oldenburg since 1995. It was used to establish the server Satel-Light, which delivers valuable information on daylight in buildings to architects and other stakeholders (Fontoynont et al., 1997). The method was also used to provide the data for successful projects like PVSAT and PVSAT-2 (Lorenz et al., 2004;Hammer et al., 2003) which went into 24/7 operational services. All CM SAF respective data sets belong to the most used and validated data sets in the world covering a wide field of applications, as can be seen from recent publications (Wild et al., 2021;Kulesza, 2021;Alexandri et al., 2021;Gardner et al., 2021b, a;Farahat et al., 2021;Yang and Gueymard, 2021;Kulesza, 2021;Kulesza and Bojanowski, 2021;Fountoulakis et al., 2021;Drücke et al., 355 2021; Daggash and Mac Dowell, 2021). CAL based SSI data are characterised by a high accuracy and homogeneity, which is a reason for the popularity, even up to quality control of ground based stations (Urraca et al., 2017(Urraca et al., , 2020. The CM SAF SSI data are regurarly validated by the CM SAF team e.g. Pfeifroth et al. (2019) but also by the user community, e.g. (Wang et al., 2018;Amillo et al., 2018;Urraca et al., 2007;Trolliet et al., 2018;Riihelä et al., 2015;Posselt et al., 2011b;Ineichen et al., 2009)  Finally, the CALSAT method is also used at DWD to generate 24/7 operational near real time data and forecasts, which are available at opendata link. In summary, the direct approach using the observable CAL to define the cloud transmission ist quite well established and is used and for the creation of radiation data sets that are widely used around the world, e.g. Selbach and Thies (2021). It enables the retrieval of monthly means with an accuracy of 5.2 W/m 2 and 7.7 W/m 2 for SSI and direct irradiance, respectively (Pfeifroth et al., 2019). Hereby, the mean absolute difference is used as metric for the accuracy. It is 370 defined as the average of the absolute values of the differences between the ground based measurements and the satellite based irradiance first over all months and then over all stations. For the validation all available BSRN stations were used.
The uncertainty in CALSAT SSI trends is with -0.8±0.4 W/m 2 /decades quite small, which shows the adequate homogeneity of the data set and its ability to identify climate trends and decadal variability.

375
Already the early CALSAT methods had significant impact. e.g. the method developed by (Möser and Raschke, 1984;Diekmann et al., 1988) was routinely applied at the German Weather Service till it was replaced by a modified SPECMAGIC (Mueller et al., 2012a) version in 2018 A similar method was also used for the GEWEX Surface Radiation Budget Project as a quality control algorithm (Gupta et al., 2001;Darnell et al., 1992). This algorithm known as Staylor algorithm was chosen by the World Climate Research Program (WCRP) SRB project for generation of surface solar irradiance for a test period of 380 several years (Whitlock et al., 1995) and has been used to produce long term data set.

Spectral resolved irradiance
The demand for spectrally resolved irradiance has increased significantly in recent years due to new solar cell technologies and increased awareness of climate change. The importance of the spectrally resolved irradiance (SRI) for the efficient planning of the climate system and enables the generation of essential meteorological variables for several application fields, e.g.
-Daylight, which is of importance for the design and planning of office buildings and health studies, e.g., concerning Seasonal Affective Disorder.
-Ultraviolet (UV)-A and UV-B, important for weather warnings concerning the UV dose -Photosynthetic Active Radiation (PAR), important for agro-meteorology.

390
However, retrieval algorithms for the generation of accurate long term series of spectral resolved irradiance covering the complete solar spectrum were not available or applied to generate CDRs until 2012. The development of SPECMAGIC (Mueller et al., 2012a), a fast and accurate method for the retrieval of spectral resolved irradiance, was a game changer. It has been used to improve the quality of the broadband radiation of the CM SAF CDRs, but also to provide the first spectral resolved solar irradiance climate data record.

395
The SPECMAGIC method is a adaption of the concept for the broadband radiation ) to Kato bands (Kato et al., 1999) using the correlated-k approach offered as part of libradtran (Mayer and Kylling, 2005). The correlated-k method is developed to compute the spectral transmittance (hence the spectral fluxes) based on grouping of gaseous absorption coefficients. However, in contrast to Mueller et al. (2009) the cloud transmission is derived with the CAL approach in SPECMAGIC (Mueller et al., 2012a) and only the clear sky irradiance is calculated with an eigenvector LUT. The CALSAT 400 method was developed for broadband radiation. Thus, a spectral correction of the broadband transmission was needed, which was done by a RTM based spectral correction factor for the broadband CAL. A detailed description of the methods is given in Mueller et al. (2012a). Further validations were presented in Amillo et al. (2015). As ground measurements of spectral resolved irradiance are extremely rare, satellite derived information constitutes the primary data source for spectral resolved irradiance within the scope of climate monitoring and solar energy applications. The validation of the spectral resolved irradiance climate 405 data record (SRI) in Ispra (Italy) shows an accuracy of the monthly mean values of spectral resolved data higher than 0.03 W/m 2 /nm for wavelengths below 1000 nm and even better for larger wavelengths (Pfeifroth et al., 2019).
Over recent decades the share of solar energy has increased enormously around the world, as briefly discussed in section 1. In contrast to fossil and atomic energy, solar resources are fluctuating and the energy yield depends on the weather. Variations in 410 solar radiation, and thus in solar energy, have to be balanced, either by increase or decrease of the production from conventional power plants or by trading energy on the electricity market. The shorter the time for this compensation, the higher the costs and the lower the efficiency of renewables. In addition, wrong forecasts of extreme weather events might also lead to grid instabilities and regional overloads. Accurate forecasts are therefore needed to enable an efficient, planing and operation of the energy production and to optimise the use of renewable energies and the reduction of CO2 emissions. Beside forecasts, 415 instantaneous near real time data are also essential for the optimisations and monitoring of solar energy systems. As a result, the demand for accurate near real time data and forecasts of solar surface irradiance has steadily increased. Satellite based cloud information is well suited for the estimation of near real time radiation data and the basis for short term forecasts, which outperforms NWP in the first hours (Lorenz et al., 2017;Urbich et al., 2019).
The satellite based short term forecast of SSI, also referred to as nowcasting, is usually done by retrieval of atmospheric mo- To our knowledge, one of the first applications in meteorology has been the utilisation of the optical flow for radar images as described by Peura and Hohti (2004). At DWD the TV-L1 method (Zach et al., 2007) was implemented in 2018 for the short-term forecast of radar reflectivity by Manuell Werner. This has motivated the adaptation of TV-L1 to CAL images (Urbich et al., 2018) in order to track the cloud movements and to extrapolate the CAL values with optical flow. In order to met the requirements of the operational near real time retrieval of CAL based on SPECMAGIC (Mueller et al., 2012a) has 435 been modified. The modified version is referred to as SPECMAGIC_NOW. The Visible Channel in the 600 nm band is used instead of broadband visible channel, which is gained from a combination of VIS006 and VIS008 . Further, a running backward statistics are performed to calculate ρ cs . In contrast to SPECMAGIC, only 20 days are used for the statistics.
Also the estimation of ρ cal has been modified. Please see section 7 for further details. The near real time data and forecasts up to +18 hours are available at the DWD open data server (opendata link).
The combination of the effective cloud albedo with TV-L 1 and SPECMAGIC_NOW delivers a novel powerful method for the short-term forecast of solar surface irradiance, as discussed in Urbich et al. (2019). Yet, for the satellite based nowcasting 2 subsequent images are needed. Before sunrise no former image in the visible is available for the estimation of the atmospheric motion vectors. Thus, the University of Oldenburg extended the Heliosat method to Infrared Images Hammer et al. (2015). For longer forecast horizons a blending with NWP models is needed (Urbich et al., 2020;Lorenz et al., 2017).

445
However, forecasting SSI is not only needed within the scope of solar energy, but is also useful for weather warnings concerning UV radiation, heat waves, droughts and evaporation.
7 Recent developments and discussion

Method
Copernicus "hoping to return the science to a less cumbersome, more elegant view of the Universe suggested a heliocentric 450 (Sun-centred) model of planetary motion" (Carroll and Ostlie, 2017) instead of the geocentric system, which was the prevailing view until then. His approach led to a much simpler description between planets and stars.
This paradigm should still be a central key for natural science. Make things as easy as possible and as complex as needed. It is always a goal of science to improve methods. However, established ways should not be forgotten by this legitimate goal. It should not be expected a priori that more complex and more cumbersome methods lead to a higher accuracy. Science demands  (Betcke et al., 2006) was to supply high-quality solar radiation data gained from the exploitation of existing Earth observation technologies by taking advantage of the enhanced capabilities of the 460 new Meteosat Second Generation Satellites. Therefore one aim was to develop a method to replace the CALSAT method by a method using cloud micro-physical parameters. During the Heliosat-3 project, the meteorologists Skartveit and Olseth reminded us not to forget the advantages of the the direct path, using CAL derived from satellite measurements. Lessons learned within Helisoat-3, CAL is still used and was not replaced by an approach using COD and r ef f at the University of Oldenburg, the leading entity of the project. Further, within CM SAF the LUT approach used in the early phase for the generation 465 of the first data sets  has been replaced by the CALSAT approach concerning cloud transmission for the generation of CDRs (Posselt et al., 2011b;Müller et al., 2015) and the generation of near real time data and forecasts (Urbich et al., 2018). For clear sky cases RTM based LUTs are still used in the CALSAT approach, beside other arguments, in order to overcome the limitation of turbidity as input (Mueller et al., 2004) .
Thus, the CALSAT approach has been not replaced by flux based methods at leading institutes and is to the knowledge 470 of the authors still the best way forward for the retrieval of cloud transmission. The approach results from the law of energy conversation and relates the observed reflection directly to the effective cloud albedo and the cloud transmission without the need for external SAL data or ADM simulations. In this terms the CALSAT method takes full benefit of the satellite measurements in the visible (Mueller et al., 2011). Both, the data sets of the University of Oldenburg, CM SAF and other entities using CAL are among the most validated data sets of the world. The validation results around the SAF radiation data 475 show that the improved CALSAT methods are in the meanwhile able to derive accurate SSI data close to the accuracy of well maintained ground based stations (Posselt et al., 2011b;Müller et al., 2015;Pfeifroth et al., 2019). The quality is such good that CAL based SSI data can be used for monitoring the quality of ground based measurements (Urraca et al., 2017(Urraca et al., , 2020 and for climate trend analysis (Pfeifroth et al., 2018). The success and impact of the CALSAT approach is impressive. Nevertheless, further improvements of CALSAT have been performed and are still under progress. Yet, many of these improvements are not 480 CALSAT specific, but represent general issues of satellite based retrieval of SSI. The following sections provide a discussion of limitations and possible improvements that are to the best knowledge of the authors.

Mountains
The pixel size of the visible MSG full disk channels (VIS006 and VIS008) is with approximately 3-5 km over the European mountains relative coarse. This leads to pixels which covers mountains and valleys and are therefore no longer spatially repre-485 sentative of the area covered. The terrain is too heterogeneous and the spatial representatives is largely limited by the resolution of the VIS006 and VIS008 channels. However, over Europe the High Resolution Visible (HRV) channel is available. It has a sub-satellite resolution of 1x1km, which means of about 1x2 km over the European mountains. The spatial representativity is significantly improved by the higher spatial resolution and is a central key for the improvement of the precision and accuracy of SSI in mountainous regions (Dürr et al., 2010;Dürr and Zelenka, 2009). tion. This correction is independent of the retrieval method for SSI. A respective correction as well as further mountain specific features has been implemented into ths CALSAT approach by Dürr and Zelenka (2009). The work of Dürr and Zelenka (2009) has been the basis for the Heliomont algorithm (Stöckli, 2013;Castelli et al., 2014), which comes with further improvements.
Further, snow covered surfaces occur with much higher frequency in mountains than in low lands. Thus, also the treatment of clouds over snow needs more attention in the mountains. Dürr and Zelenka (2009) implemented a correction for the CAL 500 retrieval over snow and demonstrated that it leads to higher accuracy.

Long Lasting clouds
All methods need cloud free pixel in order to get actual information of the surface albedo (indirect flux based methods) or the clear sky reflection ρ cs . Outdated information of SAL or clear sky reflection can lead to significant errors in SSI. The clear sky reflection and the surface albedo can only be derived from satellites in cloud free situations. Thus, long lasting clouds are 505 a serious error source for the satellite based retrieval of SSI. This is valid for all retrieval methods. Long lasting clouds can contaminate the observation of ρ cs . The minimisation of ρ leads only to the clear sky reflection if at least one pixel is cloud free. However, especially in winter at higher latitudes (e.g. northwestern Europe) it can happen that there is no cloudless sky for 20-30 days, which leads to cloud contamination and an overestimation of rho cs and, in turn, to an underestimation of CAL.

510
An option to reduce the uncertainties induced by long lasting cloud might be the use of simulated of ρ cs based on RTMs or climatological values for ρ cs as a backup. The former option has been implemented in SPECMAGIC_NOW, a SPECMAGIC modification dedicated and used for the 24/7 production of near real time data at DWD. However, the method for the decision when the backup is used is still in development and needs further evaluation. Another option is to fit the diurnal variation of ρ cs and to replace outliers by the fitting function, see Heliomont for details (Castelli et al., 2014). This approach requires 515 cloudfree skies for neighbouring slots in time and thus can only mitigate but not resolve the problem of long lasting clouds.
However, it has also to be considered that long lasting clouds occur typically during wintertime at higher latitude and are therefore associated with low absolute SSI values and hence also low absolute errors. Even with a bakup for ρ cs , CAL would be still derived directly from observations for the vast majority of cases.

Improvement of clear sky reflection, ρ cs method 520
Usually the broadband visible channels are used for the CALSAT method, either directly or as a combination of the visible channels in the spectral bands 600 nm (VIS006) and 800 nm (VIS008) according to Cros et al. (2006). However, the precision of the ρ cs detection might be improved by using the VIS006 channel as the reflection in this band is rather dark for vegetation.
This enhances the contrast between clear sky and cloudy reflection and enables a higher precision for optical thin clouds. A comparison of using different VIS channels was performed by Posselt et al. (2011a). This study showed the potential of using 525 the VIS006 channel and motivated its use for SPECMAGIC_NOW. Experiments to improve the accuracy by using percentiles instead of the original statistical method (Hammer, 2000) did not lead to an improvement. Hence, the same method is used for the calculation of ρ cs in SPECMACIC (Mueller et al., 2012a) and SPECMAGIC_NOW.
Miss-classification of snow covered surfaces as clouds or clouds as snow is a serious error source for all methods. This is also valid for the CALSAT method, but with some specificity resulting from the ρ cs method. To illustrate, let there be a certain 530 number of snow free and snow covered days within the 20-30 day period used for the retrieval of ρ cs , As ρ cs is calculated with a minimisation function ρ cs will capture always the snow free situation. For the snow covered days the snow will be treated as a thick cloud resulting in a large underestimation of SSI. This effect can be reduced if ρ cs is calculated with a Fuzzy logic approach, discussed in detail in Posselt et al. (2011b). However, as experienced by the CM SAF the Fuzzy logic approach induced other artefacts, e.g. wrong ρ cs values in the desert. There might be another option to treat this problem. An accurate 535 snow map can be used to flag snow covered regions and to correct ρ cs by simulated values if ρ cs shows too low values for snow.
Of course this approach would be a rough workaround which needs further optimisation and adjustments based on validation studies. A world wide reference for snow cover maps are provided by NIC NOAA, referred to as the IMS snow ice map (Helfrich et al., 2018). This map is based on several satellite information including microwave and ground based information and therefore not so prone to long lasting clouds. However, still melting periods of snow are sometimes not captured by IMS 540 as a result of long lasting clouds as discussed in the snow workshop (Helmert et al., 2018).
In SPECMAGIC_NOW the duration used for the slot-wise detection of ρ cs has been reduced from 30 to 20 days. This is expected to reduce the effect of snow miss-classification as cloud, but might be a handicap for long lasting clouds. However, for Germany the operational comparison with other data sets (DWD in-situ network, SPECMAGIC) have not shown any significant drawbacks concerning long lasting clouds so far.  function in a first step. Then the 95 or 98 percentile is used as ρ cal . This is done for a target region with a high cloud coverage (Atlantic ocean, South West of Southern Africa). The advantage of this approach is that the calibration is done for the same targets (clouds) for which the transmission is retrieved. It has been shown that the method leads to stable homogeneous data sets across satellite generations (Posselt et al., 2011b;Pfeifroth et al., 2018).
This approach has been slightly modified for SPECMAGIC_NOW. First of all another target region is used, following an Comparison with the former method of CM SAF for the calculation of ρ cal indicates that this method leads to less noise. As SPECMAGIC_NOW is operated 24/7 using SEVIRI as input the method of Govaerts et al. (2004) could be applied. However, 580 concerning physics the ρ cal method described above has a significant advantage. Identical channels and targets (clouds) are used for both the calibration and the retrieval of CAL. Yet, the Govaerts et al. (2004) method uses desert targets in addition, which is expected to add uncertainties to the calibration because the spectral albedo is quite different to that of clouds. At least during the development and testing of SPECMAGIC the ρ max calculation could not be improved by the use of additional desert targets. 585

Parallax correction
The geolocation of the satellite is always related to the Earth surface (sea level). This means that independent on cloud height the cloud is assumed to be at the surface. The higher the clouds and latitudes the larger is the displacement of the geolocation. functions. This approach works well for the Infrared channels. In the IR the satellite receives mainly the emission from the cloud top if the clouds are not semi-transparent, which can be assumed for a COD larger than 10. Thus, CTH can be used for 595 the needed altitude information for the parallax correction. However, the diffuse part of SSI is reflected throughout the cloud and the height needed for the parallax correction can not be assumed to be close to the top of the cloud. Therefore, the parallax correction should only applied to the direct radiation. The application of a parallax correction for direct irradiance was already postulated in 1996 by Beyer et al. (1996). A parallax correction has been implemented in the HelioMont method (Stöckli, 2013;Castelli et al., 2014), but the reduction in the error measures was marginal compared to other more dominant effects 600 (Reto Stöckli, personal communication). One reason might be that for COD larger than about 10 direct irradiance is zero. This means that diffuse radiation usually dominates SSI in cloudy situations and CTH is not well suited for the correction. Thus, the optimal definition of the cloud height for the parallax correction needs further investigations, in particular empirical studies and RTM calculations for different cloud scenes. After all, the parallax effect is not a weakness of a specific retrieval method, but results from the viewing geometry. Close to the equator, everything is fine.

Slant column correction
The geostationary orbit leads to slant column viewing geometries at higher latitudes. As a consequence the "cross section" of the clouds increases significantly, see Figure 6 for illustration. Thus, on average the cloud coverage is artificially enhanced at larger satellite zenith angles leading to an underestimation of surface solar radiation. The validation study of Riihelä et al.  et al., 2019c). The correction term for the slant geometry effect is approximated as a function of the satellite zenith angle θ sat : The correction term is afterwards applied for CAL * θ sat /1.3 < 0.55 and CAL > 0.04 This results in an improved performance of the retrieval, especially in higher latitudes (Pfeifroth et al., 2019). Yet, further empirical studies and RTM calculations are needed to improve the correction of the slant geometry effect. Again this effect is relevant for all methods. The cloud cross section increases also with increasing SZA, but here also the reflection increases as cloud optical thickness is increasing, but the slant viewing geometry of the satellite leads to an artificially increased cloud cover that does not match the ground observations.

Deep learning -neural networks
Overall, the use of artificial intelligence methods, in particular deep learning (machine learning) with neural networks, is also increasing in meteorology. This includes also the estimation of solar surface irradiance from satellites (Senkal, 2009;Yeom et al., 2019;Cornejo-Bueno et al., 2019;Dewitte et al., 2021).
However, a drawback of neural networks is the black box character and the need for retraining for new regions and satellite 625 systems. A neural network is, strictly speaking, only valid for the training framework, as only the behaviour of the training data sets can be reproduced. An application to other regions, periods or satellite instruments typically requires extensive re-training.
That is a reason why scientists started to develop spectral transfer functions to ease the adaption of the network to other satellite it is a complex regression and the term intelligence is somehow misleading. For the retrieval of solar surface irradiance the physics is well defined. Thus, it is not obvious what advantage NN could provide compared to physical approaches. However, a good application of neural networks (NN) might be the replacement of LUTs. This has be done for the retrieval of SIS (e.g. Takenaka et al. (2011)). In this study an algorithm for estimating solar radiation from space using a neural network (NN) to approximate radiative transfer calculations were developed. Similar approaches are also used for other atmospheric variables, 635 e.g. for volcanic ash (Bugliaro et al., 2021). However, with respect to SSI, the burden of cumbersome LUTs, which require hundreds of thousands of RTM calculations, is eliminated by using the hybrid eigenvector LUT concept ).
These LUTs are relatively easy to calculate and to manage. This might reduce the value of NN as replacement of LUTs within the scope of SSI. Thus, the authors believe that there is no urgent need to apply neural networks for the retrieval of solar surface radiation,in particular if the CAL approach is used for the retrieval of the cloud transmission.

The clear sky input data
3 SAFs are engaged in the retrieval of SSI. EUMETSAT triggered an intensive discussion and interaction between the SAFs in order to exploit synergies. As a consequence, a comparison of the SAF SSI data (Ineichen et al., 2009) was done and several SAF radiation workshops took place. It was discussed that all SAFs use established methods for the retrieval of the cloud transmission, but that there is a need for action on the clear sky input data. Therefore, the clear sky input data is briefly 645 discussed in the following paragraphs.

Ozone:
O 3 is a strong absorber and can be well detected from satellites. Data in good quality can be gained from from GOME / SCIAMACHY (Burrows and Chance, 1992) , TOMS (Fleig et al., 1986) and the Ozone Measuring Instrument (Ziemke et al., 2011) and the respective follow-up satellites. There is no need to use data in high temporal resolution as the spatial and 650 temproal variation of SSI is dominated by clouds in the meso-scale. Thus, observations based on polar orbiting satellites are not a handicap. Alternatively ozone data from the ECMWF models can be used (Bechtold et al., 2008). However, also climatologies can be used in good approximation for broadband SSI as a deviation of about 150 DU from the US standard atmosphere leads only to changes in SSI of about +/-5.5 W/m 2 . This is in the range of the uncertainty of solar irradiance measurements (Müller, 2012b). Water vapour (H2O g ) is a strong absorber and the total column can be well retrieved from satellites. However, H20 g can only be retrieved in cloud free skies wihout appropriate profile information. Thus, using H20 g from satellite is expected to induce a clear sky bias in H20 g . Therefore, data from NWP might be the better option as the water vapour channels are assimilated by modern state of the art NWP models (4d-VAR). Thus, this data source is strongly recommended for H20 g , e.g. CM SAF is using water vapour information from the ECMWF European Medium-Range Weather Forecasts NWP model (Dee and Uppala, 2009;Betts et al., 2003). Also OSI SAF uses data from ECMWF. The use of climatologies can lead to significant deviations for daily and hourly SSI (broadband) values and is not recommended as a primary choice.

Surface Albedo
Satellites are a good source for gridded Surface ALbedo (SAL) data. The optimum accuracy for the retrieval of SAL is of 665 about 5 % (Fell et al., 2021;Riihelä and Kallio-Myers, 2020 Shuai et al. (2020);Carrer et al. (2010Carrer et al. ( , 2018 and Fell et al. (2021). They hamper an accurate retrieval of COD, r ef f and aerosols. Surface albedo data is available from LSA-SAF COPERNICUS (Geiger et al., 2008;Carrer et al., 2010Carrer et al., , 2018 , CM-SAF (Schulz et al., 2008), COPERNICUS NOAA 675 / NESDIS (Wang et al., 2013), GEWEX (GEWEX-Quarterly) and several other sources. To the knowledge of the authors the best data source for snow coverage is the IMS snow mask Helfrich et al. (2018), however this mask is provided only in a daily resolution. Again, data from modern NWP models might be a good alternative as satellite information is assimilated. This option should be investigated in more detail.

680
Aerosol information has the highest uncertainty among the atmospheric clear sky parameter. This results from the gap of a dense ground based network and the limitations of satellite retrieval methods. The Aeronet network provide high quality data, but the station density is too low to gain an accurate gridded data set. Satellite based AOD can not fill this gap as the retrieval of aerosols (AOD and type) is ill-posed. The reflection signal is weak compared to that of SAL and clouds. Therefore, quite accurate cloud screening and accurate spectral resolved information of the surface reflection is needed. However, optical 685 thin cirrus clouds are hard to detect and the uncertainty of SAL retrieval is in the same order of magnitude as the signal from aerosols for low to medium aerosol loads (RTM study CM SAF, not published). Further, for a given AOD the reflection depends also on the composition and the shape of the aerosols (aerosol type), and visa versa. Absorbing aerosols (e.g. urban aerosols) tend to lead to a reduction of the reflection for increasing AOD whereas scattering aerosols (desert, rural, maritime... ) lead to a significant increase in the reflection . Unfortunately, the possibilities for the accurate retrieval 690 of aerosol composition and shape from operational satellites is very limited. ADMs/BRDFs are needed for the retrieval as well. Although, significant progress has been made using MODIS data (Lipponen et al., 2018), accurate estimation of aerosol information is still a great challenge. MODIS is equiped with a blue (dark) channel which favours the retrieval of aerosols as SAL is relativ dark over vegetated surfaces in this wavelength region. However, accurate calibration is a serious challenge in the blue channels due to the exposure with UV, leading to ageing of the optical devices (e.g. the diffuser). In summary, 695 satellite based aerosol data comes with serious uncertainties and the aerosol information is only provided for cloud free skies, yet, the AOD depends for hygroscopic aerosols on the relative humidity (Hess et al., 1998) and thus changes with cloudiness.
Based on the validation results and RTM studies of CM SAF, the authors are not aware of any aerosol data set that captures the temporal and spatial variation of AOD with a higher accuracy of approximately +/− 0.1. This might explain why amazingly good accuracy of SSI can be gained over Europe with a fixed visibility of about 23 km (Ineichen et al., 2009). Within this scope 700 it is important to consider that the variation in AOD is on average relative low in many regions compared to its uncertainties, and that 23 km visibility is a reasonable climatological assumption (see Aeronet measurements and Müller et al. (2016)). Thus, accurate information of the spatial and temporal aerosol distribution are needed to beat on average climatological values or fixed visibilities. As a result of the disadvantages of the individual data sources (i.e. ground based measurements, satellite based data sets and chemical transport models) a reasonable way forward might be the use of aerosol information from data 705 assimilation, the combination of numerical modelling, satellites and ground based measurements. At least for the CALSAT methods it has been shown that this approach has a great potential to provide better aerosol data sets (Mueller and Träger-Chatterjee, 2014;. Therefore, MACC reanalysis data (Inness et al., 2013) was used for the generation of the CM SAF SARAH data sets e.g. Mueller and Trentmann, 2015;Pfeifroth et al., 2019b). The MACC procedure consists of a forward model for aerosol composition and dynamics (Morcrette et al., 2009) and a data assimilation 710 procedure (Benedetti et al., 2009). The data set is available at MACC-Weblink and has been used to assess the radiative forcing by aerosols (Bellouin et al., 2013). It is recommended to consider this data set for aerosol information within the retrieval of SSI. Yet, for CAL methods, a specific feature enters the door. High fluctuations of desert aerosols (and familiar types) relative to the background, e.g. as induced by desert storms, are interpreted as clouds, because the reflection is significantly increased compared to clear sky reflection. This has to be considered and a modification of the used aerosol information might be of 715 benefit for CAL methods. An respective approach is presented and discussed in more detail in .

Turbidity
Until the Helioat-3 project Linke turbidity was usually used as input for the clear sky model applied within Heliosat methods Beyer et al., 1996;Cano et al., 1986;Rigollier et al., 2004;Perez et al., 2001) instead of separate aerosols and water vapour information. However, the use of Linke turbidity has a serious disadvantage. For the accurate 720 calculation of the ratio between diffuse and direct irradiance, information of the aerosol type and the aerosol optical depth are required (Mueller et al., 2004). An exact ratio of direct and diffuse radiation is essential for an accurate estimation of the irradiation on tilted surfaces that are typical for PV systems. The use of turbidity was therefore in contradiction to the aims of the Heliosat-3 project to support the solar energy community in its efforts for increasing the efficiency and cost-effectiveness of solar energy systems and improving the acceptability of renewables. Therefore, RTM based methods were developed (Mueller ozone separately. These concept and input information were already used in the climate community (Gupta et al. (1999)

Conclusions
Satellite-based remote sensing of solar surface irradiance (SSI) is a key component in monitoring and predicting the state of the earth's atmosphere and is also of great importance for the planing, monitoring and operation of solar energy systems.
The effective cloud albedo CAL, also reffered to as cloud index ore effective cloud fraction, can be retrieved directly from the radiances observed by the weather satellites without the need of any modelling or external informaion (Mueller et al., 2011).

745
The respective CALSAT approach was initially developed by Möser and Raschke (1984); Cano et al. (1986) andDiekmann et al. (1988). The basic relationship between observed radiances and the effectice cloud albedo (equation 8) is still the basis for state of the art retrievals and for estimating SSI with an accuracy close to that of well-maintained ground-based stations (Dürr and Zelenka, 2009;Posselt et al., 2011b;Mueller et al., 2012a;Müller et al., 2015;Castelli et al., 2014;Pfeifroth et al., 2019b).
There are good reasons why the CALSAT approach is still very prominent and widely used. It is based on the physical law 750 of energy conservation and takes full benefit of the satellite observations for the retrieval of the cloud transmission. There is no need for any ADM simulation or external information of the surface albedo or to asssume that clouds are plane-parall.
Only the observed reflections are needed to define the effective cloud albedo and thus the cloud transmission. This reduces the uncertainty induced by 3rd party information or assumptions needed to model fluxes, which in turn are necessary for the indirect methods.

755
Therefore, the authors do not see any advantage in using indirect ways, e.g. to retrieve COD and r ef f for the estimation of the cloud transmission. For the retrieval of COD and r ef f at least two visible channels are needed, which are not available for for Meteosat First Generation satellites as the MVIRI instrument is equipped with only on visible channel. This is a serious limitation for the generation of climate data records. CAL can be derived from all Meteosat satellite generations and can be easily adapted to other satellite systems. However, CALSAT methods provide only information about the cloud transmission, 760 but not about the micro-physical cloud properties. If micro-physical quantities of clouds are needed or of interest then retrieval methods based on COD and r ef f are the preliminary choice. Depending on the area of application, it can then also be useful to use COD and r ef f for the determination of the solar surface radiation in order to obtain a consistent data set.
Concepts developed within HELIOSAT-3 (Mueller et al., 2004) went eventually in the development of the methods MAGIC (Mueller et al., , 2011Posselt et al., 2011b) and SPECMAGIC (Mueller et al., 2012a). In these methods the cloud 765 transmission derived with the CALSAT approach is coupled with RTM-LUT based clear sky model. In this way, the advantages of the CALSAT approach can be fully exploited and at the same time the complete clear sky information can be used instead of only turbidity. The concept of eigenvector hybrid LUTs ) enable a significant reduction of the needed RTM calculations and is recommended therefore. SPECMAGIC was used for the generation of the first CDR of spectral resolved irradiance and is the basis of all CM SAF SARAH data sets.

770
Of course, satellite retrievals has also limitations. That is the reason why during the last decades several extensions has been done to improve the CALSAT based methods, including a specific and powerful method for mountains, the Heliomont method (Dürr and Zelenka, 2009;Stöckli, 2013). The improvements of the method cover parallax correction, slant geometry correction, snow-cloud differentiation, calibration and correction of shadowing effects. Although, significantly improvements have been achieved more studies and works are needed to gain further improvements. However, it is likely that significant increase of 775 the SSI quality can be gained from an improved accuracy of information in cloud-free skies, in particular concerning aerosol optical depth and type . For the clear sky input, due to the extension and further development of data assimilation, ECMWF data might be the best choice. Finally, the upcoming satellite generations, having a higher resolution and more spectral channels, will likely contribute to the further improvement of atmospheric input information and retrieval schemes.

780
Code and data availability. The near real time and forecast data is available at opendata.dwd.de/weather/satellite/radiation/, the CDRs at wui.cmsaf.eu, the code is available on request (GPL license). Please contact Richard.Mueller@dwd.de

Meaning of eigenvector approach
A process is called independent of other atmospheric processes if for a given deviation of an atmospheric variable (here water vapour for example) a unique scalar t H2O can be defined which fulfils the following equation.

785
RT M δH2O I 0 = t H2O · I 0 (1) For aerosols this is not the case, because no eigenvalue exists. Hence, For equal δAOD not a unique but different t aod values exist, as t depends on the value of aod (aerosol optical depth), ssa (single scattering albedo) and gg (asymmetry factor). The transmission for a given aod depends also on the values for ssa and 795 gg.
For the surface albedo, Equation 1 is also valid not for the extraterrestrial irradiance but the surface irradiance: δSAL I surf = t H2O · I surf (3) Author contributions. Richard Müller wrote the manuscript. Uwe Pfeifroth contributed to the writing.
Competing interests. We declare that we have no competing interest at present.  Urbich, I., Benidx, J., and Müller, R.: A Novel Approach for the Short-Term Forecast of the Effective Cloud Albedo, Remote Sensing, 10,