A variational algorithm aims to minimise the departures from the observations and the background profile, weighted by the expected errors of the background profile and the observations. In the case of the MWR and cloud radar, the observed measurements of radar reflectivity and brightness temperature (BT) are not the same as the variables being retrieved, known as the control variables. This necessitates the need for forward operators to convert the control variables into equivalent brightness temperatures and radar reflectivity. A cost function J, shown in Eq. (), may be calculated, where xb is the background state, B is the background error covariance matrix, y is the observation vector, F is the forward operator and R is the observation error covariance matrix. In this study, the observation vector y contains both the BASTA reflectivity (Z) of the 90 range gates corresponding closest to the heights of the 90 AROME background profile vertical levels and the MWR (BT) observed both at zenith and off-zenith elevation angles. The notation used here is that of , with the exception that F is used for the forward operator to clearly distinguish it from the Jacobian matrix, H. The (BT) vector thus corresponds to 13 channels at zenith and 4 channels at each of the 9 lower elevation angles (see Sect. ), which corresponds to a total size of 49 measurements. At all vertical gates, when the BASTA reflectivity is below the minimum detectable signal (shown in Fig. ), the reflectivity is set to this minimum detectable signal.

Radar reflectivity observations are provided with a mask defining the type of hydrometeor which the observation corresponds to (liquid water or airborne plankton, ice, or drizzle and rain). The mask detects the melting layer from the radar reflectivity and Doppler velocity gradients. For the liquid section, rain, drizzle and cloud are defined from the Doppler velocity . Retrievals are not attempted when rain is present in any pixel of the observations, due to the added complexity for making retrievals when the radome of the radar is wet (due to unknown attenuation effects). Where ice clouds are present in the observations above a liquid water cloud, retrievals of liquid water content in the lower part of the atmosphere are still performed by filtering out the ice cloud signal from the radar reflectivity. To do this, the BASTA reflectivity is set to the radar sensitivity at the range gate containing ice. The aim of the algorithm is to focus on retrievals in warm fog only, but, as the MWR is not affected by the presence of ice, this configuration still allows the additional retrievals of temperature and humidity even during the presence of ice clouds.

In this study, the background state xb is defined as the AROME model 90 levels of temperature, specific humidity and liquid water content. Background profiles taken from the AROME model were found to commonly contain a mix of different hydrometeors alongside LWC, most frequently rain or ice. As the algorithm is currently developed only for multi-layer liquid clouds, 1D-Var retrievals are not performed when the AROME background profile contains a significant amount of ice or rain. To evaluate the significance of the amount of rain and ice in the AROME background profile, radar reflectivity was simulated with only LWC and with all hydrometeors. Profiles significantly altered in the radar reflectivity simulations by other hydrometeors were not considered – as explained in . To that end, the radar reflectivity was simulated from the background profiles with all hydrometeors and then with only LWC. If the profile contained pixels with differences of more than 3 dB between the simulation with all hydrometeors and with only LWC, the other hydrometeors were considered to make up a significant portion of the cloud, and 1D-Var retrievals are not performed for this case. In the other cases, 1D-Var retrievals are performed considering rain and ice water contents as negligible. No correction is brought to the AROME LWC background profile when there is a cloud in the observation but not in the background state and vice versa. In fact, the 1D-Var algorithm should easily remove non-observed cloud layers in the background. This is more complex when the AROME background is clear but the radar observation is not. To deal with this issue, new developments are proposed during the computation of the Jacobian matrix for the cloud radar reflectivity. This is discussed later on in this section.

The 1D-Var algorithm aims to minimise the cost function for the control variable state x, for which the statistically optimal state given all the input components is considered to be found. 2J(x)=12x-xbTB-1x-xb+12(y-F(x))TR-1(y-F(x)) The minimisation of the cost function is performed by iteration, with subsequent values of x being found through Eq. (), where xi+1 is the following state vector and x0, the initial state, is equal to xb. Factor γ is a coefficient specified by the Levenberg–Marquardt descent algorithm. Here Hi represents the Jacobian matrix, the predicted sensitivity of the observation matrix to a change in state xi (Hi=∂F(xi)/∂xi). 3xi+1=xi+(1+γ)B-1+HiTR-1HiHiTR-1y-Fxi-B-1xi-xb For this experiment, the 1D-Var package maintained by the NWP Satellite Application Facility (NWPSAF; https://www.nwpsaf.eu/site/software/1d-var/, last access: 25 October 2021), modified for the 1D assimilation of ground-based MWR observations , has been extended to cloud radar observations. The forward operators F used both for MWR and cloud radar observations are described in Sect. .

The Jacobian matrix for cloud radar reflectivity was calculated through the brute force method, which involved running the forward models with small perturbations to each element of xi , due to the difficulties of linearising the forward operator. This method has been used in previous studies investigating the data assimilation of radar reflectivity and has been shown to be an effective way to estimate the Jacobian matrix . This is formulated in Eq. (), where δxk is the perturbation made to element k of the current state vector xi, for a state vector of size n, and FK(x) is the simulated observation corresponding to the Kth observation from state xi, for an observation matrix of size N. 4H=∂F(x)∂x≈F1x+δx1-F1(x)δx1F1x+δx2-F1(x)δx2…F1x+δxn-F1(x)δxnF2x+δx1-F2(x)δx1F2x+δx2-F2(x)δx2…F2xn+δxn-F2(x)δxn⋮⋮⋱⋮FNx+δx1-FN(x)δx1FNx+δx2-FN(x)δx2…FNx+δxn-FN(x)δxn The perturbation size of the forward operator with respect to the different variables has been selected according to the observed linear behaviour of the forward operator after testing different values of perturbation with changes of this size.

As previously stated, when the AROME background is clear, but the BASTA observation shows a cloud layer, the 1D-Var algorithm would not be able to add a new cloud layer due to zero values in the Jacobian matrix. In fact, where the AROME background is clear, the simulated radar reflectivity is below the minimum detectable signal and set to the radar sensitivity to be consistent with the BASTA observations. In cases where a small perturbation in the initial LWC profile is unable to create a radar reflectivity above the radar sensitivity, the difference between the two simulated reflectivities (with the perturbed profile minus the initial profile) would be zero. Consequently, the Jacobian values calculated from the brute force method would be equal to zero, which means that the inclusion of a new cloud layer in an initially dry background profile would not be possible.

Here this work differs from others , as Jacobian values are forced to be non-zero when the background profile is clear to give more flexibility to the algorithm to create a cloud layer when necessary. In order to do this, the minimum value of LWC leading to a radar reflectivity value equivalent to the radar sensitivity at that specific vertical altitude has been defined for each range gate. Jacobian values are then calculated in the neighbourhood of this defined minimum LWC values to fill in the Jacobian matrix where the background is clear.

For MWR brightness temperatures, the adjoint of the tangent linear of the fast radiative transfer model RTTOV-gb is used .

The algorithm will converge when a state xa is obtained that minimises the cost function. The errors associated with this state are specified by the analysis error covariance matrix, A (again following the notation of , and not to be confused with the averaging kernel). This may be found from Eq. (), with diagonal terms giving an estimate of the variance of the retrieval error. 5A=HTR-1H+B-1-1

Forward models, otherwise referred to as observation operators, are used to convert the control variables of the algorithm into observation type variables, the use of which are a common approach for the assimilation of observations indirectly related to the control variables .

The fast radiative transfer model RTTOV-gb of and was used as the forward model for brightness temperatures in this experiment. This gives simulated brightness temperatures at the required microwave frequencies by using a radiative transfer equation for upward-looking passive sensors. The model computes the Planck radiances emitted from the top of the atmosphere to the surface and takes into account the absorption between the height level of the emitted radiance and the surface. In RTTOV-gb, liquid water is taken into account as an absorbing species, meaning that the effects of clouds on observed microwave brightness temperatures may be taken into account . This is necessary in order to more accurately model the brightness temperatures, which is essential to retrieving accurate profiles of temperature and humidity with this methodology. It also permits the retrieval of the LWP. While several radiative transfer models are able to simulate downwelling radiance with the aforementioned capabilities, RTTOV is designed to make fast calculations and is thus highly suited to operational variational methods.

In order to simulate the radar reflectivity from the control variables, a radar simulator for vertically pointing W-band radar designed by was used. Inputs of pressure, temperature, humidity and mixing ratio of five hydrometeor types (liquid cloud, ice cloud, rain, snow and graupel) must be specified. Radar reflectivity is computed at the same resolution as the input profiles, taking into account backscattering due to the five hydrometeor types and attenuation from moist and dry air. The simulator assumes a modified gamma distribution for the size distribution of hydrometeors, consistent with the ICE3 microphysical scheme in the AROME model, with the distribution coefficients being specified as additional inputs. An evaluation of the radar simulator capability for ground-based 95 GHz cloud radar and sources of uncertainty can be found in .

The background profile is used to constrain the retrieval, and hence the more accurate the background profile is, the more accurate the retrievals are likely to be. The French convection-scale NWP model AROME was hence used to provide background profiles for this study. The AROME model has 90 vertical levels from the surface to a height of approximately 30 km and a horizontal resolution of 1.3 km over the domain centred over mainland France and covering most of western Europe .

have shown that large errors due to the spatial and temporal displacement of fog events could be expected from short-term model forecasts during fog events, leading to suboptimal background profiles for future 1D-Var retrievals. An approach to select a more adapted background profile with reference to the observed radar reflectivity within a 27 km sub-domain and within a 6 h time window, named the most resembling profile (MRP) method, was therefore proposed and was shown to significantly reduce innovation (observation minus background) error statistics. This study follows the work of by using the MRP method to provide an adequate background profile to the 1D-Var retrieval algorithm. The impact of using the MRP method will be discussed in Sect. .

The specification of the errors associated with the background profile and observations instruct the algorithm on the level of trust placed in both the observations and the background profile. The retrieved profile will take into account the relative weight of the observation errors compared to the background errors, thus the smaller the background or observation errors are, the more closely the retrieved profile is likely to agree with the background or the observations, respectively. It is the role of the background error co-variance matrix to specify the background errors to the 1D-Var algorithm. Background errors are not assumed to be independent of one another, thus co-variances between the different control variables are specified in the off-diagonal terms of this matrix.

There are two common approaches to the modelling of background error statistics: to compute innovation statistics by comparing background profiles to observations with a low observation error or to use a surrogate method, which it is assumed can approximate the errors . Due to the difficulty of gathering observations representative for all forecast conditions and the difficulty of modelling the relation of errors in different locations in 3D- and 4D-Var algorithms, surrogate techniques have been a common choice. These surrogate techniques normally involve comparing forecasts run with different lead times . The background error covariance matrix used operationally in the AROME 3D-Var is computed from a climatological dataset of 3 h ensemble forecasts derived from the operational AROME Ensemble Data Assimilation system (EDA). have demonstrated that a B matrix specifically adapted for fog exhibited significant differences compared to a B matrix designed for cloud-free areas and that the use of this led to an improved analysis during fog conditions. The B matrix used in this experiment was generated in a similar way applying a fog mask in order to better represent background error covariances specific to fog conditions. More explanations about the B matrix computation can be found in .

The observation error covariance matrix (R matrix) is comprised of the instrumental error for each measurement plus the error contained in the forward operator used to simulate the measurement from the state vector x. For observations made with the microwave radiometer, forward model errors vary significantly between each channel. Uncertainty in the modelling of brightness temperatures resulting from uncertainties in absorption modelling has been assessed to range from 0.3 K at 22.24 GHz up to 3.18 K at 52.24 GHz in winter in the mid-latitudes . The total uncertainty due to the MWR calibration technique has been assessed to be between 0.2 and 1.2 K by . Finally, instrumental noise is below 0.5 K at all channels . Each source of uncertainty has been added in quadrature to provide the total observation error: σtot=σnoise2+σcalib2+σFM2, with σtot the total observation error, σnoise the uncertainty due to the instrumental noise, σcalib the calibration uncertainties and σFM the uncertainty due to spectroscopic errors in the radiative transfer model. The observation error covariance matrix is assumed to be diagonal, with observation error values for each channel provided in Table .

For the cloud radar errors, instrumental error of 2 dB was assumed from work on the calibration of the BASTA cloud radar . As discussed in , the primary component of forward model errors in the radar simulator comes from the hypothesis made on the assumed cloud droplet size distribution. Droplet number concentrations were found to range from 30 to 300 cm-3, and the shape parameter ν was found to range from range 2.5 to 15 for the expected cloud types and conditions. This resulted in a forward operator parameter error of approximately 3 dB when the 25th to 75th percentiles of these values were considered . A forward model error of 3 dB is thus assumed hereafter. The total expected variance is found from the variance of measurement errors plus the variance of forward model errors, and it follows that the standard deviation, which is considered as the instrumental error here, is the square root of this. The total instrumental error used for the retrievals was therefore considered to be 3.6 dB.

Despite the probability of a degree of correlation between the radar observation errors at different range gates, which could come from both the calibration uncertainties and correlations in the size distribution errors, as well as uncertainties in attenuation modelling, the R matrix was assumed to be a diagonal matrix, i.e. there was assumed to be no correlation in the observation errors. Despite this, tests were made with use of a square R matrix with high error correlations between the radar observations, which suggested that little impact would be brought to retrievals with the use of this method.

The requirements for making a synthetic database are that we have a dataset that best resembles that of the real observations. For this, the following prerequisites needed to be met: i.

the true profiles need to be physically consistent and resemble atmospheric profiles that would be observed,

To that end, the considered true profiles were generated from the 10 to 180 min AROME model short-term forecasts at the SIRTA observatory (Site Instrumental de Recherche par Télédétection Atmosphérique) between November 2018 and February 2019 . The background profiles were derived from the true profiles by perturbing the temperature, humidity and LWC profiles according to the expected AROME background error covariance matrix during fog conditions. This was done with the use of the random.multivariate_normal function in the Python NumPy package to produce random vectors that have a covariance equal to the B matrix. By adding these vectors to the true profiles, a dataset containing the error expected in the real background profiles was created. To ensure that this would reflect the conditions in which retrievals would be made, a specific fog B matrix was computed from one AROME EDA cycle valid for one fog case simulated in November 2018.

Similarly, the synthetic observation dataset could be derived by simulating the MWR brightness temperatures and then perturbing these by the expected observation error covariance matrix described in Sect. . An estimation of radar reflectivity error of 3 dB was made initially and used in the synthetic data study. This is in line with the 3.6 dB estimated uncertainty from that was later used in the application to real measurements.

From the synthetic database, forecasts involving mixed-phase cloud were excluded, as the retrieval algorithm has not yet been developed to make retrievals of ice, and where there is radar reflectivity that comes from a mixture of ice and liquid water, it is not currently possible to distinguish between the radar reflectivity signal from ice crystals and the signal from water droplets. In total, 1063 suitable profiles were found, involving a range of different synoptic conditions, with radiative fog, low stratus cases, and stratus-lowering fog found to make up most of the dataset.

In this section, 1D-Var retrievals have been carried out on the database of suitable liquid cloud profiles. A requirement for the convergence of the algorithm is made based on the value of the cost function, the normalised gradient of the cost function and the gamma factor of the Levenberg–Marquardt minimisation. Successful retrievals were made for 97 % of the profiles, with a maximum number of iterations set at 15.

Figure  shows that the background profiles present a positive bias in the LWC field. This is caused by the way in which perturbations are made to all the fields. By adding or subtracting LWC amounts from the true profiles based on the background error covariances according to a Gaussian distribution, occasionally perturbations will be made that decrease the LWC field to be below zero. As this is un-physical, any values of LWC below zero were set to zero. As this meant that a net increase was being made to the LWC field, a positive bias was seen. Figure  shows that the 1D-Var is able to correct this bias, reducing it from 0.028 g m-3 in the background to 0.004 g m-3 in the retrieval. A significant improvement is also observed in the total root-mean-square error (RMSE) from 0.047 g m-3 in the background profile to 0.018 g m-3 in the retrievals. The correlation with the true LWC values is also improved from 0.90 to 0.98.

As anticipated, the retrieval was also able to improve the values of LWP. From Fig. , the effect of the positive bias on the LWC can be seen more clearly. In fact, in the background profiles, a large positive bias of 41.5 g m-2 can be seen, with a standard deviation of LWP errors of 50.6 g m-2. The standard deviation of errors is improved by around 75 % in the retrievals to reach a value of 11.5 g m-2. It can be noted that this estimated uncertainty is much smaller than the expected 20 g m-2 when MWR are used alone to derive the LWP highlighting the potential benefit of the instrumental synergy.

Another point of interest in the study is to quantify the benefit of the dual-retrieval method compared to retrievals made with one instrument alone. As highlighted in Sect. , the microwave radiometer is sensitive to the temperature and humidity profiles, but only the integrated value of LWC (the LWP) . The radar, meanwhile, is sensitive to the LWC at each range gate observed but has very little sensitivity to the temperature and humidity. Because of this, it was not expected that retrievals made with radar observations alone would result in changed temperature and humidity values. However, if appropriate cross-correlations between variables are used during the 1D-Var algorithm, we can expect that an improvement in LWC increments could positively impact the temperature and humidity increments. Additionally, by better locating the cloud in the vertical, we should also improve the radiative transfer simulation of MWR channels sensitive to humidity and LWC during the 1D-Var minimisation. If dual-instrument retrievals were statistically found to be better than retrievals with the radar alone, this would suggest the benefit of a single multi-instrumental algorithm compared to separate algorithms.

The statistics of the retrieved vertical profiles presented in Fig.  confirm the previous conclusion concerning an initial LWC positive bias in the background profile, with a peak of 0.03 g m-3 bias at 500 m. This bias is fairly well corrected for in the retrievals with absolute biases smaller than 0.01 g m-3. The standard deviation of background and retrieval minus the truth statistics also show that the most benefit is brought to the retrievals of liquid water content when both instruments are used. It can be noted that the cloud radar shows the largest benefit of either instrument alone, but the additional MWR information manages to decrease the standard deviation with respect to truth profiles at all vertical gates, with the largest improvement above 1 km. An overall accuracy of 0.02 g m-3 above 400 m and 0.04 g m-3 below 400 m is expected when both MWR and cloud radar measurements are used, giving a relative standard deviation of errors of around 20 %. When the MWR alone is used, the expected retrieval error is approximately twice the one obtained with the synergistic retrieval but already managed to decrease the background error from 0.09 to 0.05 g m-3 at 250 m.

The retrieval uncertainty due to the forward model error and forward model parameter uncertainty may also be investigated. As mentioned in Sect. , several parameters must be specified to the radar simulator to prescribe the cloud droplet size distribution. As these parameters may not be representative of the observed cloud droplets, this induces a certain amount of error into the retrieval. From , the covariance of the error in the retrieval due to errors in the forward model parameters, Ab, may be calculated from Eq. (). 6Ab=GHbRbHbTGT,7G=HTR-1H+B-1-1HTR-1, where Rb is the covariance matrix of forward model parameter errors, Hb is the sensitivity of the forward model to the prescribed parameters (analogous to Eq. ) and G is called the gain (or contribution function) matrix. Ab represents the error covariances in the retrieval due to the assumptions about the forward model parameters. In , an analysis of expected error in two of the droplet size distribution parameters – the total droplet concentration N and the shape parameter ν – was conducted with the aid of previous literature on fog and cloud droplet distributions. These uncertainties can be used to estimate the matrix Rb. In this study, it was found that the total droplet concentration could be expected to range from 30 to 300 cm-3 and that the shape parameter varied from 2.5 to 15. It was assumed that this matrix was diagonal, i.e. that the error in the two parameters was not correlated, and error was not correlated between different retrieval height levels.

In order to avoid directly calculating Hb, the reflectivity from a profile can be simulated with the radar simulator and then once again by making a small perturbation to the simulator parameters. By finding the difference between the first and second simulations and dividing this by the perturbation size, the matrix Hb may be approximated in the manner described in Sect. 2.2.

The retrieval error resulting from errors in forward model parameters was estimated for one profile with a maximum LWC of 0.3 g m-3. From the square root of diagonal components of matrix Ab, calculated from Eq. (), model parameters were found to contribute between 0.01 and 0.025 g m-3 to the total retrieval error for liquid water content. If the droplet concentration is known, the contribution of parameter error falls to between 0.006 and 0.014 g m-3.

The effect of the assumed instrumental error was also investigated, with retrievals being made with a cloud radar instrumental error of 1, 3 and 6 dB, whilst the same synthetic database of observations and background profiles was used. There were no significant differences seen by reducing the observation error from 3 to 1 dB; however, when the instrumental error was increased to 6 dB, the retrieval accuracy was degraded slightly by 0.01 g m-3, as shown in Table .

Figure  also shows the statistics for the temperature and humidity fields. As expected, the use of MWR observations significantly decreases the temperature errors with respect to the truth profiles from approximately 1.3 K in the background to 0.7 K in the retrieval at 200 m altitude. At 200 m a.g.l., the standard deviation of errors is slightly improved for the dual retrieval compared to the microwave-only retrieval, though these differences cannot be considered to be significant.

It can be noted that the radar alone lacks any meaningful sensitivity to either variable, as shown in Fig. c–f. Though attenuation of radar reflectivity is affected by the temperature and humidity of the air below the backscattering target, the perturbation of humidity and temperature from the background profile needed to bring about a change in radar reflectivity, relative to their respective errors, is far higher than the relative change in LWC needed to bring about a change in reflectivity relative to LWC background errors. However, changes to the LWC field can impact the retrieval of temperature and humidity. In fact, as the B matrix contains the cross correlation of errors between all retrieved quantities (i.e. temperature, humidity and LWC for 90 height levels, which are the same as those in the AROME model), a larger perturbation in the LWC field at a certain height could be associated with a larger perturbation in temperature or humidity, should the correlation between the errors of the two variables be strong. When the radar is used in conjunction with the microwave radiometer, the effect of this is generally seen to a greater extent.

Such an impact is seen in the humidity retrievals in Fig. e and f, where there is little difference between the standard deviation of errors in the radar-only retrieval compared to the background profile. The standard deviation of errors did, however, become slightly reduced in the dual-retrieval configuration compared to the microwave-only retrieval. However, for the altitudes where this improvement is seen, there persists a negative bias for the dual radar plus microwave radiometer retrievals that is larger than for the microwave-only retrievals.

Due to the existing positive bias in the LWC background profiles, as the retrievals using the cloud radar involve a net reduction in the LWC, the positive cross correlation of background errors between LWC and specific humidity in this region (for variables at the same height level), could encourage too large reductions in the specific humidity field and the appearance of this negative bias in the humidity retrievals.

A similar impact is observed in Fig.  for temperature retrievals. In fact, the only height at which the standard deviation of temperature retrieval errors with cloud radar alone are visibly different from the standard deviation of background errors is at 200 m a.g.l., the height at which the largest errors in the LWC field are observed and the radar-only retrieval bias is the largest. It should be noted that the bias in the temperature retrievals at this height increases to 4 times the value of bias in the dual retrieval.

It can be useful to know how many independent pieces of information are used in the retrievals. Modern microwave radiometers contain many channels, between which there is often a high degree of correlation. The number of independent observations is therefore considerably lower than the number of channels. The measurement uncertainty will also affect the information content in the retrievals, with a larger uncertainty resulting in a lower information content. In , this problem is formalised by relating the number of independent columns of information to the concept of degrees of freedom. We may then ask how many of the degrees of freedom of measurement are related to noise and how many are related to signal.

To calculate the number of independent pieces of information that are used in the 1D-Var retrieval, Eq. () may be used: 8DFS=trI-AB-1. Here, I is the identity matrix, tr refers to the trace of a matrix and A is the analysis error covariance matrix described earlier. We may also split the degrees of freedom for signal (DFS) into the three variables being retrieved, showing the DFS for temperature, humidity and liquid water content independently. The mean DFS for all the synthetic profile retrievals made is shown in Table .

For the liquid water content retrievals, we expect the radar to contribute the most to degrees of freedom for signal compared to the microwave radiometer measurements. As the DFS coming from a radar is likely increased as the number of range gates affected by a cloud layer increases, the relative DFS for the liquid water content retrievals is calculated instead of the absolute DFS. The relative DFS for LWC is the DFS for LWC divided by the total number of levels containing non-zero LWC values. While MWR provides very few independent pieces of information on the LWC profile with a relative DFS of 2.7 %, a relative DFS of 37 % is obtained by the single use of the cloud radar. The synergistic use of both MWR and cloud radar manages to increase the relative DFS up to 38.2 %. These results agree well with a previous study by .

When only MWR observations are used, with respective values of 2.31 and 0.75, the DFS between the surface and 30 km a.s.l. for temperature and humidity was smaller than that found in other works . However, these studies used climatological background profiles, which typically have larger background errors. The DFS measures the sensitivity of the retrieved profiled to changes in the true profile. The larger the background errors are assumed to be, the more weight will be given to the observations, which will change with respect to the true profile while the background remains fixed. Therefore, the more relative weight given to the observations, the higher the DFS will be. As a dynamic background taken from an NWP model is used in this study, the background errors are considerably lower than those of a climatological background by approximately a factor of 10 for temperature and factor of 5 for humidity, resulting in a lower DFS similarly to what has been observed in . It may be noted that, as expected, almost no information can be extracted about temperature and humidity from the radar alone. However, through the background error covariance matrix correlations, the synergistic retrievals slightly decrease the temperature DFS while increasing the humidity DFS. The cumulative DFS may also be examined. This shows at which altitudes the signal used in the retrievals comes from and is shown in Fig. . At a given altitude, the higher the rate of change of cumulative DFS, the greater the signal used in the retrievals. It may be seen that most information from the temperature retrievals is found in the lowest 750 m, whereas for humidity it is found mainly between 1000 to 3000 m. It should be noted here that most humidity information brought by the measurements is significantly above the fog layer. For the SOFOG-3D field campaign, 80 % of events measured with the tethered balloon had a maximum fog top height lower than 200 m, and thus a limited improvement could be expected.

The South-west Fog 3D Experiment for Process Studies (SOFOG-3D) field campaign took place between October 2019 and March 2020 and was an observational field campaign focused on fog . In addition to numerous other measurements across the south-west of France, including a microwave radiometer network , the main measurement site was located near the village of Saint-Symphorien in the department of Gironde, 50 km south of Bordeaux. Here, the BASTA cloud radar and HATPRO microwave radiometer were deployed, making continuous measurements throughout the 6-month period. During fog episodes, intensive observation periods (IOPs) took place. During these IOPs, a tethered balloon to which a cloud droplet probe (CDP), optical particle counter (OPC) and cloud condensation nuclei counter (CCNC) were attached was also used for vertical profiling of both aerosol and cloud microphysical properties. During the IOPs, radiosondes recording temperature, humidity and wind were also launched two to four times per night. Table  notes the instruments at the supersite that were used in this study.

Measurements from the CDP were averaged by total flow volume over each 10 s period. Although the binned distribution of liquid water droplet diameters is provided by the CDP, only the total number concentration and the LWC were considered in this study.

One issue with using radar reflectivity to make retrievals of liquid water content can be the presence of pollen, insects and other non-meteorological airborne particles (sometimes referred to as airborne plankton). These can affect radar reflectivities in the boundary layer and can be difficult to distinguish from signal caused by clouds. In this study, to ensure that radar reflectivity observations were caused by cloud droplets only, the cloud base height and visibility measurements were used to ensure that only cloud radar reflectivities corresponding to either a fog or a cloud layer were used.

The minimum observable radar reflectivity is found from the signal to noise ratio. This therefore depends on the height of the gates, as both the noise and the maximum reflected power change with altitude. The sensitivity of the radar was found to range from -52 dBZ at 75 m to -33 dBZ at 1000 m (gates above 1000 m were not investigated due to the very small amount of radar gates not affected by ice particles above that altitude).

In order to investigate potential biases in the cloud radar reflectivity or radar simulator, Fig.  shows the comparison between the simulated reflectivity from the CDP measurements and the BASTA radar reflectivity. In this study, both the LWC and droplet number concentration from the CDP are directly used in the radar simulator. It can be noted that significant discrepancies can be observed between the two reflectivities, as observed between 04:30 and 06:00 UTC, with a large under-estimation of the simulated reflectivity from the CDP measurements compared to the BASTA measurements.

Though the exact reason for this is not perfectly known, it could come partially from temperature dependencies of certain components of the radar , from the CDP underestimating the LWC (e.g. through missing droplets or the mis-sizing of droplets) or other effects that were not accounted for. It should be noted that the maximum droplet size observable by the CDP is 50 µm. Due to the fact that the larger droplets have a disproportionately large impact on radar reflectivity, the presence of these droplets could have contributed to this effect. In fact, have attempted to characterise the uncertainty of the CDP instrument in studies using both glass beads and water droplet generators, wherein a known size distribution is observed by the instrument and measurements are compared to the distribution. Two common problems experienced by the CDP are coincidence error, where two droplets cross the beam at the same time and are interpreted to be one droplet, and mis-sizing, normally caused by droplets traversing the edge of the beam and being interpreted as smaller droplets. The liquid water content observation from the CDP is calculated from the sum of the mass of all droplets observed within the sampled volume; hence, the uncertainty in droplet diameter and concentration results in an even greater error for measurements of LWC, as the LWC is proportional to the third power of droplet diameter.

Although the CDP can only measure droplets of up to 50 µm, very few observations were recorded of cloud droplets with a diameter over 40 µm, and the assumed droplet size distribution predicted that droplets over this size would account for only 0.15 % of the volume of cloud liquid water observed, for an LWC of 0.12 g m-3. However, the modal diameter sizes observed often had a non-negligible difference compared to that predicted by the assumed droplet distribution in the forward model (for the same LWC). It is therefore possible that a larger proportion of droplets with a diameter bigger than 50 µm were present and not observed by the CDP.

An additional phenomenon was also observed by and hypothesised to be due to cloud inhomogeneities at a scale smaller than the cloud radar sampling volume. These could be particularly pertinent at the fog top height, where radar gates could be only partially covered by LWC. In regards of the observed biases between the BASTA- and CDP-simulated reflectivities, and in order to be able to validate 1D-Var retrievals with CDP measurements consistent with the cloud radar observations, a time-dependent bias correction was performed on the radar observations. To that end, the reflectivity bias is computed as the 60 min average of the differences between the cloud radar and CDP-simulated reflectivities and then subtracted from the received power. Figure  shows the overall improved agreement between the CDP-simulated and cloud radar reflectivities after bias corrections. The bias-corrected radar reflectivities are then used in the following sections.

Brightness temperatures from MWR observations were also bias-corrected with a similar method as detailed in . This method is based on a daily monitoring of differences between observed brightness temperatures and simulated brightness temperatures from the AROME model 1 h forecasts during clear-sky conditions only. This MWR unit was also calibrated with the use of liquid nitrogen just before the experiment. The obtained bias correction is thus relatively small, with absolute values from 0.1 to 1.8 K, the largest biases being due to spectroscopic modelling errors in the oxygen band.

In this study, one fog event taking place on 7–8 March 2020 was examined due to the length of the event and the thickness of the fog layer. This fog event had a large spread of LWC values; it also went through phases of thinning, a phase with a cloud aloft and a phase of lifting from the surface before dissipation. The event also had a relatively long duration of 9 h, with in situ measurements made over 11 h.

In general, reliable radar signal is available from the third gate (37.5 m) during the night and the fourth gate (50 m) during the day. It was also shown in the synthetic data study that despite the liquid water path retrieval having a smaller errors compared to the HATPRO microwave-only retrieval, an error of 11.5 g m-2 was present in retrievals of the liquid water path. For this reason, the retrievals of liquid water content were expected to show good results of liquid water content only for fog layers with at least a thickness of 50 m.

As may be seen from the radar reflectivity observation shown in Fig. , an initial thin low cloud present around 21:00 UTC lowered to the surface to form a thin fog layer. This grew in thickness as cooling took place throughout the night. Observations with the tethered balloon began at around 23:00 UTC and continued until 10:00 UTC the following day, at which time the fog layer had lifted to form a low stratus cloud before completely dissipating. Figure  also shows the time series of retrieved temperature profiles and LWP from the MWR alone using the manufacturer neural networks implemented in the instrument software. Pre-fog stable conditions with a thermal inversion close to the surface up to 03:00 UTC may be observed on this figure. Between 03:00 and 08:00 UTC, the temperature profile is almost iso-thermal within the first 300 m, corresponding to the fog mature phase. Around 08:00 UTC, at the time of fog dissipation, the increase in surface temperature due to the heating of the sun may also be identified. The LWP retrievals show values below 20 g m-2 until 00:00 UTC, followed by a large increase up to 80 g m-2 during the fog mature phase.

The radar and radiometer were situated in close proximity but were separated from the tethered balloon by a distance of up to 300 m. This distance varied as a function of the balloon height and the wind speed. From a scanning version of the cloud radar stationed close to the supersite, large variations in values of radar reflectivity were observed with horizontal spatial displacements. To ensure that comparisons of measurements made by the CDP were therefore comparable to the retrievals, it had to be ensured that all instruments were observing a similar fog layer. In order to do this, a screening procedure has been applied to remove from the statistical analyses cases from which the BASTA cloud radar differ from the CDP-simulated reflectivity by more than 3 dB. This difference took into account the instrumental and radar reflectivity simulator errors, meaning that differences greater than this must come from one of the previously mentioned effects that could account for differences between observed reflectivity and that simulated from the CDP. The most likely explanation of the discrepancy between the observed radar reflectivity and the radar reflectivity simulated from the CDP is the inhomogeneities that are present, as highlighted in Sect. .

In Fig. , the AROME forecasts corresponding to the closest time (within a 3 h windows) and location of the observation and the 1D-Var retrievals are shown. Also shown is the MRP background, with the retrievals using this as a background. With reference to the radar reflectivity shown in Fig. a, it is evident that the presence of LWC is better modelled in the retrieval compared to the AROME background.

The initial lowering of cloud at 21:00 UTC is clearly seen in the retrieval, whilst the fog event begins with a thin layer at the surface in the model at 22:00 UTC. From the CDP measurements of the fog event investigated, a noticeable variation in the total droplet number concentration was observed, shown in Fig. . It can be seen that the number concentration frequently diverges from the assumed concentration of 150 cm-3 used in the radar simulator, from zero (evidently in the absence of LWC) to a maximum of 300 cm-3, which was observed in the middle of the fog layer where the LWC was relatively large (approximately 0.3 g cm-3). The lowest values of number concentration, where the CDP was inside a cloud or a fog layer with an LWC above 0.5 g cm-3, were around 50 g cm-3 – significantly higher than the lower bound of number concentration of fog events observed in some studies .

The retrievals were made with different configurations at a 1 min time resolution, in order to estimate the sensitivity of the retrieval to certain parameters. In this section, 1D-Var retrievals will be directly compared to CDP LWC measurements when the BASTA reflectivity and the CDP-simulated reflectivity are comparable. Here, the two observations were considered comparable for a radar and CDP observation made at the same time and valid at the same height, with a difference in the simulated and directly observed radar reflectivity of 3 dB. However, several CDP measurements at 10 s resolution are located within each 1D-Var gate with a variable grid size between around 10 m at surface and 80 m at 1 km. To take into account the CDP measurement variability within each 1D-Var gate, the 1D-Var LWC retrieval is compared to the median CDP measurements within certain spatial and temporal bounds. To that end, only CDP measurements within 10 m of the height corresponding to the middle of the 1D-Var gate are taken into account in the calculation of the median. If the time needed by the tethered balloon to sense the whole ±10 m bin is spread over several minutes, an average of the 1D-Var retrievals within the time of ascent of the tethered balloon is directly compared to the median CDP LWC measurements.

The sensitivity of the 1D-Var algorithm to different settings is investigated by quantifying the impact on the LWC accuracy.

We first examine the effects of an improved background profile, found by using the MRP method to select the AROME background profile closest to the observation within a 27 km domain and 6 h time window . To this end, retrievals were made by with the MRP background and the background corresponding to the nearest time and location of observation. For the case in question, it is notable that the closest AROME model in time and location was able to forecast the overall structure of the second half of the fog event fairly well (Fig. ), and it is during this period that most of the in situ observations were made. Because of this, the standard deviation of LWC errors in the nearest background profiles, found through comparisons to the in situ observations, was very similar to that of the MRP background, with values close to 0.08 g cm-3 (Table ). As shown in Table , the standard deviation of LWC errors was reduced for retrievals using both background profiles. The observed impact was, however, slightly better when the nearest background profile was used, with a 0.009 g cm-3 lower standard deviation of errors and a 0.012 g cm-3 lower bias. The differences in LWC retrieval statistics with both backgrounds are probably small enough to be considered within the spatial variability and uncertainty of the CDP measurements within each radar vertical gate, which was found to be 0.02 g cm-3. As mentioned earlier, it is possibly due to the reasonably well-described fog structure from the nearest AROME background when the CDP measurements were performed during this unique fog event which prevents the benefit of the MRP method being seen. However, further analysis was performed concerning the better performance of the 1D-Var algorithm using the nearest profile compared to retrievals using the MRP background despite the larger number of clear layers in the nearest background profile at range gates covered by fog or low clouds in the observation.

As was highlighted in Sect. , one possible reason arises from the calculation of the Jacobian matrix, which was handled so that diagonal components of the matrix could not be zero as should be the case when the background profile is clear. Forcing Jacobian values to be non-zero even during clear-sky conditions was found to improve our results but probably hampers the benefit of the MRP, which successfully corrected the fog vertical structure in the first part of the event. In fact, it can be noted that at the beginning of the fog event, the nearest profile tended to underestimate the number of levels containing non-zero LWC, whilst the MRP was able to better approximate the cloud structure as expected. The impact of not being able to force non-zero Jacobian values where the initial hydrometeor contents of a pixel is zero was then investigated, as this is generally the case with variational frameworks using tangent linear approximation of the forward model due to the faster computation time. To that end, the MRP method was again compared to the use of the nearest profile when the Jacobian values were set to zero when the background is clear.

Figure  highlights that for times when the nearest background poorly predicts the pixels containing LWC, the ability to replicate cloud in the retrieval is greatly degraded compared to when the Jacobians are forced to be non-zero (see Fig.  for comparison). Table  shows that whilst the algorithm is able to converge more that 99 % of the time for both background profile methods when the Jacobians are always non-zero, this is reduced to 84 % and 88 % when allowing Jacobians to be zero for the nearest and MRP backgrounds, respectively. Another point to note from the table is that the median number of iterations needed for convergence was improved and even divided by two where non-zero Jacobians are forced through the vertical profile when using the MRP method, meaning that less time and computational power is needed to make the retrievals thanks to the MRP method. Further evaluation will be performed in the future to more deeply investigate the benefit of the MRP method, especially during cases when the closest AROME background in time and location shows larger discrepancies with the observed reflectivity.

Due to the better performance of the nearest profile on this specific fog case, however, the next evaluation of different 1D-Var configurations was performed using the nearest profile and allowing non-zero LWC Jacobians even when the background profile is clear.

Figure  shows the scatter plots comparing both the AROME background and the retrieved LWC of three configurations of the algorithm to the CDP LWC measurements. Comparisons were made by finding the median LWC observed within 120 s of the retrieval and within 10 m of the retrieval height level. Error bars marked on the plots show the minimum and maximum values of LWC observed by the CDP over this time range and height difference for the retrieval in question. It may be seen that for certain comparisons, a wide range of LWC values are observed, which leads to uncertainties in the accuracy of the comparisons of ±0.15 g cm-3.

Whilst fog was predicted in the model after 06:00 UTC, with values that were fairly close to those observed, the fog event began with a large cluster of values where no LWC was predicted but was observed. Figure b shows that the retrieval made using the nearest background profile improves all statistical measures of the LWC field, with a decreased RMSE from 0.096 g cm-3 in the background to 0.066 g cm-3 in the 1D-Var retrievals. The correlation is also significantly increased from 0.56 to 0.72.

Two other configurations tested are also shown in Fig. c and d to evaluate the impact of the background error covariance matrix cross-correlations and the impact of microphysical assumption errors in the radar simulator. Figure c shows the retrievals made with a block-diagonal B matrix in contrast with a full B matrix with cross-correlations adapted to fog events. Though an improved LWC field is obtained even with a block-diagonal background error covariance matrix compared to the AROME background, it can be noted that the 1D-Var analyses are slightly degraded compared to the retrieval with a fully correlated B matrix. In fact, the largest degradation is observed on the correlation coefficient, which is decreased from 0.7 to 0.6. This could suggest that the information from the increments in the temperature and humidity profiles can provide information useful to the retrieval of LWC as long as proper cross-correlations between variables can be defined.

Optimal estimation techniques often derive background error covariance matrices from observations (such as ), which could very likely be improved through improved measurements of fog and cloud properties. However, this methodology requires that long-term in situ measurements are made for each site, which constitutes a serious drawback. As explained in Sect. 2.5, the B matrix for this work was produced through an ensemble data assimilation. It is thus likely that an improvement in the understanding and the modelling of fog processes in high-resolution models such as AROME could lead to more accurate background profiles with smaller background error covariances between different variables and model layers in the B matrix. In the future, improved measurements of fog and cloud properties could be useful for evaluating the efficacy of the ensemble data assimilation of generating representative background error matrices.

Another configuration to be investigated evaluated the impact of forward model errors due to inaccurate approximation on droplet number concentration. To that end, the total droplet number concentration initially fixed at 150 cm-3 was changed to the observed number concentration from the CDP (Fig. ).

As can be seen in Fig. d, a significant improvement in the 1D-Var retrievals is observed. In fact, the RMSE is decreased from 0.066 to 0.049 g cm-3, and the correlation coefficient is significantly increased from 0.7 to 0.8. This indicates that a non-negligible portion of the retrieval error is indeed due to the errors in the assumptions of size distribution of droplets. This sensitivity study demonstrated that optimal 1D-Var retrievals with increased accuracy can be obtained when forward model errors can be limited and an optimal background error covariance matrix is used.

The impact of synergistic benefit of using the two instruments in real-world conditions was also investigated. As is shown in Fig. , retrievals were run with the same configurations shown in Fig.  b) but with only one instrument. Figure a shows liquid water content retrievals made using only the microwave radiometer, and Fig. b shows the retrievals made with only the cloud radar. As expected, the radiometer is not able to accurately describe the distribution of liquid water content alone, and only small increases in the correlation coefficient, from 0.57 to 0.62, are found when compared to the AROME model. With the radar-only configuration, results were slightly degraded compared to those found from the synergistic retrievals, with a correlation coefficient of 0.64 compared to 0.71 and a standard deviation of errors of 0.61 compared to 0.60 in the synergistic setup. Single-instrument retrievals of temperature and humidity at the times of the radiosonde launches were also performed to investigate whether any benefit could be added to the radiometer retrievals through the inclusion of a cloud radar. As was found in the synthetic dataset studies, the retrievals made with only the radiometer were not statistically worse than with both instruments, indicating that information from the cloud radar does not significantly contribute to retrievals of humidity and temperature.

It may of course be seen from the errors in the fog case in question that the retrieval errors were larger than those found from the synthetic dataset study. This was expected, and there are several factors that could contribute to this. The first is that in the synthetic study, it is assumed that the background and observation errors may be perfectly modelled. The dataset of background profiles will contain the errors specified in the B matrix, and similarly the observational dataset will contain the errors specified in the R matrix. However, these matrices are only estimations of the true errors. Additionally, observations are supposed to be unbiased, which might not be the case, and even though a bias correction is proposed in this study, it may not be optimal. Another reason for the apparent increase in errors in this study compared to the synthetic data study probably comes from the fact that the in situ measurements used in this study themselves contained substantial error. Indeed, the error in LWC measurement by similar CDP instruments has been estimated to be up to 50 % . Compared to the CDP measurement variability that was used in Fig.  to quantify the error when comparing the retrieved LWC with the CDP LWC, the expected CDP measurement errors are of the same order of magnitude as the CDP variability for the highest values of LWC (>0.25 g m-3), but they should be smaller for the lowest LWC values (<0.1 g m-3). The variability in the measurement can be seen from the error bars on the scatter plots, with the range of CDP LWC measurements in some cases being greater than 100 % of the median observed value. An error in the comparison could also arise from the fact that the observations made from the balloon platform were of a much smaller sampling size and taken from a location up to 300 m away from the radar and radiometer field of view. Though an attempt to reduce the impact of this error was made through the permitting of comparative statistics only when the difference between observed radar reflectivity and radar reflectivity simulated from the CDP was less than 3 dB, a difference in size distribution in the droplets could not be ruled out.

Finally, to check the validity of temperature and specific humidity retrievals as expected from previous studies using MWR alone , a verification was performed through checks against radiosonde observations. In the field campaign, radiosondes were launched up to four times per day, meaning that fewer points of comparison could be made between observations and retrievals per fog event, compared to for the LWC observations from the tethered balloon. To make the verification over a statistically significant number of observations, all radiosoundings launched during the campaign IOPs were thus taken into account.

It can be seen in Fig.  that in both temperature and humidity retrievals, the standard deviation of errors is reduced when compared to the nearest background profile, which was the AROME model valid at the time and location of the observations, and this shows a much higher impact for temperature compared to specific humidity. It may be noted, however, that both the temperature and the humidity retrievals do not improve by the magnitude shown in the synthetic dataset study. Whilst the temperature retrievals were more accurate than the AROME model throughout the boundary layer, this improvement tended to range from 10 % to 20 %, in contrast to the 40 % seen in the synthetic dataset study. It should also be noted that there was a degradation in the temperature bias below 500 m. For the specific humidity retrievals, the retrievals showed smaller improvements of up to 10 % between 1000 and 2000 m a.g.l.

As mentioned in , 1D-Var retrievals are quite sensitive to the choice of the background error covariance matrix, especially for humidity retrievals and temperature retrievals within the first kilometre. One possible reason for the smaller improvement obtained on the temperature and humidity retrievals could thus come from a non-optimal B matrix used during the minimisation. In fact, it should be noted that for the application to real observations, the same static fog B matrix valid for November 2018 has been reused. However, our validation takes into account all of the SOFOG-3D radiosoundings launched at the supersite during atmospheric conditions prone to fog occurrences according to the AROME model forecasts, but only nine of them were found to be launched within a fog layer. The temperature and humidity retrievals might thus be representative of stratus clouds and clear conditions more than fog conditions, making the fog SIRTA B matrix potentially suboptimal. Further evaluation will thus be conducted in the future to adapt the fog B matrix to the time and location of the observation to make it more consistent with the real observation errors.