AMTAtmospheric Measurement TechniquesAMTAtmos. Meas. Tech.1867-8548Copernicus PublicationsGöttingen, Germany10.5194/amt-9-2425-2016Retrieval algorithm for rainfall mapping from microwave links in a cellular communication networkOvereemAartaart.overeem@knmi.nlhttps://orcid.org/0000-0001-5550-8141LeijnseHiddehttps://orcid.org/0000-0001-7835-4480UijlenhoetRemkohttps://orcid.org/0000-0001-7418-4445Hydrology and Quantitative Water Management Group, Wageningen University, P.O. Box 47, 6700 AA, Wageningen, the NetherlandsRoyal Netherlands Meteorological Institute (KNMI), P.O. Box 201, 3730 AE, De Bilt, the NetherlandsAart Overeem (aart.overeem@knmi.nl)1June2016952425244418July20157August201514April20164May2016This work is licensed under a Creative Commons Attribution 3.0 Unported License. To view a copy of this license, visit http://creativecommons.org/licenses/by/3.0/This article is available from https://amt.copernicus.org/articles/9/2425/2016/amt-9-2425-2016.htmlThe full text article is available as a PDF file from https://amt.copernicus.org/articles/9/2425/2016/amt-9-2425-2016.pdf
Microwave links in commercial cellular communication networks hold a promise
for areal rainfall monitoring and could complement rainfall estimates from
ground-based weather radars, rain gauges, and satellites. It has been shown
that country-wide (≈ 35 500 km2) 15 min rainfall maps can be
derived from the signal attenuations of approximately 2400 microwave links in
such a network. Here we give a detailed description of the employed rainfall
retrieval algorithm. Moreover, the documented, modular, and user-friendly
code (a package in the scripting language “R”) is made available, including
a 2-day data set of approximately 2600 commercial microwave links from the Netherlands. The purpose of this paper is to promote rainfall mapping
utilising microwave links from cellular communication networks as an
alternative or complementary means for continental-scale rainfall monitoring.
Introduction
Accurate rainfall observations with high spatial and temporal resolution are
needed for hydrological applications, agriculture, meteorology, weather
forecasting, and climate monitoring. However, there is a lack of accurate
rainfall information for the majority of the land surface of the earth,
notably from ground-based weather radars . Moreover,
the number of reporting rain gauges is dramatically declining in Europe,
South America, and Africa. report a decline of
approximately 50 % in the period 1989–2006 for GPCC, version 5.0.
Satellites are often the only source of rainfall information. Despite their
increasing coverage and spatio-temporal resolution, measurement errors and
sampling uncertainties limit the stand-alone applicability of satellite
rainfall products
e.g..
This calls for alternative and complementary sources of rainfall information.
Since 2006 various studies have shown that microwave links from operational
cellular communication networks may be used for rainfall monitoring for
various networks and climates
e.g..
The ability to observe other types of precipitation, such as snow, is limited
however.
Left: illustration of a telephone tower. The electromagnetic signals
transmitted from the directional antenna of one cellular communication tower
to another are attenuated by rainfall. Right: map of the Netherlands with
locations of the employed link paths (1527) from the cellular communication
network for 9 September, 16:00 UTC–11 September, 08:00 UTC (2011). These
were part of one network of one of the three providers in the Netherlands.
The white circles show the locations of the two weather radars operated by
the Royal Netherlands Meteorological Institute (KNMI).
A link is defined as a radio connection from one telephone tower to another
telephone tower, whereas a link path describes the path between two telephone
towers. Many links are full duplex, i.e. those links measure in two
directions over the same link path, in which case we have (according to our
definition) two links but only one link path. The basic principle of
rainfall estimation using microwave links is as follows. Rainfall attenuates
the electromagnetic signals transmitted from the directional antenna of one
telephone tower to another (Fig. ). The received power
at one end of a microwave link as a function of time is stored operationally
by communication companies to monitor the quality of their networks. From the
decrease in received signal level with respect to the reference signal level,
being representative of dry weather, the rainfall-induced path-integrated
attenuation can be calculated. This can be converted to an average rainfall
intensity over the path of a link. Rainfall estimates from networks of
individual links could in turn potentially be employed to create near
real-time rainfall maps. This is particularly interesting for (developing)
countries where few surface rainfall observations are available. For
instance, , who report on a Rain Cell Africa workshop held
in Burkina Faso in March 2015, clearly demonstrate the relevance and interest
to accelerate the uptake of this new measurement technique on the African
continent. Despite the sympathy from scientists and representatives of
meteorological services and telecommunication companies, to date no
user-friendly computer code for microwave link data processing and rainfall
mapping has been made publicly available. That is the motivation of this
paper.
Based on a 12-day data set
have shown that country-wide (≈ 35 500 km2)
15 min rainfall maps can be derived from received signal powers of microwave
links in a cellular communication network. Hence, further upscaling this
novel source of rainfall information is the logical next step toward
continental-scale rainfall monitoring. The underlying rainfall retrieval
algorithm is briefly described in and largely based on
. The purpose of this paper is to provide a detailed
description of the algorithm employed by and the
corresponding computer code, which is needed for a successful implementation
by potential users. Moreover, sensitivity analyses are performed with respect
to two threshold values of the wet–dry classification method and the outlier
filter threshold value. Finally, the transferability of the code to other
networks and climates is extensively discussed.
The code is freely provided as R package “RAINLINK” on
GitHub
https://github.com/overeem11/RAINLINK
. It contains a
working example to compute link-based 15 min rainfall maps for the entire
surface area of the Netherlands for 40 h from real microwave link data as
used in . This is a working example using actual data from
an extensive network of commercial microwave links, for the first time in the
scientific literature, which will allow users to test their own algorithms
and compare their results with ours. Note that link data are utilised in a
stand-alone fashion to obtain rainfall maps; i.e. data from rain gauges,
weather radars, or satellites are not combined with the link data.
The basic theory of rainfall estimation employing microwave links has already
been described, e.g. in , , and . The core
of the rainfall retrieval algorithm consists of the (path-averaged) rainfall
intensity R (mm h-1) being estimated from microwave (path-averaged)
specific attenuation k (dB km-1) using a power-law R–k relation
:
R=akb.
Coefficients a (mm h-1 dB-b kmb) and exponents b (-)
depend mainly on link frequency. The rainfall retrieval algorithm consists of
the following steps: (1) preprocessing of link data; (2) wet–dry
classification; (3) reference signal determination; (4) removal of outliers due
to malfunctioning links; (5) correction of received signal powers; and (6) computation of mean path-averaged rainfall intensities. Below attention is
given to the main retrieval issues associated with link-based rainfall
estimation.
Received signal powers occasionally decrease during non-rainy periods,
resulting in non-zero rainfall estimates, e.g. caused by reflection of the
beam or dew formation on the antennas (see for an
overview). A reliable classification of wet and dry periods is needed to
prevent this rainfall overestimation. Different classification methods have
been proposed, some of which can be applied to received powers or signal
attenuations when they are sampled at very high frequencies (often for
research purposes), e.g. every 6 or 30 s or even at 20 Hz. For instance, present a spectral time series analysis,
and Markov switching models. Here the so-called “nearby
link approach” is employed termed “link approach” in these
papers, which was derived for application to
minimum received powers over a time interval (15 min in
). A common operational sampling strategy for
commercial microwave links is to obtain a minimum and maximum received power
per 15 min interval . Hence,
methods designed for frequently sampled attenuation data cannot be applied to
data from such operational networks. However, the nearby link
approach cannot be applied if spatial link densities are too low.
provide more information on different methods of wet–dry
classification and reference signal determination. Sometimes powers are
sampled every second , every minute
, or only once or twice every 15 min
. Other approaches to identify rainy and non-rainy spells
make use of auxiliary sources but are not considered in this paper. For
instance, radar data have been utilised in the “radar approach”
, and geostationary satellite data in the “satellite
approach” . An advantage of the nearby link approach
is that it solely employs link data; i.e. it does not depend on auxiliary
data.
Attenuation due to wet antennas gives rise to overestimation of rainfall and
needs to be compensated for
.
The sampling strategy is also an important error source, e.g. one sample
every 15 min leads to large deviations due to unresolved rainfall variability
.
This paper is organised as follows. First a description of the required
microwave link data is given. Next, the rainfall retrieval algorithm and the
interpolation methodology are described. The results section illustrates the
different steps of the rainfall retrieval algorithm including rainfall
mapping. Also sensitivity analyses of parameters of the algorithm are
provided, as well as a comparison between the performance of two
interpolation methods. Next, the algorithm and its applicability to other
networks and regions are discussed. Finally, conclusions are provided.
DataMicrowave link data: characteristics and preparations
In order to compute path-averaged rainfall intensities, received signal
powers were obtained from Nokia microwave links in one of the national
cellular communication networks in the Netherlands, operated by T-Mobile NL.
The minimum and maximum received powers over 15 min intervals were provided,
based on 10 Hz sampling. The transmitted power was almost constant. Here the
data have a resolution of 1 dB, and the majority of these Nokia links used
vertically polarised signals. The data format required by the code is given
in Appendix .
Data from the working example were obtained from 9 September, 08:00 UTC, to
11 September, 08:00 UTC (2011), to estimate rain (Overeem et al.,
2016a). Figure shows the
locations of the links which can be used to estimate rainfall (on average
2473 links and 1527 link paths over all 160 time intervals of 15 min). Data
from another day are used to illustrate the rainfall retrieval algorithm, but
these data are not released with this paper. In addition, a 12-day validation
data set, which includes the data from the working example, is used for
sensitivity analyses of parameters of the rainfall retrieval algorithm. This
data set is from June, August, and September 2011. and
provide more information on the characteristics of
microwave links from this 12-day data set. All link data are from an
independent validation data set; i.e. they have not been used to calibrate
the rainfall retrieval algorithm.
Gauge-adjusted radar rainfall depths
use a gauge-adjusted radar data set with a spatial
resolution of approximately 0.9 km2 and a temporal resolution of 5 min
to calibrate the microwave link rainfall retrieval algorithm. Here this radar
data set, from another period than the calibration period, is utilised to
validate link-based rainfall maps. More information on the derivation of this
data set can be found in . The
data set is freely available at the climate4impact portal (Overeem et al.,
2016b).
Methodology
This section describes the entire processing chain from received signal
powers to rainfall maps. The code is provided via GitHub as the
R
http://www.r-project.org/
package called “RAINLINK” (version
1.11)
https://github.com/overeem11/RAINLINK
, which is distributed
under the terms of GNU General Public License version 3 or later. Table gives an overview of the (sub)functions needed for rainfall
retrieval and mapping. Table provides an overview of variables used in (sub)functions in the rainfall retrieval
algorithm. Table shows the parameters used in the rainfall
retrieval algorithm and their default values. First, preprocessing of link
data is performed (Appendix ).
Overview of functions needed for processing link data
from received signal powers to rainfall maps. Italicised (sub)functions are
optional. A choice has to be made between bold subfunctions. The script
“Run.R” can be employed to determine which (sub)functions are being
applied.
StepFunctionSubfunctionDescription1PreprocessingMinMaxRSL–Preprocessing of linkdata2WetDryNearbyLinkApMinMaxRSLWet–dry classification with nearby link approach3RefLevelMinMaxRSL–Reference signal level determination4OutlierFilterMinMaxRSLa–Remove outliers5CorrectMinMaxRSLCorrection of received signal powers6RainRetrievalMinMaxRSLCompute mean path-averaged rainfall intensitiesMinMaxRSLToMeanRConvert minimum and maximum to mean rainfall intensities7InterpolationInterpolate path-averaged rainfall intensitiesIntpPathToPointCompute path-averaged rainfall intensities for unique link pathsAssign path-averaged intensities to points at middle of link pathsClimVarParamCompute values of sill, range, and nugget of spherical variogrammodelOrdinaryKrigingInterpolate link rainfall intensities by ordinary kriging usingassigned values of sill, range, and nugget of spherical variogrammodel from ClimVarParam or by manually supplying them asfunction argumentsIDWApply inverse distance weighted interpolation8RainMapsLinksTimeStepLink rainfall maps for time interval of link dataRainMapsRadarsTimeStepGauge-adjusted rainfall maps for time interval of link dataRainMapsLinksDailyDaily link rainfall maps from link data at given time intervalRainMapsRadarsDailyDaily gauge-adjusted rainfall map for specified radar filePolygonsMake data frame for polygonsToPolygonsRainValues of rainfall grid are assigned to polygonsReadRainLocationbExtract (interpolated) rainfall depth for supplied latitude andlongitude9PlotLinkLocationsPlot a map with the link locations
a Outliers can only be removed when “WetDryNearbyLinkApMinMaxRSL” has been run.b This subfunction can also be used as a function to extract
(interpolated) rainfall depths from a data frame of (interpolated) rainfall
values for supplied latitude and longitude.
Most important variables used in the (sub)functions of the rainfall retrieval algorithm.
Name in (sub)functionSymbol in textUnitDescriptionaamm h-1 dB-b kmbCoefficient of R–k power lawAmaxAmaxdBMaximum rain-induced attenuationAminAmindBMinimum rain-induced attenuationbb–Exponent of R–k power lawDateTimeNAUTCDate and timeDryNA–Should interval be considered dry forreference level determination? (0 = wet; 1 = dry)FFdB km-1 hComputed for filter to remove outliersFrequencyfGHzMicrowave frequencyIDID–Unique link identifierPathLengthLkmPath lengthRmean〈R〉mm h-1Path-averaged rainfall intensityPminPmindBMinimum received powerPminCorPminCdBCorrected minimum received powerPmaxPmaxdBMaximum received powerPmaxCorPmaxCdBCorrected maximum received powerPrefPrefdBReference levelXStartNA∘ (km)Longitude (or easting) of start of microwave linkXEndNA∘ (km)Longitude (or easting) of end of microwave linkYStartNA∘ (km)Latitude (or northing) of start of microwave linkYEndNA∘ (km)Latitude (or northing) of end of microwave link
Values of the parameters used in the rainfall
retrieval algorithm. All these parameter values can be modified. The
configuration file “Config.R” can be utilised to load all parameter
values.
Variable descriptionSymbol and unitValueDependent onWet–dry classification Radiusr (km)15Spatial correlation of rainfallMinimum number of available3(surrounding) linksNumber of previous hours over which– (h)24max(Pmin) is to be computed (alsodetermines period over which cumulativedifference F of outlier filter is computed)Minimum number of hours– (h)6needed to compute max(Pmin)Thresholdmedian(ΔPL) (dB km-1)-0.7Spatial correlation of rainfallThresholdmedian(ΔP) (dB)-1.4Spatial correlation of rainfallThreshold (step 8 in Appendix )– (dB)2Reference signal level Period over which reference level is– (h)24to be determinedMinimum number of hours that should– (h)2.5be dry in preceding periodOutlier filter Outlier filter thresholdFt (dB km-1 h)-32.5Malfunctioning of linksRainfall retrieval Wet antenna attenuationAa (dB)2.3Rainfall intensity, number of wet antennas,antenna coveraCoefficientα (-)0.33Time variability of rainfallbCoefficient of R–k power lawa (mm h-1 dB-b kmb)3.4–25.0Drop size distribution, frequencycExponent of R–k power lawb (-)0.81–1.06Drop size distribution, frequencyc
a Here Aa is fixed.b Here α is fixed.c To some extent also on polarisation,
temperature, drop shape, and canting angle distribution (this has not been
taken into account).
Here values have been computed from one data set of
measured drop size distributions (p. 65 in ).
Classification of wet and dry periods: the nearby link approach
In order to define wet and dry periods, it is assumed that rain is correlated
in space and hence that several links in a given area should experience a
joint decrease in received signal level in the case of rain. A time interval
is labelled as wet if at least half of the links in the vicinity (default
radius is 15 km, but this can be modified to better match other time
intervals) of the selected link experience such a decrease. A detailed
description of this classification algorithm can be found in
Appendix .
Note that this processing step is optional. The user can also decide not to
apply a wet–dry classification, which may be the only option in areas with
low spatial link densities.
Determination of reference signal level
The performed classification of rainy and non-rainy time intervals serves two
purposes: (1) it allows for determining an accurate reference signal level or
base level, which needs to be representative of dry weather; (2) it prevents
non-zero rainfall estimates during dry weather.
The reference signal level Pref is computed for each link and time
interval separately from the minimum and maximum received signal powers
(dBm), Pmin and Pmax respectively:
P¯=Pmin+Pmax2 (in dBm) is computed for each
time interval classified as dry in the previous 24 h (including the
present time interval);
Pref is the median of P¯ over all dry time intervals. If the
number of dry time intervals represents less than 2.5 h over the previous
24 h, Pref, and hence the rainfall intensity, is not
available and so not computed.
If no wet–dry classification has been applied, the reference level is
determined over all time intervals in the previous 24 h. The periods of
2.5 and 24 h are the default values and can be modified.
Filter to remove outliers
Malfunctioning links can cause outliers in rainfall retrievals. These
outliers can be removed by using a filter that is based on the assumption
that rainfall is correlated in space. The filter discards a time interval of
a link for which the cumulative difference between its specific attenuation
and that of the surrounding links (i.e. within a default radius of 15 km)
over the previous 24 h (default value; including the present time interval)
becomes lower than the outlier filter threshold (dB km-1 h-1). This
criterion is applied to specific attenuation derived from uncorrected minimum
received power . Imagine that the default value of 32.5 dB km-1 h-1 is uniformly distributed over all time intervals in a 24 h
period. This implies a maximum specific attenuation of approximately 1.35 dB km-1 (32.5 dB km-1 h-1 divided by 24 h) per time interval. This
corresponds to a daily rain accumulation of approximately 120 mm for a 38.9 GHz link
and 750 mm for the least sensitive, 13 GHz, link. Hence, a time
interval of a chosen link will only be discarded if the rainfall amounts
during the previous 24 h period are substantial. It is therefore highly
unlikely that this filter would discard real rain.
The value of the cumulative difference, F, is computed as follows:
F=∑t=-24 h+Δt0ΔPL,tS-median(ΔPL,t)Δt,
where t is the time interval, t=0 being the present time interval for
which F needs to be computed, and Δt is the time interval in hours
(0.25 h in the working example). A link is not used to estimate rainfall if
F<Ft. Note that ΔPLS, median(ΔPL), and F are computed in the nearby link approach and are
based on the minimum received powers, the superscript S referring to
the selected link for which rainfall is to be computed. Running the outlier
filter is optional.
Correction of received powers
Subsequently, corrected minimum (PminC) and maximum (PmaxC) received powers are computed for each time interval.
PminC=Pminifwet AND Pmin<Pref,Pref otherwise PmaxC=PmaxifPminC<Pref AND Pmax<Pref,Pref otherwise
In case of no wet–dry classification the time interval of Pmin is always considered wet.
Values of coefficients in the relationship to convert specific
attenuation to rainfall intensity for
frequencies ranging from 6 to 50 GHz. The grey-shaded area denotes the
37.0–40.0 GHz range. Note the logarithmic vertical scale in the left
figure. Here values have been computed from one data set of measured drop
size distributions (p. 65 in ; solid lines). The values
recommended by the International Telecommunication Union ,
meant for computing specific attenuation for given rain rates and for
worldwide application, are also plotted (dashed and dotted lines).
Computation of path-averaged rainfall intensities
Here the path-averaged rainfall intensities are computed from the corrected
minimum and maximum received signal powers. The minimum and maximum
rain-induced attenuation are calculated for each link and time interval
using
Amin=Pref-PmaxC,Amax=Pref-PminC.
Next, the minimum and maximum path-averaged rainfall intensities are computed:
〈Rmin〉=aAmin-AaLH(Amin-Aa)b,〈Rmax〉=aAmax-AaLH(Amax-Aa)b,
with H the Heaviside function (if the argument of H is smaller than 0,
H=0; else H=1). Aa is meant to correct for attenuation due
to wet antennas (dB) and assumed to be constant, e.g. independent of rain
rate and frequency. The coefficients a (mm h-1 dB-b kmb) and
b (-), provided in a file on GitHub, are valid for vertically polarised
signals (Fig. ), which will usually be employed for
microwave links. Utilising these coefficients for horizontally polarised
signals will generally only produce small errors in the retrieved rainfall
estimates. The values recommended by the International Telecommunication
Union , meant for computing specific attenuation for given
rain rates and for worldwide application, are also plotted. Differences up to
10 % are found for the value of the exponent b from ITU (dashed and dotted
lines) compared to that obtained from drop-size distribution data from the Netherlands (solid lines).
For the link frequencies employed in this study (between 12.8 and 40.0 GHz)
the value of the exponent b is close to 1 (Fig. ,
right). , , and
show that this near-linearity only leads to small errors
in rainfall estimates.
The assumed temporal sampling strategy only provides a minimum and maximum
received power over a given time interval. The goal is to obtain a reliable
mean path-averaged rainfall intensity over this time interval. This is
achieved by computing the mean path-averaged rainfall intensity as a weighted
average:
〈R〉=α〈Rmax〉+(1-α)〈Rmin〉,
where α is a coefficient that determines the relative contributions of
the minimum (〈Rmin〉) and maximum (〈Rmax〉) path-averaged
rainfall intensity (mm h-1) during a time interval. The values for
Aa (2.3 dB) and α (0.33) have been taken from
, who use a 12-day calibration data set from June and July
2011 (which has not been used for validation). They compare daily link-based
rainfall depths with gauge-adjusted radar retrievals to calibrate the
rainfall retrieval algorithm.
The figure illustrates how path-averaged link rainfall depths are interpolated to link rainfall maps.
First, path observations (left panel) are assigned to points (centre panel).
Next, ordinary kriging is applied to obtain rainfall maps (right panel). For time interval
20:30–20:45 UTC on 10 September 2011.
Rainfall maps
Path-averaged rainfall intensities from microwave links are spatially
interpolated to obtain rainfall maps. The user can choose to apply ordinary
kriging (OK) employing a spherical variogram model
or to apply inverse distance weighted (IDW)
interpolation on link rainfall data. OK and IDW are well suited for dealing
with heterogeneously distributed data locations. OK requires a variogram
model, but unfortunately it is impossible to robustly estimate such
variograms for each time interval separately due to the sparsity of rainfall.
Hence, a more robust procedure is followed. The parameter values can be
supplied by the user or they can be computed as follows. The sill and range
of an isotropic spherical variogram model have been expressed as a function
of day of year (DOY) and duration (1–24 h) using a 30-year rain gauge data
set from the Netherlands . This data set does not
overlap with the link data set. If needed the relationships can be
extrapolated to time intervals shorter than 1 h. The nugget is set equal to
0.1 times the sill. Note that these equations and the optimal values of their
coefficients have been found to be useful for the Dutch climate. Hence, they
may need adjustment for other climatic settings. See Appendix
for a detailed description of the interpolation
algorithm.
Figure shows an example of the interpolation
procedure for a given 15 min time interval. The left panel shows the
locations of the microwave links, where the colour denotes the rainfall
depth. Next, these path-averaged rainfall depths are assigned to the middle
of the link, i.e. considered as point measurements (centre panel). This is
done to simplify the interpolation procedure. The right panel shows the
corresponding interpolated rainfall map (OK). Interpolated rainfall maps can
be visualised by using the functions provided in the package. This is
described in more detail in Appendix .
From received signal powers to cumulative rainfall depths (one day,
one link): minimum and maximum received powers and path-averaged,
gauge-adjusted radar rainfall intensities (a); corrected minimum and
maximum received powers, reference signal levels, and path-averaged,
gauge-adjusted radar rainfall intensities (b); minimum and maximum
path-averaged link rainfall intensities (c); mean path-averaged link
and gauge-adjusted rainfall intensities (d); cumulative
path-averaged link and gauge-adjusted radar rainfall depths (e); map
with location of microwave link in the city centre of Amsterdam, the
Netherlands (f). Period is 30 August, 08:00 UTC–31 August, 08:00 UTC
(2012).
Fifteen-minute rainfall maps from 10 September 2011, 20:30–20:45 UTC,
for links only (left) and radars plus gauges (right) for the Netherlands.
Spatial resolution is approximately 0.9 km2. Values below 0.1 mm are not
shown. The lines denote the locations of the employed microwave links (left).
This figure is part of the working example.
Daily rainfall maps for links only (left) and radars plus gauges
(right) for the Netherlands. Spatial resolution is approximately 0.9 km2.
Period is 10 September, 08:00 UTC–11 September, 08:00 UTC (2011). Values below
1.0 mm are not shown. The lines denote the locations of the employed
microwave links (left). This figure is part of the working example.
ResultsIllustration of steps in rainfall retrieval algorithm
Here the steps in the rainfall retrieval are illustrated. Starting points are
the minimum and maximum received powers over a given time interval (15 min in
this case), as shown in Fig. a for one
link during 24 h. As expected, there is a strong negative correlation
between the minimum received powers and the path-averaged gauge-adjusted
radar rainfall intensities, considered to be the ground truth. Next, the
corrected received powers and the reference level are shown (Fig. 4b). In this
particular example the received powers are hardly corrected. A decrease
around 08:30 UTC, probably not related to rainfall, is corrected for. In
contrast, a rain-induced decrease just before 17:00 UTC is removed. The mean
path-averaged link rainfall intensities (Fig. 4d) and the cumulative link rainfall
depths (Fig. 4e) correspond well with the gauge-adjusted radar-based values. In total
approximately 60 mm was observed, resulting from two convective events with
tens of millimetres in a couple of hours.
Illustration of plotting capability of R visualisation function
“RainMapsLinksTimeStep” in package RAINLINK: 15 min rainfall map from 10
September 2011, for links only for Amsterdam, the Netherlands. Spatial
resolution is approximately 0.9 km2. Values below 0.2 mm are not shown.
The lines denote the locations of the employed microwave links. The number at
the red cross is the rainfall depth at that location, of which the name is
provided in the title caption. This figure is part of the working example.
Sensitivity analyses of the threshold values for the wet–dry
classification. The 2-D contour plots display relative mean error (%; black
lines) and, in colours, CV (left) or ρ2 (right) in path-averaged
15 min rainfall depths as a function of median(ΔP) and
median(ΔPL). The default threshold values are
indicated by the black dot. No outlier filter has been applied.
Rainfall mapping
Link rainfall maps are compared to gauge-adjusted radar rainfall maps in
Figs. and for 15 min and
daily rainfall accumulations respectively. These are obtained using ordinary
kriging with a climatological spherical variogram model. The left panels show
the link-based rainfall maps with the locations of the employed links for the
nearby link approach with outlier filter. The radar rainfall maps are given
in the right panel. The cellular communication network is able to detect the
rainfall patterns for the 15 min interval, although deviations are found with
respect to radars combined with rain gauges. Note that some large areas do
not have link data. The daily rainfall maps are from 10 September 2011, 08:00 UTC, to 11 September 2011, 08:00 UTC, and reveal link-based rainfall depths
larger than 26.0 mm. These local high rainfall depths occurred in a few
hours, i.e. convective rainfall. In general, the link network is able to
correctly determine the spatial rainfall patterns. These examples demonstrate
a successful application of microwave links to estimate rainfall. Figure illustrates the capability of the rainfall
mapping functions to produce a (local) rainfall map of high graphical
quality.
Sensitivity analyses
For all sensitivity analyses the same values for Aa (2.3 dB) and
α (0.33) are employed, i.e. as obtained for the nearby link approach
with outlier filter using the default parameter values (Table ).
Validations are performed on 15 min path-averaged link rainfall depths or
link rainfall maps from the 12-day validation data set .
No threshold values regarding the minimum rainfall depths are applied in the
comparisons. Metrics are computed for residuals, i.e. the link minus the
gauge-adjusted radar rainfall depths. The sensitivity analyses are based on
12 rainy days from the summer. Such analyses may yield different results for
other seasons and rainfall types.
Nearby link approach wet–dry classification
A sensitivity analysis is performed for the two threshold values of the
wet–dry classification, where median(ΔP) is varied from -5 to
0 dB with a step size of 1 dB (because the resolution of the employed data
set is 1 dB), and median(ΔPL) is varied from -2 to 0 dB km-1, with a step size of 0.1 dB km-1. Although these threshold
values could potentially become positive, this will hardly happen or only
small positive values will be obtained. If both threshold values are 0 dB (km-1) this can therefore be considered representative for the situation
without wet–dry classification.
The 2-D contour plots in Fig. display relative
mean error (%; black lines) and coefficient of variation (CV) or
squared correlation coefficient (ρ2) (colours) in path-averaged 15 min
rainfall depths as a function of median(ΔP) and
median(ΔPL). The default threshold values are
indicated by the black dot. Given the resolution of 1 dB, the default value
for median(ΔP), -1.4 dB, implies that median(ΔP) should be lower than -1 dB (hence the black dot is plotted at 1 dB). In
order to focus on the performance of the wet–dry classification with respect
to the situation without wet–dry classification, the outlier filter has not
been applied here. A wet–dry classification generally leads to a clear
improvement in terms of ρ2 and CV upon a situation without
wet–dry classification.
The contour plots show that application of median(ΔPL)
is not necessary and that the best results are obtained for
median(ΔP) values around -4 dB, i.e. much lower than the
default value. Note that the default values of Aa and α have been
used, where an outlier filter was applied. This may be the reason for the
variability in relative mean errors. Finding optimal values of Aa and
α for every combination of median(ΔP) and
median(ΔPL) was considered too computationally
demanding and could have compensated for errors in wet–dry classification
for each combination of both threshold values.
The nearby link approach with outlier filter and default settings for the
parameter values (Table ), gives a CV of the residuals
of 3.84, a squared correlation coefficient ρ2 of 0.54, and a relative
bias in the mean of 10.5 % compared to the reference: gauge-adjusted,
path-averaged radar rainfall depths. Removing step 8 (Appendix ) from the nearby link approach does hardly change these
validation results. Therefore, this step can now also be discarded by
supplying a function argument. In this step the previous two time intervals
and the next time interval are classified as wet if the present time interval
is rainy and has more than 2 dB of attenuation for the link under
consideration.
Sensitivity analyses of the threshold value for the outlier filter.
Performance in terms of ρ2, CV, and relative mean error (%) as a
function of outlier filter threshold for path-averaged 15 min rainfall depths. The grey vertical line indicates the
default threshold value chosen for the results in this paper. Also shown is
the fraction of data which is left after applying the outlier filter for a
chosen threshold value.
Outlier filter
Performance of link rainfall estimates clearly deteriorates if the outlier
filter is not applied. The CV of the residuals is much larger, 6.03,
the ρ2 is much lower, 0.35, and the relative bias in the mean becomes
much larger, 24.2 % (compare to previous paragraph). The latter may be
related to the fact that Aa and α have been calibrated on a
data set where the outlier filter has been applied.
Now the performance for a wide range of threshold values of the outlier
filter is investigated for path-averaged 15 min rainfall depths. For the default threshold value of -32.5 dB km-1 h-1, denoted by the grey line, good results are obtained in terms of ρ2
and CV (Fig. ). Results improve for
increasing threshold values at the expense of a severe decline in the amount
of link data. The best choice for a threshold value is somewhat arbitrary,
i.e. a range of values gives good results, while still having many link data.
The relative mean error decreases from more than +20 % to less than -15 %
from Ft=-300 to 0 dB km -1 h-1. If for each value of
Ft the coefficients Aa and α would have been
calibrated, the relative mean error would be expected to be more constant.
Apparent are the jumps in ρ2 and CV for certain values of
Ft. It seems that these are related to some very large “rainfall
depths”, which point to malfunctioning links. Note that F is the
cumulative difference between a link's specific attenuation and that of the
surrounding links. Hence, such jumps for strongly negative values of
Ft are likely not caused by rain or (sources of) errors affecting a
whole region.
Performance of ordinary kriging versus inverse distance weighted interpolation
Here the 12-day data set of 15 min path-averaged link rainfall depths is
employed to obtain interpolated 15 min rainfall maps (0.9 km2), which are
aggregated to daily rainfall maps (0.9 km2). The nearby link approach with
outlier filter and default parameter values has been applied. These maps are
computed for two interpolation methodologies: (1) OK with a
spherical climatological variogram model, i.e. the default method; (2) IDW interpolation with an inverse distance weighting
power of 2.0.
First, the 15 min rainfall maps are studied. The following metric values are
obtained for IDW: relative mean error of 8.3 %, CV of 3.30, and a
ρ2 of 0.41. OK performs better, since the relative mean error is
clearly lower (1.5 %), CV is similar (3.29), and ρ2 is clearly
higher (0.48). Despite the high spatial resolution of the obtained rainfall
maps (0.9 km2) compared to the link density, still reasonable results are
obtained in terms of ρ2, but values of CV are high.
Next, the daily rainfall maps are investigated. The following metric values
are obtained for IDW: relative mean error of 8.2 %, CV of 0.51, and
a ρ2 of 0.70. OK performs somewhat better, since the relative mean
error is clearly lower (1.4 %), CV is slightly larger (0.54), and
ρ2 is somewhat higher (0.73). Hence, the improvement of OK with respect
to IDW is particularly evident at the 15 min scale. Note that the results for
the daily timescale are much better than those for the 15 min timescale.
The performance of IDW could become better if another value of the inverse
distance weighting power would be selected.
DiscussionComputation time
Using one processor (Intel i7; Linux operating system) the entire processing
up to and including the interpolation takes around 10 min, i.e. to obtain
15 min link-based rainfall maps for 40 h based on data from on average
2381 links and 2473 links in total (representing 1527 unique link paths).
Not applying a wet–dry classification leads to a decrease in computation time
of 1 min. Most time is consumed by the OK interpolation (8 min). The
IDW interpolation only takes 2 min (for an inverse distance weighting
power of 2.0). A similar amount of time would be needed in case
multiple processors would be used to run multiple periods, each processor
running its own period. In order to reduce computational time it could also
be worthwhile to divide a region into subregions, particularly to speed up
the wet–dry classification. Another option would be to transfer the algorithm
to a high-level programming language.
Transferability of code
The developed rainfall retrieval algorithm or its parameter values will
likely need adaptation for (optimal) rainfall estimation for other networks
and climates around the world. Nevertheless, many networks have similar
characteristics as those in the Netherlands. A 15 min sampling strategy,
either instantaneous or minimum and maximum values, is common and has been
used in several other networks
e.g..
We advocate a pragmatic approach to first apply this algorithm to data from
those networks with minimum and maximum received signal levels and assess the
quality of the derived rainfall maps. This is certainly relevant for areas
(e.g. developing countries) where few reference rainfall data are available
to calibrate the rainfall retrieval algorithm. Suboptimal parameter values or
interpolation methods may still provide meaningful rainfall estimates for
other networks or climates.
As a next step the parameters of the algorithm could be adapted to local
conditions, for instance based on recommendations from the International
Telecommunication Union . Then it could be decided whether
further modifications of the algorithm would be needed or not. For poorly
gauged regions we suggest optimising the algorithm, including interpolation
methodology, employing data from a region with a similar climate and network,
for which sufficient ground-truth rainfall data are available.
Coefficients and thresholds have been optimised for 15 min intervals; hence
their values may not be appropriate for other time intervals. Their values
can be easily changed by modifying function arguments.
Even if network characteristics are very different, e.g. in terms of sampling
strategy, large parts of our code could still be used to develop algorithms
suited for the specific needs of these networks. Although the rainfall
retrieval algorithm contains several empirical parameters, the developed
methods are not merely statistical in nature. Their general principles hold
for other networks as well. For instance, the principle of the wet–dry
classification, where data from surrounding links are used to distinguish
between wet and dry periods, makes use of the general fact that rainfall is
correlated in space, although decorrelation distances can vary between
different climates.
In case of very different sampling strategies, e.g. mean or instantaneous
received signal powers for shorter or longer time intervals, the code should
be modified. Although many networks have constant transmitted powers
, other
networks may not operate with constant transmitted powers. For links using
Automatic Transmit Power Control (ATPC) the transmitted power can become
higher in case of a reduced received power at the end of the link, to
compensate for large losses along the link path. In this case the transmitted
power also needs to be known and should be taken into account in the rainfall
retrieval algorithm . Note that a sampling
strategy where minimum and maximum transmitted and received powers are
available over a chosen time interval may result in additional errors in case
of ATPC. This is because the timing of minimum received and transmitted
powers does not necessarily coincide, which also holds for the maximum
received and transmitted powers.
Ideally, the employed values for a and b should match the polarisation of
the link. If information on the latter is available, the code could be easily
adapted to incorporate this.
Parameter values
Table gives an overview of the parameters of the rainfall
retrieval algorithm, their default values, and the factors influencing them.
This can help to assess which parameters will change for other regions and
networks. Some parameters are already modelled as function of frequency,
others do not seem to be sensitive to frequency, whereas again others should
ideally be optimised for different frequencies.
Wet–dry classification with nearby link approach
Some of the signal fluctuations that occur in dry weather may also occur
during rainy periods. The algorithm does not correct for such errors.
Furthermore, nearby links may suffer simultaneously from signal fluctuations
not caused by rainfall, resulting in a poor performance of the nearby link
approach.
The parameters concerning the wet–dry classification have been optimised
using data from another period and are based on received signal level data
stored at 0.1 dB resolution . These values have been
applied in and in the current manuscript to an
independent data set from another brand of links with slightly different
antenna covers and a coarser 1 dB power resolution. The sensitivity analysis
shows that the parameter values from are suboptimal but
give a clear improvement in link-based rainfall estimates compared to the
case without wet–dry classification.
The chosen radius depends on the spatial correlation of rainfall and is,
hence, not frequency dependent. Because networks are designed in such a way
that ΔP will be similar for different microwave frequencies, lower
frequencies are utilised for longer link paths and vice versa. Lower
frequencies experience less rainfall-induced specific attenuation compared to
higher frequencies. Hence, median(ΔP) is nearly independent of
frequency.
Ideally, a sensitivity study should be carried out to find optimal threshold
values for the wet–dry classification in case of other networks and climates.
This will be computationally expensive in case of large data sets.
It is to be expected that the nearby link approach will also work for other
temporal resolutions than 15 min. This may require a smaller radius and a
higher link density for higher temporal resolutions and may allow a larger
radius and lower link density for lower temporal resolutions. For data
sampled at very high frequencies, i.e. 1 s, the nearby link approach may
become too computationally expensive. This could be circumvented by first
averaging signal attenuations over longer durations before applying this
approach.
Reference level determination
Note that a default 24 h period is considered for determining the reference
level and for one step in the nearby link approach. Such a relatively long
period is chosen, among other reasons, to increase the probability to obtain
at least 2.5 h of dry periods, which helps to determine the reference level
more accurately. Note that both periods can be modified.
Outlier filter
The filter to remove outliers deals with specific attenuation; i.e. it does
not explicitly take into account frequency. It is also suitable for other
time interval lengths. Nevertheless, its threshold value is very high, which
makes it unlikely that actual rainfall is filtered out accidentally,
irrespective of the employed frequency. The outliers are likely caused by
malfunctioning links. Perhaps melting precipitation also plays a role.
Hence, it makes sense to apply a frequency-independent threshold value. Note
that the threshold value may need to be optimised for other networks and
climates. The sensitivity analysis on a 12-day data set shows that the chosen
default value of Ft is suitable but a range of values shows a
similar performance.
Wet antenna attenuation and sampling strategy
The wet antenna attenuation correction Aa will not only compensate
for wet antennas. The value of Aa found in the calibration will
also be influenced by other errors. Moreover, in case of rain along the link
path no, one, or both antennas can be wet, whereas this correction is always
applied, and Aa itself has also been optimised on data where the
number of wet antennas can vary. In addition, the correction does not depend
on rainfall intensity. Hence, applying Aa should be seen as a
pragmatic approach towards correcting for wet antennas.
The coefficient α is employed to obtain mean path-averaged rainfall
intensities from minimum and maximum path-averaged rainfall intensities and
is expected to depend on the time variability of rainfall. Note that the
value of α presented here is based on data available at 15 min time
intervals. It is expected that this value will be different for disparate
time intervals. use the known distribution of rainfall
intensities at the point scale to weigh minimum and maximum received signal
levels.
The optimised values of Aa and α are representative of
summer months in the Netherlands and appear to be relatively close to each
other for different frequency classes. Application to data from other months
will generally only lead to a small decrease in performance (not shown).
Hence, application of the existing values of Aa and α to
other rainfall types can still give reasonably good rainfall estimates.
Nevertheless, it is advised to recalibrate Aa and α in
case of other networks or climates. This may be achieved by comparing
link-based rainfall estimates with high-quality (gauge-adjusted) weather
radar data, which may provide full coverage over the network, as in
. Alternatively, path-averaged rainfall intensities may be
estimated from rain gauge data. This may require a dedicated research
experiment , preferably with several rain gauges
along the link path.
utilise a research link with high temporal resolution for
testing the proposed rainfall retrieval algorithm. By applying the rainfall
retrieval algorithm, mean path-averaged rainfall intensities can be derived.
These can be compared to the corresponding true values obtained from the
received signal powers sampled at 10 Hz. Hence, the same instrument, a microwave link, is used
to assess the ability of the retrieval algorithm to deal with the sampling
strategy. Since the same instrument is used representativeness errors are not
present. The probability distribution of minimum and maximum rainfall
intensities could be studied in order to obtain more reliable mean rainfall
intensities. Moreover, such an experiment may also help to assess attenuation
due to wet antennas caused by dew or rain, its dependence on rainfall
intensity or antenna cover type, and the time it takes for antennas to dry
following a rain event. Preferably a rain gauge or disdrometer should be
available near the antenna. A research link is particularly useful when the
microwave frequency, path length, and antenna cover are representative of the
links in a cellular communication network. Finally, see, for instance,
and for other wet antenna
attenuation correction methods.
Interpolation methodology
The employed interpolation methodology, ordinary kriging, may not be
specifically suited for other regions with different rainfall climatologies.
The methodology developed in could be optimised for
other climates. This requires long rainfall time series. The assumed
stationarity and isotropy will often be violated .
However,
violation of assumptions does not automatically imply that the interpolation
method is not useful. Further, the path-averaged link rainfall intensities
are assumed to be point measurements. Hence, it is recommended to improve the
interpolation methodology, e.g. by treating the rainfall values as line
observations instead of as point observations, which is expected to have the
largest impact at local scales for areas with high link densities or at areas
with long links. Using data from the same 12 days as employed in
, find that link rainfall retrieval
errors themselves are the source of error that contributes most to the
overall uncertainty in rainfall maps from a commercial link network. Errors
due to mapping, i.e. interpolation methodology and link density, play a
minor, albeit non-negligible role for the same network as utilised in this
study. Hence, despite the limitations of the interpolation methodology its
usefulness has been confirmed . Further,
study the performance of simulated link rainfall maps
as a function of link density. They show that even for low spatial link
densities reasonable results can still be obtained.
It may be interesting to apply a tomographic approach in order to obtain
link-based rainfall maps . Such an approach
can potentially reconstruct the two-dimensional distribution of rainfall from
a set of one-dimensional transmission data from many different (nearby or
even intersecting) paths . Hence, it would take the line
character of the attenuation measurements into account, in contrast to the
current approach where a line measurement is assigned to a point at the
middle of the link path.
The estimated sill and range will become less accurate by extrapolating to
time intervals shorter than 1 h, such as 15 min. use
rain gauge data from England to quantify spatial correlation for short time
intervals, e.g. 1 and 15 min. These kind of studies can be useful to improve
the interpolation of link-based rainfall intensities.
R–k relationship
The relationship between path-averaged rainfall intensity and path-averaged
specific attenuation is commonly employed in other studies. The provided
parameter values of a and b are available for frequencies ranging from 1 to 100 GHz. The value of the exponent b is close to 1 for the frequencies
employed in this study, which range from 12.8 to 40.0 GHz. Frequencies between
37.0 and 40.0 GHz are denoted by the grey-shaded area in Fig. , which contains 81 % of the links from the working example,
having values of b very close to 1 (right). For other frequencies the
corresponding value of b often deviates more from 1 (Fig. , right). High rainfall variability along the link path will
lead to overestimation for b<1 and underestimation for b>1.
Although exponents are often not exactly equal to 1, they are much closer to
1 than the ones typically used in radar reflectivity–rain rate relations.
For example, assessed the influence of spatial variability on the
link path for these two frequencies. They show that the under- or
overestimation will generally be small. In the tropics this problem will be
more pronounced because of the high spatial rainfall variability.
Particularly for long links operating at low frequencies (e.g. 7 GHz), this
may lead to overestimation. Also note that obtain quite
accurate rainfall estimates using a 7 GHz microwave link with a length of 29 km in Burkina Faso, having a tropical climate. Previous work has demonstrated
that this error is limited for temperate climates such as experienced in the Netherlands . Moreover, 82 % of the links in
our network have a length shorter than 5 km, even 97 % of the links are
shorter than 10 km. The average link length is only around
3 km.
Conclusions
It has been shown in several studies that microwave links from cellular
communication networks can be used to retrieve rainfall information. For
instance, country-wide (≈ 35 500 km2) 15 min rainfall maps can be
obtained from received signal powers of microwave links .
In this paper a detailed description is given of the algorithm of
. The accompanying code is made publicly available as the R
package RAINLINK via GitHub under the condition of version 3 or later of the
GNU General Public License. The modular programming facilitates users to
adapt the code to their specific network and climate conditions or to only
employ one or some functions of RAINLINK. We hope that RAINLINK will promote
the application of rainfall monitoring using microwave links in poorly gauged
regions around the world.
We invite researchers to contribute to RAINLINK to make the code more
generally applicable to data from different networks and climates. Ideally,
the code should be tested on data sets containing all seasons, for varying
networks and regions. Such an endeavour worldwide is currently difficult to
achieve. It would require an enormous effort and it would also require data
sharing among researchers, which is still not that easy to accomplish due to
confidentiality requirements often imposed by telecommunication companies.
One may wonder whether the technology is bound to disappear due to the
introduction of fibre optical cable networks. For instance, for the provided
link data set the majority of the links does not exist anymore due to network
renewal and deployment of underground fibre optical cable networks. Whereas
the Netherlands is at the forefront internationally concerning deployment of
underground fibre optical cable networks for telecommunication between base
stations, the country is expected to still have several thousands of links
(from cellular telecommunication companies and others) in 2025. For other
countries and continents the uptake of fibre optics will be significantly
slower, lagging behind at least 5–20 years. Moreover, construction of fibre
optical cable networks may not be feasible or economically viable in many
mountainous or rural areas around the world. Therefore, we expect this type
of cellular communication infrastructure to still be around for several
decades worldwide (Ralph Koppelaar, T-Mobile NL, personal
communication, 2016). Why not attempt to use this
existing infrastructure as a complementary source of rainfall information, in
particular in those areas around the world with very few rain gauges, let
alone weather radars?
Although rain gauges, radars, and satellites have been specifically designed
to measure rainfall, all of these instruments face their own challenges. It
is well known that radar rainfall estimates generally deteriorate for longer
ranges from the radar. Geostationary satellite observations have a time
resolution of typically 15 min but are often very indirect (e.g. estimates
through cloud physical properties; ). Low-Earth Orbit
satellites usually have long revisit times. Despite (new) satellite missions,
microwave link data can still become important for ground validation of or
merging with satellite rainfall products. For instance, the IMERG product of
the new GPM mission provides gridded rainfall products every 30 min covering
60∘ N–60∘ S with a spatial resolution of 0.1 ∘. This is certainly a major step forward with
respect to TRMM, but one has to recognise that the rainfall retrieval
algorithm heavily relies on temporal interpolation and, depending on the
product, additional data sources, such as rain gauges, since the actual
satellite revisit time is typically several hours. Moreover, links measure
rainfall close to the ground, which is not the case for weather radar and
satellites, and at spatio-temporal scales relevant for meteorology and
hydrology (typically 1 s–15 min; 0.1–20 km). Even if rain gauges are
present, the number of links will often be an order of magnitude larger than
the number of rain gauges in a region. The larger spatial density of links
has been demonstrated in our previous work to compensate for their lower
accuracy with respect to rain gauges. Hence, rainfall information from
cellular telecommunication networks is promising for hazardous weather
warning, flood forecasting, food production, drought monitoring, etc.
Finally, although it is indeed difficult to obtain transmitted and received
signal level data from telecommunication companies, researchers have managed
to obtain data for a limited, but expanding, number of countries (namely,
Australia, Brazil, Burkina Faso, Czech Republic, France, Germany, Israel,
Kenya, Pakistan, Sweden, Switzerland, and the Netherlands).
To conclude, we feel that merging of rainfall data from different sources (if
available) will often yield the best rainfall estimates. For instance,
satellite data could be used for wet–dry classification to prevent non-zero
link-based rainfall estimates during dry periods . We
believe that the main potential for rainfall estimation using microwave links
is found in areas with few surface rainfall observations. In addition,
development of a merged link-satellite rainfall product seems an interesting
opportunity.
Data availability
The data from the “Radar precipitation climatology” (Overeem et al.,
2016b), i.e. the gauge-adjusted radar rainfall data set, as well as the
“RAINLINK microwave link data set” (Overeem et al., 2016a), are freely
available for all parties. The radar data set can be obtained from
http://climate4impact.eu/impactportal/data/catalogbrowser.jsp?catalog=http://opendap.knmi.nl/knmi/thredds/radarprecipclim.xml.
The link data set can be found at
https://github.com/overeem11/RAINLINK/tree/master/data.
Required data format
The code is designed for estimating rainfall from minimum and maximum
received powers over time intervals of a given length, as that is the way in
which cellular telecommunication companies typically store their data. The
time interval does not have to be an integer but should be equidistant. The
time interval length is automatically computed by the RAINLINK package. For
each link and time interval the following variables are needed:
microwave frequency f (GHz), minimum and maximum received power
Pmin and Pmax (dBm), date and end time of observation
(YYYYMMDDhhmm, i.e. year (2011), month (09), day (11), hour (08), minutes
(00): 201109110800), path length L (km), coordinates (latitude and
longitude) of start and end of link in WGS84 (degrees; default, also another
coordinate system may be chosen), and unique link identifier (ID), which
should remain the same over the entire processed period. A full-duplex link
should have two IDs, one for each link direction. Note that a header should
be provided as given in the data file from the example. The order of the
columns does not matter, as long as the variable name matches the column. IDs
are handled as strings. Hence, not only integers, but, for instance, also
alphanumeric IDs can be used.
A user can supply microwave link data for an arbitrary period. Note that
missing link data are allowed; i.e. the code will work when a time interval has
no data. However, many missing data or a too short period may lead to
rainfall intensities not being calculated.
In this paper data from one network have been utilised. In case data from
more networks, either from the same or from different providers, are
available, these can simply be combined into one data frame. The only
requirements are that unique link identifiers are employed and that these
networks use the same sampling strategy.
Preprocessing of link data
The processing starts with the preprocessing of link data using the function
“PreprocessingMinMaxRSL”, which does the following.
Microwave link data are supplied as function argument.
Select only those links with microwave frequencies in chosen range (here 12.5–40.5 GHz; almost all T-Mobile
NL links used to operate in this range). The chosen frequencies can be
supplied as function arguments.
For each unique link identifier a time interval is removed if it contains more than one record.
If no link data are available anymore for the selected unique link identifier, perform the previous step for the next unique link
identifier.
For each unique link identifier it is checked whether its frequency, link coordinates, or path length vary
during the considered period. If this is the case for one of these variables,
the link is discarded for this particular considered period.
A data frame is provided as output.
Repeat these steps for each unique link identifier.
Wet–dry classification with nearby link approach
A step-by-step description of the classification algorithm (function
“WetDryNearbyLinkApMinMaxRSL”) is given below and mainly obtained from
. The classification is run for each link for the period for which data are provided. Note that running this wet–dry
classification is optional.
The link coordinates are converted to an azimuthal equidistant cartesian coordinate system (easting
and northing of start of link, easting and northing of end of link; km).
Select a link.
All links for which both end points are within a chosen radius (default 15 km) from either end of the
already selected link are selected as well.
Continue if at least three surrounding links have been selected for the considered time interval for which
the link in step 2 has data, otherwise the link is not used for that time interval. If the link is part of a full-duplex link,
the other link is counted as surrounding link.
Calculate the attenuation ΔP=Pmin-max(Pmin) and specific attenuation
ΔPL=Pmin-max(Pmin)L for each
link and each time interval. max(Pmin) is the maximum value of
Pmin over the previous number of hours including the present time
interval (default 24 h). Note that max(Pmin) is only computed if
at least a minimum number of hours of data are available (default 6 h);
otherwise it is not computed and no rainfall intensities will be retrieved.
The median values of ΔP and ΔPL are computed over all selected links for each
time interval.
If median(ΔPL)<-0.7 dB km-1 and median(ΔP)<-1.4 dB the
time interval is classified as wet for the link selected in step 1.
If max(Pmin)-Pmin>2 dB for a given time interval that is classified as wet, the
previous two time intervals and the next time interval are classified as wet
for the link selected in step 1.
All time intervals that have not been classified as wet are classified as dry for the link selected in step 1.
Repeat these steps for all other links.
Note that a radius of 10 km is used in , whereas a
default radius of 15 km is used here and in , which
allows estimating rainfall in areas with lower spatial link densities. The
15 km radius is representative of the decorrelation distance of convective
rainfall in the Netherlands in case of a time interval of 15 min. For the
often occurring stratiform rainfall this distance will be (much) longer. Step
4 has been slightly altered with respect to .
The threshold values in steps 7 and 8 have been obtained from
, who optimise them by visual comparison with the
gauge-adjusted radar data set of path-averaged rainfall intensities employing
data from 2009 (NEC links with 0.1 dB power resolution). Step 8 was used
because rainfall is generally correlated in time, and sometimes very local,
so that it does not occur at surrounding links.
If the algorithm would be rerun for another period, for instance one time
interval later, this can result in different rainfall estimates compared to
the preceding run. This is due to step 8 of the wet–dry classification
methodology, which may also classify the previous 30 min as rainy. In order
to obtain the same rainfall estimates for different runs, the algorithm would
need to be slightly modified or one should wait 30 min before applying the
algorithm. Another option is to not apply step 8, which can be supplied as
function argument. The rainfall retrieval algorithm is suitable for real-time
application, for which the reclassification of previous time intervals is of
no consequence, because rainfall intensity of present time interval is of
interest.
Interpolation methodology
Here an extensive description of the interpolation algorithm is given, which
is carried out by calling the function “Interpolation”. It performs the
following steps.
Convert the supplied interpolation grid with coordinates (longitude and latitude; two columns) to an
azimuthal equidistant cartesian coordinate system. In the example the
interpolation grid is in WGS84 (degrees). A radar grid is used, where the
coordinate of each grid cell represents the middle of the radar pixel. The coordinates of the supplied link data are also converted to an azimuthal equidistant cartesian
coordinate system (easting and northing; km).
Select the mean path-averaged link rainfall intensities for each time interval.
Then it calls the subfunction “IntpPathToPoint”, which does the following:
Compute the coordinates belonging to the middle of the links.
Determine the unique coordinates of the middle of the links.
Calculate the average rainfall intensity for each unique set of coordinates. This implies that data
from full-duplex links are averaged. If another link happens to have the same
middle of the link path, its rainfall intensity is taken into account in the
averaging.
Next, three different interpolation methodologies are available, which can be
chosen by supplying a function argument:
Inverse distance weighted interpolation on link rainfall data (subfunction “IDW”). The inverse
distance weighting power should be supplied as function argument.
Ordinary kriging with spherical variogram model. Its parameter value nugget, sill, and range
can be defined by the user as function arguments.
Ordinary kriging with spherical variogram model with climatological parameter values based on a 30-year
rain gauge data set. These are computed for the DOY as obtained from
the input data frame with microwave link data, thus taking into account
seasonality in spatial rainfall correlation. The subfunction “ClimVarParam”
computes these parameter values.
For the last methodology the spherical variogram parameters can be computed
as follows (power-law scaling in cosine function parameter;
):
r=15.51D0.09+2.06D-0.12cos2π(DOY-7.37D0.22)3654,C=0.84D-0.25+0.20D-0.37cos2π(DOY-162D-0.03)3654,C0=0.1C,
where r is the range (m), C is the partial sill (mm2 h-2),
C0 is the nugget (mm2 h-2), DOY is day of year, and
D is the duration (h), i.e. the time interval of the rainfall intensities
which are to be interpolated. The nugget is basically the semi-variance at
zero distance, which can be interpreted as very-fine scale variability or as
measurement uncertainty. The sill is the variance at very large distances,
and the range is the distance at which the variance does not increase any
more (this is equivalent to the distance beyond which the field is completely
decorrelated, i.e. ρ(r)=0).
The spherical variogram takes the following form:
γ(h)=C32hr-12hr3+C0ifh≤rC+C0ifh>r,
where h is the distance (m).
The interpolation is performed for the provided grid. For ordinary kriging
the 50 (default value) nearest observations are used in order to reduce
computational time (subfunction “OrdinaryKriging”). Negative rainfall
values occur regularly, and are replaced by zero values. Finally, the
function “Interpolation” only gives the rainfall intensity (mm h-1)
as output. Each row corresponds to the same row from the interpolation grid.
Visualisation of rainfall maps
The rainfall maps can be visualised using the functions described for step 8
in Table . Both link and radar rainfall maps can be obtained for
the time interval or a daily duration. The function reads a file where each
pixel in the grid is described by a polygon with four corners (WGS84
coordinates; degrees). In this way rainfall values are correctly plotted at
the location of the grid pixels. The functions produce maps of high graphical
quality which are customisable. For instance, background map (OpenStreetMap
or GoogleMaps), legend and title names, location, number and extend of
rainfall classes, and colour palette can be modified in the function
arguments.
Acknowledgements
We gratefully acknowledge Ronald Kloeg and Ralph Koppelaar from T-Mobile NL
for providing the cellular communication link data. We thank Marc Bierkens
(Utrecht University) for his advice concerning kriging and Manuel Rios Gaona
(Wageningen University) for programming part of the kriging script. We thank
Claudia Brauer (Wageningen University) for assisting with modular programming
and GitHub. This work was financially supported by the Netherlands Technology
Foundation STW (project 11944). The review comments by three anonymous
referees and Maik Heistermann helped us to significantly improve the
readability of the paper and the functionality of the code. Edited by: G. Vulpiani
References
Atlas, D. and Ulbrich, C. W.: Path- and area-integrated rainfall measurement
by microwave attenuation in the 1–3 cm band, J. Appl. Meteorol., 16,
1322–1331, 1977.Berne, A. and Uijlenhoet, R.: Path-averaged rainfall estimation using
microwave links: Uncertainty due to spatial rainfall variability, Geophys.
Res. Lett., 34, L07403, 10.1029/2007GL029409, 2007.Bianchi, B., Rieckermann, J., and Berne, A.: Quality control of rain
gauge measurements using telecommunication microwave links, J. Hydrol., 492, 15–23, 10.1016/j.jhydrol.2013.03.042,
2013.Chwala, C., Gmeiner, A., Qiu, W., Hipp, S., Nienaber, D., Siart, U., Eibert,
T., Pohl, M., Seltmann, J., Fritz, J., and Kunstmann, H.: Precipitation
observation using microwave backhaul links in the alpine and pre-alpine
region of Southern Germany, Hydrol. Earth Syst. Sci., 16, 2647–2661,
10.5194/hess-16-2647-2012, 2012.Cuccoli, F., Baldini L., Facheris L., Gori S., and Gorgucci E.:
Tomography applied to radiobase network for real time estimation of the rainfallrate fields, Atmos. Res., 119, 62–69, 10.1016/j.atmosres.2011.06.024,
2013.Doumounia, A., Gosset, M., Cazenave, F., Kacou, M., and F. Zougmore:
Rainfall monitoring based on microwave links from cellular telecommunication networks:
First results from a West African test bed,
Geophys. Res. Lett., 41, 6016–6022, 10.1002/2014GL060724, 2014Giuli, D., Toccafondi, A., Biffi Gentili, G., and Freni, A.: Tomographic
reconstruction of rainfall fields through microwave attenuation measurements,
J. Appl. Meteorol., 30, 1323–1340, 10.1175/1520-0450(1991)030<1323:TRORFT>2.0.CO;2,
1991.Gosset, M., Kunstmann, H., Zougmore, F., Cazenave, F., Leijnse, H., Uijlenhoet, R., Chwala, C., Keis, F., Doumounia, A., Boubacar, B., Kacou,
M., Alpert, P., Messer, H., Rieckermann, J., and Hoedjes, J.: Improving rainfall
measurement in gauge poor regions thanks to mobile telecommunication
networks, B. Am. Meteorol. Soc., 97, ES49–ES51, 10.1175/BAMS-D-15-00164.1, 2016Heistermann, M., Jacobi, S., and Pfaff, T.: Technical Note: An open source
library for processing weather radar data (wradlib), Hydrol. Earth
Syst. Sci., 17, 863–871, 10.5194/hess-17-863-2013, 2013.Hou, A. Y., Kakar, R. K., Neeck, S., Azarbarzin, A. A., Kummerow, C. D., Kojima, M.,
Oki, R., Nakamura, K., and Iguchi, T.: The global precipitation
measurement mission, B. Am. Meteorol. Soc., 95, 701–722,
10.1175/BAMS-D-13-00164.1, 2014International Telecommunication Union: Recommendation ITU-R P.838-3, Specific
attenuation model for rain for use in prediction methods, available at:
https://www.itu.int/dms_pubrec/itu-r/rec/p/R-REC-P.838-3-200503-I!!PDF-E.pdf (last access: 23 May 2016), 2005Joyce, R. J., Janowiak, J. E., Arkin, P. A., and Xie, P.: CMORPH: A
method that produces global precipitation estimates from passive microwave
and infrared data at high spatial and temporal resolution, J. Hydrometeorol.,
5, 487–503, 10.1175/1525-7541(2004)005<0487:CAMTPG>2.0.CO;2, 2004
Kharadly, M. M. Z. and Ross, R.: Effect of wet antenna attenuation on
propagation data statistics, IEEE T. Antenn. Propag., 49, 1183–1191, 2001.Kidd, C. and Huffman, G.: Global precipitation measurement, Meteorol. Appl.,
18, 334–353, 10.1002/met.284, 2011.Lorenz C. and Kunstmann, H.: The hydrological cycle in three state-of-the-art
reanalyses: Intercomparison and performance analysis, J. Hydrometeorol.,
13, 1397–1420, 10.1175/JHM-D-11-088.1, 2012.Leijnse, H., Uijlenhoet, R., and Stricker, J. N. M.: Rainfall measurement
using radio links from cellular communication networks, Water. Resour. Res.,
43, W03201, 10.1029/2006WR005631, 2007a.Leijnse, H., Uijlenhoet, R., and Stricker, J. N. M.:
Hydrometeorological application of a microwave link: 2. Precipitation.
Water Resour. Res., 43, W04417, 10.1029/2006WR004989, 2007b.
Leijnse, H.: Hydrometeorological application of microwave links –
Measurement of evaporation and precipitation, PhD thesis, Wageningen
University, Wageningen, the Netherlands, 2007c.Leijnse, H., R. Uijlenhoet and Stricker, J. N. M.: Microwave link rainfall
estimation: Effects of link length
and frequency, temporal sampling, power resolution, and wet antenna
attenuation, Adv. Water. Resour., 31, 1481–1493,
10.1016/j.advwatres.2008.03.004, 2008.Leijnse, H., Uijlenhoet, R., and Berne, A.: Errors and uncertainties in
microwave link rainfall estimation explored using drop size measurements and
high-resolution radar data, J. Hydrometeorol., 11, 1330–1344, 10.1175/2010JHM1243.1, 2010.Messer, H. and Sendik, O.: A new approach to precipitation monitoring, IEEE
Signal. Proc. Mag., 110–122, 10.1109/MSP.2014.2309705, 2015.
Messer, H. A., Zinevich, A., and Alpert P.: Environmental monitoring by
wireless communication networks, Science, 312, p. 713,
2006.
Minda, H. and Nakamura, K.: High temporal resolution path-average rain
gauge with 50-GHz band microwave, J. Atmos. Ocean. Tech., 22, 165–179,
2005.Olsen, R. L., Rogers, D. V., and Hodge, D. B.: The aRb relation in the
calculation of rain attenuation, IEEE T. Antenn. Propag., AP-26 (2),
318–329, 1978.Overeem, A., Buishand, T. A., and Holleman, I.: Extreme
rainfall analysis and estimation of depth-duration-frequency curves using
weather radar, Water Resour. Res., 45, W10424,
10.1029/2009WR007869, 2009a.Overeem, A., Holleman, I., and Buishand, T. A.: Derivation of a 10-year
radar-based climatology of rainfall, J. Appl. Meteorol. Clim., 48, 1448,
10.1175/2009JAMC1954.1, 2009b.Overeem, A., Leijnse, H., and Uijlenhoet, R.: Measuring urban rainfall using
microwave links from commercial cellular communication networks, Water
Resour. Res., 47, W12505, 10.1029/2010WR010350, 2011.Overeem, A., Leijnse, H., and Uijlenhoet, R.: Country-wide rainfall maps from
cellular communication networks, P. Natl. Acad. Sci. USA,
110, 2741–2745, 10.1073/pnas.1217961110, 2013.Overeem, A., Leijnse, H., and Uijlenhoet, R.: The RAINLINK microwave link
data set, available at:
https://github.com/overeem11/RAINLINK/tree/master/data, last access 23
May 2016a.Overeem, A., Leijnse, H., and Uijlenhoet, R.: Radar precipitation
climatology, available at:
http://climate4impact.eu/impactportal/data/catalogbrowser.jsp?catalog=http://opendap.knmi.nl/knmi/thredds/radarprecipclim.xml, last access: 23
May 2016b.Rayitsfeld, A., Samuels, R., and Zinevich A.: Comparison of two methodologies
for long term rainfall monitoring using a commercial microwave communication
system, Atmos. Res., 104–105, 119–127, 10.1016/j.atmosres.2011.08.011, 2012.Rios Gaona, M. F., Overeem, A., Leijnse, H., and Uijlenhoet, R.: Measurement
and interpolation uncertainties in rainfall maps from cellular communication
networks, Hydrol. Earth Syst. Sci., 19, 3571–3584,
10.5194/hess-19-3571-2015, 2015.
Rios Gaona, M. F., Overeem, A., Leijnse, H., and Uijlenhoet, R.: First-year
evaluation of GPM-rainfall over the Netherlands: IMERG Day-1 Final Run, J. Hydrometeorol., in review,
2016.Roebeling, R. A. and Holleman, I.: SEVIRI rainfall retrieval and
validation using weather radar observations, J. Geophys. Res.,
114, D21202, 10.1029/2009JD012102, 2009.Schleiss, M. and Berne, A.: Identification of dry and rainy periods
using telecommunication microwave links, IEEE Geosci. Remote. S., 7,
611–615, 10.1109/LGRS.2010.2043052, 2010.Schleiss, M., Rieckermann, J., and Berne, A.: Quantification and modeling
of wet-antenna attenuation for commercial microwave links, IEEE
Geosci. Remote. S., 10, 1195–1199, 10.1109/LGRS.2012.2236074, 2013
Schuurmans, J. M., Bierkens, M. F. P., Pebesma, E. J., and Uijlenhoet, R.:
Automatic prediction of high-resolution daily rainfall fields for multiple
extents: The potential of operational radar, J. Hydrometeorol., 8,
1204–1224, 10.1175/2007JHM792.1, 2007.Sorooshian, S., Hsu, K.-L., Gao, X., Gupta, H. V., Imam, B., and
Braithwaite, D.: Evaluation of PERSIANN system satellite-based
estimates of tropical rainfall, B. Am. Meteorol. Soc., 81, 2035–2046,
10.1175/1520-0477(2000)081<2035:EOPSSE>2.3.CO;2, 2000Uijlenhoet, R., Cohard, J.-M., and Gosset, M.: Path-average rainfall
estimation from optical extinction measurements using a large-aperture
scintillometer, J. Hydrometeorol., 12, 955–972, 10.1175/2011JHM1350.1, 2011.Upton, G. J. G., Holt, A. R., Cummings, R. J., Rahimi, A. R., and
Goddard, J. W. F.: Microwave links: The future for urban rainfall measurement?, Atmos. Res., 77, 300–312, 10.1016/j.atmosres.2004.10.009, 2005Van de Beek, C. Z., Leijnse, H., Torfs, P. J. J. F., and Uijlenhoet, R.: Seasonal
semi-variance of Dutch rainfall at hourly to daily scales, Adv. Water
Resour., 45, 76–85, 10.1016/j.advwatres.2012.03.023, 2012.
Van het Schip, T. I., Overeem, A., Leijnse, H., Uijlenhoet, R., Meirink,
J. F., and van Delden, A. J: Rainfall measurement using cell phone links:
classification of wet and dry periods using satellites, Meteorol.
Appl., in review, 2016.Villarini, G., Mandapaka, P. V., Krajewski, W. F., and Moore, R. J: Rainfall
and sampling uncertainties: A rain gauge perspective, J. Geophys. Res.,
113, D11102, 10.1029/2007JD009214, 2008.Wang, Z., Schleiss, M., Jaffrain, J., Berne, A., and Rieckermann, J.: Using
Markov switching models to infer dry and rainy periods from telecommunication
microwave link signals, Atmos. Meas. Tech., 5, 1847–1859,
10.5194/amt-5-1847-2012, 2012.Zinevich, A., Alpert, P., and Messer, H.: Estimation of rainfall fields
using commercial microwave communication networks of variable density,
Adv. Water Resour., 31, 1470, 10.1016/j.advwatres.2008.03.003, 2008.Zinevich, A., Messer, H., and Alpert, P.: Frontal rainfall observation by
a commercial microwave communication network, J. Appl. Meteorol. Clim.,
48, 1317, 10.1175/2008JAMC2014.1, 2009.