To be able to estimate rain rates with a microwave link, information of the attenuation along the path is necessary. This attenuation is strongly correlated with the rain rate along the path. To determine the attenuation, the transmitted (TX-level) and the received power (RX-level) of the links have to be known. For older CML hardware, the TX-level is typically constant, but modern systems support an adjustment of the TX-level via automatic transmission power control to compensate attenuation events and keep the RX-level in the receiver's optimal range. Hence, recordings of the RX-level, and very often also TX-levels, are necessary, ideally at a time resolution of minutes to capture the temporal variability of rainfall.

One of the biggest advantages of the hydrometeorological usage of CML networks is that they already exist, are well maintained and provide a good spatial coverage in inhabited areas . The large countrywide CML networks are operated by cell phone providers or by CML manufacturers for a cell phone provider. Smaller networks are operated locally, e.g., by companies to establish wide area networks (WAN) between different remote sites.

For the operators of CML networks, the attenuation that is caused by rainfall is only unwanted disturbance, since it perturbs the communication along the CML in case of strong attenuation events. In periods with extremely heavy rainfall this may even lead to total loss of connection.

Strong attenuation events are not the only cause of failures in a CML network, though. Hardware and software problems occur and have to be fixed promptly to assure constant high availability of a network's full functionality. Hence, the CML networks are continuously monitored. But for the monitoring, neither the exact point in time, nor the exact amount of precipitation that may have caused a system failure is important.

As consequence the network operators store TX- and RX-level data only in a coarse resolution (15 min, hourly, daily) or do not store that data at all.

For the purpose of monitoring, modern CML systems support the storage of a so-called performance report, which typically contains the TX-level and RX-level. Depending on the hardware type and the chosen settings these reports are generated every 15 min, every hour or daily. The most common setting is a temporal resolution of 15 min and the recording of the minimum and maximum values within this period. The TX- and RX-level resolution is typically 0.3 or 1 dB, depending on the hardware. These performance data are stored, e.g., by T-Mobile in the Netherlands and by cell phone providers in Israel and are made available for research. Data transfer from the providers to the researches is usually made in chunks on daily or irregular basis.

In Germany the situation with our cooperation partner Ericsson is different. Ericsson operates the CML network for T-Mobile, but it does not store the CML performance reports. Hence, in our initial approach, we used small data loggers directly at the CML towers to record the RX-level, which is available at the analog output of the automatic gain control (AGC) of older hardware . The TX-level was constant at the CMLs which we equipped. Data were sent to a database from the data loggers via GSM every day. This approach had the advantage of a very high power resolution of the RX-level, about 0.03 dB, only limited by the resolution of the module's analog-to-digital converter. However, it also had one main drawback. Equipping one link involved not only the costs for the data acquisition module (approximately Eur 100) but also the expenses for an Ericsson technician who did the actual installation at the towers. Hence, using dedicated data acquisition modules at the towers was not suitable for an extension to a countrywide coverage, involving several thousands of CMLs.

In joint cooperation with Ericsson Germany we thus developed a custom data acquisition solution which can easily be scaled up for a larger number of CMLs and which at the same time improves the temporal resolution compared to the available data from the performance reports.

Just recently, have mentioned a similar approach, also using a custom data acquisition software running within an operational CML network. In contrast to the software that we are presenting here, which handles multiple parallel requests of CML data, their software is only capable of serial data acquisition. Hence, it seems not to be suitable to provide data for a large number of CMLs with a high and constant temporal resolution.

Since CMLs are built to form communication networks, it is easy to communicate with an individual CML when one has access to the network. Each CML is accessible via its IP address, which is typically from a private IP address space. For service and configuration related communication the Simple Network Management Protocol (SNMP) is used, which was developed to allow standardized communication between network attached devices, i.e., routers, printers or computers.

A SNMP request is defined by a target IP address and the SNMP object identifier (OID). The OID defines the actual command that has to be evoked. The OIDs are defined in the management information base (MIB). There are general MIBs which define OIDs that every system supports, like sysUpTime, which would request the uptime of the system. For additional functionality the MIBs can be extended. For example, genEquipRfuStatusTemperature is the OID to query the hardware temperature of the radio unit of a Ceragon IP10 CML.

In large CML networks, like the ones operated by the cell phone providers, SNMP is used to remotely configure the CMLs and continuously monitor them for failures. Querying the TX- and RX-level is a typical functionality of the monitoring capabilities and is, to our knowledge, supported on all modern CMLs.

The OIDs to request the TX- and RX-level are not standardized and differ from CML hardware to CML hardware, even for the same manufacturer. For example, xfRFCurrentInputPower is the base OID for requests of the current output power for an Ericsson MINI-LINK Traffic Node CML while “gnOduStatusXReceiveLevel” is the one for a Ceragon IPMax and “genEquipRfuStatusRxLevel” for a Ceragon IP10 system. For each system they have to be looked up in the MIBs which are available to the CML owners from the manufacturers.

Since modern CML systems like the Ericsson MINI-LINK Traffic Nodes support the operation of several CMLs managed by one main control unit, the derivation of the correct OID requires a further step, though. The base OID has to be extended to specify which sub-system, hence which CML, has to be accessed. Furthermore, some modern systems make it possible to query data not only for the near end of a CML, to which the SNMP requests are sent, but also for the opposite site, the far end. This also has to be encoded in the OID.

Appendix  explains in detail how the correct OIDs for different systems are built.

Based on the knowledge of the SNMP communication capabilities of the CMLs and respecting the security requirements of CML operators, the concept of our custom CML data acquisition was built on the following findings:

Necessary CML data can be queried via SNMP.

Our open-source real-time SNMP-based CML data acquisition software is written in Python and hence was given the name pySNMPdaq.

To assure a constant timing of the simultaneous SNMP requests, pySNMPdaq is divided into three types of sub-processes using the Python standard multiprocessing library:

Timer is the timer Process which continuously triggers events at a fixed temporal resolution. The smallest time step is 1 s, with an accuracy of 10 ms. It avoids the typical drift of a simple while-sleep loop construct by syncing to the hardware clock of the system it is running on.

Figure  shows a schematic of the interplay of the three different process types.

The main requirements, besides software dependencies, to successfully run pySNMPdaq are

access to a computer with SNMP connectivity to the CMLs

When the above requirements are met, a list of IP addresses, each with a set of OIDs, has to be compiled and the global configuration file has to be adjusted to define sampling intervals and data handling. We provide examples for both required files in the github repository together with a step-by-step guide on how to generate the listing of IP addresses and OIDs. After the files have been set up correctly, pySNMPdaq can be started and will run as a background process automatically.

When run in an operational CML network, care should be taken to not interfere with its normal operation. Hence a slow increase of the number of CMLs and a reasonable sampling frequency should be considered.

Our test runs at the Ericsson test facility and additional calculation of the caused traffic (see Sect. ) show that a continuous data acquisition of several hundred CMLs at a 1 s interval is theoretically possible and should not impair the network. This has not been tested operationally in a large network, though. A sampling interval of 1 min was agreed upon with Ericsson for the data acquisition of their operational CMLs.

In summer 2014 we started to run pySNMPdaq operationally on a server in the CML network that Ericsson Germany operates as backhaul for the T-Mobile cell phone net. Slowly increasing the number of simultaneously queried CMLs, we are currently receiving data from 450 Ericsson MINI-LINK TN CMLs every minute with a time delay of only several seconds.

Figure  shows a record of one CML over the course of 3 days. The CML has a length of 7.9 km and a frequencies of 22.078 and 23.086 GHz for the two directions. Plotted is the difference between the transmitted and received level (TX-RX level). Quantization is 1 dB for the TX-level and 0.3 dB for the RX-level. The CML is configured as a 1 + 1 protection system using a second transmit-and-receive unit at the near end as a hot standby. This configuration is often used at sites where the outdoor units, which contain the transmitter and receiver, are mounted in places which are difficult to reach. There, an exchange of an outdoor unit is often more expensive than the unit itself.

To give the reader an idea of the impact of rainfall on the attenuation along the CML path, Fig.  also shows the hourly records of a DWD (German Meteorological Service) rain gauge, which is located about 4 km from the CML. Please note that the different temporal aggregation and the different spatial coverage of both measurements must lead to differences in the actual observed values and the corresponding time stamps. Nevertheless, a good correlation between path attenuation (TX-RX level) and rain rate is evident. A quantitative analysis, which would require the derivation of rain rates from CML data, is beyond the scope of this article. For a validation of CML-derived rain rates and an explanation of the required processing we hence refer the reader to, e.g., or .

The lower part of Fig.  is a zoomed-in version showing one of the strong attenuation events in more detail. One can see that the three channels (far-near, near-far, far-nearprotect) agree very well. Interestingly there are, however, small differences even between far-near and far-nearprotect, which operate in the same direction and share the transmitter at the far end. The differences hence must be caused by small variations in the two receivers and the subsequent rounding to the quantized values that are available via SNMP.

Besides the operation for the CMLs in the Ericsson cell phone backhaul network, we also installed pySNMPdaq at the Bayerische Zugspitzbahn, the operator of the local alpine ski resort. They connect the summit stations, e.g., on Germany's highest peak, the Zugspitze, with the valley via Ceragon CMLs. Since their network only comprises four CMLs, computational demand was low and pySNMPdaq was run on a low performance netbook placed at the IT center of the Bayerische Zugspitzbahn. Since the operation of this CML network is not as crucial as the one that serves as the cell phone network backhaul, we were allowed to run the data acquisition with a temporal resolution of 1 s.

Figure  gives an example of TX-RX level records. In addition, similar to Fig. , the hourly rain rates from a nearby DWD rain gauge are shown. Note that the Ceragon CMLs have 1 dB quantization of the received signal level. The transmit level was constant. In the zoomed-in plot the high temporal resolution is revealed. Since data at 1 s intervals are not necessary for rain rate estimation, the quantization could be compensated by averaging. In situations with low fluctuations of the signal level, this however leads to artifacts.

The operation at the Bayerische Zugspitzbahn also proved that pySNMPdaq is capable of acquiring data from different types of CML hardware, since three of the four Ceragon CMLs are IP10 models and one is an IPMax system.

As has been mentioned in Sect. , pySNMPdaq can acquire data for CML networks with heterogeneous hardware as long as the CML hardware supports querying the received, and if necessary the transmitted signal level. To our knowledge this is true for most, if not all, modern CML systems. The only challenge is to derive the correct OIDs for each hardware type.

Given its flexibility regarding CML hardware, pySNMPdaq should be usable in all CML networks worldwide.

Since it is crucial that the data acquisition software does not impair the stability of the SNMP communication within an operational CML network, the additional data traffic must not overload the network. To estimate the data traffic limits it is important to know the SNMP bandwidth of the CMLs and the topology of the CML network.

To our knowledge the CMLs have a SNMP service channel with a bandwidth between 64 and 192 kbps s-1. A typical SNMP request of four values (TX- and RX-levels for the near end and the far end) causes a data traffic of approximately 500 bytes, 250 bytes for the request, 250 bytes for the response, mostly caused by the package overhead. That is, even the slowest SNMP service channel with 64 kbps, equal to 8 kilobytes per second, could theoretically handle 16 of those requests every second.

Hence, as long as the number of SNMP requests that are routed via one CML is small, there should not be any problems with the additional data traffic. This will however depend on the topology of the network. The CML network of Ericsson in Germany for example is structured in small hubs of CMLs which are connected via DSL to the IT center. That is, there is no big accumulation of SNMP requests to one or very few CMLs. For other networks, that may differ and it is up to the future user of pySNMPdaq to evaluate the network structure and use a safe number of parallel requests and a safe sampling rate.

Given a certain upper boundary for the number of simultaneous SNMP requests, the number of CMLs that are included in the data acquisition could be raised by staggering the requests into batches. For example, 1000 simultaneous requests could be sent out every 5 s, each time for 1000 different CMLs, but repeated in the same order every minute. This way, not all CMLs will be sampled simultaneously, but the sampling rate for each individual CML would still be constant.

The current implementation of pySNMPdaq does not yet support staggered simultaneous requests. We will add this feature in the coming month, though, and it will be made available with the updated version in the github repository.

The maximum number of simultaneous SNMP requests is not only limited by the bandwidth constraints of the CML network. Each parallel Process of the operating system and each interprocess communication data structure like a Queue or an Event requires additional memory.

Figure  shows the memory usage of the current implementation of pySNMPdaq for different numbers of parallel processes, i.e., simultaneous SNMP requests. We tested on a machine with only 8 GB of RAM, but the fitted second-order polynomial yields a memory usage of approximately 7.7 GB for 1000 parallel processes and of approximately 25.8 GB for 2000 parallel processes. The memory usage could probably be optimized by exchanging parts of the current interprocess communication with a low-level solution, e.g., by using flags in shared memory. We do, however, not see the current need to optimize the memory usage, because the CML network operators will want to keep a safety margin regarding data traffic. Hence, they will limit the number of simultaneous SNMP requests anyway, in our case to around 1000.

To overcome both the computational demand and the data traffic limitations, a solution for large and countrywide CML networks could also be to use several data acquisition servers which are spatially distributed across the network.