Doppler lidar (DL) applications with a focus on turbulence measurements sometimes require measurement settings with a relatively small number of accumulated pulses per ray in order to achieve high sampling rates. Low pulse accumulation comes at the cost of the quality of DL radial velocity estimates and increases the probability of outliers, also referred to as “bad” estimates or noise. Careful filtering is therefore the first important step in a data processing chain that begins with radial velocity measurements as DL output variables and ends with turbulence variables as the target variable after applying an appropriate retrieval method. It is shown that commonly applied filtering techniques have weaknesses in distinguishing between “good” and “bad” estimates with the sensitivity needed for a turbulence retrieval. For that reason, new ways of noise filtering have been explored, taking into account that the DL background noise can differ from generally assumed white noise. It is shown that the introduction of a new coordinate frame for a graphical representation of DL radial velocities from conical scans offers a different perspective on the data when compared to the well-known velocity–azimuth display (VAD) and thus opens up new possibilities for data analysis and filtering. This new way of displaying DL radial velocities builds on the use of a phase-space perspective. Following the mathematical formalism used to explain a harmonic oscillator, the VAD’s sinusoidal representation of the DL radial velocities is transformed into a circular arrangement. Using this kind of representation of DL measurements, bad estimates can be identified in two different ways: either in a direct way by singular point detection in subsets of radial velocity data grouped in circular rings or indirectly by localizing circular rings with mostly good radial velocity estimates by means of the autocorrelation function. The improved performance of the new filter techniques compared to conventional approaches is demonstrated through both a direct comparison of unfiltered with filtered datasets and a comparison of retrieved turbulence variables with independent measurements.

Doppler lidars (DLs) are widely used for measurements of atmospheric wind and turbulence variables in different application areas, such as wind energy, aviation, and meteorological research (

At the Meteorological Observatory Lindenberg – Richard Aßmann Observatory (MOL-RAO) the interest in long-term operational DL profile observations for both wind and turbulence variables is motivated by different application aspects. The data can be helpful in analyzing and interpreting the kinematic properties of the vertical structure of the atmospheric wind and turbulence under different weather conditions and states of the ABL during the course of the day (e.g., stable ABL, convective mixed ABL, transitions between different ABL states). In addition, the profile information can be useful for regular validation purposes of atmospheric numerical models. This includes not only modeled wind profiles but also the performance of turbulence parameterizations (e.g., TKE closure) used to describe subgrid-scale processes. Due to increasingly higher model resolutions and the associated changes in the applicability and relative importance of parameterization schemes, long-term DL-based turbulence measurements are also interesting when it comes to developing appropriately adapted parameterization approaches that meet these new requirements.

A variety of scanning techniques and retrieval methods for vertical profiles of wind and turbulence variables based on DL measurements have been developed (

Because of the strength of the

To overcome the issues described above, new filter methods were developed in the course of implementing the retrieval method by

The DL measurements serving as the basis for this work were taken at the boundary-layer field site Falkenberg (in German: Grenzschichtmessfeld, GM, Falkenberg), which is an open field embedded in a flat landscape, with main wind directions from WSW, located about 5 km to the south of the MOL-RAO observatory site. The flat terrain characteristics meet the requirements for the application of the turbulence measurement approach by

At GM Falkenberg a StreamLine DL from the manufacturer HALO Photonics with the specifications given in Table

Instrument specifications of the HALO Photonics StreamLine DL operated at MOL-RAO.

Note that with StreamLine XR DL systems HALO Photonics by Lumibird offers a further development of the StreamLine series. XR systems operate with larger pulse length in order to increase the range, depending on the presence of scattering particles in the atmosphere. The larger pulse length, however, reduces the spatial resolution of the measurements along the line of sight (LOS), which is not an option for measurements in the ABL if the focus is on the detection and investigation of small-scale structures.

For the measurements carried out in this work the relevant DL output variables are the radial velocity estimates

Examples for measurements from one and the same conically scanning Doppler lidar. Each column represents measurements during a 30 min interval at different times and range gates (i.e., measurement heights along the line of sight) which are characterized by different kinds of noise (left: noise-free, middle: type A noise, right: type B noise). The plots of each row depict the measurements from different perspectives. The first row shows a time series plot of the radial velocities (

Typical examples for DL78 measurements that differ in terms of their noise characteristics are shown in Fig.

All three measurement examples have been taken with the same DL system under identical configuration (e.g.,

Concerning the differences in the bad estimate distributions, to the authors' knowledge up to now there have been no user reports about nonuniform bad estimate distributions in DL measurements available. A uniform distribution of bad estimates indicates that the noise component of the spectrum of the lidar signal is white noise

In the previous section it has been shown that DL radial velocity measurements obtained using the measurement strategy proposed in

Different filtering techniques to separate reliable data from noisy measurements can be found in the literature. A closer look at the underlying principles of radial velocity quality assessment allows a rough subdivision into two categories of filtering methods: (1) one category makes use of additional parameters from post-processing of Doppler spectra and (2) the other uses statistical analysis tools applied to time series of DL radial velocity estimates. The method behind the first category is the well-known SNR thresholding technique. Methods representing the second category are, for instance, the median absolute deviation (MAD) originating from

The signal-to-noise ratio (SNR) is determined from the Doppler spectra and is defined as the ratio between the signal power and the noise power. The first bears the meaningful information in a measurement and the latter is considered to be an unwanted signal contribution that is blurring this information. The higher the level of signal power and the smaller the level of noise power, the better the SNR and thus the quality of the radial velocity estimate.

In practice, DL users are often faced with deciding on a suitable SNR threshold value (SNR

The turbulence retrieval proposed by

Doppler velocity vs. SNR plot from conically scanning Doppler lidar measurements with

Methodically different from the SNR thresholding technique is the consensus averaging (CNS) method introduced by

Schematic representation of a possible practical implementation of the CNS (consensus) averaging method based on an approach by

VAD plot examples from conical DL measurements with

If the focus is on the derivation of turbulence variables, including the determination of variances caused by eddies in turbulent flow, the problem of this CNS approach is that it requires an a priori estimate of the variance which is actually being attempted to be derived. If

Another limitation of the CNS filtering technique is that it expects a uniform distribution of bad estimates for a successful application. This becomes evident by considering the CNS filtering results for the measurement example shown in Fig.

The filtering techniques discussed in the previous section are not efficient enough if DL measurements with both the highest possible data availability and a probability of bad estimates close to zero are required. New filtering techniques with improved performance concerning this demand are presented in this section. In particular, depending on the measurement's noise characteristics two different approaches (referred to as

The VV90D perspective represents a diagram in a rectangular coordinate system where each radial velocity value

Examples of a graphical visualization of radial velocity measurements from a conically scanning DL using the framework of the VV90D perspective for two cases with stationary

A quantitative description of the VV90D ring structures is provided with the diagrams shown in Fig.

Compared to the commonly used VAD visualization technique, the framework of the VV90D perspective represents an alternative way of displaying radial velocity measurements from a conically scanning DL and opens up new possibilities for data analysis at the same time. In the next section it will be shown how this framework can be used to develop suitable filtering techniques of bad estimates in noisy DL data.

Two different coarse filtering techniques are presented next. The underlying ideas are motivated by characteristic features of good and bad estimates in the VV90D diagram, which will be briefly explained. For this purpose the noise-contaminated measurement examples of type A and type B from Sect.

Examples of noise-contaminated DL measurements over a 30 min time period analyzed using the framework of the VV90D perspective. The upper

One specific property emerging from the analysis of noisy DL data above is that if subsets

Examples of the outcome of coarse filter I. The upper

Results that can be obtained using coarse filter I applied to the measurement examples of Fig.

The underlying idea of coarse filter II makes use of another specific property that can be derived from the analysis of noise-contaminated measurements using the framework of the VV90D perspective. It has already been described above that circular rings

Example of the treatment of noise-contaminated DL measurements over a 30 min time period illustrated using the framework of the VV90D perspective in combination with intermediate results when coarse filter II has been applied. The upper

Examples of the outcome of coarse filter II. The upper

For the measurement examples shown in Fig.

The coarse filter results presented in Sect.

The median absolute deviation (MAD) is a well-known statistical tool for outlier detection in measured datasets

Looking at the pre-filtered DL measurements from the VAD perspective (Fig.

Intermediate results of the two-stage MAD filter applied to the outcome of coarse filter II for the type A measurement example shown in Fig.

The main advantage of coarse filter I over coarse filter II is the better performance with respect to the detection of bad estimates, which makes the two-stage MAD filter as a follow-up filter step of coarse filter I redundant at this point. The disadvantage of coarse filter I, however, lies in the strong rejection of many obviously good estimates (see Sect.

Results of the filter for post-processing applied to the outcome of coarse filter I (see Fig.

It has been shown in Sect.

Results of the filter for post-processing applied to the outcome of coarse filter II in combination with a follow-up two-stage MAD filter step (see Figs.

So far the re-activation step of falsely rejected good estimates introduced above has been discussed in the context of a post-processing of filter results after applying coarse filter I. Even if an unjustified data loss of good estimates after an application of coarse filter II is not that substantial the above-described re-activation step can also be applied to the outcome of coarse filter II. However, for the reasons mentioned in Sect.

Finally, it should be mentioned that with the re-activation of initially discarded data in the identified corridor of good estimates there is always a risk of returning a certain number of bad estimates if the raw measurements were contaminated with noise. Since bad estimates can be distributed over the whole measurement space of

In the previous subsections the limits of the usability of approach I and approach II depending on the type of noise have been discussed. The type of noise, however, is not the only factor affecting the applicability of the two different filtering techniques. Their success is also linked to the strength and temporal evolution of the wind during the measurement period. This becomes obvious by comparing the filter results of approach I and approach II for both type A and type B noise (Figs.

Overview of the final filter results of approach I and approach II for DL measurement examples contaminated with type A noise. The examples of each column represent four selected atmospheric conditions with respect to the wind situation: weak and stationary wind (category I), strong and stationary wind (category II), weak and nonstationary wind (category III), and strong and nonstationary wind (category IV). The panels in the first row show the time series of the respective RAW data of the DL radial velocity measurements over a measurement interval of 30 min. The panels in the second (third) row show both the RAW data and the filter results of approach I (approach II) in different colors.

Same as in Fig.

Comparing the filter results for measurements with type A noise the outcomes of approach I and approach II are equally good for category I and II (Fig.

Knowledge and insights gained from the overview given in this section are important to develop a strategy to implement the filtering techniques for operational use. More detailed information on a strategy that can be used for an implementation of approach I and approach II is given in Appendix

Depending on the filtering technique used, the decision about which of the radial velocities are classified as good or bad estimates can turn out very differently. Hence, it is to be expected that differently pre-filtered DL measurements may result in differences in the retrieved turbulence variables. This section aims to demonstrate that due to the higher sensitivity of the newly introduced filtering techniques concerning both the rejection of bad estimates and the acceptance of good estimates, the quality and data availability of DL-based turbulence measurements (e.g., TKE retrieved following

In order to be able to assess the quality of TKE variables based on differently pre-filtered DL measurements, as an independent reference, sonic data from a 99 m tall meteorological mast are used. The measurements were performed with a USA-1 sonic anemometer (Metek GmbH) at a sampling rate of 20 Hz, and the raw data were processed with EddyPro (LiCor Inc.) software. The mast is operated at GM Falkenberg at a distance of about 80 m towards SSW from the DL system.

Comparison of Doppler-lidar-based TKE at 95 m height with data from a mast equipped with a sonic device at 90 m height. The measurement period was from 18 May to 17 July 2021. Each column represents the comparison of different TKE products based on differently pre-filtered input data. SNR threshold filtered data with SNR

Results of an intercomparison of TKE retrievals based on differently pre-filtered DL data versus sonic TKE measurements are summarized in Fig.

Considering the Doppler lidar TKE retrieval based on a pre-filtering of the measurements using the SNR threshold approach, the problems that arise in connection with routine turbulence measurements (see Sect.

One additional interim note should be given on the high sonic TKE value of

Another way to compare TKE variables retrieved from differently pre-filtered DL measurements is based on the use of a so-called Bland and Altman plot (Fig.

With increasing altitudes the probability of bad estimates in DL radial velocity measurements increases. This is due to the typically decreasing aerosol density, which is also noticeable in weaker SNR values. Therefore, the actual robustness of the filtering methods can be tested much better at larger measuring heights, where the methods have to cope with an increased occurrence of noise. Unfortunately, the tower-based measurements at GM Falkenberg are only provided up to 99 m height so that for higher altitudes no independent references for a comparison are available. For that reason an alternative way based on the following strategy is used. In the subsequent analysis, DL TKE values obtained using the SNR threshold method for noise filtering are used as an alternative reference intended to replace the missing sonic data at higher altitudes. This can be motivated by two arguments. On the one hand, it could be shown that the comparison of TKE values based on SNR pre-filtering with independent sonic data provided the best results (see Fig.

Comparison of TKE from DL based on differently pre-filtered measurement data prior to the TKE retrieval as proposed in

Comparisons of TKE products based on SNR pre-filtered DL data with those based on CNS filtering and CF filtering for measurement heights between 45 and 500 m are shown in Fig.

First test measurements for a desired routine application of DL turbulence measurements based on the approach outlined in

The drawbacks of the frequently used filtering techniques motivated our work to seek new ideas for filter methods that can be applied to noise-contaminated measurements from conically scanning DL systems with low pulse accumulation. They should allow for both accurate noise filtering and the largest possible data availability. Two different approaches (I + II) were pursued in order to account for possible emerging differences in the noise distribution. Their basic structure consists of two parts, namely a coarse filter and a so-called filter for post-processing. Although each approach has a different coarse filter (I + II) they are applied in both cases based on a newly introduced framework of the VV90D perspective. By plotting the time series of radial velocity measurements (V) from conically scanning DL against the same measurement series but with a phase shift by 90° (

The results obtained with both newly introduced filter approaches were qualitatively and quantitatively verified. While the qualitative verification was based on a purely visual assessment of the filter results, the quantitative verification was based on an evaluation of the TKE that was calculated using the filtered measurements as input data for the turbulence retrieval. Because still included bad estimates after the filtering process would introduce large errors in the final TKE product this is an indirect way to verify the filter results. It could be shown that the deficiencies in the filtered time series and the related problems regarding data availability and quality of derived turbulence variables emerging through an application of traditional filter methods have been significantly reduced with the new filter approach. In this way, we have found a solution to deal with noise-contaminated DL measurements if low pulse accumulations for the radial velocity estimates are used. Therewith we have also created a basis to be able to use the turbulence retrieval as outlined in

This new filter method can also be applied beyond the application described here generally to other DL applications using conical scans. One example could be DL wind gust retrievals based on a scan mode as described in

For the configuration of the Doppler lidar to measure turbulence variables as proposed by

First a configuration file

A quantification of the occurrence of noise in a series of radial velocity measurements based on a conical scan is feasible by means of the ACF. Noise-free radial velocity measurements based on a conical scan geometry follow a sinusoidal curve when plotting

Assuming horizontally homogeneous and stationary wind field conditions, DL measurements taken along the scanning circle are described through

Background noise characteristics are best analyzed based on DL measurements at high altitudes where atmospheric signals are unlikely due to low aerosol density. DL measurements from three different StreamLine DL systems (DL78, DL172, DL177) at 1729 m height and from four different StreamLine XR DL systems (DL44, DL146, DL143, DL161) at 1737 m height are shown in Figs.

Measurements from three different Doppler lidar StreamLine systems at 1729 m height. Panels

Measurements from four different Doppler lidar StreamLine XR systems at 1737 m height. Panels

An equation for uncertainty estimates of DL radial velocity is given in

Using the framework of the VV90D perspective, the critical radius

Noise characteristics of the Doppler lidar DL78. The data represent a 30 min time interval at range gate number 90. The panels on the left show

From a practical point of view there are four further issues which are worth pointing out for DL users when applying coarse filter II.

The systematic intercomparison of both filtering techniques in Sect.

The combined application of both approaches presents another challenge for the implementation process. From a visual perspective on the measurement data it is easy to differentiate between type A and type B noise. However, for a routine processing an automated decision-making strategy would be required. This could be arranged as follows: first one could apply both filter approaches for each measurement interval under consideration. In doing so one obtains for each measurement interval two differently filtered datasets which do not necessarily have to be the same. If one knows the good radial velocities, one also knows the bad ones at the same time. Hence, with correct filtering the distributions of the bad estimates can provide useful information about the type of noise occurring over the measurement interval. In the case of type A noise one would expect uniformly distributed data, whereas in the case of type B noise a maximum close to zero would be characteristic (see also Sect.

For DL measurements without noise issues (Fig.

Time series plots of Doppler lidar radial velocity

Doppler lidar datasets used for the analysis including radial velocity measurements (level 1 data) and retrieved wind and turbulence products (level 2 data) are available via the ZFDM repository of the Universität Hamburg (

CD and EP performed the measurements. EP conceived the investigations, did the formal analysis of the data, and developed the filter methods. EP implemented the filter methods in continuous interaction with CD. CD investigated the transferability of the filter methods to other scan strategies. EP visualized the data and wrote the manuscript draft. CD and EP discussed and finalized the paper together.

The contact author has declared that none of the authors has any competing interests.

Publisher’s note: Copernicus Publications remains neutral with regard to jurisdictional claims made in the text, published maps, institutional affiliations, or any other geographical representation in this paper. While Copernicus Publications makes every effort to include appropriate place names, the final responsibility lies with the authors.

We thank Ronny Leinweber for his support in configuring the Doppler lidar and creating the level 1 data. We thank Markus Kayser for introducing the idea of a coordinate transformation of VAD data into the phase-space perspective. Frank Beyrich is acknowledged for valuable contributions to the final writing of the paper. Thanks to Volker Lehmann for comments on the paper.

This paper was edited by Robin Wing and reviewed by two anonymous referees.