For several decades, light detection and ranging (LiDAR; also spelled lidar) has been used to profile the atmosphere and retrieve information on aerosols, trace gases, and atmospheric density, temperature, and wind see, e.g.,. Following the invention of the laser, first observations of tropospheric clouds were reported in the early 1960s. Soon thereafter more powerful lasers and sensitive detectors led to detection of stratospheric aerosols by lidar e.g.,. But it took until the early 1980s before the lidar technology was developed far enough to enable measurements of atmospheric density and temperature in the mesosphere . In contrast to their tropospheric counterparts, the mesospheric lidars were rather complex experiments requiring a great deal of labor to set up and operate, with some systems filling entire buildings . Hence, these lidars were run only during campaigns, or, e.g., in the case of stations belonging to the Network for the Detection of Atmospheric Composition Change, operation was limited to certain days per week when the weather forecast looked favorable and trained operators were available. The intermittent operation not only limited the amount of data but also made statistical studies that require a dense temporal sampling of the atmosphere, e.g., the investigation of the evolution of atmospheric gravity wave events e.g.,, next to impossible. Gravity wave climatologies which do not require a dense sampling were published by, e.g., , , , , , and .

In recent years, a number of autonomous tropospheric lidar systems have been developed to address the shortcomings of the earlier manually operated instruments and increase the data output . But until today, to the knowledge of the authors, no attempts were made to build autonomous middle atmosphere lidars. There may have been several factors contributing to the stalled development. First, lidars capable of sounding the mesosphere require a much higher sensitivity given the exponential decrease in air density with altitude. Consequently, mesospheric lidars use powerful lasers, large aperture receiving telescopes, and highly efficient receivers, which makes some of the solutions generally used in the development of autonomous tropospheric lidars impractical, as, for instance, a window covering the telescope to protect it from the environment. Second, because of the technical challenges and lower interest in the middle atmosphere as compared to the troposphere, there are only a few groups operating middle atmosphere lidar instruments.

The primary objective of the Compact Rayleigh Autonomous Lidar (CORAL) is the demonstration of a fully autonomous lidar system which can be used for studying atmospheric dynamics in the stratosphere and mesosphere. That the instrument should be capable of fully automatic observations was not seen as a practical feature but rather as pure necessity resulting from the lack of human effort to operate the instrument. For the same reason, the instrument should require only a bare minimum of maintenance work. In other words, the instrument should happily sit by itself, monitor itself, and collect atmospheric measurements whenever weather conditions allow for optical soundings. Human interaction should be limited to approximately weekly downloads of scientific data and yearly maintenance. Moreover, CORAL should be transportable, fully independent of infrastructure expect for electrical power, robust enough to withstand environmental conditions from the tropics to the Arctic and Antarctic, easy to replicate, and in relative terms low cost. In short, we wanted to develop a lidar system which can be set up at some remote location and left there for years collecting atmospheric data, much like ceilometers are used today by the weather services. If such a system was possible, it would surely mark the transition from the conventional, laboratory style and labor-intensive lidar systems commonly in use today and run by lidar experts to a new generation of operational lidar systems which can be run by experts and non-experts alike. There are several benefits expected from such a new generation of lidars.

As the cost of lidar operators contribute significantly to the operating costs of conventional lidars, the use of autonomous systems will bring the cost per operating hour down. Lower costs will enable a more widespread use of lidar systems for atmospheric research and climate monitoring.

The purpose of this paper is threefold. First, we want to demonstrate the functionality of an autonomous operated middle atmosphere lidar. Second, given the advances in computer power and software development tools, this paper will demonstrate that the building of an autonomous lidar instrument is not overly complicated. Third, as we will argue in the discussion (Sect. ), the large and continuous data sets produced by autonomous instruments facilitate advances in science that are hardly possible with conventional human-operated lidar instruments. Following this agenda, we describe the lidar instrument in Sect. , followed by the description of the software used for autonomous lidar operation in Sect. . In Sect. , we briefly discuss our implementation of the temperature retrieval.

Development of CORAL started in 2014 as a copy of the German Aerospace Center's first mobile middle atmosphere lidar system, TELMA (Temperature Lidar for Middle Atmosphere Research), which was employed with much success during the deep propagating gravity wave experiment (DEEPWAVE) field campaign in New Zealand . CORAL measures atmospheric density in the altitude range 15–95 km and thus covers most of the stratosphere and mesosphere. The system uses a pulsed laser at 532 nm wavelength as light source and a receiver equipped with several channels for detecting both elastic scattering and inelastic scattering at 608 nm wavelength. Atmospheric temperature is retrieved by hydrostatic integration of the measured density profiles .

The lidar instrument is integrated into an 8 foot (2.4 m) steel container (see Fig. ) which serves both as transport container and enclosure during lidar operation. The container is divided into two compartments: an air-conditioned room accommodates the transmitter, receiver, and data acquisition systems, while the telescope is located in a separate room with a large hatch in the roof for direct access to the sky. The technical specifications of the lidar instrument are summarized in Table .

Our implementation of the temperature retrieval is based on the integration method developed by . In the absence of aerosols and absorption by trace gases such as ozone and after taking into account the geometric factor z-2 stemming from the formation of spherical waves of the scattered light, the lidar return signal can be assumed as proportional to the Rayleigh backscatter of the atmosphere e.g.,. Because Rayleigh scattering is directly proportional to the number density of air, the Rayleigh backscatter profile equates to a profile of relative number density. In the retrieval, the atmosphere is split up in discrete layers, and the relative number density is evaluated at these layers from layer averages of the Rayleigh backscatter profile assuming that the number density varies exponentially across layers. Using the ideal gas law and assuming hydrostatic equilibrium, temperatures are calculated by integrating relative number densities from top to bottom. One complication arises from the requirement to start the integration at infinity in which the density is zero. Since this is impractical because of the large noise in lidar measurements at very high altitudes, the integration is instead split into two parts. Above a certain reference altitude z0, the integration is assumed to evaluate to a reference temperature T0 that is then used to initialize or “seed” the integration of the second part starting at z0. Typically, z0 is chosen as the upper boundary of the lidar return profile where the signal is still reliable, and the corresponding T0 is taken from other measurements or models e.g. The condition “no aerosols present” can be relaxed if inelastic (Raman) scattering, e.g., for nitrogen molecules, is used instead of the elastic scattering to derive profiles of relative number density.

In our implementation of the retrieval, the basic preparatory steps prior to the integration are binning the photon count data to a 100 m vertical grid and a desired temporal resolution, the correction of detector dead-time effects, subtraction of the background which is estimated from photon count profiles between 130 and 200 km altitude, correction of the two-way Rayleigh extinction, range-correction by multiplication with the range squared, and vertical smoothing to 900 m effective vertical resolution to improve the signal-to-noise ratio (SNR). After performing these steps, temperatures are computed by integration of the profiles of relative density. This process is repeated independently for each detector channel. The question is now, where do we get the seed value from? We start off with the nightly mean profile of the far-channel which we seed with an approximately co-located SABER (Sounding of the Atmosphere using Broadband Emission Radiometry instrument on the Thermosphere Ionosphere Mesosphere Energetics Dynamics, TIMED, satellite) profile at typical altitudes of 98–108 km. As coincidence criteria, we chose a maximum horizontal distance of 1000 km between the mean location of the satellite measurement and the location of CORAL. Because the density profiles from the lower channels overlap with an upper channel, we can then seed the retrieval of the mid-channel temperature profile with a temperature value from the far-channel. In a similar way, both the low channel and Raman channel are seeded with a value taken from the mid-channel temperature profile. The altitude where the integration starts and the seed value is taken is determined by the SNR N*/N, where N is the number of detected photons per 100 m bin and N* the desired signal (background subtracted). We define the seed altitude as the maximum altitude with SNR>4 (far-channel) and SNR>15 (all other channels). The criterion SNR>4 at 100 m bin width translates to a relative uncertainty in pseudo-density of 8.3 % in the case of the default resolution of 900 m and 6.5 % for 1500 m resolution. For reference, a threshold of 10 % was used by .

The individual temperature profiles are then merged into a single continuous profile. In order to guarantee a smooth transition from one profile to another, we compute a weighted average in the overlapping region using the weighting function wz=0.5+0.5cos⁡πz-zt/Δz within the transition region starting at altitude zt and vertical extent Δz. The bottom of the upper of the two profiles is typically chosen as zt and Δz=2km. Figure  shows an example of individually retrieved temperature profiles and overlapping regions. Note the large discrepancy between the Raman temperature profile (channel 4) and the elastic temperature profile (channel 3) at about 20 km altitude, which is caused by stratospheric aerosols. We mitigate the impact of stratospheric aerosols by transitioning to Raman temperatures below 32 km altitude when merging profiles.

In order to achieve higher temporal resolutions, we implemented the iterative approach sketched in Fig. a. After retrieving the nightly mean profile seeded with SABER data, we bin the photon count data to overlapping 120 min wide bins that are offset by 30 min. Temperature profiles are then retrieved from these binned data using seed values taken from the nightly mean profile. This works because the SNR of the 120 min binned photon count profiles is always lower (or equal) to the SNR of the nightly mean profile for the same altitude. The result of such a coarse temperature retrieval is shown in Fig. b. Having completed the 120 min retrieval, we bin the photon count data to 60 min resolution using 15 min offsets from one bin to the next bin and start the process over again. This time the seed values are taken from the 120 min temperature profiles. In the next iteration, the temporal resolution is increased to 30 min using seed values taken from the 60 min profiles. Iteration by iteration, a pyramid with ever increasing resolution is built up until the desired resolution is reached and at which point the algorithm stops. In Fig. b and c, we show an example to demonstrate the effect of increasing resolution on temperature profiles. Where the 120 min retrieval reveals only large-scale structures, in this case signatures of an internal gravity wave with a period of ∼6 h and 22 km vertical wavelength, the high-resolution retrieval (Fig. c) yields a multitude of fine details including smaller-scale waves. The semidiurnal tide with a period of 12 h may also be present in Fig. b and c but is overshadowed by internal gravity waves with much larger amplitudes. show measurements of tides acquired with a predecessor instrument of CORAL above New Zealand with peak amplitudes of ∼6 K at 85–90 km altitude (their Fig. 12).

The implementation of our retrieval also allows us to increase the vertical resolution to, e.g., 500 and 300 m in regions where the SNR is sufficient. A high vertical resolution is especially important for retrieving accurately the large vertical temperature gradients induced by large-amplitude waves that are on the verge of becoming convectively unstable. Due to the SNR required, generally, these very high-resolution retrievals are limited to altitudes of 30–70 km.

A common way for estimating the uncertainty of retrieved temperatures Tz is computing the error propagation in the hydrostatic integration of the density profiles. Here, the assumed errors of the density profiles are the photon count uncertainties Nz scaled accordingly. In our implementation, we use a different approach. Starting with the photon count profile Nz and its uncertainty Nz, we perform a set of 200 Monte Carlo experiments for each profile. In these experiments, the number of detected photons per bin N is replaced with N+αN, with α being random numbers drawn from a Gaussian distribution with a standard deviation of one and zero mean. Then we apply the data reduction steps described above and run the retrieval separately for each of the 200 synthetic photon count profiles. In a last step, the final temperature profile is computed as the mean of all 200 retrieved profiles, and its uncertainty is given by the standard deviation.

The Monte Carlo method has the big advantage that all data reduction steps are included in the assessment of temperature uncertainties ΔTz in a completely natural way. That also applies to the initial seed temperatures taken from the SABER profile during the first iteration, i.e., retrieval of the nightly mean profile. Of course, the seed temperature is also fraught with uncertainty due to true measurement errors but also because the SABER measurements may have been acquired up to 1000 km away from the location of CORAL. In order to include the impact of variations in seed temperature, we generate a set of seed profiles in the form Tseedz+αΔTz, one for each of the 200 synthetic photon count profiles, with α being again random numbers that are different for each z and have a standard deviation of one and zero mean. In the case of the SABER profile, ΔTz is assumed constant with a value of 20 K, whereas for subsequent iterations, we use the uncertainty of the temperature profile retrieved in the previous iteration. This scheme ensures that the initialization error caused by the SABER profile is passed on to all lidar temperature profiles, resulting in robust uncertainty estimates for retrieved profiles.

The CORAL instrument is the first middle atmosphere lidar that is capable of operating fully autonomously for extended periods. By putting computer software in charge and not relying on human operators to start, monitor, and stop the instrument, CORAL is destined to acquire atmospheric profiles whenever weather conditions allow for optical soundings. This not only maximizes the data return but also minimizes potential sampling biases. As demonstrated in Fig. , CORAL also operates in marginal weather and records data during short windows with gaps in the cloud layer, an opportunity normally not seized by conventional lidars because of the waste of time of the human operator. We believe that probing the atmosphere as often as possible is critical to capturing the true state of the atmosphere, in particular with regard to atmospheric gravity waves. There is a long-standing and ongoing discussion as to whether lidars in general underestimate wave activity because, being optical instruments, lidars are typically run during stable weather and clear skies conditions. However, for example, it is reasonable to assume that strong forcing of mountain waves occurs in stormy weather conditions, which are often accompanied by clouds. In contrast to conventional lidars, CORAL may operate under these conditions and thus capture strong wave responses not previously observed by other lidars. A study trying to quantify this potential “nice weather” bias is in preparation.

On the other hand, frequent observations – even if they are short – can reveal new insights in the evolution of gravity wave events and the question about their intermittency. detected large-amplitude and long-lasting (several days) mountain waves in CORAL measurements above Río Grande in southern Argentina. The magnitude of this event would have been certainly heavily underestimated given the coarser sampling of conventional lidars. compared CORAL gravity wave measurements acquired above Sodankylä, northern Finland, to gravity wave potential energy densities retrieved from the ECMWF IFS. In their study, showed that IFS forecasted the evolution of wave events reasonably well although wave amplitudes in the upper stratosphere were heavily underestimated. Again, the high cadence of the CORAL measurements turned out to be crucial for this study. A third example highlighting the importance of frequent observations is the work by who investigated the influences of source conditions on mountain wave penetration into the mesosphere above New Zealand using a predecessor instrument of CORAL. Finally, analyzed CORAL data for signatures of downward propagating gravity waves. Since these are relatively rare events, collecting as much observations as possible greatly improves the chances for finding cases in which the downward propagating waves are not masked by interference with strong upward propagating waves. During a previous campaign in the Bavarian Forest in Germany, CORAL also was able to capture a rare midlatitude noctilucent cloud event .

The above examples clearly demonstrate the scientific value of autonomous lidar systems. However, we also note that site selection plays an important role in the data return of an instrument as even the most powerful lidars can not operate if there is a constant thick cloud layer above. Based on our experience, sites in the lee of mountains are good places for setting up optical instrumentation. Figure  shows nightly mean profiles acquired by CORAL at Río Grande, southern Argentina, in the lee of the southern Andes. Over the 12-month period, CORAL conducted observations on 243 nights, which equals about two out of three nights or 66.4 %. To our knowledge, no other lidar instrument achieved a similar high cadence over such a long period. report 74 observations with a daylight-capable iron lidar in 2011 (20 %). state 154 nights with sodium lidar observations in 2012–2016 (8 %), 519 Rayleigh lidar observations in 1994–2013 (7 %), and 302 Rayleigh lidar observations in 2005–2015 (8 %). The comparison of latter numbers (7 %–20 %) with CORAL observations (66 %) shows the extraordinariness of the CORAL data set.

The reliability of CORAL measurements is demonstrated in Fig.  which shows a comparison between a radiosonde and a lidar temperature profile. The coincident radiosonde sounding was acquired on 16 November 2019 during the Southern Hemisphere Transport, Dynamics and Chemistry (SOUTHTRAC) campaign . The radiosonde was launched at the Río Grande airport at ∼1 km distance from the location of CORAL at 00:09 UTC. Almost 2 h later, a hole in the cloud layer of ∼35 min duration enabled lidar observations and the retrieval of a temperature profile with 20 min integration period centered at 02:13 UTC. This period coincides with the upper part of the radiosonde sounding. Both profiles are in nearly perfect agreement with a mean difference of <0.6 K between 28 and 33 km altitude. While mean differences increase to ∼2.7 K at the bottom of the lidar profile, we should keep in mind that the radiosonde crossed these altitudes about 1 h earlier and atmospheric conditions have likely changed in the meantime. The lidar measurement was acquired in fully automatic mode demonstrating that the conscan method achieved beam overlap and tracking in less than 10 min after a cold start of the system. Furthermore, the comparison shown in Fig.  supports the conclusion that range-dependent defocusing of the telescope is not significant as otherwise we should see a dramatically diverging lidar temperature profile at altitudes below 20 km (see Sect. S2 in the Supplement).

From an operational point of view, after the completion of the testing phase, CORAL exceeded all expectations. The instrument has been collecting data on a routine basis for the last 13 months without requiring any on-site service and is still operating normally as of October 2020. That not only demonstrates the robustness of the system but has also proven the very importance of the continuation of the long-term observations given the ongoing travel restrictions due to the COVID-19 pandemic. The performance of the lidar is slowly degrading due to a buildup of dust on the telescope mirror and laser turning mirror. However, the lidar system has enough performance reserves to continue observations for another year before cleaning will be necessary. The other limiting factor is the lifetime of the deionization cartridge in the primary cooling loop of the laser, which normally lasts about 1 year. Accepting a higher risk of failure, based on our experience the laser can be operated with the same cartridge for up to 2 years. The addition of a conductivity meter for monitoring the coolant is planned as part of a larger upgrade of the lidar instrument.

Most of the teething problems in the early days of CORAL were caused by or related to software problems. For example, a race condition in the software of one of the two MDMs resulted in the telescope hatch not closing during an approaching rain shower, causing flooding of the telescope compartment. In another incident, a configuration error prevented the successful transfer of authority from one MDM to the other when the first MDM was disabled by a faulty power supply. As a result, environmental control was lost, and the freezing of the primary cooling loop of the laser led to permanent damage and made replacement necessary. These examples show that quality assurance in software development is of similar criticality for the success of an instrument like CORAL as the design of the hardware. Whereas most of the hardware comprises off-the-shelf components, the software that empowers CORAL to make autonomous observations is unique. The same applies to the beam tracking system described in Sect. , which to our knowledge is the first application of the conical scan method to middle atmosphere lidars. Again, the hardware is rather simple and straightforward to implement in lidars, and it is the software that represents a major advancement in technology.

Based on the above examples, we argue that, in the past, the software was an often overlooked and underappreciated aspect in the development of lidar technologies. That even may have been a natural consequence given that most lidars are built by physicists and engineers who usually are not trained software developers. In order to facilitate the development of new technologies including advances in hardware and software, we propose to include trained software developers in lidar teams. Furthermore, we encourage the lidar community to share algorithms and software. Based on our experience with CORAL, it takes considerable efforts and resources to develop software and test it. In our opinion, it is a waste of resources if every group has to start from scratch developing more or less the same set of tools.

This work has described a new autonomous middle atmosphere temperature lidar that is capable of performing fully automatic observations over extended periods, potentially years. This capability represents a major advancement over conventional lidars that are operated only during campaigns or during certain days per week. Not only does the automatic system result in significantly reduced operating costs as no personnel is needed to run the lidar, but the high cadence of the observations also enables new scientific studies. Thus, the CORAL system is a valuable tool for the scientific community, and its success may prompt the development and installation of a whole new class of middle atmosphere lidars that can be used for a broad range of scientific studies, e.g., atmospheric gravity wave research, climate monitoring, and satellite data validation. In order to facilitate scientific progress and seed the development of CORAL-type lidars, we will make the CORAL software available to the community.