In the DIAL technique, the measured backscattered photocounts, Nobs(z, λi), for a laser pulse at wavelength λi are given by the lidar equation .

Nobs(z,λi)=1C(λi)O(z)z2β(λi,z)exp⁡[-2τ(z,λi)]+B(z), where C(λi) is the lidar constant, which contains the efficiency of the system, the telescope area, and the emitted number of photons at each wavelength, O(z) is the overlap function of the lidar, β(z, λi) is the atmospheric backscattering coefficient, τ(z, λi) is the atmospheric optical depth, and B(z) is the background photon counts. The atmospheric optical depth is given by τ(z,λ)=∫z0zσO3λ,Tz′nO3z′2+αλ,z′+∑eσe(λ)nez′dz′, where z0 is the altitude of the station, σO3(λi) is the ozone absorption cross section at the specific altitude and wavelength, T(z′) is the atmospheric temperature, nO3(z) is the ozone number density to be measured, α(λ,z) is the atmospheric extinction coefficient, which includes both Rayleigh and Mie scattering extinction coefficients, and ∑eσe(λ)ne(z) is the extinction by other absorbers (like SO2 and NO2). In major volcanic eruptions the abundance of SO2 gas in the stratosphere can significantly perturb ozone retrievals. However, SO2 only stays in the stratosphere for 30 to 40 d . In general, the amount of SO2 mixing ratio in the stratosphere is negligible. At midlatitudes, the uncertainty of ozone number density due to absorption by NO2 reaches a maximum of 0.4 % between 25 and 30 km of altitude. Thus, the effect of NO2 on ozone retrievals is not significant, and the third term of Eq. () is small (Brasseur et al., 1999; Godin-Beekmann et al., 2003).

For many lidar systems, at count rates below about 1 MHz, the relation between the true counts and the observed signal is linear. However, for higher counts, the detector's response may not be linear. This relation for the non-paralyzable detectors is 3Nobs=Ntrue1+γNtrue, and for the paralyzable ones is 4Nobs=Ntrueexp⁡(-γNtrue), where Nobs is the observed counts, Ntrue is the true counts, and γ is the dead time. In the traditional method, the lidar measurements should be corrected for the effect of dead time. If the value of the dead time is not known, an empirical fit can be used to estimate the dead time value . It is also well known that for high-intensity systems the output of the photomultiplier tube (PMT) can show an excess of counts some time after the signal intensity is maximum, a “tail” that is called signal-induced noise (SIN) . In fact, SIN is the residual signal originating from high signal intensities at low altitudes. It adds up with the background signal and is visible at altitudes at which the signal-to-noise ratio (SNR) is very small . Using a mechanical chopper to block high-intensity light from approaching the detector is the most practical way to avoid SIN. It is important to consider the noise component from the upper altitude of lidar signals. In many lidars the background is a constant and the effect of SIN is not detected. If present, SIN is modeled using an exponential function of the form

5B(z)=eexp⁡(-fz)+g, where the fitting coefficients e, f, and g are analytically determined . The SIN is more pronounced for the online wavelength, and for most nights its effect on the off-line wavelength is negligible; hence, a constant background can be used.

In the traditional method, the derivative of the ratio between the online and off-line signals is calculated. The ozone number density can be retrieved as follows: 6nO3(z)=-12Δδo3(z)ddzln⁡N(λ1,z)-B1(z)N(λ2,z)-B2(z)+δnO3(z), where N(λ1, z) and N(λ2, z) are, respectively, the online and off-line signals at altitude z, B1(z) and B2(z) are the background signals, and ΔδO3(z) is the differential absorption cross section between the two wavelengths. δnO3(z) is a correction term for the effect of differential Rayleigh and Mie scattering and the differential absorption by other absorbers. More details can be found in , , and .

In the traditional ozone retrieval algorithm, several corrections are applied to the raw (Level 0) counts to produce corrected photocounts. For high count rates, the dead time of the counting system is determined and a nonlinearity correction is applied. Depending on the configuration of the lidar, channels with different gains may be merged (“glued”) to produce a single ozone profile. Determining the optimized height to merge the channels is typically done empirically. In the DIAL technique, the rapid decrease in sensitivity to ozone in the upper stratosphere is another important consideration. Low-pass filters are used to reduce the noise of the signals. For an ideal low-pass filter, the transfer function of all frequencies between 0 and the cutoff frequency, νc, is 1, and the transfer function from νc to 1 is 0, where the reduced frequency ν is defined as ffN and fN is the Nyquist frequency. The final vertical resolution of the signal, Δzf, varies by the order of filter, which depends on the cutoff frequency and the initial vertical resolution Δzi: 7Δzf=νcΔzi. A detailed discussion on the digital filtering and vertical resolution can be found in and .

In the lower stratosphere, perturbations in the ozone profiles are well detected; however, depending on the number of points in the filter (order of filter), the perturbation can be largely attenuated and cause negative or positive biases. For higher altitudes, because of the lower SNR, the vertical resolution is decreased. Different numerical filters have been tested to optimize ozone retrievals. In all these techniques, to overcome the SNR decrease, the number of coefficients in the filters is increased with altitude .

The OEM is an inverse method in which the Bayesian theorem is used to find the probability distribution function (PDF) of the state of interest. Let x=(x1,x2,…,xn) be the state vector and y=(y1,y2,…,yn) be the vector of the measurements. The relation between the measurements and the state vector is 8y=F(x,b)+ϵ, where F(x,b) is called the forward model. The forward model describes our understanding of the physics of the measurements as well as the instrument's characteristics. Here, b is the model parameter vector, which contains additional parameters needed in the forward model, and the noise in the measurements is the vector ϵ. In lidar measurements, the photon counts follow a Poisson distribution. However, for a count rate greater than 10 to 20, the PDF of the corresponding error tends toward a Gaussian distribution. Therefore, using the Bayesian approach and assuming a Gaussian PDF for all quantities, for a given measurement y, the most likely state of x is found by minimizing the following cost function: J(x)=y-Fx^,bTSy-1y-Fx^,b9+x^-xaTSa-1x^-xa, where Sy is the covariance matrix of the measurements, xa is the a priori profile, which is an initial guess for the state vector, and Sa is the associated a priori covariance matrix. Typically, the cost is normalized to the number of measurements, and a cost of around 1 indicates a good retrieval.

As the forward model is nonlinear, the Marquardt–Levenberg method is used to find the state vector. The optimized solution for the state vector x occurs when the following iteration converges. xi+1=xi+1+γiSa-1+KiTSyKi-110KiTSy-1y-Fxi-Sa-1xi-xa Here, K=dFdx is the Jacobian of the forward model, and γi is a damping factor for the iteration. A comprehensive description of the application of the Marquardt–Levenberg method to OEM can be found in .

Our first-principle OEM retrieval uses the lidar equation as the forward model and the raw counts are the measurements. The lidar equation for the true counts is Ntruez,λon=Cλonz2βz,λonΓO3λon,zΓatmλon,z+Bλon(z)Ntruez,λoff=Cλoffz2βz,λoffΓO3λoff,zΓatmλoff,z11+Bλoff(z), where λon and λoff represent the online and off-line channels, ΓO3(λon,off,z) and Γatm(λon,off,z) are, respectively, the ozone and atmospheric transmissions in each wavelength, Cλon and Cλoff are the lidar constants, and Bλon(z) and Bλoff(z) are the background counts. For the stratospheric ozone measurements, in the altitude region of retrieval, the overlap is complete, and thus we have not included it in our forward model. Depending on the characteristics of the data acquisition system, the true counts are related to the observed counts by either Eq. () or (). In multichannel systems, our forward model calculates the online and off-line wavelengths for both high-altitude and low-altitude channels. The transmissions are defined as 12ΓO3,atm(λi,z)=e-2τO3,atm, where the optical depth τO3,atm is previously defined in Eq. (). Both atmospheric optical depth and atmospheric backscattering coefficients have contributions due to scattering from molecules and aerosols: τatm=τmol+τaer=∫z0zσRnair(z)+α(z)dz13βatm=βaer(z)+βaer(z), where βair(z) and βaer(z) are the corresponding air and aerosol backscattering coefficients. The online and off-line coefficients are related through the following equation:

14βaer(λoff)=βaer(λon)λoffλon-a, where for aerosols the Ångstrom coefficient a equals approximately 1, and for molecular scattering the Ångstrom coefficient a equals 4. In this paper, we only considered the clean-night condition. Therefore, the aerosol contribution to the process is not included, but it could be in the future.

Due to the presence of SIN in the online channel, the background is assumed to be a function of height in the form of Eq. (), while due to a negligible presence of SIN in the off-line channel, a constant background is used. If necessary, it is possible and easy to assign any reasonable analytic function for the background in both channels. Therefore, if needed the background for the off-line channel can be assumed as a function of height as well. Using the above forward model, the ozone and air density profiles, the background coefficients, the dead time, and the lidar constants for the four channels are simultaneously retrieved. Other parameters in the forward model are treated as model parameters. Hence, they are fixed but considered a source of uncertainty on the retrieval (b model parameter uncertainty) contributing to the total uncertainty budget (see Table ).

The statistical uncertainty of the retrieved quantities and the model parameter uncertainties are calculated as follows: Sm=GySyGyT15Sf=GyKbSbKbTGyT, where Sm, Sf, Sb are the covariances of the retrieval noise, the forward model parameter error, and the error covariance of the model parameters. The gain matrix, Gy=dx^dy, gives the sensitivity of the retrieval to the measurements, while Kb=dFdb is the Jacobian of the forward model with respect to b.

Figure  shows the averaged counts over 4 h of measurements for two different channels at online and off-line wavelengths on the night of 26 July 2017. The coincident ozonesonde is launched within 1 h after the start of the measurement and takes approximately 2 h to reach 30 km. For each retrieval, the averaging kernel matrix is calculated. The averaging kernel is a diagnostic variable that describes how the retrieval sees changes in the real atmosphere. Therefore, it contains information on the sensitivity (area of the averaging kernel function) and on the smoothing (shape of the averaging kernel function) of the retrievals. Ideally the averaging kernel is a unity matrix preserving any change in the retrieved quantity from the a priori state. The area is defined as the vector product Au, where u is a unit vector. When the retrieval comes solely from the measurements, then the area equals 1, and at altitudes at which the a priori profile is contributing to the retrievals the area decreases; an area equal to 0 would mean nothing is being retrieved.

Figure  shows the averaging kernels for the ozone density. The dashed line shows that the averaging kernel for ozone density equals 1 up to 42.7 km, and thus below this altitude the retrieval is independent of the a priori profile. Ozone is a minor constituent in the atmosphere; due to the poor SNR of signals at higher altitudes, the sensitivity of the averaging kernel decreases. Here, the retrieval falls back to the a priori values.

In a good retrieval, the difference between the forward model and the measurements, which is called the residual, should be within the uncertainty of the measurements. Figure  shows the residual plots, which confirm that our forward model has correctly characterized the physics of the atmosphere and is capable of retrieving the quantity of interest.

The OEM retrieval grid starts at 500 m and increases to 700 m at 18 km. The full width half maximum of the averaging kernel at each height is defined as the vertical resolution of the retrieval. At lower altitudes, the averaging kernel is broad, and the retrieval resolution is close to the spacing of the retrieval grid (for this specific retrieval around 500 m). As shown in Fig. b, by increasing the altitude, the retrieval resolution consequently decreases such that at 40 km the resolution is 2.8 km. Traditionally, the vertical resolution decreased by height as well. Figure a shows the vertical resolution of the retrieval in both traditional methods and the OEM. At the first 2 km of retrieval the OEM provides a better retrieval resolution; however, from 14.5 to 17 km the traditional method has a better resolution. At around 17 km both methods show the same retrieval resolution; however, the traditional resolution decreases faster such that at 42.2 km the retrieval resolution is around 7 km. The trade-off between the retrieval resolution and the retrieval uncertainty should be considered when comparing the methods, and the reader is referred to the discussion below.

Having a poorer vertical resolution leads to a better (that is, smaller) retrieval uncertainty. As shown in Fig. b, the statistical uncertainty of the retrievals for the traditional method is around 12 % at 15 km (when the vertical resolution is 200 m and the low-altitude Rayleigh channel is used) and it decreases to less than 1 % at 25 km (when the vertical resolution is around 2 km and the high-altitude Rayleigh channel is used). In contrast, the statistical uncertainty of retrieval in the OEM is around 10 % at 15 km (when the vertical resolution is 500 m) and decreases to 2.2 % at 25 km (when the vertical resolution is 700 m).

To demonstrate the mentioned trade-off in the OEM, we increased the correlation length of the a priori from 1000 to 1500 m in the lower altitudes (below 18 km) and from 1400 to 5500 m in higher altitudes (above 18 km). As a result, the retrieval has a poorer vertical resolution and smaller retrieval uncertainties. Assuming a higher correlation length indicates that at each altitude, the retrieved ozone density is dependent on the ozone distribution above and below the indicated altitude; thus, the retrieved ozone density looks smoother.

The vertical resolution and uncertainty for the traditional method as well as for the OEM with low and high correlation lengths are plotted in Fig. .

In the traditional method, the relation between the final vertical resolution and the retrieval uncertainty is defined as follows: 17ϵs∝(AΔzf3P0ta)-12, where ϵs is the retrieval uncertainty, A is the area of the telescope, Δzf is the final vertical resolution, P0 is the emitted power, and ta is the acquisition time . Assuming that the traditional method has the same vertical resolution as the OEM, using the above relation we can calculate the retrieval uncertainty, which corresponds to the higher vertical resolution. Despite the difference in the vertical resolution values, at altitudes below 20 km, both the traditional method and the OEM have similar uncertainties (the difference is less than 1 %). At altitudes above 20 km, assuming that the traditional method has the same vertical resolution as OEM, the retrieval uncertainty in the traditional method is calculated. Figure  shows the comparison between OEM uncertainty and the modified traditional uncertainty for altitudes above 20 km. From 20 to 35 km the difference between the uncertainties is insignificant (less than 1 %), while above 35 km the difference grows to 4.5 %.

The traditional ozone profile can be calculated at a similar vertical resolution to our OEM retrieval. The statistical uncertainty of the traditional analysis, using the same vertical resolution as our OEM, is shown in Fig. . Below 30 km both methods provide the same uncertainties; however, above this altitude the OEM uncertainty is smaller. The OEM's smaller statistical uncertainty at higher altitudes increases more slowly than for the traditional method due to the contribution of the a priori profile, which adds additional information. However, in the OEM retrieval an increased contribution from the regularization term of the solution means the response function becomes less than 1. Below 30 km the a priori profile has a small contribution to the final retrieval (as the response function is ∼1), but between 30 and 40 km the a priori profile has a greater contribution. Above 40 km the response function decreases rapidly (Fig. ).

Figure  shows our retrieved ozone density compared to the sonde measurements and the traditional retrieval. Consistent with Fig.  we have plotted the OEM retrievals for two different sets of correlation lengths. The ozonesonde measurements have better vertical resolutions compared to the DIAL measurements, albeit with larger random uncertainty. Also, the sonde profiles show more vertical structure of the ozone distribution. Compared to the traditional retrieval, the OEM can successfully catch many of these variations.

In order to account for the higher vertical resolution of the ozonesonde measurements, we use the OEM averaging kernels to “degrade” (smooth) the sonde profile using 18xsmoothed=Axsonde+(In-A)xa, where In is the unity matrix, and xsmoothed is the smoothed sonde profile. Figure a shows the percentage difference between the smoothed sonde and the OEM (in blue) as well as the percentage difference between the smoothed sonde and the traditional profile (in red). The difference between the sonde and the traditional analysis at 14 km is greater than 100 %. Figure b shows a comparison between the two lidar methods. For higher altitudes (above 25 km) the difference between the two retrievals is less than the statistical uncertainty of the measurement. However, for lower altitudes (between 14 and 21 km) the difference between the two methods is significant.

To investigate the effect of a priori profiles on retrievals, the OHP climatology and the US standard model were used to retrieve ozone density (see Fig. ). The OEM retrievals resulting from these two a priori profiles as well as the traditional retrieval are plotted in Fig. a. As shown in the panel (b) of this figure, below 35 km the difference between the two OEM retrievals is less than 0.5 %. Above this altitude, the percentage difference between the two methods reaches 2.5 %, which is much smaller than the retrieval uncertainty at altitudes above 35 km. Thus, the choice of a priori has a small effect on the retrievals.

The OEM provides a complete systematic and statistical uncertainty budget. Figure a shows the uncertainty of the OEM ozone retrieval shown in Fig. . The forward model parameters, the Rayleigh cross sections, the ozone absorption cross section, and the temperature profiles assumed for the ozone cross section contribute to the systematic uncertainty of the retrieval. Below 20 km, these uncertainties are comparable with the statistical uncertainty; however, in the higher altitudes systematic uncertainties are less than 1 %. The Rayleigh-scatter cross section uncertainty, at the bottom of the retrieval, is around 7 %, while at higher altitudes the uncertainty decreases to less than 1 %. These values agree with the Rayleigh-scatter uncertainty of 8 %, which is calculated in the uncertainty budget. The ozone absorption cross section for the 308 nm channel reached a maximum of 4 % at the bottom of the retrieval, which is higher than the calculated uncertainty of 1 % in . The uncertainty due to temperature is less than 0.05 %. The uncertainty due to the ozone absorption cross section at the 355 nm channel is negligible as well.

The calculated OEM uncertainty can be compared with the traditional uncertainty budget. Figure b shows the uncertainty of the traditional ozone profile. The Rayleigh-scatter cross section uncertainty has a maximum value of 8 % at the bottom of the profile, while above 20 km it becomes less than 1 %. This result is consistent with the uncertainty calculated by our OEM retrieval. In the traditional analysis, for an isothermal atmosphere, the ozone absorption cross section uncertainty at 308 nm is 3 %. The ozone absorption cross section uncertainty in our OEM retrieval is similar to , whose Monte Carlo simulations allowed temperature to vary with height. In the traditional analysis, the background aerosol uncertainty is also calculated, which impacts the ozone profile by less than 1 % in the lower stratosphere. Aerosols are currently being added to the OEM forward model as a model parameter. The statistical uncertainty of the traditional analysis at higher altitudes (above 25 km) is smaller compared to the OEM, which as explained earlier is the result of having a larger vertical resolution. However, as shown in Fig.  (the black dotted lines), the OEM retrievals also have smaller statistical uncertainties if the vertical resolution increases. As discussed previously (Fig. ), for the traditional analysis using a similar vertical resolution to our OEM, the statistical uncertainty of the traditional method will be larger than for the OEM retrievals in the upper stratosphere due to the regularization term in the OEM.

The acceptable range of ozone retrieval extends from 12 to 42.7 km. The averaging kernel of the air density extends much higher, as the air density contributes to both backscattering coefficients and the extinction coefficient terms in the forward model. Therefore, in air density retrievals, the maximum height of acceptable retrieval is 70.2 km. However, we show the retrievals below 42.7 km to be consistent with the ozone density retrievals. As shown in Fig. a, the relative air density profile is retrieved as well.

To validate our result, we used the nitrogen Raman spectrum at 386.7 nm. The off-line wavelength is transmitted to the atmosphere at the 355 nm channel, and the corresponding Raman wavelength is received at the 386.7 nm channel. The Raman channel is not sensitive to the aerosol contents of the atmosphere, and the wavelength is not absorbed by ozone (off-line Raman channel). Thus, the atmospheric backscattering and extinction terms are mostly determined by the air density. This makes the Raman off-line channel a good candidate for our validation. We can assume that N(λoff,z)∝nairz2.

Using the above relation, the relative air density profile can be generated. The relative air density is scaled against the OEM retrieval of air density, and the percentage difference is calculated (Fig. b).

As shown in the figure, the difference between the scaled relative air density generated from the Raman counts and the OEM relative air density is less than 10 %. However, in higher altitudes (above 35 km) the difference can reach up to 50 %. This difference is governed by the higher measurement noise in the Raman channel. This result provides confidence that the density retrieval is reasonable. Figure c shows the uncertainty of the relative air density retrieval. For the air density retrieval the statistical uncertainty is small (around 0.1 % at the bottom of the retrieval). The Rayleigh-scatter cross section uncertainty is small as well, and the ozone absorption cross section uncertainties are negligible.

The OHP analysis employing the traditional method uses a different value of saturation correction for each wavelength. In our OEM code, we are retrieving four different dead times, each corresponding to one of the channels. For a priori values, we are using the provided x value, which is discussed earlier in this section. As shown in Table , the retrieved dead time values for 26 July 2017 are similar to the provided x values. The only major difference is detected for the online low-altitude channel, for which the x value is 4.6 ns and the retrieved value is 5.05 ns.

The ozone profiles retrieved using our OEM for the nights of 14 and 20 July, the coincident sonde measurements, and the traditional ozone retrievals are shown in Fig. . The night of 14 July 2017 includes 4.5 h of measurements. The retrieval extends from 9.6 to 40.2 km. Above 16 km, the difference between the two traditional methods and the OEM retrieval is within the statistical uncertainty of the measurements. Between 16 and 19 km the difference between the OEM and the sonde becomes large; this is coincidental with the two peaks measured by the sonde at these two altitudes (see Fig. ). After smoothing (degrading) the sonde profiles the two picks are much smoother, and this causes the large difference between the OEM and sonde profiles (Fig. ). For 20 July 2017 the retrieval is computed using 4 h of measurements. The ozone retrieval extends from 11 to 36.8 km. The differences between the two methods are within the retrieval uncertainty (Fig. ). For both nights the difference between the sonde and the calculated profiles below 13.5 km is larger than 80 % and is thus not shown here. These two additional nights help to demonstrate that the OEM can produce high-quality ozone density profiles that are consistent with the traditional profiles found using the traditional method.