Figure  shows the true exponential extinction coefficient profiles (see Table ) with aerosol optical thicknesses of 0.2, 0.6 and 1.0. In addition, an a priori profile is depicted with the same scale height (SH = 1 km) but with AOT = 0.18. In general, an a priori profile should comprise known information of the true atmosphere. Ancillary measurements are frequently used to gather information for a mean profile. Here, we try to evaluate the performance of the algorithm by using an a priori profile close to the true profile (E1) but also for atmospheres with highly varying aerosol loads (E2, E3).

The retrieval uses vertical grid steps of 50 m for synthetic data because smaller step widths introduce retrieval noise, whereas larger steps result in vertical structure being overly smoothed. Since the retrieval tends to overestimate the upper layers due to missing sensitivity (see Fig. ), a height-dependent a priori variance was used for the preparation of the a priori covariance matrix Sa. The measurement covariance matrix Se consists of a constant signal-to-noise (SNR) value, which is multiplied with the measurement for each LOS to get absolute covariances. The Tikhonov parameter is changed in the first examples to demonstrate the retrieval response to this parameter. The iterations stop when convergence is reached or after 60 steps. Convergence in parameter space was set to 0.0001 km-1. In addition, convergence of the rms of measured and simulated DSOT differences was set to 0.001.

On the left side of Fig.  the retrieval results for different Tikhonov parameters γ are shown. E1, which is close to the a priori profile (dashed grey line) is retrieved well for all γ values. In general, smaller Tikhonov parameters give too much weight to the measurement by reducing smoothing and introducing retrieval noise, which leads to oscillations. Large values smooth the solution in the direction of the a priori profile. When the true atmosphere is far away from the a priori profile, the algorithm does not perform satisfyingly. For E2, strong smoothing (large γ values) leads to more or less straight lines in which insufficient curvature can be found, whereas weak (or no) smoothing means the beginning of oscillations indicated by the reduction of the bottom extinction. No parameter value is able to retrieve the proper bottom extinction. This can be found in the same way for E3, where all results show a clear lack of accuracy. This is not surprising as the a priori profile cannot be understood as the best guess or median of the atmosphere any more.

Several authors have already highlighted retrieval problems from MAX-DOAS measurements under highly variable (in time and space) atmospheric conditions e.g.. Nevertheless, studies that focus on improvements of the regularization or a priori information are rare with respect to MAX-DOAS retrievals. Here, we use a priori pre-scaling to cope with strong deviations of the true atmosphere from the a priori estimate. For this purpose, the BOREAS aerosol retrieval is started on a reduced vertical and spectral grid from a zero profile (all a priori extinction values were set to be zero) as a priori information. In Eq. (15), the a priori profile x0 is set to 0 but the remaining terms stay unchanged in order to retrieve a reliable AOT value without limiting the solution by a certain profile shape. Since the retrieval works directly on O4 absorption, aerosol optical thickness is in general the most reliable retrieval quantity. The AOT from this pre-calculation is used to scale the a priori profile for the main run by calculating a scaling factor from the initial a priori AOT and the new value, which is applied to all initial extinction coefficients before the main inversion. Note that the only a priori information used both within the pre-run and within the main run is the Tikhonov parameter. This method performs well when the a priori shape is similar to the true atmospheric profile shape (right side of Fig. ). However, no improvement is found when the shape of the a priori profile deviates strongly from the true atmospheric condition but no deterioration is found either (see box profiles in Appendix A). On the right-hand side of Fig. , clear improvements can be seen in both the overall shape and bottom extinction. Nevertheless, the profiles tend to deviate more strongly at the surface, especially when the aerosol load is higher, which was also found by . The relative differences in retrieved AOT are high for the unscaled retrieval but can be significantly reduced with a priori pre-scaling (see Table ). Further improvements could be made by increasing the allowed number of iterations together with stronger convergence criteria (not shown), but the average retrieval time would increase as well.

A comparison of all three profiles and their optimal γ ranges with and without pre-scaling indicates that each combination of a priori and true profile has a specific optimal Tikhonov parameter which might differ strongly between each scenario. In addition, the SNR is another quantity which has a large impact on the retrieval. In general, SNR values might differ strongly throughout a day, because dynamic ranges of exposure and integration times are frequently used to operate the CCD in the ideal saturation range when the number of photons might differ strongly (e.g. due to different viewing geometries). In Fig.  we varied SNR (as a representation of the measurement error) and the Tikhonov parameter γ from 500 to 5000 and from 0.25 to 30 respectively to find the best parameter range for the optimization of AOT and the bottom extinction (BOT). The figure shows results for the strong aerosol scenario E3 (E1, E2 and the box scenarios are shown in the appendix). The relative differences to the true value are colour coded with bright spots, indicating better retrievals.

For scenario E3, high γ (or low SNR) values apply too much smoothing (not enough weight on the measurement), which ends in strong AOT / BOT underestimations of the retrieval results. Furthermore, there is an ideal range of SNR values for γ≤2 where the AOT relative difference is ≤10 %. In this specific SNR / γ range, the bottom values do not show the best results. On the other hand, BOT values are optimized for 3≤γ≤10 for SNR =2500 and SNR =3000 where the AOT does not have the lowest difference. As a consequence, due to unequal vertical sensitivity, the choice of a priori constraints and parameterization, in some cases either AOT or the shape of the profile can be optimized. Stable solutions for the optimization of both profile characteristics could not be identified in the range of parameterization, regularization and convergence. As a result, a reasonable compromise for optimum values with respect to both AOT and BOT would be in the range SNR = 2000–3000 with low Tikhonov values 2≤γ≤5. Note that the SNR range between 3500 and 4000 has larger relative deviations than the surrounding settings which could not be found for other scenarios. This is assumed to be either a unique solution, where the range of settings leads to poor retrieval results, or a numerical instability.

In contrast to this pragmatic approach, the so-called L-curve approach was found to be insufficient as it already assumes good regularization; i.e. the data residual and the parameter norm (x-xa) can be described well by a specific regularization parameter , which was not found for the application of MAX-DOAS profiling.

The optimal-estimation-based trace gas profile retrieval faces similar problems to the regularization difficulties discussed for the aerosol retrieval in the previous section. In many studies, an a priori variance of a fixed percentage of the a priori profile is used in combination with the differential slant column density errors from the DOAS fit to constrain the solution e.g.. Here, we show that this is an insufficient approach, since the fitting errors might be too small, giving too much weight to the measurement when the NO2 concentration is high and the a priori profile is not a good estimate of the true atmosphere.

Again, a priori pre-scaling is used to achieve a better estimate of the true atmospheric conditions. In contrast to the aerosol retrieval, which needed a pre-run for a better first guess, the trace gas a priori pre-scaling is achieved using the 30∘ elevation angle measurement. In the geometric approximation (see Eq. ), the differential AMF is 1, which leads to ΔS30=V. This vertical column density V is used as a pre-scaling value for the a priori profile.

The different NO2 exponential profiles, based on the parameters in Table , are shown in Fig.  together with the a priori profile (box profiles can be found in Appendix ). Note that we used an a priori profile close to E2. These profiles are a good representation of the daily variability in an urban region, where high NO2 concentrations are found during phases of commuter traffic (rush hours) in the morning and late afternoon and low NO2 loads are found in between.

Figure  depicts the profiling results without a priori pre-scaling on the left side and with pre-scaling on the right side. Different regularization ratios are achieved by varying parameter g in Eq. (), which varies the measurement error by g. The retrieval is done on the same vertical grid as in the previous section with similar variances for the a priori covariance matrix. In contrast to a constant SNR, we use variances in the range of typical DOAS fit errors of NO2 for the measurement covariance matrix, which can be understood as the standard approach of the MAX-DOAS profiling community. For small elevation angles and high NO2 concentrations, the relative ΔS error is often lower than 1 % and sometimes even lower than 0.3 %, indicating a high reliability in the measurement but neglecting errors due to pointing inaccuracies of the telescope or horizontal inhomogeneities which affect the profiling result but not the measurement.

The results again show the importance of pre-scaling. Whereas the E2 scenario is well retrieved for all g ranges without pre-scaling, the other scenarios tend to oscillate or underestimate the profile. Note that the standard regularization ratio (g=1) does not lead to satisfying results for E3. Applying pre-scaling leads to more accurate profiles, especially for E3.

The specific optimal g range for each scenario depends not only on the a priori profile and the true atmosphere but also on the variance values, their height dependencies and the definition of off-diagonal elements. However, the quantification of this optimal range is a difficult task for real data due to the lack of knowledge about the true atmosphere. Manual variations of g as well as iterative approaches are conceivable. Possible criteria for finding the best g values are, for example, the rms between measurement and retrieval quantity or ancillary measurements. Here, we used fixed values which were chosen to decrease oscillations while maintaining the best surface concentration.

Aerosol profiles were validated using three different instruments. The retrieved AOT was compared with values from an AERONET station (AErosol RObotic NETwork, ). The level 2 AOT at 440 nm was scaled with the Angström exponent from the ratio 440 nm/675 nm to calculate the AOT at 477 nm (see Fig. ). Due to measurement intervals varying between 4 and 30 min, we decided to resample the AERONET signal. The errors from AERONET instruments are usually in the range of 0.01 (VIS, IR) to 0.02 (UV) . Here, we used a constant error of 0.01.

The bottom extinction coefficient of the retrieved aerosol profiles was compared with in situ PM10 concentrations from the National Air Quality Monitoring Network (NAQMN) operated by the Dutch National Institute for Public Health and the Environment (RIVM). The NAQMN measurement site Wielsekade is located at a distance of around 900 m from the MAX-DOAS station. The site is listed as a regional background station and the PM10 measurement principle is beta attenuation (Thermo Fisher Scientific FH62I-R). The instrument provides data on 1 min intervals but only the hourly values are validated by RIVM. These hourly values were resampled on the MAX-DOAS times and the minute values were used to calculate the errors as standard deviation for the data points shown (see Fig. ).

Furthermore, we used AERONET-scaled ceilometer near-surface extinction as a further validation data set. The ceilometer (CHM15k Nimbus) was operated by the Royal Netherlands Meteorological Institute (KNMI) and sampled backscattering signals every hour at a wavelength of 1064 nm. The integrated backscattering signal was divided by the 1020 nm AERONET AOD to get a conversion factor which was applied to the backscattering signal for the conversion into extinction coefficients. This new ceilometer profile was again scaled with AERONET AOT at 477 nm. The error is expressed as the standard deviation calculated from the temporal and vertical averaging.

Figure  shows the time series and the scatter plot of AERONET- and BOREAS-derived AOT. The BOREAS data were filtered for profiles which reached the iteration limit during the retrieval. Grey areas indicate clouds within the specified time period.

In the scatter plot, BOREAS error bars were calculated by integrating the total error (Eq. ) vertically. On the first 2 cloud-free days, the temporal variability shows a similar pattern with a slight offset between the instruments, most likely introduced by an underestimation of the AOT by BOREAS. Due to the limited sensitivity of MAX-DOAS profiling for higher altitudes (see Fig. ), elevated aerosols do not contribute to BOREAS AOT but can be measured by direct Sun instruments like AERONET Sun photometers.

In the evening, the results of both instruments vary more. Different reasons for this finding are possible. First, a developing planetary boundary layer (PBL) might exceed the vertical extension where BOREAS has sufficient sensitivity. Second, when pointing towards the Sun, saturation of the CCD becomes a problem, leading to low integration times, which decrease SNR and fitting quality. In addition, RTM calculations might introduce uncertainties at high SZA, when the aerosol load is high and the light path is strongly increased. Furthermore, the aerosol phase function used leads to uncertainties, as the forward scattering peak might be underestimated by the Henyey–Greenstein parameterization. The other days show a higher variability due to an increase in cloudy scenes. Note that, especially in the morning and around noon, clouds may influence the instruments in various ways because of their different azimuthal viewing directions. The BOREAS profile retrieval cannot be considered reliable when one or several elevation angles point at clouds during the measurements, as intensity and light path vary in unpredictable ways. This might also lead to large iteration numbers or to no convergence at all. On 15 September, the temporal patterns of the two instruments differ strongly. Although the extinction values around noon agree well, AERONET's AOT decrease before and the increase afterwards cannot be found this strongly by BOREAS. Especially in the evening, both curves deviate, indicating highly variable atmospheric conditions, which might also be introduced by a change in wind direction (see Fig. ). The aerosol load on the last 2 days is much lower because of rainfall in the time between 16 and 22 September. In the morning of 23 September, several profiles were discarded due to exceeded iteration limits.

The correlation coefficient of 0.83 is high and shows the general good agreement on 3 of 5 days. The regression line indicates the above-mentioned small underestimation of the AOT by BOREAS.

Figure  shows the comparison of near-surface aerosol values. Extinction coefficients from BOREAS and the ceilometer are depicted together with PM10 concentrations of the NAQMN in situ instrument. Note that the y axis for PM10 concentrations was chosen to match the results of BOREAS on 14 September. The NAQMN in situ sampler and BOREAS show a good temporal agreement within the first 3 days. Only in the morning hours of the 13 and 15 September, do data from the two instruments show a larger spread. This deviation is most likely due to the insufficient vertical resolution of MAX-DOAS profile retrievals when the PBL has not yet evolved and near-surface aerosol loads are dominant (see Figs.  and  for further details). On 15 September, early morning clouds might also have an impact on the deviation. Again, a larger variability for BOREAS results can be found for the last 2 days, when the retrieval was influenced by broken clouds. The in situ instruments show a smoother daily variation here. The ceilometer bottom extinction shows a good agreement with BOREAS and NAQMN on the first day and an underestimation on the second day. On 15 September, bottom values are influenced by thick clouds which might interfere with the backscattering signal of altitude ranges below the clouds (see Fig. ). The day 23 September shows more variability than the first 2 days but this can be found for the other instruments as well.

The correlation of BOREAS bottom extinctions with data from both validation instruments is high, indicating good agreement on cloud-free days. The correlation with ceilometer near-surface values is high as well but the regression line shows a general underestimation of ceilometer near-surface values in comparison to BOREAS.

Figure  shows the aerosol layer height found by the ceilometer for all days. Especially during the first 3 days, an upward-extending PBL can be identified from noon to the late afternoon. Only on 15 September, does a more or less stable residual PBL seems to exist in the morning hours. The other days show high variability in the morning with layer heights from 200 to 600 m with more or less individual high signals, which might be produced by smaller clouds at higher altitudes: 24 September shows a stable layer around 200 m and a diffuse developing PBL in the afternoon.

The underlying extinction coefficient profiles for the above-discussed bottom extinction and AOT values can be seen in Fig. . In the top row, temporal and vertically averaged ceilometer profiles are depicted. In the middle row, these data were smoothed to the MAX-DOAS vertical resolution by the application of BOREAS averaging kernels (xnew=xapri+A(xceilo-xapri); see ). The first day shows a good agreement between BOREAS and ceilometer near-surface extinction with a small offset between the curves. In the morning, both instruments find the aerosol load mainly located in the lowermost layers. Beginning around noon, the PBL starts developing with a maximum height of 2 km in the afternoon found by the ceilometer. BOREAS cannot resolve this increasing PBL as well as the averaged ceilometer but the AK smoothed data indicate that the reason might be limited sensitivity to the top boundary of the PBL.

On 15 September, the averaged ceilometer data show high and thick clouds in the morning and a rising PBL in the afternoon. In the AK-smoothed data, these clouds cannot be identified any more. BOREAS introduces elevated aerosol layers which can be understood as a retrieval artefact due to these cloudy scenes. In the afternoon, BOREAS finds an upward expansion in the PBL similar to the ceilometer, with the exception of the last profile, which was influenced by the telescope pointing towards the Sun. BOREAS extinction values are smaller in the PBL, which is a consequence of the already explained underestimation of BOREAS AOT to AERONETs AOT, which was used for the backscatter signal scaling.

Several instruments were used for the validation of BOREAS nitrogen dioxide profiling results. The VCD results were validated with the help of a Pandora instrument (no. 128) operated by LuftBlick and a NO2 lidar from RIVM. In addition, three NO2 sondes launched on 15 September by KNMI were also used.

The Pandora instrument was used with its direct Sun capability to retrieve total nitrogen dioxide columns on hourly time steps (see Fig. ). Since the measurement time did not match to BOREAS scans, Pandora values were resampled. The stratospheric column was subtracted based on the approach of by using OMI stratospheric columns during the Cabauw overpass (L2 OVP, ). The data were quality filtered and the errors were provided by LuftBlick and are based on .

The NO2 lidar did not measure as frequently as Pandora and MAX-DOAS instruments but performed several measurements on 15 September. One lidar scan took approximately 30 min with the temporal midpoints of the scans on a non-regular temporal grid which necessitated resampling. The scan was done on different elevation angles which enabled the retrieval of vertical profiles with altitude points up to 2.5 km. For the VCD calculation, these profiles were vertically integrated. Errors were provided by the RIVM team and Gaussian error propagation was applied for the calculation of VCD errors. In addition to the remote-sensing instrument, NO2 sondes were launched on 15 September around 05:15, 08:04 and 10:25 UTC. The sondes provide NO2 VMR as well as pressure, temperature and altitude data for the conversion to concentrations, which were vertically integrated up to 4 km.

Near-surface concentration validation was done with the help of in situ (NAQMN, ICAD, CAPS) and remote-sensing instruments (lidar and LP DOAS). The in situ samplers NAQMN, CAPS and ICAD were operated by RIVM, the Royal Belgian Institute for Space Aeronomy (BIRA-IASB) and the Institute of Environmental Physics in Heidelberg (IUPH) respectively. In addition to the aerosol bottom concentration, NAQMN also provided NO2 concentrations (Teledyne API 200E; see Sect.  for details on data handling). IUPH and BIRA-IASB created a common NO2 in situ data set from IUPHs ICAD (Iterative CAvity DOAS, ) and BIRAs CAPS (Environment SA, AS32M-CAPS, Cavity Attenuated Phase Shift, ). This joint data set filled gaps in the individual measurement series and fixed a scaling issue for CAPS. BIRA-IASB operated two different CAPS instruments, which were installed at the 27 and 200 m levels of the Cabauw measurement tower. The joint ICAD/CAPS data set as well as the 200 m CAPS data were averaged over 15 min time steps. For both instruments (ICAD/CAPS) an error of 1 ppb was assumed.

In addition to the in situ instruments, a long-path DOAS (LP DOAS) provided by IUPH was used for the validation. Four reflectors attached at different altitude levels of the measurement tower (12.7, 47, 107 and 207 m) provided unique and well-defined light paths which enabled the calculation of concentration on several altitudes. The instruments' elevation measurements led to profiles every 30 min. Errors were calculated by the IUPH team.

Nitrogen dioxide profiles were provided only by the NO2 lidar and sonde launches. The LP DOAS did not cover enough altitude levels for a comparison. The conversion from trace gas VMR to concentrations for all data sets was done with the meteorological data from the CESAR observatory .

Figure  depicts the comparison of VCD for different instruments. The agreement with Pandora is good on all days with larger deviations in the morning and on 15 September in general with the exception around noon. Deviations in the morning might be (similarly to the AERONET comparison) due to spatial differences in the NO2 distribution, when Pandora points to the east and MAX-DOAS to the west. This argument is supported by the lidar measurement on 15 September that agreed well with BOREAS and shared the same azimuthal viewing direction. In the afternoon of that day, BOREAS VCD increased faster than Pandora and lidar, with a strong overestimation for the last profile at 16:00 when the Sun was low. Here, we can exclude horizontal gradients as a reason for the differences, because of the same viewing direction of lidar and MAX-DOAS. On the same day, NO2 sondes proved that the BOREAS VCD was correct only during the ascent (triangle with edge at the top) of the second launch. The matching descent (triangle with edge at the bottom) for the first flight was approximately 38 km in the north-westerly direction between Amsterdam and Rotterdam and agrees more with Pandora columns, indicating differences due to azimuthal viewing directions. The strong differences in VCD for ascent and descent of the first and second sonde launches show high temporal and spatial variability of the NO2 concentration, which is supported by all remote-sensing instruments.

The correlation with Pandora is slightly better than with lidar but the regression parameter shows that Pandora VCDs are higher than those from BOREAS, whereas the lidar regression is closer to 1.

In Fig. , near-surface NO2 concentrations are depicted. Both in situ data sets (NAQMN and ICAD/CAPS) agree well with each other and show larger differences only in the morning of 14 September. The LP DOAS profiles at 12.7 and 47 m show similar concentrations to the in situ instruments. Differences in the morning of the first 2 days between the lower and the higher LP DOAS values indicate a strong vertical inhomogeneity, which was also found for the aerosols (see Sect. ).

BOREAS surface concentrations agree very well with all data sets except during the morning hours. On 13 September, in addition to the ICAD/CAPS data at the 27 m level (blue), the 200 m points are shown (green). The anti-correlated behaviour in the morning hours confirms the strong inhomogeneity and proves that nitrogen dioxide was mainly concentrated close to the surface at that time. The mean values of both CAPS instruments (yellow line) show a better agreement with BOREAS than the 27 and 200 m data.

It seems that BOREAS cannot fully resolve this thin near-surface layer and retrieves rather smooth profiles instead of sharp concentration peaks (see also the discussion about vertical sensitivity in Sect. ). This is again in agreement with the LP DOAS results on 24 September, where the 200 m level LP DOAS finds similar concentrations to BOREAS. Note that the NO2 lidar has a finer resolution within the lowest 100 m, which might be the reason it depicts similar concentration numbers to the other instruments in the morning hours of 15 September. As a rough estimation of the thickness of the individual thin layers in the morning, we divided BOREAS VCD by the ICAD/CAPS concentration and found that the layer height lies around 200 m (06:00–08:00), 300 m (09:00) and 400 m (10:00) on 13 September. The mean AK FWHM for the surface layer on that day is 214 m, indicating that the lowermost layer can be resolved only from around 09:00 UTC where the curves approach each other. A good agreement is found at 10:00 UTC when the concentration is focused in a layer twice as thick as BOREAS surface resolution.

The correlations are high for all instruments with the highest value for NAQMN and the NO2 lidar. The in situ instruments show similar correlations, but a slight underestimation of BOREAS can be found with slopes between 1.22 and 1.44.

Figure  depicts NO2 profiles for BOREAS, lidar and sondes on 15 September. Similarly to the aerosol profile comparison, averaging kernels were applied to lidar and sondes measurements. The vertical profiles of the unsmoothed lidar show the previously discussed high vertical inhomogeneity. We can clearly identify a high concentration close to the surface and another elevated NO2 layer with altitudes varying between 100 and 500 m. The unsmoothed sondes agree well with the lidar measurements only at the ascent of the second launch. After the application of averaging kernels, the two distinct layers with high concentrations are smoothed to one layer with NO2 concentrated at the surface, again indicating a lower vertical sensitivity of MAX-DOAS profiling. In addition, smoothed and unsmoothed lidar find larger concentrations for altitudes over 500 m before 08:00 UTC, which was not found by BOREAS and the sonde profiles.