Maximizing the scientific application of Pandora column observations of HCHO and NO₂

As part of the Pandonia Global Network (PGN), Pandora spectrometers are widely deployed around the world. These ground-based remote-sensing instruments are federated such that they employ a common algorithm and data protocol for reporting on trace gas column densities and lower-atmospheric profiles using two modes based on direct-sun and sky-scan observations. To aid users in the analysis of Pandora observations, the PGN standard quality flagging procedure assigns flags to the data indicating high, medium, and low quality. This work assesses the suitability of these data quality flags for filtering data in the scientific analysis of formaldehyde (HCHO) and nitrogen dioxide (NO₂), two critical precursors controlling tropospheric ozone production. Pandora data flagged as high quality assure scientifically valid data and are often more abundant for direct-sun NO₂ columns. For direct-sun HCHO and for sky-scan observations of both molecules, large amounts of data flagged as low quality also appear to be valid. Upon closer inspection of the data, independent uncertainty is shown to be a better indicator of data quality than the standard quality flags. After applying a filter to independent uncertainty, Pandora data flagged as medium or low quality in both modes can be demonstrated to be scientifically useful. The utility of this filtering method is demonstrated by correlating contemporaneous but independent direct-sun and sky-scan observations. When evaluated across 15 Pandora sites in North America, this new filtering method can recover as much as 90 % of data that would have previously been discarded. This method suggests that standard PGN criteria for atmospheric variability and the normalized root mean squared error are too stringent, as they are responsible for downgrading most of the recovered data. A method is also developed for combining the direct-sun and sky-scan observations into a single dataset by accounting for biases between the two observing modes and differences in measurement integration times. These combined data provide a more continuous record that is useful for interpreting Pandora observations against other independent variables such as hourly observations of surface ozone. When Pandora HCHO columns are correlated with surface ozone measurements, data filtered by independent uncertainty exhibit similarly strong and more robust relationships than high-quality data alone. These results suggest that Pandora data users should carefully assess data across all quality flags and consider their potential for useful application to scientific analysis. The present study provides a method for maximizing use of Pandora data with the expectation of more robust satellite validation and comparisons with ground-based observations in support of air quality studies.

1 Introduction

The Pandora spectrometer is a ground-based ultraviolet–visible (UV-VIS) remote-sensing instrument utilized to validate space-based UV-VIS sensors and understand local air quality (Herman et al., 2009, 2019; Wang et al., 2010; Spinei et al., 2018; Tzortziou et al., 2012; Szykman et al., 2019; Judd et al., 2019; Verhoelst et al., 2021; Di Bernardino et al., 2023). The Pandora instrument was developed to observe column nitrogen dioxide (NO₂) in direct-sun mode, in addition to the more traditional sky-scan observations obtained using multi-axis differential optical absorption spectroscopy (MAX-DOAS). Once calibrated, direct-sun observations are less impacted by atmospheric parameters that influence the air mass factor (AMF) and that also cause uncertainty in space-based retrievals, such as the vertical gas profile or surface albedo (Cede et al., 2006; Herman et al., 2009). More recently, Pandora instrument retrievals of column formaldehyde (HCHO) have become available, expanding opportunities to evaluate the importance of both nitrogen oxides (NO_x) and volatile organic compounds (VOCs) as precursors of ozone and secondary aerosol. This was enabled by the removal of instrument components made of Delrin, which were discovered to outgas HCHO and interfere in its detection (Spinei et al., 2021).

The Pandonia Global Network (PGN) is a NASA- and ESA-sponsored ground-based network of Pandora spectrometers supporting in situ and remotely sensed air quality monitoring and satellite validation activities (https://www.pandonia-global-network.org, last access: 27 February 2025). The major advantages of the PGN are uniform instrument design, homogeneous calibrations, centralized data monitoring and processing, and real-time data distribution (Herman et al., 2009, 2018, 2019; Spinei et al., 2018; Cede, 2024). Pandora instruments are calibrated by the NASA Goddard Space Flight Center and deployed to local operators around the world who have joined the PGN. Consistent retrievals across the network are performed by LuftBlick Earth Observation Technologies and are hosted on the PGN website (Cede, 2024). Presently, there are more than 130 Pandora instruments worldwide and over 60 across the USA (Szykman et al., 2019; Chang et al., 2022).

Figure 1 Block diagram of a Pandora instrument and its two viewing geometries.

Pandora instruments have provided valuable information on NO₂ retrieval biases from the polar-orbiting Sentinel-5P TROPOspheric Monitoring Instrument (TROPOMI) (Judd et al., 2020; Verhoelst et al., 2021; Park et al., 2022; Ialongo et al., 2020) and the Ozone Monitoring Instrument (OMI) (Tzortziou et al., 2012; Reed et al., 2015; Herman et al., 2019). The high temporal availability of Pandora data has allowed for deep investigation of local meteorology using column NO₂ and has shown how well that is captured by air quality models (Goldberg et al., 2017; Choi et al., 2020; Tzortziou et al., 2022, 2023; Wang et al., 2023; Adams et al., 2023). The relatively new formaldehyde retrievals from Pandora (Spinei et al., 2018, 2021; Cede et al., 2025) are beginning to be incorporated into model and satellite validation efforts (Herman et al., 2018) and show promise for the characterization of the local ozone photochemical environment (Schroeder et al., 2016; Travis et al., 2022).

Figure 2 (a) Pandora standard schedule of operations (UTC hour) on a clear-sky day, 27 September 2022, at the University of Houston site for different observing modes with a zoomed-in 18 min window for both L0 (b) and L2 (c) file information. The black points in the L0 schedule show the start of each measurement time, and the underlying red line is the total time in the respective mode. The L2 data (c) show the alternating retrieval of NO₂ (without filter) and HCHO (with the U340 filter) column from direct-sun and sky-scan modes.

Geostationary satellite observations provide a new challenge in interpreting NO₂ and HCHO columns at unprecedented spatial and temporal scales (Kim et al., 2017; Judd et al., 2018). Pandora instruments will be the main source of validation data for both NO₂ and HCHO columns for the Tropospheric Emissions: Monitoring of Pollution (TEMPO) geostationary air quality satellite launched over the USA in 2023 (Szykman and Liu, 2023; Zoogman et al., 2017). Pandora instruments are also being used to validate the Geostationary Environmental Monitoring Spectrometer (GEMS) satellite retrievals over East Asia (Kim et al., 2023). Given the growing need for Pandora data to support satellite observations and model development, improved methods for ensuring data availability and quality are needed. In this work, a data filtering method is presented for assessing the suitability of HCHO and NO₂ column data beyond the standard quality flags provided by PGN. This method takes advantage of the independent but contemporaneous measurement modes of Pandora operation (direct sun and sky scan) to analyze Pandora data quality and maximize the availability of data for scientific application. The method presented here is but one approach to examine Pandora data quality. The user community should continue to carefully consider how to best ensure data quality while preserving the most data possible.

2 Data and methodology

2.1 Pandora measurements and data retrieval

The Pandora spectrometer is an instrument for ground-based remote sensing of a range of trace gases, including NO₂, HCHO, O₃, and SO₂, with a temporal resolution on the order of seconds to minutes. Pandora is a passive UV-VIS spectrometer system equipped with a responsive sun tracker that points the optical head sensor at the sun or sky with a spatial accuracy of up to ±0.1°. A basic description of the Pandora instrument is summarized here, with additional details provided in Sect. S1.1 in the Supplement and available in Cede et al. (2024). Briefly, light is transmitted through fiber optical cables to a UV-VIS low stray-light spectrometer (Herman et al., 2009; Tzortziou et al., 2012). The spectrometer covers a spectral range of 280–530 nm with a spectral resolution of 0.6 nm. To minimize the dark-current noise, the spectrometer is maintained at a stable temperature of 15–20 °C (based on the site location) using a thermal electrical cooler. The Pandora spectrometer operates in two daytime viewing geometries, namely, direct sun (DS) and scattered sunlight or sky scan (SS) (Fig. 1), with varying degrees of temporal resolution spanning from milliseconds to minutes depending on the mode of operation.

The Pandora retrieval involves several sequential steps or data processing levels that are all archived by PGN. Raw measurement spectra are archived as L0. L1 data include corrections for instrumental characteristics such as dark signal, nonlinearity, flat field, temperature sensitivity, simple stray-light removal, and wavelength corrections. L2Fit data include spectral fitting, which enables the calculation of slant columns relative to the reference spectrum employing DOAS analysis (Herman et al., 2009; Cede et al., 2022; Gebetsberger and Tiefengraber, 2023). L2 data result from the conversion of slant columns to vertical columns utilizing geometrical AMFs for direct-sun and analytical methods for sky-scan observations. Sky-scan observations also include profile and surface concentrations. These products are briefly described in Sect. S1.1. The reported accuracy of the DS total vertical column of NO₂ is $\mathrm{2.7}\times {\mathrm{10}}^{\mathrm{15}}\mathrm{molecules}{\mathrm{cm}}^{-\mathrm{2}}$ (0.1 Dobson unit, where $\mathrm{1}\mathrm{DU}=\mathrm{2.7}\times {\mathrm{10}}^{\mathrm{16}}\mathrm{molecules}{\mathrm{cm}}^{-\mathrm{2}}$), with a precision of $\mathrm{2.7}\times {\mathrm{10}}^{\mathrm{14}}\mathrm{molecules}{\mathrm{cm}}^{-\mathrm{2}}$ (0.01 DU) (Herman et al., 2009). For HCHO DS, a small statistical error up to 6 % and a notable systematic error up to 26 % have been reported (Spinei et al., 2018, 2021), while the recent Pandora Delrin-free instruments require more assessment to quantify accuracy and precision. The Pandora SS measurements still lack robust validation; however, the CINDI-2 intercomparison included a Pandora (called a NASA instrument) instrument and reported a bias for measurements in the SS mode of about $-\mathrm{0.05}\times {\mathrm{10}}^{\mathrm{16}}\mathrm{molecules}{\mathrm{cm}}^{-\mathrm{2}}$ for the HCHO column and about $-\mathrm{0.02}\times {\mathrm{10}}^{\mathrm{16}}\mathrm{molecules}{\mathrm{cm}}^{-\mathrm{2}}$ for the NO₂ column against the median of the participating instruments (Tirpitz et al., 2021; Verhoelst et al., 2021). Additional details on current Pandora software, recent hardware changes, and the retrieval scheme are discussed in Sect. S1.1.

Figure 3 Percentage contribution of high- (0, 10), medium- (1, 11), and low- (2, 12) quality flags for 15 Pandora sites for observations in both DS (a, c) and SS (b, d) modes during 2021–2022. Percentage contributions are shown for NO₂ in the top panels and HCHO in the bottom panels.

2.2 Pandora observational schedule

Pandora instruments are locally operated and mainly run on the standard schedule alternating between DS and SS mode, although there is an option for user-defined schedules. The DS and SS modes also alternate between an open (unfiltered) mode for NO₂ and a U340 (filtered for maximum transmission near 340 nm) mode for HCHO observations (Cede et al., 2024; Spinei et al., 2018). A typical example of the standard Pandora schedule on a clear-sky day is shown in Fig. 2a, with the typical total hours per day for each routine annotated. Typically, Pandora spends about 1.5 h each day in DS NO₂ (SQ) and DS HCHO (SS, not to be confused with “sky-scan”, refers to the quick direct-sun scan using the U340 filter; see Sect. S1.2) modes and about 1.5 h each day in SS mode for lower-tropospheric columns of NO₂ and HCHO (EO, which refers to the quick elevation scan with the open filter) and HCHO only (EU, which refers to the quick elevation scan with the U340 filter). Longer, more detailed sky scanning (EL, which refers to the detailed elevation scan with U340 filter, or EK, which refers to the detailed elevation scan with the open filter; see Sect. S1.2) occurs for a total of about 1–2 h for profile observations. Find sun (FS) routines occur throughout the day but only take a short time (a few seconds) over the course of each hour. Typically, Pandora spends 65 % of the time in SS mode and 30 % in DS mode under clear-sky conditions. Further details of all Pandora routine acronyms are described in Sect. S1.2 in the Supplement and in the Pandora user manual (Cede et al., 2024). The inset panels in Fig. 2b and c provide a closer view of the schedule for an 18 min window as the instrument switches between DS and SS modes. The varying effective measurement times (t_eff) illustrate the observation frequency for each mode and product. Generally, the Pandora instrument possesses the ability to capture solar spectra at adjustable integration times ranging from 2.5 ms to 4 s, with an overall measurement duration of approximately 40 s, encompassing a few dark-current measurements (Cede et al., 2024). The integration time for DS NO₂ (with no filter) under bright sun is about 4 ms, and roughly 4000 spectra are acquired repeatedly and averaged to obtain very high signal-to-noise ratios with high precision (Herman et al., 2018). For DS HCHO (with the U340 filter), a longer integration time (30–1000 ms) is used; hence, there are fewer measurements for HCHO compared to NO₂. Typically, in the DS mode, there are about 5 times more observations of NO₂ compared to those of the HCHO column per scan (Fig. 2c). In the SS mode, elevation scans are most effective when taken along an azimuth pointing away from the sun to minimize the effect of scattering. The zenith angle scans can vary by site, with the lowest scan typically occurring between 89° and 85° depending on viewing conditions available at the site to the highest zenith angle (0°), which is utilized as the reference during the spectral fitting (details of different zenith angles are provided in Sect. S1.2). The extended SS measurements are taken at multiple zenith angles (Sect. S1.2) to derive lower-tropospheric columns and profiles typically up to 2–4 km.

Figure 4 Correlation between contemporaneous DS and SS observations of the HCHO column for different quality flag combinations. Data are from the University of Houston Pandora site over 2 years (2021–2022).

3 Pandora data quality flags and an alternative data filtering method

The PGN software suite performs retrievals for NO₂ and HCHO and provides a measure of data quality to guide users (Cede et al., 2024, 2025). Reduced data quality can arise from instrumental and/or atmospheric sources. Data flags provide information on both the data quality and data assurance level (Gebetsberger et al., 2022). Data quality is indicated in the units position, with a 0, 1, or 2 indicating high, medium, and low quality, respectively. While data that are preliminary but not yet quality assured are useable for science, this status is indicated by a 1 in the tens position. Data that are unusable for whatever reason are indicated by a 2 in the tens position. Thus, data flagged as 0 or 10 are considered useable and of high quality, and users are cautioned to pay additional scrutiny to data flagged as 1 or 11 (medium quality). In contrast, users are discouraged from using any data flagged as 2 or 12 (low quality). These quality flags are assigned in various stages from L1 (raw data) to L2fit (spectral fit data) to L2 (processed data) to ensure data quality through multiple checks at each stage. At the L1 stage, data are flagged into either low- or medium-quality bins based on instrument-related issues such as excessive dark counts, detector saturation, a dark count that differs significantly from the dark map for too many pixels, different effective temperature, and unsuccessful dark-background fitting. In the L2fit stage, where spectral fitting is performed, data are further flagged based on factors including the quality of the fit, the wrms limit (normalized root mean square of fitting residuals weighted with independent uncertainty), and wavelength shift. Finally, at the final retrieval L2 stage, factors including retrieval error and atmospheric variability are used to flag data into the medium- or low-quality groups. More description of the data quality flags is provided in Gebetsberger et al. (2022).

Table 1 Correlations (R²) between hourly surface ozone and HCHO observations from direct sun and sky scan for high-quality data flags alone and for data using the new filtering method. The analysis is for four Pandora sites during September 2022. Each site is located near (within 500 m of) an ozone monitoring instrument. The number (n) is the available hourly matchups for each site.

Following the suggestion of the Pandora user manual (Cede et al., 2024, 2025), many previous studies with Pandora v1.8 data utilize only high-quality data (0, 10) (e.g., Park et al., 2022; Verhoelst et al., 2021; Liu et al., 2023). However, when using only high-quality-flagged data, a large fraction of observations is often eliminated. Figure 3 shows quality flag statistics across the 15 Pandora sites evaluated in this study (listed in Table S1 in the Supplement). They have been selected since they fall under recent NASA airborne campaigns that covered areas of the eastern USA and Houston, TX. These are the Tracking Aerosol Convection Interactions Experiment – Air Quality (TRACER-AQ) in 2021, the Student Airborne Research Program (SARP-East) in 2023, and the Synergistic TEMPO Air Quality Science (STAQS) study in 2023. Across these sites, high-quality-data flags dominate only for DS NO₂ observations during the 2021–2022 period, typically being between 60 % and 80 %. For other observations, low-quality-data flags often dominate, on average accounting for 57 %, 45 %, and 46 % of data for SS NO₂, DS HCHO, and SS HCHO, respectively, across the evaluated sites, making much of the data unavailable to support field campaign analysis.

Figure 5 Frequency distribution of independent uncertainty in three quality flags: high (blue), medium (green), and low (red) for (a) column HCHO direct-sun, (b) column HCHO sky-scan, (c) column NO₂ direct-sun, and (d) column NO₂ sky-scan observations at the University of Houston Pandora site. The cumulative distributions for each panel are inset with the cutoff values used in the present analysis shown as a dashed vertical black line. Data values exceeding the cutoff are shaded in gray for the frequency distributions.

The schedule of Pandora operations shown in Fig. 2 demonstrates the contemporaneous nature of DS and SS observations. Figure 3 further shows that the quality flags for the two observing methods can often differ substantially. This enables an independent assessment of data quality by taking advantage of the expected autocorrelation of contemporaneous (within 5 min) DS and SS observations for different quality flag combinations. An example is provided in Fig. 4, showing HCHO observations from the University of Houston Pandora site. Note that there are nine possible combinations of high-, medium-, and low-quality data pairs. More than half of those data pairs (57 %) include one low-quality observation. Low-quality SS observations correlate equally well or better with medium- and high-quality data, as do data pairings including only medium- and high-quality data. There is more scatter in the data that include low-quality DS observations (right column of panels in Fig. 4), but there is still evidence of a large number of well-correlated data points, even for pairings of low-quality data for both DS and SS modes. Given the independent nature of the DS and SS observations, it is worthwhile to examine the role of measurement uncertainty in these correlations.

Figure 6 Correlation between contemporaneous DS and SS observations of the HCHO column for different quality flag combinations at the University of Houston Pandora site over 2 years (2021–2022). Data points are colored by cutoff values used for HCHO column independent uncertainty values. Gray indicates points for which both DS and SS uncertainties are less than the cutoff; other colors indicate points for which only DS (cyan), only SS (yellow), or both DS and SS (red) uncertainties exceed the cutoff value.

Figure 7 Correlation between contemporaneous DS and SS observations of the HCHO column for different quality flag combinations after applying a new filtering method to the University of Houston Pandora site for 2021–2022. Data points are colored based on the count within each hexagonal bin (grid size of 100).

Figure 5 shows the frequency distribution of independent uncertainty for high-, medium-, and low-quality Pandora HCHO and NO₂ column observations for both DS and SS mode for nonnegative vertical columns at the University of Houston Pandora site. Independent uncertainty is a measure of the photon noise across the detector pixels that is propagated to slant columns for each retrieval and is provided in the L2 product. Figure 5 shows that there is significant overlap in the independent uncertainty distribution across all three quality flags for each measurement mode. The figure inset shows the corresponding cumulative probability distributions and cutoff value (dotted black line) that preserves almost all high-quality data while including a significant amount of overlapping data flagged as medium or low quality. In the present analysis, the cutoff value for independent uncertainty is defined as μ+3σ of the independent uncertainty distribution for high-quality data, where μ is the mean and σ is the standard deviation. The gray shaded area in Fig. 5 shows the portion of the data falling outside this limit. For the University of Houston Pandora, the cutoff values are $\mathrm{0.37}\times {\mathrm{10}}^{\mathrm{15}}\mathrm{molecules}{\mathrm{cm}}^{-\mathrm{2}}$ for DS independent uncertainty and $\mathrm{1.79}\times {\mathrm{10}}^{\mathrm{15}}\mathrm{molecules}{\mathrm{cm}}^{-\mathrm{2}}$ for SS independent uncertainty in HCHO column measurements. For NO₂ column measurements, the cutoff values are $\mathrm{0.62}\times {\mathrm{10}}^{\mathrm{14}}\mathrm{molecules}{\mathrm{cm}}^{-\mathrm{2}}$ for DS independent uncertainty and $\mathrm{1.81}\times {\mathrm{10}}^{\mathrm{14}}\mathrm{molecules}{\mathrm{cm}}^{-\mathrm{2}}$ for SS independent uncertainty. In this method, Pandora data with independent uncertainty lower than the cutoff limit are further scrutinized for scientific utility regardless of their data flag.

Figure 8 Correlation between contemporaneous DS and SS observations of the NO₂ column for different quality flag combinations at the University of Houston Pandora site for 2021–2022. Data points are colored by the cutoff values used for NO₂ column independent uncertainty values. Gray indicates points for which both DS and SS uncertainties are less than the cutoff; other colors indicate points for which only DS (cyan), only SS (yellow), or both DS and SS (red) uncertainties exceed the cutoff value.

Figure 9 Correlation between contemporaneous DS and SS observations of the NO₂ column for different quality flag combinations after applying a new filtering method to the University of Houston Pandora site for 2021–2022. Data points are colored based on the count within each hexagonal bin (grid size of 150).

Figure 10 The new data filtering method used in the present analysis to increase the scientific availability of Pandora observations.

Figure 11 Quality flags (QFs) for each Pandora site in 2021–2022 (left bars) colored by green (high), blue (medium), and red (low) for (a) NO₂ DS, (b) NO₂ SS, (c) HCHO DS, and (d) HCHO SS. The adjacent gray bars show the total observation counts after applying the new filtering method.

Figure 12 Relationship between the Pandora HCHO column and surface ozone observations at the University of Houston site during TRACER-AQ in September 2021. Scatter plots between the sky-scan HCHO column and surface ozone for (a) only high-quality flags and (c) data with the new filtering method. Scatter plots between the direct-sun HCHO column and surface ozone for (b) only high-quality flags and (d) data with new filtering method. The HCHO column is given in Dobson units ($\mathrm{1}\mathrm{DU}=\mathrm{2.69}\times {\mathrm{10}}^{\mathrm{16}}\mathrm{molecules}{\mathrm{cm}}^{-\mathrm{2}}$).

Figure 13 Comparison of airborne GCAS trace gas columns and Pandora columns of NO₂ (panels a and b) and HCHO (panels c and d) during TRACER-AQ for Houston Pandora sites (Aldine, La Porte, and University of Houston). Comparisons are shown for both direct-sun (panels a and c) and sky-scan (panels b and d) modes.

Figure 14 (a) Mean differences between the DS and SS columns for HCHO and NO₂ over the 15 Pandora sites for 2021–2022. The NO₂ stratospheric column for each site is also provided. The bottom panels show the variation in mean column difference with solar zenith angle (SZA) for the NO₂ column (b) and HCHO columns (c).

This proposed data filtering method is evaluated in Fig. 6 for HCHO observations at the University of Houston Pandora site, showing how specific data pairs are related to the independent uncertainty cutoff values. Data pairs with uncertainty falling above the cutoff for DS only (blue points) and for both DS and SS (red) are almost exclusively limited to the low-quality-flagged data (right column and bottom row of panels in Fig. 6). These points also account for most of the scatter in the correlations. Data pairs with uncertainty falling above the cutoff for SS data only (yellow) tend toward larger values of HCHO in the column. This calls for a second filtering step in which observations with independent uncertainties of less than 10 % of the column amount are retained even if the absolute independent uncertainty exceeds the cutoff value. This preserves a small but important set of large column abundances that are important for exploring the full dynamic range of column variability.

Finally, there can be a few rare outliers (<0.1 %) associated with unusually large wrms values even after applying this independent uncertainty cutoff. For this analysis, we empirically determine that these outliers are associated with unusually high values for DS and SS data of high wrms and for SS data of maximum horizontal distance (MHzD). These outliers can be removed with thresholds for wrms of 0.01 and for MHzD of 20 km. Specific column numbers from Pandora files for the variables used in this analysis are provided in Table S2 in the Supplement to facilitate the application of this method.

Figure 7 shows the filtered data for the University of Houston Pandora site colored by data density. After the filtering method is applied, correlations notably improve for combinations that include DS low-quality-flagged data. Among all pairings ($R^{\mathrm{2}}=\mathrm{0.72}-\mathrm{0.86}$), data quality flags no longer appear to be a useful predictor of correlation between contemporaneous SS and DS observations. This suggests that all remaining data may be scientifically useful regardless of the quality flag. Figure 7 also shows that there is a bias between SS and DS observations for the HCHO column. This is not unexpected, given that SS observations are limited to the lower troposphere, but as discussed later in Sect. 3.3, this bias must be addressed for applications where it is advantageous to combine SS and DS observations into a larger unified dataset.

Figure 8 shows the same plot as Fig. 6 but for NO₂ columns at the University of Houston Pandora site. There are more total pairs in this comparison since DS NO₂ observations are more frequent than those of HCHO (see Fig. 2). Again, a reasonable correlation is observed between all the quality flags ($R^{\mathrm{2}}=\mathrm{0.68}-\mathrm{0.73}$). For NO₂, the filtering method leads to very few data being removed; thus correlations are essentially unchanged in Fig. 9 for the filtered data. Again, there is a bias, with DS values being greater than SS values. This is mainly due to stratospheric NO₂, which is only detected in the DS mode (see discussion in Sect. 3.3). Figures 8 and 9 show that for NO₂, the frequent DS retrievals enable multiple DS measurements to be matched to a single SS observation within the 5 min matching window.

The method demonstrated in Figs. 4 through 9 is summarized in Fig. 10. It can be applied to any PGN site with contemporaneous DS and SS observations. Results of applying the method are shown in Fig. 11 for the 15 Pandora sites falling under recent NASA airborne campaigns. Data from 2021 and 2022 are used for all sites, although La Porte, Texas, only has data from August 2021 to February 2022. The bars in Fig. 11 show the occurrences of the high- (green), medium- (blue), and low- (red) quality-flagged data across these 15 Pandora sites. The amount of available data varies considerably between sites. This can be due to a number of reasons, including instrument down time, the use of nonstandard observing schedules, and periods of data flagged as unusable (20, 21, or 22). The gray bars show the amount of usable data after applying the new filtering method. For HCHO data, 35 % is high quality on average. This increases to 77 % on average when the new method is applied. For NO₂, this improvement is from 46 % to 97 %.

In Sect. S1.3 in the Supplement, data quality flags are traced for the University of Houston Pandora dataset used in Figs. 6 through 9. For DS HCHO, most data are flagged as medium or low quality, primarily due to a violation of the data quality indicators (DQ) for atmospheric variability. Data are also flagged due to high wrms and saturated data. The data recovery procedure shown in Fig. 10 restores 34 % of data flagged for high wrms and 70 % of data flagged for high atmospheric variability. The large atmospheric variability values are assumed to be related to cloudiness. However, there appears to be no correspondence between atmospheric variability values and actual sky conditions (Fig. S1 in the Supplement). For SS HCHO, flagging primarily occurs due to saturation or dark-count issues in the L1 stage and high wrms in the L2fit stage. Unlike DS measurements, atmospheric variability is rarely triggered for sky-scan HCHO despite observations showing a tendency toward greater atmospheric variability. Of the flagged data, 75 % of saturated or dark-count-affected data and 70 % of high-wrms-flagged data are recovered. For NO₂, DS data are primarily flagged for exceeding atmospheric variability thresholds, while SS data are flagged for high wrms values. This analysis suggests that the PGN criteria for atmospheric variability, wrms, dark counts, and saturated data may be overly stringent. Of these PGN criteria, wrms is the only one considered in the method outlined in Fig. 10, but the threshold value is empirically determined independent of the flagging. Data users are encouraged to continue to pay attention to the impact of wrms on Pandora data quality for specific instruments.

As noted in Fig. 10, the first step in verifying the success of the method is to demonstrate the quality of correlation across all quality flag combinations. To further strengthen the case for using this method, independent observations are used in the following sections to demonstrate more robust scientific analyses. Evaluation of biases between DS and SS observations and prospects for combining them into a single dataset are also explored.

3.1 Independent assessment of Pandora scientific data quality with HCHO : O₃ analysis

Recent work has shown a strong correlation between surface O₃ and column HCHO (Schroeder et al., 2016; Travis et al., 2022). This relationship is used here to test the effectiveness of the new filtering method on Pandora HCHO column observations for the University of Houston Pandora site during the TRACER-AQ period (September 2021). High-resolution (5 min) collocated surface ozone measurements were obtained from the Texas Commission on Environmental Quality. The Pandora HCHO column measurements in SS and DS modes are temporally matched (within 5 min) with surface ozone data between 10:00 and 18:00 local time, when photochemistry is most active. This analysis compares results using only high-quality-flagged Pandora data to results using Pandora data filtered with the new method.

Figure 12 shows the relationship between the HCHO column and surface ozone using DS and SS observations. When the new filtering method is applied, the number of data points increases 5-fold for SS (181–955) and 2-fold for DS (275–696). In both cases, the R² remains similar or improves. However, the substantial increase in data density will lead to more robust scientific analysis of the HCHO : O₃ relationship. Similar results were obtained for other months outside the TRACER-AQ period that are not shown here.

This analysis is expanded in Table 1 to include Pandora sites that are located within 500 m of an Environmental Protection Agency (EPA) regulatory monitor reporting hourly ozone. Data are for the month of September 2022, which had the best data availability. Table 1 summarizes the resulting correlation between hourly averaged column HCHO in SS and DS modes and hourly surface ozone. Generally, similar or improved correlation and increased data availability (14 %–163 %) were observed after applying the new filtering method. However, over the New Brunswick site in DS mode, a lower correlation (0.60–0.52) was observed after applying the new method. This was caused by the new method capturing an ozone event (>60 ppb) that appeared to be decoupled from high HCHO columns on 19 September 2022. This independent analysis shows that the recovered data can both improve the analysis of the HCHO : O₃ relationship and provide confidence in the scientific value of the new filtered dataset.

3.2 Independent assessment of Pandora scientific data quality with airborne remote sensing

GEO-CAPE airborne simulator (GCAS) observations (Nowlan et al., 2018; Judd et al., 2020) provide an independent dataset of column NO₂ and HCHO observations to compare against Pandora over Houston, TX during the TRACER-AQ period (September 2021). Looking downward as it flies overhead at ∼8.5 km, GCAS measures a partial column from the aircraft altitude down to the surface that includes the polluted boundary layer and a large portion of the less polluted free troposphere. Additionally, the GCAS tropospheric column measurements have been shown to strongly correlate with in situ aircraft observations for NO₂ (R²=0.89) and HCHO (R²=0.54), with column differences in magnitude within 10 % (Nowlan et al., 2018). Figure 13 shows the comparison of DS and SS observations with GCAS NO₂ and HCHO. To minimize the temporal and spatial mismatch between the Pandora and GCAS observations, GCAS was spatially filtered to include the nearest cloud-free GCAS pixels within 2.5 km of the Pandora location, and Pandora was temporally filtered to include observations in a 15 min window centered on the time of the GCAS overpass.

Both the DS and SS NO₂ columns are well-correlated with GCAS (R²=0.78 and 0.64, respectively). Some scatter can be attributed to GCAS uncertainties being on the order of 25 % (Judd et al., 2020; Nawaz et al., 2024) compared to a priori uncertainties related to surface reflectivity and the trace gas profile. For DS NO₂ columns, there is a clear offset, with Pandora columns exceeding GCAS values. The slope near unity and the intercept of $\mathrm{3}\times {\mathrm{10}}^{\mathrm{15}}\mathrm{molecules}{\mathrm{cm}}^{-\mathrm{2}}$ are consistent with the stratospheric amount that would be absent in the GCAS observations below the aircraft. Despite a few outliers (Fig. 11b), there is no notable systematic bias between GCAS and SS NO₂. This is consistent with the SS NO₂ observations being limited to the most polluted lower few kilometers of the atmosphere. It could be argued that GCAS values tend to be slightly higher than SS observations of NO₂ due to the larger altitude range observed.

The correlations between the GCAS and Pandora HCHO columns also show general agreement (R²=0.50 and 0.51) but are weaker, primarily due to the greater uncertainty in HCHO measurements for both instruments. For HCHO, the bias is in the SS rather than the DS observations. This is due to the lack of significant HCHO in the free troposphere and stratosphere; thus, total column values associated with DS observations are more consistent with those of GCAS. Since HCHO is created photochemically, it tends to have a weaker vertical gradient in the polluted boundary layer compared to NO₂, which is much more weighted towards the surface where it is emitted. Thus, the limited altitude range of SS observations is more likely to miss some fraction of lower-atmospheric HCHO and to be biased low with respect to both DS and GCAS HCHO.

Overall, these comparisons are encouraging for both GCAS and Pandora with differences that are consistent with expectations based on the differences in DS and SS viewing sensitivities.

3.3 Direct-sun and sky-scan intercomparison of column measurements

Given that the Pandora instrument spends a significant amount of time in both DS and SS modes, maximum data availability for scientific applications would be achieved by further addressing the causes of the differences in their resulting measurements. DS and SS column measurements are expected to differ in their absolute number due to limitations in the vertical sensitivity (for SS measurements) and retrieval method (Herman et al., 2009; Frieß et al., 2019) and physical differences between tropospheric and stratospheric gas amounts. SS columns generally profile from the surface up to 2–4 km, while DS columns observe the total column throughout the entire atmosphere. Due to its short lifetime and major secondary source from chemistry, the HCHO column has a weak gradient from the surface through the depths of the boundary layer (Schroeder et al., 2016; Crawford et al., 2021). This suggests that the DS HCHO total column should be greater than the SS lower-tropospheric column, and the difference should be largely dependent on the vertical sensitivity of the SS measurement. For the NO₂ column, the stratospheric contribution is the largest influence on the differences between the SS and DS column retrievals. To obtain a tropospheric DS NO₂ column, the NO₂ stratospheric column can be subtracted from the DS NO₂ total column using the stratospheric climatology provided by the PGN in their product based on Brohede et al. (2007).

Figure 14a shows the mean differences between the DS and SS columns for HCHO and NO₂ over the 15 Pandora sites in 2021–2022. The bias for HCHO on average is $\mathrm{3.4}\times {\mathrm{10}}^{\mathrm{15}}\mathrm{molecules}{\mathrm{cm}}^{-\mathrm{2}}$, with a larger bias at Hampton University in Virginia and in New Haven, Connecticut ($>\mathrm{5}\times {\mathrm{10}}^{\mathrm{15}}\mathrm{molecules}{\mathrm{cm}}^{-\mathrm{2}}$), and the smallest bias in Bayonne, New Jersey. The mean HCHO column in DS mode over these 15 sites averaged about $\mathrm{1}\times {\mathrm{10}}^{\mathrm{16}}\mathrm{molecules}{\mathrm{cm}}^{-\mathrm{2}}$, making the bias between DS and SS mode approximately 35 % of the column.

For NO₂, the mean difference was calculated after removing the stratospheric contribution (see red line in Fig. 14a; Fig. 14 is without removing stratospheric contribution). The remaining bias was less than $\mathrm{1}\times {\mathrm{10}}^{\mathrm{15}}\mathrm{molecules}{\mathrm{cm}}^{-\mathrm{2}}$ or about 20 % of the tropospheric column ($\mathrm{5}\times {\mathrm{10}}^{\mathrm{15}}\mathrm{molecules}{\mathrm{cm}}^{-\mathrm{2}}$) on average. Higher average NO₂ columns ($>\mathrm{10}\times {\mathrm{10}}^{\mathrm{15}}\mathrm{molecules}{\mathrm{cm}}^{-\mathrm{2}}$) were observed in New York (Manhattan, Bronx, and Queens), which also had three of the four highest biases observed. The highest bias was at Bayonne and appears to be anomalous, demonstrating that bias comparisons across the network are a useful tool for identifying potential problems at specific sites. In this case, Bayonne's large NO₂ bias between SS and DS deserves a closer look by the site operator and may be related to a reference spectrum or calibration issue.

Figure 14 also shows how the NO₂ (Fig. 14b) and HCHO (Fig. 14c) biases between DS and SS change with the solar zenith angle (SZA). One reason to expect changes in the bias with SZA is the larger uncertainty in AMF calculations and larger stray-light contribution at high SZA (Herman et al., 2009). However, no consistent patterns in bias behavior versus SZA were found for either HCHO or NO₂. Pandoras in Aldine and at the University of Houston showed increasing biases with increasing SZA for HCHO, but the bias decreases with SZA in New Haven and Toronto and showed no variations with SZA for the New York Pandora instruments. The NO₂ column biases as a function of SZA showed no notable variation and were mostly constant around a mean value. Additionally, stray light is minimized in Pandora DS retrievals by subtracting the average signal below 290 nm from the spectra (Cede et al., 2024). However, residual stray light could contribute to the large difference between the DS and SS HCHO columns. Plotting the residual stray light as a function of SZA (Fig. S1) showed that it decreased or remained constant (∼0.3 %) with increasing SZA, which suggests that the stray-light contribution to the column might not increase significantly at higher SZA. There is also no evidence for a seasonal dependence of the difference between SS and DS observations (Fig. S2 in the Supplement for the University of Houston), unlike that reported for the total ozone biases between Pandora and Brewer observations (Tzortziou et al., 2012).

Finally, the importance of differences in the pointing azimuth and solar azimuth angle was investigated since 1) the Pandora SS analytical method may be less accurate when the solar azimuth and pointing azimuth are close (Cede et al., 2024) and 2) when the difference is large, the retrievals may be sampling different air masses. Figure S3 in the Supplement shows no systematic biases as a function of the difference between the pointing azimuth and solar azimuth across all 15 sites. This suggests that retrieval uncertainties and vertical sensitivity are most likely responsible for the mean biases between the SS and DS columns. This allows for the possibility of combining the two datasets for higher temporal coverage.

Table 2 Correlations (R²) between hourly surface ozone and the column HCHO from sky scan, direct sun, and combined data at the University of Houston site during TRACER-AQ (September 2021). Results include data with high quality only and with the new filtering method. The number of analyzed hours (n) is given for each case.

3.4 Temporally combining DS and SS data

Section 2.2 discussed the standard schedule for Pandora observations with alternating measurements in the DS and SS modes and the use of UV and open filters for the HCHO and NO₂ observations, respectively. The DS NO₂ retrieval is relatively fast compared to DS HCHO measurements, which are less frequent. For SS measurements, both HCHO and NO₂ are comparable and take much longer than the DS measurements do (Fig. 2c). In order to be combined, these time differences as well as biases diagnosed in the previous section must be addressed.

With the goal of creating a more continuous data record of hourly HCHO columns from Pandora that is suitable for comparison with hourly in situ measurements of surface ozone at EPA sites, Eq. (1) combines data for DS and SS modes by correcting for the mean bias and then accounting for the difference in measurement times. Each column is weighted by the ratio between measurement time in each mode and the total measurement time during the hour for both the DS and SS modes. DS values for the NO₂ column must first include subtraction of the stratospheric NO₂ contribution. SS observations are corrected by adding the mean bias value (Fig. 14a). The summation of the weighted values results in an hourly Pandora HCHO column.

$$\begin{array}{}\text{(1)}& \begin{array}{rl}{\displaystyle }\mathrm{\Delta }{X}_{\text{combined}}=& {\displaystyle }{\sum }_{i=\mathrm{1}}^{\text{1\hspace{0.17em}h}}{X}_{\text{DS}}\left[i\right]\times {\displaystyle \frac{t_{\text{eff}}\left[i\right]}{\text{Tot}}}\\ {\displaystyle }& {\displaystyle }+{\sum }_{j=\mathrm{1}}^{\text{1\hspace{0.17em}h}}\mathit{\left\{}{X}_{\text{SS}}\right[j]+\text{MB}\mathit{\}}\times {\displaystyle \frac{\mathrm{\Delta }t\left[j\right]}{\text{Tot}}},\end{array}\end{array}$$

where t_eff is the effective measurement time for each DS observation (see column 3, “effective duration of measurement” in the Pandora data files); Δt is the estimated measurement time for each SS observation (see Sect. S1.2 for details); Tot is the total integrated measurement time in a 1 h period for both the DS and SS measurements; and X_DS[i] is the direct-sun column and X_SS[j] is the sky-scan column for the ith and jth measurements of each hour, respectively. MB is the mean bias of the HCHO and NO₂ columns between the DS and SS measurements.

Table 2 shows the performance of the relationship between HCHO and hourly surface ozone at the University of Houston during September 2021 for SS, DS, and the combined dataset. The hourly correlation remains similar compared to high-resolution data (5 min), as discussed in Fig. 12; however, there is a smaller number of observations. The combined dataset with the new filtering method increased the temporal coverage of Pandora observations on the hourly data compared to using only one mode and high-quality-flagged data. Applying the new filtering method to the combined data results in the broadest data coverage, with a correlation that is better than or equivalent to the other subsets of data.

4 Summary and conclusions

The growing Pandonia Global Network of Pandora spectrometers provides a ground-based network of remote-sensing observations to both validate satellite trace gas retrievals and understand local air quality. Pandora provides information on the diurnal variability in multiple species (e.g., NO₂ and HCHO) using two independent viewing geometries: direct-sun (DS) and sky-scan (SS) modes. The Pandora quality assurance procedure recommends using high-quality-flagged data, exercising caution with medium-quality-flagged data, and not using low-quality-flagged data. Unfortunately, high-quality data can often be only a small fraction of the total set of observations, except for the DS NO₂ columns.

In response to the overly stringent data flagging procedure, a framework has been described for analyzing Pandora observations by taking advantage of the two independent modes of operation. This includes a new filtering method as an alternative to using quality flags to identify scientifically useful observations. The need to filter data is not new, as Pandora users have selected filtering criteria with some level of subjectivity in years prior to development of the PGN quality flagging procedure (Herman et al., 2009; Judd et al., 2019; Tzortziou et al., 2022).

The method keys in on the range of independent uncertainty values for Pandora data flagged as high quality. Given the large overlap in independent uncertainty for high-, medium-, and low-quality-flagged data, the μ+3σ value of independent uncertainty for the high-quality-flagged data defines a cutoff that preserves almost all high-quality data and includes a large amount of medium- and low-quality-flagged data. A final step in the method is to restore observations with a uncertainty percent of less than 10 %. For a much smaller number of data points, wrms>0.01 (normalized root mean square of the fitting residuals weighted with the independent uncertainty) and SS MHzD>20 km (maximum horizontal distance) should also be filtered out.

Given the independence and contemporaneous nature of DS and SS observation modes, the correlation of adjacent (within 5 min) observations between modes provides evidence for scientifically useful data in the medium- and low-quality-flagged data. Unfiltered observations already exhibit some degree of correlation for data with medium- and low-quality flags, but for filtered data, quality flags appear to have no influence on correlations for various pairing of DS and SS quality flags. This is particularly important, as it means that the method can recover as much as 90 % of data that would have previously been discarded. Further analysis suggests that the standard PGN data flags for atmospheric variability and wrms are too stringent based on the quality of data recovered that exceed PGN criteria for those values.

The strong relationship between column HCHO and surface ozone provided a method for testing the new filtering method. At the University of Houston and four additional sites with nearby ozone monitors, filtered data resulted in a similar or improved relationship between column HCHO and surface ozone, with 14 %–63 % more data than when using high-quality data alone.

Airborne GCAS observations from the TRACER-AQ campaign in Houston provided an independent set of NO₂ and HCHO column observations to compare against filtered Pandora DS and SS observations. The two datasets correlated well, but biases were also observed related to differences in the total column DS observations versus SS observations that have sensitivity limited to the lowest 2–4 km of the atmosphere.

Combining SS and DS data provided better hourly data coverage after correcting for biases between the two modes. Across 15 Pandora sites in the domain of recent NASA field campaigns, the HCHO column biases ranged from 0.8×10¹⁵–$\mathrm{6}\times {\mathrm{10}}^{\mathrm{15}}\mathrm{molecules}{\mathrm{cm}}^{-\mathrm{2}}$ (8 %–60 % of the average column of $\mathrm{1}\times {\mathrm{10}}^{\mathrm{16}}\mathrm{molecules}{\mathrm{cm}}^{-\mathrm{2}}$). The NO₂ tropospheric column biases were less than $\mathrm{1}\times {\mathrm{10}}^{\mathrm{15}}\mathrm{molecules}{\mathrm{cm}}^{-\mathrm{2}}$ (20 % of the average column of $\mathrm{0.5}\times {\mathrm{10}}^{\mathrm{16}}\mathrm{molecules}{\mathrm{cm}}^{-\mathrm{2}}$, except for at the Bayonne, NJ, Pandora). The biases in NO₂ and HCHO between DS and SS observations were examined for trends with solar zenith angle (SZA), viewing azimuth angle, and residual stray light. No systematic behavior in the biases was found.

A more continuous hourly Pandora dataset was constructed for comparison with the hourly surface ozone. This dataset required correcting the site-specific biases in DS and SS observations as well as weighting the observations based on their different measurement durations. The combined hourly dataset showed similar or improved correlations with surface ozone, with a 30 %–40 % increase in hourly data coverage.

The proposed filtering method and improved representation of hourly average conditions by combining DS and SS data offer the community useful strategies for using Pandora observations to their fullest potential for validating geostationary satellite observations and informing local air quality.