Evaluation and optimization of ICOS atmosphere station data as part of the labeling process

The Integrated Carbon Observation System (ICOS) is a pan-European research infrastructure which provides harmonized and high-precision scientific data on the carbon cycle and the greenhouse gas budget. All stations have to undergo a rigorous assessment before being labeled, i.e., receiving approval to join the network. In this paper, we present the labeling process for the ICOS atmosphere network through the 23 stations that were labeled between November 2017 and November 2019. We describe the labeling steps, as well as the quality controls, used to verify that the ICOS data (CO₂, CH₄, CO and meteorological measurements) attain the expected quality level defined within ICOS. To ensure the quality of the greenhouse gas data, three to four calibration gases and two target gases are measured: one target two to three times a day, the other gases twice a month. The data are verified on a weekly basis, and tests on the station sampling lines are performed twice a year. From these high-quality data, we conclude that regular calibrations of the CO₂, CH₄ and CO analyzers used here (twice a month) are important in particular for carbon monoxide (CO) due to the analyzer's variability and that reducing the number of calibration injections (from four to three) in a calibration sequence is possible, saving gas and extending the calibration gas lifespan. We also show that currently, the on-site water vapor correction test does not deliver quantitative results possibly due to environmental factors. Thus the use of a drying system is strongly recommended. Finally, the mandatory regular intake line tests are shown to be useful in detecting artifacts and leaks, as shown here via three different examples at the stations.

1 Introduction

Precise greenhouse gas monitoring began in 1957 at the South Pole and in 1958 at the Mauna Loa observatory (Keeling, 1960; Brown and Keeling, 1965; Pales and Keeling, 1965). Over these 60 years of data, CO₂ levels have risen by about 100 ppm (parts per million) in the atmosphere. CO₂ and other greenhouses gases are a major source of climate forcing (IPCC, 2014), and following Mauna Loa measurements, several monitoring networks (Prinn et al., 2018; Andrews et al., 2014; Fang et al., 2014; Ramonet et al., 2010) and coordinating programs (WMO, 2014) have been developed over time to monitor the increasing mixing ratios in different parts of the world and quantify the relative roles of the biospheric oceanic fluxes and anthropogenic emissions. Initially, the goal was to measure greenhouse gases at background stations to get data representative of large scales. Later, more and more regional stations and networks were established in order to get more information on regional to local fluxes. Indeed, this is especially relevant in the context of monitoring and verifying the international climate agreements (Bergamaschi et al., 2018).

The Integrated Carbon Observation System (ICOS) is a pan-European research infrastructure (https://www.icos-ri.eu, last access: 28 December 2020) which provides highly compatible, harmonized and high-precision scientific data on the carbon cycle and greenhouse gas budget. It consists of three monitoring networks: atmospheric observations, flux measurements within and above ecosystems, and measurements of CO₂ partial pressure in seawater. Its implementation included a preparatory phase (2008–2013; EU FP7 project reference 211574) and a demonstration experiment until the end of 2015 when ICOS officially started as a legal entity. ICOS was first designed to serve as a backbone network to monitor fluxes away from main anthropogenic sources. The concentration gradients between European sites are typically of only a few parts per million on seasonal timescales. These small atmospheric signals combined with atmospheric transport models are used to deduce surface fluxes. For these atmospheric inversions, a high-precision and integrated network is mandatory. As a precise example, Ramonet et al. (2020) show that a strong drought in Europe like the one seen in summer 2018 produces an atmospheric signal of only 1 to 2 ppm.

During the preparatory phase and the demonstration experiment, standard operating procedures for testing the instruments and measuring air in the most precise and unbiased way were defined. Data management plans were created and required IT infrastructure such as databases, and the quality control software tools were developed. In addition to the monitoring networks and the head office which is the organizational hub of the entire ICOS research infrastructure, central facilities have been built to support the production of high-quality data. This ensures traceability, quality assurance/quality control (QA/QC), instrument testing, data handling and network support with the aim of standardizing operations and measurement protocols. The central facilities are grouped as follows: the Flask and Calibration Laboratory (CAL-FCL, Jena, Germany) for greenhouse gas flask and cylinder calibration (linking the ICOS data to the WMO calibration scales), the Central Radiocarbon Laboratory (CAL-CRL, Heidelberg, Germany) for radiocarbon analysis, and three thematic centers for atmosphere, ecosystem and oceans.

The thematic centers are responsible for data processing, instrument testing and developing protocols in collaboration with station principal investigators (PIs). Regular monitoring station assembly (MSA) meetings facilitate discussions of technical and scientific matters.

The Atmosphere Thematic Center (ATC; https://icos-atc.lsce.ipsl.fr/, last access: 28 December 2020) is divided into three components: the metrology laboratory (MLab) responsible for instrument evaluation, protocol definition and PI support, the data unit responsible for data processing, code development and graphical tools for PIs, both located in Gif-sur-Yvette, France, and finally the MobileLab in Helsinki, Finland, tasked with the audit of the stations during and after the labeling process.

One very important task for the ATC is ensuring that the stations reach the quality objectives defined within ICOS, which are based on the compatibility goals of the WMO (WMO, 2018) and detailed in the ICOS atmosphere station specifications (ICOS RI, 2020a). To do so, a so-called “labeling process” has been developed to firstly assess the relevance of a new measurement site, as well as the adequacy of the human and logistical resources available with the ICOS requirements. Afterwards, an evaluation of the first months of measurement. is carried out, verifying compliance with the ICOS protocols. The Carbon Portal (https://www.icos-cp.eu/, last access: 28 December 2020, Lund, Sweden), which is responsible for the storage and dissemination of data and elaborated products (such as inversion results or emission maps), is associated with the labeling process via PID/DOI attribution for the data and the provision of a web interface to gather important information needed for the labeling. The labeling process is very useful for new stations coming into the network to ensure proper setting and good measurement practice and, in the end, to be able to reach the precision and stability requirements of ICOS. For the end user, the labeling process guarantees high-quality observations with full metadata descriptions and traceable data processing.

In this paper, we present the labeling process for the ICOS atmosphere network and illustrate it through the 23 stations that have been labeled between November 2017 (first stations labeled) and November 2019. First, we describe the protocol that a station must follow to be labeled. Then, we detail the different metrics and elements that are analyzed during the labeling process to validate the quality level. Afterwards, we present the 23 labeled atmosphere stations, and in a third part, we discuss results and findings from these stations as seen during the labeling process.

2 Protocol and metrics of the labeling process

To be labeled, an ICOS atmosphere station has to follow the guidelines and requirements defined in the ICOS atmosphere station specifications (ICOS RI, 2020a; hereafter referred as the AS specifications) and the labeling document (ATC-GN-LA-PR-1.0_Step2info.pdf; available on the ATC website under section Documents, Public documents, Labelling or at https://box.lsce.ipsl.fr/index.php/s/uvnKhrEinB2Adw9?path=/Labelling#pdfviewer, last access: 28 December 2020). The AS specifications are discussed and updated if necessary every 6 months at the Monitoring Station Assembly meetings that include the PIs of all the ICOS stations and representatives of the central facilities. The goal of these specifications is to allow each site to reach the performances required by the ICOS atmosphere data quality objectives, which principally adhere to the WMO guidelines (WMO, 2018) for greenhouse gas observations but are elaborated on more in the AS specifications and presented in Sect. 2.4 below.

The labeling process of atmosphere stations has been defined as a three-step process: Step 1: evaluation of the station location and infrastructure; Step 2: station performances; and Step 3: official and formal ICOS data labeling by the ICOS general assembly (composed of representatives of the member and observer countries of ICOS and meeting twice a year).

To pass Step 1, stations must submit information about the infrastructure of their site, its location and its proximity to anthropogenic sources like cities or main roads. This is done through the Carbon Portal interface (https://meta.icos-cp.eu/labeling, last access: 28 December 2020). The ATC uses these data to compile a report and issue recommendations to the ICOS head office which will then approve or reject the application. Usually, if there are some problematic points, the ATC first contacts the PI to see if improvements can be done to meet the requirements or ask for additional documents. ICOS atmosphere is mainly focused on tall tower sites measuring regional signals but accepts a limited number of a high-altitude and coastal sites (ICOS RI, 2020a).

Once Step 1 is approved, the station can be built, equipped and set up to fulfill the AS specifications. Once the near-real-time dataflow to the ATC database is established (Hazan et al., 2016), stations can apply for Step 2. The time lapse between Step 1 and Step 2 can vary greatly depending on the site. Indeed, in the case of already existing stations, they are entering ICOS with running instruments and historical datasets already and need only small changes such as getting the calibration cylinders from the CAL-FCL and modifying some procedures to have their data processed into the database before beginning Step 2. Others will have the whole construction of the tall tower and shelter and the installation of lines to achieve first.

During Step 2, a phase of measurement optimization begins: the initial test period. This is done in close collaboration between the station PI and the ATC through routine sessions of data evaluation (usually every month). This period typically lasts 4 to 6 months to gather data to evaluate their quality. The period may be prolonged if needed. If data meeting the AS specifications are available prior to the application of Step 2, the initial test period can be shorter.

During the initial test period, the requirements detailed hereafter are asked for from the station PI in order to be able to analyze all the data in a uniform way for all sites.

2.1 General requirements

The ICOS atmosphere network aims to provide high-precision measurements of greenhouse gases, and the priority is CO₂ and CH₄ which represent the main anthropogenic greenhouse gases (GHGs). In situ measurements of N₂O, the third most important contributor to the additional radiative forcing, were not required in the initial phase of ICOS due to the difficulty in finding at that time reliable instruments able to provide the expected precision (Lebegue et al., 2016). This gas is expected to become mandatory in the near future of ICOS. Flask sampling is required at Class 1 stations for quality control of in situ measurements and to provide additional trace gases measurements like N₂O, H₂ and CO₂ isotopes (Levin et al., 2020). Other parameters are required in order to support the interpretation of the GHG variabilities, like CO as a tracer of combustions, and meteorological parameters are required to characterize the local winds, vertical stability along tall towers and weather conditions (pressure, temperature, relative humidity). The eddy covariance fluxes have been selected as well with the idea of characterizing the local surface fluxes from either biogenic and/or anthropogenic activities and of monitoring possible long-term changes around the ICOS sites. So far this parameter is not required for the labeling process due to logistical difficulties in performing such measurements at several atmosphere sites.

Table 1 ICOS atmosphere station parameters (from the ICOS atmosphere station specifications; ICOS RI, 2020a).

^∗ Not yet required for the labeling; see Sect. 2.1.

At the beginning of the initial test period, a station must provide at minimum continuous in situ greenhouse gas data to the database on a daily basis, and by the end, meteorological parameters (wind speed, wind direction, atmospheric temperature, relative humidity and pressure) and additional diagnostic data (room temperature, instrument and flushing pump flow rates). Table 1 shows the list of all mandatory parameters that should be provided by the stations depending on the station class. Furthermore, it also provides a list of recommended parameters. A Class 1 station will provide more parameters than Class 2, but both stations must meet the same level of data quality. Presently, the MSA has decided that labeling should not be contingent upon two Class 1 parameters: the boundary layer height and the measure of greenhouse gases and δ¹⁴CO₂ values from flask sampling. Indeed, for these two parameters, the technologies, hardware and software are still in development or in need of improvement (Feist et al., 2015; Levin et al., 2020; Poltera et al., 2017). As soon as the MSA decides to approve a technology, it should, however, be added to the station as soon as possible. Indeed, flask sampling is an additional quality control tool, as well as a way to sample species that cannot yet be measured continuously, while boundary layer heights would help improve models. For all the stations presented here, we focused on CO₂ and CH₄ continuous measurements for all sites and on CO measurements for Class 1 sites. The other species measured by some instruments such as N₂O were not assessed as they were not mandatory.

The instruments providing the data must be ICOS compliant as defined in the AS specifications. The list of accepted analyzers is regularly updated to keep up with new technologies that are continuously tested at the ATC. In the case of the GHG analyzers, all instruments operated in the network are tested at the ICOS ATC MLab following the procedure described in Yver Kwok et al. (2015). Their intrinsic performances are evaluated, as well as their sensitivities to atmospheric pressure, instrument inlet pressure, ambient temperature, other species and water vapor. In the case of water vapor, a specific correction is determined for each instrument. A test report is produced systematically to provide the specific analyzer status with regards to its compliance to the specifications. It is important to characterize the instrument performances under well-defined and controlled conditions at the ATC MLab since they will be used as a reference for the evaluation of field performances. The initial test of the analyzers also allows us to verify if the performances of the instruments are consistent with the specifications provided by the manufacturer. Over the past years, few instruments were sent back to the manufacturer due to poor performance. Other parts, such as pressure regulators, gas distribution systems and types of cylinders, are also defined in the AS specifications.

2.2 Greenhouse gas calibration requirement

For consistency and efficiency purposes over the network, a common calibration strategy has to be followed. During the initial test period, the general philosophy is to carry out frequent calibrations and quality control measurements with the aim of determining their optimal frequencies and durations. Presently, the calibration strategy for the initial period is as follow: three to four cylinders (filled with natural dry air, for which values have been assigned at the CAL-FCL and are traceable to the WMO scales; https://www.icos-cal.eu/static/images/docs/ICOS-FCL_QC-Report_2017_v1.3.pdf, last access: 28 December 2020) are each measured four times for 30 min one after the other every 15 d, leading to a total of 6 to 8 h of calibration measurements. Depending on the stability of one calibration to the other, ATC will recommend if the frequency can be reduced, but in any case, at least one calibration sequence per month is required. Cylinder numbers and positions in the sampling system at which they are connected, as well as the sequence of injections, have to be entered into the ATC configuration software. An automatic quality control of raw measurements (Hazan et al., 2016) is performed on the calibration data based on a check of instrumental parameters, such as temperature and pressure of the analyzer cavity, to ensure the instrument is working properly. For example, the typical accepted range for cavity ring-down spectroscopy (CRDS) cavity pressure is 139.8 to 140.2 Torr (186.38 to 186.92 hPa). Then a flushing period, whose duration is configured via the ATC configuration tool, is automatically filtered out. From the validated measurements, 1 min averages are calculated and then the injection means for each of the calibration gases. The different levels of data aggregation (minute, injections, cycles) are automatically checked by comparing them to predefined standard deviation threshold values (see Table 2 and Hazan et al., 2016, for more details). Moreover, the water vapor content of the calibration gas is indirectly checked with a threshold on the difference between raw data and data corrected from water vapor effects. These thresholds are defined considering the instrument performances assessed by the ATC MLab and the station sampling setup and can be modified during the initial test period. For example, for a configuration without a drier, the typical humidity threshold for CO₂ will be 0.01 ppm, but with a Nafion drier, as the dry cylinder gas will be humidified by the drying system, the threshold can be raised to 0.5 ppm. The effect of long dry measure on wet air is discussed in the next section. Finally, for calibration cylinders and any others cylinders, it is advised to set the pressure on the regulators so that the pressure at the instrument inlet is slightly above the atmospheric pressure, thereby limiting a possible leakage contamination. However, the pressure should not be set at a value that is too high or too low in order to avoid a significant pressure jump while passing from cylinder to ambient air measurement (which is usually done at an instrument pressure inlet below the atmospheric pressure). Indeed, laboratory tests have shown that transitory biases appear during step pressure change at the instrument inlet: the higher the step, the longer the return to equilibrium. In consequence, a large inlet pressure difference between ambient air and cylinder gas may result in an artifact which will not have time to disappear over the time we are measuring the samples. During the initial test of the instrument, an acceptable range for the pressure is determined to help the PI set the regulators.

Table 2 ATC MLab typical thresholds for calibration quality control (in standard deviation, SD). The minute data SD takes into consideration the SD of each minute of the injection. The injection average SD takes into consideration the SD of all the minutes of one injection, and the cycle average SD takes into consideration the SD of the two to three injections.

2.3 Quality control requirements

2.3.1 Target tank measurements

An important element of our quality control strategy for greenhouse gas measurements is to regularly measure a target gas of known concentration. On a daily basis, we analyze air sampled from a short-term target (every 7–10 h during the initial test), and after each calibration, we do the same with air from a long-term target, as shown in Fig. 1. This ensures continuity in the quality control as the long-term target should last more than 10 years. Therefore, its chosen mixing ratio is relatively high (450 ppm for CO₂ for background sites) compared to actual ambient air values in order to follow the increasing trend. It is recommended to send the long-term target, as well as the calibration set, for recalibration approximately every 3 years to CAL-FCL to investigate and take into account any possible composition changes in the gases, especially for CO.

Figure 1 Target gas injections for 1 month for CO₂ (ppm or µmol mol⁻¹) shown as the difference of calculated vs. assigned mixing ratios. The short-term target is plotted in green, while long-term data are in brown. The calibration dates are shown by the light orange open circles. Cylinder number (D******), mean values (±X), point-to-point variability (Ptp) and difference to the assigned value (Diff) are displayed above the figure.

Figure 1 shows the difference between the assigned and measured values over 1 month for the short-term (in green) and the long-term (in brown) targets. The instrument calibration dates are indicated at the bottom of the plot by the open orange circles. In this example, we can notice that after a calibration, the short-term target is significantly different from the other injections. Indeed, after about 6 h of dry air injection, the cavity is extremely dry compared to the usual injections after wet ambient air. This effect, seen only in cavity ring-down spectroscopy (CRDS) analyzers (which, however, make up the majority of the CO₂ ∕ CH₄ ∕ CO instruments in ICOS atmosphere), is thought to be due to residual water on the pressure sensor in the instrument cavity (Reum et al., 2019). The extent of the effect is dependent on the analyzer, and thus this effect is important to assess as it allows us to improve the bias estimate based on the target. Indeed, the mole fraction assigned by the CAL-FCL, as well as the target measured directly at the end of the calibration sequence in the field, is given in extremely dry conditions. While the instrument variability should be assessed with the short-term target measured regularly within ambient air, the measurement bias should be assessed only with the long-term target and the short-term target in extremely dry conditions, i.e., the target measured directly after calibration for an analyzer not equipped with an ambient air dryer. Depending on the instrument, this potential bias is more or less pronounced but does not exceed 0.05 ppm for CO₂ and 0.4 ppb for CH₄. It is part of the uncertainty of the water vapor correction estimated during the initial test at the Mlab. Moreover, one of the final tests is to compare the tested instrument with the Mlab reference instrument whose samples (air and cylinder gas) are all dried. This allows us to evaluate the weight of this bias.

2.3.2 Manual data quality control

All the data processed by ICOS ATC go through an automatic quality control (QC) based on various criteria (Hazan et al., 2016). However, as a second and final quality control step, the PI of the station has to review and validate the data on a regular basis using the logbook information from the station (e.g., contamination due to maintenance on site). No data is flagged invalid without an objective reason.

To harmonize the quality control, the ATC provides dedicated software tools and organizes mandatory training that must be attended before beginning the operation at the station. The data validation is done with a software developed at the ATC directly in the server. On a daily basis, the station PI checks the ATC data products generated daily on the ATC website (https://icos-atc.lsce.ipsl.fr/dp, last access: 28 December 2020) as an early detection of any issue related to the analyzers, sampling lines, data transmission or processing. On a weekly basis, raw greenhouse gas data have to be checked and (in)validated using ATC QC software via a flagging scheme. Raw data are reviewed day by day. For valid data, we can choose additional information such as “quality assurance operation” or “non-background conditions”, but this is not mandatory. Data have to be invalidated only for an objective reason which has to be chosen from a list to be able to carry on with the QC. The reasons can be (non-exhaustive list): “calibration Issue”, “flushing period”, “maintenance with contamination”, “inlet leakage”, etc. On a monthly basis, the hourly average of greenhouse gas and meteorological data must also be verified.

During the initial test period, regular online meetings take place with the PIs and the ATC to review the data and assist the PIs. Their purpose are as follows:

• to exchange expertise between ATC and the PIs, for example, on how to QC local events (spikes) and how to interpret the data products; for example, for the spikes, a spike detection algorithm has been developed and is automatically applied to the data (El Yazidi et al., 2018);

• to make sure the data are regularly controlled;

• to benefit from local knowledge to explain patterns detected in the time series.

2.3.3 Intake line and water vapor correction tests

ICOS atmosphere station specifications also require the station PIs to perform tests on the intake lines to investigate potential leaks and artifacts. These tests are extremely important because the target measurements, as described previously, do not make it possible to check for leaks in all parts of the air sampling lines. Consequently, the PIs perform dedicated tests every 6 months inside the measurement shelter on the different parts of the sampling system (including filters, valves, etc.) and every year for the entire sampling lines running outside. Ideally, for this last test, a test gas should be injected through each sampling line on the outside structure (tower) in order to test the leakage and the inner surface artifacts. The test through the whole sampling line also allows us to calculate the sample residence time. However, for convenience, we propose replacing this test by a leak test of these outside sampling lines for lines younger than 10 years as for these lines, we expect that contamination like bacterial buildup that could cause biases will remain rare and the leak test will suffice to identify cracks.

Figure 2 Station schematics with injection points in blue for the shelter test. Example of Trainou tower, France. Different parts include valves, filters, pressure gauges and pumps. One element of each type is given in the legend for clarity.

Dry air from a cylinder is measured first as close to the instrument as possible but without disconnecting the line to the instrument, and this measure is considered the reference. Then, for the shelter test, the same gas is injected upstream of all the sampling parts in the shelter and at the outside sampling line connection point in the shelter, usually a filter, as shown in Fig. 2 which shows a typical setup for an ICOS station. The two injection points are shown in the figure by the blue circles. During this test, it is important to adjust the applied pressure of the test gas cylinder in order to reproduce the same pressure conditions as when ambient air is sampled. To avoid emptying the cylinder, the flushing pumps are off during the test upstream of all the sampling parts. If no significant bias is observed, we can consider that there are no leaks and that no component is causing an artifact. Significant means higher than or very close to the WMO compatibility goals taking into account the water vapor uncertainty and the bias of the instrument determined at the ATC MLab. In the case of the entire lines, the test can be done either as per the shelter test but with the gas injected through the whole line (usually by connecting the tops of the spare line and of the intake line and injecting the gas at the bottom of the spare line) or by closing the top of the intake line, creating a vacuum and then checking if this vacuum is holding over time. This last test only informs us about the presence of a leak but is easier to perform. The test through the whole line is recommended for lines older than 10 years. In addition to the regular frequency, these leak tests must be carried out after any modification of the sampling lines.

Another source of uncertainty and possible biases is the water vapor effect on the measured gases. Tests at the MLab have shown that these effects can change over time and in a different way for each species and are visible to all instruments and technologies tested up to now (CRDS, Fourier transform infrared spectroscopy, FTIR, off-axis integrated cavity output spectroscopy, ICOS-OA). If the instrument has not been tested at the MLab within the last year and no drying system is used, the PI needs to perform a new water vapor assessment to evaluate if the water vapor correction has indeed changed over time. This test consists of injecting with a syringe at least three times a small droplet (0.2 mL) directly in the inlet of the analyzer or through a filter when the analyzer does not have an internal filter to humidify a dry gas from a cylinder (with ambient air mixing ratios) and letting it dry to obtain the profile of the trace gas vs. the amount of water vapor (Rella et al., 2013).

2.4 Metrics for the station labeling

The decision taken by ICOS general assembly to label a station or not has to be based on objective criteria known in advance and common for all sites. During the initial test period, different metrics are thoroughly investigated to make sure the measurements meet the ICOS specifications and quality standards required. We detail them below and illustrate each of them with a figure from one labeling period or one site that we used in the reports. As a result, the figures do not show all stations or the most recent period but are here to illustrate how the report content looks and what information can be derived. Most of the figures are automatically generated regularly and are available on the ATC website (https://icos-atc.lsce.ipsl.fr/dp), but some are specifically produced for the labeling reports.

2.4.1 Percentage of data validated by station PIs

Quality control by the PIs is paramount to ensure the high quality of the ICOS dataset. When preparing the reports, we make sure that the quality control is done as detailed in the AS specifications, i.e., weekly for the greenhouse gas air raw data and monthly to every 2 months for the hourly means or injections of greenhouse gas (air and quality control gases) and meteorological data.

Figure 3 Example of data validation for six stations: CMN (Monte Cimone, Italy), IPR (Ispra, Italy), LIN (Lindenberg, Germany), PAL (Pallas, Finland), TOH (Torfhaus, Germany) and TRN (Trainou, France). Dark colored data are data controlled by the software only. Light colored data are controlled by the PI. Green is valid and red invalid. On the left, the first column shows the station acronym, the second column the ICOS ID of the instrument and the third the sampling height.

Figure 3 shows the status of hourly data validation at a given time for six stations. The hourly means are computed automatically using minute means which are themselves computed using raw data. If there is at least one valid value from the raw data within a given minute, the corresponding minute mean is considered valid. Similarly, if there is at least one valid minute mean to compute an hourly mean, the hourly mean is considered valid. There is no automatic quality control criterion applied to the hourly means; the criteria are only applied to the raw data. Valid data are shown in green and invalid data in red. Dark colors indicate automatic (by the ATC software) validation prior to the manual data inspection by the PI (light colors). The dark red color will usually indicate flushing periods or instrument failure (hence no data when the database expects some) that are automatically flagged. For each station, the analyzers are identified by their unique ICOS ID attributed by the ATC. This is the number shown in the second column in the graph. The third column shows the sampling heights. In this plot, however, we will mostly focus on the amount of manual validation to ensure that the data have been indeed controlled by the PIs. All interventions of the PIs to flag data are done through the ATC software and are recorded for traceability. When raw data are rejected, the PI has to select a reason for the problem within a predefined list of 11 issues (such as flushing period, instrument failure, maintenance, etc.). The PIs first validate the data on the raw level, then every 1 to 2 months on the hourly level. The second validation of the hourly dataset aims to verify the longer-term consistency of the time series. Every time a data flagging is performed either on the raw or hourly data, a reprocessing is automatically applied to the other aggregated levels (raw, minute, hour) for consistency.

2.4.2 Percentage of air measurement vs. calibrations or target and flushing time

When switching from one sample to the other, there is some flushing period (defined by the PI in agreement with the ATC) that is automatically removed from the valid data. Figure 4 allows us to evaluate whether the time spent measuring “invalid” air is not too high compared to the time measuring “valid” air. It is mostly important for tall towers that switch between many levels and may end up spending too much time flushing. In the figure, when the percentage does not reach 100 %, it means that the station has not yet provided data for the whole year. For example at Lindenberg (LIN), the analyzer for CO₂ and CH₄ was at least running since October 2017, but the CO ∕ N₂O analyzer was installed only in August 2018, thus showing only 2 months of data.

Figure 4 Data distribution between ambient air and target and calibration gases for the same six stations as Fig. 3 for CO₂, CH₄ and CO. Calibration is in red, target in blue and air in green/gray. Gray and darker colors are invalid data. Less than 100 % data availability means that the instrument was installed less than 12 months ago. The first line shows the station acronym and the second line the instrument ICOS ID. In the case of LIN, two instruments are used to measure CO₂, CH₄ and CO.

2.4.3 Optimized stabilization time to flush the sampling system

Related to the previous metric, to ensure the optimal time spent measuring ambient air and also to save calibration and target gases, the stabilization time needed for each gas tank is evaluated and optimized where necessary. If we observe that from one gas to the other, the stabilization period is significantly different, it can be a sign of a leak or problem in the setup that will be reported to the PI. The sample is considered stable when the difference between a given minute-averaged data point in the gas injection and the last injection point (after 30 min of measurement) is lower than 0.015 ppm for CO₂, 0.25 ppb for CH₄ and 1 ppb for CO. These thresholds are determined considering the WMO recommendations (WMO, 2018) and expectations from the instrument performances (see Yver Kwok et al., 2015).

Figure 5 Difference between the last injection-minute-averaged data point and the rest of the injection for cylinder samples for Monte Cimone station (CMN) for instrument 590. All the injections over the last 6 months are averaged. Short-term target is in dark blue, long-term target in light blue and calibration in red. Dashed red lines show the thresholds. Vertical lines on each point show the minute standard deviation.

In Fig. 5, we show the average differences for all tanks over a period of 6 months at Monte Cimone (CMN). During that time, the short-term target has been injected 335 times and the long-term target 15 times, and there are 240 injections for the calibrations (15 calibration sequences with four cylinders and all cycles taken into account, here four). For the calibrations, we show the average difference of all the calibration cylinders and cycles. Here, we see that the short-term target and the calibration air stabilize faster than the long-term target, about 6 vs. 18 min for CO₂. This can be a sign of a leak or more likely be due to the fact that the long-term target is measured only once every 2 weeks, and as a consequence, the pressure regulator installed on the tank is flushed less often and requires a longer flushing time to reduce possible cumulative artifacts related to the pressure regulator's inner parts in static mode.

2.4.4 Instrumental drift and optimization

The instrument response may drift over time, which is usually the case for some ICOS-compliant CRDS analyzers of CH₄ (Yver Kwok et al., 2015). This drift is corrected by the data processing using regular calibration sequences. Depending on the drift rate and its linearity, the frequency of the calibration may have to be adapted. Following the observed time evolution of the calibration gases allows us to track if one gas is behaving significantly differently than the others, which could be caused by a drift in the cylinder or a leak in the setup (Fig. 6). For some instruments, such as the ones measuring CO and N₂O, we also observe short-term drifts on the scale of hours to days. In this case, we use a “short-term working standard” (STWS or reference) to correct for such a drift. This standard is calibrated by the twice-monthly calibrations. By looking at the last calibration injections, we can assess the number of cycles needed to reach the required stability and optimize this calibration sequence (Fig. 7). The first cycle is always rejected by default as the samples are not yet well dried. The stability is estimated for the two to three following injections. In the examples presented in Fig. 7, we see that after the first rejected injection, the spread between the other cycles is below 0.01 ppm for CO₂. This shows that reducing the calibration sequence from four cycles to three is possible to save gas without reducing quality.

Figure 6 Evolution of the analyzer's raw output when measuring different calibration gases with respect to the assigned values over a year at Trainou station (TRN) for instrument 472. Each calibration cylinder is shown with a different color. Assigned values are indicated on the right for each cylinder (D******).

Figure 7 Average of each cycle injection for the last calibrations over 3 months at Svartberget station (SVB) for instrument 464. Green dots are data used for the calibration correction, and red is rejected for stabilization. The number of calibrations is shown on the top right. Assigned values are on the top left. Cylinder number (D******) is shown at the top of each panel.

2.4.5 Temperature dependence of the instruments

A few of the instruments tested at the ATC MLab have shown a significant sensitivity of the GHG measurements to temperature. On site, the temperature variation is supposed to be small, but in case of problems with the air conditioning, we can use the target gas measurement to evaluate the impact of the temperature changes on the measurements, as seen in Fig. 8. Currently, there is no correction derived from the MLab tests applied in the ICOS data processing. For instruments showing a significant variability due to the temperature or other parameters (i.e., leading to the WMO goals within the observed range of temperature being exceeded), it is recommended to use a short-term working standard in order to correct the short-term variability induced by these sensitivities. The target tank cannot be used for this or any correction as it is a quality control gas which is taken into account for data uncertainty assessment, contrary to the short-term working standard.

Figure 8 Temperature influence on the measurements at Puy de Dôme station (PUY) for instrument 473 and the target cylinder D337581. On the top: greenhouse gas and instrument temperature (Tdas) measurements against time. On the bottom: greenhouse gas measurements against instrument temperature. In most of the cases, no dependencies are seen.

2.4.6 Meteorological measurements

Meteorological parameters are mandatory as they are used to analyze the atmospheric signals measured at the station location and associate them with regional or large-scale processes. During the initial test period, the ATC checks that the sensors are compliant with the list from the AS specifications and that the data are transmitted correctly to the ATC data unit database for all mandatory levels (see Fig. 9). The ATC checks the data availability and consistency. The ATC performs simple filtering on the raw data based on valid ranges (min/max values) for the five mandatory species, which are pressure, temperature, relative humidity, wind speed and wind direction. Except for relative humidity, the data are also marked as invalid if the measurement is constant for more than X min in a row. X is set to 10 for the wind variables and to 60 for the other species. This criterion is used to cope with blocked sensors. ATC is also working on a model vs. measurement comparison with the European Center for Medium-range Weather Forecasts (ECMWF) data to highlight potential drifts or outliers. In terms of instrumentation, ATC is working on instating a 2-year recalibration of the humidity sensors that are the ones drifting the fastest over time. If meteorological data are sent to the database at the same time as the greenhouse gas data and not at the end of the test period, they can be used to understand the variability in the greenhouse gas data.

Figure 9 Meteorological parameters for 1 month at Hyltemossa station (HTM). From top to bottom: atmospheric pressure, relative humidity, atmospheric temperature, wind direction and wind speed. The data at the different levels are plotted with different colors.

2.4.7 Diagnostic parameters

For the diagnostic parameters (room temperature, instrument and flushing pump flow rates), in a similar way to the meteorological parameters, the ATC checks that they are available and consistent. If they are present over the whole test period, they can be used to monitor that the room temperature is well controlled and that the instrument and flushing pump flow rates are as stable as expected. Higher flow rates can indicate leaks, whereas decreases over time will most probably indicate that filters are getting clogged and need maintenance or that there is an obstruction in the sampling line (see Fig. 10 bottom panel). The measure of the flow rates is also important to estimate the time delay between the air sampling at the top of the sampling lines and the measurement in the analyzer. This delay can be significant for the highest level of a tall tower and needs to be known to correctly attribute a timestamp to the measured air. Finally, the instrument flow rates can be used to estimate the lifetime of the gas cylinders.

Figure 10 Diagnostic parameters for 1 year at Pallas station (PAL). From top to bottom: instrument flow rate, room temperature and sampling line flushing flow rate.

2.4.8 Time series and associated uncertainties

For most of the stations that enter labeling Step 2, data have already been collected before the initial test period which allows an analysis of the previous year (Fig. 11) and previous month (not shown) and the ability to plot a wind rose (not shown) allocating the mixing ratios to the wind direction and intensity for the whole year and by season. This figure is of interest to evaluate the influence of different sources that can surround the site at a more or less large scale. On the yearly figure, ATC looks for patterns in the target gases (biases, drifts), data gaps and outliers, whereas the ambient air signals are much more visible in the figures of shorter periods. To allocate mixing ratios to wind sectors or to compare two instruments measuring the same species, all instruments and sensors have to have access to a time server and update their clock regularly.

Figure 11 Hourly averaged greenhouse gas measurements for 1 year at Torfhaus station (TOH) for instrument 457. The different levels and targets are plotted with different colors. Ambient air is plotted on the left and target measurements on the right. Calibrations are shown with open orange circles. Invalid data are shown at the bottom of each plot. Cylinder number (D******), mean values (±X), point-to-point variability (Ptp) and difference to the assigned value (Diff) are displayed above the target gas plots. Measured GHGs are shown in the different panels from top to bottom.

Figure 12 Last year of greenhouse gas measurements along with estimated uncertainties at Jungfraujoch station (JFJ) for instrument 225. Continuous measurement repeatability (CMR) and long-term repeatability (LTR) are calculated as in Yver Kwok et al. (2015). The short-term target bias is calculated as the difference between the hourly average of the short-term target injections and the value assigned by the FCL-CAL. In the top panel, the ambient air data are compared to the MHD (Mace Head, Ireland) marine smooth curve, derived from atmospheric measurements made at Mace Head, a historical European background site. The smooth curve is calculated using NOAA's CCGCRV function (https://www.esrl.noaa.gov/gmd/ccgg/mbl/crvfit/crvfit.html, last access: 28 December 2020, Thoning et al., 1989).

With the measurement of the target gases, uncertainties comparable to the ones estimated in the MLab during the initial test are calculated, as well as the bias to the CAL-FCL assigned values, as shown in Fig. 12. These values are compared to the MLab values and, if very different, can be a hint that the setup has introduced a problem that needs to be identified and solved. Details on the calculations of these uncertainties are found in Yver Kwok et al. (2015). In brief, CMR stands for continuous measurement repeatability and is calculated here using the monthly average of the standard deviations of short-term target raw data over 1 min intervals. Long-term repeatability (LTR) is the standard deviation of the averaged short-term target measurement intervals over 3 d. Here, we can see that before November 2017, the target variability was high leading to high and variable LTR and bias. After November 2017 and a change of parts in the sampling setup that we discuss in Sect. 4.6, the LTR and bias show a significant improvement.

3 Presentation of the 23 labeled stations

The 23 labeled stations described here passed Step 1 between 2016 and 2019 (see Table 3). For most of them, this was a straightforward step. For four of them (Ispra, IPR, Observatoire de l'Atmosphère du Maïdo, RUN, Lutjewad, LUT, and Karlsruhe, KIT), additional documents and preliminary studies were requested mainly to address potential local contaminations. After Step 1 approval, they entered Step 2 up to 2.5 years later depending on the existing infrastructure and instrumentation. At the end of the initial test period, a scientific report summing up the setup done during this period and the resulting data was sent to the PI, and the station was proposed for labeling. The stations were then approved by the ICOS general assembly.

Table 3 Information about the 23 labeled stations: site name, three-letter acronym, coordinates, sampling level heights, station surroundings, class, date of the first ICOS compliant data and date of labeling. The abbreviations a.s.l. and a.g.l. signify above sea level and above ground level, respectively.

In November 2017, the first four ICOS atmosphere stations were labeled following this procedure. In May 2018, the next seven were approved. In November 2018, another six were labeled. In May and November 2019, two and four, respectively, were approved. Seven stations are located in Germany, three in mainland France, three in Sweden, three in Finland, two in Italy (with one operated by the Joint Research Centre of the European Commission), one in the Netherlands, one in Norway, one in Switzerland, one in Czech Republic and one on La Réunion island, operated jointly by the Belgian and French national networks. A total of 10 countries plus the European Commission out of 12 ICOS RI (research infrastructure) member countries are represented.

Figure 13 Map showing the 23 labeled stations before 2020. The colors show when the station was labeled: first purple (November 2017), then blue (May 2018), green (November 2018), pink (May 2019) and red (November 2019) clouds. On the right, a zoomed in map shows the 21 labeled stations located in mainland Europe. © OpenStreetMap contributors 2020. Distributed under a Creative Commons BY-SA License. See Table 3 for the acronyms and more details about each station.

The 23 stations cover the majority of western Europe with the most southerly in Italy, the furthest north in Svalbard in the Arctic Circle and one located in the Indian Ocean in the Southern Hemisphere (see Fig. 13 and Table 3). The first ICOS compliant data date from the end of 2015 for two German stations (which began measurements following the ICOS procedure before the operational phase and so the Step 1 application), and the more recent stations have had compliant data since September 2019. A total of 15 stations are continental sites equipped with tall towers with up to six air sampling levels. Four are classified as mountain stations, two as coastal sites with one sampling level and two as remote sites.

The AS specifications also provide guidelines for sampling periphery such as regulators, sampling valves and tubing. This allows a high level of standardization while allowing flexibility for the PIs to design the setup. For the gas distribution to the analyzer, the required equipment is a rotary valve from Valco (model EMT2SD). Alternative options may be accepted after proving their suitability (dead volume, material compatibility, absence of leakages). A drier is recommended but not required.

The 13 sites use the required rotary valve to switch between levels and quality control gases. Three use only solenoid valves, while the last seven use the required valve for the quality control gases but solenoid valves to switch between levels. These valves have proven suitable during audits run by the MobileLab on two of these sites or during the intake line tests run every 6 months.

Four sites (Hyltemossa, HTM, Norunda, NOR, Svartberget, SVB, and Hyytiälä, SMR) equipped with several sampling heights on tall towers use buffer volumes in order to have more hourly representative data at each sampling level. At the Swedish sites, buffer volumes of 8 L are used with an integration time from 3.8 to 4.9 min and a flushing rate between 1.6 to 2.1 L min⁻¹. At SMR, buffer volumes of 5.6 L are flushed at 0.325 L min⁻¹, which gives an integration time of 16.9 min. However, they lose the information about the short-term variability of mixing ratios, which is essential for the application of the spike detection algorithm and therefore important for sites that often experience this type of signal. Of the 19 sites that do not use buffer volumes, 8 (Monte Cimone, CMN, Jungfraujoch, JFJ, Lutjewad, LUT, Pallas, PAL, Puy de Dôme, PUY, Observatoire de l'Atmosphère du Maïdo, RUN, Utö, UTO, and Zeppelin, ZEP) sample at a single height.

During the initial test period, two sites (Observatoire Pérenne de l'Environnement, OPE, and Křešín u Pacova, KRE) were using cryogenic water traps to dry the air. IPR was using a compressor chiller set at a dew point of 5 ^∘C. ZEP was using a Nafion membrane through which all samples pass, and 19 sites were not drying the air for their CRDS measurements. Out of these 19, 4 sites (Lindenberg, LIN, Karlsruhe, KIT, Ochsenkopf, OXK, and Steinkimmen, STE) were also using OA-ICOS N₂O ∕ CO instruments and using the recommended Nafion drier. In May 2020, seven sites (CMN, HTM, PUY, SVB, RUN, Trainou, TRN, JFJ) which were not equipped with a drier during the test period installed a Nafion drier for all their samples, while Hohenpeissenberg (HPB) and Gartow (GAT) added a Nafion drier for their OA-ICOS N₂O ∕ CO instruments. IPR stopped using its chiller in October 2018 and then installed a Nafion drier in May 2020.

Figure 14 Data distribution between ambient air and target and calibration gases for the 23 stations for CO₂, CH₄ and CO over the past year. Calibration is in red, target in blue and air in green/gray. Gray and darker colors are invalid data. The 100 % level is reached when data are available for the full year. For some stations, two instruments are used to measure CO₂, CH₄ and CO (see their ICOS ID on the second line). For others, due to instrumental failure, a new instrument replaced the previous one. See Table 3 for the acronyms and more details about each station.

In Fig. 14, the availability and distribution of data over the past year for CO₂, CH₄ and CO is shown for the 23 stations. Calibration is in red, target in blue and air in green/gray. Gray and darker colors are invalid data. The 100 % level is reached when data are available for the full year. For some stations, two instruments are used to measure CO₂, CH₄ and CO. For other sites, due to instrumental failure, a new instrument has replaced the previous one, and together the data availability reaches 100 %. Table 4 shows for each station and species, and aggregating instruments if needed, the percentage of ambient air and the percentage of rejected data (automatically and manually, including flushing for air and cylinders) over the past year or available period if shorter. Thus the percentages will look different than in the figure for stations that had not data for a full year.

Table 4 Percentage of ambient air and invalid data (manual and automatic, including flushing for air and cylinders) over the whole dataset at each station, aggregating instruments if more than one was used, over the period March 2019–March 2020 or over the available period within these dates if shorter (for stations labeled in November 2019 notably). CO percentages are indicated between brackets when different, i.e., when measured with another analyzer than CO₂ and CH₄ and installed later.

4 Some lessons learned from labeling 23 stations: data, troubleshooting and maintenance

4.1 Calibrations

During the initial test period, all stations followed the recommendations and ran calibrations with four cycles of 30 min injections every 2 weeks. At the end of this phase, it was noticed that for most of the stations, three cycles were enough to get precise results and were thus proposed as a solution to save gas. By 2020, 9 out of 23 had opted to reduce their number of calibration cycles. For most of the stations, it was recommended to keep a 2 week frequency mostly to accommodate for the random variation in CO for the CO₂ ∕ CH₄ ∕ CO CRDS analyzer. Table 5 shows the instrumental drift observed during the test period over at least 6 months at each station for the different greenhouses gases. For the length of the injections (detailed in the next section), it was more variable: for 14 stations, the performances made it possible to reduce by 5 to 10 min the injection length, while for the other nine sites, it was advised to stay with the same schedule or even to increase the flushing time from 10 to 15 min.

Table 5 Instrumental drift calculated as the difference between the first and last data points divided by the number of years. NS: not significant. NA: not applicable, i.e., species not measured or not enough data to estimate a trend. In the case of CO CRDS, the calibration usually shows variability but no drift.

4.2 Tank stabilization time

Table 6 presents the average, minimum and maximum stabilization times observed at the stations for the different types of gases (short-term and long-term targets and calibration), while the individual results for each station are shown in Fig. 15. On average, all samples are stable after 10 min for all species with some being stable after only 1 min. In the case of CO, the high values at 30 min were not due to a problem in the setup of the station but rather to the fact that the two instruments concerned were very noisy (as shown in the tests performed at the ATC MLab prior to installation at the station), and thus the values were systematically ranging above and below the threshold. For CH₄, the CRDS lines at STE showed an outlier with a short-term target only stable after 21 min. Another one (at SMR) had a long-term target slightly above the others and was stable after 15 min.

Table 6 Cylinder sample stabilization time estimated during the initial test period. The mean, minimum and maximum (mean; min; max) for the 23 stations are shown.

^∗ Too noisy for the algorithm to find the stabilization time.

Figure 15 Stabilization time (minutes) for CO₂, CH₄ and CO at the 23 stations at the time of the labeling. Red shows the calibration, green the long-term target and blue the short-term target. On the x axis, the trigram of the station and the ICOS ID of the analyzer are shown. Data from CRDS and OA-ICOS analyzers are shown.

For CO₂, whose criterion is the more stringent, the time to reach stability is higher and with a larger spread over the stations. However, only five sites had samples needing more than 15 min to reach stability. For three sites (CMN, LIN and SVB), only the long-term target was of concern probably due to the fact that it is injected less often, as discussed in Sect. 2.4.3. A longer stabilization time was found for CO₂ for the calibration for GAT CRDS lines and for STE CRDS lines for both CO₂ and CH₄ for the short-term target. Leak tests were recommended to identify the problems. At GAT, the stabilization time is now equivalent for all types of samples between 10 and 12 min. For STE as well, the stabilization time is now reduced to about 12 min for both CO₂ and CH₄.

4.3 Uncertainties

Figures 16, 17 and 18 show the uncertainties and bias of the short-term target for CO₂, CH₄ and CO, as defined in Fig. 12 and discussed in Sect. 2.4.8. They are calculated using 1 year of data. For each station, we show two boxes and a dot. The left box (in pink when large enough to see) uses data from the year before the date of the end of the initial test period. The right box (in blue when large enough to see) uses the last year from March 2019 to March 2020. The red dot shows the MLab initial test values. For example, for a site labeled in May 2018, we use data from 15 April 2017 to 15 April 2018. For the sites labeled last, the periods are almost the same. For some sites, instruments have changed since the labeling, and thus there is one box per instrument.

Figure 16 Uncertainties and bias to the short-term target for CO₂, defined as in Fig. 12. The red dot shows the minute CMR and LTR from the MLab initial tests. The left box (pink) is calculated using data from the year prior to labeling. The right box (blue) is calculated using data from March 2019 to March 2020. For GAT, 489 was prior to 413. For JFJ, 226 replaced 225. At OPE, 729 is running in parallel with 379. At ZEP, 529 was prior to 591. The x axis shows the site trigram and ICOS ID of the analyzer. The black lines in the bias plot show the WMO compatibility goals.

Figure 17 Uncertainties and bias to the short-term target for CH₄, defined as in Fig. 12. The red dot shows the minute CMR and LTR from the MLab initial tests. The left box (pink) is calculated using data from the year prior to labeling. The right box (blue) is calculated using data from March 2019 to March 2020. For GAT, 489 was prior to 413. For JFJ, 226 replaced 225. At OPE, 729 is running in parallel with 379. At ZEP, 529 was prior to 591. The x axis shows the site trigram and ICOS ID of the analyzer. The black lines in the bias plot show the WMO compatibility goals.

Figure 18 Uncertainties and bias to the short-term target for CO, defined as in Fig. 12. The red dot shows the minute CMR and LTR from the MLab initial tests. The left box (pink) is calculated using data from the year prior to labeling. The right box (blue) is calculated using data from March 2019 to March 2020. For GAT, 489 was prior to 413. For JFJ, 226 replaced 225. At OPE, 729 is running in parallel with 379. At ZEP, 529 was prior to 591. The x axis shows the site trigram and ICOS ID of the analyzer. The black lines in the bias plot show the WMO compatibility goals.

For all species, the CMR and LTR are close to the values calculated at the Mlab, and the bias for all sites is mostly within the WMO recommendations. This shows that the setup of the station has not decreased the instrument performances seen at the MLab. Some of the sites show outliers in the left boxes (initial test period) due to problems that have been solved during the initial test period. At HTM, the instrument is known to be poor for CO. It was replaced between May 2018 and September 2019 by an instrument performing better (not shown). Unfortunately, due to a storm at the station, this instrument was damaged, and the old one was reinstalled. At JFJ, the reason for the CO₂ scatter is discussed in Sect. 4.6. For the CO, the outlier in the LTR is due to a single injection that was about 4 ppb higher than the other injections. No particular reason could be identified, but the subsequent injections went back to the normal values. Outliers at SVB in the recent period are related to two problems that have s since been solved. During winter 2019/2020, the PI noticed a large temperature variation in the shelter. After inspection, they found holes in the walls and plugged them in February 2020. At SMR, the outliers are linked to a failure in the instrument pump, which finally broke down and was then changed.

4.4 Intake line tests

During the initial test period or within the next 6 months, 17 out of the 23 stations performed intake line tests. During this exercise, the bias between a reference measurement taken at a free valve port and measurements taken upstream of all the ambient air sampling parts inside the shelter at the outside sampling line connection (see Fig. 2) was calculated and is shown in Fig. 19 as the measured value minus the reference value. This value compared to the WMO compatibility goal and the MLab instrument performances help to determine if there is a leak or an artifact in the system. As seen in Fig. 19, only three stations found a significant bias when testing their system. This highlights the quality of the work done during the setup, as well as the right choices of parts that the ATC deemed compliant. This also demonstrates the importance of carrying out the intake line tests regularly as the target gas measurements alone will not show leakages or artifacts in the ambient air sampling system. Out of the three sites, TOH experienced a positive bias (measured value minus reference value) and hence a leak. However, the subsequent test showed that the leak was coming from the tubing attached to the test gas cylinder and not from the setup itself. KRE and TRN found a negative bias which implies a CO₂ absorption. In the case of KRE, it was attributed to a water trap. When changed, the bias disappeared. For TRN, it was due to a piece of stainless steel tubing that may have been contaminated. Even though the nature of the contamination in the stainless steel tubing has not been clearly identified, it seems related to a water effect on the tubing's inner surface. For both stations, only one sampling level was affected. Station PIs from all ICOS stations have been warned about such possible contamination with this material and advised to always use new tubing when doing modifications in the sampling system.

Figure 19 CO₂ difference between the reference injection and the injection before the shelter's first element. The different colors and shapes show the difference at each sampling levels (green dot: lower level; orange triangle: second level; purple square: third level; pink cross: fourth level). The same color and shape at one given site indicate that several tests have been performed (e.g., TOH). The black lines show the WMO compatibility goals.

4.5 Water correction test

During the initial test period or within the next 6 months, 18 out of the 23 stations performed the water vapor correction assessment test (hereafter called droplet test), described earlier in Sect. 2.3.3. Out of the five that did not, one is using a drying system, and one had its instrument tested less than a year before the initial test period. Eight sites showed a drift in CO₂ up to 0.05 ppm yr⁻¹ at 3 % humidity but mostly insignificant at lower values. No significant drift was observed for CH₄. Most drifts were seen for CO. A total of 11 sites showed a drift from 2 to 7 ppb yr⁻¹ at 3 % and still from 0.5 to 5 ppb yr⁻¹ at 1 % humidity. Only six sites had no data gaps in the water vapor profile.

Two main conclusions can be drawn from the analysis of the water correction test reports: two-thirds of the sites did not manage to obtain a full water vapor profile covering all the water vapor levels. Often, levels around 0.5 % and above 2 % are data depleted. Another 15–20 droplet tests have been performed on instruments not yet labeled, and they show the same pattern: it is difficult to perform a good droplet test despite following all the steps of the protocol. This can be due to factors such as the atmospheric pressure, the room temperature, the filter at the analyzer inlet, etc. However, even with missing data points, the shape of the water vapor influence on CO₂, CH₄ and CO is still visible and gives us qualitative information about the water correction drift over time. Here, we observe no significant drift in CH₄, about half of the analyzers show drifting in CO₂, and two-thirds of the analyzers show a drift in CO. This CO drift is significant and can reach up to 7 ppb yr⁻¹ at 3 % humidity and still up to 5 ppb yr⁻¹ at 1 %. However, when looking at more recent droplet tests, the CO drift seems to stabilize after a few years. These tendencies are illustrated in Fig. 20. Due to the still low number of tests and the known possible erratic behavior of the water vapor effect on CO (Zellweger et al., 2019), this tendency needs to be confirmed. In any case, this highlights either the need for a quantitative way of estimating the water correction on site or the need for drying. ICOS ATC strongly recommends the installation of a drier, which can be a Nafion membrane (no maintenance, controlled artifact with an MD model from Perma Pure, following the reflux method described in Welp et al., 2013) or a cryogenic water trap (at least at −50 ^∘C, regular maintenance, no artifact).

Figure 20 Example of water vapor correction assessment over time for CO. HB stands for humidity bench, while Droplet is for the droplet test described in the text. The humidity bench allows a much more controlled and precise sensitivity test. The two sites' droplets show a large drift in the water vapor correction compared to the initial tests.

4.6 Troubleshooting

Thanks to the works of the PIs following good practice and the ICOS specifications from the installation of their stations, the initial test period did not highlight a high number of cases with major problems. However, for five stations, the labeling had to be postponed by 6 months. In the case of JFJ, the labeling was postponed due the instability of the target gas measurements. For CO₂, the standard deviation reached 0.1 ppm with a bias of 0.09 ppm (see Fig. 16). At that time, a Perma Pure PD series polytube Nafion drier was used, and degradation of it or the counterflow pump was suspected. Changing the pump did not solve the problem, but as soon as the Nafion was removed, the CO₂ standard deviation dropped to 0.02 ppm and a bias of 0.02 ppm. This type of Nafion has subsequently been investigated by the ATC and deemed unsuitable for ICOS use as the possible artifacts induced by polytube Nafion can be significant due to their large inner surface and can be problematic. ICOS ATC recommends the use of single tube Nafion instead.

Another problem was detected at HTM that potentially affected NOR and SVB as the three stations were using the same system on their tall towers. The lower sampling level with a low flow rate showed a positive bias in CO compared to the other sampling levels that could not be explained by the origin of air masses. The intakes in the tower were heated to avoid condensation or ice. This bias was caused by those heated intake cups. It is unclear whether the excess CO originated from the heater or from a plastic part. To be on the safe side, the heating was turned off and the plastic parts replaced by Teflon ones. After these modifications, the CO bias was no longer observed. About 4 months of data were invalid for this lower level at HTM as a result. For the fifth station, KRE, the delay was caused by problems in a faulty drying system, as explained above.

5 Conclusions

In this paper, we have presented the process used to label ICOS atmosphere stations. This process ensures that the produced data are of high quality. For the 23 labeled sites, the PIs implemented the specifications and completed training at ATC before beginning the initial test phase or just after. This allowed for high-quality data with only a few problems detected. The problems were handled, and solutions were implemented to get good quality data. Other stations would benefit from the lessons learned during the certification process. From these data, we can draw conclusions on different topics.

• Regular calibrations (twice a month) are important in particular to follow the CRDS analyzer’s CO variability.

• Reducing the number of calibration injections (from four to three) in a calibration sequence is possible and allows us to extend the calibration gas lifespan and maximize atmospheric measurement time.

• It is often possible to reduce the time of the target and calibration gas injections, but it still has to be determined site by site depending on the instrument's intrinsic performances and the stabilization time (mainly dependent on the upstream sampling system).

• Measuring the target in ultra-dry conditions (i.e., after the calibration cycles) is important to evaluate potential biases in the assigned values.

• The on-site water correction test does not deliver quantitative results for any species. A better way has to be devised to reevaluate the water vapor correction. The use of a drying system is recommended for all analyzers, either a Nafion membrane or a cryogenic water trap (at least at −50 ^∘C).

• Regular intake line tests are useful for detecting artifacts and leaks.