Constraining the accuracy of flux estimates using OTM 33A

Other Test Method 33A (OTM 33A) is a near-source flux measurement method developed by the Environmental Protection Agency (EPA) primarily used to locate and estimate emission fluxes of methane from oil and gas (O&G) production facilities without requiring site access. A recent national estimate of methane emissions from O&G production included a large number of flux measurements of upstream O&G facilities made using OTM 33A and concluded the EPA National Emission Inventory underestimates this sector by a factor of ∼2.1 (Alvarez et al., 2018). The study presented here investigates the accuracy of OTM 33A through a series of test releases performed at the Methane Emissions Technology Evaluation Center (METEC), a facility designed to allow quantified amounts of natural gas to be released from decommissioned O&G equipment to simulate emissions from real facilities (Fig. 1). This study includes test releases from single and multiple points, from equipment locations at different heights, and spanned methane release rates ranging from 0.16 to 2.15  kg h⁻¹. Approximately 95 % of individual measurements (N=45) fell within ±70 % of the known release rate. A simple linear regression of OTM 33A versus known release rates at the METEC site gives an average slope of 0.96 with 95 % CI (0.66,1.28), suggesting that an ensemble of OTM 33A measurements may have a small but statistically insignificant low bias.

1 Introduction

Methane is a potent greenhouse gas, and emissions from the oil and gas (O&G) sector are thought to account for roughly 30 % of total methane emissions in the United States (U.S. EPA, 2019). “Upstream” O&G activities (extraction, production, etc.) are thought to contribute the bulk of emissions within the O&G sector (Alvarez et al., 2012; Zavala-Araiza et al., 2015; Alvarez et al., 2018). However, attempts to quantify O&G methane emissions are hindered by inaccurate emission inventories, a lack of measurements, and variability between basins (Allen, 2016; Schwietzke et al., 2017; Robertson et al., 2017; Alvarez et al., 2018; Omara et al., 2018). For example, basin-wide aircraft measurements of methane emissions from different O&G basins find emissions are generally higher than official inventories published by the U.S. Environmental Protection Agency (EPA; e.g., Karion et al., 2013; Pétron et al., 2014; Brandt et al., 2014; Karion et al., 2015; Schwietzke et al., 2017; Peischl et al., 2018), but the scale of aircraft measurements give little insight into the exact source of emissions on the ground. Site- and component-level measurements are therefore necessary for improving emission estimates of the O&G production sector (Brandt et al., 2014). Existing near-source studies of O&G basins suggest the majority of large, uncontrolled emissions are the result of faulty equipment that may not be noticed for some time (Zavala-Araiza et al., 2017; Omara et al., 2018), emphasizing the need for permanent or semi-permanent monitoring technologies instead of infrequent manual inspections (Coburn et al., 2018; van Kessel et al., 2018). However, more permanent approaches are still under development and must be approved as equivalent monitoring technologies before they can replace existing EPA-approved Leak Detection and Repair (LDAR) methods like optical gas imaging (OGI). Annual or semi-annual LDAR programs already in place rarely quantify total emissions from a site, and the efficacy of these programs depends on many factors including employee experience, leak size, and meteorological variables like wind speed and temperature (Ravikumar et al., 2016, 2018). This makes LDAR programs an important tool for finding leaks and reducing emissions, but they often do not explicitly quantify or provide data of the actual emission rate from production sites, and this limits usefulness for improving emission inventories. In the absence of OGI-equivalent continuous monitoring approaches, both basin- and site-level emission estimates have been gathered using a number of different techniques, all with strengths and weaknesses.

Figure 1 METEC facility with nine of the 11 release points circled. Release points include (clockwise from top of tank) tank candy cane, tank thief hatch, tank front flange, wellhead Kimray packing, wellhead hand valve packing, separator burner fuel supply, separator Kimray vent, separator pressure relief valve (PRV), and separator house PRV. Not pictured: wellhead lubricator flange and wellhead pressure gauge. The University of Wyoming (UW) mobile laboratory is in the background.

One approach is to measure emissions at an O&G production facility. On-site measurement teams typically detect emissions from malfunctioning components via OGI, which can be quantified using high-volume samplers. The drawbacks of on-site measurements include difficulty measuring emission sources rich in volatile organic compounds (VOCs; Brantley et al., 2015), the inability to reach all emission sources (such as the tops of free-standing tanks), difficulty measuring intermittent sources, and the time required for each inspection (Brantley et al., 2014; Bell et al., 2017; Ravikumar et al., 2018). Site access requirements also introduce the possibility of changes in operation when measurement teams are on-site (Alvarez et al., 2018).

The tracer flux ratio (TFR) technique estimates methane emissions by multiplying the observed concentration ratio of methane to a tracer by the known emission rate of the tracer. TFR has been used in both ground-based (Roscioli et al., 2015; Yacovitch et al., 2015) and airborne applications (Daube et al., 2019), though only ground-based approaches have been used for O&G facilities. TFR can quantify all emissions at an O&G facility, and can often differentiate between emissions from relatively close facilities without the need for site access, though access can improve flux estimates (Roscioli et al., 2015). TFR does not require an atmospheric transport model and is therefore insensitive to uncertainties in atmospheric stability and turbulence. Limitations of TFR include the reliance on downwind roadways of sufficient distance (∼0.5–2 km) and reliable wind direction (Omara et al., 2018; Roscioli et al., 2015). Drawbacks of using TFR to estimate methane emissions include the amount of time required to estimate emissions from one site (2.5–2.8 sites d⁻¹; Yacovitch et al., 2017), and the need to transport and release compressed tracer gases (some of which are flammable such as acetylene) near O&G facilities.

As mentioned previously, airborne mass flux measurements have been used to estimate methane emissions from multiple O&G basins (eg., Karion et al., 2013, 2015; Peischl et al., 2015, 2016, 2018; Pétron et al., 2014; Schwietzke et al., 2017). Meteorological requirements (like a fully developed planetary boundary layer and consistent wind direction) make these measurements difficult, especially for expansive O&G basins such as the Permian basin in Texas and New Mexico (Peischl et al., 2018). Emission estimates of individual production sites via aircraft measurements are also possible, but measurement sites typically need to have relatively large emissions and are limited by aircraft range, turning radius, and favorable meteorological conditions (Caulton et al., 2014; Lavoie et al., 2015, 2017; Conley et al., 2017). Additionally, airborne sampling must occur during the day to meet meteorological requirements, and diurnal variability of emissions associated with on-site maintenance could impact aircraft-based emission estimates in some basins (Schwietzke et al., 2017; Vaughn et al., 2018; Zaimes et al., 2019).

A final type of measurement technique used to estimate emissions from O&G production facilities – and the focus of this study – are downwind measurements that estimate emissions by using the methane mixing ratio and wind measurements to derive the source flux. Downwind emission flux estimates are made using parameters measured in the field combined with additional parameters found with Gaussian or atmospheric dispersion models (Brantley et al., 2014; Rella et al., 2015; Caulton et al., 2018; Robertson et al., 2017; Lan et al., 2015; Foster-Wittig et al., 2015). Downwind measurements do not require site access, but may not be able to identify or capture all sources on-site, especially buoyant ones. Similar to TFR, these techniques require downwind roadways (50–200 m away) and consistent wind direction. Operator-approved site access can improve OTM 33A measurement success in regions with limited downwind roadway infrastructure or complex topography. Though sampling time can be considerably faster than TFR or on-site techniques, it is hard to measure enough sites to get a representative sample (and therefore a flux) of an entire O&G basin (Harriss et al., 2015). As a whole, all of the emission measurement techniques mentioned here are only representative of a timescale between seconds and hours, and therefore it is challenging to use them to capture emissions sources with large temporal variability (U.S. EPA, 2014; Brantley et al., 2014; Robertson et al., 2017; Bell et al., 2017; Caulton et al., 2018; Vaughn et al., 2018).

This study focuses on a ground-based mobile emissions measurement approach, Other Test Method 33A (OTM 33A). OTM 33A is among the most common downwind methods, along with TFR, used to measure methane and VOC fluxes from O&G sources (Brantley et al., 2014, 2015; Robertson et al., 2017). A recent study by Bell et al. (2017) compared on-site, OTM 33A, and TFR measurement techniques in the Fayetteville Shale. The results of the Bell et al. (2017) study suggest OTM 33A only captured ∼40 %–60 % of emissions measured or estimated by on-site teams in the Fayetteville Shale when the dominant emission source was an on-site direct measurement rather than a simulated emission source. OTM 33A had a larger low bias when manual or automated unloadings were measured. Manual or automated unloadings occur when the well pressure is not great enough to move liquids from the geologic formation, preventing gas flow to the pressurized sales line. To maximize the pressure differential, the well is vented directly to the atmosphere in order to remove accumulated liquids. This process can be performed manually or automatically, and may use a plunger to assist with liquid removal. This creates an emissions plume with high vertical velocity. It is likely the majority of this plume would pass over the mobile laboratory unless perfect conditions and road access generate a downwind measurement site 200 m or less from the source. The results of the Bell et al. study add uncertainty to recent national methane emission estimates, which relied heavily on OTM 33A measurements in five O&G basins (Alvarez et al., 2018). However, the Alvarez et al. (2018) study also found that basin-wide emission estimates based on OTM 33A facility measurements agreed with airborne basin-wide flux estimates to within measurement uncertainty. Additionally, no significant low bias (>10 %) was detected in numerous (>100) OTM 33A test releases, conducted by multiple groups (Brantley et al., 2014; Robertson et al., 2017). These test releases were all single point-source releases conducted in open terrain without obstacles, which may not be a reliable comparison to the types of emission sources experienced in O&G fields. The discrepancy between results of Bell et al. (2017) study and previous test releases, along with the potential significant impact on national emission estimates, motivated the suite of more realistic test releases described here.

2 Materials and methods

2.1 Mobile laboratory

The University of Wyoming (UW) mobile laboratory is a customized Freightliner Sprinter van. The front of the van is equipped with a horizontal mast that projects instrumentation and the inlet at a fixed height of 4 m above the ground slightly beyond the vehicle's front bumper. Meteorological instruments on the mast include a 3-D sonic anemometer and an all-in-one compact weather station. The mast also includes a camera, an AirMar differential GPS, and a Teflon inlet (1∕4 in. o.d.) for gas-phase species. Ambient air is pulled through the Teflon inlet at a rate of 6.5 L min⁻¹. For the test releases described here, the laboratory was instrumented with a G2204 Picarro Cavity ring-down spectrometer (CRDS) which has been modified to measure water vapor and dry methane concentrations at a frequency of 2 Hz. The Picarro has an additional meter of 1∕8 in. OD Teflon tubing that branches from the main inlet line, resulting in a total sample transit time through the inlet to the instrument of 1 s. This lag is accounted for during data processing. Additionally, the van contains a battery bank which allows the instrumentation and data acquisitions system to be used while the vehicle engine is turned off.

2.2 Instrument calibration

The Picarro response was tested using two methane–zero air mixtures certified by the National Institute of Standards and Technology (NIST; 2.538±0.05 ppm, 101±5 ppm), and ultra-high-purity zero air (UHPA) at intervals throughout the campaign to confirm stability and accuracy. The instrument was always within ±0.01 ppm of the lower NIST standard, ±1 ppm of the higher standard, and ±0.003 ppm of zero when tested with UHPA. The 5 s instrument precision is ±0.002 ppm. Due to the observed instrument stability and accuracy, no calibration adjustments were made to methane concentrations during data processing.

2.3 OTM 33A measurement method

OTM 33A is one of the EPA Geospatial Measurement of Air Pollution Remote Emission Quantification (GMAP-REQ) techniques that was designed to observe, characterize, and/or quantify emissions from a variety of sources, though OTM 33A has been used most often to measure emissions from O&G operations (U.S. EPA, 2014; Thoma, 2012; Brantley et al., 2014, 2015; Robertson et al., 2017). While several quantification approaches are possible with OTM 33A, the one most commonly employed is an inverse Gaussian approach, which is the focus of this manuscript. OTM 33A has three operational parts: concentration mapping, source characterization, and emission rate quantification. Detection of emissions occurs by driving downwind of possible emission sources in an attempt to transect an emissions plume, measure the ambient background trace gas mixing ratio, and, if possible, to rule out any emissions from upwind sources. Source characterization includes observations of temporal variability and emissions composition. If enhancements of methane or other trace gases are detected during downwind transects of a possible source, the laboratory is parked 20–200 m directly downwind within the emission plume to quantify emissions. Care is taken to position the mast directly into the dominant wind direction to minimize impact from turbulent eddies around the vehicle. Once the laboratory is safely positioned, the vehicle is turned off and an OTM 33A flux measurement begins. During the ∼20 min measurement, 2 Hz measurements of wind direction (in x, y, and z), wind speed, temperature, and the methane mixing ratio are collected and time-stamped with a universal data system time. Meanwhile, distance to the possible emission sources relative to the mast of the laboratory are measured using a TruePulse laser range finder (Model 200). If possible, the most likely emission source is identified using an infrared camera (FLIR GF300). Site photos and observations are also collected.

The OTM 33A analysis program, written in MATLAB (2015), estimates an emission mass flux, Q [g s⁻¹], by using the Gaussian dispersion equation (Eq. 1). The terms of this equation are found as follows. First, the lowest 5 % of measured mixing ratios during the ∼20 min measurement are averaged and considered ambient background, which was around 1.9 ppm (±0.15 ppm) of methane for this study. The background value is subtracted from the data to yield methane enhancement. The analysis program bins observed methane enhancements by wind direction into 10^∘ bins (Fig. 2a), and then calculates the average methane enhancement observed in that wind bin. A plot of methane enhancement vs. wind direction is then generated and fit to a Gaussian distribution (Fig. 2b). The Gaussian fit's apex is C_peak [g m⁻³]. To determine the expected spreading of the emission plume, the program calculates atmospheric stability indicator (ASI) values. The ASI values are based on the standard deviation of the two-dimensional wind direction (horizontal spreading), and the standard deviation in vertical wind speed (vertical spreading), also known as the turbulent intensity. The horizontal and vertical ASI values are averaged together into a point Gaussian indicator (PGI) value, which parameterizes the vertical and horizontal plume spread experienced during the OTM measurement. There are seven PGI values which correspond to Pasquill stability classes A–D (Brantley et al., 2014; U.S. EPA, 2019). The PGI and measured source distance are used as inputs to a lookup table that gives the plume dispersion in two dimensions, σ_y [m] and σ_z [m]. The average wind speed $\stackrel{\mathrm{‾}}{U}$ [m s⁻¹] is also calculated for the same time periods that the methane enhancements are observed.

$$\begin{array}{}\text{(1)}& Q=\mathrm{2}\times \mathit{\pi }\times {\mathit{\sigma }}_y\times {\mathit{\sigma }}_z\times \stackrel{\mathrm{‾}}{U}\times C_{\mathrm{peak}}\end{array}$$

Equation (1) does not include any terms for ground reflection of the plume, plume buoyancy/velocity, or differences in height of the emission source and measurement inlet. OTM 33A assumes a single emission point. For this reason, OTM 33A is best suited for measuring O&G facilities with equipment concentrated in one area that have downwind roadways. OTM 33A struggles to quantify plumes with a particularly high vertical velocity or buoyancy (such as manual unloadings, lit or unlit flares, or very hot emissions). In this scenario, the calculated C_peak will not represent the center of the emission plume, leading to underestimations of these sources (Bell et al., 2017). The estimated lower detection limit of the method is 0.01 g s⁻¹ (0.036 kg h⁻¹; Brantley et al., 2014).

Figure 2 Summed methane enhancement and total number of data points in each 10^∘ wind bin (a). Average methane enhancement per 10^∘ wind bin and Gaussian fit (b). Goodness of fit parameter R is calculated following Eq. (S1) in the Supplement.

A series of built-in data quality indicators (DQI) will flag an OTM 33A flux estimate for a variety of reasons, including poor Gaussian fit, inadequate sampling time within the emission plume, too-variable wind speed or direction, or a maximum methane enhancement that is too small. Flags are then added up, and measurements are broken into categories that represent the probability that an OTM measurement is a good flux estimate. For the current study, the same approach as Robertson et al. (2017) and Bell et al. (2017) was used where most of the Category 1 and a few Category 2 measurements that were only flagged for low methane concentrations (max enhancement less than 100 ppb above background) were considered. Occasionally, measurements with very few DQI flags (Category 1 measurements) will be thrown out after review of the Gaussian fit or if IR camera images suggest we are missing most of the emission plume. Full descriptions of the DQI can be found in Sect. S1.2 in the Supplement, Robertson et al. (2017), Brantley et al. (2014), and in the EPA's documentation (U.S. EPA, 2014).

2.4 Test releases

The University of Wyoming performed two sets of test releases to assess the ability of OTM 33A to quantify methane emissions. The first set of tests, the Christman Field Test Releases (CF-TR), were conducted in conjunction with Colorado State University in July and August of 2014 at the abandoned Christman Airfield in Fort Collins, CO. These releases consisted of two configurations, a simple point source (an opened gas cylinder) and manifold (an elevated ∼2 m length of PVC pipe with many perforations). Neither source of methane gas was obstructed, and they were, in essence, single point sources, one slightly broader than the other. Release rates were set using calibrated mass flow controllers and are correct to within 5 %. These tests spanned a variety of release rates (0.2 to 2 kg h⁻¹) and were staged in an open field with no obstructions (clear line of site) between the single methane source and mobile lab. Winds ranged from 2 to 8 m s⁻¹ from the S/SE. The calculated PGI ranged from 2 to 6, which roughly correspond to Pasquill–Gifford stability classes A–D. Mean measurement distance was 78 m, with a range of 34–174 m. Details of these results are reported in Snare (2015) and Robertson et al. (2017).

The more-recent set of tests were performed at the Methane Emissions Technology Evaluation Center (METEC) in Fort Collins, CO in June of 2017. METEC contains multiple faux O&G facilities ranging in size and complexity with decommissioned O&G equipment that has been plumbed to release a known amount of natural gas (>94 % methane) from a multitude of points. For this study, we used one METEC site representative of a small O&G facility that included a condensate storage tank, separator, and wellhead, all of which were plumbed to be possible emission sources, 11 of which were used in this study (Fig. 1). This resulted in 15 release configurations that had from one to three release points at different heights (0.33–4 m), up to 6 m apart from one another. The relative complexity of the site also introduced obstructions (the methane release would have to flow around a large tank or other piece of equipment to reach the mobile lab) which could potentially impact release quantification. Releases spanned 0.17 to 2.15 kg h⁻¹ and were controlled by combining flows from a number of critical orifices, resulting in a four σ release error less than 5 %. Meteorological conditions ranged from sunny to partly cloudy, with average winds from 2 to 9 m s⁻¹ from the E/SE. The calculated PGI ranged from 3 to 6, which roughly correspond to Pasquill–Gifford stability class B–D. Mean measurement distance was 114 m with a range of 53–195 m. One to two duplicate OTM 33A measurements were attempted at different distances for each of the 15 unique METEC test releases configurations.

3 Results

23 OTM 33A test releases were measured during the CF-TR; 21 passed the data quality indicators (DQI; U.S. EPA, 2014) and were included in this analysis. Thirty-four test releases were measured during the METEC-TR, of which 24 passed the DQI and were included in this study. A similar success rate, ∼70 %, has been observed in the majority of the basins measured by the University of Wyoming (Robertson et al., 2017). Of the 24 successful measurements during the METEC test releases, there were 10 replicate measurements (same release configuration but different OTM 33A measurement distances). The following analysis explores different statistical approaches to constrain the error associated with individual OTM 33A measurements and to assess the accuracy and precision of an ensemble of OTM 33A measurements. The latter analysis is especially important given this is how OTM 33A measurements are often scaled up to estimate basin-wide emissions from O&G.

3.1 Evaluating the accuracy of OTM 33A

3.1.1 Percent error analysis

$$\begin{array}{}\text{(2)}& \mathit{\%}\text{Error}={\displaystyle \frac{\text{OTM 33A flux}-\text{known release}}{\text{known release}}}\times \mathrm{100}\end{array}$$

Percent error (Eq. 2) was calculated for each individual measurement made during the test releases. A histogram of percent error for both the CF-TR and METEC-TR indicate a large range of over- and underestimations are possible using OTM 33A (Fig. 3). Percent error ranges from −75 % to 50 % and −60 % to 170 % for CF-TR and METEC-TR, respectively. Figures 4 and 5 show that the larger percent errors correlate with smaller release rates, with OTM 33A generally overestimating smaller releases. Of the CF-TR data, 68 % fall within ±28 % of the known release, which is the 1σ error used by Robertson et al. (2017) and similar to the error reported by the EPA of 72 % of measurements within ±30 % of the known release (Brantley et al., 2014). Of the METEC-TR data, 68 % are within ±38 % of the known release, perhaps suggesting that a slightly higher 1σ error is appropriate, especially if measuring emissions fluxes less than 0.5 kg h⁻¹. For the combined set of test releases, greater than 85 % of the data are within ±50 % of the known value, and 95 % of the data are within ±73 %. If a Gaussian curve is fit to all of the test release data (N=45), the 95 % confidence interval is found to be +54 % to −84 %, suggesting a low bias of −15 % and a 2σ error of ±69 % (Fig. S5 in the Supplement). The rounded 2σ confidence interval for test releases of ±70 % would become 0.58q and 3.33q when q is an OTM 33A estimate made of an unknown emission source in an O&G basin. The number of replicate measurements of METEC release configurations were too small to perform a similar statistical analysis (N=10), but multiple measurements did not decreased the mean OTM 33A measurement error (14.7 % for replicate measurements, 13.1 % for all measurements). Replicate measurements have been shown to improve flux estimates but at the expense of measuring a number of unique sites (Brantley et al., 2014).

Figure 3 Histogram of percent error of the OTM 33A flux estimate for both Christman and METEC test releases. Data are binned in 10 % error bins. Positive percent error corresponds to OTM 33A overestimating the known release rate.

Figure 4 Scatterplot of test release error and release rate. Positive percent error corresponds to OTM 33A overestimating the known release rate.

Figure 5 Box plot of METEC and Christman OTM 33A release errors binned by release rate. The rectangle contains the median value, while the edges represent the 25th and 75th percentiles. Box whiskers include the rest of the data (100 % coverage). Positive percent error corresponds to OTM 33A overestimating the known release rate.

3.1.2 Ordinary least-squares regression

Another approach to assess the performance of OTM 33A is using an ordinary least-squares (OLS) regression applied to a correlation plot of the OTM 33A flux estimate versus the known release rate. Assuming the OTM-measured flux and known release rate converge at (0,0) yields OLS slopes of 0.91 for CF-TR and 0.92 for METEC-TR (Fig. 6). This suggests OTM 33A may have a $\sim -\mathrm{10}$ % negative bias when an ensemble of measurements are considered. Notably, the increased complexity of the METEC-TR did not yield a more significant bias like that reported by Bell et al. (2017). Statistical analysis of the residuals to assess point leverage and possible outliers supports the validity of an OLS approach for the test release data. Removing the largest outlier found with the Cook's test improves both OLS fits slightly to 0.97 for METEC-TR and 1.1 for CF-TR. Residual plots are included in Sect. S2.

Figure 6 Correlation plot of OTM 33A-measured flux versus known release rates. Intercept is set to (0,0).

95 % confidence intervals (CI) for the OLS fit were calculated through bootstrapping following the method detailed in Robertson et al. (2017), whereby the linear regressions of bootstrapped data sets are calculated to assess the range of possible regressions. Bootstrapping was used because it does not require an assumption of normally distributed data (unlike the Gaussian fit approach used in Sect. 3.1.1). Using this method, the OLS correlation slopes have a mean (and 95 % CI) for the CF-TR of 0.96(0.56,1.47) and 0.96(0.66,1.28) for the METEC-TR.

3.1.3 Bland–Altman analysis

Because the rate of methane releases for both the METEC and Christman tests are known to within a small margin of error (<5 %), OLS regression, which assumes no error in the independent variable, is a reasonable approach. However, OLS analysis is weighted by larger release rates and may not give an accurate representation of OTM 33A performance at all methane emission rates. Bland–Altman (BA) analysis removes this bias by considering the difference between the test release and OTM measurements (known release – OTM flux) as a function of a known release rate (Fig. 7; Giavarina, 2015). Bland–Altman analysis also assumes that the method difference (y axis) comes from a normal distribution. Kolmogorov–Smirnov statistical tests supporting the normality of the method difference can be found in Sect. S3. For BA analysis, if the 2σ range of method difference includes zero, the methods are considered to be statistically equivalent, i.e., without bias (Giavarina, 2015). The BA plot also illustrates the amount by which OTM can overestimate (negative numbers) or underestimate (positive numbers) the known release. On average, the CF-TR and METEC-TR both underestimate the known releases, with mean differences of 0.028 and 0.025 kg h⁻¹, respectively. However, since the 2σ interval includes zero, the BA analysis identifies no statistical difference between the OTM 33A flux estimate and the known release rate.

Figure 7 Bland–Altman analysis of Christman and METEC test releases. Bold black lines represent the mean difference between the test release and the OTM measurement, while the dashed yellow and red lines indicate 1 and 2 standard deviations of the method difference.

3.1.4 Orthogonal distance and variance-weighted least-squares regression

Other approaches for minimizing the influence of larger release rates on the OLS fit include orthogonal distance regression (ODR) and variance-weighted least-squares regression (VWLS). These methods take into account error in both the x and y variables, and require that each measurement has an independent uncertainty estimate on both axes. Since uncertainty of OTM 33A flux estimates is taken as a fixed percentage of the estimated value, these methods tend to perceive a higher confidence (smaller absolute uncertainty) in smaller estimates, and a lower confidence (higher absolute uncertainty) in larger estimates. This results in a fit with a low bias since the estimates with smaller absolute uncertainty are strongly weighted, and less weight is given to estimates with larger uncertainties. To examine these approaches, the METEC-TR are used as an example below.

Applying a measurement uncertainty of ±50 % (representing the percent error that roughly 85 % of the data points are within) for each OTM measurement and the metered uncertainty for each METEC-TR in kg h⁻¹ yields an ODR slope of 0.79±0.09 when the intercept is set to (0,0) (Fig. 8). A lower slope of 0.67±0.1 is found using the VWLS method. In this case the ODR and VWLS regressions suggest the OTM flux estimates are 20 %–33 % lower than the known releases, where an OLS regression indicates the method is only 8 % low. Total emissions estimated by OTM 33A (23.074 kg h⁻¹) are 2.5 % lower than the total known emission rates (23.67 kg h⁻¹), suggesting OLS regression is a better fit for this data set.

Figure 8 Correlation plot of known release and OTM 33A flux estimate for the METEC test releases. Plot includes OLS (slope=0.92), ODR (slope=0.79), and VWLS (slope=0.67) regressions.

VWLS and ODR should be used with caution where the measurement uncertainty is not independent of the measurement (i.e., a constant fractional error), because this may discriminate against data points of a larger magnitude depending on uncertainty in the other (x or y) variable. Bell et al. (2017) used a VWLS regression to compare OTM 33A measurements to on-site measurements of O&G production facilities in the Fayetteville Shale, which yielded a correlation of $\mathrm{0.41}(+\mathrm{0.51},-\mathrm{0.17})$. The 95 % CI of the VWLS regression are calculated through bootstrapping the regression and considering the uncertainties in both the study on-site estimate and the OTM 33A estimate for each data point. Repeating this analysis using an OLS regression without an intercept results in a correlation of $\mathrm{0.39}(+\mathrm{0.39},-\mathrm{0.15})$ (Fig. S6). If an OLS regression were chosen instead of the VWLS, the conclusion by Bell et al. (2017) that OTM 33A produced lower emissions estimates than the on-site measurement results would not have been effected. The total mass flux measured by OTM 33A ($\mathrm{13}(+\mathrm{5.3},-\mathrm{2.1})$ kg h⁻¹) and on-site teams ($\mathrm{19}(+\mathrm{7.7},-\mathrm{3})$ kg h⁻¹) also supports the conclusion that the OTM 33A flux estimate was biased low relative to the on-site measurements at these paired facilities.

3.2 Sensitivity analysis

3.2.1 OTM 33A sensitivity to source distance

Because OTM 33A assumes a point source, the distance to the release point has a large influence, as this impacts the modeled plume spread and therefore the final calculated flux. The importance of an accurate source distance in Gaussian plume modeling has been noted in previous studies (Lan et al., 2015; Caulton et al., 2018). During the METEC-TR, the University of Wyoming measurement team had site access and we were able to determine the exact emission point(s) using an IR camera. With this knowledge, we were able to calculate the exact source distance, or the average distance in the case of multiple emission sources. In the field, site access is often not available and it is often not possible to detect the most likely emission point(s). For this reason, the average distance of possible emission points is used when calculating source distance.

OTM 33A sensitivity to source distance was tested two ways for the METEC test releases. The following test was performed during the data analysis stage, and compared the flux estimated using the average distance of all the components that could be sighted with the range finder from the van (e.g., wellhead, separator, tank) to the flux estimated using the distance to the known release point or point distance (identified using the FLIR camera). Although the well pad measured at the METEC facility was quite small (∼6 m by 6 m), the average source distance was larger than the specific source distance ∼60 % of the time. The change in the OTM 33A flux (ΔFlux) as a result of changing the measurement distance (ΔDistance) was found using Eqs. (3) and (4).

$$\begin{array}{}\text{(3)}& {\displaystyle }& {\displaystyle }\mathit{\%}\mathrm{\Delta }\text{Distance}={\displaystyle \frac{\text{Average Distance}-\text{Point Distance}}{\text{Average Distance}}}\times \mathrm{100}\text{(4)}& {\displaystyle }& {\displaystyle }\mathit{\%}\mathrm{\Delta }\text{Flux}={\displaystyle \frac{\text{Average OTM}-\text{Point OTM}}{\text{Average OTM}}}\times \mathrm{100}\end{array}$$

A correlation plot of %ΔDistance and %ΔFlux suggests that for a 5 % change in source distance, the OTM 33A flux estimate would increase by almost 10 % (Fig. 9a). In terms of mass error, the OTM flux estimated by the average or specific source distance has very little impact on the over- or underestimation of the METEC known release (Fig. 9b). Allowing this fit to have an intercept changes the linear fit to $y=\mathrm{0.978}x-\mathrm{0.03}$, a negligible difference. Source distance-related error is small in the context of the ±70 % measurement error, but this analysis underscores how determination of the exact emission point can further reduce errors in the field.

Figure 9 (a) Correlation between percent change in distance ($(\text{Avg. distance}-\text{point})/\text{point}\cdot \mathrm{100}$), and resulting percent change in OTM flux ($(\text{OTM avg}-\text{OTM})/\text{OTM}\cdot \mathrm{100}$). (b) OTM 33A flux mass error compared to the METEC known release when using average versus known source distance.

OTM 33A sensitivity to distance was also tested in the field during the METEC test releases. For configurations that had both a “closer” (generally<70 m) and “farther” (generally>100 m) measurement distance for replicate measurements, the closer measurement had a flux estimate closer to the known release 78 % of the time (Sect. S1.4). The average distance of the closer replicate measurements (78 m) is comparable to the average measurement distances for the CF-TR of 78 m, smaller than the mean METEC-TR distance of 114 m, and larger than the measurement distances during the Arkansas campaign of 46 m (20–113 m; Robertson et al., 2017; Bell et al., 2017). For both the CF-TR and the METEC-TR, there is no obvious increase in percent error as measurement distance increases (Fig. 10a), suggesting the underestimation reported by Bell et al. (2017) cannot be blamed solely on closer measurement distances.

3.2.2 OTM 33A error – sensitivity to wind speed

One hypothesized reason for the underestimation of OTM 33A compared to on-site methods reported in the Bell et al. (2017) study is the lower wind speeds (<2 m s⁻¹) experienced in that study. The CF-TR and METEC-TR both had wind speeds higher than 2 m s⁻¹, making an absolute conclusion impossible, but for the wind speeds measured there is no obvious trend between the mean measurement wind speed and OTM 33A error (Fig. 10b).

Figure 10 OTM 33A percent error compared to the measurement distance (a). OTM 33A percent error compared to the mean measurement wind speed (b). Positive percent error is OTM overestimating the known release.

3.2.3 OTM 33A error – sensitivity to number of sources and source height

The METEC-TR included multiple emission points, both slightly above and below the sampling inlet height. There is no obvious trend between the number of release points and percent error, though the sample size for two or more sources is relatively small (N=6). The height of the sources tested also show no obvious influence on OTM 33A accuracy (Fig. 11).

Figure 11 Percent error of the METEC-TR depending on source height or average height (for multiple sources). Positive error is OTM 33A overestimating the known release rate. Icons indicate the size bin of the known release rate.

3.3 Ensemble mass flux

OTM 33A measurements are often used to find an average emission rate per well or per facility in an O&G basin (Robertson et al., 2017; Alvarez et al., 2018). To assess the accuracy of the mean of a number of OTM 33A measurements, the mean mass flux measured by OTM 33A is compared to the mean mass flux of the known release through bootstrapping. The bootstrapping approach is used to generate more statistically robust results without the need for assuming Gaussian distributions. The OTM 33A flux estimates and known releases (including their respective measurement uncertainties) are sampled with replacement, summed, and compared following Robertson et al. (2017). This approach suggests that the addition of complexity in the METEC-TR did not significantly impact the accuracy of OTM 33A (Fig. 12), and for both sets of test releases there is a large amount of overlap between the OTM 33A and known release distributions (Fig. S6). These results also indicate OTM 33A does not drastically underestimate the total emissions for an ensemble or group of measurements, and that scaling-up mean emissions measured with OTM 33A to an entire basin is a valid approach.

Figure 12 Probability density function (PDF) of bootstrapped mean mass emission flux for OTM measurements and the known releases. Mean and 95 % CI in kg h⁻¹ are as follows. Christman: known release – (0.54(0.37,0.75)), OTM – (0.51(0.34,0.73)). METEC: known release – (0.85(0.58,1.13)), OTM – (0.84(0.60,1.11)).

Figure 13 summarizes measurement conditions experienced during four University of Wyoming field campaigns (Robertson et al., 2017) and the test releases. Data from the four O&G basins represented a significant fraction of data, along with other field campaigns used by Alvarez et al. (2018) to generate an estimate of national methane emissions. Measurement conditions in Fayetteville, Arkansas (AR), were notable for closer source distance, lower wind speeds, and generally more unstable atmospheric conditions. All of these variables could have influenced the low bias reported by Bell et al. (2017) and are conditions that should be replicated (if possible) in future test releases.

Figure 13 Summary of accepted OTM 33A measurements from field deployments and test releases (right of vertical line). Basins from Robertson et al. (2017). Upper Green River Basin, Wyoming (UGRB), Uintah Basin, UT (UB), Denver–Julesburg Basin, CO (DJ), Fayetteville, Arkansas (AR). Mean statistics (from left to right) are as follows. Distance (m): 98, 114, 83, 51, 114, 78. Flux (kg h⁻¹): 2.41, 6.99, 1.51, 1.27, 0.51, 0.96. Mean wind speed (m s⁻¹): 5.3, 4.2, 3.1, 2.9, 4.1, 4.9. Stability class: 5.0, 4.9, 5.3, 3.5, 5.0, 5.4.

4 Conclusions

The more realistic test releases described in this study build on preexisting test releases and suggest a single OTM 33A measurement can have a 2σ error of ±70 %. Analysis of both the simple CF-TR and more complex METEC-TR indicate that under these measurement conditions and release rates, an ensemble of OTM 33A may have a slight negative bias (∼5 %) when compared to a known release rate through an OLS model. The mean and 95 % CI found through bootstrapping are 0.96(0.56,1.47) and 0.96(0.66,1.28) for the CF-TR and METEC-TR, respectively. The 40 %–60 % underestimation reported in the Bell et al. (2017) study was not replicated during either test release experiment.

OTM 33A flux estimates are sensitive to the assumed source distance, with a +5 % change in source distance corresponding to a $\sim +\mathrm{10}$ % change in the OTM flux. However, the error caused by uncertainty in source distance is small compared to the measurement method error determined through these test releases. During field measurements, uncertainty in source distance can be mitigated by having site access and using an IR camera to detect the emission source(s). Uncertainty did not correspond to wind speeds observed during the test releases, but was relatively higher for smaller release rates. Sensitivity of OTM 33A to the number or height of emission sources was inconclusive.

For both test release experiments, the maximum release rates (2–2.15 kg h⁻¹) were constrained by available resources and facility throughput and, while they represent a large fraction of emission rates observed in the field, they do not fully encompass the dynamic range of emissions observed in an O&G basin. The bootstrapped mean emission rates from four O&G basins measured by the University of Wyoming range from 0.68 to 3.7 kg h⁻¹ (Robertson et al., 2017), suggesting the range of these test releases may not be representative of the largest emission rates observed in the field (Fig. 13).

OTM 33A has been used to estimate mean facility emissions and basin-wide facility emissions in a number of O&G basins. The mean mass fluxes and 95 % CI for each test release experiment are not statistically different. This analysis lends confidence to national emission estimates from the O&G production sector using OTM 33A measurements. Despite the OTM 33A estimated limit of detection (0.01 g s⁻¹) and relative overestimation of smaller release rates, the analyses reported here and the study by Bell et al. (2017) suggest that OTM 33A does not overestimate an ensemble of flux estimates.