the Creative Commons Attribution 4.0 License.
the Creative Commons Attribution 4.0 License.
Quantifying the uncertainties in thermal-optical analysis of carbonaceous aircraft engine emissions: An interlaboratory study
Abstract. Carbonaceous particles, such as soot, make up a notable fraction of atmospheric particulate matter and contribute substantially to anthropogenic climate forcing, air pollution, and human health. Thermal-optical analysis (TOA) is one of the most widespread methods used to speciate carbonaceous particles and divides total carbon (TC) into the operationally defined quantities of organic carbon (OC; carbon evolved during slow heating in an inert atmosphere) and elemental carbon (EC). While multiple studies have identified fundamental scientific reasons for uncertainty in distinguishing OC and EC, far fewer studies have reported on interlaboratory reproducibility. Moreover, existing reproducibility studies have focused on complex atmospheric samples. The real-time instruments used for regulatory measurements of aircraft engine non-volatile particulate matter (nvPM) mass emissions are required to be calibrated to the mass of EC determined by TOA of the filter-sampled emissions of a diffusion flame source. However, significant differences have been observed in the calibration factor for the same instrument based on EC content determined by different calibration laboratories. Here, we report on the reproducibility of TC, EC, and OC quantified using the same TOA protocol, instrument model (Sunset 5L), and software settings (auto split-point: Calc405) across five different laboratories and instrument operators. Six unique data sets were obtained, with one laboratory operating two instruments. Samples were collected downstream of an aircraft engine after treatment with a catalytic stripper to remove volatiles. We compared laboratory-reported uncertainties with actual variability in the data set, the difference of which (dark uncertainty) was substantial. Interlaboratory (dark) contributions increase uncertainties by a factor of 1.2 – 1.6 relative to the laboratory-reported uncertainties, even for these relatively simple samples (combustion particles downstream of a stripper), resulting in uncertainties of 26 % (k = 2) for EC. Uncertainties were a little larger for EC than for OC. These results indicate that current TOA uncertainties are underestimated and should be adjusted upwards to reflect these interlaboratory differences.
- Preprint
(1019 KB) - Metadata XML
- BibTeX
- EndNote
Status: closed
-
RC1: 'Comment on amt-2024-1', Anonymous Referee #2, 18 Mar 2024
This study comprehensively evaluates the previously poorly quantified inter-laboratory uncertainty in TOA analysis used to derive EC, OC, and TC. Aerosol exhaust from a helicopter engine was collected on 20 filters in accordance with regulatory civil aviation specification for nvPM mass instrument calibration. The comprehensive analysis presented in this study underscore the importance of including the inter-laboratory contribution to EC, OC, and TC uncertainties as it was found to be significant. I enjoyed reading your manuscript, which is well written, includes informative figures, and provides a detailed description of the methods and uncertainty analysis. However, I felt that the implications and limitations of your work were not discussed extensively. Below are a few general comments and minor suggestions for the authors to consider.
Minor comments:
- Line 19: You’ve omitted to say that regulatory aircraft nvPM mass can be directly calibrated on a gas turbine, not just a diffusion flame.
- Line 27: “Uncertainties were a little larger for EC than for OC”; I suggest merging with previous sentence and directly report OC uncertainty rather than using “a little larger”.
- Line 200-204: It is unclear to me how the 'dark inter-laboratory uncertainty', as shown in Figure 5 and Table 2, could be lower for TC than for EC, given that TC is derived from EC. Can you clarify and discussed in the main text if relevant?
- The sections 3.1 and 3.2 titles aren’t very clear, I suggest renaming them (for example 3.1 could be EC, OC and TC uncertainty and 3.2 could be EC/OC uncertainty).
- Line 287-288: ”uncertainties are poorly captured by existing estimates for these measurements”; Can you clarify that you mean by existing estimates? Do you mean lab-reported or what has been reported in the literature or both? Can you also expand on “except perhaps for measurements of OC” because I didn’t get that impression from your literature review in the introduction.
- L304: It would be useful to introduce the term “metrological” earlier in the manuscript to ensure that all readers are familiar with the term and its significance.
General comment:
- Are there any instruments other than the Sunset 5L commercially available and used by competent laboratories to make TOA measurements? Could your reported uncertainties be specific to that instrument? I suggest discussing this where relevant.
- Something that caught my attention is that your calculated cumulated uncertainties (e.g., 26% (k = 2) for EC) from your highly controlled study (i.e., nvPM, identical filters, known composition) are higher than that reported from other studies (Schauer et al., Ten Brink et al, Panteliadis et al., Brown et al.), yet you mentioned that atmospheric samples from these studies should have higher uncertainties than yours. Why do you think you estimated a higher overall uncertainty than say, Panteliadis et al.? It would be interesting to discuss these differences in more details.
- Could the way each laboratories take their punches and/or their handling of the punches be responsible for the laboratory bias you reported (could one laboratory take their punch in a way that led to a systematic bias)?
- I suggest adding more discussion on the implications and limitations of your work. By that, I mean discussing alternatives to TOA and how to reduce TOA uncertainty for regulatory aircraft nvPM mass emissions (use manual split, more thorough calibration procedures and quality controls, alternative calibration methods, etc).
Citation: https://doi.org/10.5194/amt-2024-1-RC1 - AC1: 'Reply on RC1', Timothy Sipkens, 03 May 2024
-
RC2: 'Comment on amt-2024-1', Anonymous Referee #1, 19 Mar 2024
Review of “ Quantifying the uncertainties in thermal-optical analysis of carbonaceous aircraft engine emissions: An interlaboratory study”
This article presents an interesting and pertinent study on the estimation of measurement uncertainties on the total carbon (TC), elemental carbon (EC) and organic carbon (OC) contents, measured with the instrument Sunset 5L owned by five certified laboratories worldwide. Six equally shared punches from 20 samples probed from the exhaust of a helicopter engine were distributed for analysis among the chosen laboratories. Each instrument user respected the NIOSH5040 analysis protocol. The database was analyzed with statistical multilevel models to identify or predict the uncertainty bias among the samples.
The article is well-written and fits the topic of this journal. The obtained results are highly interesting for research topics such as nvPM emissions measurement protocols, atmospheric measurements, and aviation emissions, measurements, and protocols. There are a few arguably contrasting points that deserve to be put in a better light or clarified.
General remarks to clarify
- The text does not clearly state when the samples were obtained. Is it the same work as Olfert et al. (2017) or another specific study? Please clarify this aspect.
- Despite the significant experimental work in the referenced work Olfert et al., 2017, this article does not state which engine operating conditions were used for the obtained samples. It may not seem relevant to the authors, but why not have well-identified conditions in which nvPM was produced by the engine? I think this information is essential since the same operating conditions of the engine were used for three sets of samples loaded with 50, 100 and 250 µg/m3 of soot, while the last sample loaded with 500 µg/m3 of soot was obtained by increasing the RPM of the engine. It is well known that changing the engine's operating conditions will impact the structure and morphology of soot particles/nvPM. Isn’t this contradictory with what the authors state in the paragraph from the introduction containing Lines 74 to 77? The reader can find additional information about the sampled particles on the filter by identifying the operating conditions in the specified article if the work is common and even though the detailed statistical analysis did not identify any correlation between the filter loadings and uncertainties (lines 183-184).
- The filter holder from Figure 1 contains two filter holders in series. Was the second filter analyzed for some residual TC content, as presented in the work of Corbin et al. (2020)?
- It is surprising that the different loadings of the samples do not affect the uncertainty measurement of the three quantities measured by the Sunset instrument. This finding deserves a more detailed discussion since studies show that the loading of the filter impacts the uncertainty measurement of the thermo-optical analysis measurements.
- The use of the word structure (lines 183, 225, 229) and structural trends (line 227) can sometimes be misleading in the text for readers who are not specialized in multilevel statistical analysis. Please be more specific where it is the case; such as data/uncertainty structure or something that fits better in the context.
Specific comments:
Figure 2 - to which sample corresponds to the obtained data? it is worth mentioning.
Figure 6 - what represents the error bars in the bottom graphs with the Relative value [%] since it is mentioned that the error bars are excluded for clarity?
Line 37: remove on from the sentence “… mass on collected …”
Line 110: “darkness …” can be replaced with “coverage …” to differentiate from dark uncertainty
Line 136: remove the from the sentence “… and the their uncertainty …”
Line 199: The authors mentioned, “These filters coincide with cases where the overall variance is larger and represent a minority of cases.” Please be more specific when selecting a criterion for the value of the variance to eliminate the sample in question. Either be specific and justify why this selection was made or mention if you referred to data outliers.
Citation: https://doi.org/10.5194/amt-2024-1-RC2 - AC2: 'Reply on RC2', Timothy Sipkens, 03 May 2024
-
AC3: 'Comment on amt-2024-1', Timothy Sipkens, 03 May 2024
Beyond changes in response to the reviewers below, discussions with other researchers have led us to update the statistical model used in the analysis. This was done for several reasons:
1. To increase the consistency with the language in ISO 5725, including the use of repeatability, reproducibility, within-laboratory uncertainties, and between-laboratory uncertainties. This is better aligned with Panteliadis et al., who used the cited standard for analysis.
2. To update the symbols to reflect those used in ISO 5725. For example, reproducibility is now represented by sR.
3. To update the statistical model with the objective to better compute within-laboratory uncertainties, where, rather than relying on laboratory-reported uncertainties, within-laboratory uncertainties are computed as part of the MCMC procedure. The increases consistency with previous literature.
4. Removal of the consensus value contributions, as these were found to be highly correlated with the between-laboratory uncertainties, misrepresenting the overall uncertainties.
In concert with these changes, the text has been updated to reflect the new MCMC outputs and the figures have been updated accordingly. While the decomposition of the uncertainties has changed, the overall conclusions of the manuscript remain roughly the same. The reframing of the problem allows for better comparisons to the literature (for instance, with Panteliadis et al. due to similarities with ISO, and with Conrad and Johnson who only assessed repeatability, albeit for a range of conditions).
The change in the statistical model had a minor impact on the overall reproducibility:
EC: 17.0 % > 16.5 %
OC: 17.4 % > 8.0 %
TC: 15.4 % > 12.1 %
OC was most impacted. This change resulted from not relying on the laboratory-reported uncertainties and better accounting for the lack of systematic errors in the measurements.We have also added calculations for EC/TC ratios, including notes on incorporating correlation between the EC and TC measurements, to the body of the manuscript and have added a comparison to an application of the ISO 55725-2 method to the Suppemental Information.
The above changes have not been marked in the manuscript, as this would mask other changes specific to the reviewer’s comments. However, this amounts to substantial changes to Sec. 2.2, 3.1, and the latter portion of 3.2.
The data will also be added as Supplemental Information.Citation: https://doi.org/10.5194/amt-2024-1-AC3
Status: closed
-
RC1: 'Comment on amt-2024-1', Anonymous Referee #2, 18 Mar 2024
This study comprehensively evaluates the previously poorly quantified inter-laboratory uncertainty in TOA analysis used to derive EC, OC, and TC. Aerosol exhaust from a helicopter engine was collected on 20 filters in accordance with regulatory civil aviation specification for nvPM mass instrument calibration. The comprehensive analysis presented in this study underscore the importance of including the inter-laboratory contribution to EC, OC, and TC uncertainties as it was found to be significant. I enjoyed reading your manuscript, which is well written, includes informative figures, and provides a detailed description of the methods and uncertainty analysis. However, I felt that the implications and limitations of your work were not discussed extensively. Below are a few general comments and minor suggestions for the authors to consider.
Minor comments:
- Line 19: You’ve omitted to say that regulatory aircraft nvPM mass can be directly calibrated on a gas turbine, not just a diffusion flame.
- Line 27: “Uncertainties were a little larger for EC than for OC”; I suggest merging with previous sentence and directly report OC uncertainty rather than using “a little larger”.
- Line 200-204: It is unclear to me how the 'dark inter-laboratory uncertainty', as shown in Figure 5 and Table 2, could be lower for TC than for EC, given that TC is derived from EC. Can you clarify and discussed in the main text if relevant?
- The sections 3.1 and 3.2 titles aren’t very clear, I suggest renaming them (for example 3.1 could be EC, OC and TC uncertainty and 3.2 could be EC/OC uncertainty).
- Line 287-288: ”uncertainties are poorly captured by existing estimates for these measurements”; Can you clarify that you mean by existing estimates? Do you mean lab-reported or what has been reported in the literature or both? Can you also expand on “except perhaps for measurements of OC” because I didn’t get that impression from your literature review in the introduction.
- L304: It would be useful to introduce the term “metrological” earlier in the manuscript to ensure that all readers are familiar with the term and its significance.
General comment:
- Are there any instruments other than the Sunset 5L commercially available and used by competent laboratories to make TOA measurements? Could your reported uncertainties be specific to that instrument? I suggest discussing this where relevant.
- Something that caught my attention is that your calculated cumulated uncertainties (e.g., 26% (k = 2) for EC) from your highly controlled study (i.e., nvPM, identical filters, known composition) are higher than that reported from other studies (Schauer et al., Ten Brink et al, Panteliadis et al., Brown et al.), yet you mentioned that atmospheric samples from these studies should have higher uncertainties than yours. Why do you think you estimated a higher overall uncertainty than say, Panteliadis et al.? It would be interesting to discuss these differences in more details.
- Could the way each laboratories take their punches and/or their handling of the punches be responsible for the laboratory bias you reported (could one laboratory take their punch in a way that led to a systematic bias)?
- I suggest adding more discussion on the implications and limitations of your work. By that, I mean discussing alternatives to TOA and how to reduce TOA uncertainty for regulatory aircraft nvPM mass emissions (use manual split, more thorough calibration procedures and quality controls, alternative calibration methods, etc).
Citation: https://doi.org/10.5194/amt-2024-1-RC1 - AC1: 'Reply on RC1', Timothy Sipkens, 03 May 2024
-
RC2: 'Comment on amt-2024-1', Anonymous Referee #1, 19 Mar 2024
Review of “ Quantifying the uncertainties in thermal-optical analysis of carbonaceous aircraft engine emissions: An interlaboratory study”
This article presents an interesting and pertinent study on the estimation of measurement uncertainties on the total carbon (TC), elemental carbon (EC) and organic carbon (OC) contents, measured with the instrument Sunset 5L owned by five certified laboratories worldwide. Six equally shared punches from 20 samples probed from the exhaust of a helicopter engine were distributed for analysis among the chosen laboratories. Each instrument user respected the NIOSH5040 analysis protocol. The database was analyzed with statistical multilevel models to identify or predict the uncertainty bias among the samples.
The article is well-written and fits the topic of this journal. The obtained results are highly interesting for research topics such as nvPM emissions measurement protocols, atmospheric measurements, and aviation emissions, measurements, and protocols. There are a few arguably contrasting points that deserve to be put in a better light or clarified.
General remarks to clarify
- The text does not clearly state when the samples were obtained. Is it the same work as Olfert et al. (2017) or another specific study? Please clarify this aspect.
- Despite the significant experimental work in the referenced work Olfert et al., 2017, this article does not state which engine operating conditions were used for the obtained samples. It may not seem relevant to the authors, but why not have well-identified conditions in which nvPM was produced by the engine? I think this information is essential since the same operating conditions of the engine were used for three sets of samples loaded with 50, 100 and 250 µg/m3 of soot, while the last sample loaded with 500 µg/m3 of soot was obtained by increasing the RPM of the engine. It is well known that changing the engine's operating conditions will impact the structure and morphology of soot particles/nvPM. Isn’t this contradictory with what the authors state in the paragraph from the introduction containing Lines 74 to 77? The reader can find additional information about the sampled particles on the filter by identifying the operating conditions in the specified article if the work is common and even though the detailed statistical analysis did not identify any correlation between the filter loadings and uncertainties (lines 183-184).
- The filter holder from Figure 1 contains two filter holders in series. Was the second filter analyzed for some residual TC content, as presented in the work of Corbin et al. (2020)?
- It is surprising that the different loadings of the samples do not affect the uncertainty measurement of the three quantities measured by the Sunset instrument. This finding deserves a more detailed discussion since studies show that the loading of the filter impacts the uncertainty measurement of the thermo-optical analysis measurements.
- The use of the word structure (lines 183, 225, 229) and structural trends (line 227) can sometimes be misleading in the text for readers who are not specialized in multilevel statistical analysis. Please be more specific where it is the case; such as data/uncertainty structure or something that fits better in the context.
Specific comments:
Figure 2 - to which sample corresponds to the obtained data? it is worth mentioning.
Figure 6 - what represents the error bars in the bottom graphs with the Relative value [%] since it is mentioned that the error bars are excluded for clarity?
Line 37: remove on from the sentence “… mass on collected …”
Line 110: “darkness …” can be replaced with “coverage …” to differentiate from dark uncertainty
Line 136: remove the from the sentence “… and the their uncertainty …”
Line 199: The authors mentioned, “These filters coincide with cases where the overall variance is larger and represent a minority of cases.” Please be more specific when selecting a criterion for the value of the variance to eliminate the sample in question. Either be specific and justify why this selection was made or mention if you referred to data outliers.
Citation: https://doi.org/10.5194/amt-2024-1-RC2 - AC2: 'Reply on RC2', Timothy Sipkens, 03 May 2024
-
AC3: 'Comment on amt-2024-1', Timothy Sipkens, 03 May 2024
Beyond changes in response to the reviewers below, discussions with other researchers have led us to update the statistical model used in the analysis. This was done for several reasons:
1. To increase the consistency with the language in ISO 5725, including the use of repeatability, reproducibility, within-laboratory uncertainties, and between-laboratory uncertainties. This is better aligned with Panteliadis et al., who used the cited standard for analysis.
2. To update the symbols to reflect those used in ISO 5725. For example, reproducibility is now represented by sR.
3. To update the statistical model with the objective to better compute within-laboratory uncertainties, where, rather than relying on laboratory-reported uncertainties, within-laboratory uncertainties are computed as part of the MCMC procedure. The increases consistency with previous literature.
4. Removal of the consensus value contributions, as these were found to be highly correlated with the between-laboratory uncertainties, misrepresenting the overall uncertainties.
In concert with these changes, the text has been updated to reflect the new MCMC outputs and the figures have been updated accordingly. While the decomposition of the uncertainties has changed, the overall conclusions of the manuscript remain roughly the same. The reframing of the problem allows for better comparisons to the literature (for instance, with Panteliadis et al. due to similarities with ISO, and with Conrad and Johnson who only assessed repeatability, albeit for a range of conditions).
The change in the statistical model had a minor impact on the overall reproducibility:
EC: 17.0 % > 16.5 %
OC: 17.4 % > 8.0 %
TC: 15.4 % > 12.1 %
OC was most impacted. This change resulted from not relying on the laboratory-reported uncertainties and better accounting for the lack of systematic errors in the measurements.We have also added calculations for EC/TC ratios, including notes on incorporating correlation between the EC and TC measurements, to the body of the manuscript and have added a comparison to an application of the ISO 55725-2 method to the Suppemental Information.
The above changes have not been marked in the manuscript, as this would mask other changes specific to the reviewer’s comments. However, this amounts to substantial changes to Sec. 2.2, 3.1, and the latter portion of 3.2.
The data will also be added as Supplemental Information.Citation: https://doi.org/10.5194/amt-2024-1-AC3
Viewed
HTML | XML | Total | BibTeX | EndNote | |
---|---|---|---|---|---|
383 | 70 | 24 | 477 | 16 | 21 |
- HTML: 383
- PDF: 70
- XML: 24
- Total: 477
- BibTeX: 16
- EndNote: 21
Viewed (geographical distribution)
Country | # | Views | % |
---|
Total: | 0 |
HTML: | 0 |
PDF: | 0 |
XML: | 0 |
- 1