Introduction

Regulation

Air pollution is widely accepted as a major contributory factor in respiratory and other medical conditions [1,2,3,4], along with the widely accepted negative consequences for the environment [5,6,7]. Regulation from governments is therefore in place to control the release of hazardous substances to the air, e.g. the Industrial Emission Directive, IED [8], for industrial emissions.

The IED sets emission limit values (ELV) for each pollutant and requires that industrial sources like large combustion plants (LCPs) and waste incineration plants (WIPs) monitor their emissions to water and air to enforce these limits. All LCPs with thermal capacity ≥ 100 MW must be fitted with an automated measuring system (AMS) to continuously provide emission monitoring capability. As regulation, the IED sets uncertainty requirements that the AMS has to meet to ensure the measurements are suitable.

Standardisation framework

A framework of international standards provides procedures to control how an AMS is run to ensure reliable, accurate and consistent measurements. EN 14181:2014 (stationary source emissions—quality assurance of automated measuring systems [9]) describes the quality assurance procedures required to ensure that an AMS will be capable of providing data that meet the measurement uncertainty requirements in the IED.

The quality assurance framework is split into three quality assurance levels (QALs) covering: the suitability of the AMS for the intended measurements (QAL1), the calibration of the AMS (QAL2) and the control of its ongoing operations (QAL3). The standard also outlines an annual surveillance test (AST) which fits between QAL2 and QAL3.

To meet the suitability requirement (QAL1), the AMS manufacturers have to demonstrate that it conforms to the requirements in EN 15267 [10,11,12,13], through both laboratory and field testing. The plant operator is then required to ensure that a suitable conforming AMS is chosen. The AMS must be installed within a suitable location as defined by the manufacturer and according to EN 15259 [14], in order to ensure that the AMS will provide measurements with a total uncertainty that will meet the requirements of the IED.

Calibration (QAL2) is carried out by specialist testing laboratories following the procedure in EN 14181. The AST is a periodic check of the calibration that is also carried out by testing laboratories. The operator is responsible for implementing the procedures for monitoring drift and precision (QAL3) as laid out in EN 14181.

The AMS must be calibrated after being installed at the measurement location, but assuming it remains stable (passing QAL3) and the AST each year demonstrates that the calibration function is still valid, the AMS only needs to undergo another QAL2 if the plant or instrument conditions change (e.g. maintenance of the AMS, change in plant fuel, etc.) or up to once five (three for WIP) years have passed since the last calibration, as required by EN 14181.

EN 14181 is the main standard relating to QA/QC for an AMS, but this is far from stand-alone.

As already mentioned, the QAL1 testing relies on the methodology in EN 15267, together with the guidance for calculating total uncertainty from EN ISO 14956 [15]. EN 15259 sets requirements for locating the AMS to ensure the site will be suitable.

The AMS is calibrated by comparison with a reference method. This can be a standard reference method (SRM), which will be described in a species-specific standard (e.g. EN 14791:2017 [16] for sulphur oxides; EN 14792:2017 [17] for nitrogen oxides; etc.), or an alternative method (AM). Any AM has to demonstrate its equivalence to the appropriate SRM, according to the methodology in EN 14793:2017 [18].

Requirements of EN 14181

The QAL1 procedure is designed to ensure that the chosen AMS is suitable for making the required measurements, is installed correctly in a location that meets the measurement requirements and can achieve the measurement uncertainties required by the legislation (i.e. the IED).

QAL2 involves making parallel measurements with the AMS and a reference method. The results are used to create a calibration function for the AMS, fitting its results to the reference method. A minimum of fifteen parallel measurements taken over a minimum of 3 days are required to ensure that the calibration function is representative. The valid calibration range of the instrument is set based on the range of measurements made during the QAL2, to demonstrate that it is suitable. The QAL2 also incorporates a number of functional tests (e.g. linearity) to validate the correct operation of the AMS.

QAL3 is a periodic testing to make sure the AMS stays within its specifications during operation. It involves measurement of zero and span samples which are compared with the previous QAL3 measurements to identify drift and/or loss of precision over time. Control charts are used to track the results and assess when action is necessary. The required frequency of these tests is based on the QAL1 results which identify the maximum allowed interval to ensure that drift is detected while maintaining measurement uncertainty levels within the prescribed limits.

The IED requires an AMS to be compared to a reference method once per year, but to reduce operational cost and disruption an AST can be used rather than a full QAL2. Functional tests are still carried out, but only five parallel measurements are required since an AST is only checking that the current calibration function is still valid. This means a full QAL2 is only required every 5 years or when the AMS fails the QAL3 or AST.

In summary, if an AMS meets all the requirements of EN 15267-3 and is run according to procedures in EN 14181 with regular QAL2, QAL3 and AST, it can be assumed by the operator that the AMS will fulfil the uncertainty requirements under the IED.

The application of EN 14181

Given this assumption on AMS performance, it is important that the processes in EN 14181 have been thoroughly tested and validated. Some efforts have been made to investigate the uncertainty of aspects of the process (e.g. QAL2 [19]; QAL3 [20]), while Graham [21] looked at industry experience implementing the standard. In particular, Graham investigated the performance of two forms of control chart included in EN 14181, suggesting modifications to reduce false positives and make plotted results clearer for the user. While QAL3 can be automated in some AMS, Graham cautions against complete automation as it can be difficult to detect faults, for example if the pressure is getting low in the reference gas cylinder. Direct injection of reference gases can also lead to problems as it will not test the sampling system, with undetected leaks diluting measurement samples when in normal operation.

Modelling EN 14181

NPL has modelled a gas analyser operating under the quality framework of EN 14181 in order to investigate the propagation of uncertainty, using Monte Carlo simulation (MCS) techniques. Particular attention has been paid to the effects of high uncertainty on the reference measurements. The AMS is calibrated to this, so the reference method is considered free of bias, which might not be the case.

To model the entire process of an instrument measuring over the lifespan of a calibration function, i.e. 5 years, is immensely challenging. In particular, to deal with variations in method encountered when variables change, the model had to implement branching decision trees to allow different model repeats to follow alternative methodologies, something not found by the authors in the literature.

As an example, during the QAL2, parallel measurements are made with the AMS and SRM, but these measurements are not required to be at set concentrations so will vary with the gas stream. Depending on the distribution of the measurements, the calibration function can be calculated in one of three ways. Well-distributed data over the full measurement range allow the testers to use method A, while methods B and C cope with clustering of data, C for low-level clusters with all values < 15 % of ELV and B for any higher-level clustering. The model includes a decision tree structure, so such variations can occur within the larger process (Fig. 1).

Fig. 1
figure 1

Flow chart for the calibration procedure (QAL2), with three methods for generating the calibration function for the instrument based on the suitability of the data used in the calibration, indicating the requirement to implement branching decisions in the MCS

Full size image

Methodology

Model principles

The model represents all the processes involved in running an AMS operating under EN 14181, i.e. the measurements and the quality assurance regime. The model simulates the processes over a period of up to 5 years, providing information on the uncertainty in the reported values in that period. Each measurement made with the AMS will have associated uncertainties, which are all randomly generated according to MCS and the expected uncertainties for the individual variables. Table 1 summarises the uncertainty variables included in the model.

Table 1 List of the uncertainty variables within the MCS model
Full size table

The model has been created using the R programming language [22] which is well suited to the statistical requirements of MCS.

Input data and assumptions

Measurement data for an actual site can be provided as input data for the model. This can be used as the true emissions on which the perturbations will be applied to initiate the MCS. This allows the model to simulate appropriate levels of emissions for the relevant application being investigated.

When an AMS is tested to EN 15267-3 as part of QAL1, it is characterised for a variety of performance-related areas to check it will meet the minimum required standards. The performance characteristic certificate for the AMS therefore contains the majority of measurement uncertainty variables to be used by the model to generate appropriate perturbations in the MCS (e.g. zero and span drift rates, repeatability, etc.).

The model is based on the real use of a gas analyser instrument under EN 14181, but, like any model, it has to include some assumptions in order to function. Some aspects of the testing under EN 14181 are assumed and not specifically modelled (e.g. the linearity test that is part of the AST). This can be justified since they relate to constant sources of error which will not drift over time, so the offset will be included within a standard calibration. The model takes 30 min average measurements, assuming that there is no downtime, including timeless quality assurance testing, so every measurement is made regardless of procedural requirements (e.g. running AST). This is obviously a major abstraction compared to real field measuring where downtime can represent ten per cent or more of a data set, but it is a justified assumption as it has no effect on the uncertainty of the AMS results that are covered within the measurement period.

The ambient temperature data used in the tests are simulated, based on a sine curve to represent seasonal variation. The model does not include other effects of weather since instruments are generally deployed in some form of enclosure so would not be influenced in this way. Shewhart and exponentially weighted moving average (EWMA) control charts do not allow adjustments to be applied when QAL3 tests are failed, so the model only implements the CUSUM control charts that reduce the need for additional calibrations by making adjustments when drift is detected.

Validation

During development, each part of the model was tested with simplified artificial data sets. Initially, these kept everything constant to ensure that the functions were performing correctly before testing with more complex data. Testing was done independently for each section of the model and repeated after integration of multiple sections to verify operations were still successful. The full model was run with various scenarios for which the correct emission values had been calculated to demonstrate that the model was functioning correctly.

In particular, the QAL2 module was tested by running the model with data from real QAL2 tests to check it would select the correct algorithm and generate the expected calibration function. Results were consistent across all repeats and matched the calibration function from the QAL2 the data were from, thus verifying the ability of the model to accurately reproduce real-world results.

QAL2 failure rates as a function of SRM uncertainty

SRM uncertainty could potentially be a major issue since the reference method is assumed to be free of bias and able to provide a true indication of the actual conditions being measured by the AMS. To investigate this effect, the QAL2 module was run with all uncertainty values set to zero except SRM measurement uncertainty. The model was repeatedly run with the same data, varying only the SRM uncertainty to demonstrate the likelihood of this causing a failure of the QAL2 test. Part of the QAL2 is a variability test to check whether the differences between the AMS and SRM are within the maximum permissible uncertainty. If this test does not pass, the QAL2 has been failed and the parallel measurements must be repeated. As uncertainty on the SRM measurements increases, so too does the chance that the AMS and SRM measurements will diverge far enough to fail a QAL2, even with a perfect AMS instrument. The model counts the number of attempts required to pass the QAL2 in each repeat. In general, this should be one unless the error is sufficient to generate failures. The model was run with 1000 Monte Carlo repeats, with the number of attempts required to create a valid calibration function in each repeat being recorded.

Results and discussion

Fabricated data for model validation

For these tests, all measurement uncertainties were set to zero, except the variable being tested, which was run with uncertainty set at 1 % intervals from zero to 25 %. Each run consisted of 1000 MCS repeats to produce representative results. Fifteen measurement values were set to evenly cover the measurement range, which is the ideal situation since clustering of the data can lead to the generation of inappropriate calibration functions and therefore failed QAL2 tests.

Validation of model

The SRM uncertainty can only affect the QAL2 and AST result (i.e. the calibration function). The overall uncertainty should vary along with any increase or decrease in SRM uncertainty due to the relative quality of the calibration function. Table 2 indicates overall uncertainty as the standard deviation of all the MCS results. As expected, the overall uncertainty increases along with SRM uncertainty due to increasingly poor calibration functions being created.

Table 2 Results from testing the effect of varying SRM uncertainty on the overall uncertainty of annual emissions
Full size table

The overall uncertainty will only vary between years due to changes in the calibration function (i.e. after a failed AST). At low SRM uncertainty values, the AST should not fail with an error-free AMS, and the results in Table 2 show no year-to-year variation up to 11 % SRM uncertainty. Above 15 %, the annual uncertainties are varying every year due to the increasing likelihood of failing the AST and having to recalibrate.

This combination of checks demonstrates any problems with the modelling of the SRM uncertainty value. This approach of isolating individual variables for testing was then used to validate the remaining uncertainty parameters.

Application of model

Due to the ability to independently adjust variables, it is possible to model the effects of any potential changes before they are implemented in practice. For example, a proposition to alter the frequency of QAL3 testing could be investigated with the model, indicating the effect on overall measurement uncertainty that any increase or decrease in testing would cause. Increasing frequency would improve uncertainty by detecting issues more quickly, but would lead to increased AMS downtime and costs for the operator who is likely to want a reduced burden of testing. The model would indicate if total uncertainty requirements could be maintained with fewer tests.

Suitability of new AMS systems could be assessed quickly and cheaply using the model, based on the product conformity certificate and the current plant data. This would indicate the expected performance of a potential new system, allowing the operator to understand the long-term effect of such a decision before investing.

The model could also be used to aid AMS manufacturers in the development and evolution of new instruments. A sensitivity analysis run in the model could identify the most important contributions to overall measurement uncertainty, allowing the development of new systems that optimise those variables.

QAL2 failure rates as function of SRM uncertainty

As previously mentioned, the SRM uncertainty is vital to the QAL2 process. If the SRM uncertainty is high, it could result in the SRM values deviating too far from the AMS, leading to a failed QAL2, even if the AMS is performing well. Allowable uncertainty for the SRM varies by species, but for SO2 it can be up to 20 %. To investigate if false failures are likely to be occurring, the authors have tested the failure rates for QAL2 in the model with different levels of SRM uncertainty and a zero-error AMS, so any failures must be due to the SRM uncertainty.

In reality, a certain level of failures is expected, so an acceptable false failure rate of 5 % was adopted. The MCS of the QAL2 process was run with 1000 repeats for SRM uncertainty values from 10 to 20 %, in 2 % intervals. The number of attempts to successfully generate a calibration function was recorded to demonstrate the failure rates (Fig. 2).

Fig. 2
figure 2

Failure rates for QAL2 tests with varying levels of SRM uncertainty. Above 15 % SRM uncertainty, a significant number of failures start to occur, and at the 20 % SRM uncertainty limit 43.7 % of QAL2 tests result in failure with 17.6 % requiring 3+ attempts to pass the QAL2

Full size image

The results show that when the SRM uncertainty rises to 16 % or higher there are a significant number of QAL2 failures purely due to the SRM performance. Failed QAL2 testing has significant impacts for the operators as they have to pay for additional testing and even potentially unnecessary repairs to the AMS. Since the SRM is assumed to give an accurate measurement, calibrating to it when there are high SRM uncertainties will lead to inaccurate AMS data unknowingly being recorded until an improved calibration function is generated, potentially greatly reducing data quality.

Conclusions

The validation process demonstrated that the model was functionally correct and generating realistic levels of overall uncertainty.

The model results demonstrate the effect of SRM uncertainty, illustrating that even if the AMS is perfect the current limits are not sufficient to ensure that QAL2 testing will be completed successfully. A true AMS with realistic uncertainties would expect even higher failure rates, so continuing with an uncertainty limit as high is not acceptable. Limiting SRM uncertainty to 15 % would ensure that no AMS system would be likely to fail a QAL2 purely because of testing with a poorly performing SRM, resulting in significant savings for plant operators from avoidable QAL2 tests.

In reality, you do not see the rate of QAL2 failures demonstrated by the model for the highest SRM uncertainties. This indicates that the reference methods being used at present are already achieving uncertainties well below the 20 % limit value. The model results demonstrate that to achieve similarly low failure rates the SRMs that are currently in use must be operating at better than 15 % SRM uncertainty so would not be affected by any reduction in the limit value to that level.

The model can demonstrate that EN 14181 is capable of controlling the measurement uncertainty to within the limits required by the IED. Additionally, it can test the effect on overall uncertainty of any changes to the process before they are implemented in the field to ensure that EN 14181 continues to meet the requirements. Operators could use the model to test alternative AMS systems to ensure they will be suitable, thereby reducing the costs involved in implementing and maintaining measurement systems to meet the needs of current and future legislation.