Introduction

A qualitative property is one that cannot be described meaningfully by a number, for example, biological or chemical species, nucleic acid or protein sequence, or a classification such as “hard” or “soft” for drinking water. The concept of “qualitative” properties includes those that can be expressed on either a nominal scale (where values can only be equal or unequal) or ordinal scales, on which values can be meaningfully ordered. The number of reference materials (RMs) including certified reference materials (CRMs) with certified qualitative properties offered by reference material producers (RMPs) has increased steadily over the past years. The Technical Committee of the International Organization for Standardization, ISO/TC 334 (formerly ISO/REMCO), therefore proposed in 2018 to develop an International Standard providing guidance on the production of this type of RMs. In 2024, this has led to the publication of ISO 33406 Approaches for the production of reference materials with qualitative properties [1]. This paper gives a brief overview of the content of ISO 33406 and its relationship to other standards for RM production.

Need for guidance on the production of qualitative RMs

ISO 17034:2016 General requirements for the competence of reference material producers [2] specifies general requirements that RMPs must meet when producing RMs. ISO 17034 addresses RM production in general, including the additional production requirements for CRMs. According to ISO/TC 334, RMs and CRMs can have quantitative or qualitative property values, including the option to have both types of values [2]. However, the main emphasis of ISO 17034 lies on RMs with quantitative property values and on approaches applicable to both RMs with quantitative and RMs with qualitative property values. Specific information related to the production of RMs with qualitative values is limited. While examples for RMs and CRMs with qualitative property values are already available [3], the wide range of materials makes it difficult to set mandatory technical requirements. ISO/TC 334 accordingly opted to write an International Standard providing guidance for the production of RMs with qualitative property values, as opposed to setting requirements, in the same way that ISO 33405:2024 Reference materials – Approaches for characterization and assessment of homogeneity and stability [4] provides guidance but does not add mandatory requirements beyond those of ISO 17034. Consequently, ISO 33406 does not use the term shall, which would refer to a mandatory requirement, but uses should or may to indicate that the standard provides recommendations or options (permission), respectively. Details on the use of shall, should and may by ISO is available in [5].

Scope and positioning of ISO 33406

ISO 17034 outlines the general requirements for RMPs and notes that further guidance is given in ISO 33401:2024 Reference materials – Contents of certificates, labels and accompanying documentation and ISO 33405. Approaches outlined in these documents, which were developed by ISO/TC 334, meet the requirements of ISO 17034, but alternative ways may exist as well [6]. Figure 1 illustrates the relationship of ISO 33406 to ISO 17034 and ISO 33405.

Fig. 1
figure 1

Relationship between ISO 17034, ISO 33405 and the new International Standard ISO 33406

ISO 33406 specifically provides guidance on the production of RMs having one or more assigned qualitative property values. As can be seen from Fig. 1, it builds on the requirements established in ISO 17034 and the additional guidance given in ISO 33405. More specifically, ISO 33406 describes where the concepts of traceability and uncertainty of qualitative property values address characteristics similar to those used in the metrology of quantitative properties, provides guidance on how to express uncertainties, and on how to establish traceability (both metrological and documentary (i.e. based on documents proving provenance), as appropriate) for qualitative property values.

Terminology

Most of the RM terminology in ISO 33406 uses terms from ISO 17034, ISO Guide 30:2015 Reference materials – selected terms and definitions [7] and ISO 33401:2024 Reference materials — Contents of certificates, labels and accompanying documentation [8]. General metrology terms follow ISO/IEC Guide 99 (the “VIM”) [9]. Therefore, all four are normative references to ISO 33406.

In the absence of an agreed international definition, ISO/TC 334 decided to use qualitative property (without formal definition) as superordinate for nominal and ordinal properties, with categorical properties as synonym, following usage common in the statistical sciences [10]. For RMs, nominal properties are a particular kind of qualitative property, allowing identity or difference as the only meaningful comparisons between their values. In contrast to this, ordinal properties have values that can be ordered (i.e. ranked) from smallest to largest or from lowest to highest, but for which arithmetic operations (like subtraction or division) are not meaningful, even when their values are represented numerically. An example is cancer staging where roman numbers between I and IV are assigned to indicate the extent to which a cancer has grown and spread.

Characterization—Knowledge about the origin of the material

One way to characterize an RM with qualitative properties is to use knowledge of the origin of the material. This is also called material provenance, used here to refer to the origin and/or place of origin. To support characterization based on provenance, the RMP should obtain documentary or other evidence about the origin of the material that allows tracking the material from the final RM packaging back to the origin.

Procedures implemented by the RMP should cover handling of the material, prevention of contamination by other materials, and replacement of the material by any other.

Ideally, characterization is based on documented information to confirm the knowledge of the origin of the material, supported by additional evidence to confirm identity of the material. For biological materials, provenance can include evidence of parentage or continuous culture from an authentic specimen. In the case of value assignment based on provenance, the RMP should maintain and make available documentation of the provenance of the material, including the source of the material, changes in ownership, and documentation also about the assignment of value, including the source of the material. Procedures need to be in place to avoid contamination.

Characterization—Identity based on measurements

Some qualitative characteristics are determined by measuring quantitative properties of a material and classifying the material based on the results. For example, drinking water can be classed as “hard” or “soft” based on measurements of calcium and magnesium (as carbonate) [11]. Whenever characterizing the identity of a material based on measurements, the aspects outlined in ISO 17034 and ISO 33405 need to be considered. Characterization can be based on measurement results from one or several methods targeting reliable, identifying characteristics. For example, identity of a pure organic compound of small-to-medium molecular mass can be based on measurement of properties such as melting point, CHN analysis and spectroscopic data on line frequency and intensity from nuclear magnetic resonance (NMR) and infrared spectroscopy. Differently from RMs with quantitative properties, slight heterogeneity and instability of a quantitative material does not necessarily change the assigned identity. For example, changes in the level of dryness do not change the species of a plant used as an RM. Where applicable, a suitable assessment or statement of purity of the RM should be made. However, where the identification methods used do not yield unique, characteristic measured values, knowledge of the source of the raw material is vital for the certification of identity.

Characterization—Identity based on qualitative tests and expert opinion

Identity is sometimes determined by expert judgement, usually based on observations of characterizing properties, or by comparison with a paradigmatic exemplar. Likewise, some RMs or reference data are recognized by the relevant community of users. Plant or animal species are often determined by comparisons with voucher specimens.

For defined chemical substances of small-to-medium molecular mass, a combination of several methods should be chosen that probe different properties of the candidate RM. Together with information on the raw material and its processing steps, the sampling and transport to the RMP, the selected methods should establish the identity of the material. If detailed published criteria (e.g. pharmacopeia criteria for identification) exist, the choice of methods can be restricted to those listed.

As stipulated in ISO 17034, all test and measurement procedures used must be properly validated according to ISO/IEC 17025:2017 General requirements for the competence of testing and calibration laboratories [12] and the results must fulfil the requirements for metrological traceability. Examining control materials together with the RM helps to ensure that characterization methods were applied correctly. Also, comparison with published procedures is useful. In the ideal case, results agree with the published criteria, and the identity of the RM can be established with high confidence.

A range of statistical approaches can in principle be used in value assignment to RMs with qualitative properties. ISO 33406 does not list all of them, and further guidance can be found in [10].

Metrological traceability and other references in qualitative determinations

The concept of metrological traceability is well understood in the case of measurement of quantitative properties, and the requirements for establishing metrological traceability in such cases are given in, for example, ISO/IEC 17025. These requirements also apply for tests, based on measurement, that yield a qualitative result, and to any conditions of measurement or testing that are specified in the documented procedure. For example, a test for flammability might specify a particular temperature or humidity or specify sample pre-treatment conditions such as drying times. These conditions are expected to be maintained using suitably calibrated equipment. ISO 33406 recognizes this and applies the same provisions for any quantitative determinations or conditions used in assignment of a qualitative value to an RM. Whenever a qualitative value assignment depends upon measurements or tests that include quantitative conditions, ISO 33406 recommends that these measurements and/or tests should meet the metrological traceability requirements of ISO/IEC 17025, and a statement of metrological traceability relating to the relevant measurements and/or tests should be included in the documentation accompanying the RM.

Value assignment for qualitative properties can depend on comparison with reference data or with another RM already value assigned for the same qualitative property (such as an authentic exemplar of a herbal species). The concept of metrological traceability does not apply to this kind of comparison. Here, ISO 33406 adds a new recommendation; any reference used for such comparison should be stated in the documentation accompanying the RM if the value assignment depends critically upon the comparison. By “critically” ISO 33406 means that the comparison is essential to the value assignment. The recommendation parallels recommendations for statements of metrological traceability that require identification of the principal calibration standard, but not (for example) of any quality control materials used [13].

Measurement uncertainty and confidence in qualitative values

Every assignment of value to a property is surrounded by uncertainty and it is important to provide users of these RMs with guidance on the reliability of the assigned value. For quantitative properties with numerical values, the techniques for measurement uncertainty evaluation are well defined and described in ISO/IEC Guide 98–3 (the “GUM”) [14]. However, these techniques are not applicable to the evaluation and expression of uncertainties associated with the value assignment of qualitative properties. Additionally, the meaning of “uncertainty” as it applies to values on ordinal scales or to nominal values is not well established. ISO 33406 accordingly provides separate recommendations for addressing measurement uncertainty associated with any intermediate measurements or conditions, and for conveying the reliability of the assigned qualitative value.

If the assignments of the qualitative property depend on measurement of one or more quantitative values, the RMP’s confidence in the assigned qualitative values is impacted by the control of uncertainties in test conditions and the uncertainty related to intermediate measurements. ISO 33406 accordingly recommends that the RMP should ensure that measurement uncertainties are minimized to have no significant impact on the confidence in the assigned qualitative value, either through controlling test conditions within tolerances, by demonstrating that the uncertainty does not influence the test outcome significantly, or by a combination of both of these actions.

When assigning a qualitative value based on comparing a measured value with one or more limits, the RMP can use the associated measurement uncertainty, together with assumptions about the associated probability distribution, to estimate the probability of false classification. This can then be used to demonstrate that the uncertainty associated with the measured value(s) has no material impact on confidence in the resulting classification.

Turning to the case of uncertainty or confidence in qualitative assigned values, terminology has yet to be firmly established. The term “measurement uncertainty” cannot currently be applied to qualitative values, as it is specific to measurement of quantities. ISO 33406 accordingly uses the term confidence to refer to the degree of belief that the RMP has in the value assigned to a qualitative property. The general principle in ISO 33406 regarding confidence in the assigned value is that the RMP “should provide sufficient information with the RM to allow the user to assess whether the confidence claimed for the assigned value meets their needs”. Because this is a new area for RM standardization, ISO 33406 includes additional guidance on the nature of such information.

The state of the art in expressions of confidence for qualitative RM values is not yet well advanced, though some examples exist (see, for example, example 5D in [15]). Recognizing the limited state of the art, ISO 33406 does not require quantitative statements of confidence in assigned qualitative values. It does, however, permit RMPs to include such statements where appropriate. Such a statement can be, for example, a likelihood ratio or a probability distribution on the set of possible values of the qualitative property. It can also use confidence measures derived from false response rates, such as sensitivity and specificity, often referred to as diagnostic sensitivity and specificity [16]. ISO 33406 includes an annex providing substantial information about many of these possibilities. Further information can also be found in the Eurachem/CITAC Guide Assessment of the performance and uncertainty in qualitative chemical analysis [17].

ISO 33406 also permits confidence to be expressed qualitatively, using an ordinal scale (e.g. most confident, very confident, moderately confident), provided that the scale is explained to the end user.

Finally, ISO 33406 includes a list of additional information that can be used to help the end user understand the confidence in an assigned qualitative value. In addition to quantitative and qualitative indications of confidence above, the information can include the basis for the assigned value; information on the origin of the material; information on preparation procedures and contamination control; information on the performance of relevant test methods; and information on any reference data used.

Provisions on homogeneity and stability testing

Homogeneity and stability are as important for qualitative RMs as for RMs with quantitative properties. However, in contrast to the well-established range of statistical approaches for value assignment, homogeneity testing, stability testing and commutability assessment for quantities, few statistical approaches have been widely applied to the characterization of qualitative properties. ISO 33406 accordingly includes guidance on a range of experimental approaches and data analysis methods to support homogeneity and stability assessment for RMs with qualitative properties.

When selecting statistical approaches for such RMs, ISO 33406 recommends that RMPs prioritize experimental methods that offer quantitative information over qualitative tests, as they provide more detailed and informative results. Additionally, reliable test methods should be prioritized to differentiate between test process failures and actual defects or changes in a candidate RM, especially when qualitative test methods have an appreciable false response rate. Furthermore, statistical analysis of qualitative test results should be chosen to accommodate low or zero proportions of discordant responses, as RMs are expected to be homogeneous and stable.

Assessment of homogeneity

According to ISO 17034, the RMP must assess the homogeneity of the RM in its final packaged form to ensure its fitness for purpose. The RMP needs to address between-unit homogeneity for batch certification and within-unit homogeneity, if the user is permitted to take multiple sample aliquots from the RM unit. Within-unit homogeneity is usually addressed by giving a minimum sample size. A general exception is a certified qualitative property individually assigned for every unit of an RM; in this case, it is not necessary to perform a further test of the between-unit homogeneity for that property.

ISO 33406 recommends that homogeneity of the RM should be assessed for all the assigned property values as per the requirement of ISO 17034. Homogeneity of the assigned qualitative property value(s) in an RM should be assessed as response to test methods directly targeting the assigned nominal property value(s) and to any other methods listed in the intended use section of the RM document. This is because the end user is generally reliant on consistency of response of their test methods, and the response can be affected by (for example) the distribution of potential interfering species in a material.

ISO 33406 gives extensive guidance on experimental designs for homogeneity studies. Where it is possible to study homogeneity using quantitative methods, ISO 33406 refers RMPs to ISO 33405, which provides general experimental designs and sampling strategies for homogeneity studies. For homogeneity studies using a test method giving a qualitative result, ISO 33406 provides additional guidance.

A number of factors need to be considered in planning a homogeneity study for a material with a qualitative assigned value. Perhaps the most important is the number of RM units for study. For homogeneity studies using a test method giving a qualitative result, the number of RM units studied needs to be increased, relative to the number required for quantitative assigned values, to provide a sufficiently narrow confidence interval for the estimated proportion of defective units. A second factor is the reliability of the chosen test method. Where the chosen test method is known to have an essentially negligible false response rate, replicate testing may be omitted. This strategy implies that between-unit inhomogeneity cannot be distinguished from within-unit inhomogeneity or from chance failures in the test method, unless additional information on, for example, test method variability is available. Finally, the minimum number of units to be sampled can depend on the type of criteria for providing the qualitative test results; that is, whether the criteria are quantitative (such as comparison of a measured value with limits) or qualitative.

For RMs certified for qualitative properties, ISO 33406 recommends the number of units chosen for the homogeneity study should be set based on sampling guidance for inspection by attributes as described in the ISO 2859–1:1999 Sampling procedures for inspection by attributes — Part 1: Sampling schemes indexed by acceptance quality limit (AQL) for lot-by-lot inspection [18], ISO 2859–4:2020 Sampling procedures for inspection by attributes — Part 4: Procedures for assessment of declared quality levels [19] or similar guidance. It is, however, noted that these standards generally suggest a very large number of units if the intended proportion of defective items is very low. Because of this, experimental homogeneity studies carried out on RMs with assigned qualitative properties are likely to be limited to a check for unexpected gross heterogeneity.

Statistical approaches for homogeneity assessment for qualitative properties

Two general statistical approaches can be applied for homogeneity assessment for qualitative properties. Either it is confirmed that the proportion of defective RM units is below a chosen threshold, or sufficient replicate testing is carried out to provide a check for a significant between-unit difference in the probability of non-compliant responses. Either approach can be used to calculate an upper confidence limit for the proportion of non-compliant RM units.

An example of the first of these approaches is the use of inspection by attributes as described in ISO 2859–1 and ISO 2859–4 making use of an acceptance quality limit, specifying the worst tolerable process average (for the proportion of defective units) and an inspection level, recommended to resemble the normal quality. The second approach can use well-established statistical tests such as the chi-squared test, and ISO 33406 gives a procedure for applying a chi-squared test for homogeneity assessment. A procedure is also given for determining an upper confidence limit for the proportion of defective RM units. References are also given for CATANOVA [20] and ORDANOVA [21], which can be used for nominal and ordinal values, respectively.

Assessment of stability

According to ISO 17034, the stability of the RM with respect to the assigned qualitative value(s) given on the RM document must be assessed. The RMP needs to assess the short-term stability of the RM to guarantee that suitable transport conditions are selected, and the RM’s long-term stability to ensure appropriate storage conditions and the stability during the use by the end user. The RMP also needs to provide a period of validity for the qualitative value as well as instructions for use.

The stability of qualitative values in an RM is assessed by evaluating the stability of responses to test methods targeting those values. This can be done by measuring the change in the proportion of units responding correctly to qualitative test methods, or by studying quantitative properties that indicate material degradation. The outcome of stability studies should allow the RMP to make an informed recommendation about the stability of the assigned properties of the RM during the period of validity and to give instructions for handling and use of the material on the RM document.

Experimental designs and statistical approaches for stability assessment for qualitative properties

The simplest experimental design for assessing the stability of quantitative or qualitative properties is to expose a large number of units to planned storage conditions and to test a fixed number of units at intervals. ISO 33405 refers to this as a classical stability study. An alternative is the isochronous design, in which testing is deferred until the end of the study, after moving RM units periodically to reference conditions. Both the classical and isochronous design provide an estimated proportion of failed units at each of a series of times. An alternative experimental stability assessment approach can be used if the RM units can be measured non-destructively.

ISO 33406 lists several statistical approaches for assessing the stability of qualitative properties. One approach involves using a contingency table, in a similar manner to that for homogeneity studies, adapted to test for differences in failure proportions at different time points; this would indicate potential instability. It is important to note that significant results from this test only indicate differences between groups and do not consider ordering information or interval size for different time or temperature points; ISO 33406 accordingly recommends additional inspection for any trends in proportion of units failing.

More sophisticated approaches are also suggested. One is the application of logistic or probit regression to counts or proportions of failed units or to individual failures, and guidance is given on the choice between the two. Finally, survival analysis, also known as reliability analysis in engineering, can be used to study cumulative failures over time. Survival analysis typically requires the use of methods that can cope with censored observations, which occur when an RM unit has not failed by the end of a stability study period, resulting in a failure time “greater than” the study duration [22]. ISO 33406 gives a number of survival analysis models that can be useful for stability studies of qualitative RMs.

Commutability assessment

The commutability of an RM relates to the ability of an RM, characterized by a reference procedure, to act as a calibrator or quality control (QC) material for a second measurement or testing procedure applied to routine test materials. Commutability is important for qualitative property RMs that are intended for use as positive or negative controls in a qualitative testing procedure and where different test procedures can respond very differently to different types of test materials.

ISO 33406 recommends that a statement relating to commutability is provided where it is relevant to the end use. This should include sufficient information to enable the end user to judge whether the material is suitable for the intended use without additional qualification, or whether the end user needs to undertake further study. The statement is expected to indicate whether any commutability study has been undertaken; the results of any such study; and any differences between the RM and typical test materials that could adversely affect commutability.

To support the commutability statement, ISO 33406 provides general guidance on the need for commutability assessment and, particularly, the need for experimental study. It provides for a number of approaches to commutability assessment, including consideration of the physical, chemical and biological properties of the material; review of available literature; and experimental commutability assessment. For experimental studies, the recommendations depend on whether the RM is intended for use with test methods. When use with only one test method is envisaged, it is considered sufficient to show that the RM responds as expected to that procedure. For RMs intended to be used with multiple test procedures, consistent response to all intended test procedures is recommended.

Application-specific guidance

In addition to the general guidance above, ISO 33406 includes annexes covering two important application areas: nucleic acid RMs and protein RMs certified for sequence, biological species or (for proteins) identity of one or more proteins. The annexes include brief technical guidance on the main activities needed to ensure reliable sequence information or identity. For proteins, brief guidance is also given on database use and on data archiving.

Conclusion and future directions

The recently released International Standard ISO 33406 focusses on the production of RMs with one or more assigned qualitative properties. It supports and complements the general requirements given in ISO 17034, which RMPs must meet when producing RMs, by giving specific guidance related to the production of RMs with qualitative properties. In addition to guidance on value assignment, homogeneity and stability assessment and commutability assessment for RMs with qualitative property values, ISO 33406 shows how metrological traceability applies, how analogous relations to qualitative reference values and reference data should be treated, and how RMPs can evaluate and convey the reliability of qualitative property values assigned to RMs.

ISO 33406 has several informative annexes providing further information and explaining possible ways to express confidence in qualitative values and listing statistical procedures, which can be used in conjunction with RMs with qualitative property values. Examples of homogeneity and stability studies of RMs with qualitative properties are given and possible ways for how to produce DNA and protein RMs with assigned qualitative properties are outlined in greater detail.

ISO/TC 334 accordingly supports RMPs producing RMs with qualitative properties with the guidance provided in ISO 33406 and hopes to further boost and harmonize the production of such RMs. Depending on RMP’s experience in using ISO 33406 and on progress made in the field, it may be possible to include firm requirements as well as guidance in future editions of the International Standard.