FormalPara Key Summary Points:

Why carry out this study?

The performance of the commonly used nucleic acid amplification platforms for detection of Mycobacterium tuberculosis complex has not been extensively evaluated in Taiwan.

We hypothesized that different nucleic acid amplification platforms may perform differently in the diagnosis of pulmonary tuberculosis and in excluding non-tuberculous mycobacteria.

What was learned from the study?

The nucleic acid amplification tests overall had a relatively high sensitivity and specificity in detecting Mycobacterium tuberculosis complex.

Amplicor and Xpert had better performance than in-house PCR in excluding non-tuberculous mycobacteria.

Our results suggest that replacing in-house PCR with other nucleic acid amplification platforms might be of benefit in a real-world setting.

Introduction

Tuberculosis (TB) is an infectious disease caused by Mycobacterium tuberculosis complex (MTBC). Smear-positive pulmonary TB is the most infectious form of TB. The World Health Organization (WHO) announced that an estimated 10 million people had TB and 1.5 million people died from TB globally in 2020, making TB the second leading infectious killer after coronavirus disease 2019 [1]. Moreover, an estimated 3.9% and 21% of new TB cases and previously treated cases, respectively, had rifampin-resistant (RR) or multidrug-resistant (MDR) TB worldwide in 2015 [2]. However, TB incidence has decreased in Taiwan. The Taiwan Centers for Disease Control reported annual notified TB incidence of 72.5 and 33.0 per 100,000 population in 2005 and 2020, respectively. Meanwhile, a reported 2.0% of new TB cases and 7.0% of previously treated cases had RR or MDR TB in 2020 [3].

Nucleic acid amplification (NAA) assays are an important advancement in the diagnosis of TB. Previous studies have demonstrated that NAA tests have better accuracy than smear microscopy and a shorter turnaround time than culture [4,5,6]. Several NAA platforms are available in Taiwan, including Roche Cobas Amplicor Mycobacterium tuberculosis assay (Amplicor) (Grenzach-Whylen, Germany) and Xpert MTB/RIF assay (Xpert) (Cepheid, Sunnyvale, CA, USA). Amplicor targets the 16S ribosomal RNA gene for amplification and detection of MTBC. It can differentiate TB and non-tuberculous mycobacteria (NTM) but cannot detect drug resistance in TB. The Xpert assay amplifies the rpoB gene to detect MTBC and probe for mutations within the 81-bp rifampin resistance- determining region of the rpoB gene to detect rifampicin resistance. The diagnosis of pulmonary TB begins with identifying individuals suspected of having TB, followed by sputum examinations. NAA tests are usually performed in sputum smear-positive cases to exclude NTM in Taiwan. In 2013, the WHO issued a conditional recommendation that Xpert, rather than conventional smear microscopy and culture, may be used as the initial diagnostic test in all adults and children suspected of having TB [7].

The Taiwan guidelines for the diagnosis and treatment of TB recommend NAA tests together with smear and culture as the initial diagnostic tools in all presumptive TB cases [8]. However, the Xpert test is not yet covered by the Taiwan National Health Insurance system. This is probably because the Xpert assay is more expensive than the other NAA platforms available in Taiwan, even though the Xpert assay is increasingly being used, while use of Amplicor and in-house polymerase chain reaction assays (in-house PCR) is decreasing, at least in part because they are time-consuming and labor-intensive techniques. Moreover, the performance of Xpert has not been comprehensively compared to other NAA platforms in Taiwan.

We hypothesized that different NAA platforms may perform differently in the diagnosis of pulmonary TB and in excluding NTM. Therefore, the aim of this study was to extensively investigate the accuracy of NAA tests overall followed by comparing the performance between the commonly used NAA platforms in Taiwan, including in-house PCR, Amplicor and Xpert, in a real-world setting.

Methods

Study Design, Setting and Population

This multi-center, cross-sectional study was organized by Taipei Medical University-affiliated Wanfang Hospital (TMUWH) as the investigation center. Hospitals taking care of a relatively high number of TB patients throughout Taiwan were invited to participate in this study, including National Taiwan University Hospital, Taipei Veterans General Hospital, Taichung Veterans General Hospital, Ministry of Health and Welfare Taoyuan General Hospital (MOHW-TGH), Ministry of Health and Welfare Chest Hospital (MOHW-CH) and Hualien Tzu Chi Hospital (HTCH). The study was approved by the Institutional Review Boards and Ethics Committees of the participating hospitals (N201903076, 2019-11-007BC, N201903076, IRB108-269-B, IRB_201702013RIND, CE18193A). The need for informed consent was waived because the study was based on a retrospective electronic medical chart review.

Patients with presumptive pulmonary TB, defined as those who had both respiratory symptoms and abnormal pulmonary radiographic findings suggesting TB disease, and who underwent NAA tests ordered according to physician judgment from January 2017 to December 2018 at seven participating hospitals, were enrolled in this study. These patients were classified into a frontline NAA group and an add-on NAA group. Frontline NAA was defined as an NAA test requested before or concomitantly with smear microscopy as the initial diagnostic tool; add-on NAA was defined as an NAA test requested after obtaining the results of smear microscopy mainly to exclude NTM.

The recommendation in Taiwan was to collect three sputum samples for smear examination for acid-fast bacilli and culture for MTBC for patients suspected of having pulmonary TB although, in fact, some patients did not submit three specimens [8]. The NAA test was encouraged for rapid diagnosis of TB and was highly recommended for positive smear to exclude NTM. Meanwhile, the NAA was done using the same culture sample. Expenditures for all examinations were covered by the national health insurance. Moreover, all of the participating hospitals performed Ziehl-Neelsen stains for smear examination. Among the seven participating hospitals, TMUWH, MOHW-TGH, MOHW-CH and HTCH used both Löwenstein-Jensen medium and Mycobacteria Growth Indicator Tube (MGIT) for culture while the others used MGIT only for culture. Identification was made for any positive culture for mycobacterium to exclude NTM. However, species identification for NTM was not always performed. Particularly, two participating hospitals used Amplicor, three used Xpert, and two used both Xpert and in-house PCR tests.

Data Collection

The investigators reviewed and collected data using a standardized questionnaire from electronic medical charts. The collected data included demographic characteristics, type of specimen, date of submitting sputum, date that the results were reported, results of sputum smear microscopy and culture for mycobacteria, type of NAA platform, and the date and results of the NAA tests.

Outcome Assessment

The diagnostic performance (sensitivity, specificity and predictive values) of smear microscopy and the NAA tests in the detection of MTBC was assessed using at least one sputum culture positive for MTBC as a positive reference standard and using all sputum cultures negative for MTBC as a negative reference standard [9, 10]. To investigate the performance of the NAA tests in excluding NTM, we performed both patient- and specimen-based analyses to quantify the false-positive rate of the NAA tests in patients infected with NTM. In patient-based analysis, any positive NAA test in patients infected with NTM was classified as being false positive, even if the specimen that was tested positive for NAA and the specimen that was culture positive for NTM were different specimens. In specimen-based analysis, a positive NAA test in patients infected with NTM was classified as being false positive only if the specimen that was tested positive for NAA and the specimen that was culture positive for NTM were the same specimen.

Sample Size

We did not know the number of Xpert, Amplicor and in-house PCRs performed in the participating hospitals. Thus, we only estimated the total sample size of NAA tests required based on the advice of Bujang et al. [11]. Assuming the prevalence of TB among those who had NAA tests was 5%, the required sample size for 80% power to detect NAA with a sensitivity of 70% or 80% was 3100 or 2140, respectively. We thus planned to assess 4000 or more individuals with NAA tests to ensure that we had a sufficient sample size to assess the performance of the NAA assays.

Data Management and Statistical Analysis

Electronic data were managed using a password protected personal computer at TMUWH. Stata Version 15 (Stata Corp LP, College Station, TX, USA) was used for data analysis. Categorical variables were tabulated as frequency and percentage. We computed and compared differences and 95% confidence intervals (CIs) of the differences in sensitivity, specificity and false-positive rate among NTMs between the different platforms using the chi-squared test.

Results

Figure 1 shows that a total of 4271 presumptive TB patients underwent sputum NAA tests for the diagnosis of pulmonary TB at the seven participating hospitals from 2017–2018 throughout Taiwan. Of the 4271 patients, 4140 had sputum culture results and 131 did not (contamination). Of these 4140 patients, 4126 had valid NAA results and 14 did not. Thus, analysis of the performance of NAA tests was done for the 4126 patients with valid sputum culture and NAA results. Table 1 shows that, of these 4126 patients, 1935 (46.9%) were smear negative, 2161 (52.4%) were smear positive, and 30 (0.7%) had no smear results; 907 (22.0%) were culture positive for MTBC, 1100 (26.7%) were culture positive for NTM, and 2119 (51.4%) were culture negative; 1417 (34.3%) had valid frontline NAA results, 2648 (64.2%) had valid add-on NAA results, and 61 (1.5%) were unclassified because of a lack of smear results (n = 30) or because the dates of smear examinations were not recorded (n = 31). Of the 4126 patients, 879 (21.3%), 2756 (66.8%) and 552 (13.4%) underwent Amplicor, Xpert and in-house PCR tests, respectively; 61 underwent both Xpert and in-house PCR tests.

Fig. 1
figure 1

Patient enrollment flow chart at the participating hospitals throughout Taiwan. HTCH Hualien Tzu Chi Hospital, MOHW-CH Ministry of Health and Welfare Chest Hospital, MOHW-TGH Ministry of Health and Welfare Taoyuan General Hospital, NAA nucleic acid amplification, NTUH National Taiwan University Hospital, TCVGH Taichung Veterans General Hospital, TMUWH Taipei Medical University-affiliated Wanfang Hospital, TPEVGH Taipei Veterans General Hospital

Full size image
Table 1 Diagnostic performance of NAA tests compared with positive sputum cultures for Mycobacterium tuberculosis
Full size table

Overall, 20.8% (860/4126) of the patients had a positive NAA test result, including 17.7% (251/1471) of those with frontline NAA results, 22.4% (594/2648) of those with add-on NAA results, 3.9% (75/1935) of those with a smear-negative result, and 35.9% (776/2161) of those with a smear-positive result. The sensitivity and specificity of NAA tests overall were 83.2% (95% CI 80.8–85.7%) and 96.7% (95% CI 96.1–97.4%), respectively. There was no significant difference between frontline NAA and add-on NAA with overlapping 95% CIs (Table 1). The sensitivity of NAA among the smear-negative cases was 33.1% (95% CI 26.0–40.3%), and the specificity was 98.9% (95% CI 98.4–99.4%) (Table 1).

Among the 879 participants with valid sputum Amplicor test results, the sensitivity and specificity were 82.2% (95% CI 77.5–86.8%) and 97.1% (95% CI 95.8–98.4%), respectively. The sensitivity of frontline Amplicor was lower than that of add-on Amplicor (difference 20.2%, 95% CI 4.1–36.3%); however, the sample size of the frontline Amplicor group was relatively small. The sensitivity of Amplicor among the smear-negative cases was low (33.3%, 95% CI 20.0–46.7%) (Table 2).

Table 2 Diagnostic performance of Roche Cobas Amplicor Mycobacterium tuberculosis assay (Amplicor)
Full size table

Of the 2756 subjects with valid sputum Xpert results, 1575 (57.1%) were smear-negative, 1154 (41.9%) were smear-positive, and 27 (1.0%) had no smear results; 1151 (41.8%) had frontline Xpert results, 1553 (56.3%) had add-on Xpert results, and 52 (1.9%) were unclassified because of missing data. The sensitivity of the Xpert assay overall was 83.8% (95% CI 80.7–86.9%), with a specificity of 97.6% (95% CI 97.0–98.2%). The sensitivity and specificity between frontline Xpert and add-on Xpert were not significantly different with overlapping 95% CIs. The sensitivity for Xpert among smear-negative cases was only 32.1% (23.3–40.9%) (Table 3).

Table 3 Diagnostic performance of the Xpert MTB/RIF assay (Xpert)
Full size table

Of the 552 participants who had valid in-house PCR results, 133 (24.1%) were smear negative, 416 (75.4%) were smear positive, and 3 (0.5%) had no smear results; 182 (33.0%) had frontline in-house PCR results, 361 (65.4%) had add-on in-house PCR results, and 9 (1.6%) were unclassified because of missing data. The sensitivity for in-house PCR was 82.4% (95% CI 75.7–89.1%), and the specificity was 91.8% (95% CI 89.2–94.4%). The sensitivity of in-house PCR among the smear-negative cases was relatively low (50.0%, 95% CI 19.0–81.0%) (Table 4).

Table 4 Diagnostic performance of in-house PCR assay (in-house PCR)
Full size table

There was no significant difference in sensitivity between Amplicor and Xpert (difference 1.6%, 95% CI − 7.2–4.0%), between Amplicor and in-house PCR (difference 0.2%, 95% CI − 8.4–7.9%) or between Xpert and in-house PCR (difference 1.4%, 95% CI − 6.0–8.7%). However, the specificity of in-house PCR was significantly lower than that of Amplicor (difference 5.3%, 95% CI 2.4–8.2%) and Xpert (difference 5.8%, 95% CI 3.1–8.5%).

Of the 4096 patients with smear and culture results, 2161 (52.8%) were smear positive, of whom 731 (33.8%) were culture positive for TB and 880 (40.7%) were culture positive for NTM. In addition, 1935 (47.2%) were smear negative, of whom 166 (8.6%) were culture positive for TB and 206 (10.6%) were culture positive for NTM. The sensitivity of smear microscopy in the detection of MTBC was 81.5% (95% CI 79.0–84.0%), with a specificity of 55.3% (95% CI 53.6–57.0%), positive predictive value (PPV) of 33.8% (95% CI 31.8–35.8%) and negative predictive value (NPV) of 91.4% (95% CI 90.2–92.7%). There was no significant difference in sensitivity between NAA and smear tests (difference 1.7%, 95% CI − 1.8–5.2%); however, the specificity of smear tests was significantly lower than that of NAA tests (difference 41.4%, 95% CI 39.6–43.2%).

Of the 4126 patients with NAA tests, 1100 (26.7%) were infected with NTM, including 44.6%, 20.2% and 31.5% of those who underwent Amplicor, Xpert and in-house PCR tests, respectively (Table 5). In patient-based analysis, the false-positive rate of NAA among the patients infected with NTM was 3.2%, including 0.8% for Amplicor, 2.7% for Xpert and 10.3% for in-house PCR. In specimen-based analysis, the false-positive rate of NAA tests was 1.6%, including 0.3% for Amplicor, 0.7% for Xpert and 7.7% for in-house PCR (Table 5). In-house PCR had a significantly higher false-positive rate among the specimens that were culture positive for NTM than Amplicor (difference 7.4%, 95% CI 3.4–11.5%) and Xpert (difference 7.0%, 95% CI 2.9–11.0%).

Table 5 Patient- and specimen-based analyses of the performance of NAA tests in excluding NTM
Full size table

Discussion

Our results revealed that the NAA tests had relatively high sensitivity, specificity, PPV and NPV. In addition, the performance of frontline NAA and add-on NAA tests did not differ significantly, except that the sensitivity of frontline Amplicor was lower than that of add-on Amplicor. There was no significant difference in sensitivity between the NAA tests and smear microscopy; however, the specificity of smear microscopy was substantially lower than that of the NAA tests. The three NAA platforms that we evaluated had a relatively low sensitivity in patients with smear-negative pulmonary TB. Furthermore, in-house PCR had a significantly lower specificity compared to Amplicor and Xpert, in part because in-house PCR had a relatively high false-positive rate among the patients infected with NTM.

Several studies have validated the performance of Amplicor, Xpert and in-house PCR in the diagnosis of pulmonary TB in different settings [12,13,14,15,16,17], providing evidence to support the use of NAA tests as an initial diagnostic test for the rapid detection of pulmonary TB.

Amplicor has been used in Taiwan for 2 decades. It can be loaded with 24 samples for simultaneous amplification and is usually used in laboratories with a relatively high volume of tests. Eing et al. reported that, compared to in-house PCR, Amplicor exhibited a relatively low sensitivity (91.1% versus 66.3%) and similar specificity (99.9% versus 99.7%) for the diagnosis of pulmonary TB [13]. Cohen et al. also reported that the sensitivity of in-house PCR was higher than that of Amplicor in hospitalized patients [18]. However, we found that Amplicor had a similar sensitivity and better specificity compared to in-house PCR in our settings.

Amplicor and Xpert have rarely been compared. Patel et al. compared Amplicor and Xpert for the diagnosis of TB meningitis and reported that they had similar sensitivity and specificity [19]. Antonenka et al. reported that Xpert and Cobas Taqman MTB assay had a similar performance in detecting MTBC in respiratory specimens [20]. Our results revealed that Xpert and Amplicor had similar sensitivity and specificity. Moreover, both Amplicor and Xpert had better specificity than in-house PCR for the diagnosis of pulmonary TB, while the sensitivity between these two platforms was similar.

The sensitivity of smear microscopy in the detection of MTBC was relatively high (81.5%) in our study samples, and there was no significant difference in sensitivity between the NAA and smear tests. This likely was largely because a relatively high proportion of the NAA tests was performed as an add-on test for positive smear to exclude NTM, which may have introduced selection bias. However, the specificity of smear tests was significantly lower (55.3%) than that of the NAA tests. We have been conducting a prospective randomized study on NAA tests in the diagnosis of pulmonary TB in Taiwan, which may provide additional insights into this issue.

The relatively low sensitivity of NAA tests for smear-negative specimens (33.1%) is a concern and is not sufficiently reliable to exclude TB. This finding is likely because there were very few bacilli in the respiratory samples and is consistent with a previous study [21]. Roche Diagnostics (Rotkreuz, Switzerland) has replaced the Cobas Amplicor MTB test with the Cobas TaqMan MTB test, which has a better specificity than the Cobas Amplicor MTB test [22]. However, the sensitivity of the Cobas TaqMan MTB test for smear-negative specimens remains unsatisfactory [21]. Cepheid (Sunnyvale, CA, USA) developed the Xpert MTB/RIF Ultra (Xpert Ultra) to overcome the limitations of the Xpert MTB/RIF assay [14]. Dorman et al. conducted a prospective multicenter diagnostic accuracy study comparing the diagnostic performance of Xpert Ultra with that of Xpert MTB/RIF and concluded that the sensitivity of Xpert Ultra was superior to that of Xpert MTB/RIF in patients with paucibacillary TB and in patients with human immunodeficiency virus infection [15]. However, this increase in sensitivity of Xpert Ultra came at the expense of a decrease in specificity, especially in patients with a previous history of anti-TB treatment [15]. Further studies are needed to investigate whether the sensitivity of Xpert Ultra is superior to that of Xpert MTB/RIF in Taiwan.

Recently, the frequency of discovery of NTM has increased in Taiwan. One previous study conducted in northern Taiwan showed that NTM accounted for 56.9% of cultures testing positive for mycobacteria during 2000–2012, while another Taiwanese multicenter study found that the incidence rate of NTM lung disease was 46.0 episodes per 100,000 hospital-based patient-years during 2010–2014 [23,24,25]. Therefore, NAA tests are increasingly used to rapidly differentiate TB and NTM in patients with a positive-smear result in Taiwan. Our results revealed that Amplicor and Xpert performed similarly well in excluding NTM with a relatively low false-positive rate. However, in-house PCR had a significantly higher false-positive rate among patients infected with NTM compared to Amplicor (7.7% versus 0.3%) and Xpert (7.7% versus 0.7%). This is consistent with the findings of previous reports in which the false-positive rate of in-house PCR (12.1%, 7/58) among subjects with a positive culture for NTM was higher than that of Amplicor (6.9%, 4/58) and Xpert (0.8%, 1/122) [18, 19]. This highlights discrepancies between the laboratory practice of in-house PCR, the National Committee for Clinical Laboratory Standards guidelines, and the manufacturers’ recommendations. Taken together, these findings suggest that it may be beneficial to replace in-house PCR with other NAA platforms in clinical practice. Of note, Xpert may report false-positive results for five NTM species (M. abscessus, M. marinum, M. smegmatis, M. phlei and M. aurum) [26, 27].

This study has several strengths. It was implemented in seven hospitals throughout Taiwan with a relatively high number of TB patients. This allowed us to enroll a large number of participants, which is representative of our setting. Furthermore, the real-world data used in this study reflect clinical practice, and thus our findings will be useful for the development of national policy. There are also several limitations to this study. A substantial proportion of the NAA tests were add-on tests and thus may have introduced a selection bias toward a relatively high sensitivity of smear microscopy in the detection of MTBC. Moreover, there was heterogeneity of laboratory methods and regulations between the hospital laboratories. We did not include non-respiratory specimens, thus our findings may not be generalizable to patients suspected of having extrapulmonary TB. Also, some clinical information was not collected, and we were not able to explore the causes of false-positive NAA results. Lastly, NTM species identification was not performed in all subjects infected with NTM because of the retrospective study design. Therefore, we were not able to investigate the complete profile of NTM species that caused false-positive NAA results.

Previous studies have found that routine NAA testing for all specimens from patients suspected of having TB in a low prevalence setting is not as cost-effective as that in high TB burden countries [28,29,30]. Our findings showed that Amplicor and Xpert had a similar performance for diagnosing pulmonary TB. However, Amplicor requires multiple samples per run and does not detect rifampicin resistance in TB. Xpert is more expensive than Amplicor, but it can manage specimens individually and can detect rifampicin resistance. In-house PCR had a lower specificity and a relatively high false-positive rate in excluding NTM in this study, and thus it may not be a good option until its performance has been improved. Taiwan has a medium burden of TB (annual reported TB incidence of 33.0 per 100,000 population in 2020) and a relatively low prevalence of rifampicin resistance (2.0% among new TB cases and 7.0% among previously treated cases had RR or MDR TB in 2020) [3, 8]. In laboratories with a relatively high volume of tests per day, whether first performing Amplicor as the initial diagnostic test followed by Xpert for Amplicor-positive specimens to detect rifampicin resistance would be more cost-effective than the use of Xpert as the initial diagnostic test needs to be investigated. In contrast, for hospitals that manage a relatively low number of TB patients, using Xpert rather than Amplicor to manage specimens individually would likely be a better option to achieve a shorter turnaround time.

Conclusions

Although NAA tests had a relatively high sensitivity and specificity in detecting MTBC in our settings, the sensitivity values of Amplicor, Xpert and in-house PCR in the detection of TB among smear-negative pulmonary TB cases were all relatively low. Amplicor and Xpert performed better than in-house PCR in excluding NTM, suggesting that it might benefit from replacing in-house PCR with other NAA platforms in a real-world setting.