Abstract
Background and Aims
Rapid and accurate detection of COVID-19 is crucial for mitigation of the pandemic. We evaluated the performance of six molecular kits and the effect of several factors on the performance of the kits.
Materials and Methods
Two hundred and four nasopharyngeal samples were collected from participants aged ≥18 years at the Baruch Padeh Medical Center Poriya, Israel, between June and August 2020. Samples were tested by: Allplex 2019‐nCOV Assay (Seegene), Real‐Time Fluorescent RT‐PCR Kit for Detecting SARS-2019-nCoV (BGI Genomics), Xpert® Xpress SARS-CoV-2 test (Cepheid), Simplexa® COVID-19 Direct Kit (Focus Diagnostics), BD SARS-CoV-2 Reagents for BD MAX™ System (BD), and Logix Smart™ Coronavirus Disease 2019 (COVID-19) Test kit (CO-DIAGNOSTICS).
Results
Xpert® Xpress SARS-CoV-2 test and Logix Smart™ COVID-19 Kit had the highest (91.2%) and the lowest (74.5%) sensitivity, respectively. Symptoms were a predictor of a positive result. Traditional assays had a higher minimum cycle threshold (min Ct), i.e. detected lower viral load, compared to rapid assays (p = 0.012). Samples of symptomatic participants had lower min Ct, than samples of asymptomatic participants (p < 0.001). Additionally, the more genes were detected, the lower the min Ct (p < 0.001), indicating that a greater percentage of the viral genome was amplified.
Conclusions
Taken together, most assays had overall good performance. Since several factors affect the performance of kits, each laboratory must be familiar with its kit’s limitations in order to produce the most reliable results.
Similar content being viewed by others
Avoid common mistakes on your manuscript.
We evaluated the sensitivity and specificity of six molecular kits that detect COVID-19 and investigated several factors that may affect the performance of the kits. |
We found that Xpert® Xpress SARS-CoV-2 test had the highest sensitivity (91.2%). |
The performance of the kits was affected by the presence of symptoms, number of genes the kit detected, and type of RNA extraction. |
1 Introduction
More than 1 year ago, an outbreak of a severe acute respiratory syndrome Coronavirus 2 (SARS-CoV2) emerged in Wuhan, China, and rapidly spread globally [1]. Considering there were no specific therapies, one successful strategy for mitigating this pandemic spread was a rapid, accurate and sensitive detection of the virus, enabling early isolation of infected individuals [2].
To this end, a real-time reverse-transcriptase PCR (RT-PCR) using nasopharyngeal samples was chosen as the gold standard for COVID-19 detection and several assays have been rapidly developed [3, 4]. Among these assays are the traditional RT-PCR assays, that require sample preparation, including virus deactivation and RNA extraction, prior to amplification and detection of the viral genome. Other diagnostic tests are automated, rapid molecular assays, that include an automated process for all steps of RT-PCR from sample preparation to a final result.
Nevertheless, infection control is still limited since RT-PCR only assesses the viral RNA presence in the sample and cannot indicate on the virus viability and infectivity [5]. Thus, a negative result cannot exclude a previous infection, and must be combined with the patient’s history, clinical observations and available epidemiological information [2]. On the other hand, a patient with a positive result is not necessarily infectious.
In this study, we compared the performance of four traditional molecular assays and two automated, rapid tests that have been used worldwide by experienced clinical diagnostic laboratories to detect COVID-19: Allplex 2019‐nCOV Assay (Seegene), Real‐Time Fluorescent RT‐PCR Kit for Detecting SARS-2019-nCoV (BGI Genomics), Simplexa® COVID-19 Direct Kit (Focus Diagnostics), BD SARS-CoV-2 Reagents for BD MAX™ System (BD), Xpert® Xpress SARS-CoV-2 test (Cepheid), and The Logix Smart™ Coronavirus Disease 2019 (COVID-19) Test Kit (CO-DIAGNOSTICS). We also investigated the effect of several factors on the performance of the kits.
2 Materials and Methods
2.1 Study Population
The study group included 204 patients aged ≥18 years, who were admitted to the Baruch Padeh Medical Center Poriya, Israel, between June 2020 and August 2020 for COVID-19 molecular testing.
The Israeli Ministry of Health approved this study as part of validation of kits for detection of COVID-19. Since there was no utilization of personal data of the participants (except for the presence of symptoms that were linked to specimens and these were de-identified), the Israeli Ministry of Health waived the need for Helsinki Ethics Committee approval and for participants’ consent.
Out of 204 participants, 102 were positive and 102 were negative for the presence of COVID-19 RNA, according to the AllplexTM 2019‐nCOV Assay, which was considered our reference kit. Of the 102 COVID-19-positive participants, 77 were tested due to symptom presentation, while the other 25 had no symptoms and were tested due to exposure to an infected individual. Symptoms included fever or chills, cough, shortness of breath/difficulty breathing, fatigue, muscle or body aches, headache, recent loss of taste or smell, sore throat, congestion or runny nose, nausea or vomiting, and diarrhea.
2.2 Sample Collection
Samples were collected from the tonsillar areas and the posterior pharynx of patients using synthetic fiber swabs with thin flexible plastic shafts. The swabs were inserted into test tubes containing Universal Transport Medium for viruses (UTM) and sent to the clinical microbiology laboratory at the Baruch Padeh Medical Center, Poriya, Israel, for the detection of COVID-19.
2.3 Molecular Testing for COVID-19
2.3.1 Xpert Xpress SARS-CoV-2 Test
The Xpert Xpress SARS-CoV-2 test (Cepheid, Sunnyvale, CA, USA) was performed using the GeneXpert Instrument Systems (Cepheid, Sunnyvale, CA, USA) that perform automated sample preparation, nucleic acid extraction, amplification, and detection of target sequences using real-time PCR assays. The Xpert Xpress SARS-CoV-2 test was performed inside a disposable cartridge that contains RT-PCR reagents according to the manufacturer’s instructions. Briefly, the sample tube was mixed and then 300 µL of the sample-in-UTM was transferred to the sample chamber of the Xpert Xpress SARS-CoV-2 cartridge using the supplied transfer pipette. The GeneXpert cartridge was loaded onto the GeneXpert Instrument System platform and results were obtained within 47 min. Results were automatically interpreted by the instrument’s software. Table S1 in the Electronic Supplementary Material (ESM) describes the characteristics of all tested kits, including the specific gene targets.
2.3.2 Simplexa™ COVID-19 Direct Test
The Simplexa™ COVID-19 Direct test (Focus Diagnostics, Cypress, CA, USA) was performed on the LIAISON® MDX instrument (Focus Diagnostics), which, like the GeneXpert Instrument systems, performs an automated RT-PCR reaction from sample preparation to a final result. The kit is supplemented with 24 reaction mix vials and a Direct Amplification Disc, which contains two wedges for each sample, with one wedge for the reaction mix (designated as ‘R’) and the second wedge designated as ‘S’) for the sample. For each sample, the content of one reaction mix tube (50 µL) was transferred to the ‘R’ wedge, followed by pipetting 50 µL of the sample-in-UTM to the ‘S’ wedge. Then, the disk was loaded on the LIAISON® MDX instrument. Results were obtained within 90 min and automatically interpreted by the instrument’s software.
2.3.3 BD SARS-CoV-2 Reagents for BD MAX™ System
The BD SARS-CoV-2 Reagents for BD MAX™ System (BD Diagnostics, Franklin Lakes, NJ, USA) is a real-time RT-PCR test intended for the qualitative detection of COVID-19 RNA in respiratory samples. This assay includes RNA extraction, RT-PCR reaction and detection of the viral nucleic acid. For each sample, 750 µL of the sample-in-UTM was transferred to a BD MAX™ TNA-3 Sample Buffer Tube provided with the kit. For each sample, one Unitized Reagent Strip was loaded with one BD MAX™ ExK™ TNA-3 Extraction Tube, one BD MAX™ TNA MMK Master Mix Tube and one BD SARS-CoV-2 Reagents for BD MAX™ System Primers and Probes Tube. The Unitized Reagent Strips and required number of BD MAX™ PCR Cartridges were placed into the BD MAX™ System (BD Diagnostics). The Sample Buffer Tubes were then placed into the BD MAX™ System Racks corresponding to the Unitized Reagent Strips. Thermal cycling was performed at 58 °C for 20 min, followed by 95 °C for 5 min, and 45 cycles of 95 °C for 5 s, and 58 °C for 40 s.
The BD MAX™ System automatically interprets the test results when reaction ends (within ~59 min).
2.4 RNA Extraction for Non-Automated Assays
Viral RNA was extracted from 140 µL of the sample-in-UTM using the QIAamp Viral RNA Kit and the QIAcube automated spin-column purification kit (QIAGEN GmbH, Hilden, Germany) according to the kit's protocol using the QiIAqube instrument (QIAGEN).
2.5 AllplexTM 2019‐nCOV Assay
The Allplex™ 2019-nCoV Assay (Seegene, Seoul, South Korea) was designed to detect the COVID-19 RNA in human respiratory samples. This assay was chosen by the Israel Ministry of Health as the gold standard for COVID-19 detection. It was therefore used as the reference standard for this analysis. For each sample, 5 µL of 2019-nCoV MOM (containing oligonucleotides and amplification and detection reagent), 5 µL of RNase-free Water, 5 µL of 5X Real-time One-step Buffer, and 2 µL of Real-time One-step Enzyme were added to a well of 96 PCR microplate. Then, 8 μL of RNA from each sample or positive or negative controls was transferred into the specific well. The PCR plate was loaded into a Bio Rad CFX96TM Real-Time Detection System (Bio Rad), and reaction conditions were set as follows: one cycle at 50 °C for 20 min, followed by a cycle at 95 °C for 15 min and 45 cycles at 94 °C for 15 s and 58 °C for 30 s. Results were obtained within ~70 min. Interpretations of results was performed according to the manufacturer's instructions.
2.6 Real‐Time Fluorescent RT‐PCR Kit for Detecting SARS-CoV-2 Assay
The Real‐Time Fluorescent RT‐PCR Kit for Detecting SARS-CoV-2 (BGI Genomics, Yantian District, Shenzhen, China) was designed to detect the COVID-19 RNA in human respiratory samples. For each sample, 18.5 µL of SARS-CoV-2 Reaction Mix and 1.5 µL of SARS-CoV-2 Enzyme Mix were transferred into a well of 96 PCR microplate. Then, 10 μL of RNA from each sample or positive or negative control was transferred into the specific well. The PCR plate was loaded into a Bio Rad CFX96TM Real-Time Detection System (Bio Rad) and reactions conditions were set as following: 1 cycle at 50 °C for 20 min, followed by a cycle at 95 °C for 10 min and 40 cycles at 95 °C for 15 s and 60 °C for 30 s. Results were obtained within ~ 60 min. Interpretation of results was performed according to the manufacturer’s instructions.
2.7 The Logix Smart™ Coronavirus Disease 2019 (COVID-19) Test Kit
The Logix Smart™ Coronavirus Disease 2019 (COVID-19) Test kit (CO-DIAGNOSTICS, Lake City, UT, USA) was designed to detect the COVID-19 RNA in human respiratory samples. For each sample, 5 µL of Master Mix were transferred into a well of 96 PCR microplate. Then, 5 μL of RNA from each sample or positive or negative control was transferred into the specific well. The PCR plate was loaded into a Bio Rad CFX96TM Real-Time Detection System (Bio Rad) and reaction conditions were set as follows: one cycle at 45 °C for 15 min, followed by one cycle at 95 °C for 2 min and 50 cycles at 95 °C for 3 s and 55 °C for 32 s. Results were obtained within ~47 min. Interpretation of results was performed according to the manufacturer's instructions.
2.8 Statistical Analysis
Each sample was tested twice by each assay and there was no difference in the results of the two repeats.
Since our laboratory is a clinical microbiology laboratory, we did not evaluate copies/mL and only performed validation of the assays. We rely on the evaluation that was performed by each manufacturer regarding copies/mL and limit of detection.
We used the Allplex™ 2019-nCoV Assay as the reference method for calculating sensitivity, specificity, and negative and positive predictive values. Therefore, specimens that were found to be positive or negative by the Allplex™ 2019-nCoV Assay were defined as ‘True Positive’ or ‘True Negative’, respectively. Fisher’s exact test was applied for analyzing the differences between the sensitivities of the assays. Agreement rates were calculated as the percentage of samples that had the same results as the reference kit, the Allplex™ 2019-nCoV Assay, out of the total samples.
A Wilcoxon rank sum test was performed to analyze differences in number of positive results between symptomatic and asymptomatic participants. Odds ratio analysis was performed to investigate whether symptoms are predictive of a positive result.
A Chi-square test was applied to investigate the association between agreement and kit type (vs. rapid test) or between agreement and similarity of genes.
A Wilcoxon rank sum test was performed to analyze differences in the minimum cycle threshold (min Ct) (equates to the highest viral load) of agreement cases and disagreement with the reference kit or between symptomatic and asymptomatic participants. A paired t-test was performed to analyze differences in the mean minimum Ct value between traditional and rapid tests.
A repeated-measures ANOVA was applied to test differences in Ct values between PCR kits based on number of detected genes.
Statistical significance was determined with p value < 0.05. Data were analyzed using the R (R Core Team, 2020) software, version 4.0.2.
3 Results
Two hundred and four participants were enrolled in the study, with 102 positive and 102 negative for the presence of COVID-19 RNA, according to the AllplexTM 2019‐nCOV Assay, which was considered our reference kit. Of the 102 COVID-19-positive participants, 77 were tested due to the presence of symptoms, while the other 25 had no symptoms and were tested due to exposure to/contact with an infected individual.
3.1 Performance of Kits, in Comparison with Allplex™ 2019-nCoV Assay
Our main aim was to evaluate the performance of traditional and rapid molecular tests for COVID-19 detection, compared to our reference kit, the Allplex™ 2019-nCoV Assay.
The samples of all 102 COVID-19- negative participants were detected as negative by Logix Smart™ COVID-19 Kit, Xpert® Xpress SARS-CoV-2 test and Simplexa®COVID-19 Direct Kit.
Two (1.96%) out of 102 negative samples were detected as positive by the BD SARS-CoV-2 Reagents for BD MAX™ System (BD) and the Real-Time Fluorescent RT-PCR Kit for Detecting SARS-2019-nCoV had 8 (7.8%) false-positive results. All kits had false-negative results, with Logix Smart™ COVID-19 Kit having the highest (26/102, 25.5%) and the Xpert® Xpress SARS-CoV-2 test having the lowest (9/102, 8.8%) number of false-negative results (Table S2, ESM).
As presented in Table 1, among the rapid tests, the highest agreement levels with the Allplex™ 2019-nCoV Assay occurred with the Xpert® Xpress SARS-CoV-2 test (95.6%), which also had the highest sensitivity (91.2%) and the highest negative predictive value (92%) of all tests.
The lowest sensitivity assay among all assays was the Logix Smart™ COVID-19 Kit, with 74.5% sensitivity. All assays had high specificities and positive predictive values. The negative predictive values were in the range of 79.7–92.
Among the traditional tests, the BD SARS-CoV-2 Reagents for the BD MAX™ System had the highest agreement (91.2%) levels with the Allplex™ 2019-nCoV Assay and the highest sensitivity (88.2%).
3.2 Percentage of True Positive Results Detected by the Different Assays
Among the traditional molecular assays, the Real-Time Fluorescent RT-PCR Kit for Detecting SARS-2019-nCoV had detected the highest percentage (88.2%) of true positive (TP) results (according to the reference kit). Additionally, this assay had detected the highest percentage of TP results among both symptomatic and asymptomatic participants (93.5% and 76%, respectively) (Table 2).
The percentage of TP results of symptomatic participants was higher than that of asymptomatic participants in all the traditional assays.
Regarding the rapid molecular assays, the highest percentage of TP results was detected by the Xpert® Xpress SARS-CoV-2 test for all participants and for symptomatic and asymptomatic participants (91.2%, 97.4% and 72%, respectively). As with the traditional assays, the percentage of TP detected by the rapid tests was higher when participants had symptoms.
3.3 Percentage of True Positive Results According to the Number of Detected Genes
We were interested whether the percentage of TP results detected by the assays is affected by the number of detected genes. We categorized the number of genes detected according to the number of genes each kit detects (and not according to the test results of each gene). Therefore, three-genes-kit refers to Allplex 2019‐nCOV Assay, two-genes-kit refers to Simplexa® COVID-19 Direct Kit (Focus Diagnostics), BD SARS-CoV-2 Reagents for BD MAX™ System (BD), and Xpert® Xpress SARS-CoV-2 test (Cepheid).
As shown in Table 3, the percentage of TP results increased with the number of detected genes. Additionally, a higher number of TP results were obtained among asymptomatic participants with three genes (25, 100%) compared to two genes (22, 88%). Similarly, the number of TP results among symptomatic participants increased with the number of genes detected (Table 3).
3.4 The Association Between Symptoms and Positive Results
One of this study’s aims was to investigate whether the kits’ results are affected by symptom presentation. For each positive sample, we looked at the number of positive tests by specimen and compared these numbers between symptomatic and asymptomatic participants. As shown in Fig. 1, most positive samples among symptomatic participants were interpreted as positive by either five or six kits, while samples of asymptomatic participants were determined as positive by four or five kits (p < 0.001).
Next, we evaluated the odds ratio, with symptoms as predictive of a positive result (Table 4). Logistic regression analysis revealed that the odds ratio (OR) for positive results when the participant had symptoms was higher than 1, with Xpert® Xpress SARS-CoV-2 having the highest OR (14.6). The OR was statistically significant for all kits except of the Logix Smart™ COVID-19 Kit.
3.5 Factors That Affect Agreement With the Gold Standard Assay
One aim was to investigate factors that may influence the agreement with our reference kit, the Allplex™ 2019-nCoV Assay. First, we looked at the min Ct for each sample. We divided the positive results into results with agreement with the Allplex™ 2019-nCoV Assay and results with disagreement, and compared the min Ct of each group. We found that when the kits disagreed with the reference kit, the min Ct was higher compared to the agreement cases (min Ct = 38.9 and 31.26, respectively) (p < 0.001) (Fig. 2).
Second, we tested whether the kit’s type (rapid test or traditional assays) affects the agreement of the kit with the reference kit. A Chi-square test that compared the proportion of agreement between rapid tests (88.2%) or traditional assays (85.5%) with the reference kit did not find any association between type of kit and agreement with the Allplex™ 2019-nCoV Assay (p = 0.238)
Third, we investigated whether the agreement of kits with the Allplex™ 2019-nCoV Assay was affected by similarity of detected genes. No difference was found in the proportion of results with agreement between kits with common genes (86%) as detected by the Allplex™ 2019-nCoV Assay and kits with different genes (86.9%) (p = 0.75).
3.6 Factors That Affect the Min Ct
In Real-Time PCR, the Ct indicates reciprocally on the viral load. High Ct values may indicate low viral load that can be found at the beginning or at the end of disease. Additionally, for each test there is a range of accepted Ct for a positive result interpretation. Therefore, we wanted to explore factors that might influence the min Ct for each sample.
First, we compared the min Ct between the rapid tests and the traditional assays, and found that the traditional assays had a higher min Ct compared to the rapid tests (30.75 and 29.12, respectively) (p = 0.012) (Fig. 3A).
Boxplot of minimum Ct value (for all kits) for different factors. The minimum, maximum and median (indicated by a bold line) are presented for each graph. (a) The effect of method types on minimum (Min) Ct, *p < 0.05; (b) the effect of symptom presence on Min Ct, ***p < 0.001; and (c) the effect of number of detected genes on Min Ct, ***p< 0.001 for the comparison of 3-gene kits and 2-gene kits with 1-gene kit, #p < 0.05 for the comparison of 3-gene kits with 2-gene kits. One-gene kits refer to Real‐Time Fluorescent RT‐PCR Kit for Detecting SARS-2019-nCoV (BGI Genomics) and The Logix Smart™ Coronavirus Disease 2019 (COVID-19) Test kit (CO-DIAGNOSTICS); 2-gene kits refer to Simplexa® COVID-19 Direct Kit (Focus Diagnostics), BD SARS-CoV-2 Reagents for BD MAX™ System (BD), and Xpert® Xpress SARS-CoV-2 test (Cepheid); the 3-gene kit is the Allplex™ 2019-nCoV Assay
Second, we found an association between symptoms presence and min Ct, with samples from symptomatic participants having lower min Ct than samples of asymptomatic participants (29.4 and 33.44, respectively) (p < 0.001) (Fig. 3B).
Another factor that affected the min Ct was the number of genes detected by kits. The more genes that were detected, the lower the min Ct (p < 0.001) (Fig. 3C); the average min Ct values were 33.9, 28.8, and 26.57 for one-gene, two-gene and three-gene kits, respectively. A post hoc analysis revealed a significant difference between one-gene-kits and two-genes kits (p < 0.001), between one-gene kits and three-genes kits (p < 0.001) and between two-gene kits and three-gene kits (p = 0.016).
4 Discussion
The main aim of the current study was to evaluate the performance of six molecular tests for COVID-19 detection and to allocate factors that may affect the tests’ performance. The Food and Drug Administration Emergency Use Authorizations (FDA-EUA) have approved various molecular tests for the detection of COVID-19 [6]. These assays differ in many characteristics, starting from the sample type and the sample medium, through the extraction procedure, number and identity of detected genes, turnaround time, number of samples per run, and limit of detection [6]. Therefore, any laboratory that performs COVID-19 molecular tests should be familiar with its kit’s advantages and disadvantages.
We found that Xpert® Xpress SARS-CoV-2 test had the highest sensitivity among all tests, compared to the reference kit. This finding was not surprising in light of data from previous publications that have presented even higher sensitivities of this assay (97–100%), compared to the current study [7,8,9]. Additionally, a recent report that compared the performances of seven different primer-probe sets concluded that primers that target the N2 or the E genes, as in the Xpert® Xpress SARS-CoV-2 test, have higher sensitivity, compared to primers with other gene targets [10].
The least sensitive assay among all assays was the Logix Smart™ COVID-19 Kit, with 74.5% sensitivity. However, we have not find any previous publication for a reference. Nevertheless, it was shown that primers targeted against the RdRp gene were less sensitive compared to primers targeted against N2 and E genes [10]. Thus, this evidence can explain the relatively low sensitivity of the Logix Smart™ COVID-19 Kit, which amplifies the RdRp gene.
It should be mentioned that although various studies have evaluated some of the assays that were tested in the current study, several differences in study design (i.e., the extraction process, the study participants, etc.) may affect the different performance measures’ results. For example, in the current study, all assays were compared to the Allplex™ 2019-nCoV Assay. Different studies have used other assays as their reference kit [7,8,9, 11, 12]. Therefore, the performance of a specific kit may change when compared to a different assay.
It is known that various factors can contribute to variations in kit performances, including different gene targets and different primer-probe sets, the need for RNA extraction step, limits of detection, etc. [5].
One common factor that affected all tests was the presence of patient symptoms. As was shown earlier, the percentage of true positive results was higher among symptomatic participants compared to asymptomatic participants in all assays. As we also saw, symptomatic patients had lower Ct values compared to asymptomatic patients, so we assume the lower Ct values contribute to the higher TP percentage. Additionally, the OR analysis has found that symptoms were predictive of a positive result in most kits. These findings are probably associated with viral load differences; it was shown that the highest viral load was detected at symptom onset and decreased during the first 10 days [13]. Another study has reported that the Ct was lower when sampling was performed close to symptom onset and increased as the gap between symptom onset and sampling day increased [7]. Therefore, we believe that symptomatic participants have a higher viral load compared to asymptomatic patients, leading to a lower Ct determination and higher probability of interpretation as a positive result.
Another factor that was investigated in the current study with regard to kit performance is the number of detected genes. We showed that the larger the number of genes detected, the larger the number of true positive results, among all participants and among the sub-groups of symptomatic and asymptomatic participants. A former study has presented an enhanced sensitivity of assays with two gene targets compared to a one-gene target kit [12]. It is known that the COVID-19 virus has evolved over quite a short period of time. Whenever there is a change in the genome area to which the primer-probe set is usually attached, the test's sensitivity may decrease. Therefore, detecting more than one gene target can reduce the risk for sensitivity decrease due to viral evolution. As proof of this assumption, a recent study has screened available SARS-CoV-2 genomes, and found single nucleotide mutations that influenced the annealing of all RT-PCR assays tested in the study. They concluded that reduction in kit performance can be minimized with the implementation of more than one gene target [14].
Our next aim was to investigate factors that affect the agreement of kits with the reference kit. We found that disagreement cases were characterized with higher min Ct values. In addition to the fact that each kit has a different cut-off, there is an ongoing debate regarding the cut-off Ct that should be used to interpret a positive result. The reason for this argument is associated with evidence that the viral load and therefore the virus culturing rate decreases as the Ct increases [13, 15,16,17,18]. For example, Singanayagam et al. [13] found a probability of 8% to culture the virus from samples with a Ct above 35 [13]. Thus, we assume that the disagreement cases in our study represent samples with a low viral load.
Other factors that we explored in relation to agreement with the reference kit were the kit's type (rapid test or traditional assays) and similarity of detected genes. No associations were found between these factors and agreement with the Allplex™ 2019-nCoV Assay.
Our last goal was to investigate factors that affect the min Ct. First, to our surprise, we found a higher min Ct in the traditional assays compared to the rapid tests. It is known that tests with no nucleic acid step usually contain a large number of amplification inhibitors [19]. Therefore, we expected a slightly higher Ct in the rapid tests. Our results may suggest that the tested rapid tests overcome this disadvantage of amplification inhibitors.
Second, samples from symptomatic participants had a lower min Ct than samples from asymptomatic participants. This finding supports our assumption that symptomatic patients have a higher viral load compared to asymptomatic patients.
The last factor that affected the min Ct was the number of genes detected by kits. The more genes that were detected, the lower the min Ct. As mentioned earlier, as the virus has gained a considerable number of mutations since its first appearance, the binding efficiency of several primer-probe sets to their target genes may have been reduced, which may affect kit performance. Therefore, detecting more genes will compensate for a reduction in kit performance.
The study has several limitations. First, we used the Allplex™ 2019-nCoV Assay as the reference kit based on a decision by the Israeli Ministry of Health. A wider evaluation should be performed with other kits serving as the reference kit. Second, we had a limited number of samples and specifically samples of asymptomatic patients.
5 Conclusions
In conclusion, our results suggest that most assays for COVID-19 detection are characterized with overall good performance measurements. The differences in the various kit performances are derived from both controlled (such as sampling efficacy) and uncontrolled factors (presence of symptoms). Therefore, each laboratory must be familiar with its kit disadvantages and limitations in order to produce the most reliable results.
References
Zhu N, Zhang D, Wang W, Li X, Yang B, Song J, Zhao X, Huang B, Shi W, Lu R, et al. A novel coronavirus from patients with Pneumonia in China, 2019. N Engl J Med. 2020;382:727–33. https://doi.org/10.1056/NEJMoa2001017.
Sharfstein JM, Becker SJ, Mello MM. Diagnostic testing for the novel coronavirus. JAMA. 2020;323:1437–8. https://doi.org/10.1001/jama.2020.3864.
Tang YW, Schmitz JE, Persing DH, Stratton CW. Laboratory diagnosis of COVID-19: current issues and challenges. J Clin Microbiol. 2020. https://doi.org/10.1128/JCM.00512-20.
Rabi FA, Al Zoubi MS, Kasasbeh GA, Salameh DM, Al-Nasser AD. SARS-CoV-2 and coronavirus disease 2019: What we know so far. Pathogens. 2019;2020:9. https://doi.org/10.3390/pathogens9030231.
Sule WF, Oluwayelu DO. Real-time RT-PCR for COVID-19 diagnosis: challenges and prospects. Pan Afr Med J. 2020;35:121. https://doi.org/10.11604/pamj.supp.2020.35.24258.
Oh H, Ahn H, Tripathi A. A closer look into FDA-EUA approved diagnostic techniques of COVID-19. ACS Infect Dis. 2021;7:2787–800. https://doi.org/10.1021/acsinfecdis.1c00268.
Procop GW, Brock JE, Reineks EZ, Shrestha NK, Demkowicz R, Cook E, Ababneh E, Harrington SM. A comparison of five SARS-CoV-2 molecular assays with clinical correlations. Am J Clin Pathol. 2021;155:69–78. https://doi.org/10.1093/ajcp/aqaa181.
Lephart PR, Bachman MA, LeBar W, McClellan S, Barron K, Schroeder L, Newton DW. Comparative study of four SARS-CoV-2 Nucleic Acid Amplification Test (NAAT) platforms demonstrates that ID NOW performance is impaired substantially by patient and specimen type. Diagn Microbiol Infect Dis. 2021;99: 115200. https://doi.org/10.1016/j.diagmicrobio.2020.115200.
Ulhaq ZS, Soraya GV. The diagnostic accuracy of seven commercial molecular in vitro SARS-CoV-2 detection tests: a rapid meta-analysis. Expert Rev Mol Diagn. 2021. https://doi.org/10.1080/14737159.2021.1933449.
Nalla AK, Casto AM, Huang MW, Perchetti GA, Sampoleo R, Shrestha L, Wei Y, Zhu H, Jerome KR, Greninger AL. Comparative performance of SARS-CoV-2 detection assays using seven different primer-probe sets and one assay kit. J Clin Microbiol. 2020. https://doi.org/10.1128/JCM.00557-20.
Liotti FM, Menchinelli G, Marchetti S, Morandotti GA, Sanguinetti M, Posteraro B, Cattani P. Evaluation of three commercial assays for SARS-CoV-2 molecular detection in upper respiratory tract samples. Eur J Clin Microbiol Infect Dis. 2021;40:269–77. https://doi.org/10.1007/s10096-020-04025-0.
Mostafa HH, Hardick J, Morehead E, Miller JA, Gaydos CA, Manabe YC. Comparison of the analytical sensitivity of seven commonly used commercial SARS-CoV-2 automated molecular assays. J Clin Virol. 2020;130: 104578. https://doi.org/10.1016/j.jcv.2020.104578.
Singanayagam A, Patel M, Charlett A, Lopez Bernal J, Saliba V, Ellis J, Ladhani S, Zambon M, Gopal R. Duration of infectiousness and correlation with RT-PCR cycle threshold values in cases of COVID-19, England, January to May 2020. Euro Surveill. 2020. https://doi.org/10.2807/1560-7917.ES.2020.25.32.2001483.
Penarrubia L, Ruiz M, Porco R, Rao SN, Juanola-Falgarona M, Manissero D, Lopez-Fontanals M, Pareja J. Multiple assays in a real-time RT-PCR SARS-CoV-2 panel can mitigate the risk of loss of sensitivity by new genomic variants during the COVID-19 outbreak. Int J Infect Dis. 2020;97:225–9. https://doi.org/10.1016/j.ijid.2020.06.027.
Wolfel R, Corman VM, Guggemos W, Seilmaier M, Zange S, Muller MA, Niemeyer D, Jones TC, Vollmar P, Rothe C, et al. Virological assessment of hospitalized patients with COVID-2019. Nature. 2020;581:465–9. https://doi.org/10.1038/s41586-020-2196-x.
Perera R, Tso E, Tsang OTY, Tsang DNC, Fung K, Leung YWY, Chin AWH, Chu DKW, Cheng SMS, Poon LLM, et al. SARS-CoV-2 virus culture and subgenomic RNA for respiratory specimens from patients with mild coronavirus disease. Emerg Infect Dis. 2020;26:2701–4. https://doi.org/10.3201/eid2611.203219.
van Kampen JJA, van de Vijver D, Fraaij PLA, Haagmans BL, Lamers MM, Okba N, van den Akker JPC, Endeman H, Gommers D, Cornelissen JJ, et al. Duration and key determinants of infectious virus shedding in hospitalized patients with coronavirus disease-2019 (COVID-19). Nat Commun. 2021;12:267. https://doi.org/10.1038/s41467-020-20568-4.
La Scola B, Le Bideau M, Andreani J, Hoang VT, Grimaldier C, Colson P, Gautret P, Raoult D. Viral RNA load as determined by cell culture as a management tool for discharge of SARS-CoV-2 patients from infectious disease wards. Eur J Clin Microbiol Infect Dis. 2020;39:1059–61. https://doi.org/10.1007/s10096-020-03913-9.
Wilson D, Yen-Lieberman B, Reischl U, Warshawsky I, Procop GW. Comparison of five methods for extraction of Legionella pneumophila from respiratory specimens. J Clin Microbiol. 2004;42:5913–6. https://doi.org/10.1128/JCM.42.12.5913-5916.2004.
Acknowledgements
We thank Mr. Wadie Abu Dahoud for the statistical analysis.
Author information
Authors and Affiliations
Contributions
Conceptualization, L.J. and A.P.; methodology, L.J., M.A., and A.P.; validation, L.J., M.A., and A.P.; formal analysis, L.J. and M.A.; investigation, L.J., M.A., and A.P.; data curation, L.J., M.A., and A.P.; writing – original draft preparation, L.J., M.A., and A.P.; writing – review and editing, L.J., M.A., and A.P.; visualization, M.A. and A.P.; supervision, M.A. and A.P.; project administration, M.A. and A.P. All authors have read and agreed to the published version of the manuscript.
Corresponding author
Ethics declarations
Funding
The study received no funding.
Conflicts of Interest
The authors have no conflicts of interest to declare.
Availability of Data and Material
The datasets generated and analyzed during the current study are available from the corresponding author on reasonable request.
Ethics Approval
This study was considered as part of validation of kits for detection of COVID-19. Since there was no utilization of personal data of the participants, the Israeli Ministry of Health waived the need for Helsinki Ethics Committee approval and for participants’ consents.
Consent
Patient consent was waived by the Israeli Ministry of Health. This study was considered as part of validation of kits for detection of COVID-19. Since there was no utilization of personal data of the participants, the Israeli Ministry of Health waived the need for Helsinki Ethics Committee approval and for participants’ consents.
Supplementary Information
Below is the link to the electronic supplementary material.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License, which permits any non-commercial use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc/4.0/.
About this article
Cite this article
Jerbi, L., Azrad, M. & Peretz, A. Evaluation of Factors that Affect the Performance of COVID-19 Molecular Assays Including Presence of Symptoms, Number of Detected Genes and RNA Extraction Type. Mol Diagn Ther 26, 229–238 (2022). https://doi.org/10.1007/s40291-021-00574-y
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s40291-021-00574-y