Introduction

Coronavirus disease 2019 (COVID-19) has affected 545 million people globally till July 1, 2022 [1], since it was declared pandemic by the World Health Organization on March 11, 2020 [2]. Diagnostic testing for SARS-CoV-2 is a critical component to the overall prevention and control strategy for COVID-19. Countries should have national testing strategy with clear objectives that can be adapted according to changing epidemiological situations, available resources and tools, and country-specific contexts. It is crucial that all testing for SARS-CoV-2 is linked to public health measures to ensure appropriate clinical care and support and to carry out contact tracing to break the chain of transmission [3].

The “gold standard” for clinical diagnostic detection of SARS-CoV-2 remains laboratory-based (moderate and high complexity) nucleic acid amplification test (NAAT), such as quantitative reverse transcription polymerase chain reaction (qRT-PCR); however, specialized instruments and expertise are required [4, 5]. Rapid antigen detection immunoassays are particularly suited for point-of-care testing (POCT), as they can easily be performed and interpreted without equipment, are inexpensive, and improve turnaround times. Moreover, results returned by a recently launched antigen assay appeared to correlate better with patient infectiousness than RT-PCR results [6].

Since mid-2020, cheaper and faster diagnostic tests that detect specific antigens for SARS-CoV-2 infection have become commercially available, and some have reached the WHO Emergency Use List. Antigen tests generally have similar specificity, but are less sensitive than most NAATs. Antigen level in specimens collected either before the onset of symptoms, or late in the course of infection, may be below the test detection limit. This can result in a negative antigen test result, while more sensitive tests, such as most NAATs, may return a positive result. The antigen test has comparable sensitivity to laboratory-based NAAT when the viral load in the specimen is high and the person is most likely to be most infectious. Therefore, WHO recommends the use of antigen rapid diagnostic tests (Ag-RDTs) that meet the minimum performance requirements of 80% sensitivity and 97% specificity, compared to NAAT in suspected cases of COVID-19. WHO recommends that these requirements can be used for primary case detection and contact tracing, during outbreak investigations, and for monitoring trends in disease incidence in the communities [3].

World Health Organization (WHO) provided technical assistance to Ministry of Health, Republic of Indonesia, to develop the Ministerial Decree No. HK.01.07/MENKES/446/2021 on the “Use of Antigen Rapid Diagnostic Test in Testing of Coronavirus Disease 2019 (COVID-19).” The decree provides guidance on how antigen-detecting rapid diagnostic tests can be used for SARS-CoV-2 diagnosis as well as the criteria for products that can be used in Indonesia, including those that are listed in WHO Emergency Use Listing [7]. Therefore, Directorate of Supervision of Medical Devices and Household Health Supplies—Ministry of Health of Indonesia, established several testing laboratories including our laboratory, to ensure the validity of Ag-RDTs proposed by some companies [8]. The Ag-RDT kits must be valid for the issuance of certificate for marketing authorization. In this study, we evaluated the eligibility of the three proposed Ag-RDT kits and analyzed the results.

Methods

Sample collection and study protocol

Three hundred and thirteen samples were collected in Institute of Tropical Disease, Universitas Airlangga, Surabaya, Indonesia, during March—May 2020.

Using flocked swabs, and following appropriate safety precautions, trained staffs collected nasopharyngeal and oropharyngeal samples per participant and placed the swab in a 3-mL viral specimen collection tube (Shandong Chengwu Medical Products Factory, Shandong, China). In the laboratory, the swabs were examined by using qRT-PCR and Ag-RDT kits for COVID-19.

The three Ag-RDT kits (namely kit A, kit B, and kit C) were evaluated. Two of them were made in South Korea and Singapore, and the other was made domestically. Each kit should be evaluated as follows: (1) using around 30 samples with cycle threshold value (Ct) ≤ 25 by qRT-PCR, around 30 samples with Ct > 25 by qRT-PCR, and around 30 samples negative by qRT-PCR, chosen/selected randomly; (2) nasopharyngeal swabs from new cases (24–48 h) were inserted into the kit buffer.

SARS-CoV-2 testing

RNA extraction and qRT-PCR

RNA was extracted using QIAamp Viral RNA Mini Kit (Qiagen, Hilden, Germany). Then, qRT-PCR assay was performed on RNA extracts to detect viral RNA by The SuperScript III Platinum One-Step qRT-PCR Kit (Invitrogen, Carlsbad, CA) targeting the nucleocapsid using 2019-nCoV RUO kit (primers and probes for N1, N2 and RP) (IDT, Singapore), completed with 2019-nCoV_N positive control (as positive control) and Hs_RPP30 positive control (as internal control) (Integrated DNA Technologies, Inc., Iowa, USA). The amplification was performed on QuantStudio 1 Real-Time PCR System (Applied Biosystems, Foster City, CA, USA) according to the manufacturer’s recommendations. Samples with Ct of the two target genes < 40 were considered positive. If only 1 gene shows Ct < 40, it will be retested by using a commercial kit (The STANDARD M nCoV Real-Time Detection Kit, SD BIOSENSOR, Inc., Korea), which was commonly used for public service in our laboratory. Our laboratory has obtained a certificate of proficiency testing for SARS-CoV-2 PCR, organized by WHO. We used a positive control in addition to the positive control provided by the kit, at each running of qRT-PCR.

Ag-RDT

Ag-RDT was performed following the manufacturer’s instruction (reading at 15–20 min). It is a ready-to-use test which allows rapid and qualitative detection of SARS-CoV-2 antigen in nasopharyngeal secretions.

The three Ag-RDT kits for COVID-19 evaluated in this study use a sandwich immunodetection method employing a simple-to-use lateral flow test format. Ag-RDTs are usually comprised of a plastic cassette with sample and buffer wells, a nitrocellulose matrix strip, with a test line with bound antibody specific for conjugated target antigen–antibody complexes and a control line with bound antibody specific for conjugated antibody. They are based on immunochromatography and show the presence of SARS-CoV-2 Ag by a colored test line. In the case of SARS-CoV-2 RDTs, the target analyte is often the virus’ nucleocapsid protein, preferred because of its relative abundance [9, 10].

Statistical analysis

The criteria used for the performance of the Ag-RDT kits were sensitivity, specificity, NPV, PPV, and accuracy. The qRT-PCR was considered as the gold standard for this evaluation; therefore, positive and negative samples by molecular techniques were considered true positive and true negative samples, respectively. Analyses were performed using R program version 4.1.1 (https://www.r-project.org). The difference between SARS-CoV-2 qRT-PCR Ct and the results of the three Ag-RDT kits was determined by one-way ANOVA test in the R program; P value < 0.05 was considered significant.

Results

We examined 270 of obtained 313 nasopharyngeal samples from persons with suspected COVID-19. The examined samples met the criteria for the validation tests of the three kits. They were sent from private clinics and public and private hospitals in East Java Province, mostly from Surabaya City.

The performances (sensitivities, specificities, NPVs, PPVs, and accuracy) of the three kits are described in Table 1. For samples with Ct > 25, all the three kits could not detect SARS-CoV-2 Ag (0% sensitivity) but showed a high specificity (100%). Meanwhile, for samples with Ct ≤ 25, kit C was the best (76.7% sensitivity and 100% specificity) and almost met the minimum performance requirements of ≥ 80% sensitivity and ≥ 97% specificity. The PPV of the three kits was 100%, but their NPVs ranged from 63% (kit B) to 84.8% (kit C). Kit C showed the best accuracy (89.9%).

Table 1 Performance of Ag-RDT kit A, kit B, and kit C
Full size table

The positive result of qRT-PCR is determined by Ct values of the two target nucleocapsid genes (N1 and N2). In Fig. 1, the reactive results by kit A and kit B were obtained from samples showing a large range between Ct values of the two genes. Meanwhile, the reactive results by kit C were obtained from samples showing a more consistent Ct values between the two genes than the other two kits. All the reactive results were obtained from samples with Ct value ≤ 25. More reactive results were produced by kit C. In Fig. 2, compared to the non-reactive results by the other two kits, those by kit B were obtained from samples showing a largest range in Ct values between the two genes. Some non-reactive results of kit B were obtained from samples showing very low Ct values (< 15) of one target gene while high Ct of the other target gene. The consistent Ct values (≤ 25) of the two genes seem to increase the number of reactive results from the Ag-RDT kit. However, there were no significant differences between the reactive results by the three Ag-RDT kits (A, B, and C) to the Ct values of N1 (P = 0.16) and N2 (P = 0.22).

Fig. 1
figure 1

Ct values of N1 and N2 target genes from qRT-PCR among reactive results by the three Ag-RDT kits (A–C)

Full size image
Fig. 2
figure 2

Ct values of N1 and N2 target genes from qRT-PCR among non-reactive results by the three Ag-RDT kits (A–C)

Full size image

Discussion

In the ongoing COVID-19 pandemic, diagnostic testing for SARS-CoV-2 is crucial to limit the spread of the virus as well as appropriately manage infected patients [11]. There is a need to develop reliable rapid tests for infection control with early notification of cases to enable effective outbreak management [10]. Many rapid tests have been developed to detect SARS-CoV-2 proteins in respiratory samples.

The performances of these rapid antigenic tests depend on several factors including the patient factors (i.e., symptoms and immune status), viral factors (i.e., viral load and structural variation in the target antigen), specific protein target detected in the assay (i.e., produced in higher or lower concentration, higher mutation rates or not), product design and quality (i.e., potential cross-reactivity and quality of packaging), transportation and storage, quality of the specimen, capability of the operator, and how it is processed [3, 11].

The COVID-19 Ag kits have several advantages such as the ease and fast achievement of the test, the rapid answer, the lower cost, and the non-requirement of special equipment or skills compared with molecular techniques; however, many studies showed that the rapid tests are suffering from poor sensitivity [11]. In this study, the three Ag-RDT kits could detect SARS-CoV-2 with low–medium sensitivity (40–76.7%) in nasopharyngeal samples with Ct ≤ 25. The maximum Ct values in samples showing reactive results using kit A, kit B, and kit C were less than 25 (24.96, 24.40, and 23.42, respectively). Several samples show a very large difference in Ct values of the two genes; the lower Ct value of them is more associated with the reactive results from the Ag-RDT kit. All the three kits could not detect samples with Ct > 25 (0%). The overall specificity of the three kits for all samples (with Ct > 25 and Ct ≤ 25) was 100%. We used CDC 2019-Novel Coronavirus (2019-nCoV) Real-Time RT-PCR Diagnostic Panel targeting nucleoprotein genes (N1 and N2) which is a highly accurate test [12]; however, variation in the obtained samples might influence the performance of the evaluated kits.

The three kits evaluated in this study used nucleoprotein (NP) of SARS-CoV-2, the same as the common Ag-RDT kits for COVID-19, due to a potential biomarker for the early diagnosis of COVID-19. As the NP is highly immunogenic, it is abundantly expressed in almost all coronavirus infections. It was reported that the NP (nucleocapsid) gene is highly conserved and stable, with more than 90% amino acid homology with SARS-CoV and a low mutation rate [13,14,15]. However, Lesbon et al. (2021) found the mutations in NP gene in 99 of the 14,346 sequenced samples in Brazil. Among the 17 representative samples, they demonstrated as follows: a deletion of 18 nucleotides (Del28877-28894) (82%), a substitution of GGG to AAC (28881–28883) (12%), and a frameshift mutation caused by deletion (Del28877-28878) (6%). These mutations causing the six amino acids’ deletion and two amino acids’ substitutions occurred in the linker region of NP [16]. We found that more reactive results were generated by kit C, where the tested samples showed more consistent Ct values (< 25) of the two target genes. The consistent Ct values (< 25) of the two genes seem to increase the number of reactive results from the Ag-RDT kit. The possible mutations in the NP contained of the samples with different Ct value ranges between N1 and N2 genes might influence the results of the two kits (kit A and kit B).

The previous studies have performed validity tests of some commercial RDTs. Lindner et al. (2020) reported sensitivities of 74.4 to 79.5% in agreement with PCR assays, which meant that up to 25% positive cases were not detected by lateral flow assays [17]. Krüttgen et al. found that the sensitivity of a known commercial rapid antigen assay with samples with a Ct of < 25, 25—< 30, 30—< 35, and >  = 35 was 100%, 95%, 44.8%, and 22.2%, respectively. They concluded that the lateral flow devices were only useful for patients with high viral loads (Ct < 30) [18]. Scola et al. (2020) added that there was a correlation between successful isolation of virus in cell culture and Ct value of qRT-PCR (targeting E gene), and they deduced that nasopharyngeal samples from patients with Ct > 33–34 are not contagious. They also suggested that sensitivity of amplification based on gene E detection would be less sensitive than ORF1ab or N genes [19]. The Ag-RDT may have a limited suitability for the determination of the SARS-CoV-2 infection status of patients, especially in the early or late phase of the infection typically associated with a low viral load. However, differentiation between contagious and non-contagious individuals may be possible with this assay [18]. In our study, we classified the positive samples into samples with Ct ≤ 25 and samples with Ct > 25. We performed qRT-PCR targeting N1 and N2 genes. There were no samples with Ct > 25 showing reactive results by all the three kits. Only kit C almost met the minimum performance requirements of ≥ 80% sensitivity and ≥ 97% specificity for samples with Ct ≤ 25. Kit A and kit B had already been validated in India; the results also showed both were not approved [20]; however, in the USA, kit A was authorized by Food and Drug Administration (FDA) under an Emergency Use Authorization Only (EUA) for use by authorized laboratories certified under the Clinical Laboratory Improvement Amendments (CLIA). The variability in results between tests (which is not reflected in the manufacturer-reported data) indicates the need for independent validations. It is likely due to variability in the populations tested, as head-to-head performance showed a comparable sensitivity. A previous review highlights the importance of performing tests in accordance with the manufacturers’ recommended procedures and in alignment with standard diagnostic evaluation and reporting guideline [21]. Besides, Haage et al. (2021) recommended to ensure appropriate transport and storage conditions of the Ag-RDT kits for accurate use of SARS-COV-2 Ag-RDTs in tropical settings, due to more robust across temperature fluctuations. Short- and long-term exposure to elevated temperatures likely impairs sensitivity of several SARS-CoV-2 Ag-RDTs that may translate to false-negative test results at clinically relevant virus concentrations compatible with inter-individual transmission [22].

Most of the currently available Ag-RDTs are based on LFIA platform, which is using antigen–antibody reaction as a detection principle, and are performed by the unidirectional flow of sample over a test strip. Most of them lose out to RT-qPCR in terms of sensitivity, and they may also show cross-reactivity to other-related viruses. Funabashi et al. [23] developed Rapiim SARS-CoV-2 to improve the sensitivity of SARS-CoV-2 Ag-RDT by using the optical waveguide-based biosensor technology [24] and a pair of highly specific monoclonal antibodies (mAbs) targeting SARS-CoV-2 nucleocapsid protein (NP) antigen [25]. The US FDA has only approved the Ag-RDTs that use NP as a target region [26], but this may cause the cross-reaction with NP from other coronaviruses. This could be overcome by using N-terminally truncated NP (DN-NP), which is highly specific for SARS-CoV-2 [25, 27]. Funabashi et al. (2021) found that Rapiim SARS-CoV-2-N was able to detect all samples with high and medium viral titers, while it could detect 64.7% (95% CI 47.8–78.6%) samples in the low virus titer, with a detection limit of 9.3 × 104 copies/ml and exhibited no cross-reactivity with related viruses (100% specificity) [23]. Several improvement methods could reduce the number of false negatives while screening patients with low viral load in the early stages of infection.

This rapid test was thought to be used as a first-line COVID-19 diagnostic test to possibly reduce the number of qRT-PCR testing in cases of positive results; however, it still required confirmation for negative results. Some commercial kits on the market recommended that their assays are not approved as a stand-alone diagnostic and the limitations that the assay should be performed “in patients with clinical symptoms” or that the test “is not intended to detect from defective (non-infectious) virus during the later stages of viral shedding.” They cannot be recommended for broad use in any setting in which reliable diagnostics are crucial to avoid spreading of the virus, such as hospitals and long-term care facilities for the risk groups, especially as limited information on host and viral factors influencing shedding of SARS-CoV-2 antigens and their correlation to infectious viruses impede any prognosis on infectivity [10]. There is scarce evidence on the accuracy of these tests in widespread testing of asymptomatic populations in different settings [28]. Kruttgen et al. [18] in 2020 and Lefever et al. [29] in 2021 have performed some validation studies, but in smaller samples or including symptomatic patients.

The limitation of this study was several factors which may affect the results of these validity tests that could not be controlled in this study, especially the viral factors such as the presence of viral antigen variation and the expression or shedding of viral antigen in the tested samples.

In conclusion, our data showed that the proposed three SARS-CoV-2 Ag-RDT kits have not met the minimum performance requirements of ≥ 80% sensitivity for samples with Ct ≤ 25. The validation test should be encouraged by the policymakers in national health care for the issuance of certificate for marketing authorization. The poor sensitivity of the COVID-19 Ag kits leads to false-negative results, which in these times of pandemic can be of great consequence. Some factors might influence the results of evaluation such as variation of genes/proteins and transportation–storage of the kits; therefore, several improvement methods should be used to increase their performance, even though they cannot achieve RT-PCR performance and should not be used alone for COVID-19 diagnosis.