Introduction

Clinical registries are interactive real-time databases recording the detailed information of patients, including specific diagnoses, clinical conditions, and procedures [1]. A clinical registry is typically customized to fulfill its major purposes, including describing the natural history of diseases, treatments, medications, and their clinical efficacy and cost-effectiveness, as well as disease outcomes and safety and quality of care issues [2, 3]. In 1974, the World Health Organization (WHO) pioneered these registries in epidemiological and clinical research [4], and since then disease registry systems (DRSs) have turned into organized systems to improve the quality of care in the healthcare system and progress in medical research [5, 6]. Disease registry systems provide opportunities to conduct high-quality medical research, advance diagnostic and therapeutic clinical practice guidelines, and improve the quality of healthcare services, patient outcomes, and resource and financial management. In addition, DRSs can show us the best way to strategic purchasing and conduct cohort and clinical trials based on real data [7, 8], especially for rare diseases [9].

However, using databases for research and audit and answering specific questions require high-quality data and resolving the weaknesses of DRSs [10]. Quality control (QC), as an integrated system, is an important component of the quality management of data registries, helping in dynamic monitoring and evaluating the effectiveness of the construct and process of data registration according to predetermined goals. As with any project, DRSs should be evaluated by a supervisor appointed by the founder. This process helps DRSs improve themselves and correct their errors, such as missing data, delayed follow-up, and data duplication [4, 11, 12]. Therefore, clinical DRSs need to be validated and improved as their quality assessment results are provided to policymakers and health insurance companies and are used for making public-heath related decisions.

In Iran, DRSs started to grow 30 years ago when the cancer registry and then other national registries such as trauma, spinal defects, and newborn anomalies were established [13,14,15,16]. In 2014, a national DRS program was created aiming to integrate at least 20 DRSs in Iran. Until November 2018, a total of 71 clinical DRSs were active in the medical universities and health institutions affiliated with Iran’s Ministry of Health and Medical Education, among which Shahid Beheshti University of Medical Sciences (SBMU), with six DRSs, ranked fifth until then. This university now hosts around 50 DRSs for different medical specialties. The pillar of dynamic monitoring of these DRSs is to validate their different QC dimensions and integrate them into a meaningful whole to provide comprehensive coverage while keeping these dimensions organized and integrating them with the health information system (HIS).

Because it is inapplicable to interpolate all internationally used QC dimensions into all DRSs, we here developed a checklist to monitor the most important QC dimensions including comparability [17], reliability, and validity [18], completeness [19], timeliness [11], accessibility [20], efficiency, and duplication [2]. The second purpose was to evaluate the quality of 49 DRSs.

Methods

The present study had a mixed-method design and was conducted in two consecutive qualitative and quantitative phases. In the qualitative phase, a checklist was developed to assess the QC of 49 active DRSs established by the research centers, hospitals, and educational departments affiliated with the Shahid Beheshti University of Medical Sciences (SBMU). In these DRSs, the data were launched using unique and standard software, and crude data were transferred from actively supervised registries approved by SBMU. Registries with no recorded data were excluded.

Ethical consideration

This study’s procedures were approved by the Ethics Committee of SBMU under the registration number IR.SBMU.RETECH.REC.1400.577.

Qualitative phase: checklist development

A working group consisting of 13 experts was formed to develop a checklist through a panel group discussion and brainstorming. The working group’s members included epidemiologists (n = 3), medical informatics specialists (n = 3), social medicine specialists (n = 3), health policymakers with experience in the field of DRSs (n = 2), and two DRS professionals. First, an initial form of the checklist was designed based on the standards released by the Ministry of Health of Iran, and then it was further updated based on the key points extracted from relevant articles retrieved by systematically searching different databases. Finally, a checklist with 29 items was developed, and each item was scored on a five-point Likert scale from "not important" (one score) to "very important" (five scores). A total of three panel-group discussion sessions were held with the participation of all 13 experts. During the first session, questions with scores of 4 or 5 were kept and further examined in the subsequent panel-group sessions, and questions obtaining a score of < 4 were omitted. At the end of the third panel-group discussion, a checklist with 23 items was finalized.

The questions were developed in a way to evaluate different QC aspects of the registries, including structure management, data sources, data elements, registry software, recording processes, registry outcomes, user training, and the performance of the QC subcommittee.

The final checklist approved by the panel group was presented to an examiner team, whose members had no previous encounter with the research topic, to assess the face validity and understandability of the questions. In this step, content validity was numerically calculated using two indicators: Content Validity Index (CVI) and Content Validity Ratio (CVR). Items with CVI scores less than 0.7 were omitted. Considering that our expert team had 13 members, the acceptable CVR was designated above 0.56 based on the Lawshe “minimum CVR value”. The reliability of the checklist was verified based on Cronbach’s α.

Training courses

In this step, eight examiners were requested to designate the level of QC for each item of the checklist based on an organized guideline presented to them during two training courses. The examiner team consisted of individuals who were registrars, researchers, executive directors, quality experts, administrators, and supervisors who were familiar with the process of data registration. All examiners (n = 8) participated in the training courses with a total duration of four hours. Finally, the QC of each registry was separately checked by the examiners using the provided checklist. During the training courses, the executive director of the registry provided the related documents and reports and briefly explained data collection methods. All examiners were then asked to rate each QC dimension for the registry according to the checklist under the supervision of the head of the team. All examiners independently investigated the registry. In order to calculate the agreement between examiners, the checklist was completed twice for two of the registries (# 25 and #26) at an interval of three months between June and August 2021. The average of Kappa agreement obtained was beyond 80%, which is considered acceptable.

Quantitative phase: data collection

During the evaluation step, four examiner teams assessed the QC of the registries from August to November 2021. On the examination day, the executive director presented the annual reports of the registry to examiner teams. Then the registry was rated for different items available on the checklist.

Data quality dimensions

There is a need to accurately define and regularly monitor all QC dimensions, some of which have been well discussed and defined in various fields of medicine [17]. Comparability, completeness, and validity are considered key QC items in most registries [21]. In the present study, we focused on six QC areas to assess our DRSs, including 1) Comparability: intra-organizational consistency of data over time allowing for comparison [17]; 2) Completeness: the data collected matching the data expected to describe a specific entity [19]; 3) Timeliness: collecting and sharing data within a reasonable time to be used for intended purposes [11]; 4) Interpretability: ease of understanding the data [22]; 5) Accessibility: ease of access to the data; users’ being informed of what data are being collected and where they are located; and 6) Validity: adhering to the rules or definitions applicable to the data during data collection [2].

A rating scale of 0 to 650 was used to rank the data during quality analysis. This rating scale was then transformed into a percentage system from 0 (the lowest quality) to 100% (the highest quality). A cut-point of 70% of the mean score based on a consensus reached by the panel group was considered desirable for different QC domains, including completeness, timeliness, accessibility, validity, comparability, and interpretability, whose individual cut-points were obtained as 111.3, 45.8, 70.4, 26.3, 5.6, and 20.57, respectively.

Results

During the study’s qualitative phase, a total of 29 items were extracted corresponding to the goals of the Ministry of Health and according to the literature in order to rank the quality of each registry. A satisfactory level of agreement was observed among the panelists with regard to the final 23 QC items. Regarding content validity assessment, the CVI and CVR of these 23 items were 0.79 and 0.58, respectively, suggesting the good content validity of the checklist items. Cronbach’s alpha coefficients for all QC domains were higher than 0.69, indicating acceptable internal consistency. This checklist was used to evaluate six QC domains for DRSs, including completeness (Q1, Q2, Q3, Q5, Q6, Q7, Q10, Q17, and Q18), timeliness (Q11, Q21, and Q22), accessibility (Q12, Q20, and Q23), validity (Q9, Q13, and Q14), comparability (Q16), and interpretability (Q4). These dimensions were ranked based on the total scores from the summing of related questions. Furthermore, data duplication, confidentiality, and understanding of the memorandum (MOU) were separately evaluated by the questions of Q8, Q15, and Q19, respectively (Table 1).

Table 1 Quality control checklist for disease registry programs
Full size table

During the study’s quantitative phase, the registries were assigned specific serial numbers. An overview of the 49 registries approved by SBMU has been shown in Table 2. Most of the registries (81.6%, n = 40) were focused on diagnosis/treatment; six of them (12.2%) recorded treatment outcomes and other registries were related to different scopes of diagnosis (n = 1,#49), procedures (n = 1, #47), and side effects of treatments (n = 1, #30). As illustrated in Fig. 1, most of the registries were in the field of neurology (n = 8), followed by pediatrics (n = 7) and cancer management (n = 6).

Table 2 Overview of 49 quality control registry’s in Shahid Beheshti University of medical science
Full size table
Fig. 1
figure 1

Registries established by Shahid Beheshti University of Medical Sciences, Iran, in different health fields

Full size image

Table 3 shows the ranking of the registries studied based on their QC scores. The highest rank (96.1%) belonged to registries #22 and #42, and the lowest ranking (20%) was related to registry #49.

Table 3 Total score and rank obtained of disease registry programs of Shahid Beheshti University of Medical Sciences in 2021
Full size table

Table 4 presents the mean score of each QC domain for DRSs based on the total scores from the summing of related questions. Regarding the acceptable quality cut-point (i.e., > 70% of the mean score of each domain), out of 49 DRSs evaluated, 48 (98%), 46 (94%), 41 (84%), 38 (77.5%), 36 (73.5%), and 32 (65.3%) registries obtained quality scores in the domains of interpretability, accessibility, completeness, comparability, timeliness, and validity, respectively. In this study, the rate of recording duplicated data was low (12.2%), which can be explained by the development of electronic registries with unique national ID numbers.

Table 4 Scores of six domains of quality control of 49 registries
Full size table

Discussion

In this study, we developed a customized checklist to assess six quality domains in 49 DRSs established by SBMU in 2021. Our results demonstrated that all the registries had acceptable interpretability, accessibility, comparability, and completeness. In addition, the timeliness and validity domains acquired the lowest quality ranks.

There are several methodological problems with data quality assessment, one of which is the lack of a comprehensive standardized method for this purpose. The quality of data varies between different practices, and data quality needs to be assessed based on unique requirements in various fields [21, 23, 24]. Comparability, completeness, and validity are considered key elements during data quality assessment [21, 23]. Faulconer and de Lusignan suggested an 8-step statistical method for assessing the quality of the data of patients with chronic obstructive pulmonary disease [24]. Therefore, we developed a specific checklist to assess the QC of our DRSs in different fields.

Only a high degree of completeness will ensure that the incidence and prevalence rates estimated in DRSs are close to their ‘true’ values. Most data QC dimensions overlap with each other, and their interpretations are vague due to ambiguous definitions or even a lack of standard definitions. Two of the most frequently cited data QC dimensions are “accuracy” and “completeness”. It may be difficult to accurately estimate the completeness of registries because this entity is influenced by the proportion of patients introduced to registries by healthcare centers, the ratio of those refusing to be referred, and the total number of patients in the study population [25]. In our investigation, the overall rate of completeness was obtained at 84%, which was in line with the study conducted by Fung et al. on Singapore’s cancer registries, reporting a completeness rate of 98.1% [26]. One possible reason for lower completeness in some of our DRSs may be the short time passing from their establishment (#11, #19, and #27). Despite all the limitations such as the relatively short period of the study, in a study by Lee et al., completeness of 90–100% was reported for a registry of operative sectors (e.g., operating surgeons, consulting surgeons, and the hospital). Interestingly, auditing revealed that the registry’s completeness reached 100% after resolving deficiencies [27]. Another explanation for this variation in completeness may be differences in the number of patients with specific disorders such as Parkinson's disease (#1), SMA (#2), pediatric migraine (#8), tracheal stenosis following intubation (#15), and pediatric nephrotic syndrome (#21). In addition, nationwide recruitment for a number of our registries (#42, #43, and #44) could have contributed to their high completeness rates. Also, some medical procedures should be registered before their costs can be reimbursed by insurance companies (registries #12, #23, and #24). Another factor increasing the completeness of data recording can be the proven utility of this practice amongst health professionals in the registeration centers (#10, #18, #22, #36, #37, and #41). It is worth noting that using this checklist, we were unable to determine the proportion of eligible patients who decided not to be enlisted in relevant registries, increasing the likelihood of overestimating completeness in these registries.

Using standard internationally approved definitions for recording and reporting data boosts the level of comparability of registries [11]. In our study, there was limited standardization regarding the definitions used in registries, leading 11 out of 49 registries to have unacceptable comparability in terms of diagnostic and therapeutic elements. It is worth mentioning that in our checklist, only one question (Q16) was related to comparability, limiting the ability of this checklist to reliably assess this quality domain compared to other dimensions investigated by multiple questions. Comparability has been reported to vary considerably in different registries. In a study in Russia, only four cancer registries out of 10 studied registries met international standards [28]. However, cancer registries in Singapore were reported to have a high level of comparability [26]. Low comparability is the main barrier to achieving an interoperability framework, and one potential solution to this problem is to develop a team of specialists and experts to standardize definitions across all DRSs.

High timeliness allows for the real-time recording of diagnoses, procedures, and other relevant data in DRSs. Although there are currently no international regulations for assessing timeliness, timely reporting of information is a foremost priority for all registries [1, 11]. There are rare reports on the timeliness of DRSs [29], and the definition of this term in the context of data registries should be exactly determined. In general, timeliness refers to the rapidity at which a registry can collect, process, and report reliable and complete data [30]. In our checklist, timeliness was defined as the date on which the database was ‘frozen’ to calculate annual statistics for issuing an official report. This period comprised two intervals: the time until receipt announcement (ie., from the date of diagnosis to the day of receiving the report) and the processing time (i.e., from the date of report receiving until data availability). In our report, in 36 out of 49 registries (73.5%), physicians or nurses reported the data at the time of patient visits or shortly afterward, but five registries had delays in submitting annual reports. Also, eight of the registries preferred to postpone the publication of their results to attain better completeness to the cost of undesirable timeliness. Evaluating the timeliness and several other quality dimensions of a pediatric mortality surveillance system in Iran showed that this system successfully fulfilled timeliness criteria mainly because the managers were committed to holding a monthly committee for monitoring childhood mortality and the immediate reporting of infectious diseases [31]. Moreover, an evaluation of timeliness at the Cancer Registry of Norway during 1953–2005 showed that the median time for diagnosis of a new case reduced from over 525 days in 2001 to 261 days in 2005 [29]. Another study showed that the timeliness of the diseases was low based on the national reporting of the disease surveillance system [32]. Therefore, implementing electronic data recording and employing dedicated and well-trained staff can improve the timeliness of registries in reporting their data.

Accessibility is defined as the ease of access to data for users, rendering the data more available and making it possible for others to confirm the registry’s results [33]. As almost all of our registries used the same unique software, they acquired high accessibility (94%). This unique software facilitates the generation of meaningful and credible information from diverse sources, decreasing the occurrence of potential errors during the data entry process and facilitating access to the data. In line with our results, a study by Azadmanjir et al. aimed to identify and address hurdles to data accessibility at the National Spinal Cord Injury Registry of Iran, a registry relying on primary data sources. Their expert panel selected 174 data quality items, including accessibility and usefulness in quality-of-care assessment in emergency settings [34].

Validity is the extent to which the data registered can be assessed in terms of accuracy and relevant rules or definitions [35]. A high validity rate (91.9%) was reported for Singapore’s cancer registries, which could be attributed to the fact that these registries were focused on a specific field [26]. In the present study, nearly one-third of our registries (35%) had low validity, mainly due to the lack of uniform definitions for items due to the variety of DRS fields. This observation highlights the importance of employing uniform, transparent, and accurate definitions by all registries according to existing guidelines and classifications.

Interpretability is defined as the ease of understanding of data for users, presenting one of the main challenges faced by data registries and requiring a clinical framework for data interpretation and scoring [2]. Registered data can be interpreted only if they are specifically assigned to exact endpoints [22]. In our checklist, this quality domain was assessed by a single question (Q4), asking about the presence or absence of a data dictionary at each registry, and almost all (98%) of the registries investigated obtained a score higher than the cut-point. There are no previous reports on the interpretability of registered data in DRSs, so we could not compare our observation with other studies in terms of this QC domain.

The main strength of the present investigation included the development of a customized checklist to assess data quality in Iran’s DRSs in various fields in terms of all main QC standard indicators. One of the limitations of this study was that we confined our search to DRSs established by SBMU in Tehran, which could decrease the generalizability of our findings. Second, most of these DRSs mainly rely on the reports generated by governmental hospitals, and it is possible that the private hospitals or health service centers in different regions might not be covered.

Conclusion

In this study, a customized checklist was developed to assess the quality of the data recorded in DRSs, which could be considered a proof of concept for future investigations. Our results demonstrated that most of DRSs had high degrees of interpretability, accessibility, comparability, and completeness. However, their timeliness and validity needed to be improved. As the DRSs of the SBMU acquired a high degree of quality control in most of the studied domains allows for greater confidence in the use of the qualified data to improve the healthcare system and the possibility of integrating data with national healthcare data.