Introduction

Tuberculosis (TB) was a major public health problem in the world [1, 2]. In order to deal with the challenge of TB, governments of all countries had established TB control model in line with their national conditions, and actively carried out actions around the work objectives of each stage to achieve the goal of “End TB”. China was one of the 30 countries with high TB burden in the world, and the situation of TB control was severe. Implementing anti-TB institutions cooperation was the basic strategy for TB control in China. Since 2010, the new TB control model of trinity form had been implemented gradually in all regions [3]. In the new TB control model, the Centers for Disease Control and Prevention (CDC) was responsible for planning and management, the designated hospitals was responsible for diagnosis and treatment of patients, and the primary health care institutions was responsible for tracking and management of patients, that combine into the trinity form of “the anti-TB institutions cooperation which responsible for TB prevention, treatment and management together”. See the Fig. 1. The new TB control model of trinity form was expanded from the previously model that TB control being undertook by CDC alone. Under the new TB control model, the primary TB control institutions (including CDC in county-level and the designated hospitals in county-level and the primary health care institutions) undertook specific work that discovery, report, diagnosis, treatment and management of TB patients, and played an important role. The work in the primary TB control institutions involved multiple departments and interests, and was an organic whole. So the comprehensive evaluation should be carried out based on an indicators framework, in order to improve the working capacity of the primary TB control institutions. Especially after the COVID-19 pandemic [4,5,6], it was of great practical significance to carry out comprehensive evaluation of the performances in primary TB control institutions in the process of restoring the service supply of TB control and accelerating to realization the global goal of “End TB”. However, there was no an unified indicators framework for carrying out the comprehensively evaluation to the performances in primary TB control institutions, and there were few relevant studies in China.

Therefore, in this study, Delphi method and analytic Hierarchy Process (AHP) were used to establish an indicators framework for comprehensively evaluating the performances in primary TB control institutions under the new TB control model of trinity form, in order to provide theoretical and evidence-based basis for carrying out the comprehensive and effective evaluation of TB control in various in Guangxi and the other regions of China.

Fig. 1
figure 1

The new TB control model of trinity form in Guangxi, China

Full size image

Materials and methods

Study site

The Guangxi Zhuang Autonomous Region (Guangxi), with a population of 50 million, was an underdeveloped southwestern region where the TB epidemic was relatively serious in China [7,8,9]. At present, the new TB control model of trinity form had been implemented in all areas of the region in Guangxi. Previous studies had shown that there were many problems and difficulties in TB control of Guangxi, such as insufficient financial investment, weak ability in primary TB control institutions, common delay in treatment of patients, heavy disease economic burden and so on. After the impact of the epidemic of COVID-19, TB control would face more severe challenges. There was an urgent need to comprehensively evaluate the performance in primary TB control institutions based on a scientific indicators framework, so as to provide reference for the continuous improvement the quality of TB control. Our study was conducted in Guangxi with the period from September 2021 to December 2021.

Selection of experts

Using objective sampling method, 14 experts who had at least 10 years working experience and engaged in TB prevention and control and public health management were selected from health committee, CDC, TB designated hospitals and university of Guangxi, China.

Establishing of the initial indicators framework

Taking “tuberculosis (TB)”, “indicators framework”, “evaluation” as the key word, the relevant literature were searched online in PubMed, Embase, Cochrane Lirary databases, China national knowledge infrastructure(CNKI), Wanfang Database and so on. The relevant policy and regulation and reports from the year 2011 to 2021 were consulted on the official websites of national and local health commissions and CDC. And then, an initial indicators framework including 2 first-level indicators,10 second-level indicators and 69 third-level indicators were established after discussing by the research group.

Screening indicators by Delphi method

Delphi method, also known as expert survey method, was initiated and implemented by the RAND Corporation in 1946 [10, 11]. Later, the method was widely used in business, military, education, health care and other fields, and showed its superiority and applicability in the process of use [12]. The Delphi method was essentially a feedback anonymous letter inquiry method. Its general process was: after obtaining the opinions of experts on the problem to be predicted, the result would be sorted out, summarized and made statistics, and then gave anonymous feedback to the experts, asked for opinions again, concentrated and gave feedback again, until a consensus was got. Delphi method had three characteristics, that were anonymity, multiple feedback, group statistical response [10,11,12,13]. The characteristics of Delphi method made it the most effective judgment and prediction method, and the results were widely representative and reliable.

Two rounds of questionnaires consultation through email were carried out in the study. In the first round of consultation, the contents of the questionnaire included: the purpose of consultation, the basic information of the experts, the explanation of the indicators, and the opinion column for the modification or deletion were setting to collect the opinions from the experts. The second round of consultation was designed basing on the statistical summary of the first round of consultation results. After two rounds of consultation, the consensus was got in this study.

Quantitative assignment of options

The degree of importance, degree of familiarity and judgment basis of indicators were quantitative assignment separately by the following standards. ① Degree of importance: “Unimportant” was set as 1, “not too important” as 2, “moderately important” as 3, “Relatively important” as 4, and “Very important” as 5. ② Degree of familiarity: “unfamiliar” was set as 0.2, “not very familiar” as 0.6, “relatively familiar” as 0.8, and “very familiar” as 1.③ Judgment basis: it was divided into “theoretical analysis”, “practical experience”, “reference to domestic and foreign materials” and “intuition” according to the conventional; and the degree of influence for those above were divided into large, medium and small, respectively, and being given different quantitative values. See the Table 1.

Table 1 Quantitative assignment of the judgment basis and the degree of influence
Full size table

Evaluation parameters of the results

There were four parameters. ①Coefficient of active participation of experts: it was expressed by the recovery rate of questionnaire to evaluate the experts’ concern for the study. ②Authority coefficient of experts(Cr) : it was calculated according to the score of expert’s judgment (Ca) and the score of expert’s familiarity (Cs) to evaluate the value of experts’ advice, the calculation formula was Cr=(Ca + Cs)/2. ③Degree of concentration of experts’ opinions: It was expressed by the mean of quantitative assignment of the degree of importance and the full score ratio to evaluate the importance of indicators. ④Degree of coordination of experts’ opinions: It was expressed by the coefficient of variation (CV) and Kendall’s harmony coefficient (W) to evaluate the consistency of experts’ opinions.

Selection criteria of indicators

Three criteria were formulated for selecting indicators.①The mean of importance was above 4.30; ②The authority coefficient was above 0.800; ③The coefficient of variation(CV) was less than 0.200; Besides, it was combined with the expert’s opinion and the working practice at the same time, then to determine the choice of indicators.

Data analysis

The software Microsoft Excel was used to input data of consultation, and the software IBM SPSS Statistics for Windows, Version 25.0 was used to conduct statistical analysis on the results, the methods including descriptive statistic and Kendall’s W test, etc. A p-value < 0.05 was considered significant. The software Yaahp10.3 was used for Analytic Hierarchy Process (AHP), the processes including drawing the hierarchical structure model diagram, determining the scale of Saaty according to the mean of importance of indicators in the second round results, establishing a judgment matrix, using group decision-making, conducting consistency test, and finally determining the combination weights of indicators at all levels.

Results

Basic information of experts

A total of 14 experts were included in this study, with an average age of (43.3 ± 7.549) years old, the distribution of gender, organization, professional title, and degree of education of experts were showed in the Table 2.

Table 2 Basic information of experts (n = 14)
Full size table

Coefficient of active participation of experts

In each round of consultation, 14 questionnaires were distributed and 14 valid questionnaires were recovered, with the effective rate of 100.0% in both rounds.

Authority coefficient of experts (cr)

The results of the first round of consultation showed that the values of Cr of the 2 first-level indicators were 0.921 and 0.904, respectively; the values of Cr among 10 s-level indicators range from 0.803 to 0.914, with an average value of (0.877 ± 0.034); the values of Cr among 69 indicators of third-level range from 0.796 to 0.907, with an average of(0.848 ± 0.285); The results of the second round of consultation showed that the values of Cr among 39 indicators of third-level range from 0.802 to 0.900, with an average value of (0.860 ± 0.244).

Degree of concentration of experts’ opinions

The results of the first round of consultation showed that the mean of importance of the 2 first-level indicators were 5.00 and 4.21, the standard deviation were 0.000 and 0.426, and the full score ratio were 100.0% and 21.4%, respectively; The mean of importance among 10 s-level indicators range from 4.29 to 5.00, the standard deviation was 0.000 to 0.726, and the full score ratio was 42.9–100.0%; The mean of importance among 69 indicators of third-level range from 3.86 to 4.93, the standard deviation was 0.267 to 1.167, and the full score ratio was 21.4–92.9%, and the mean of importance among 23 indicators were less than 4.30. The results of the second round of consultation showed that the mean of importance among 39 indicators of third-level range from 3.93 to 5.00, the standard deviation was 0.000 to 1.072, and the full score ratio was 35.7–100.0%, and the mean of importance among 3 indicators were less than 4.30.

Degree of coordination of experts’ opinions

The results of the first round of consultation showed that the coefficient variation (CV) of the 2 first-level indicators were 0.000 and 0.101, respectively, and Kendall’s harmony coefficient (W) was 0.786 (x2 = 11.000, P = 0.001); The variable coefficient (CV) among 10 s-level indicators range from 0.000 to 0.169, Kendall’s harmony coefficient (W) was 0.201 (x2 = 25.323, P = 0.003); The variable coefficient (CV) among 69 indicators of third-level range from 0.054 to 0.303, Kendall’s harmony coefficient (W) was 0.161 (x2 = 152.869, P = 0.000), and the value of CV among 30 indicators were above 0.200. The results of the second round of consultation show that the variable coefficient (CV) among 39 indicators of third-level range from 0.000 to 0.273, Kendall’s harmony coefficient (W) is 0.169 (x2 = 89.928, P = 0.000), and the value of CV among 2 indicators were above 0.200.

Results of selection of indicators

According to the selection criterion of indicators which had been formulated, after the consultation of the first round, there are 30 third-level indicators were deleted; After the consultation of the second round, there were 2 third-level indicators are deleted; At last, an indicators framework was established, which included 2 first-level indicators, 10 second-level indicators and 37 third-level indicators. See the Table 3.

Result of weights of indicators

The results of analytic hierarchy process(AHP) showed that the consistency test of all levels of indicators are CI < 0.10, which indicating that the weight of each indicator was acceptable. The weights of the first-level indicators were the combined weights. But the weights of second-level and third-level indicators were the relative weights. The calculation formula for the combined weight of the second-level and third-level indicators were: the combined weight of the second-level indicator = the combined weight of the first-level indicator *the relative weight of second-level indicator; The combined weight of the third-level indicator = the combined weight of the second-level indicator *the relative weights of third-level indicator. See the Table 3.

Table 3 The importance and the weights of indicators at all levels in the framework for comprehensively evaluating the performances in primary TB control institutions
Full size table

Discussion

Delphi method was a mature and easy subjective decision-making technology after years of development. It was widely used in the field of medical and health services. In the process of establishing the indicators framework, this method could synthesize the theory and practical experience of experts from different professional backgrounds to make comprehensive judgement and evaluation [10,11,12,13]. This study was based on the existing research theories and methods [14,15,16,17,18]. Firstly, the initial indicators framework was established based on literature review, which had a good theoretical basis. Then, 14 experts were invited to take part in Delphi expert consultation, experts come from different organizations such as health committee, CDC, TB designated hospitals and universities in Guangxi, who engaged in TB control and public health management, with bachelor’s degree or above, and 78.6% of who had deputy senior professional titles or above, which showed that the experts had good representation and abundant professional knowledge, practical skills or management experience. In the two rounds of consultation, the coefficient of active participation of experts was 100.0%, which indicating that the enthusiasm of experts to participate in this study was high. The authority coefficient of experts (Cr) showed the experts’ theoretical understanding and practical experience about the indicators, and it was generally considered that when the value of Cr was equal to or above 0.7 was accepted. In this study, the average value of Cr in the two rounds of consultation was above 0.8, which showed that the opinion of experts had a good guidance for establishing the indicators framework. After each round of consultation, indicators were screened according to the selection criteria of indicators which had been formulated before, and the opinions of experts were summarized and feedback. Finally, after two rounds of consultation, the experts’ opinions tended to be consistent and meet the requirements of Delphi method, and the indicators framework was established, which included 2 first-level indicators, 10 second-level indicators and 37 third-level indicators, and could reasonably reflect the core contents and performances of TB control at primary TB control institutions.

Analytic hierarchy process (AHP) was a qualitative and quantitative, systematic and hierarchical analysis method, which was widely used in sociological, economics, management, engineering and other fields [19,20,21,22]. The indicators framework for comprehensively evaluating the performances in primary TB control institutions being established in this study was a multi-level and multi-indicators composite system, and the importance of each level and indicator needs to be determined scientifically. Using AHP method to determine the weights of indicator was to comprehensively calculate the weight coefficient of indicator by comparing the relative importance of each indicator at the same level, and the reliability and validity of the weights of indicators being determined were ensured by checking the consistency of the judgment matrix. In this study, the result of AHP was based on the score of the importance of indicators and the group decision of experts, which had good rationality and differentiation.

The indicators framework for comprehensively evaluating the performances in primary TB control institutions in this study took into account both objective performances and subjective effects, and covered all aspects of the work, which fully reflected the characteristics and focused of the new TB control model of trinity form.

From the view of the contents covered by the indicators, the objective performances included: discovery, report, registration, treatment and management of patients, and health education, quality control, training and coordination, a total of 8 aspects, that fully reflected the work flow of the new TB control model of trinity form. Among them, discovery and treatment of patients contain more third-level indicators, there were 7 third-level indicators and 6 third-level indicators, respectively, those were the focus works of the primary TB control institutions. It was in line with the basic TB control strategy in China, that was to discovery and treatment of patients timely, to control the source of infection, and to reduce the spread of TB in the population [23]. These two aspects were also the focus of study on TB control performance, which were used to evaluate the achievement of the objectives of national TB control plan [24,25,26,27]. Quality control was very important for TB control, which contained 9 third-level indicators, the content involved laboratory internal quality control, diagnosis and treatment process, infection control and adverse drug reactions reporting, and it could comprehensively evaluate the overall quality and ability of TB control. The subjective effects included: the satisfaction rate of medical staff and patients, each aspect containing a third-level indicator. Those two aspects were used to evaluate the satisfaction about the treatment of work for medical staff and the costs of treatment for patients. It was in line with the current situation of the two major challenges that insufficient human resources and heavy disease economic burden of patients, which were the mainly limitations to the performance of the primary TB control institution in Guangxi and the other regions of China [28].

From the view of the feasibility of indicators, each indicator had a clear definition and data sources, the data sources of objective performances came from National TB Management Information System and the daily work record for TB control; the data of subjective effect could be obtained by questionnaire survey in practical work.

From the view of the weights of indicators, among the first-level indicators, the objective performances had a high weight of 0.5444, which indicating that evaluating the specific tasks achievement was the main point of the performance evaluation. Among the second-level indicators, in the aspect of objective performances, indicators with the weights ranked in the top three were discovery, treatment and management of patients, that the combined weights were 0.0743, 0.0732, 0.0679 in turn; In the aspect of subjective effects, the weight of the satisfaction of the patients was higher of 0.2284, which indicated the concept of the patient-centered in the work of primary TB control institutions [29]. Among the third-level indicators, indicators with the highest weight in each aspect were as follows respectively: examination rate of sputum smear, timely reporting rate, registration rate of therapy, receiving rate of treatment, rate of regular medication, completion rate of health education, qualified rate of sputum smear, completion rate of training tasks; satisfaction rate of medical staff about the treatment of work; and satisfaction rate of patients about the costs of treatment, those indicators indicated the top priority of work, and had a great significance for effectively evaluating the performance of TB control in primary TB control institutions.

Prospects for the next work

The reliability and operability of the indicators framework for comprehensively evaluating the performances in primary TB control institutions in this study needed to be tested through empirical research. Therefore, the next work was to apply the indicators framework to the TB work practice in Guangxi, test the reliability and validity of the indicators framework, and continuously improve based on the needs and development of practice, so as to ensure the objectivity, authenticity and accuracy of the indicators.

Conclusion

In summary, this study focused on the performance of primary TB control institutions, following the scientific, comprehensive and practical principles, based on a large number of literature and policy guidelines at home and abroad, and an indicators framework fit for primary TB control institutions was established, which can provide a scientific and reasonable evaluation tool for comprehensively evaluating the performance of primary TB control institutions under the new TB control model of trinity form in Guangxi. In addition, our study conducted consultation with experts in Guangxi, so the results maybe more suitable for the work practice in Guangxi, and could also provide reference for the other regions in China.