A model for priority setting of health technology assessment: the experience of AHP-TOPSIS combination approach

Background In recent times, the use of health technologies in the diagnosis and treatment of diseases experienced considerable and accelerated growth. The goal of the present study was to describe the designated pilot MCDM (Multiple Criteria Decision Making) model for priority setting of health technology assessment in Iran. Methods Relevant articles were sought and retrieved from the most appropriate medical databases, including the Cochrane Library, PubMed and Scopus via three separate search strategies, using MESH and free text until March, 2015. Retrieved criteria were questioned from health technology assessment experts in two rounds and the relative weight for valid criteria was finally obtained from paired wise comparison method. After extraction of relative weights based on the aforementioned procedure, TOPSIS (The Technique for Order of Preference by Similarity to Ideal Solution) priority setting model was designed. The stated model was applied for assessing three technologies (adenosine, tissue plasminogen activator and mechanical thrombectomy) which were available for projects call of Iranian health technology assessment department in order to determine applicability of the model for practical purpose. Results Nine criteria, including efficiency/effectiveness, safety, population size, vulnerable population size, availability of alternative technologies, cost effectiveness in other countries, budget impact, financial protection, quality of evidence, were extracted by the Iranian health technology assessment experts. The relative weights of these criteria were as follows 0.12, 0.2, 0.06, 0.08, 0.08, 0.13, 0.08, 0.09, and 0.15, respectively. Finally TOPSIS pilot model was designed by three health technologies and nine criteria relative weights. Results showed that, the applicability of the stated model was suitable and as the pilot testing, tissue plasminogen activator was the first priority, adenosine was second and mechanical thrombectomy was third for performing health technology assessment by the Iranian ministry of health and medical education. Conclusion According to the results of this study, this model with nine effective criteria and their relative weights and in combination with TOPSIS approach could be used with suitable applicability by health technology assessment department in deputy of curative affairs and food and drug organization for determination of research priorities in health technology assessment.


Background
Acceptance of health technologies in health systems within the last decades has experienced some intensive changes. The belief in abiding with the acceptance of health technologies within 1960s and 1970s gradually was replaced by an increasing suspicion of society towards the credibility of advanced health technologies. Nowadays, politicians and the public have an ambiguous opinion about the use of a new health technology [1].
Within recent years, the use of health technologies for diagnosis and treatment of diseases experienced considerable and accelerated growth. The proper use of these technologies may considerably help in the diagnosis and treatment of diseases. On the other hand, unlimited and unrestricted entry of these technologies may result in induced demand by service providers [2].
Given, the number of health technologies which required assessment was more than available resources; so all health technology assessment organizations must prioritize their research projects [3]. Various studies have been applied for this purpose worldwide until now, such as Canadian Drug and Health Technologies Organization (CADTH) which utilized the analytical hierarchy process (AHP) for priority setting of health technologies [4]. Another existing model in relation to priority and selection of health technologies was EVIDEM (Evidence and Value: Impact on Decision Making) model in Canada [5]. Nonetheless, there is no consensus on appropriate priority setting between health technology assessment organizations [3]. The main goal of the present study was to describe the designated pilot MCDM model for priority setting of health technology assessment in Iran.

Methods
In order to achieve the aforementioned goal, relevant articles until March, 2015, based on three separate strategies using MESH and free text were sought for using the Cochrane Library, Pub Med and Scopus. The criteria and sub criteria (in pre assessment and assessment phase, at this stage, the criteria of assessment phase were also included, on the assumption that there may be some of these criteria which can be used in pre assessment phase in Iranian health system context) obtained from systematic review were questioned by experts of health technology assessment context via questionnaire (these expert were selected from official committee of HTA in Iranian ministry of health and medical education and during of this research just 11 ones of them were available for inclusion). At first, to determine the content validity as pilot, experts of health technology assessment were provided with questionnaire in three forms, including (a.) determination of face validity for each item (as two-options: it is clear and demonstrative or it is not clear and demonstrative), (b.) determination of content validity rate (CVR) for each item (as three-options: it is necessary, useful but not necessary or not necessary) and (c.) determination of content validity index (CVI) for each item (as five-options: completely irrelevant, irrelevant, relatively irrelevant, relevant or completely relevant). Sub criteria from which the minimum content validity was obtained was also included in the final questionnaire phase. In this questionnaire, the importance of each of these sub criteria was questioned by experts in these five options context (very slightly important, slightly important, moderately important, important, and very important). For this purpose, a 75 % cut point was used. This implied that for each one of these sub criteria, 75 % of experts who voted important or very important in the process of health technology assessment priority setting were included in the analytic hierarchy process (AHP) phase for extraction of their relative weight through paired wise comparisons. In the analytical hierarchy process, relative weight was obtained by paired comparison matrix. For a consistent paired comparison matrix, weight was calculated through normalization of elements of each column. However, if the matrix was inconsistent, to obtain the weight, four major methods were introduced, including 1 -least squares, 2 -least logarithm squares, 3 -Eigen vector, 4 -approximate methods [6]. The first three methods involve tedious calculations, thus some approximate methods have been proposed which are of lower accuracy (that is acceptable) and involve less and simpler calculations. In this study, geometric mean method was used. The geometric mean of elements of each row was calculated; later, the obtained vector was normalized to obtain the weight vector. Inconsistency rate of each matrix A was calculated based on the following steps: In this formula I.I.R. denotes the random matrix inconsistency index. This matrix was calculated for matrices whose numbers were completely randomly selected. In each matrix, the product of dividing the inconsistency index by random matrix inconsistency matrix with the same dimension will be a suitable criterion for judgment about matrix inconsistency which is called inconsistency ratio. If this number is smaller or equal to 0.1, system consistency is acceptable; otherwise the judgments must be revised [6].
After extraction of relative weights based on the above procedure, TOPSIS priority setting model was designed. Essential principles of this method are as follows: a. Appropriateness of each index must be uniform in ascending (descending) order (whatever r ij is higher, appropriateness also higher or vice versa) so that the best existing value of an index denotes its ideal and the worst existing value thereof shows negative ideal for it. b. The interval of an option from ideal (or negative ideal) may be calculated as Euclidean distance (from second exponent) or as the sum total of absolute value of linear distances (known as block gaps) that is dependent on the exchange and replacement rate among indices.
First step: this involves the conversion of existing decision making matrix to a matrix descaled using following formula: Formula (3): Second step: this involves creating weighted descaled matrix by assumption of vector W as input to the algorithm.
Formula (4): Where N D is a matrix having the score of indices therein descaled and comparable and W n × n is a diametrical matrix having only its main elements as nonzero.
Third step: This involves determination of ideal solution and negative ideal solution For ideal option A + and negative ideal A -, the following definition is provided: Formula (5): n js referred to the costg j Forth step: This involves calculation of separation size (distance).
The distance between option i and ideals based on Euclidean method is as follows: Fifth step: This involves calculation of relative closeness of A i to the ideal solution. This relative closeness is defined as follows: Formula (7): As observed, if And will have: And in case: Therefore, whatever A i is closer to the ideal solution A + , value of cl i will be closer to the unit.
Sixth step: This involves ranking of the options. The existing options of supposed problem can be ranked based on the descending order cl i + [7,8].
Ultimately, the first designed model was examined by pilot test using the last health technology assessment projects call of The Iranian Department of Health Technology Assessment, including 3 technologies (Adenosine, Tissue Plasminogen Activator (tPA), Mechanical Thrombectomy) and the results were provided to the experts for judgment.

Systematic review
After screening of retrieved papers via PRISMA framework, from 7012 papers, 40 were left included in the final phase. Criteria for selection of health technologies (in pre-assessment and also in assessment phase) were categorized into the six main themes such as 1 -health outcomes, 2 -disease and target population, 3 -technology alternatives, 4 -economic aspects, 5 -evidence and 6 -other factors. Maximum frequency in health outcomes related to health effects/benefits (8 studies), maximum frequency in disease and target population was related to disease severity (12 studies), maximum frequency in alternatives was related to number of alternatives (2 studies), maximum frequency in economic aspects was related to cost-effectiveness (15 studies), maximum frequency in evidence was related to quality of evidence (4 studies) and in other factors, max frequency was related to issues concerning the health system (11 studies) ( Table 1).

Phase 1: Content validity verification
Sub criteria obtained from the systematic review phase that were classified into six dimensions were designed as three questionnaires, including 54 sub criteria and one question for extraction of extra criteria. To determine the content validity of 11 health experts, technology assessment were questioned in three ways: (a.) determination of face validity for each item (as two-option: it is clear and demonstrative or it is not clear and demonstrative), (b.) determination of content validity ratio (CVR) for each item (as three-option: it is necessary, useful but not necessary or not necessary) and (c.) determination of content validity index (CVI) for each item (as five-option: completely irrelevant, irrelevant, relatively irrelevant, relevant or completely relevant). Firstly, content validity index (CVI) was calculated for each item such that out of 55 items, 21 items were obtained following a minimum of 0.79 which included health benefits, clinical impacts, efficacy/effectiveness, potential to increase the life, safety, population size, the number of potential beneficiaries, the impact on target groups, the size of vulnerable populations, the availability of alternative technologies, costeffectiveness, costs, budget impact, financial protection, the quality of evidence, number of evidence, the issues related to the health system, the issues related to the availability of expertise, equity in better access to health care and the question of additional criteria. Shortly following this, content validity ratio was calculated for each item such that, out of 54 sub criteria, all questioned experts had consensus on the necessity of only 9 sub criteria (health benefits, clinical benefits, efficacy/effectiveness, safety, cost-effectiveness, costs, technology price, quality of the evidence and issues related to rate availability of expertise) in this framework and obtained a score higher than score of Lawshe table for 11 experts. Whereas, Lawshe clarified that each item with more than half the total population of people had consensus on its necessity, and had a level of content validity (to the same rate of people that voted to its necessity) [9].
Thus, in addition to 9 sub criteria that had obtained the required score of CVR, a sub criteria greater than 50 % of people (including 50 % itself and higher) voted to its necessity and a CVI greater than 0.79 was obtained, kept and included into the final questionnaire (for assessment of their importance). These sub criteria were as follows: clinical impacts, efficiency/effectiveness, individual health benefits, safety, population size, number of potential beneficiaries, effect on targeted group, vulnerable population size, availability of alternative technologies, quality of evidence, number of evidence, budget impact, cost effectiveness, financial protection, issues related to health system and availability rate of experience and specialty and additional question for other criteria. Based on face validity, according to the opinion of experts, 4 sub criteria including individual health benefits, clinical impacts, efficiency/effectiveness, and effect on targeted group due to their overlap were all combined in the efficiency/effectiveness sub criteria. In addition, in the same manner, sub criteria of number of potential beneficiaries was combined with the population size, costs combined with cost effectiveness while number of evidence was renamed as the adequacy of the evidence. Thus, in the final questionnaire, 13 sub criteria were included based on 6 dimensions of conceptual model of study (Table 2).  Other criteria -Phase 2: Determining importance of each sub criteria Thirteen obtained sub criteria in previous phase were included into the final questionnaire phase based on the same process, 9 sub criteria, including Efficiency/effectiveness, safety, Population size, vulnerable population size, availability of alternative technologies, budget Impact, cost effectiveness of technology in other countries, financial protection, quality of evidence that obtained a greater than 75 % importance as viewpoint of questioned experts, were included in the paired wise comparisons questionnaire as the final 9 effective criteria in the priority setting of health technology assessment and this finalized the extraction of relative weight (Table 3). To assess the reliability of the final questionnaire, data obtained from each sub criterion were entered into SPSS software as Likert spectrum and the Cronbach's alpha was calculated with its value at 0.697. This demonstrated an acceptable reliability of the final questionnaire.
Moreover, no consensus was made between questioned experts in relation to the additional criteria individually mentioned by the experts. These criteria include Ethical issues, burden of disease, information value, the elasticity of demand for technology, procedures and methods of diagnosis and treatment, the external dependence of the use of technology, the development and application of technology in the world and the indication of technology.

Phase 3: Identification of relative weights of final criteria
Nine final effective criteria selected based on the opinion of experts were designed during the first and second phases as paired comparisons questionnaire and in another case they were presented to nine questioned experts in the first and second phases (at this phase, two experts of the previous phases were not available) in order to extract the relative weight of each one of the 9  Less or equal to 60 % final criteria. After filling the 9 questionnaires aforementioned, the information of these questionnaires was entered into an excel sheet as 9 matrixes (9*9) and to extract the final weight and equalizing the data of these 9 matrixes, a final matrix was formed such that in each one of its entries was included the geometric mean of its corresponding entries in 9 matrixes (9*9). The relative weigh of each criterion was extracted from the final formed matrix through normalization of geometric mean of each line thereof. Later, the obtained inconsistency ratio for the system was calculated, which was less than the range of 0.1 (IR = 0.03) and demonstrated consistency between extracted weights (Table 4).

Phase 4: Piloting the priority setting model
According to the inquiry from health technology assessment department of ministry of health, and medical education, 3 health technologies that were requested for health technology assessment, were selected and TOP-SIS priority setting pilot model was implemented on them. These technologies included the following: a. Adenosine b. Tissue Plasminogen Activator (tPA) c. Mechanical Thrombectomy Furthermore, to fill the decision matrix for execution of model pilot, values 1 to 5 were designed for each one of 9 final criteria, and data related thereto were extracted from available evidence for the three pilot technologies: (Tables 5, 6, 7, 8, 9, 10, 11, 12 and 13).

The data obtained from evidence about three abovementioned technologies
Adenosine for the treatment of supraventricular tachycardia [11] Status of efficiency/effectiveness criteria: This medication with a dose of 12 mg quickly and effectively terminated proximal episodes of the supraventricular tachycardia (relative code = 5) [12]. Status of safety criteria: The use of this medication resulted in a slightly bad side effect such as confusion, nausea, redness of face etc. (relative code = 4) [13]. Status of population size criteria: Supraventricular tachycardia prevalence was 2.25 cases in a general 1000-persons population. Therefore, the size of target population of this medication in Iran, considering its 80 million population was about 180 thousand peoples (relative code = 4) [11]. Status of vulnerable population size criteria: The probability of supraventriclar tachycardia in women was 2 times more than men, and majorly in pregnancy and fertility ages (within 15-30 years) (relative code = 4) [11]. Status of alternative technologies availability criteria: Verapamile is the alternative medication that has equal efficiency and relatively equal safety and mostly available in the health centers (relative code = 2) [11,13]. Status of quality of evidence criteria: Concerning this medication, there is a systematic review study of randomized controlled studies of meta-analysis type with a high quality (relative code = 5) [13]. Status of cost effectiveness criteria: The data about this criterion was unspecified (relative code = 1). Status of budget impact criterion: The data about this criterion was unspecified (relative code = 1). Status of financial protection criterion: The data about this criterion was unspecified (relative code = 1).
Tissue plasminogen activator (tPA) for the treatment of stroke [14] Status of efficiency/effectiveness criteria: The use of this medication resulted in a 0.07 increase of lifetime together with 0.24 increase of QALY (relative code = 3) [15]. Status of safety status: In comparison to placebo, the use of this technology had higher risk of intracranial hemorrhage (relative code = 3) [16]. Status of population size criteria: prevalence of stroke in west countries was about 12000 per million peoples [17]. The prevalence and incidence of stroke in Iran was significantly higher than western countries [18]. Thus, in Iranian population, assuming 80 million people, the population size under impact of this disease will be higher than one million people (relative code = 5). Status Table 6 Coding of Safety Criteria (as qualitative estimation or using quantitative indicators available in the safety context): Improvement rate in reduction of risk or incidence of side effects arising out of utilization of technology in the target population   [16]. Status of quality of evidence criteria: In relation to this technology, there was a high quality randomized multicenter controlled trial (elative code = 3) [16]. Status of cost effectiveness criteria: The incremental cost effectiveness ratio for all patients consuming this medication was about 6255 Dollars per QALY. This value was obtained for USA. Considering her per capital GDP in 2014 as equal to 53000$. This value is rather highly cost effective (relative code = 5) [15]. Status of budget impact criteria: The data in relation to this criterion was unspecified (relative code = 1). Status of financial protection criteria: The data in relation to this criterion was unspecified (relative code = 1).
Mechanical thrombectomy for the treatment of stroke [20] Status of efficiency/effectiveness criteria: This technology resulted in a 0.68 increase of QALY (relative code = 1) [21]. Status of safety criteria: This technology was safe which could demonstrate higher rate of symptomatic bleeding than medicinal methods (relative code = 3) [20]. Status of population size criteria: Prevalence of stroke in west countries was about 12000 per million peoples [17]. The Prevalence and incidence of stroke in Iran was significantly higher than western countries [18]. Thus, in Iranian population, assuming 80 million people, the population size under impact of this disease will be higher than one million people (relative code = 5). Status of vulnerable population size criteria: The incident of stroke was mostly prevalent among Iranian women (within 51-53 %) and majorly occurred within seventh decade of life (relative code = 4) [19]. Status of alternative technologies availability criteria: This method was the only    therapeutic method for patients with incident of stroke who were not treated by clot solving medications, since the use of this method resulted in failure with respect to them (during first 3-4 h after this occurrence) (relative code = 5) [20]. Status of quality of evidence criteria: In relation to this technology, there was health technology report from randomized r controlled trial with high quality (elative code = 5) [20]. Status of cost effectiveness criteria: The incremental cost effectiveness ratio for all patients taking this medication was about 16000 Dollars per QALY. This value was obtained for USA and considering per capital GDP of this country in 2014, which equaled 53000$, this value is rather highly cost effective (relative code = 5) [21]. Status of budget impact criteria: The data in relation to this criteria was unspecified (relative code = 1). Status of financial protection criterion: The data in relation to this criterion was unspecified (relative code = 1).

Final TOPSIS pilot model
Finally TOPSIS pilot model was designed by 3 health technologies and relative weights. The results showed that tissue plasminogen activator treatment was the first priority, adenosine was second and mechanical thrombectomy was third in performing health technology assessment by Iranian ministry of health and medical education (Table 14). Finally, the pilot model was presented to an internal panel which was made up of 6 experts of health technology assessment and was approved.

Discussion
Generally, this study which was applied based on the combined model of analytic hierarchy process and TOP-SIS, was presented for the first time in the world, in the health technology assessment priority setting context. The differences and similarities between this model and main model of Canadian Agency for Drugs and Technologies in Health are explained in terms of finalized criteria in this model. The criteria used in this model which extracted using a systematic review which included a number of alternatives and access such as, budget impact, clinical impact, controversial nature of technology, disease burden, economic impact, moral and legal issues, quality and level of evidences, expected level of interests, time period of assessment and its effect on health and clinical medicine policies, variety in direction for use and application of technology. The criteria used in the model of this study included efficiency/effectiveness, safety, population size, vulnerable population size, availability of alternative technologies, cost effectiveness in other countries, budget impact, financial protection, and quality of evidences. Efficiency/effectiveness, availability of alternative technologies, quality of evidences and budget impact as common items and controversial nature of technology, disease burden, economic impact, moral and legal issues, expected level of interests, time period of assessment and its effect on health and clinical medicine policies, variety in direction for use and application of technology in Canadian model and safety, population size, vulnerable population size, cost effectiveness in other countries and financial protection in the model of this study were assumed as differences. In relation to multi criteria decision making model used in this study, it is noteworthy that in Canadian model, overall relative weight extraction, as well as, final priority setting were performed based on analytic hierarchy process, whilst in the model of this study, only relative weight was extracted for determination of exact weight using final purpose which implied that, health technology assessment priority setting in Iran, through paired comparisons and no judgment of experts, were applied on the final model. This subject was designed in order to adhere to evidence based medicine (EBM) and its evidence pyramid using TOPSIS model and ascertain the use of codes extracted from evidence related to the technology, which can prevent the fully and unlimited interventions of medical beneficiary groups for selection of health technologies that are assumed as the stewardship scope.
In fact, out of 11 criteria acquired from systematic review phase of Canadian model and mentioned earlier, 6 criteria including disease burden, clinical impact, alternative technologies, budget impact, economic impact and available evidence were selected as the main criteria by health technology assessment experts of that country.   literature review as well as qualitative study was used in the model design, followed by a test for health technology assessment priority setting. To test the model and method, 3 round Delphi model was implemented using electronic questionnaire. Accordingly, Delphi technique was invented as a method for making consensus in the health technology assessment priority setting for Latvia. Criteria affecting this model included health benefit, evidence, assessment scheduling, expected level of policy makers' benefits, social, legal and moral concepts [22]. Two criteria of health benefit and evidence were common criteria in the model designed in the present study.
The designed model of this study may be compared to the health technology assessment priority setting model in Netherlands. In this model, various methods were used for classification, scoring and weighting the policy making criteria, including actual disease burden, potential health benefit, number of patients, direct intervention costs for each patient, financial consequences of intervention implementation over time and its impact on the health system policies [23]. Three criteria, including health benefit, number of patients and financial consequences were assumed as the three common criteria in the model designed for this study. Overall, the model designed in the present study in comparison to previous models in this context had innovation in the light of effective criteria because of its additional safety, vulnerable population size, cost effectiveness of technology in other countries and financial protection to the previous models. In addition, as mentioned before, technically this is the first time that a combined model of analytic hierarchy process and TOPSIS has been used in this relation.

Conclusion
According to the results of this study, this model with nine effective criteria and their relative weights and in combination with TOPSIS approach could be used with suitable applicability by health technology assessment department in deputy of curative affairs and food and drug organization for determination of research priorities in health technology assessment. Considering the increasing growth of health technologies, ministry of health and medical education, using this model, can allocate its research budgets to health technologies assessment which are of priority and importance to the health system and society. It may promote the final impact of health technology assessment reports on the macro health policies and optimize the allocation of financial resources of health sector.

Limitations
This model is not applicable for prioritization of orphan drugs.