Data envelopment analysis efficiency in the public sector using provider and customer opinion: An application to the Spanish health system

Measuring the relative efficiency of a finite fixed set of service-producing units (hospitals, state services, libraries, banks,...) is an important purpose of Data Envelopment Analysis (DEA). We illustrate an innovative way to measure this efficiency using stochastic indexes of the quality from these services. The indexes obtained from the opinion-satisfaction of the customers are estimators, from the statistical view point, of the quality of the service received (outputs); while, the quality of the offered service is estimated with opinion-satisfaction indexes of service providers (inputs). The estimation of these indicators is only possible by asking a customer and provider sample, in each service, through surveys. The technical efficiency score, obtained using the classic DEA models and estimated quality indicators, is an estimator of the unknown population efficiency that would be obtained if in each one of the services, interviews from all their customers and all their providers were available. With the object of achieving the best precision in the estimate, we propose results to determine the sample size of customers and providers needed so that with their answers can achieve a fixed accuracy in the estimation of the population efficiency of these service-producing units through the use of a novel one bootstrap confidence interval. Using this bootstrap methodology and quality opinion indexes obtained from two surveys, one of doctors and another of patients, we analyze the efficiency in the health care system of Spain.


Introduction
Data Envelopment Analysis (DEA) has become a widely used technique to compare the efficiency of serviceproducing units because it easily handles the multiple outputs characteristic of public sector production, is nonparametric and does not require input price data [34]. The popular application is with deterministic information and classic DEA models [3,10] [1,37] or [25]), universities and research institutes [26], government [17], public libraries [22,23], schools [35], public transport services [15] or banks [28].
In recent years, some approaches consider the opinion of the customer to be crucial for measuring DEA efficiency in public services ( [5,16]). Consumer satisfaction-opinion surveys are a common tool for building opinion indexes, which measure the quality of the service, and to be used as output variables in DEA ( [20,27,31,32,35,40,41] and [42]).
Besides customer opinion-satisfaction (output), provider opinion-satisfaction (input) is also a fundamental protagonist in public service-producing units (SPUs), our decisionmaking units. The provider's positive opinion-satisfaction concerning the service offered may result in a better customer opinion concerning the service received. For instance, a city's public bus drivers with a good opinion of their salary, timetable, partners, driven vehicles, etc., may influence a better satisfaction-opinion of the travellers using the service. One way to measure the service quality offered is with satisfaction-opinion indexes obtained through survey samples carried out with the providers. In Tapia et al. [40][41][42] the protagonism of the opinion of the providers is not considered, that is, the inputs are deterministic and only the outputs are estimated using a customer sample. Introducing the opinion-satisfaction of the providers as estimated indexes increases the field of application of this work, where opinion-satisfaction indexes, estimated from a sample of providers (inputs) and a sample of customers (outputs), are used as information to measure the DEA efficiency of the public services. To do so, it is necessary to determine the sampling design, the sample size and the estimators of the satisfaction-opinion indexes of both the providers and the customers in each SPU.
In practice, this stochastic input and output information is available in many services, as it is increasingly common to conduct opinion-satisfaction surveys for both providers and customers. For instance, the sample data used in the Spanish health system application of Section 5.
The efficiency obtained with DEA models, using the opinion indexes estimated with the survey answer of samples of providers and consumers as data, will be an estimation of the population DEA efficiency. This population efficiency is an unknown non-evaluable parameter, since it would be necessary to use the indexes obtained with the opinion of all the customers and all the providers of all the services as data and this is a census of the entire population [40]. This statistical analysis of the DEA efficiency gives rise to the problem of determining the sample size of customers and providers needed to guarantee an a priori fixed accuracy in the estimation, with confidence interval, of the DEA efficiency of each public service, which is the object of our investigation. Liu et al. [30] considered the statistical analysis and sampling process as an important direction for handling the DEA. Ceyhan and Benneyan [7] investigated the impact of the sample size on the measures of efficiency when the DEA problem was carried out on values that include such estimated proportions as defect, satisfaction, mortality, or adverse event rates estimated from samples. Nevertheless, they do not propose a solution to the necessary sample size in order to control the error in the measures of efficiency. The problem of calculating the sample size of customers necessary when the objective is to estimate, with a fixed precision in the estimate error, the population efficiency in a finite set of public services using stochastic data output (customer opinionsatisfaction indexes) and known (non-stochastic) data input was resolved in Tapia et al. [40,42].
In this paper, we propose a solution to determine the customer and provider sample sizes needed to estimate the DEA efficiency in public services with bootstrap confidence intervals and a fixed accuracy. These intervals capture the random variations introduced in the DEA analysis by using outputs and inputs estimated with a sample. So far, Simar and Wilson's [38,39] methodology has been the most common for measuring efficiency with bootstrap confidence intervals in such public services as health care ( [11]), universities and research institutes [4], government [6], public libraries [29], schools [19], tourism [2], banks [28] or public transport services [21]. The problem posed by Simar and Wilson is different from ours because the stochastic character of the inputs and outputs is different. In Simar and Wilson, the stochastic character of the input and output information comes from considering the set of available SPUs, , as a sample from an infinite population and the sample observations in are realizations of identically, independently distributed (iid) random variables with a probability density function with support over can produce , [14]. In this work, the set of available SPUs are the only units whose efficiency we wish to evaluate. Therefore, we do not consider them a sample as in the classical DEA models, [9]. The stochastic character of our output and input information comes from the fact that these data are opinion-satisfaction indexes that it is necessary to estimate by taking independent random samples in each SPU, and the probability, in the statistical model, depends on the sample design used in each SPU.
In Shwartz [37], a random sample of patients arriving for health care services were bootstrap resampled to obtain data input and interval estimates of the DEA efficiency. These interval estimates are conservative (large) and the problem of the patient sample size necessary to obtain the desired accuracy in the estimation of the DEA efficiency is not resolved.
In other approaches, where samples in public services are used to estimate the data (Charles2014), the efficiency is estimated with stochastic DEA models. These models, which use LP problems subject to constraints defined in terms of probability, are also called chance-constrained problems. The deterministic characterization of "efficient" is then changed by the probabilistic characterization "probably efficient". A vast number of papers show a wide range of uses of chance-constrained programming (CCP), including [8,9,12,33] or [43]. One of the main advantages of the technique presented in this study is its simplicity, as we use the original DEA models with constant (CCR) and variable returns-to-scale (BCC), that is, linear programming (LP) problems subject to deterministic constraints [3,10].
In this paper, we therefore examine the implications of input and output data estimated with a provider and a customer sample, respectively, for the performance analysis of public services using DEA analysis techniques. In Section 2, we examine the nature of the problem. Section 3 presents a theoretical method to determine the customer and provider sample size needed to estimate the population DEA efficiency with bootstrap confidence interval, which is examined in Section 4. Section 5 includes the empirical application in the Spanish health system we have undertaken. Section 6 contains our conclusions. All the software is our own elaboration using MATLAB and it is included in Appendix II.

Nature of the problem
We consider a finite fixed set of SPUs, service-producing units, one provider interview to estimate inputs and one customer interview to estimate outputs. For example, in each hospital of a homogeneous set of , it is possible to estimate the general satisfaction of the personnel, or their annual time of formation, or other personal opinion indexes (input data) by interviewing a sample of the personnel. It is also possible to estimate the satisfaction of the patients attention they have received, the human and material resources of the hospital, etc., by interviewing to a patient sample (output data). The main problem approached in this paper is to estimate the unknown parameter population DEA efficiency. The population efficiency, or census efficiency, is unknown because, in order to know it, it would be necessary to interview all the providers and all the customers (census) of each one of the public services and, with this population data, to obtain the opinion indexes and use them as input/output data in the classic LP model CCR or BCC of Table 1; in real applications, these censuses are completely non-viable.
The error made in estimating the input/output data with samples is transferred to the DEA efficiency estimation. In this study, we propose a methodology that guarantees a reasonable quality of the DEA efficiency estimation.
Formally, in the jth service-producer unit (SPU for short), we consider the finite provider population U j U 1j U N x j j of size and the customer population where is the answer of the kth provider of the jth SPU to the ith opinion provider item, i.e., each customer W hj 1 is a quantitative vector where is the answer of the hth customer of the jth SPU to the rth customer opinion item. The population input and output data are opinion indexes obtained as a function of the provider and customer population answers, in general: These functions and can be of any type, with or without weights, whenever they admit an interpretation of population opinion-satisfaction indexes as input and population opinion-satisfaction indexes as output.
Using the data X j Y j 1 in the Table 1 models, we obtain the population DEA efficiency scores 1 ... . If it were possible to carry out a census and to know these population indexes, the information would be fixed or deterministic and the character of the problem would not be stochastic.
A lack of knowledge concerning the population information input and output makes the taking of samples to estimate 1 ... necessary. In the SPU , let U 1j U n x j j U j and W 1j W n y j j W j be random samples of size and of the provider and customer populations, respectively. To obtain the estimators of the input indexes Eq. 5 and the output indexes Eq. 6, the same functions and are used with the random samples: Having observed the provider and customer sample answers u kj 1 1 and w hj 1 1 , respectively, Eqs. 9 and 10 are the estimates of the input and output indexes, respectively; that is, the values that the estimators Eqs. 7 and 8 take with the sample answers of the customers and providers: Using X j Y j 1 in the Table 1 models, we obtain the estimators 1 of the population efficiency scores 1 , on the understanding that the DEA model is maximized, or minimized, with the data x j y j 1 in order to obtain the estimation of the estimator . Therefore, our statistical model corresponds to independent, random samples in each SPU, that is, the sample space is 1 , where samples u kj of size and samples w hj of size in SPU , and the probability depends on the sample design used.
The first objective of this paper, having fixed 0 1 and 0 1 , is to obtain the provider and customer sampling size, and , respectively, to estimate and and the estimator such that: The second objective is to determine a confidence interval for the populational efficiency, in each SPU, with a fixed accuracy.

How many providers and customers need to be interviewed?
The provider and customer sample size problem Eq. 11 can only be analytically resolved in the case of one input and one output and the CCR model. A rigorous proof of all the results are provided in the Appendix I.

CCR model with one provider and one customer opinion index
Let us consider these assumptions:  , let be the sampling size in the SPU , such that 6 1 (17) and be the sampling size such that 1 .

CCR or BCC model with two or more provider and/or customer opinion indexes
In this section, we report on our simulation study to check that Theorem 2 and Corollary 1 also work in the BCC model with two or more estimated provider and/or customer opinion indexes.
If we consider items (the same in all SPUs) to estimate the provider opinion indexes (inputs) 1 with 1 , Remark 1 calculates the sample size necessary to achieve 6 We propose to determine the provider sample size in the SPU as 1 ... (25) i.e., if we consider items (the same in all SPUs) to estimate the customer opinion indexes (outputs) the provider sample size in the SPU is determined as where is the sample size necessary to achieve 6

Simulation study
We use the [13] health center data ( where , and , are the original value doctor, nurse and outpatient, inpatient of the jth health center, columns 2, 3, 4 and 5 of Table 2, respectively. We consider the population mean to the simulated answers to the provider and customer items in the jth center, 1 ... 12, to simulate the population inputs and outputs, columns 3, 4, 5 and 6 in Table 3:   1  2  1  1  1  2   1  2  1  1  1  2 The last two columns in Table 3 show the population efficiency scores CCR and BCC with output orientation.
To check the relation between sample size, estimation of the input/output indexes and estimation of the DEA efficiency, using Theorem 2 and Corollary 1 and these simulated population data, we follow the next steps: i. In the jth health center, 1 12, a previous simple random sample without replacement of 25 providers  Table 4 shows the sampling sizes, and , obtained for the jth health center in the last iteration, for the two values of and 0.1. In health center 3 or 6, the customer sample size increases up to 6 times when fixing a maximum 0.2 to 0.1. The probabilities approximated with Eq. 29 take the value one for all the health centers, the two values of , the CCR-O or BCC-O model and with 0.1. Therefore, the confidence intervals for the population  [13] efficiency score , obtained with the samples of the size of Table 4 and Corollary 1, are very conservative. However, in the next section, we will see that these same sample sizes allow less conservative bootstrap DEA efficiency confidence intervals to be obtained.

Description of the bootstrap efficiency confidence interval technology
Bootstrap uses resampling to estimate the value of a parameter of a population ( [18]). For the problem suggested in Section 2, we propose bootstrap resampling of the samples of the provider and customer answers to the opinion item to obtain confidence intervals for the population efficiencies, following these steps: i. Having fixed and a probability 1 , we determine the sample sizes and , in the SPU , using Corollary 1, Remark 1 and Eqs. 25

Simulation study
To illustrate the bootstrap efficiency confidence interval methodology and to check the estimation quality of the DEA population efficiency obtained, a simulation was performed using the population model simulated from Table 3. First, we fixed 0.2 and the confidence 1 0.9 to calculate the provider and customer sample size to estimate the input and output data; supposing a simple random sample without replacement, these sample sizes are columns 2 and 3 in Table 4.  Table 5 shows the approximate confidence of the bootstrap intervals, output orientation DEA models. We observe that the control of the coverage level 1 leads to the achievement of the confidence of the bootstrap efficiency interval required by the experimenter.
The amplitude of the bootstrap efficiency confidence intervals is analysed with the approximation of the expected value of the bounds 1000 =1 1000 and 1000 =1 1000 .
In conclusion, the bootstrap efficiency confidence interval methodology has the advantage that, after determining the provider and customer sample size using Corollary 1, Remark 1 and Eqs. 25 and 26, the experimenter can achieve the confidence required, 1 , to estimate the population efficiency.

Application to the Spanish health system
This section provides an empirical analysis of health production for Spain's 18 Autonomous Communities (CCAA).
Spain's Health Ministry has, for some time, been compiling the statistic "Health Barometer" (HB), where a group of individuals is selected in each CCAA, and a questionnaire is carried out to test the health system. One of the survey question blocks take the opinion of the individual concerning the attention provided by the doctors of primary attention and pediatrics in Spain with the following items: Output-oriented BCC model Either from your personal experience or in your own opinion, we would like you to evaluate the following aspects of the public health service, concerning the attention provided by the GP or the paediatrician. Do so using the scale of 1 to 10, where 1 means 'totally unsatisfactory' and 10 means 'totally satisfactory'.
(P-1) The attention received from the healthcare personnel (P-2) The time dedicated by the doctor to each patient (P-3) The confidence and security that the doctor transmits (P-4) The information received concerning your health problem (P-5) The time between making the appointment and the visit to the doctor.
A principal components analysis (PCA) is carried out over these 5 items.The PCA is a statistical technique for reducing the variable dimensionality of the dataset, minimizing information loss. It does so by creating new uncorrelated variables that successively maximize the explained variance of the dataset ( [24]). The first component (PCA1) obtained, interpretable as the size of satisfaction, explains 98.1% of the variability. The sample mean of the answers to this component in every CCAA is our estimated output index, interpreted as the patient mean satisfaction with the CCAA's healthcare system (column 9 in Table 7).
The input data is estimated using the results of the "Survey on the current situation of GPs in Spain", carried out by the Spanish Medical Colleges Organization (OMC) in 2015 on the population of Spanish GPs and paediatricians. We use the following items:  Table 7 show the   Table 8 shows the results of the estimation point i and bootstrap interval of the population efficiency scores, with confidence 1 0.9, in each of Spain's CCAAs, considering variable returns-to-scale and output orientation. The CCAAs in which the hypothesis of an efficient public health service (DEA efficiency equal one) is rejected are 3, 4, 5, 8, 9 10, 11, 12, 13, 17 . The CCAAs which are benchmark for the rest are 2 6, 7, 14 , 15 , because the upper and lower confidence interval bounds have value one. In general, the efficiency in all CCAAs is good, the inferior bound of the confidence interval is superior to 0.9 in all the cases, except in the CCAA 12 , according to our results, the CCAA with the least efficient health service.

Conclusions
The approach presented in this paper provides a step towards producing valid estimates of technical efficiency in public services, using provider and customer satisfactionopinion indexes estimated with samples. These indexes measure the service quality from the perspective of both the provider and the customer.
We have developed statistical results for comparing the efficiencies of public services. These results are novel in the sense that: (i) We resolve the problem of determining the customer and provider sample size necessary to estimate the opinion indexes and the population efficiency with an accuracy fixed a priori; (ii) We build confidence intervals for the population DEA efficiency using bootstrap replicates of the providers sample and the customers sample in each public service; (iii) It is possible to achieve the level of confidence of the bootstrap efficiency confidence interval required by the experimenter (iv) The DEA models used are the original linear programming models; (iv) The new approach can be readily implemented. As far as we know, the approach of this paper has not been attempted in the literature.
While this study provides a useful methodology to measure public service efficiency, its limitations should also be acknowledged. First, the results can only be proven analytically for the CCR case, with one input and one output. Second, to obtain the input and output data, a provider and customer opinion survey is necessary. Finally, the presented methodology also has to allow deterministic inputs and/or output data to be considered.
This statistical efficiency methodology, used with opinion indexes from doctors and patients allows us to conclude that, in Spain, there are ten autonomous communities that can improve their efficiency and five autonomous communities that act as benchmarks for the rest.
The results of this paper can have other important implications in practice. It can also be used to measure the efficiency in all services with users and providers, for instance, markets, health care, banks, casinos, schools, universities or public transport.