Abstract
Measuring the relative efficiency of a finite fixed set of service-producing units (hospitals, state services, libraries, banks,...) is an important purpose of Data Envelopment Analysis (DEA). We illustrate an innovative way to measure this efficiency using stochastic indexes of the quality from these services. The indexes obtained from the opinion-satisfaction of the customers are estimators, from the statistical view point, of the quality of the service received (outputs); while, the quality of the offered service is estimated with opinion-satisfaction indexes of service providers (inputs). The estimation of these indicators is only possible by asking a customer and provider sample, in each service, through surveys. The technical efficiency score, obtained using the classic DEA models and estimated quality indicators, is an estimator of the unknown population efficiency that would be obtained if in each one of the services, interviews from all their customers and all their providers were available. With the object of achieving the best precision in the estimate, we propose results to determine the sample size of customers and providers needed so that with their answers can achieve a fixed accuracy in the estimation of the population efficiency of these service-producing units through the use of a novel one bootstrap confidence interval. Using this bootstrap methodology and quality opinion indexes obtained from two surveys, one of doctors and another of patients, we analyze the efficiency in the health care system of Spain.
Similar content being viewed by others
Avoid common mistakes on your manuscript.
1 Introduction
Data Envelopment Analysis (DEA) has become a widely used technique to compare the efficiency of service-producing units because it easily handles the multiple outputs characteristic of public sector production, is non-parametric and does not require input price data [34]. The popular application is with deterministic information and classic DEA models [3, 10], for instance, in health care [1, 37] or [25]), universities and research institutes [26], government [17], public libraries [22, 23], schools [35], public transport services [15] or banks [28].
In recent years, some approaches consider the opinion of the customer to be crucial for measuring DEA efficiency in public services ([5, 16]). Consumer satisfaction-opinion surveys are a common tool for building opinion indexes, which measure the quality of the service, and to be used as output variables in DEA ([20, 27, 31, 32, 35, 40, 41] and [42]).
Besides customer opinion-satisfaction (output), provider opinion-satisfaction (input) is also a fundamental protagonist in public service-producing units (SPUs), our decision-making units. The provider’s positive opinion-satisfaction concerning the service offered may result in a better customer opinion concerning the service received. For instance, a city’s public bus drivers with a good opinion of their salary, timetable, partners, driven vehicles, etc., may influence a better satisfaction-opinion of the travellers using the service. One way to measure the service quality offered is with satisfaction-opinion indexes obtained through survey samples carried out with the providers. In Tapia et al. [40,41,42] the protagonism of the opinion of the providers is not considered, that is, the inputs are deterministic and only the outputs are estimated using a customer sample. Introducing the opinion-satisfaction of the providers as estimated indexes increases the field of application of this work, where opinion-satisfaction indexes, estimated from a sample of providers (inputs) and a sample of customers (outputs), are used as information to measure the DEA efficiency of the public services. To do so, it is necessary to determine the sampling design, the sample size and the estimators of the satisfaction-opinion indexes of both the providers and the customers in each SPU.
In practice, this stochastic input and output information is available in many services, as it is increasingly common to conduct opinion-satisfaction surveys for both providers and customers. For instance, the sample data used in the Spanish health system application of Section 5.
The efficiency obtained with DEA models, using the opinion indexes estimated with the survey answer of samples of providers and consumers as data, will be an estimation of the population DEA efficiency. This population efficiency is an unknown non-evaluable parameter, since it would be necessary to use the indexes obtained with the opinion of all the customers and all the providers of all the services as data and this is a census of the entire population [40]. This statistical analysis of the DEA efficiency gives rise to the problem of determining the sample size of customers and providers needed to guarantee an a priori fixed accuracy in the estimation, with confidence interval, of the DEA efficiency of each public service, which is the object of our investigation. Liu et al. [30] considered the statistical analysis and sampling process as an important direction for handling the DEA. Ceyhan and Benneyan [7] investigated the impact of the sample size on the measures of efficiency when the DEA problem was carried out on values that include such estimated proportions as defect, satisfaction, mortality, or adverse event rates estimated from samples. Nevertheless, they do not propose a solution to the necessary sample size in order to control the error in the measures of efficiency. The problem of calculating the sample size of customers necessary when the objective is to estimate, with a fixed precision in the estimate error, the population efficiency in a finite set of public services using stochastic data output (customer opinion-satisfaction indexes) and known (non-stochastic) data input was resolved in Tapia et al. [40, 42].
In this paper, we propose a solution to determine the customer and provider sample sizes needed to estimate the DEA efficiency in public services with bootstrap confidence intervals and a fixed accuracy. These intervals capture the random variations introduced in the DEA analysis by using outputs and inputs estimated with a sample. So far, Simar and Wilson’s [38, 39] methodology has been the most common for measuring efficiency with bootstrap confidence intervals in such public services as health care ([11]), universities and research institutes [4], government [6], public libraries [29], schools [19], tourism [2], banks [28] or public transport services [21]. The problem posed by Simar and Wilson is different from ours because the stochastic character of the inputs and outputs is different. In Simar and Wilson, the stochastic character of the input and output information comes from considering the set of available SPUs, Sn, as a sample from an infinite population and the sample observations in Sn are realizations of identically, independently distributed (iid) random variables with a probability density function with support over \(P=\left \lbrace (x,y) | x \text { can produce } y\right \rbrace \), [14]. In this work, the set of available SPUs are the only units whose efficiency we wish to evaluate. Therefore, we do not consider them a sample as in the classical DEA models, [9]. The stochastic character of our output and input information comes from the fact that these data are opinion-satisfaction indexes that it is necessary to estimate by taking independent random samples in each SPU, and the probability, in the statistical model, depends on the sample design used in each SPU.
In Shwartz [37], a random sample of patients arriving for health care services were bootstrap resampled to obtain data input and interval estimates of the DEA efficiency. These interval estimates are conservative (large) and the problem of the patient sample size necessary to obtain the desired accuracy in the estimation of the DEA efficiency is not resolved.
In other approaches, where samples in public services are used to estimate the data (Charles2014), the efficiency is estimated with stochastic DEA models. These models, which use LP problems subject to constraints defined in terms of probability, are also called chance-constrained problems. The deterministic characterization of “efficient” is then changed by the probabilistic characterization “probably efficient”. A vast number of papers show a wide range of uses of chance-constrained programming (CCP), including [8, 9, 12, 33] or [43]. One of the main advantages of the technique presented in this study is its simplicity, as we use the original DEA models with constant (CCR) and variable returns-to-scale (BCC), that is, linear programming (LP) problems subject to deterministic constraints [3, 10].
In this paper, we therefore examine the implications of input and output data estimated with a provider and a customer sample, respectively, for the performance analysis of public services using DEA analysis techniques. In Section 2, we examine the nature of the problem. Section 3 presents a theoretical method to determine the customer and provider sample size needed to estimate the population DEA efficiency with bootstrap confidence interval, which is examined in Section 4. Section 5 includes the empirical application in the Spanish health system we have undertaken. Section 6 contains our conclusions. All the software is our own elaboration using MATLAB and it is included in Appendix A.
2 Nature of the problem
We consider a finite fixed set of L SPUs, service-producing units, one provider interview to estimate m inputs and one customer interview to estimate s outputs. For example, in each hospital of a homogeneous set of L, it is possible to estimate the general satisfaction of the personnel, or their annual time of formation, or other personal opinion indexes (input data) by interviewing a sample of the personnel. It is also possible to estimate the satisfaction of the patients attention they have received, the human and material resources of the hospital, etc., by interviewing to a patient sample (output data). The main problem approached in this paper is to estimate the unknown parameter population DEA efficiency. The population efficiency, or census efficiency, is unknown because, in order to know it, it would be necessary to interview all the providers and all the customers (census) of each one of the L public services and, with this population data, to obtain the opinion indexes and use them as input/output data in the classic LP model CCR or BCC of Table 1; in real applications, these censuses are completely non-viable.
The error made in estimating the input/output data with samples is transferred to the DEA efficiency estimation. In this study, we propose a methodology that guarantees a reasonable quality of the DEA efficiency estimation.
Formally, in the jth service-producer unit (SPUj for short), we consider the finite provider population \({\mathbf {U_{j}}}= \left \{{{\mathbf U}}_{{\mathbf 1}{\mathbf j}},{\dots } ,{{\mathbf U}}_{{{\mathbf N}}_{{\mathbf {x_{j}}}{\mathbf j}}} \right \} \) of size \(N_{x_{j}}\) and the customer population \( {\mathbf {W_{j}}}=\left \{{{\mathbf W}}_{{\mathbf 1}{\mathbf j}},{\dots } ,{\mathbf W}_{\mathbf {{N_{y_{j}}} j}} \right \} \) of size \(N_{y_{j}}\). Each provider is a quantitative vector \({{\mathbf U}}_{{\mathbf {kj}}} =\left (U_{k1j},{\dots } ,\ U_{kmj}\ \right )\), where Ukij is the answer of the kth provider of the jth SPU to the ith opinion provider item, i.e., each customer \({{\mathbf W}}_{{\mathbf { hj}}}=\left (W_{h1j},{\dots } ,\ W_{hsj}\ \right )\) is a quantitative vector where Whrj is the answer of the hth customer of the jth SPU to the rth customer opinion item. The population input and output data are opinion indexes obtained as a function of the provider and customer population answers, in general:
These functions f and g can be of any type, with or without weights, whenever they admit an interpretation of m population opinion-satisfaction indexes as input and s population opinion-satisfaction indexes as output.
Using the data \({\left \lbrace \left (\mathbf {X_{j}}, \ {{{\mathbf Y}}}_{\mathbf j}\right )\right \rbrace }_{j=1,{\dots } ,L}\) in the Table 1 models, we obtain the population DEA efficiency scores \( {\left \lbrace \varphi _{j} \right \rbrace }_{j=1, ..., L} \). If it were possible to carry out a census and to know these population indexes, the information would be fixed or deterministic and the character of the problem would not be stochastic.
A lack of knowledge concerning the population information input and output makes the taking of samples to estimate \({\left \lbrace \varphi _{j} \right \rbrace }_{j=1, ..., L} \) necessary. In the SPUj, let \( \left \{{{\mathbf U}}_{{\mathbf 1}{\mathbf j}},{\dots } ,{{\mathbf U}}_{\mathbf {n_{x_{j}} j}} \right \}\subset {\mathbf {U_{j}}} \) and \( \left \{{{\mathbf W}}_{{\mathbf 1}{\mathbf j}},{\dots } ,{{\mathbf W}}_{\mathbf {n_{y_{j}} j}}\right \} \subset {\mathbf {W_{j}}}\) be random samples of size \(N_{x_{j}}\) and \(N_{y_{j}}\) of the provider and customer populations, respectively. To obtain the estimators of the input indexes Eq. 5 and the output indexes Eq. 6, the same functions f and g are used with the random samples:
Having observed the provider and customer sample answers \(\left \lbrace {\mathbf {u_{kj}}= \left (u_{k1j}, \dots , \ u_{kmj}\right ) }\right \rbrace _{ {k=1,{\dots } ,{n_{x_{j}}}} }\) and \(\left \lbrace {\mathbf {w_{hj}}=\left (w_{h1j}, \dots , \ w_{hsj}\right ) }\right \rbrace _{ {h=1,{\dots } ,{n_{y_{j}}}} }\), respectively, Eqs. 9 and 10 are the estimates of the input and output indexes, respectively; that is, the values that the estimators Eqs. 7 and 8 take with the sample answers of the customers and providers:
Using \({\left ({\widehat {{\mathbf X}}}_{\mathbf j}, \ {\widehat {{\mathbf Y}}}_{\mathbf j}\right )}_{j=1,{\dots } ,L}\) in the Table 1 models, we obtain the estimators \( {\left ({{\widehat {\varphi } }_{j}}\right )}_{j=1,\dots ,L} \) of the population efficiency scores \( {\left ({{\varphi }_{j}}\right )}_{j=1,\dots ,L} \), on the understanding that the DEA model is maximized, or minimized, with the data \({\left ({\widehat {{\mathbf x}}}_{\mathbf j}, \ {\widehat {{\mathbf y}}}_{\mathbf j}\right )}_{j=1,{\dots } ,L}\) in order to obtain the estimation \( {{\widehat {\omega } }_{j}}\) of the estimator \( {{\widehat {\varphi } }_{j}}\).
Therefore, our statistical model \( \left ({\Omega }, P \right ) \) corresponds to independent, random samples in each SPU, that is, the sample space is \({\Omega }={\prod }_{j=1}^{L}{\Omega }_{j}\), where \( {\Omega }_{j}=\left \{ \text {samples} \mathbf {u_{kj}} \text {of size} n_{x_{j}} \text { and} \text { samples} \mathbf {w_{hj}} \text {of size} n_{y_{j}}\right .\) \(\left . \text {in SPU}_{j} \right \rbrace \), and the probability P depends on the sample design used.
The first objective of this paper, having fixed δ ∈ (0,1) and \(\alpha \in \left (0,1\right )\), is to obtain the provider and customer sampling size, \(N_{x_{j}}\) and \(N_{y_{j}}\), respectively, to estimate \({\widehat {X}}_{j} \) and \({\widehat {Y}}_{j} \) and the estimator \({\widehat {\varphi }}_{j} \) such that:
The second objective is to determine a confidence interval for the populational efficiency, in each SPU, with a fixed accuracy.
3 How many providers and customers need to be interviewed?
The provider and customer sample size problem Eq. 11 can only be analytically resolved in the case of one input and one output and the CCR model. A rigorous proof of all the results are provided in the Appendix I.
3.1 CCR model with one provider and one customer opinion index
Let us consider these assumptions:
-
C1
Fixed L SPUs.
-
C2
One provider population opinion index \(\left \lbrace X_{j}\right \rbrace _{j=1,{\dots } ,L}\) as input and one customer population opinion index \(\left \lbrace Y_{j}\right \rbrace _{j=1,{\dots } ,L}\) as output. We consider \(\left \lbrace {\widehat {X}}_{j}\right \rbrace _{j=1,{\dots } ,L}\) and \(\left \lbrace {\widehat {Y}}_{j}\right \rbrace _{j=1,{\dots } ,L}\) to be the corresponding estimators.
-
C3
CCR model Eq. 1 with output orientation (CCR-O).
Let \({(Z_{j}=\frac {Y_{j}}{X_{j}})}_{j=1,\ {\dots } ,\ L}\) and \({({\widehat {Z}}_{j}=\frac {{\widehat {Y}}_{j}}{{\widehat X}_{j}})}_{j=1,{\dots } ,L}\). In this situation and with the given notation, the population efficiency obtained with the CCR-O model and its estimator are:
Having fixed p ∈ (0,1), we thus consider the sets of Ω:
The following Lemmas 1, 2 and 3 are used to prove Theorem 1 which, in this particular case, establishes the relation between the accuracy of the provider and customer opinion estimation indexes and that of the population CCR-O efficiency estimation.
Lemma 1
Under assumptions C1, C2 and C3, with the given notation, fixed p ∈ (0,1) and m such that \( Z_{m}={max}_{j=1,{\dots } ,\ L}\left \{Z_{j}\right \}\), then:
Lemma 2
Under assumptions C1, C2 and C3, with the given notation, fixed p ∈ (0,1), m and Zm defined as in Lemma 1 and l such that :
Consider the set of Ω
then:
m = l implies Al = B
m≠l implies Al ∩ Am ⊂ B
Lemma 3
Under assumptions C1, C2 and C3, with the given notation, fixed p ∈ (0,1) and for any \( j=1,\ {\dots } ,\ L\) we have that
Theorem 1
Let us consider the assumptions C1, C2 and C3. Having fixed p ∈ (0,1) and \(\alpha \in \left (0,1\right )\), if \(\ P\left (A_{j}\right )\ge 1-\alpha \ \ \forall j=1,{\dots } ,L\), then
Lemma 4 is an instrumental result, used to prove Theorem 2 and Corollary 1, which establishes how to determine the provider and customer sample size in each SPU, so the CCR-O efficiency estimator has the precision fixed in Eq. 11.
Lemma 4
Under the hypotheses of Theorem 1, if
then
Theorem 2
Let us consider the assumptions C1, C2 and C3. Having fixed p ∈ (0,1) and \(\alpha \in \left (0,1\right )\), for every \(j=1,\ \dots , \ L\), let \(N_{x_{j}}\) be the sampling size in the SPUj, such that
and \(N_{y_{j}}\) be the sampling size such that
then
Corollary 1
Let us consider the assumptions C1, C2 and C3. Having fixed δ ∈ (0,1) and \(\alpha \in \left (0,1\right )\), for every \(j=1,\dots ,L\), let \(N_{x_{j}}\) be the sampling size in the SPUj, such that
and \(N_{y_{j}}\) be the sampling size such that
then
Remark 1 gives the explicit formula to obtain the sample size under the usual simple random sample without replacement sample design.
Remark 1
If the design in each SPU is simple random sampling without replacement, and the output (i.e. input) is the mean of all the answers of the population to a survey item, then the sampling size \(n_{\theta _{j}}\) that it verifies
is ([36])
with \(n_{oj}=\frac {{\tau }^{2}_{1-{{\alpha }_{1} }/{2}}}{{\left (p\theta _{j}\right )}^{2}}{{\sigma }_{\theta _{j}}}^{2}\) and \({\tau }_{1-{\alpha _{1} }/{2}}={\phi }^{-1}\left (1-{\alpha _{1} }/{2}\right )\), where \({{\sigma }_{y_{j}}}^{2}\) is the population variance and ϕ the normal standard distribution function.
3.2 CCR or BCC model with two or more provider and/or customer opinion indexes
In this section, we report on our simulation study to check that Theorem 2 and Corollary 1 also work in the BCC model with two or more estimated provider and/or customer opinion indexes.
If we consider m items (the same in all SPUs) to estimate the m provider opinion indexes (inputs) \(\left (X_{1j},\ {\dots } ,\ X_{mj}\right )\) with \(\left ({\widehat {X}}_{1j}, \dots , {\widehat {X}}_{mj}\right ) \), Remark 1 calculates the sample size \(n_{x_{ij}}\) necessary to achieve
We propose to determine the provider sample size \(N_{x_{j}}\) in the SPUj as
i.e., if we consider s items (the same in all SPUs) to estimate the s customer opinion indexes (outputs) the provider sample size \(N_{y_{j}}\) in the SPUj is determined as
where \(n_{y_{rj}}\) is the sample size necessary to achieve
3.2.1 Simulation study
We use the [13] health center data (Table 2) to simulate a population model: in the jth health center a population size of providers and customers, \(N_{x_{j}} \) and \(N_{y_{j}}\), are generated from a random uniform distribution, according to the intervals \( \left [10000, 50000 \right ] \) and \( \left [30000, 80000 \right ] \), respectively. For each provider, we generate two item answers \( \left (U_{k1j}, U_{k2j} \right ) \), i.e., for each customer \( \left (W_{h1j}, W_{h2j} \right ) \), from a bivariate normal distribution as
where docj, nursj and outj, inpj are the original value doctor, nurse and outpatient, inpatient of the jth health center, columns 2, 3, 4 and 5 of Table 2, respectively. We consider the population mean to the simulated answers to the provider and customer items in the jth center, j = 1,...,12, to simulate the population inputs and outputs, columns 3, 4, 5 and 6 in Table 3:
The last two columns in Table 3 show the population efficiency scores CCR and BCC with output orientation.
To check the relation between sample size, estimation of the input/output indexes and estimation of the DEA efficiency, using Theorem 2 and Corollary 1 and these simulated population data, we follow the next steps:
-
i.
In the jth health center, \( j=1,\dots ,12 \), a previous simple random sample without replacement of 25 providers \(\left (n_{x_{j}}^{(0)}=25 \right ) \) is taken to estimate the two inputs \( \left ({\widehat {x }}_{1j}^{(0)}, {\widehat {x }}_{2j}^{(0)} \right ) \), and their variances \(\left ({\widehat {\sigma }}_{1jx}^{2(0)}, {\widehat {\sigma }}_{2jx}^{2(0)}\right ) \) using the sample means and the sample quasi-variances, respectively, i.e., we estimate the two outputs \(\left ({\widehat {y }}_{1j}^{(0)}, {\widehat {y }}_{2j}^{(0)} \right ) \) and their variances, \( \left ({\widehat {\sigma }}_{1jy}^{2(0)}, {\widehat {\sigma }}_{2jy}^{2(0)} \right ) \) with a previous simple random sample without replacement of 25 customers \( \left (n_{y_{j}}^{(0)}=25 \right ) \).
-
ii.
Fixed δ = 0.1 or 0.2 and 1 − α = 0.9 as in Corollary 1, and with the estimates of step i., the sample sizes, \(N_{x_{j}}\) and \(N_{y_{j}}\), are determined using Eqs. 23, 25 and 26.
-
iii.
In the jth health center, the simple random samples without replacement of size \(N_{x_{j}}\) and \( n_{y_{j}} \) are taken and the inputs \(\left ({\widehat {x }}_{1j} ,{\widehat {x }}_{2j}\right ) \) and outputs \(\left ({\widehat {y }}_{1j} ,{\widehat {y}}_{2j}\right ) \) are estimated. With the data \(\left \lbrace {\left ({\widehat {x }}_{1j} ,{\widehat {x }}_{2j},{\widehat {y }}_{1j} ,{\widehat {y}}_{2j}\right ) }\right \rbrace _{j=1,...,12}\), the estimated efficiencies \({\left \lbrace {\widehat {\omega }}_{j}\right \rbrace }_{j=1,...,12}\) are obtained, maximizing the LP model Eqs. 3 or 4 with output orientation.
-
iv.
One thousand iterations of step iii. are carried out obtaining, for the jth health center, 1000 estimated efficiency scores \( {\left \lbrace {{\widehat {\omega }}_{j}^{(k)}}\right \rbrace }_{k=1, ..., 1000} \) and 1000 intervals
$$ H_{j}^{(k)}=\left[{\widehat{\omega }}_{j}^{(k)}-\delta,min\left\{{\widehat{\omega }}_{j}^{(k)}+\delta,1\right\}\right]; \ k=1,{\dots} ,1000 $$(28) -
v.
The probability \(P\left (\left \vert {}{\widehat {\varphi {}}}_{j}-{\varphi {}}_{j}\right \vert {}\leq \delta \right ) \) is approximated by calculating
$$ C_{j}=\frac{1}{1000}{\sum}^{1000}_{k=1}{I_{\left( {\varphi }_{j}\in H_{j}^{(k)}\right)}} $$(29)
Table 4 shows the sampling sizes, \(N_{x_{j}}\) and \(n_{y_{j}} \), obtained for the jth health center in the last iteration, for the two values of δ and α = 0.1. In health center 3 or 6, the customer sample size increases up to 6 times when fixing a maximum δ = 0.2 to 0.1.
The probabilities \(P\left (\left \vert {}{\widehat {\varphi {}}}_{j}-{\varphi {}}_{j}\right \vert {}\leq {}\delta {}\right ) \) approximated with Eq. 29 take the value one for all the health centers, the two values of δ, the CCR-O or BCC-O model and with α = 0.1. Therefore, the confidence intervals for the population efficiency score φj, obtained with the samples of the size of Table 4 and Corollary 1, are very conservative. However, in the next section, we will see that these same sample sizes allow less conservative bootstrap DEA efficiency confidence intervals to be obtained.
4 Description of the bootstrap efficiency confidence interval technology
Bootstrap uses resampling to estimate the value of a parameter of a population ([18]). For the problem suggested in Section 2, we propose bootstrap resampling of the samples of the provider and customer answers to the opinion item to obtain confidence intervals for the population efficiencies, following these steps:
-
i.
Having fixed δ and a probability 1 − α, we determine the sample sizes \(N_{x_{j}}\) and \(N_{y_{j}}\), in the SPUj, using Corollary 1, Remark 1 and Eqs. 25 and 26.
-
1.
In the SPUj, we take a provider and a customer simple random samples without replacement \( \left \lbrace \mathbf {u_{kj}}= \left (u_{k1j}, ..., u_{kmj} \right )\right \rbrace _{k=1, ..., n_{x_{j}}} \) and \( \left \lbrace \mathbf {w_{hj}}= \left (w_{h1j}, ..., w_{hsj} \right )\right \rbrace _{h=1, ..., n_{y_{j}}} \), respectively, to estimate the provider and customer opinion indexes \( \left ({\widehat {{x}}}_{1j},\dots ,{\widehat {{x}}}_{mj} \right ) \) and \( \left ({\widehat {{y}}}_{1j},\dots ,{\widehat {{y}}}_{sj} \right ) \), respectively, for example, with the sample means:
$$ \begin{array}{@{}rcl@{}} && \widehat x_{ij}=\frac{{\sum}_{k=1}^{n_{x_{j}}} u_{kij}}{n_{x_{j}}}; i=1, ..., m; j=1, ...L \end{array} $$(30)$$ \begin{array}{@{}rcl@{}} && \widehat y_{rj}=\frac{{\sum}_{h=1}^{n_{y_{j}}} w_{hrj}}{n_{y_{j}}}; r=1, ..., s; j=1, ...L. \end{array} $$(31) -
ii.
In the SPUj, we take a bootstrap sample with replacement \( \left \lbrace \mathbf {u_{kj}}^{*}\right \rbrace _{k=1, ..., n_{x_{j}}} \) from \( \left \lbrace \mathbf {u_{kj}}\right \rbrace _{k=1, ..., n_{x_{j}}} \), i.e., \( \left \lbrace \mathbf {w_{hj}}^{*}\right \rbrace _{h=1, ..., n_{y_{j}}} \) of size \(n_{y_{j}} \) from \( \left \lbrace \mathbf {w_{hj}}\right \rbrace _{h=1, ..., n_{y_{j}}} \), with which we obtain the bootstrap version of the m inputs, \(\mathbf {{\widehat {x}}_{j}^{*}}=\left (\widehat {x}_{1j}^{*}, ..., \widehat {x}_{mj}^{*}\right )\), and s outputs, \(\mathbf {{\widehat {y}}_{j}^{*}}=\left (\widehat {y}_{1j}^{*}, ..., \widehat {y}_{sj}^{*}\right )\). With the data \( {\left \lbrace \left (\widehat {x}_{1j}^{*}, ..., \widehat {x}_{mj}^{*},\widehat {y}_{1j}^{*}, ..., \widehat {y}_{sj}^{*}\right )\right \rbrace }_{j=1,{\dots } ,L}\) and the DEA model of Table 1, we obtain the bootstrap version of the estimated DEA efficiency, \( {\left ({\widehat {\omega }^{*}}_{j}\right ) }_{j=1,...,L} \).
-
iii.
The step iii. is repeated B times and the B bootstrap versions of the estimated DEA efficiency for the SPUj, j = 1,...,L, are \({\left \{{\widehat {\omega }}^{*(b)}_{j}\right \}}_{b\text {=1,{\dots } ,\ }\ B}\).
-
2.
In the SPUj, the observed percentile bootstrap confidence interval for the population efficiency score φj is obtained, having fixed a coverage intention of level \(\text {1-}\alpha ^{\prime }\), as
$$ I_{j}=\left[ {\widehat{\omega}}^{*(\alpha^{\prime}/2)}_{j},{\widehat{\omega}}^{*(1-\alpha^{\prime}/2)}_{j}\right] $$(32)where \({\widehat {\omega }}^{*(\alpha )}_{j}\) is the α-percentile of the B values \({\left \{{\widehat {\omega }}^{*(b)}_{j}\right \}}_{b\text {=1,{\dots } ,\ }\ B}\).
4.1 Simulation study
To illustrate the bootstrap efficiency confidence interval methodology and to check the estimation quality of the DEA population efficiency obtained, a simulation was performed using the population model simulated from Table 3.
First, we fixed δ = 0.2 and the confidence 1 − α = 0.9 to calculate the provider and customer sample size to estimate the input and output data; supposing a simple random sample without replacement, these sample sizes are columns 2 and 3 in Table 4.
The steps ii.-v. are iterated 1000 times, fixing the confidence of the bootstrap efficiency interval \(1-\alpha ^{\prime }=0.9 \ \text {or} \ 0.95\) and 2000 resampling Bootstraps (B = 2000).
The confidence of the bootstrap interval for the DEA efficiency in the SPUj is approximated with:
where \( {\left \{{I_{j}}^{(k)}=[{I_{j}}^{\left (k\right )L},\ {I_{j}}^{(k)U}]\right \}}_{k=1, ..., 1000} \) are the 1000 bootstrap efficiency confidence intervals obtained in step v.
Table 5 shows the approximate confidence of the bootstrap intervals, output orientation DEA models. We observe that the control of the coverage level \(1-\alpha ^{\prime }\) leads to the achievement of the confidence of the bootstrap efficiency interval required by the experimenter.
The amplitude of the bootstrap efficiency confidence intervals is analysed with the approximation of the expected value of the bounds
Table 6 shows the approximation of the expected values of the bounds of the bootstrap efficiency confidence interval, considering the BCC model with output orientation. If we look at the SPUs in which the expected efficiency confidence interval contains the one, {1, 2, 4, 7, 10, 11, 12}, these SPUs coincide with the efficient population units (value 1 in column 9 from Table 3). As expected, the increase in the trust \(1-\alpha ^{\prime } \) of the interval Bootstrap leads to an increase in the amplitude.
In conclusion, the bootstrap efficiency confidence interval methodology has the advantage that, after determining the provider and customer sample size using Corollary 1, Remark 1 and Eqs. 25 and 26, the experimenter can achieve the confidence required, \(1-\alpha ^{\prime }\), to estimate the population efficiency.
5 Application to the Spanish health system
This section provides an empirical analysis of health production for Spain’s 18 Autonomous Communities (CCAA).
Spain’s Health Ministry has, for some time, been compiling the statistic ”Health Barometer” (HB), where a group of individuals is selected in each CCAA, and a questionnaire is carried out to test the health system. One of the survey question blocks take the opinion of the individual concerning the attention provided by the doctors of primary attention and pediatrics in Spain with the following items:
Either from your personal experience or in your own opinion, we would like you to evaluate the following aspects of the public health service, concerning the attention provided by the GP or the paediatrician. Do so using the scale of 1 to 10, where 1 means ’totally unsatisfactory’ and 10 means ’totally satisfactory’.
(P-1) The attention received from the healthcare personnel
(P-2) The time dedicated by the doctor to each patient
(P-3) The confidence and security that the doctor transmits
(P-4) The information received concerning your health problem
(P-5) The time between making the appointment and the visit to the doctor.
A principal components analysis (PCA) is carried out over these 5 items.The PCA is a statistical technique for reducing the variable dimensionality of the dataset, minimizing information loss. It does so by creating new uncorrelated variables that successively maximize the explained variance of the dataset ([24]). The first component (PCA1) obtained, interpretable as the size of satisfaction, explains 98.1% of the variability. The sample mean of the answers to this component in every CCAA is our estimated output index, interpreted as the patient mean satisfaction with the CCAA’s healthcare system (column 9 in Table 7).
The input data is estimated using the results of the “Survey on the current situation of GPs in Spain”, carried out by the Spanish Medical Colleges Organization (OMC) in 2015 on the population of Spanish GPs and paediatricians. We use the following items:
-
(C1) Workload as number of patients attended per day, answering 1 if the workload is normal or low (inferior to 40 patients) and 0 if the workload is high.
-
(C2) Occupation of the team of doctors, answering 1 if the occupation is normal or low and 0 if it is high.
-
(C3) Time dedicated to ongoing training.
The sample means of the doctors, answers to these items in each CCAA estimate the three opinion indexes used as inputs: the proportion of doctors with a normal or low workload \(\left (\mathbf {\overline {C1}} \right ) \), the proportion of teams of doctors with a normal or low occupation \(\left (\mathbf {\overline {C2}} \right ) \), and the mean time doctors dedicated to ongoing training \(\left (\mathbf {\overline {C3}} \right ) \). From our point of view, an increase in the value of these input indexes in the population of doctors of a CCAA would lead to a bigger satisfaction in the population of patients attended in the same CCAA. Columns 2 and 3 of Table 7 show the population and sample size of doctors (i.e., columns 7 and 8 of patients).
Table 8 shows the results of the estimation point \(\left ({\widehat {{\mathbf \omega }}}_{{\mathbf i}}\right )\) and bootstrap interval of the population efficiency scores, with confidence \( 1-\alpha ^{\prime }=0.9 \), in each of Spain’s CCAAs, considering variable returns-to-scale and output orientation. The CCAAs in which the hypothesis of an efficient public health service (DEA efficiency equal one) is rejected are {3, 4, 5, 8, 9 10, 11, 12, 13, 17}. The CCAAs which are benchmark for the rest are {2 6, 7, 14 , 15}, because the upper and lower confidence interval bounds have value one. In general, the efficiency in all CCAAs is good, the inferior bound of the confidence interval is superior to 0.9 in all the cases, except in the CCAA {12}, according to our results, the CCAA with the least efficient health service.
6 Conclusions
The approach presented in this paper provides a step towards producing valid estimates of technical efficiency in public services, using provider and customer satisfaction-opinion indexes estimated with samples. These indexes measure the service quality from the perspective of both the provider and the customer.
We have developed statistical results for comparing the efficiencies of public services. These results are novel in the sense that: (i) We resolve the problem of determining the customer and provider sample size necessary to estimate the opinion indexes and the population efficiency with an accuracy fixed a priori; (ii) We build confidence intervals for the population DEA efficiency using bootstrap replicates of the providers sample and the customers sample in each public service; (iii) It is possible to achieve the level of confidence of the bootstrap efficiency confidence interval required by the experimenter (iv) The DEA models used are the original linear programming models; (iv) The new approach can be readily implemented. As far as we know, the approach of this paper has not been attempted in the literature.
While this study provides a useful methodology to measure public service efficiency, its limitations should also be acknowledged. First, the results can only be proven analytically for the CCR case, with one input and one output. Second, to obtain the input and output data, a provider and customer opinion survey is necessary. Finally, the presented methodology also has to allow deterministic inputs and/or output data to be considered.
This statistical efficiency methodology, used with opinion indexes from doctors and patients allows us to conclude that, in Spain, there are ten autonomous communities that can improve their efficiency and five autonomous communities that act as benchmarks for the rest.
The results of this paper can have other important implications in practice. It can also be used to measure the efficiency in all services with users and providers, for instance, markets, health care, banks, casinos, schools, universities or public transport.
References
Almeida Botega L, Andrade MV, Guedes GR (2020) Brazilian hospitals? performance: an assessment of the unified health system (SUS). Health Care Manag Sci 23:443–452
Assaf A (2010) Bootstrapped scale efficiency measures of UK airports. J Air Transport Manag 16(1):42–44
Banker RD, Charnes A, Cooper WW (1984) Some models for estimating technical and scale inefficiencies in data envelopment analysis. Manag Sci 30(9):1078–1092
Barra C, Zotti R (2016) Measuring efficiency in higher education: an empirical study using a bootstrapped data envelopment analysis. Int Adv Econ Res 22(1):11–33
Bayraktar E, Tatoglu E, Turkyilmaz A, Delen D, Zaim S (2012) Measuring the efficiency of customer satisfaction and loyalty for mobile phone brands with DEA. Expert Syst Appl 39(1):99–106
Benito B, Solana J, Moreno MR (2014) Explaining efficiency in municipal services providers. J. Product. Anal. 42(3):225–239
Ceyhan ME, Benneyan JC (2014) Handling estimated proportions in public sector data envelopment analysis. Ann Oper Res 221(1):107–132
Charles V, Kumar M (2014) Satisficing data envelopment analysis: an application to SERVQUAL efficiency. Measurement 51:71–80
Charnes A, Cooper WW (1959) Chance-constrained programming. Manag Sci 6(1):73–79
Charnes A, Cooper WW, Rhodes E (1978) Measuring the efficiency of decision making units. Eur J Oper Res 2(6):429–444
Chowdhury H, Zelenyuk V (2016) Performance of hospital services in Ontario: DEA with truncated regression approach. Omega 63:111–122
Cooper WW, Deng H, Huang Z, Li SX (2002) Chance constrained programming approaches to technical efficiencies and inefficiencies in stochastic data envelopment analysis. J Oper Res Soc 53(12):1347–1356
Cooper W. W., Seiford L. M., Tone K (2006) Introduction to data envelopment analysis and its uses: with DEA-solver software and references. Springer Science & Business Media
Cooper W. W., Seiford L. M., Zhu J (2011) Handbook on data envelopment analysis. Springer Science & Business Media
Daraio C, Diana M, Di Costa F, Leporelli C, Matteucci G, Nastasi A (2016) Efficiency and effectiveness in the urban public transport sector: a critical review with directions for future research. Eur J Oper Res 248(1):1–20
De Witte K, Geys B (2013) Citizen coproduction and efficient public good provision: Theory and evidence from local public libraries. Eur J Oper Res 224(3):592–602
Diewert WE (2011) Measuring productivity in the public sector: some conceptual problems. J Prod Anal 36(2):177–191
Efron B, Tibshirani RJ (1994) An introduction to the bootstrap. CRC press
Essid H, Ouellette P, Vigeant S (2014) Productivity, efficiency and technical change of Tunisian schools: a bootstrapped Malmquist approach with quasi-fixed inputs. Omega 42(1):88–97
Førsund FR (2017) Measuring effectiveness of production in the public sector. Omega 73:93–103
Gil Ropero A, Turias Dominguez I, Cerbán Jiménez MM (2019) Bootstrapped operating efficiency in container ports: a case study in Spain and Portugal. Ind Manag Data Syst 119(4):924–948
Hammond CJ (2002) Efficiency in the provision of public services: a data envelopment analysis of uk public library systems. Appl Econ 34(5):649–657
Hemmeter JA (2006) Estimating public library efficiency using stochastic frontiers. Public Finance Rev 34(3):328–348
Jobson JD (1992) Applied multivariate data analysis. Volume II: Categorical and Multivariate Methods. Springer
Kohl S, Schoenfelder J, Fügener A (2019) The use of Data Envelopment Analysis (DEA) in healthcare with a focus on hospitals. Health Care Manag Sci 22:245–286
Korhonen P, Tainio R, Wallenius J (2001) Value efficiency analysis of academic research. Eur J Oper Res 130(1):121– 132
Lee H, Kim C (2014) Benchmarking of service quality with data envelopment analysis. Expert Syst Appl 41(8):3761–3768
Li Y, Chiu Y, Lin T, Huang Y. Y. u. (2019) Market share and performance in Taiwanese banks: min/max SBM DEA. TOP 2:233–252
Liu ST, Chuang M (2009) Fuzzy efficiency measures in fuzzy DEA/AR with application to university libraries. Expert Syst Appl 36(2):1105–1113
Liu W, Wang Y-M, Lyu S (2017) The upper and lower bound evaluation based on the quantile efficiency in stochastic data envelopment analysis. Expert Syst Appl 85:14–24
Mayston DJ (2015) Analysing the effectiveness of public service producers with endogenous resourcing. J Prod Anal 44(1):115–126
Mayston DJ (2017) Data envelopment analysis, endogeneity and the quality frontier for public services. Ann Oper Res 250(1):185–203
Olesen OB, Petersen N (1995) Chance constrained efficiency evaluation. Manag Sci 41 (3):442–457
Ruggiero J (1996) On the measurement of technical efficiency in the public sector. Eur J Oper Res 90(3):553–565
Santín D, Sicilia G (2017) Dealing with endogeneity in data envelopment analysis applications. Expert Syst Appl 68:173–184
Särndal C-E, Swensson B, Wretman J (2003) Model assisted survey sampling. Springer Science & Business Media
Shwartz M, Burgess JF, Zhu J (2016) A DES based composite measure of quality and its associated data uncertainty interval for health care provider profiling and pay-for-performance. Eur J Oper Res 253 (2):489–502
Simar L, Wilson PW (1998) Sensitivity analysis of efficiency scores: How to bootstrap in nonparametric frontier models. Manag Sci 44(1):49–61
Simar L, Wilson PW (2000) A general methodology for bootstrapping in non-parametric frontier models. J Appl Stat 27(6):779–802
Tapia JA, Salvador B, Rodríguez JM (2018) Data envelopment analysis in satisfaction survey research: sample size problem. J Oper Res Soc 69:7,1096–1104
Tapia JA, Salvador B, Rodríguez JM (2019) Data envelopment analysis efficiency of public services: bootstrap simultaneous confidence region. SORT-stat Oper Res Trans:337–354
Tapia JA, Salvador B, Rodríguez JM (2020) Data envelopment analysis with estimated output data: Confidence intervals efficiency. Measurement 152(107364)
Zandkarimkhani S, Mina H, Biuki M (2020) A chance constrained fuzzy goal programming approach for perishable pharmaceutical supply chain network design. Ann Oper Res 295:425–452
Acknowledgements
The authors thank the Editor and two anonymous referees for their constructive comments and suggestions that have helped to improve the quality of the paper.
Funding
Open Access funding provided thanks to the CRUE-CSIC agreement with Springer Nature.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of Interests
Author 1 declares that he has no conflict of interest. Author 2 declares that he has no conflict of interest.
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Appendices
Appendix I
Proof of Lemma 1
The first equality is obtained by
and, dividing the upper and lower extremes of Eq. 35 by the lower and upper extremes of Eq. 36, respectively, we obtain
The second equality is obtained by clearing \({\widehat {Z}}_{j},\) and the third by dividing by Zm.
2.1 Proof of Lemma 2
If m = l, from Lemma 1, it follows that Al = B.
If m≠l, then \({\widehat {Z}}_{m}<{\widehat {Z}}_{l}\) and Zl<Zm and, if Al ∩ Am is verified, it follows that:
2.2 Proof of Lemma 3
Dividing the first inequality that defines the set B by \(\left (1+p\right )\) and multiplying by \(\left (1-p\right )\), we obtain:
Now, for \(j=1,{\dots } ,L\), as \( {\widehat {\varphi }}_{j}=\frac {{\widehat {Z}}_{j}}{{\widehat {Z}}_{l}} \), we have that:
Dividing the second inequality that defines the set B by (1 − p) and multiplying by (1 + p), we have:
Given that \(j=1,{\dots } ,L\): we get \({\widehat {\varphi }}_{j}\frac {{\widehat {Z}}_{l}(1+p)}{Z_{m}\left (1-p\right )}\le {\widehat {\varphi }}_{j}\frac {\left (1+p\right )^{2}}{\left (1-p\right )^{2}}\) and, as \( {\widehat {\varphi }}_{j}=\frac {{\widehat {Z}}_{j}}{{\widehat {Z}}_{l}}\), we get \( \frac {{\widehat {Z}}_{j}(1+p)}{Z_{m}\left (1-p\right )}\le {\widehat {\varphi }}_{j}\frac {\left (1+p\right )^{2}}{\left (1-p\right )^{2}}\).
As from Lemma 1, \(A_{j}=\left (\frac {{\widehat {Z}}_{j}(1-p)}{Z_{m}\left (1+p\right )}\le {\varphi }_{j}\le \frac {{\widehat {Z}}_{j}(1+p)}{z_{m}\left (1-p\right )}{\mathbf \ }\right )\), then, whenever Aj ∩ B happens, we have
2.3 Proof of Theorem 1
The first inequality follows from Lemma 3, the second from Lemma 2, and the last inequality follows because the events Aj are independent.
2.4 Proof of Lemma 4
If \({\widehat {\varphi {}}}_{j}\frac {\left (1-p\right )^{2}}{\left (1+p\right )^{2}}\leq {}{\varphi {}}_{j}\Longrightarrow {}{\widehat {\varphi {}}}_{j}\left (1-\frac {4p}{(1+p)^{2}}\right )\leq {}{\varphi {}}_{j}\Rightarrow {}{\widehat {\varphi {}}}_{j}-{\varphi {}}_{j}\leq {}\frac {4p}{(1+p)^{2}}{\widehat {\varphi {}}}_{j}\leq {}\frac {4p}{(1+p)^{2}}\); the last inequality follows from \(0\leq {}{\widehat {\varphi {}}}_{j}\leq {}1.\)
If \(min\left \{1,{\widehat {\varphi {}}}_{j}\frac {(1+p)^{2}}{\left (1-p\right )^{2}}\right \}=1\Longrightarrow {}{\widehat {\varphi {}}}_{j}\frac {\left (1+p\right )^{2}}{\left (1-p\right )^{2}}\geq {}1\Longrightarrow {}{\widehat {\varphi {}}}_{j}\geq {}\frac {\left (1-p\right )^{2}}{\left (1+p\right )^{2}}\Longrightarrow {}{\widehat {\varphi {}}}_{j}+\frac {4p}{(1+p)^{2}}\geq {}\frac {\left (1-p\right )^{2}}{\left (1+p\right )^{2}}+\frac {4p}{\left (1+p\right )^{2}}=1\) and \(min\left \{1,{\widehat {\varphi {}}}_{j}+\frac {4p}{(1+p)^{2}}\ \right \}=1\).
If \({\varphi {}}_{j}\leq {}min\left \{1,{\widehat {\varphi {}}}_{j}\frac {(1+p)^{2}}{\left (1-p\right )^{2}}\right \}={\widehat {\varphi {}}}_{j}\frac {(1+p)^{2}}{\left (1-p\right )^{2}}\)\(\Longrightarrow {}{\varphi {}}_{j}\leq {}{\widehat {\varphi {}}}_{j}\left (\frac {1}{1-\frac {4p}{\left (1+p\right )^{2}}}\right )\Longrightarrow {}{\varphi {}}_{j}\left (1-\frac {4p}{\left (1+p\right )^{2}}\right )\leq {}{\widehat {\varphi {}}}_{j}\Longrightarrow {}{\varphi {}}_{j}-{\widehat {\varphi {}}}_{j}\leq {}\frac {4p}{\left (1+p\right )^{2}}{\varphi {}}_{j}\leq {}\frac {4p}{(1+p)^{2}}\); the last inequality follows from 0 ≤ φj ≤ 1.
Proof of Theorem 2
The result follows from Theorem 1, Lemma 4 and \(P(A_{j})=P\left (\left \vert {\widehat {X}}_{j}-X_{j}\right \vert {}\leq p X_{j}\right ) \cap \left (\left \vert {\widehat {Y}}_{j}-Y_{j}\right \vert {}\leq {p}Y_{j}\right ))=P\left (\left \vert {\widehat {X}}_{j}-X_{j}\right \vert {}\leq p X_{j}\right ) P\left (\left \vert {\widehat {Y}}_{j}-Y_{j}\right \vert {}\leq {p}Y_{j}\right ).\)
Appendix II
The software to reproduce the methods is available in the following link https://uvaes-my.sharepoint.com/:f:/g/personal/jesus_tapia_uva_es/EpnYVaKm1OZMpKUyyrI-bS0BNUj7aDumIquTR7Ipxt422A?e=GvrDtG
This file contains the following programs:
-
–
Program to take survey sampling without replacement ``mas.m”
-
–
Program to take survey sampling with replacement ``mascon.m”
-
–
Programs to obtain the sample size: ``masnp.m” ``masn.m” ``masnvar.m”
-
–
Program to obtain efficiency bootstrap confidence intervals ``percboot2outp2inp2pob.m”
-
–
Program to obtain efficiency CCR ``ccrout.m”
-
–
Program to obtain efficiency BCC ``bccout.m”
-
–
Program to obtain the intervals and the results of the simulation ``CIE15ybootalfab01.m”
-
–
Simulation data ``datcoop2i2ost2pob.mat”
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Tapia, J.A., Salvador, B. Data envelopment analysis efficiency in the public sector using provider and customer opinion: An application to the Spanish health system. Health Care Manag Sci 25, 333–346 (2022). https://doi.org/10.1007/s10729-021-09589-7
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10729-021-09589-7