Quality of Service Ranking by Quantifying Partial Compliance of Requirements

Abstract

While there is not much discussion on the importance of formally describing and analysing quantitative requirements in the process of software construction; in the paradigm of API-based software systems it could be vital. Quantitative attributes can be thought as attributes determining the Quality of Service – QoS provided by a software component published as a service. In this sense, they play a determinant role in classifying software artifacts according to specific needs stated as requirements. In previous works we presented an efficient, and fully automatic, analysis technique for establishing Service Level Agreements – SLA. Such a proposal relays on describing QoS contracts as convex specifications, and compliance checking is performed by the application of an analysis algorithm based on state of the art techniques used in hybrid system verification. Such a technique succeeds in offering a procedure for determining SLA but fails in the more realistic scenario where, potentially, no service fully satisfies the requirements. In this scenario the running application may still prefer to invoke the service that offers the best chances of successfully executing with values for QoS attributes meeting the requirements satisfactorily.

In this work we propose and implement a metric for automatically quantifying partial satisfaction of QoS requirements, leading to a way of ranking services according to such notion of partial compliance.

Research partly supported by the European Unions Horizon 2020 research and innovation programme under the Marie Sklodowska-Curie grant agreement No 778233. Carlos G. Lopez Pombo’s research is supported by Universidad de Buenos Aires through grant UBACyT 20020170100544BA, and Consejo Nacional de Investigaciones Científicas y Técnicas through grant PIP 11220130100148CO.

1 Introduction

Distributed software resulting from paradigms such as Service-Oriented Computing (SOC) or the API’s economy is based on the idea of constructing software artifacts by composing services provided by third parties and registered in repositories. This envisages a generation of applications which, at run-time, are transparently reconfigured by the intervention of a dedicated middleware with the capability to bind a running application that has certain requirements to a service capable of fulfilling them, subject to the negotiation of a Service Level Agreement – SLA; in this way, services can collectively fulfill certain business goals [4].

Requirements can be formalised as contracts between software components, and satisfaction of such contracts is usually dealt with by checking whether a judgment of the form $Pr \vdash Rq$ holds or not, where $Pr$ is the provision contract and $Rq$ is the requirements contract. In the literature, functional requirements have been identified as those describing what the system has to do, while non-functional ones are assumed to characterise how the system develops the behaviour described by the functional ones.

In this work we focus on Quantitative requirements as a proper subclass of non-functional requirements of a software system. From our perspective, quantitative requirements formalise the admissible values of specific attributes that can be interpreted over a metric space [2]. Such attributes are referred to as Quantitative attributes. For practical reasons, the real numbers constitute a good candidate over which these can be interpreted, formalised and analysed.

In this sense, quantitative requirements might be used to characterise the Quality of Service – QoS provided by a software component available as service. While services may have the same functional behaviour, they might differ on their non-functional one (for example, a service may offer low-speed computation at a very low cost while another, functionally equivalent one, might be faster but more onerous); a motivation shared with other works like [14]. Therefore, quantitative requirements might be considered as a way to classify functionally equivalent services by their QoS.

In [12] we presented an efficient, and fully automatic, analysis procedure for checking a judgment of the shape $Pr \vdash Rq$ where $Pr$ is the provision contract and $Rq$ is the requirements contract, thus serving the purpose of establishing SLA. Such an approach guarantees the selection of a service satisfying the requirements of the executing application but it does not provide any insight when there is no service whose provision contract is fully compliant with the requirement contract. A geometrical interpretation of the judgment $Pr \vdash Rq$ expresses that all the satisfying values of real attributes of $Pr$ also satisfy $Rq$. Thus, a negative answer just means that there is at least one value satisfying $Pr$ and not satisfying $Rq$ pushing the application to abort its execution because there is no possible SLA on the required QoS.

In this paper we propose an automatic procedure¹ for evaluating partial compliance of QoS contracts by estimating what we call the inclusion ratio: the intersection of the set of satisfying values of the provision and requirement contracts interpreted relative to the size of the set of values satisfying the provision contract. This metric leads to a natural way of ranking services. While the problem of raking services according to their QoS has been studied and different approaches to tackle the problem have been proposed, in general, these are based on monitoring run-time behavior of specific attributes [1, 9] or by socially sharing the users’ opinion [11, 15] but, to the best of our knowledge, there is none focussing on QoS ranking where the QoS is formally specified and SLA is guaranteed through an automated procedure.

2 Formalisation and Analysis of Quantitative Attributes

The selection of a service requires establishing an SLA, part of which is the QoS agreement. Given a formal QoS requirement contract $Rq$; a service that declares a formal QoS provision contract $Pr$ will be a good candidate only if $Pr \implies Rq$ holds. In [12] we adopted monotone Satisfiability Modulo Convex (SMC) formulae [13] as specification language and proposed a method to check QoS compliance. SMC formulae are defined as quantifier-free formulae in conjunctive normal form, with atomic propositions ranging over a subset of the propositional variables and convex constraints². Then, given two SMC formulas $Pr$ and $Rq$, the proposed method to check whether $Pr \vdash Rq$ holds or not is done by determining if the formula $Pr \wedge \lnot Rq$ is not satisfiable.

Such a procedure does not provide any hint on what to do when no service registered in the repository fully satisfies the requirements. That is, no QoS provision contract satisfies the QoS requirements contract.

A different way of formalising QoS contracts is as a set of real values satisfying the specification. Next we present this alternative formalisation of quantitative non-functional requirements. Throughout the rest of the paper, letters $\mathcal {I}$, $\mathcal {J}$ and $\mathcal {K}$ will be used to denote index sets.

Definition 1

Let $\mathcal {X}$ be a set of real variables. We define a QoS specification on quantitative attributes $\mathcal {X}$ to be $\left\langle \mathcal {X}, \alpha \right\rangle$, with $\alpha$ being a formula with shape $\bigvee _{i \in \mathcal {I}} \mathtt {P}_i$. Each $\mathtt {P}_i$ is a convex polytope expressed as $\bigwedge _{k \in \mathcal {K}_i} f_{i, k} ({\mathop {x}\limits ^{\rightarrow }}) \ \mathcal {R}\ 0$, where $\mathcal {R} \in \{<, \le \}$ such that each $f_{i, k} ({\mathop {x}\limits ^{\rightarrow }}) \ \mathcal {R}\ 0$ is a convex constraint. The set of QoS specifications over the set of quantitative attributes $\mathcal {X}$ will be referred to as $QoS\_spec (\mathcal {X})$.

Let $\left\langle \mathcal {X}, \alpha \right\rangle \in QoS\_spec (\mathcal {X})$. We define a valuation for $\alpha$ as $v: \mathcal {X} \rightarrow \mathbbm {R}$. Satisfaction is a relation $\models \subseteq [\mathcal {X} \rightarrow \mathbbm {R}] \times QoS\_spec (\mathcal {X})$ defined as ${\mathop {x}\limits ^{\rightarrow }} \models \alpha$ if and only if there exists $i \in \mathcal {I}$, such that for all $k \in \mathcal {K}_i$, $f_{i, k} ({\mathop {x}\limits ^{\rightarrow }}) \ \mathcal {R}\ 0$. The set of values satisfying $\alpha$ is defined as $[\![ \alpha ]\!] = \left\{ {\mathop {x}\limits ^{\rightarrow }} | {\mathop {x}\limits ^{\rightarrow }} \models \alpha \right\}$.

Essentially, the set of admissible values of the quantitative attributes $\mathcal {X}$ in a QoS_spec is characterised by a set of convex polytopes over $\mathcal {X}$. Notice that, under some reasonable assumptions, SMC formulae can be translated to specifications in $QoS\_spec ({\mathcal {X}})$ by simply iterating the satisfying valuations of the boolean abstraction of the formulae, and collecting the polytopes determined by the positive literals.

A specification like this can capture many quantitative aspects of a software system. For example, the formulae below show a specification of an API-based software application requiring a service to be paid exclusively for the time it is used. Consider as relevant quantitative attributes a) $perSec$: the cost per second of the session, and b) $maxWait$: the maximum waiting time. Then, such attributes can be formalised by a QoS specification $\left\langle \left\langle \mathcal {X}\right\rangle , \alpha \right\rangle$ where $\mathcal {X} = \{ perSec , maxWait \}$. To characterise a service that is more expensive when it is faster a possible $\alpha$ would be the disjunction of the following formulae:

$$ \begin{array}{r} 0< maxWait \le 100 \wedge 0.1 \le perSec< 0.3, \\ 100 < maxWait \le 1000 \wedge 0.0 \le perSec \le 0.1 \end{array} $$

A geometrical interpretation of the formula $Pr \implies Rq$ leads to the possibility of measuring different degrees of partial compliance, which range between total compliance (all values allowed by $Pr$ are accepted by $Rq$) and no compliance at all (no value allowed by $Pr$ is accepted by $Rq$).

Based on the previous definitions, proving the formula $Pr \vdash Rq$ is to check whether $[\![ Pr ]\!] \subseteq [\![ Rq ]\!]$. We propose to compute the volume of the intersection between $[\![ Pr ]\!]$ and $[\![ Rq ]\!]$ relative to the volume of $[\![ Pr ]\!]$, referred to as inclusion ratio. This indicator quantifies what is the percentage of the QoS values allowed by the provision contract that is actually accepted by the requirements contract. For us, this indicator serves the purpose of quantifying the partial compliance of $Rq$ by $Pr$; therefore, functionally compliant services can be chosen from a ranking built by using inclusion ratio as the ordering criterion.

2.1 On the Complexity of the Volume Computation

In the literature, the volume of a polytope is invariably defined in terms of the sum of the volumes of a certain decomposition of such polytope into a family of convex ones [6]. Fortunately, Definition 1 assists us by already expressing the set of values in a QoS contract as the union of the sets of values of each member of a family of convex polytopes, expressed as $Pr = \bigvee _{i \in \mathcal {I}} \mathtt {P}_i$ and $Rq = \bigvee _{j \in \mathcal {J}} \mathtt {R}_j$. As neither family is guaranteed to be a partition, volumes must be computed following the Principle of inclusion-exclusion [10, Chap. 4]. Then, the inclusion ratio is calculated as follows:

• First: Compute the volume (denoted as $\#$) of intersection between $Pr$ and $Rq$:

  $$\begin{aligned} \# \left( \bigcup _{i \in \mathcal {I}} [\![ \mathtt {P}_i ]\!] \cap \bigcup _{j \in \mathcal {J}} [\![ \mathtt {R}_j ]\!]\right)= & {} \sum _{\emptyset \not = \mathcal {K} \subseteq \mathcal {I} \times \mathcal {J}} (-1)^{|\mathcal {K}| - 1} \# \left( \bigcap _{(i,j) \in \mathcal {K}} \left( [\![ \mathtt {P}_i ]\!] \cap [\![ \mathtt {R}_j ]\!]\right) \right) \end{aligned}$$

  (1)

• Second: Compute the volume of $P$:

  $$\begin{aligned} \# \left( \bigcup _{i \in \mathcal {I}} [\![ \mathtt {P}_i ]\!]\right) = \sum _{\emptyset \not = \mathcal {K} \subseteq \mathcal {I}} (-1)^{|\mathcal {K}| - 1} \# \left( \bigcap _{k \in \mathcal {K}} [\![ \mathtt {P}_k ]\!] \right) \end{aligned}$$

  (2)

• Third: Compute the inclusion ratio of $Pr$ in $Rq$ as: $\frac{\#( [\![ Pr ]\!] \cap [\![ Rq ]\!] )}{\#( [\![ Pr ]\!] )}.$

In the late 80’s Dyer et al. [3] and Khachiyan [8] proved the complexity of computing the volume of a convex polytope to be #P-Hard. The first estimation algorithm was presented by Kannan et al. in [7]. Later in [5], an estimation algorithm based on the Multiphase Monte-Carlo algorithm is proposed and the authors show that it can efficiently handle instances of dozens of dimensions with high accuracy. The complexity is shown to be $O^*(m \cdot n^3)$, where n is the dimensions, m is the number of constraints, and the Soft-O notation ($O^*$) omits logarithmic factors. The estimation method tool is distributed under the name PolyVest.

The implementation of the algorithm presented above for computing the inclusion ratio requires $2^{|\mathcal {I}|\cdot |\mathcal {J}|} + 2^{|\mathcal {I}|}$ invocations to the function computing the volume of a convex polytope. Such complexity forces us to consider a further dimension of approximation in order to make the calculation of the inclusion ratio viable in practice. In the next section we study the appropriateness of considering only some of the intersections involved in the calculation as an estimation of the volume.

3 Experimental Results

Our first research question is related to the complexity of the exact computation of the volume of a polytope $\mathcal {P}$. For the sake of understanding the limits imposed by such complexity, the research question will be stated in terms of a single polytope decomposed as a family of convex polytopes.

RQ1: How does the complexity of $\# \left( \bigcup _{i \in \mathcal {I}} \mathcal {P}_i \right)$ impacts in practice?

From our experiments³ it is possible to identify the exponential nature of the computation and that the majority of cases could not be completed within the time budget, hence the method cannot be applied to more realistic case-studies. The applicability of the technique, therefore, rests on the possibility of computing a good and efficient approximation of the volume operator. A natural way of doing this is by considering only a limited number of terms of the sum in Eq. 1. The next proposition formalises the intuition that subsequent partial sums represent an upper, and a lower, bound for the exact value of the volume.

Proposition 1

Let $P = \{P_i\}_{i \in \mathcal {I}}$ and $0< n < |\mathcal {I}|$, with $n \equiv 0 \pmod {2}$ then

$$ \sum _{\begin{array}{c} \emptyset \not = \mathcal {K} \subseteq \mathcal {I} \\ 0< |\mathcal {K}| \le n \end{array}} (-1)^{|\mathcal {K}| - 1} \# \left( \bigcap _{k \in \mathcal {K}} [\![ \mathtt {P}_k ]\!] \right) \le \# (P) \le \sum _{\begin{array}{c} \emptyset \not = \mathcal {K} \subseteq \mathcal {I} \\ 0 < |\mathcal {K}| \le n + 1 \end{array}} (-1)^{|\mathcal {K}| - 1} \# \left( \bigcap _{k \in \mathcal {K}} [\![ \mathtt {P}_k ]\!] \right) \ . $$

Computing an approximation of a magnitude requires the introduction of a metric for measuring the effectiveness of such estimation. As usual, the relative error between the magnitude x and an approximate value $x_0$ is defined to be $\frac{|x_0 - x|}{|x|}$. The next proposition provides upper bounds for the relative error that do not depend on the exact value v of the volume but on two consecutive approximations: $\{v_{k-1}, v_k\}$ or $\{v_k, v_{k+1}\}$.

Proposition 2

Let $v, v_{k-1}, v_k, v_{k+1} \in \mathbb {R}$, then:

• if $0< v_{k-1}< v < v_{k}$, then $\frac{v_k - v}{v} \le \frac{v_k - v_{k-1}}{v_{k-1}}$ (denoted $\rho _k$),

• if $v_{k}> v> v_{k+1} > 0$, then $\frac{v_k - v}{v} \le \frac{v_k - v_{k+1}}{v_{k+1}}$ (denoted $\sigma _k$)

• if $v_{k-1} \ge v_{k+1} > 0$, then $\rho _k \ge \sigma _k$.

Using these bounds, and considering Proposition 1, we can design an algorithm for computing the approximate volume of a polytope, by computing successive approximations until the desired value for the relative error is guaranteed. Notice that if we require a relative error equal to 0, the complexity of the algorithm is still $O (2^n)$, but when the bound is big enough to let the algorithm converge in the second approximation, the complexity drops to $O (n^2)$. Our second research question analises the practical impact of approximating the volume of a polytope using these bounds.

RQ2: How do the partial sums of the terms in the sum of Eq. 1 converge to the value $\# \left( \bigcup _{i \in \mathcal {I}} \mathcal {P}_i \right)$ with 1% and 5% tolerance?

Results⁴ show that the number of successive approximations needed for a calculation with an error within $1\%$ tolerance is between 2 and 5, while for an error within $5\%$ tolerance is between 2 and 3. In both scenarios the estimated computation greatly outperformed the exact one, being able to compute the mayority of cases within the time budget.

A natural follow-up from these results is to explore the scalability of the procedure for volume estimation. Table 1 show the maximum size of the family of polytopes whose approximate volume can be computed in one hour, considering relative errors with bounds set to $1\%$ and $5\%$. Size denotes the number of polytopes in the family and App the number of successive approximations needed in each case.

Table 1. Upper limit for computation of approximated volume.

As we mentioned in Sect. 2, efficiently computing the volume of these types of polytopes is just a means for quantifying partial compliance of QoS contracts.

RQ3: how does the performance gain obtained by approximating the volume impacts the computation of the inclusion ratio?

Our results⁵ show that as the size of the contracts increases the efficiency gain between 0%, 1% and 5% tolerance is dramatically better. There are cases where the approximate volume computation is performed in $4\%$ of the time required by the exact volume computation, and several cases for which the exact inclusion ratio fails to be computed within the 10 h time budget but becomes feasible when a bound for the relative error is introduced.

Inclusion ratio provides a way of ranking services according to partial satisfaction. Thus, it is worth knowing how an approximate computation of the volume propagates the relative error to the computation of the inclusion ratio. In the end, this will impact the actual selection of a service candidate. The next proposition establishes lower and upper bounds for the distance between approximate inclusion ratio and exact inclusion ratio.

Proposition 3

Given two volume approximations $v_k, w_k$, where the exact volumes are v, k, respectively. Let e be the bound for the relative error (i.e., $\frac{| v - v_k |}{|v|} \le e$ and $\frac{| w - w_k |}{|w|} \le e$) then:

• 1. $(1-e) \cdot v \le v_k \le (1+e) \cdot v$ (respectively $(1-e) \cdot w \le w_k \le (1+e) \cdot w$), and

• 2. $\frac{(1-e) \cdot v}{(1+e) \cdot w} \le \frac{v_k}{w_k} \le \frac{(1+e) \cdot v}{(1-e) \cdot w}$ (equivalently $\frac{1-e}{1+e} \cdot \frac{v}{w} \le \frac{v_k}{w_k} \le \frac{1+e}{1-e} \cdot \frac{v}{w}$).

Consider, as an example, the bounds obtained from Proposition 3 for the approximate inclusion ratio, when approximate volumes are computed with relative error bound to be smaller than $5\%$ and $1\%$, as we did in the previous experiments. Let $eir$ be the exact inclusion ratio, $ir 5$ the inclusion ratio computed with a relative error below $5\%$ and $ir 1$ the inclusion ratio computed with a relative error below $1\%$; then, we obtain that $0.905 \cdot eir \approx \frac{0.95}{1.05} \cdot eir \le ir 5 \le \frac{1.05}{0.95} \cdot eir \approx 1.105 \cdot eir$ and $0.98 \cdot eir \approx \frac{0.99}{1.01} \cdot eir \le ir 1 \le \frac{1.01}{0.99} \cdot eir \approx 1.02 \cdot eir$, respectively.

Our final research question focuses on effectively ranking different provision contracts using the inclusion ratio as a metric for quantifying partial satisfaction.

RQ4: What is the impact of the proposed approximation in the construction of a ranking of a set of services?

For this experiment we performed 3 examples of ranking 20 provision contracts by partial satisfaction of a single requirements contract. The datasets were synthetically generated following the guidelines used for generating the case-studies used in RQ3. Each ranking correspond to a different size of the families of polytopes as follows: a) $| Pr | = 2$, $| Rq | = 4$, b) $| Pr | = 3$, $| Rq | = 3$, c) $| Pr | = 4$, $| Rq | = 2$.

For each dataset we performed the ranking order using the exact inclusion ratio, and also using $1\%$ and $5\%$ bounds for the relative error. The results of the experiment show that in all of the examples the ranking obtained by using the exact volume computation is preserved when resorting to approximate volume computations.

4 Conclusions and Further Work

We extended previous work where we proposed a formalisation of QoS contract as convex specifications [13]. Under a geometrical interpretation, a contract can be seen as a family of convex polytopes characterising the admissible values for the quantitative attributes and a judgement of the shape $Pr \vdash Rq$ as polytope inclusion. From this perspective, we proposed a way of QoS ranking services by quantifying the volume of the intersection between $[\![ Pr ]\!]$ and $[\![ Rq ]\!]$, under the name inclusion ratio. Since this indicator quantifies the percentage of values allowed by the provision contract that are actually accepted by the requirement contract we sustain that it serves the purpose of quantifying partial compliance. The exponential nature of computing the volume of a family of convex polytopes forced us to propose a volume approximation technique based on imposing an upper bound to the relative error.

We evaluated the performance of the approximated volume computation against the exact volume computation, the scalability of the approximate volume computation and the performance of computing the inclusion ratio with respect to the exact computation. Finally, we showed examples of the impact of approximate inclusion ratio in ranking services by comparing the ideal ranking, obtained by resorting to the exact inclusion ratio, and the one obtained by resorting to approximate inclusion ratio, for different bounds to the relative error.

This technique for evaluating partial satisfaction of QoS contracts triggers what we believe is one of the most important questions regarding the automation of service broking. Under partial satisfaction of QoS contracts, we would like to establish preferences among different quantitative attributes of the different candidates. For example, if we consider two services with the same inclusion ratio, a low budget application would like to express it’s preference for cheaper services, even at the cost of degraded performance, over a more expensive and efficient one. This problem will be addressed in the near future aiming at a more realistic view of an automatic procedure for checking QoS compliance for determining SLA in a service-based architecture.