Locality-aware streaming in hybrid P2P-cloud CDN systems

Abstract

The cloud CDN has been exploited as a cost-effective and elastic way for video streaming, where the content providers only pay for resources (e.g., bandwidth, storage, Web service) that they use. As a server-client solution, the cost on the content providers, especially the bandwidth cost for delivering videos from the cloud to the users, is still high. To further mitigate the content distribution cost, it is promising to deploy the P2P streaming technology in conjunction with a cloud CDN, to construct a hybrid P2P-cloud CDN system. In this paper, we focus on optimal, locality-aware video-on-demand streaming solutions in a hybrid P2P-cloud CDN system, which achieve the best tradeoff between the costly bandwidth consumption on the cloud servers and the undesirable inter-ISP traffic incurred in the entire system. Especially, we characterize the demanded upload bandwidth in the cloud CDN and the incurred inter-ISP traffic through a number of stochastic and optimization models. We first apply a loss network model to derive the needed bandwidth capacity in the cloud CDN under any given chunk distribution pattern among the peer caches and any streaming request dispatching strategy among the ISPs, and derive the optimal peer caching and request dispatching strategies which minimize the bandwidth demand on the cloud CDN. We then investigate the necessary volume of inter-ISP traffic required to achieve the minimum cloud bandwidth, as well as the impact of limiting the inter-ISP traffic on the cloud bandwidth consumption. Based on the fundamental insights from our models and analytical results, we design a locality-aware, hybrid P2P-cloud streaming protocol, and validate its performance using extensive simulations under realistic settings.

Appendices

Appendix A: Proof of Theorem 1

Proof

Denote the graph as G = (V, E). We first construct an S-T cut of the graph $C_{st} = (\mathcal {F},\mathcal {L})$, a partition of nodes V with source node $\mathcal {S}\in \mathcal {F}$ and destination node $\mathcal {T} \in \mathcal {L}$. The cut set of C _st is the set $\{(u,v)\in E| u \in \mathcal {F}, v \in \mathcal {L} \}$. We construct a cut set as follows: We can divide x _{m, j} ’s, 1 ≤ j ≤ J, into two classes according to whether x _{m, j} is equal to $\sum _{l=1}^{M} a_{lm}r_{l,j}$ or smaller than $\sum _{l=1}^{M} a_{lm}r_{l,j}$: $\mathcal {C}_1=\left \{c_j|x_{m,j}=\sum _{l=1}^{M} a_{lm}r_{l,j}, 1 \le j \le J\right \}$, and $\mathcal {C}_2=\left \{c_j|x_{m,j}< \sum _{l=1}^{M} a_{lm}r_{l,j}, 1 \le j \le J\right \}$. There is not enough peer upload bandwidth in ISP m to serve chunks in set $\mathcal {C}_{2}$. From the KKT conditions, we can see that for all $\mathcal {A}$ which includes $c_{j}\in \mathcal {C}_{1}$, we have $\epsilon _{\mathcal {A}}=0$ according to KKT condition (7); for $\mathcal {A} = \mathcal {C}_{2}$, we have $\sum _{c_{j}\in \mathcal {C}_{2}}x_{m,j}=U_{m}\cdot \sum _{i:s_{i}\cap \mathcal {C}_{2}\neq \emptyset }N_{m}^{(i)}$ according to KKT condition (6). According to KKT condition (6), we have $\epsilon _{\mathcal {C}_{2}} > 0$. The nodes are partitioned into two sets. The set including source node is $\mathcal {F}$ = {source node $\mathcal {S}$, all nodes in $\mathcal {C}_{2}$, and nodes in Λ that are connected to nodes in $\mathcal {C}_{2}$}, the set including destination node is $\mathcal {L}$ = {all other nodes in the graph not in $\mathcal {F}$ }.

Now let us see the capacity of the constructed cut set. The cut set includes all the edges connecting $\mathcal {S}$ with all nodes in $\mathcal {C}_{1}$, and all the edges connecting $\mathcal {T}$ with the nodes in Λ which are connected to nodes in $\mathcal {C}_{2}$. The capacity of all the edges connecting $\mathcal {S}$ with all nodes in $\mathcal {C}_{1}$ is $\sum _{c_{j} \in \mathcal {C}_{1}} \sum _{l=1}^{M} a_{lm}r_{l,j}$, the capacity of all the edges connecting $\mathcal {T}$ with the nodes in Λ which are connected to nodes in $\mathcal {C}_{2}$ is $U_{m}\cdot \sum _{i:s_{i}\cap \mathcal {C}_{2} \neq \emptyset }N_{m}^{(i)}$. As we have $U_{m}\cdot \sum _{i:s_{i}\cap \mathcal {C}_{2}\neq \emptyset }N_{m}^{(i)} = \sum _{c_{j}\in \mathcal {C}_{2}}x_{m,j}$, $x_{m,j}=\sum _{l=1}^{M} a_{lm}r_{l,j}, \forall c_{j} \in \mathcal {C}_{1}$, the total capacity of the cut set is equal to $\sum _{j=1}^{J} x_{m,j}$.

The capacity of this cut set equals to the number of total concurrently served requests obtained by solving the KKT conditions. We prove that this cut is the minimum cut in the graph. Applying the min-cut max-flow theorem, we can then show that the number of total concurrently served requests is the maximum flow.

On one hand, if $\mathcal {F}$, the set including $\mathcal {S}$, contains a node c _j from $\mathcal {C}_{1}$, since the overall amount of flow from c _j to nodes in Λ is equal to the capacity of the edge from $\mathcal {S}$ to c _j , the capacity of the new cut set is no smaller than that of the original cut set with $\mathcal {F}$ excluding nodes from $\mathcal {C}_{1}$. When all nodes in Λ connecting to c _j are included together with c _j in the set $\mathcal {F}$, the capacity of the new cut is increased by $U_{m}\cdot \sum _{s_i:c_{j} \in s_{i}}N_{m}^{(i)}-\sum _{l=1}^{M} a_{lm}r_{l,j} \ge 0, c_{j}\in \mathcal {C}_{1}$, according to KKT condition (4).

On the other hand, if the set including source node $\mathcal {S}$ excludes a node c _k in $\mathcal {C}_{2}$, the edge ($\mathcal {F}$, c _k ) will be added into the new cut set, the capacity of the new cut set is increased by $\sum _{l=1}^{M} a_{lm}r_{l,j}$. If a node s _i in Λ connecting to c _k is excluded with c _k together from the set including source node $\mathcal {S}$, the edges ($\mathcal {F}$, c _j ) with c _j ∈ s _i will be added into the new cut set, the edge ( s _i , $\mathcal {L}$) will be deleted from the new cut set. Hence, the capacity of new cut set will be increased by $\sum _{j: c_{j}\in s_{i}}\sum _{l=1}^Ma_{lm}r_{l,j} - U_mN_{m}^{(i)}$, which is larger than 0 since chunks in $\mathcal {C}_{2}$ do not receive enough peer upload bandwidth to be served. Hence, the capacity of new cut set increases.

Therefore, the cut we construct with capacity $\sum _{j=1}^{J} x_{m,j}$ is the minimum cut. Applying the min-cut max-flow theorem, we have proved that the number of total concurrently served requests is equal to the maximum bipartite flow. □

Appendix B: Proof of Lemma 1

Proof

When $N_{m}^{(i)}$’s satisfy (9), let $\epsilon _{\mathcal {A}} = 0$ for all $\mathcal {A} \subset \mathcal {C}$, and $\epsilon _{\mathcal {A}} = \max \{0,\ln \eta _{m}\}$ for $\mathcal {A} = \mathcal {C}$. It is easy to verify that $x_{m,j} = \min \left \{ \sum _{l=1}^Ma_{lm}r_{l,j},\frac {\sum _{l=1}^Ma_{lm}r_{l,j}}{\eta _{m}}\right \}, 1 \le j \le J$, are solutions to the KKT conditions in Section 3.2. Hence, the average chunk miss rate is $L_{m} =1-\frac {\sum _{j=1}^{J} x_{m,j}}{\sum _{l=1}^Ma_{lm}r_{l}}= 1- \frac {\min \left \{1,\frac {1}{\eta _{m}}\right \}\sum _{j=1}^{J} \sum _{l=1}^Ma_{lm}r_{l,j}}{\sum _{l=1}^Ma_{lm}r_{l}}=1-\min \left \{1, \frac {1}{\eta _{m}}\right \}=\max \left \{ 0, 1-\right .$ $\left .\frac {1}{\eta _{m}}\right \}$. □

Appendix C: Proof of Lemma 2

Proof

We prove the caching strategy in Lemma 2 satisfies the optimal cache condition (9) in Lemma 1, by induction.

• (i) When $|\mathcal {A}| > J-B$, the intersection between any cache state s _i , 1 ≤ i ≤ W, and $\mathcal {A}$ is not empty. Hence, the L.H.S of (9) equals to $\sum _{c_{j}\in \mathcal {A}}$ $\sum _{l=1}^Ma_{lm}r_{l,j}$, which is smaller than the R.H.S of (9), $\eta _{m} \cdot U_{m} \cdot N_{m} = \sum _{l=1}^Ma_{lm}r_{l}$.

• (ii) Let $\mathcal {A}_{k}$ denote the set $|\mathcal {A}| = k$. When $|\mathcal {A}| = J-B$, we use $\mathcal {A}_{J-B}$ to denote $\mathcal {A}$. Let $\bar {\mathcal {A}}_{J-B}$ be the complementary set of $\mathcal {A}_{J-B}$. $|\bar {\mathcal {A}}_{J-B}| = B$. The intersection between the cache state $s_{i} = \bar {\mathcal {A}}_{J-B}$ and $\mathcal {A}_{J-B}$

  $$\begin{array}{ll}& \text{R.H.S of (9)} = \eta_m \cdot U_m \left(N_m - \sum\limits_{i:s_i = \bar{\mathcal{A}}_{J-B}}N_m^{(i)}\right) \\& = \eta_m \cdot U_m \left(N_m{\kern-1pt}-{\kern-1pt}N_m \cdot \prod\limits_{c_j \in \bar{\mathcal{A}}_{J-B}} \rho_j\right) \\& \ge \eta_m \cdot U_m \left(N_m - N_m\cdot \left(\frac{B}{\sum_{l=1}^Ma_{lm}r_l}\right)^B \prod\limits_{c_j \in \bar{\mathcal{A}}_{J-B}}\sum\limits_{l=1}^Ma_{lm}r_{l,j}\right) \\& \ge \eta_m \cdot U_m \left({\kern-2pt}N_m{\kern-2pt}-{\kern-2pt}N_m \cdot\left({\kern-1.5pt}\frac{B}{\sum_{l=1}^Ma_{lm}r_l}{\kern-1.5pt}\right)^B\cdot \left({\kern-1.3pt}\frac{\sum_{c_j \in \bar{\mathcal{A}}_{J-B}}\sum_{l=1}^Ma_{lm}r_{l,j}}{B}{\kern-1.3pt}\right)^B{\kern-1.5pt}\right) \\& \ge \sum\limits_{l=1}^Ma_{lm}r_l{\kern-1.3pt}-{\kern-1.3pt}\left(\sum_{c_j \in \bar{\mathcal{A}}_{J-B}}\sum\limits_{l=1}^Ma_{lm}r_{l,j}\right) \cdot \left(\frac{\sum\limits_{c_j \in \bar{\mathcal{A}}_{J-B}}\sum_{l=1}^Ma_{lm}r_{l,j}}{\sum\limits_{l=1}^Ma_{lm}r_l}\right)^{B-1} \\& \ge \sum\limits_{c_j\in \mathcal{A}_{J-B}} \sum\limits_{l=1}^Ma_{lm}r_{l,j} = \text{L.H.S of (9)}\end{array}$$

• (iii) For k ≤ J − B, suppose all $\mathcal {A}_{k}$’s satisfy (9). Consider the case for any $\mathcal {A}_{k-1}$. Let us consider a specific $\mathcal {A}_{k-1}$, there are (J − k + 1) chunks $c_{j} \notin \mathcal {A}_{k-1}$. For any $c_{j} \notin \mathcal {A}_{k-1}$, we can construct an $\mathcal {A}_{k} =c_{j} \cup \mathcal {A}_{k-1}$. Hence, we get (J − k + 1) sets of $\mathcal {A}_{k}$’s. Let $\mathcal {A}_{k}^{1}, \mathcal {A}_{k}^{2}, \ldots , \mathcal {A}_{k}^{J-k+1}$ denote them. For each $\mathcal {A}_{k}$, we apply (9),

  $$\sum\limits_{c_{j}\in \mathcal{A}_{k}^{t}} \sum\limits_{l=1}^Ma_{lm}r_{l,j}\le \eta_{m} \cdot U_{m}\cdot \sum\limits_{i:\mathcal{A}_{k}^{t}\cap s_{i}\neq \emptyset}N_{m}^{(i)}, 1\le t\le J{\kern-1.5pt}-{\kern-1.5pt}k{\kern-1.5pt}+{\kern-1.5pt}1. $$

  Sum up all the (J − k + 1) inequalities, we have,

  $$\begin{array}{l} \sum\limits_{t=1}^{J-k+1} \sum\limits_{c_{j}\in \mathcal{A}_{k}^{t}} \sum\limits_{l=1}^Ma_{lm}r_{l,j} \\ = (J-k+1)\sum\limits_{c_{j} \in \mathcal{A}_{k-1}} \sum\limits_{l=1}^Ma_{lm}r_{l,j}+\sum\limits_{c_{j} \notin \mathcal{A}_{k-1}} \sum\limits_{l=1}^Ma_{lm}r_{l,j}\\ \le \sum\limits_{t=1}^{J-k+1}\eta_{m}\cdot U_{m} \cdot \sum\limits_{i:\mathcal{A}_{k}^{t}\cap s_{i} \neq \emptyset} N_{m}^{(i)} \\ = (J-k+1)\eta_{m}\cdot U_{m} \sum\limits_{i:\mathcal{A}_{k-1}\cap s_{i} \neq \emptyset} N_{m}^{(i)} \\ \quad + \eta_{m}\cdot U_{m}\sum\limits_{i: s_{i} \cap \mathcal{A}_{k-1} = \emptyset} N_{m}^{(i)}. \end{array}$$

  Hence,

  $$\begin{array}{l} (J-k+1)\sum\limits_{c_{j} \in \mathcal{A}_{k-1}}\sum\limits_{l=1}^Ma_{lm}r_{l,j} + \sum\limits_{l=1}^Ma_{lm}r_{l} -\sum\limits_{c_{j} \in \mathcal{A}_{k-1}}\sum\limits_{l=1}^Ma_{lm}r_{l,j} \\ \le (J-k+1)\eta_{m}\cdot U_{m} \sum\limits_{i:\mathcal{A}_{k-1}\cap s_{i} \neq \emptyset} N_{m}^{(i)} \\ \quad+ \eta_{m}\cdot U_{m}(N_{m} -\sum\limits_{i:\mathcal{A}_{k-1}\cap s_{i} \neq \emptyset} N_{m}^{(i)}). \end{array}$$

  We have

  $$\sum\limits_{c_{j} \in \mathcal{A}_{k-1}}\sum\limits_{l=1}^Ma_{lm}r_{l,j} \le \eta_{m}\cdot U_{m} \sum\limits_{i:\mathcal{A}_{k-1}\cap s_{i} \neq \emptyset} N_{m}^{(i)}. $$

  Hence, Lemma 2 is proven.

□

Appendix D: Proof of Lemma 3

Proof

Suppose each peer’s cache contains B ordered positions, to cache B chunks. Consider chunk j’s position in a peer’s cache n time units after the peer starts playing videos, under the LRU algorithm, which is denoted by $s^{j}_{n}$. We first derive the position transition probabilities, and then the stationary-state probability of a peer caching chunk c _j .

Recall that a peer has a probability $\sum _{j=1}^{J}\pi _{j}\rho _{j}$ to replay a cached chunk. The probability of downloading and playing a new chunk is $1-\pi _0-\sum _{j=1}^{J}\pi _{j}\rho _{j}$. We assume a chunk being played (no matter newly downloaded or already cached) is put into position 1 in the cache. In case of a new download, the chunk in the last position of the cache, position B, will be evicted, and positions of all other cached chunks will be increased by 1 (LRU algorithm). In case of replaying a cached chunk, positions of chunks cached before this one, will be increased by 1. Given $s^{j}_{n}$, we can derive the probability distribution of chunk j’s position at time n + 1, i.e., $s^{j}_{n+1}$. For 2 ≤ b ≤ B,

$$\begin{array}{l} Pr[s^{j}_{n+1} = b | s^{j}_{n}] = \\ Pr[c_{j}'\text{s position increases by} {1} | s^{j}_{n} = b{}-{}1] \cdot Pr[s^{j}_{n}{\kern-1.5pt}={\kern-1.5pt} b{}-{}1] \\ \quad + Pr[c_{j}'\text{s position does not change} | s^{j}_{n}{\kern-1.8pt}={\kern-1.8pt}b] \cdot Pr[s^{j}_{n}{\kern-1.8pt}={\kern-1.8pt}b]. \end{array}$$

The event that chunk j’s position increases by 1 when $s^{j}_{n} = b-1$ can be divided into two disjoint cases: (i) when the peer plays a new chunk, and (ii) when the peer replays a chunk cached at a position behind b − 1:

$$\begin{array}{@{}rcl@{}} & Pr[\mbox{\(c_{j}\)'s position increases by 1} | s^{j}_{n} = b-1] = \\ & \left(1-\pi_{0} - \sum\limits_{i \neq j}\pi_{i}\rho_{i}\right)+\sum\limits_{i \neq j}\pi_{i} \cdot Pr[s^{i}_{n} > b-1]. \end{array}$$

The event that chunk j’s position does not change when $s^{j}_{n} = b$ happens when the peer replays a chunk cached at a position ahead of b:

$$\begin{array}{@{}rcl@{}} Pr[c_{j}'\text{s position does not change} | s^{j}_{n} =b] = \pi_{0} + \sum\limits_{i \neq j}\pi_{i} Pr[s^{i}_n<b]. \end{array}$$

Hence,

$$\begin{array}{@{}rcl@{}} {}Pr[s^{j}_{n+1} =b | s^{j}_{n}] &=& \left\{ \left(1-\pi_{0} - \sum\limits_{i\neq j}\pi_{i}\rho_i\right) \right. \\ &&+\left.\sum\limits_{i \neq j}\pi_{i} \cdot Pr[s^{i}_{n} > b-1] \right\}\cdot Pr[s^{j}_{n} = b-1] \\ &&+\{ \pi_{0} + \sum\limits_{i \neq j}\pi_{i} Pr[s^{i}_n<b]\} \cdot Pr[s^{j}_{n} = b]. \end{array}$$

(16)

Equation (16) shows that the next position of chunk j is only related to the previous position of chunk j. We can use a Markov chain to model the change of chunk j’s positions: state b, 1 ≤ b ≤ B, represents that chunk j is at position b in the peer’s cache; state 0 denotes that chunk j is not in the peer’s cache; $Pr[s^{j}_{n+1} =b | s^{j}_n=b-1]$ is the transition probability from state b − 1 to state b for chunk j. Using this Markov chain model, we can analyze the stationary-state distribution of chunk j’s position in a peer’s cache.

Let s ^j denote the stationary-state position of chunk j when n goes to infinity. Based on equation (16), for 2 ≤ b ≤ B, we have,

$$\begin{array}{@{}rcl@{}} & Pr[s^{j} = b ] = (1-\pi_{0} - \sum_{i\neq j}\pi_{i}\rho_{i} + \sum_{i\neq j} \pi_{i} \sum_{p=b}^{B} Pr[s^{i}_n=p]) \\ & \cdot Pr[s^{j} = b-1] +(\pi_{0} + \sum_{i\neq j} \pi_{i} \sum_{p=1}^{b-1}Pr[s^{i}_n=p])\cdot Pr[s^{j} = b]. \end{array}$$

As $\pi _{i} \rho _{i} = \pi _{i} \sum _{p=1}^{B} Pr[s^{i}_{n} = p]$, hence,

$$ Pr[s^{j} = 1]=Pr[s^{j} = 2]=\ldots=Pr[s^{j} = B]. $$

(17)

This implies that a chunk is cached in different positions with an equal probability in the steady state.

For b = 1, Pr[s ^j = 1] is the stationary probability that chunk j is cached at position 1 in the peer’s cache, which equals to the stationary probability that a peer in the VoD system is requesting and playing chunk j that we have defined earlier, i.e., π _j .

Therefore, under the LRU algorithm, the probability that a peer caches chunk j is B · Pr[s ^j = 1] = B · π _j , the proportion of peers caching chunk j is ρ _j = B · π _j as well. This is the optimal cache distribution proposed in Lemma 2. □

Appendix E: Proof of Theorem 2

Proof

Consider two cases:

• 1) The overall upload bandwidth of peers in the system is no smaller than the bandwidth demand for serving all the chunk requests, i.e., $\sum _{m=1}^MU_mN_{m}(1-\pi _0) \ge \sum _{l=1}^{M} r_{l}$. In this case, $\frac {\sum _{t,I_{t} > 0}I_{t}}{-\sum _{s,I_s<0}I_{s}} \ge 1$. Hence, for ISPs with I _m > 0,

  $$\begin{array}{@{}rcl@{}} a^{\ast}_{lm}=\left\{\begin{array}{l@{\quad \quad}l} 1: \quad \quad l = m, \\ 0: \quad \quad l\neq m, I_{l} \ge 0, \\ \frac{I_{m}}{\sum_{t,I_t>0}I_{t}}(\frac{-I_{l}}{r_{l}}) : \quad l\neq m, I_{l} \le 0. \end{array} \right. \end{array}$$

  for ISPs with I _m < 0,

  $$\begin{array}{@{}rcl@{}} a^{\ast}_{lm} = \left\{\begin{array}{l@{\quad \quad}l} \frac{U_mN_{m}(1-\pi_0)}{r_{m}} : \quad\quad l=m; \\ 0: \quad \quad l\neq m, 1\le l \le M. \end{array} \right. \end{array}$$

  First, we can verify that $a^{\ast }_{ml}$’s satisfy constraints (17): for ISPs with I _m < 0, $\sum_{l=1}^{M} a^{\ast }_{lm}r_{l} = a^{\ast }_{mm} r_{m} = U_mN_{m}(1-\pi _0)$; for ISPs with I _m ≥ 0, $\sum _{l=1}^{M} a^{\ast }_{lm}r_{l} = r_{m} + \frac {-\sum _{l,I_l<0}I_{l}}{\sum _{t,I_t>0}I_{t}}I_{m} \le U_mN_{m}(1-\pi _0)$. Hence, in any ISP, the total number of requests is fewer than or equal to the available peer upload bandwidth. $L^{\ast }_{m} = 0$ for 1 ≥ m ≥ M, which means $a^{\ast }_{ml}$’s achieve the minimum system-wide chunk miss rate.

  Second, we show that $T^{\ast }= \sum _{m=1}^{M}\sum _{l\neq m} a^{\ast }_{lm}\cdot r_{l} \cdot (1-L_{m}^{*}) = \sum _{m=1}^{M} \sum _{l\neq m} a^{\ast }_{lm} \cdot r_{l} = \sum _{m=1}^{M} (1-a^{\ast }_{mm}) \cdot r_{m} = \sum _{m,I_{m} <0} [r_{m} - U_mN_{m}(1-\pi _0)]$ is the minimum inter-ISP traffic to achieve the minimum chunk miss rate, by contradiction. Suppose there are a set of a′ _lm ’s, different from $a^{\ast }_{lm}$’s, satisfying the constraints and inducing the volume of inter-ISP traffic T ^″ < T ^∗, and still achieving the minimum chunk miss rate, $L'_{m} = L^{\ast }_{m} = 0$ for 1 ≤ m ≤ M. $T' = \sum _{m=1}^{M}\sum _{l\neq m} a'_{lm}\cdot r_{l} \cdot (1-L'_l) = \sum _{m=1}^{M} (1-a'_{mm}) \cdot r_{m} < T^{\ast }$. Hence, there exists $a'_{mm} > a^{\ast }_{mm}$. If I _m ≥ 0, then $a'_{mm} > a^{\ast }_{mm} = 1$, this contradicts with $\sum _{l=1}^{M} a'_{lm} = 1$; if I _m < 0, then $a'_{mm} > a^{\ast }_{mm} = \frac {N_mU_{m}}{r_{m}}$, $\sum _{l=1}^{M} a'_{lm} r_{m} \ge a'_{mm}\cdot r_{m} > N_mU_{m}$, this contradicts with L′ _m = 0. Hence, such a′ _lm ’s do not exist. $a^{\ast }_{lm}$’s are the optimal solutions. $T^{\ast } = \sum _{m,I_{m} <0} (r_{m} - U_mN_m)$ is the minimum volume of inter-ISP traffic.

• 2) The overall upload bandwidth of peers in the system is smaller than the bandwidth demand for serving all the chunk requests, i.e., $\sum _{m=1}^{M} U_mN_{m}(1-\pi _0) <\sum _{l=1}^{M} r_{l}$. In this case, $\frac{\sum _{t,I_{t} > 0}I_{t}}{-\sum _{s,I_s<0}I_{s}} < 1$. Hence, for ISPs with I _m > 0,

  $$\begin{array}{@{}rcl@{}} a^{\ast}_{lm}=\left\{\begin{array}{l@{\quad \quad}l} 1: \quad \quad l = m, \\ 0: \quad \quad l\neq m, I_{l} \ge 0, \\ \frac{I_{m}}{-\sum_{-s,I_s<0}I_{s}}\left(\frac{-I_{l}}{r_{l}}\right) : \quad l\neq m, I_{l} \le 0. \end{array} \right. \end{array}$$

  for ISPs with I _m < 0,

  $$\begin{array}{@{}rcl@{}} a^{\ast}_{lm} = \left\{\begin{array}{l@{\quad \quad}l} \left(1- \frac{\sum_{t,I_t>0}I_{t}}{-\sum_{s,I_s<0}I_{s}}\right)\left(\frac{-I_{m}}{r_{m}}\right)+\frac{U_mN_{m}\left(1-\pi_{0}\right)}{r_{m}}, \\ \quad \quad \quad \quad: \quad\quad l=m; \\ 0: \quad \quad l\neq m, 1\le l \le M. \end{array} \right. \end{array}$$

  First, for ISPs with I _m < 0, $\sum _{l=1}^{M} a^{\ast }_{lm}r_{l} = r_{m} +I_{m} \cdot \frac {\sum _{t,I_t>0}I_{t}}{-\sum _{s,I_s<0}I_{s}} = I_{m} \left (\frac {\sum _{t,I_t>0}I_{t}}{-\sum _{s,I_s<0}I_{s}} -1\right )+U_mN_{m}(1-\pi _0) > U_mN_{m}(1-\pi _0)$. For ISPs with I _m ≥ 0, $\sum _{l=1}^{M} a^{\ast }_{lm}r_{l} = r_{m} + \sum _{l,I_{l} <0}a^{\ast }_{lm}r_{l} = r_{m} + I_{m} = U_mN_{m}(1-\pi _0)$. Hence, $a^{\ast }_{lm}$’s satisfy condition (14). For ISPs with I _m < 0, $L^{\ast }_{m} >0$; for ISPs with I _m ≥ 0, $L^{\ast }_{m} = 0$.

  Second, we show $T^{\ast } = \sum _{m=1}^{M}\sum _{l\neq m} a^{\ast }_{lm}\cdot r_{l} \cdot$ $(1-L_{m}^{*}) = \sum _{m, I_{m} \ge 0} \sum _{l \neq m} a^{\ast }_{lm} r_{l} = \sum _{m, I_{m} \ge 0}$ $\sum _{l,I_l<0} \frac {I_{m}}{-\sum _{s,I_s<0}I_{s}}(-I_l) = \sum _{m,I_m>0}I_{m}$ is the minimum inter-ISP traffic. Suppose there are a set of a′ _lm ’s, different from $a^{\ast }_{lm}$’s, satisfying the optimal chunk request routing constraints in (14). Hence, for ISPs with I _m ≥ 0, $\sum _{l=1}^Ma'_{lm} r_{l} \ge \sum _{l=1}^{M} a^{\ast }_{lm} r_{l} = U_mN_{m}(1-\pi _0)$. We have $T' = \sum _{m=1}^{M}\sum _{l\neq m} a'_{lm}\cdot r_{l} \cdot (1-L_m) = \sum _{m=1}^{M} \left (\sum _{l=1}^{M} a'_{lm}r_{l} - a'_{mm}r_{m}\right ) \cdot \frac {U_mN_{m}(1-\pi _0)}{\sum _{l=1}^{M} a'_{lm}\cdot r_{l}} \ge \sum _{m, I_{m} > 0}e$ $\left (\sum _{l=1}^{M} a'_{lm}r_{l} - a'_{mm}r_{m}\right ) \cdot \frac {U_mN_{m}(1-\pi _0)}{\sum _{l=1}^{M} a'_{lm}\cdot r_{l}} \ge \sum _{m, I_{m} > 0}\left [ U_{m} N_{m}(1-\pi _0)-a'_{mm}r_{m}\frac {U_mN_{m}(1-\pi _0)}{\sum _{l=1}^{M} a'_{lm}\cdot r_{l}}\right ] \ge \sum _{m, I_{m} > 0}[ U_mN_{m}(1-\pi _0) -r_{m}] = \sum _{m,I_m>0} I_{m} =T^{\ast }$, as a′ _mm ≤ 1 and $\sum _{l=1}^Ma'_{lm} r_{l} \ge U_mN_{m}(1-\pi _0)$. So, T ^∗ is the minimum inter-ISP traffic.

□