Convex optimization learning of faithful Euclidean distance representations in nonlinear dimensionality reduction
 1.8k Downloads
 7 Citations
Abstract
Classical multidimensional scaling only works well when the noisy distances observed in a high dimensional space can be faithfully represented by Euclidean distances in a low dimensional space. Advanced models such as Maximum Variance Unfolding (MVU) and Minimum Volume Embedding (MVE) use SemiDefinite Programming (SDP) to reconstruct such faithful representations. While those SDP models are capable of producing high quality configuration numerically, they suffer two major drawbacks. One is that there exist no theoretically guaranteed bounds on the quality of the configuration. The other is that they are slow in computation when the data points are beyond moderate size. In this paper, we propose a convex optimization model of Euclidean distance matrices. We establish a nonasymptotic error bound for the random graph model with subGaussian noise, and prove that our model produces a matrix estimator of high accuracy when the order of the uniform sample size is roughly the degree of freedom of a lowrank matrix up to a logarithmic factor. Our results partially explain why MVU and MVE often work well. Moreover, the convex optimization model can be efficiently solved by a recently proposed 3block alternating direction method of multipliers. Numerical experiments show that the model can produce configurations of high quality on large data points that the SDP approach would struggle to cope with.
Keywords
Euclidean distance matrix Convex matrix optimization Multidimensional scaling Nonlinear dimensionality reduction Lowrank matrix Error bounds Random graph modelsMathematics Subject Classification
49M45 90C25 90C331 Introduction
The chief purpose of this paper is to find a complete set of faithful Euclidean distance representations in a lowdimensional space from a partial set of noisy distances, which are supposedly observed in a higher dimensional space. The proposed model and method thus belong to the vast field of nonlinear dimensionality reduction. Our model is strongly inspired by several highprofile SemiDefinite Programming (SDP) models, which aim to achieve a similar purpose, but suffer two major drawbacks: (i) theoretical guarantees yet to be developed for the quality of recovered distances from those SDP models and (ii) the slow computational convergence, which severely limits their practical applications even when the data points are of moderate size. Our distinctive approach is to use convex optimization of Euclidean Distance Matrices (EDM) to resolve those issues. In particular, we are able to establish theoretical error bounds of the obtained Euclidean distances from the true distances under the assumption of uniform sampling, which has been widely used in modelling social networks. Moreover, the resulting optimization problem can be efficiently solved by a 3block alternating direction method of multipliers. In the following, we will first use social network to illustrate how initial distance information is gathered and why the uniform sampling is a good model in understanding them. We then briefly discuss several SDP models in nonlinear dimensionality reduction and survey relevant errorbound results from matrix completion literature. They are included in the first three subsections below and collectively serve as a solid motivation for the current study. We finally summarize our main contributions with notation used in this paper.
1.1 Distances in social network and their embedding
The study of structural patterns of social network from the ties (relationships) that connect social actors is one of the most important research topics in social network analysis [59]. To this end, measurements on the actortoactor relationships (kinship, social roles, etc.) are collected or observed by different methods (questionnaires, direct observation, etc.) and the measurements on the relational information are referred as the network composition. The measurement data usually can be presented as an \(n \times n\) measurement matrix, where the n rows and the n columns both refer to the studied actors. Each entry of these matrices indicates the social relationship measurement (e.g., presence/absence or similarity/dissimilarity) between the row and column actors. In this paper, we are only concerned with symmetric relationships, i.e., the relationship from actor i to actor j is the same as that from actor j to actor i. Furthermore, there exist standard ways to convert the measured relationships into Euclidean distances, see [18, Sect. 1.3.5] and [8, Chp. 6].
However, it is important to note that in practice, only partial relationship information can be observed, which means that the measurement matrix is usually incomplete and noisy. The observation processes are often assumed to follow certain random graph model. One simple but widely used model is the Bernoulli random graph model [20, 52]. Let n labelled vertices be given. The Bernoulli random graph is obtained by connecting each pair of vertices independently with the common probability p and it reproduces well some principal features of the realworld social network such as the “smallworld” effect [19, 40]. Other properties such as the degree distribution and the connectivity can be found in e.g., [7, 27]. For more details on the Bernoulli as well as other random models, one may refer to the review paper [42] and references therein. In this paper, we mainly focus on the Bernoulli random graph model. Consequently, the observed measurement matrix follows the uniform sampling rule which will be described in Sect. 2.3.
In order to examine the structural patterns of a social network, the produced images (e.g., embedding in 2 or 3 dimensional space for visualization) should preserve the structural patterns as much as possible, as highlighted by Freeman et al. [23], the points in a visual image should be located so the observed strengths of the interactor ties are preserved. In other words, the designed dimensional reduction algorithm has to assure that the embedding Euclidean distances between points (nodes) fit in the best possible way the observed distances in a social space. Therefore, the problem now reduces to whether one can effectively find the best approximation in a low dimensional space to the true social measurement matrix, which is incomplete and noisy. The classical Multidimensional Scaling (cMDS) (see Sect. 2.1) provides one of the most often used embedding methods. However, cMDS alone is often not adequate to produce satisfactory embedding, as rightly observed in several highprofile embedding methods in manifold learning.
1.2 Embedding methods in manifold learning
The cMDS and its variants have found many applications in data dimension reduction and have been well documented in the monographs [8, 18]. When the distance matrix (or dissimilarity measurement matrix) is close to a true EDM with the targeted embedding dimension, cMDS often works very well. Otherwise, a large proportion of unexplained variance has to be cut off or it may even yield negative variances, resulting in what is called embedding in a pseudoEuclidean space and hence creating the problem of unconventional interpretation of the actual embedding (see e.g., [46]).
cMDS has recently motivated a number of highprofile numerical methods, which all try to alleviate the issue mentioned above. For example, the ISOMAP of [54] proposes to use the shortest path distances to approximate the EDM on a lowdimensional manifold. The Maximum Variance Unfolding (MVU) of [61] through SDP aims for maximizing the total variance and the Minimum Volume Embedding (MVE) of [51] also aims for a similar purpose by maximizing the eigen gap of the Gram matrix of the embedding points in a lowdimensional space. The need for such methods comes from the fact that the initial distances either are in stochastic nature (e.g., containing noises) or cannot be measured (e.g., missing values). The idea of MVU has also been used in the refinement step of the celebrated SDP method for sensor network localization problems [6].
It was shown in [4, 54] that ISOMAP enjoys the elegant theory that the shortest path distances (or graph distances) can accurately estimate the true geodesic distances with a high probability if the finite points are chosen randomly from a compact and convex submanifold following a Poisson distribution with a high density, and the pairwise distances are obtained by the knearest neighbor rule or the unit ball rule (see Sect. 2.3 for the definitions). However, for MVU and MVE, there exist no theoretical guarantee as to how good the obtained Euclidean distances are. At this point, it is important to highlight two observations. (i) The shortest path distance or the distance by the knearest neighbor or the unitball rule is often not suitable in deriving distances in social network. This point has been emphasized in the recent study on Email social network by Budka et al. [10]. (ii) MVU and MVE models only depend on the initial distances and do not depend on any particular ways in obtaining them. They then rely on SDP to calculate the best fit distances. From this point of view, they can be applied to social network embedding. This is also pointed out in [10]. Due to the space limitation, we are not able to review other leading methods in manifold learning, but refer to [12, Chp. 4] for a guide.
Inspired by their numerical success, our model will inherit the good features of both MVU and MVE. Moreover, we are able to derive theoretical results in guaranteeing the quality of the obtained Euclidean distances. Our results are the type of error bounds, which have attracted growing attention recently. We review the relevant results below.
1.3 Error bounds in lowrank matrix completion and approximation
As mentioned in the preceding section, our research has been strongly influenced by the group of researches that are related to the MVU and MVE models, which have natural geometric interpretations and use SDP as their major tool. Their excellent performance in data reduction calls for theoretical justification.
Our model also enjoys a similar geometric interpretation, but departs from the two models in that we deal with EDM directly rather than reformulating it as SDP. This key departure puts our model in the category of matrix approximation problems, which have attracted much attention recently from machine learning community and motivated our research.
The most popular approach to recovering a lowrank matrix solution of a linear system is via the nuclear norm minimization [22, 38]. What makes this approach more exciting and important is that it has a theoretically guaranteed recoverability (recoverable with a high probability). The first such a theoretical result was obtained by Recht et al. [48] by employing the Restricted Isometric Property (RIP). However, for the matrix completion problem the sample operator does not satisfy the RIP (see e.g., [13]). For the noiseless case, Candès and Recht [14] proved that a lowrank matrix can be fully recovered with high probability provided that a small number of its noiseless observations are uniformly sampled. See [15] for an improved bound and [26] for the optimal bound on the sample number. We also refer to [47] for a short and intelligible analysis of the recoverability of the matrix completion problem.
The matrix completion with noisy observations was studied by Candès and Plan [13]. Recently, the noisy case was further studied by several groups of researchers including [34, 41] and [32], under different settings. In particular, the matrix completion problem with fixed basis coefficients was studied by Miao et al. [39], who proposed a rankcorrected procedure to generate an estimator using the nuclear seminorm and established the corresponding nonasymmetric recovery bounds.
The impressive result in [28] roughly states that the obtained error bound reads as \(O((nr^{d})^{5}\frac{\varDelta }{r^{4}})\) containing an undesirable term \((nr^{d})^{5}\), where r is the radius used in the unit ball rule, d is the embedding dimension, \(\varDelta \) is the bound on the measurement noise and n is the number of embedding points. As pointed out by Javanmard and Montanari [28] that the numerical performance suggested the error seems to be bounded by \(O(\frac{\varDelta }{r^{4}})\), which does not match the derived theoretical bound. This result also shows tremendous technical difficulties one may have to face in deriving similar bounds for EDM recovery.
To summarize, most existing error bounds are derived from the nuclear norm minimization. When translating to the Euclidean distance learning problem, minimizing the nuclear norm is equivalent to minimizing the variance of the embedding points, which contradicts the main idea of MVU and MVE in making the variance as large as possible. Hence, the excellent progress in matrix completion/approximation does not straightforwardly imply useful bounds about the Euclidean distance learning in a lowdimensional space. Actually one may face huge difficulty barriers in such extension. In this paper, we propose a convex optimization model to learn faithful Euclidean distances in a lowdimensional space. We derive theoretically guaranteed bounds in the spirit of matrix approximation and therefore provide a solid theoretical foundation in using the model. We briefly describe the main contributions below.
1.4 Main contributions
This paper makes two major contributions to the field of nonlinear dimensionality reduction. One is on building a convex optimization model with guaranteed error bounds and the other is on a computational method.
(a) Building a convex optimization model and its error bounds. Our departing point from the existing SDP models is to treat EDM (vs positive semidefinite matrix in SDP) as a primary object. The total variance of the desired embedding points in a lowdimensional space can be quantitatively measured through the socalled EDM score. The higher the EDM score is, the more the variance is explained in the embedding. Therefore, both MVU and MVE can be regarded as EDM score driven models. Moreover, MVE, being a nonconvex optimization model, is more aggressive in driving the EDM score up. However, MVU, being a convex optimization model, is more computationally appealing. Our convex optimization model strikes a balance between the two models in the sense that it inherits the appealing features from both.
What makes our model more important is that it yields guaranteed nonasymptotic error bounds under the uniform sampling rule. More precisely, we show in Theorem 1 that for the unknown \(n \times n\) Euclidean distance matrix with the embedding dimension r and under mild conditions, the average estimation error is controlled by \(C{rn\log (n)}/{m}\) with high probability, where m is the sample size and C is a constant independent of n, r and m. It follows from this error bound that our model will produce an estimator with high accuracy as long as the sample size is of the order of \(rn\log (n)\), which is roughly the degree of freedom of a symmetric hollow matrix with rank r up to a logarithmic factor in the matrix size. It is worth to point out that with special choices of model parameters, our model reduces to MVU and covers the subproblems solved by MVE. Moreover, our theoretical result corresponding to those specific model parameters explains why under the uniform sampling rule, the MVE often leads to configurations of higher quality than the MVU. To our knowledge, it is the first such theoretical result that shed lights on the MVE model. There are some theoretical results on the asymptotic behavior of MVU obtained recently in [2, 45]. However, these results are different from ours in the sense that they are only true when the number of the points is sufficiently large.
(b) An efficient computational method. Treating EDM as a primary object not only benefits us in deriving the errorbound results, but also leads to an efficient numerical method. It allows us to apply a recently proposed convergent 3block alternating direction method of multipliers (ADMM) [3] even for problems with a few thousands of data points. Previously, the original models of both MVU and MVE have numerical difficulties when the data points are beyond 1000. They may even have difficulties with a few hundreds of points when their corresponding slack models are to be solved. In order to increase the scalability of MVU, some algorithms are proposed in [62]. Most recently, Chen et al. [16] derive a novel variant of MVU: the Maximum Variance Correction (MVC), which greatly improves its scalability. However, for some social network applications, the quality of the embedding graph form MVC is questionable, probably because there is no theoretical guarantee on the embedding accuracy. For instance, as shown in Sect. 6, for US airport network (1572 nodes) and Political blogs (1222 nodes), MVC embedding failed to capture any important features in the two networks, although it is much faster in computing time.
Moreover, We are also able to develop theoretically optimal estimates of the model parameters. This gives a good indication how we should set the parameter values in our implementation. Numerical results both on social networks and the benchmark test problems in manifold learning show that our method can fast produce embeddings of high quality.
1.5 Organization and notation
The paper is organized as follows. Section 2 provides necessary background with a purpose to cast the MVU and MVE models as EDMscore driven models. This viewpoint will greatly benefit us in understanding our model, which is described in Sect. 3 with more detailed interpretation. We report our error bound results in Sect. 4. Sect. 5 contains the theoretical optimal estimates of the model parameters as well as a convergent 3block ADMM algorithm. We report our extensive numerical experiments in Sect. 6 and conclude the paper in Sect. 7.

For any \(Z\in \mathfrak {R}^{m\times n}\), we denote by \(Z_{ij}\) the (i, j)th entry of Z. We use \({\mathbb O}^n\) to denote the set of all n by n orthogonal matrices.

For any \(Z\in \mathfrak {R}^{m\times n}\), we use \({\mathbf {z}}_{j}\) to represent the jth column of Z, \(j=1,\ldots ,n\). Let \({\mathscr {J}}\subseteq \{1,\ldots , n\}\) be an index set. We use \( Z_{{\mathscr {J}}}\) to denote the submatrix of Z obtained by removing all the columns of Z not in \({\mathscr {J}}\).

Let \({\mathscr {I}}\subseteq \{1,\ldots , m\}\) and \({\mathscr {J}}\subseteq \{1,\ldots , n\}\) be two index sets. For any \(Z\in \mathfrak {R}^{m\times n}\), we use \(Z_{{\mathscr {IJ}}}\) to denote the \({\mathscr {I}}\times {\mathscr {J}}\) submatrix of Z obtained by removing all the rows of Z not in \({\mathscr {I}}\) and all the columns of Z not in \({\mathscr {J}}\).

We use “\(\circ \)” to denote the Hadamard product between matrices, i.e., for any two matrices X and Y in \(\mathfrak {R}^{m\times n}\) the (i, j)th entry of \( Z:= X\circ Y \in \mathfrak {R}^{m\times n}\) is \(Z_{ij}=X_{ij} Y_{ij}\).

For any \(Z\in \mathfrak {R}^{m\times n}\), let \(\Vert Z\Vert _2\) be the spectral norm of Z, i.e., the largest singular value of Z, and \(\Vert Z\Vert _*\) be the nuclear norm of Z, i.e., the sum of singular values of Z. The infinity norm of Z is denoted by \(\Vert Z\Vert _\infty \).
2 Background
This section contains three short parts. We first give a brief review of cMDS, only summarizing some of the key results that we are going to use. We then describe the MVU and MVE models, which are closely related to ours. Finally, we explain three most commonly used distancesampling rules.
2.1 cMDS
cMDS has been well documented in [8, 18]. In particular, Section 3 of [46] explains when it works. Below we only summarize its key results for our future use. A \(n\times n\) matrix D is called Euclidean distance matrix (EDM) if there exist points \({\mathbf {p}}_1,\ldots , {\mathbf {p}}_n\) in \(\mathfrak {R}^r\) such that \(D_{ij}=\Vert {\mathbf {p}}_i {\mathbf {p}}_j\Vert ^2\) for \(i,j=1,\ldots ,n\), where \(\mathfrak {R}^r\) is called the embedding space and r is the embedding dimension when it is the smallest such r.
Lemma 1
For any \(X\in {{\mathbb {S}}}_h^n\), we have \(XJXJ=\frac{1}{2}\left( \mathrm{diag}(JXJ)\,\mathbf{1}^T+\mathbf{1}\,\mathrm{diag}(JXJ)^T\right) \).
2.2 MVU and MVE models
The resulting EDM \(D \in {{\mathbb {S}}}^n\) from the optimal solution of (7) is defined to be \( D_{ij} = K_{ii}  2K_{ij} + K_{jj}\) and it satisfies \(K = 0.5 \textit{JDJ}. \) Empirical evidence shows that the EDM scores of the first few leading eigenvalues of K are often large enough to explain high percentage of the total variance.
2.3 Distance sampling rules
 (i)
Uniform sampling rule The elements are independently and identically sampled from \(\varOmega \) with the common probability \({1}/{\varOmega }\).
 (ii)
k nearest neighbors (kNN) rule For each i, \((i,j) \in \varOmega _0\) if and only if \(d_{ij}\) belongs to the first k smallest distances in \(\{d_{i\ell }: i \not = \ell =1, \ldots , n \}\).
 (iii)
Unit ball rule For a given radius \(\epsilon >0\), \((i,j) \in \varOmega _0\) if and only if \(d_{ij} \le \epsilon \).
3 A convex optimization model for distance learning
Both MVU and MVE are trusted distance learning models in the following sense. They both produce a Euclidean distance matrix, which is faithful to the observed distances and they both encourage high EDM scores from the first few leading eigenvalues. However, it still remains a difficult (theoretical) task to quantify how good the resulting embedding is. In this part, we will propose a new learning model, which inherit the good properties of MVU and MVE. Moreover, we are able to quantify the embedding quality by deriving error bounds of the resulting solutions under the uniform sampling rule. Below, we first describe our model, followed by detailed interpretation.
3.1 Model description
3.2 Model interpretation
The three tasks that model (13) tries to accomplish correspond to the three terms in the objective function. The first (quadratic) term is nothing but \(\sum _{(i,j) \in \varOmega _0} ( d_{ij}^2  D_{ij} )^2\) corresponding to the quadratic terms in the slack models (7) and (8). Minimizing this term (i.e, leastsquares) is essentially to find an EDM D that minimizes the error rising from the sampling model (11).
The second term \(\langle I, \; \textit{JDJ} \rangle \) is actually the nuclear norm of \((\textit{JDJ})\). Recall that in cMDS, the embedding points in (2) come from the spectral decomposition of \((\textit{JDJ})\). Minimizing this term means to find the smallest embedding dimension. However, as argued in both MVU and MVE models, minimizing the nuclear norm is against the principal idea of maximizing variance. Therefore, to alleviate this conflict, we need the third term \( \langle \widetilde{P}_1 \widetilde{P}_1^T, \; \textit{JDJ} \rangle \).
It is easy to see that Model (13) reduces to the nuclear norm penalized least squares (NNPLS) model if \(\rho _{2}=0\) ^{1} and the MVU model (with the bounded constraints) if \(\rho _2=2\) and \(\varTheta =I\). Meanwhile, let \(\rho _2 = 2\) and \(\widetilde{D}\) to be one of the iterates in the MVE SDP subproblems (with the bounded constraints). The combined term \(\langle I, \; \textit{JDJ} \rangle  2 \langle \widetilde{P}_1 \widetilde{P}_1^T, \ \textit{JDJ} \rangle \) is just the objective function in the MVE SDP subproblem. In other words, MVE keeps updating \(\widetilde{D}\) by solving the SDP subproblems. Therefore, Model (13) covers both MVU and MVE models as special cases. the errorbound results (see the remark after Theorem 1 and Prop. 5) obtained in next section will partially explain why under the uniform sampling rule, our model often leads to higher quality than NNPLS, MVU and MVE.
Before we go on to derive our promised errorbound results, we summarize the key points for our model (13). It is EDM based rather than SDP based as in the most existing research. The use of EDM enables us to establish the errorbound results in the next section. It inherits the nice properties in MVU and MVE models. We will also show that this model can be efficiently solved.
4 Error bounds under uniform sampling rule
The derivation of the error bounds below, though seemingly complicated, has become standard in matrix completion literature. We will refer to the exact references whenever similar results (using similar proof techniques) have appeared before. For those who are just interested in what the error bounds mean to our problem, they can jump to the end of the section (after Theorem 1) for more interpretation.
Lemma 2
Now we are ready to study the error bounds of the model (13). It is worth to note that the optimal solution of the convex optimization problem (13) always exists, since the feasible set is nonempty and compact. Denote an optimal solution of (13) by \(D^*\). The following result represents the first major step to derive our ultimate bound result. It contains two bounds. The first bound (19) is on the normsquared distance between \(D^*\) and \(\overline{D}\) under the observation operator \({\mathscr {O}}\). The second bound (20) is about the nuclear norm of \(D^*  \overline{D}\). Both bounds are in terms of the Frobenius norm of \(D^*  \overline{D}\).
Proposition 1
Proof
Lemma 3
Proof
Next, combining Proposition 1 and Lemma 3 leads to the following result.
Proposition 2
Proof
 Case 1 If \({\mathbb E}\left( \langle D^*\overline{D},X\rangle ^2\right) <8b^2 \displaystyle {\sqrt{\frac{256\log (2n)}{m\log (2)}}}\), then we know from (14) that$$\begin{aligned} \frac{\Vert D^*\overline{D}\Vert ^2}{\varOmega }< 16b^2 \sqrt{\frac{256\log (2n)}{m\log (2)}}= 16b^2 \sqrt{\frac{256}{\log (2)}}\sqrt{\frac{\log (2n)}{m}}. \end{aligned}$$
 Case 2 If \({\mathbb E}\left( \langle D^*\overline{D},X\rangle ^2\right) \ge 8b^2 \displaystyle {\sqrt{\frac{256\log (2n)}{m\log (2)}}}\), then we know from (20) that \((D^*\overline{D})/\sqrt{2}\Vert D^*\overline{D}\Vert _\infty \in {\mathscr {C}}(\tau )\) with \(\tau =2r(\frac{\kappa }{\kappa 1})^2\left( \alpha (\rho _2)+2\right) ^2\). Thus, it follows from Lemma 3 that there exists a constant \(C_2'>0\) such that with probability at least \(11/n\),$$\begin{aligned} \frac{1}{2}{\mathbb E}\left( \langle D^*\overline{D},X\rangle ^2\right) \le \frac{1}{m}\Vert {\mathscr {O}}(D^*\overline{D})\Vert ^2+2048C'_2b^2\tau \varOmega \frac{\log (2n)}{nm}. \end{aligned}$$
This bound depends on the model parameters \(\rho _1\) and \(\rho _2\). In order to establish an explicit error bound, we need to estimate \(\rho _1\) (\(\rho _2\) will be estimated later), which depends on the quantity \(\left\ \frac{1}{m}{\mathscr {O}}^*(\zeta ) \right\ _2\), where \(\zeta =(\zeta _1,\ldots ,\zeta _m)^T\in \mathfrak {R}^m\) with \(\zeta _l\), \(l=1,\ldots ,m\) are i.i.d. random variables given by (18). To this end, from now on, we always assume that the i.i.d. random noises \(\xi _l\), \(l=1,\ldots ,m\) in the sampling model (11) satisfy the following subGaussian tail condition.
Assumption 3
By applying the Bernstein inequality (Lemma 2), we have
Proposition 4
Proof
Theorem 1
The major message from Theorem 1 is as follows. We know that if the true Euclidean distance matrix \(\overline{D}\) is bounded, and the noises are small (less than the true distances), in order to control the estimation error, we only need samples with the size m of the order \(r(n1)\log (2n)/2\), since \(\varOmega =n(n1)/2\). Note that, \(r= \mathrm{rank}(J\overline{D}J)\) is usually small (2 or 3). Therefore, the sample size m is much smaller than \(n(n1)/2\), the total number of the offdiagonal entries. Moreover, since the degree^{3} of freedom of n by n symmetric hollow matrix with rank r is \(n(r1)r(r1)/2\), the sample size m is close to the degree of freedom if the matrix size n is large enough. However, we emphasize that one cannot obtain exact recovery from the bound (40) even without noise, i.e., \(\eta =0\). As mentioned in [41], this phenomenon is unavoidable due to lack of identifiability. For instance, consider the EDM \(\overline{D}\) and the perturbed EDM \(\widetilde{D}=\overline{D}+\varepsilon {\mathbf {e}}_1 {\mathbf {e}}_1^T\). Thus, with high probability, \({\mathscr {O}}(D^*)={\mathscr {O}}(\widetilde{D})\), which implies that it is impossible to distinguish two EDMs even if they are noiseless. If one is interested only in exact recovery in the noiseless setting, some addition assumptions such as the matrix incoherence conditions are necessary.
Finally, we also want to compare our error bound result in Theorem 1 with the result obtained in [57, Section 7]. The results obtained in [57] is for the sensor network localization where some location points are fixed as anchors. This makes the corresponding analysis completely different. Moreover, roughly speaking, the estimation error of the secondorder cone relaxation is bounded by the square root of the distance error, which is a function of estimator (see [57, Proposition 7.2]). This means that the righthand side of the error bound obtained by Tseng [57] depends on the resulting estimator. However, the error bound proved in Theorem 1 only depends on the initial input data of problems.
5 Model parameter estimation and the algorithm
In general, the choice of model parameters can be tailored to a particular application. A very useful property about our model (13) is that we can derive a theoretical estimate, which serves as a guideline for the choice of the model parameters in our implementation. In particular, we set \(\rho _1\) by (39) and prove that \(\rho _2 =1 \) is a better choice than both the case \(\rho _2 =0\) (corresponding to the nuclear norm minimization) and \(\rho _2=2\) (MVE model). The first part of this section is to study the optimal choice of \(\rho _2\) and the second part briefly introduces a convergent 3block alternating direction method of multipliers (ADMM) algorithm, which is particularly suitable to our model.
5.1 Optimal estimate of \(\rho _2\)
This shows that \(\rho _2 =1\) is nearly optimal if the initial estimator \(\widetilde{D}\) is close to \(\overline{D}\). We will show that in terms of the estimation errors the choice \(\rho _{2}=1\) is always better than the nuclear norm penalized least squares model (\(\rho _{2}=0\)) and the minimum volume embedding model (\(\rho _{2}=2\)).
Proposition 5
If \( \Vert \widetilde{D}\overline{D}\Vert <\overline{\lambda }_{r}/2\), then \(\alpha (1)<\min \left\{ \alpha (0),\alpha (2)\right\} \).
Proof
5.2 A convergent 3block ADMM algorithm
6 Numerical experiments
In this section, we demonstrate the effectiveness of the proposed EDM Embedding (EDME) model (13) by testing on some real world examples. The examples are in two categories: one is of the social network visualization problem, whose initial link observation can be modelled by uniform random graphs. The other is from manifold learning, whose initial distances are obtained by the kNN rule. The known physical features of those problems enable us to evaluate how good EDME is when compared to other models such as ISOMAP and MVU. It appears that EDME is capable of generating configurations of very high quality both in terms of extracting those physical features and of higher EDM scores. The test also raises an open question whether our theoretical results can be extended to this case where the kNN rule is used.
For comparison purpose, we also report the performance of MVU and ISOMAP for most cases. The SDP solver used is the stateofart SDPT3 package, which allows us to test problems of large data sets. We did not compare with MVE as it solves a sequence of SDPs and consequently it is too slow for our tested problems. Details on this and other implementation issues can be found in Sect. 6.3.
6.1 Social networks
Two realworld networks arising from the different applications are used to demonstrate the quality of our new estimator from EDME.
Numerical performance comparison of the MVU and the EDME
Problems  MVU (SDPT3)  EDME  

n/\(\mathrm{edges}\)  relgap  EDMscore (%)  CPU (s)  R  EDMscore (%)  CPU (s)  
Enron  182/2097  3.25e−04  48.1  5.46  9.92e−04  100  1.05 
Facebooklike  1893/13835  4.40e−04  20.6  624.18  9.78e−04  100  165.34 
TrainBombing  64/243  9.38e−04  91.8  0.49  9.99e−04  100  0.56 
USairport2010  1572/17214  7.30e−04  69.3  31587.08  9.98e−04  100  80.53 
Blogs  1222/16714  4.71e−04  60.5  12173.31  8.40e−04  100  83.66 
k/n/\(\mathrm{edges}\)  relgap  EDMscore (%)  CPU (s)  R  EDMscore (%)  CPU (s)  

Teapots400  5/400/1050  8.45e−04  100  3.44  9.75e−04  100  3.77 
Face698  5/698/2164  2.96e−04  100  14.25  9.94e−04  100  29.73 
Digit1  6/1135/4885  7.49e−04  98.1  68.85  9.95e−04  100  39.62 
Digits19  6/1000/4394  6.57e−04  94.0  51.02  9.83e−04  100  27.66 
FreyFace  5/1965/6925  9.48e−04  86.2  214.41  8.72e−04  100  187.56 
6.2 Manifold learning
In this subsection, we test two widely used data sets in manifold learning. The initial distances used are generated by the kNN rule. We describe them below with our findings for MVU, EDME and the celebrated manifold learning algorithm ISOMAP.
6.3 Numerical performance
We tested the ISOMAP, the MVU and our proposed EDME methods in MATLAB 8.5.0.197613 (R2015a), and the numerical experiments are run in MATLAB under a Windows 10 64bit system on an Intel 4 Cores i7 3.60GHz CPU with 8GB memory.
Besides the examples mentioned before, the following examples are also tested: the Enron email dataset [17], the facebooklike social network [44], the Madrid train bombing data [9] (downloaded from [24]), the teapots data [61], the digits “1” and “9” and the Frey face images data [49]. To save space, we do not include the actual embedding graphs for these examples, but just report the numerical performance in Table 1.
We observe that the performance of EDME is outstanding in terms of numerical efficiency. Taking USairport2010 as example, MVU used about 10 h while EDME only used about 80 s. For the examples in manifold learning, the gap between the two models are not as severe as for the social network examples. The main reason is that the initial guess obtained by ISOMAP is a very good estimator that can roughly capture the lowdimensional features in manifold learning. However, it fails to capture meaningful features for the social network examples. This echoes the comment made in [10] that the shortest path distance is not suitable to measure the distances in social networks. We also like to point out that for all tested problems, EDME captured nearly \(100\%\) variance and it treats the local features equally important in terms of the leading eigenvalues being of the same magnitude.
7 Conclusions
The paper aimed to explain a mysterious situation regarding the SDP methodology to reconstruct faithful Euclidean distances in a lowdimensional space from incomplete set of noisy distances. The SDP models can construct numerical configurations of high quality, but they lack theoretical backups in terms of bounding errors. We took a completely different approach that heavily makes use of Euclidean Distance Matrix instead of positive semidefinite matrix in SDP models. This led to a convex optimization that inherits the nice features of MVU and MVE models. More importantly, we were able to derive errorbound results under the uniform sampling rule. The optimization problem can also be efficiently solved by the proposed algorithm. Numerical results in both social networks and manifold leading showed that our model can capture lowdimensional features and treats them equally important.
Given that our model worked very well for the manifold learning examples, an interesting question regarding this approach is whether the theoretical errorbound results can be extended to the case where the distances are obtained by the kNN rule. It seems very difficult if we follow the technical proofs in this paper. It also seems that the approach of [28] would lead to some interesting (but very technical) results. We plan to investigate those issues in future.
Footnotes
Notes
Acknowledgements
We would like to thank the referees as well as the associate editor for their constructive comments that have helped to improve the quality of the paper.
References
 1.Adamic, A.A., Glance, N.: The political blogosphere and the 2004 US election: divided they blog. In: Proceedings of the 3rd International Workshop on Link Discovery (2005)Google Scholar
 2.AriasCastro, E., Pelletier, B.: On the convergence of maximum variance unfolding. J. Mach. Learn. Res. 14, 1747–1770 (2013)MathSciNetzbMATHGoogle Scholar
 3.Bai, S.H., Qi, H.D.: Tackling the flip ambiguity in wireless sensor network localization and beyond. http://www.personal.soton.ac.uk/hdqi/REPORTS/EDMSNL (2015)
 4.Bernstein, M., De Silva, V., Langford, J.C., Tenenbaum, J.B.: Graph approximations to geodesics on embedded manifolds. http://isomap.stanford.edu/BdSLT, Stanford University (2000)
 5.Bhatia, R.: Matrix Analysis. Springer, New York (1997)CrossRefzbMATHGoogle Scholar
 6.Biswas, P., Liang, T.C., Toh, K.C., Ye, Y., Wang, T.C.: Semidefinite programming approaches for sensor network localization with noisy distance measurements. IEEE Trans. Autom. Sci. Eng. 3, 360–371 (2006)CrossRefGoogle Scholar
 7.Bollobás, B.: Random Graphs. Cambridge University Press, Cambridge (2001)CrossRefzbMATHGoogle Scholar
 8.Borg, I., Groenen, P.J.F.: Modern Multidimensional Scaling. Springer, Berlin (2005)zbMATHGoogle Scholar
 9.Brian, V.: Connecting the dots. Am Sci 95, 400–404 (2006)Google Scholar
 10.Budka, M., Juszczyszyn, K., Musial, K., Musial, A.: Molecular model of dynamic social network based on email communication. Soc. Netw. Anal. Min. 3, 543–563 (2013)CrossRefGoogle Scholar
 11.Bühlmann, P., Van De Geer, S.: Statistics for HighDimensional Data: Methods, Theory and Applications. Springer, Berlin (2011)CrossRefzbMATHGoogle Scholar
 12.Burges, C.J.C.: Dimension reduction: a guided tour. Found. Trend Mach. Learn. 2, 275–365 (2009)CrossRefzbMATHGoogle Scholar
 13.Candès, E.J., Plan, Y.: Matrix completion with noise. Proc. IEEE 98, 925–936 (2010)CrossRefGoogle Scholar
 14.Candès, E.J., Recht, B.: Exact matrix completion via convex optimization. Found. Comput. Math. 9, 717–772 (2008)MathSciNetCrossRefzbMATHGoogle Scholar
 15.Candès, E.J., Tao, T.: The power of convex relaxation: nearoptimal matrix completion. IEEE Trans. Inf. Theory 56, 2053–2080 (2010)MathSciNetCrossRefGoogle Scholar
 16.Chen, W., Chen, Y., Weinberger, K.Q.: Maximum variance correction with application to A* search. In: Proceedings of the 30th International Conference Machine Learning (ICML13), pp. 302–310 (2013)Google Scholar
 17.Cohen, W.W., William, W.: Enron email dataset (2009)Google Scholar
 18.Cox, T.F., Cox, M.A.A.: Multidimensional Scaling, 2nd edn. Chapman and Hall/CRC, Boca Raton (2001)zbMATHGoogle Scholar
 19.de Sola Pool, I., Kochen, M.: Contacts and influence. Soc. Netw. 1, 5–51 (1979)MathSciNetCrossRefGoogle Scholar
 20.Erdős, P., Rényi, A.: On random graphs. Publicationes Mathematicae Debrecen 6, 290–297 (1959)MathSciNetzbMATHGoogle Scholar
 21.Fan, K.: On a theorem of Weyl concerning eigenvalues of linear transformations I. Proc. Nat. Acad. Sci. 35, 652–655 (1949)MathSciNetCrossRefGoogle Scholar
 22.Fazel, M.: Matrix Rank Minimization with Applications. Ph.D. Thesis, Stanford University (2002)Google Scholar
 23.Freeman, L.C.: Graphic techniques for exploring social network data. In: Carrington, Peter J., Scott, J., Wasserman, s.(eds) Models and Methods in Social Network Analysis, vol.28 Cambridge University Press, Cambridge. p. 248–269 (2005)Google Scholar
 24.Freeman, L.C.: Freeman Datasets. http://moreno.ss.uci.edu/data.html (2010)
 25.Gower, J.C.: Some distance properties of latent root and vector methods used in multivariate analysis. Biometrika 53, 325–338 (1966)MathSciNetCrossRefzbMATHGoogle Scholar
 26.Gross, D.: Recovering lowrank matrices from few coefficients in any basis. IEEE Trans. Inf. Theory 57, 1548–1566 (2011)MathSciNetCrossRefGoogle Scholar
 27.Janson, S., Luczak, T., Rucinski, A.: Random Graphs. Wiley, Hoboken (2011)zbMATHGoogle Scholar
 28.Javanmard, A., Montanari, A.: Localization from incomplete noisy distance measurements. Found. Comput. Math. 13, 297–345 (2013)MathSciNetCrossRefzbMATHGoogle Scholar
 29.Keshavan, R.H., Montanari, A., Oh, S.: Matrix completion from noisy entries. J. Mach. Learn. Res. 11, 2057–2078 (2010)MathSciNetzbMATHGoogle Scholar
 30.Klavans, R., Boyack, K.W.: Identifying a better measure of relatedness for mapping science. J. Am. Soc. Inf. Sci. Technol. 57, 251–263 (2006)CrossRefGoogle Scholar
 31.Klopp, O.: Rank penalized estimators for highdimensional matrices. Electron. J. Stat. 5, 1161–1183 (2011)MathSciNetCrossRefzbMATHGoogle Scholar
 32.Klopp, O.: Noisy lowrank matrix completion with general sampling distribution. Bernoulli 20, 282–303 (2014)MathSciNetCrossRefzbMATHGoogle Scholar
 33.Koltchinskii, V.: Oracle inequalities in empirical risk minimization and sparse recovery problems. In: Ecole d’Eté de Probabilités de SaintFlour XXXVIII2008, vol. 2033, Springer (2011)Google Scholar
 34.Koltchinskii, V., Lounici, K., Tsybakov, A.B.: Nuclearnorm penalization and optimal rates for noisy lowrank matrix completion. Ann. Stat. 39, 2302–2329 (2011)MathSciNetCrossRefzbMATHGoogle Scholar
 35.LeCun, Y., Cortes, C., Burges, C.J.C.: MNIST. http://yann.lecun.com/exdb/mnist/ (1998)
 36.Ledoux, M., Talagrand, M.: Probability in Banach Spaces: Isoperimetry and Processes. Springer, Berlin (1991)CrossRefzbMATHGoogle Scholar
 37.Li, X., Sun, D.F., Toh, K.C.: A schur complement based semiproximal ADMM for convex quadratic conic programming and extensions. Math. Prog. 155, 333–373 (2016)CrossRefzbMATHGoogle Scholar
 38.Mesbahi, M.: On the rank minimization problem and its control applications. Syst. Control Lett. 33, 31–36 (1998)MathSciNetCrossRefzbMATHGoogle Scholar
 39.Miao, W., Pan, S., Sun, D.F.: A rankcorrected procedure for matrix completion with fixed basis coefficients. Math. Prog. (2016). doi: 10.1007/s1010701509617 MathSciNetzbMATHGoogle Scholar
 40.Milgram, S.: The small world problem. Psychol. Today 2, 60–67 (1967)Google Scholar
 41.Negahban, S., Wainwright, M.J.: Restricted strong convexity and weighted matrix completion: optimal bounds with noise. J. Mach. Learn. Res. 13, 1665–1697 (2012)MathSciNetzbMATHGoogle Scholar
 42.Newman, M.E.J.: The structure and function of complex networks. SIAM Rev. 45, 167–256 (2003)MathSciNetCrossRefzbMATHGoogle Scholar
 43.Opsahl, T.: US Airport 2010. http://toreopsahl.com/datasets/#usairports (2011)
 44.Opsahl, T., Panzarasa, P.: Clustering in weighted networks. Soc. Netw. 31, 155–163 (2009)CrossRefGoogle Scholar
 45.Paprotny, A., Garcke, J.: On a connection between maximum variance unfolding, shortest path problems and isomap. In: International Conference on Artificial Intelligence and Statistics, pp. 859–867 (2012)Google Scholar
 46.Pȩkalska, E., Paclík, P., Duin, P.W.: A generalized kernel approach to dissimilaritybased classification. J. Mach. Learn. Res. 2, 175–211 (2002)MathSciNetzbMATHGoogle Scholar
 47.Recht, B.: A simpler approach to matrix completion. J. Mach. Learn. Res. 12, 3413–3430 (2011)MathSciNetzbMATHGoogle Scholar
 48.Recht, B., Fazel, M., Parrilo, P.A.: Guaranteed minimumrank solutions of linear matrix equations via nuclear norm minimization. SIAM Rev. 52, 471–501 (2010)MathSciNetCrossRefzbMATHGoogle Scholar
 49.Roweis, S.T., Saul, L.K.: Frey Face. http://www.cs.nyu.edu/~roweis/data.html (2000)
 50.Schoenberg, I.J.: Remarks to Maurice Fréchet’s article “Sur la définition axiomatque d’une classe d’espaces vectoriels distanciés applicbles vectoriellement sur l’espace de Hilbet”. Ann. Math. 36, 724–732 (1935)MathSciNetCrossRefzbMATHGoogle Scholar
 51.Shaw, B., Jebara, T.: Minimum volume embedding. In: International Conference on Artificial Intelligence and Statistics, pp. 460–467 (2007)Google Scholar
 52.Solomonoff, R., Rapoport, A.: Connectivity of random nets. Bull. Math. Biophys. 13, 107–117 (1951)MathSciNetCrossRefGoogle Scholar
 53.Sun, J., Boyd, S., Xiao, L., Diaconis, P.: The fastest mixing Markov process on a graph and a connection to a maximum variance unfolding problem. SIAM Rev. 48, 681–699 (2006)MathSciNetCrossRefzbMATHGoogle Scholar
 54.Tenenbaum, J.B., De Silva, V., Langford, J.C.: A global geometric framework for nonlinear dimensionality reduction. Science 290, 2319–2323 (2000)CrossRefGoogle Scholar
 55.Toh, K.C., Todd, M.J., Tütüncü, R.H.: SDPT3—a MATLAB software package for semidefinite programming, version 1.3. Optim. Methods Softw. 11, 545–581 (1999)MathSciNetCrossRefzbMATHGoogle Scholar
 56.Tropp, J.A.: Userfriendly tail bounds for sums of random matrices. Found. Comput. Math. 12, 389–434 (2012)MathSciNetCrossRefzbMATHGoogle Scholar
 57.Tseng, P.: Secondorder cone programming relaxation of sensor network localization. SIAM J. Optim. 18, 156–185 (2007)MathSciNetCrossRefzbMATHGoogle Scholar
 58.Vershynin, R.: Introduction to the nonasymptotic analysis of random matrices. In: Eldar, Y.C., Kutyniok, G. (eds.) Compressed Sensing: Theory and Applications. Cambridge University Press, Cambridge (2012)Google Scholar
 59.Wasserman, S., Faust, K.: Social Network Analysis: Methods and Applications. Cambridge University Press, Cambridge (1994)CrossRefzbMATHGoogle Scholar
 60.Watson, G.A.: Characterization of the subdifferential of some matrix norms. Linear Alg. Appl. 170, 33–45 (1992)MathSciNetCrossRefzbMATHGoogle Scholar
 61.Weinberger, K.Q., Saul, L.K.: Unsupervised learning of image manifolds by semidefinite programming. Int. J. Comput. Vis. 70, 77–90 (2006)CrossRefGoogle Scholar
 62.Weinberger, K.Q., Sha, F., Zhu, Q., Saul, L.K.: Graph Laplacian regularization for largescale semidefinite programming. Adv. Neural Inf. Process. Syst. 19, 1489–1496 (2007)Google Scholar
 63.Young, G., Householder, A.S.: Discussion of a set of points in terms of their mutual distances. Psychometrika 3, 19–22 (1938)CrossRefzbMATHGoogle Scholar
Copyright information
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.