Shuffled Linear Regression with Outliers in Both Covariates and Responses

Abstract

This paper studies a shuffled linear regression problem. As a variant of ordinary linear regression, it requires estimating not only the regression variable, but also permutational correspondences between the covariates and responses. While existing formulations require the underlying ground-truth correspondences to be an ideal bijection such that all pieces of data should match, such a requirement barely holds in real-world applications due to either missing data or outliers. In this work, we generalize the formulation of shuffled linear regression to a broader range of conditions where only a part of the data should correspond. To this end, the effective recovery condition and NP-hardness of the proposed formulation are also studied. Moreover, we present a simple yet effective algorithm for deriving the solution. Its global convergence property and convergence rate are also analyzed in detail. Distinct tasks validate the effectiveness of our proposed formulation and the solution method.

Appendices

Appendix A: Supplementary information for the analyses of GSLR

1.1 A.1 Derivation of Corollary 1

Given a generic linear subspace $\mathcal {V} \subset \mathbb {R}^m$ of dimension d (hereafter denoted as $\textsf{dim}\left( \mathcal {V}\right) = d$), and a finite set of endomorphisms $\mathcal {T}$ of $\mathbb {R}^m$, homomorphic sensing (Tsakiris and Peng 2019) presents the unique recovery condition in $\mathcal {V}$ under $\mathcal {T}$. I.e., $\forall v_1, v_2 \in \mathcal {V}, \tau _1, \tau _2 \in \mathcal {T}$, when can $\tau _1\left( v_1\right) =\tau _2\left( v_2\right)$ imply $v_1 = v_2$.

For self-containment, let us first remark Theorem 1 and Proposition 3 of Tsakiris and Peng (2019), which can be jointly described by the following corollary:

Corollary 2

(Unique recovery condition of homomorphic sensing) Assuming $\forall \tau \in \mathcal {T}, \textsf{rank}\left( \tau \right) \ge 2d$, then, we have unique recovery in $\mathcal {V}$ under $\mathcal {T}$ as long as

$$\begin{aligned} \textsf{dim}\left( \mathcal {V}\right) \le \mathcal {Q}_{\tau _1, \tau _2}, \ \forall \tau _1, \tau _2 \in \mathcal {T}, \end{aligned}$$

(16)

where $\mathcal {Q}_{\tau _1, \tau _2}$ is an intermediate technical term dependent to $\tau _1$ and $\tau _2$ such that, in case $\pi _1, \pi _2 \in \Pi ^\ddagger$ are full-rank square permutation matrices, we have

$$\begin{aligned} \mathcal {Q}_{\rho \pi _1, \rho \pi _2} \ge \frac{\textsf{rank}\left( \rho \right) }{2}, \ \forall \pi _1, \pi _2 \in \Pi ^\ddagger , \end{aligned}$$

(17)

where $\rho$ denotes a coordinate projection matrix.

To apply Corollary 2 to GSLR, we shall consider $\textbf{Ax}$ as a whole for some d-dimensional $\textbf{x}$ and set $\mathcal {V} = \{\textbf{Ax} : \textbf{x} \in \mathcal {X}\}$. Since linear mapping does not increase dimension, we have $\textsf{dim}\left( \mathcal {V}\right) \le d$. Futhermore, by setting $\mathcal {T} = \Pi _k$ and the $\rho$ in Eq. (17) to the projections that map from $\Pi ^\ddagger$ to $\Pi _k$, we obtain

$$\begin{aligned} \mathcal {Q}_{\tau _1, \tau _2} \ge \frac{\textsf{rank}\left( \rho \right) }{2} = \frac{k}{2}, \ \forall \tau _1, \tau _2 \in \Pi _k. \end{aligned}$$

(18)

Using Eq. (18) as a lower bound for Eq. (16), we can immediately recognize $k \ge 2d$ as a sufficient condition for unique recovery.

1.2 A.2 Proof of Proposition 1

Proof of Proposition 1

Since the bijectivity will always be satisfied as long as Problem 2 is solved, we here show that if there exists an f to solve Problem 2 that takes the graphs G and H defined in Proposition 1 as inputs, then this f is a linear mapping from a subset of $\textbf{A}$ to $\textbf{b}$. Specifically, since $\textsf{dom}\left( f\right)$ is a finitely discrete set, an f that solves Problem 2 would be a continuous mapping. Furthermore, assuming that vertices are mapped in the form of $\textbf{A}_i$ to $\textbf{b}_i$ and $\textbf{A}_j$ to $\textbf{b}_j$, we have

$$\begin{aligned} \underbrace{f\left( \textbf{A}_i + \textbf{A}_j\right) }_{\text {edge in} {\tilde{G}}} = \underbrace{\textbf{b}_i + \textbf{b}_j}_{\hbox { edge in}\ H} = \underbrace{f\left( \textbf{A}_i\right) + f\left( \textbf{A}_j\right) }_{\hbox { vertices in}\ H} \end{aligned}$$

(19)

always holds w.r.t.  the preimage of $\textbf{b}$, indicating that f is additive. By combining the continuity and the additivity we can obtain that f is a linear mapping (Mathias 2006). $\square$ $\square$

Appendix B: Supplementary information for the convergence analyses

1.1 B.1 Proofs of the global convergence

We here show that the GSLR triplet mentioned in Sect. 4.2.2 satisfies all the conditions of Theorem 1. For self- containment, we first remark the definition of closedness of set-valued mapping:

Definition 1

(Closedness of set-valued mapping (Gunawardana and Byrne 2005)) The concept of closedness of set-valued mapping $\mathcal {F}: \mathcal {U} \rightarrow \mathcal {V}$ generalizes the notion of continuity of point-to-point mappings: $\mathcal {F}$ is closed at a point $u \in \mathcal {U}$ if

• $u_k \rightarrow u$, where $u_k \in \mathcal {U}$,

• $v_k \rightarrow v$, where $v, v_k \in \mathcal {V}$,

• $v_k \in \mathcal {F}(u_k)$,

imply $v \in \mathcal {F}(u)$. Based on this, $\mathcal {F}$ is called closed if it is closed at any points of $\mathcal {U}$.

Now we are ready to introduce the proofs:

Proof of the compactness of $\mathcal {Z}$ Since $\Pi$ is discrete and finite, $\Pi$ is compact. Since $\mathcal {X}$ is compact per Hypotheses H and unions of compact sets are still compact, $\mathcal {Z}$ is compact. $\square$

Proof of the closedness of $\mathcal {H}$ For any $\left( \textbf{P}_{t}, \textbf{x}_{t}\right)$, there exist some $\delta _1, \delta _2 \in \mathbb {R}^{+}$ that satisfy

$$\begin{aligned} \begin{aligned} \forall \textbf{P}', \ \textbf{P}' \rightarrow \textbf{P} :=\{\textbf{P}' : \textbf{P}' \in \mathcal {N}_{\delta _1}(\textbf{P}_t)\} = \{\textbf{P}_t\}, \\ \forall \textbf{x}', \ \textbf{x}' \rightarrow \textbf{x} :=\left|\mathbf {x'}, \textbf{x}_t\right|_\textbf{x} < \delta _2, \end{aligned} \end{aligned}$$

(20)

where $\mathcal {N}_{\delta _1}(\textbf{P}_t)$ is the neighborhood of $\textbf{P}_t$. Therefore, we can derive the differences of mapped $\textbf{x}$ as

$$\begin{aligned} \begin{aligned}&\mathcal {H}(\mathbf {P', x'}) \setminus \mathcal {H}(\textbf{P}_t, \textbf{x}_t) = \mathcal {H}(\textbf{P}_t, \textbf{x}_t) \setminus \mathcal {H}(\mathbf {P', x'}) \\ =&\mathcal {H}(\textbf{P}_t, \textbf{x}') \setminus \mathcal {H}(\textbf{P}_t, \textbf{x}_t) \\ =&\mathop {\textrm{argmin}}\limits _{\textbf{x} \in \mathcal {X}}\Psi (\textbf{P}_t, \textbf{x}) \setminus \mathop {\textrm{argmin}}\limits _{\textbf{x} \in \mathcal {X}}\Psi (\textbf{P}_t, \textbf{x}) = \emptyset . \end{aligned} \end{aligned}$$

(21)

Since Eq. (21) implies that the cost matrix for the k-LAP generated by the regression variable $\textbf{x} \in \mathcal {H}(\textbf{P, x})$ remains unchanged when the input of $\mathcal {H}$ switches from $\left( \textbf{P}_t, \textbf{x}_t\right)$ to $\left( \mathbf {P'}, \mathbf {x'}\right)$, so will the mapped permutation $\textbf{P}$. Therefore, we have $\mathcal {H}(\textbf{P}_{t}, \textbf{x}_{t})$ closed by further joining Eqs. (20) and (21). $\square$

Proof of the continuity and monotonicity of $\Psi$ For the continuity, since compositions of continuous mappings are still continuous, we only need to prove the continuity of $f(\textbf{P}, \textbf{x}) = \textbf{PAx}$ and $g(\textbf{P}) = \textbf{PP}^T\textbf{b}$. Since the proof roadmap for f and g is similar, we only take f as an example. Specifically, given the fact that such a function is a bilinear form over the finite-dimensional space $\left( \textbf{P, x}\right)$, we only need to show that it is separately continuous w.r.t.  both $\textbf{P}$ and $\textbf{x}$. For proof, per Hypotheses H, we have $f(\textbf{x}) = \textbf{Ax}$ continuous w.r.t.  $\textbf{x} \in \mathcal {X}$. Since $\Pi$ is finitely discrete, we also have $f(\textbf{P}) = \textbf{PA}$ continuous w.r.t.  any $\textbf{P} \in \Pi$. Therefore, we can obtain that $f(\textbf{P}, \textbf{x}) = \textbf{PAx}$ is continuous and hence so is $\Psi$.

For the monotonicity, by conducting the linear regression step w.r.t.  $\textbf{x}$ we have

$$\begin{aligned} J(\textbf{P}_t, \textbf{x}_{t+1}) \le J(\textbf{P}_t, \textbf{x}_{t}), \end{aligned}$$

(22)

where the equality holds if and only if $\textbf{x}_t$ is a critical point. Furthermore, assuming there is a temporary permutation matrix $\textbf{P}_{\textrm{tmp}}$ that randomly removes $k_{t+1} - k_t$ non-zero assignments from $\textbf{P}_t$, since the cost matrix $\textbf{D}_{ij} = \left( \textbf{A}_i\textbf{x}-\textbf{b}_j\right) ^2$ is non-negative, we can obtain that

$$\begin{aligned} J(\textbf{P}_{\textrm{tmp}}, \textbf{x}_{t+1}) \le J(\textbf{P}_{t}, \textbf{x}_{t+1}). \end{aligned}$$

(23)

Since $\textbf{P}_{t+1}$ is a global optimum of the objective function w.r.t.  $\textbf{P}$, and it preserves the same number of inliers to $\textbf{P}_{\textrm{tmp}}$, we can further derive that

$$\begin{aligned} J(\textbf{P}_{t+1}, \textbf{x}_{t+1}) \le J(\textbf{P}_{\textrm{tmp}}, \textbf{x}_{t+1}), \end{aligned}$$

(24)

where the equality holds if and only if $\textbf{P}_\textrm{tmp}$ is a critical point. Summarizing Eqs. (22)-(24) leads to:

$$\begin{aligned} J(\textbf{P}_{t+1}, \textbf{x}_{t+1}) \le J(\textbf{P}_{\textrm{tmp}}, \textbf{x}_{t+1}) \le J(\textbf{P}_{t}, \textbf{x}_{t}). \end{aligned}$$

(25)

Therefore, we can conclude that Algorithm 1 is monotonically decreasing and the equality in Eq. (25) holds if and only if $\left( \textbf{P}_t, \textbf{x}_t\right)$ is already a critical point. $\square$

1.2 B.2 Proofs of Propositions 2 and 3

Throughout this appendix, apart from the notations described in Table 1, we also use C to denote a constant whose value may vary from line to line.

To prove Proposition 2, due to the non-smoothness of our Objective (6), we following the concept of the KL-inequality generalized to sub-analytic set (Bolte et al. 2007). Let us first remark the definitions for self-containment:

Definition 2

(Kurdyka-Łojasiewicz inequality) Let f be a continuous sub-analytic function (i.e., a function whose graph is a sub-analytic set) with closed domain that maps to $\mathbb {R}$, and ${\bar{x}} \in \textsf{dom}\left( f\right)$ be a critical point, there exists a concave function $\phi \left( s\right) = Cs^{1 - \theta }$ with $\theta \in \left[ 0, 1\right)$ and $C > 0$ such that the following inequality holds for any x within the neighborhood of ${\bar{x}}$ under the conventions $0^0=1$ and $0/0=\infty /\infty =0$:

$$\begin{aligned} \phi '\left( f(x) - f({\bar{x}})\right) \textrm{dis}\left( 0, \partial f(x) \right) \ge 1, \end{aligned}$$

(26)

where $\textrm{dis}\left( 0, \partial f(x)\right) = \inf \{ \left\| x^\star \right\| : x^\star \in \partial f(x) \}$ denotes the point-to-set distance.

Definition 3

(Semi-X & sub-analytic sets) A subset $\mathcal {S}$ of $\mathbb {R}^{d}$ is called semi-algebraic if every $\textbf{x} \in \mathbb {R}^{d}$ admits a neighborhood $\mathcal {N}$ such that

$$\begin{aligned} \mathcal {S} \cap \mathcal {N} = \bigcap _i^{m} \bigcup _j^{n} \left\{ \textbf{x} \in \mathcal {N} : f_{ij}(\textbf{x}) = 0, g_{ij}(\textbf{x}) \ge 0 \right\} , \end{aligned}$$

where $f_{ij}$ and $g_{ij}$ are polynomials and $m, n < +\infty$.

A semi-analytic set generalizes the above definition by relaxing $f_{ij}$ and $g_{ij}$ to real analytic functions.

A subset $\mathcal {S}$ of $\mathbb {R}^{d}$ is called sub-analytic if every $\textbf{x} \in \mathbb {R}^{d}$ admits a neighborhood $\mathcal {N}$, and there exists a bounded semi-analytic set $\mathcal {Y} \subset \mathbb {R}^{d+o}$ such that $\mathcal {S} \cap \mathcal {N}$ is the projection from $\mathcal {Y}$ to $\mathcal {N}$:

$$\begin{aligned} \mathcal {S} \cap \mathcal {N} = \pi (\mathcal {Y}), \end{aligned}$$

where $\pi : \mathbb {R}^{d+o} \rightarrow \mathbb {R}^{d}$ is the projection. In summary, sub-analytic set broadens the semi-analytic one, and the semi-analytic set broadens the semi-algebraic one.

We also introduce the following lemma:

Lemma 1

The set of generalized permutation matrices $\Pi$ can be characterized by the following polynomials:

$$\begin{aligned} \begin{aligned} \textbf{P1} \le \textbf{1} \ \textrm{and} \ \textbf{P}^T\textbf{1} \le \textbf{1}, \\ \textbf{e}_i^T \textbf{D} \textbf{e}_i \left( \textbf{e}_i^T \textbf{D} \textbf{e}_i - 1\right) = 0, \forall i \in \left\lceil n \right\rfloor , \\ \textbf{e}_i^T \textbf{D} \textbf{e}_j = 0, \forall i, j \in \left\lceil n \right\rfloor , i \ne j, \\ \end{aligned} \end{aligned}$$

where $\textbf{e}_i$ is the one-hot vector with 1 at position i, $\textbf{D} = \{ \textbf{PP}^T$, $\textbf{P}^T\textbf{P}\}$, and $\left\lceil n \right\rfloor$ is the set of positive integers that are no greater than the size of $\textbf{D}$.

Proof of Lemma 1 This lemma is an extension of the properties of the full-rank permutation matrices, which states that a matrix is permutational if and only if it is orthogonal and doubly stochastic (Fogel et al. 2013). The latter two equations imply that the diagonal entries of $\textbf{D}$ should be within $\{0, 1\}$ and the off-diagonal ones 0. While the sufficiency is guaranteed by the definition of generalized permutation matrices, we here demonstrate that these polynomials are necessary for a matrix to be permutational by forcing $\textbf{P}$ to be binary. Specifically, assuming there is a matrix $\textbf{P} \in \Pi$ with entries $\textbf{P}_{ip}, \ldots , \textbf{P}_{iq} \in \left( 0, 1\right)$ with $1 \le p < q \le n$ in the $i^{\textrm{th}}$ row, then the $pq^{\textrm{th}}$ entry of $\textbf{D} = \textbf{P}^T\textbf{P}$ can be denoted as

$$\begin{aligned} \begin{aligned} \textbf{D}_{pq} = \sum _k \textbf{P}^T_{pk} \textbf{P}_{kq} = \sum _k \textbf{P}_{kp} \textbf{P}_{kq} \ne 0, \end{aligned} \end{aligned}$$

(27)

which contradicts the requirement that the off-diagonal entries of $\textbf{D}$ should be 0. $\textbf{D} = \textbf{PP}^T$ follows the same proof. $\square$

Now we are ready to prove Proposition 2:

Proof of Proposition 2 The first property is a direct result of the fact that $\triangledown _\textbf{x} J = 2\left( \textbf{A}^T\textbf{P}^T\textbf{PA}\textbf{x} - \textbf{PP}^T\textbf{b}\right)$ is a continuous linear mapping.

For the second property, from Lemma 1 we know that $\Pi$ is semi-algebraic (hence sub-analytic). Moreover, given the facts that $\mathcal {X}$ is sub-analytic per the assumption, Objective (6) is polynomial, and unions and intersections preserve sub-analyticity per Definition 3; we have the graph of Objective (6) sub-analytic. Joining the sub-analyticity together with the continuity mentioned in Sect. 4.2.2 implies the KL-inequality. $\square$

To prove Proposition 3, we introduce the following two lemmas:

Lemma 2

Suppose that Hypotheses H are satisfied, then for any $t^{\textrm{th}}$ step of Algorithm 1, the following equation holds true:

$$\begin{aligned} \left( \triangledown _{\textbf{x}} J(\textbf{P}_{t}, \textbf{x}_t) - \triangledown _{\textbf{x}} J(\textbf{P}_{t-1}, \textbf{x}_t), \textbf{0}\right) \in \partial J(\textbf{P}_t, \textbf{x}_t ). \end{aligned}$$

Proof of Lemma 2 Per the alternative update rule of Algorithm 1 we have

$$\begin{aligned}{} & {} \textbf{0} = \triangledown _\textbf{x} J\left( \textbf{P}_{t-1}, \textbf{x}_t\right) , \end{aligned}$$

(28)

$$\begin{aligned}{} & {} \textbf{0} \in \partial _{\textbf{P}} J\left( \textbf{P}_{t}, \textbf{x}_t\right) . \end{aligned}$$

(29)

From Eq. (28) we can obtain

$$\begin{aligned} \begin{aligned} \triangledown _\textbf{x} J\left( \textbf{P}_{t}, \textbf{x}_t\right) - \triangledown _\textbf{x} J\left( \textbf{P}_{t-1}, \textbf{x}_t\right) \in \partial _{\textbf{x}} J\left( \textbf{P}_{t}, \textbf{x}_t\right) , \end{aligned} \end{aligned}$$

(30)

which together with Eq. (29) concludes the proof. $\square$

Lemma 3

Suppose that Hypotheses H are satisfied, then for any $t^{\textrm{th}}$ step of Algorithm 1, there exists $C \in \mathbb {R}^{+}$ such that the following equation holds true:

$$\begin{aligned} J_{t} - J_{t+1} \ge C \left\| \textbf{z}_{t+1} - \textbf{z}_t\right\| ^2. \end{aligned}$$

Proof of Lemma 3 We base the proof on the quotient law of convergent sequences. In detail, let us define the sequences:

• 1. Sequence of difference between energy values: $P = \left\{ J_{t} - J_{t+1} : t \in \mathbb {N} \right\}$.

• 2. Sequence of squared difference between solutions: $Q = \left\{ \left\| \textbf{z}_t - \textbf{z}_{t+1}\right\| ^2 : t \in \mathbb {N} \right\}$.

• 3. Sequence of quotient: $S = \left\{ \frac{P_t}{Q_t} : t \in \mathbb {N} \right\}$.

Per the compactness of $\textbf{z}$ and continuity of J, we know that both $\textbf{z}$ and J are bounded. Moreover, per the global convergences mentioned in Sect. 4.2.2, for any $t \in \mathbb {N}$ we have $P_t, Q_t \ge 0$ and $P_t = 0 \Leftrightarrow Q_t = 0$. Therefore, with the convention $\frac{0}{0} = 1$, for any $s \in S$ we have $s \in \mathbb {R}_{+}$, which implies that there always exist some $C \in \mathbb {R}_{+}$ such that $\inf S \ge C$ holds. $\square$

With Lemmas 2 and 3 in hand, we now can prove Proposition 3:

Proof of Proposition 3 According to Lemma 2 and the Lipschitz continuity of $\triangledown _{\textbf{x}} J$, we have

$$\begin{aligned} \begin{aligned} \textrm{dis}\left( \textbf{0}, \partial J(\textbf{z}_t)\right)&\le \left\| \triangledown _{\textbf{x}} J(\textbf{P}_{t-1}, \textbf{x}_t) - \triangledown _{\textbf{x}} J(\textbf{P}_t, \textbf{x}_t)\right\| \\&\le L \left\| \textbf{P}_{t-1} - \textbf{P}_t\right\| \le L \left\| \textbf{z}_{t-1} - \textbf{z}_t\right\| , \end{aligned} \end{aligned}$$

(31)

where $L \in \mathbb {R}^{+}$ is the Lipschitz constant.

Assuming the energy value ${\bar{J}}$ at the critical point satisfies ${\bar{J}} = 0$ without loss of generality, combining Eq. (31) and Eq. (26) leads to

$$\begin{aligned} \phi '(J_t) = \phi '( J_t - {\bar{J}}) \ge \frac{1}{L} \left\| \textbf{z}_{t-1} - \textbf{z}_t\right\| ^{-1}. \end{aligned}$$

(32)

On the other hand, by using the concavity of $\phi$ and Lemma 3, we can obtain

$$\begin{aligned} \begin{aligned} \phi (J_t) - \phi (J_{t+1})&\ge \phi '(J_t)\left( J_t - J_{t+1}\right) \\&\ge C \phi '(J_t) \left\| \textbf{z}_{t+1} - \textbf{z}_t\right\| ^2. \end{aligned} \end{aligned}$$

(33)

Taking Eq. (32) into Eq. (33) yields

$$\begin{aligned} \begin{aligned} \left\| \textbf{z}_{t-1} - \textbf{z}_t\right\| \cdot C \left( \phi (J_t) - \phi (J_{t+1}) \right) \ge \left\| \textbf{z}_{t+1} - \textbf{z}_t\right\| ^2. \end{aligned} \end{aligned}$$

(34)

Taking square root on both sides of Eq. (34) and applying the inequality $a^2 + b^2 \ge 2ab$ result in

$$\begin{aligned} \left\| \textbf{z}_{t-1} - \textbf{z}_t\right\| + C \left( \phi (J_t) - \phi (J_{t+1}) \right) \ge 2 \left\| \textbf{z}_{t+1} - \textbf{z}_t\right\| . \end{aligned}$$

(35)

Summing Eq. (35) on both sides for $t \in \left[ p, +\infty \right)$ leads to

$$\begin{aligned} \begin{aligned} \sum _{t=p}^{+\infty } \left\| \textbf{z}_{t+1} - \textbf{z}_{t}\right\| \le \left\| \textbf{z}_p - \textbf{z}_{p-1}\right\| + C \phi (J_p), \end{aligned} \end{aligned}$$

(36)

Since $\mathcal {Z}$ is compact as mentioned in Sect. 4.2.2, there exists a B such that $\sup \{\left\| \textbf{z}_i - \textbf{z}_j\right\| : \textbf{z}_i, \textbf{z}_j \in \mathcal {Z} \} = B < +\infty$. Hence Eq. (36) yields

$$\begin{aligned} \sum _{t=p}^{+\infty } \left\| \textbf{z}_{t+1} - \textbf{z}_{t}\right\| \le C \phi (J_p) + B < +\infty , \end{aligned}$$

(37)

which concludes the proof. $\square$