Primal–Dual Proximal Splitting and Generalized Conjugation in Non-smooth Non-convex Optimization

Abstract

We demonstrate that difficult non-convex non-smooth optimization problems, such as Nash equilibrium problems and anisotropic as well as isotropic Potts segmentation models, can be written in terms of generalized conjugates of convex functionals. These, in turn, can be formulated as saddle-point problems involving convex non-smooth functionals and a general smooth but non-bilinear coupling term. We then show through detailed convergence analysis that a conceptually straightforward extension of the primal–dual proximal splitting method of Chambolle and Pock is applicable to the solution of such problems. Under sufficient local strong convexity assumptions on the functionals—but still with a non-bilinear coupling term—we even demonstrate local linear convergence of the method. We illustrate these theoretical results numerically on the aforementioned example problems.

Appendices

A data Statement for the EPSRC

The source codes for the numerical experiments are on Zenodo at [12].

Reductions of the Three-Point Condition

The following two propositions demonstrate that Assumption 3.2 (iv) is closely related to standard second-order optimality conditions, i.e., that the Hessian is positive definite at the solution ${\widehat{u}}$.

Proposition A.1

Suppose Assumption 3.2 (ii) (locally Lipschitz gradients of K) holds in some neighborhood ${\mathcal {U}}$ of ${\widehat{u}}$, and for some $\xi _x\in {\mathbb {R}}$, $\gamma _x>0$,

$$\begin{aligned} \xi _x\Vert x-{\widehat{x}}\Vert ^2 +\langle K_x(x,{\widehat{y}})-K_x({\widehat{x}},{\widehat{y}}),x-{\widehat{x}}\rangle \ge \gamma _x\Vert x-{\widehat{x}}\Vert ^2 \quad ((x,y)\in {\mathcal {U}}). \end{aligned}$$

(66)

Then (15a) holds in ${\mathcal {U}}$ with $\theta _x=2(\gamma _x-\alpha )L_{yx}^{-1}$, and $\lambda _x=L_x({\widehat{y}})^2(2\alpha )^{-1}$ for any $\alpha \in (0,\gamma _x]$.

Proof

An application of Cauchy’s and Young’s inequalities with any factor $\alpha >0$, Assumption 3.2 (ii), and (66) yields the estimate

$$\begin{aligned} \begin{aligned} \langle K_x(x',{\widehat{y}})-K_x({\widehat{x}},{\widehat{y}}),x-{\widehat{x}}\rangle +\xi _x\Vert x-{\widehat{x}}\Vert ^2&= \langle K_x(x,{\widehat{y}})-K_x({\widehat{x}},{\widehat{y}}),x-{\widehat{x}}\rangle +\xi _x\Vert x-{\widehat{x}}\Vert ^2 \\&\quad +\langle K_x(x',{\widehat{y}})-K_x(x,{\widehat{y}}),x-{\widehat{x}}\rangle \\&\ge (\gamma _x-\alpha )\Vert x-{\widehat{x}}\Vert ^2-L_x({\widehat{y}})^2(4\alpha )^{-1} \Vert x'-x\Vert ^2. \end{aligned} \end{aligned}$$

At the same time, using (16),

$$\begin{aligned} \Vert K_y({\widehat{x}},y)-K_y(x,y)-K_{yx}(x,y)({\widehat{x}}-x)\Vert \le \frac{L_{yx}}{2}\Vert x-{\widehat{x}}\Vert ^2. \end{aligned}$$

Therefore (15a) holds if we take $\theta _x\le 2(\gamma _x-\alpha )L_{yx}^{-1}$ and $\lambda _x=L_x({\widehat{y}})^2(2\alpha )^{-1}$. $\square$

Proposition A.2

Suppose Assumption 3.2 (ii) (locally Lipschitz gradients of K) holds in some neighborhood ${\mathcal {U}}$ of ${\widehat{u}}$ with $L_y(x)\le {\bar{L}}_y$, and that

$$\begin{aligned} \Vert K_{xy}(x,y')-K_{xy}(x,y)\Vert \le L_{xy}\Vert y'-y\Vert \quad (u,u'\in {\mathcal {U}}) \end{aligned}$$

for some constant $L_{xy} \ge 0$. Assume, moreover, for some $\xi _y\in {\mathbb {R}}$, $\gamma _y>0$ that

$$\begin{aligned} \xi _y\Vert y-{\widehat{y}}\Vert ^2 +\langle K_y({\widehat{x}},{\widehat{y}})-K_y({\widehat{x}},y),y-{\widehat{y}}\rangle \ge \gamma _y\Vert y-{\widehat{y}}\Vert ^2 \quad ((x,y)\in {\mathcal {U}}). \end{aligned}$$

(67)

Then (15b) holds in ${\mathcal {U}}$ with $\theta _y=2(\gamma _y-\alpha _1)(1+\alpha _2)^{-1} L_{xy}^{-1}$, and $\lambda _y=({\bar{L}}_y^2(2\alpha _1)^{-1}+(1+\alpha _2^{-1})L_{xy}\theta _y)$ for any $\alpha _1\in (0,\gamma _y]$, $\alpha _2>0$.

Proof

An application of Cauchy’s and Young’s inequalities with any factor $\alpha >0$, Assumption 3.2 (ii), and (67) yields the estimate

$$\begin{aligned} \begin{aligned} \langle K_y(x,y)-K_y(x,y')&+K_y({\widehat{x}},{\widehat{y}})-K_y({\widehat{x}},y),y-{\widehat{y}}\rangle +\xi _y\Vert y-{\widehat{y}}\Vert ^2 \\&\ge \langle K_y(x,y)-K_y(x,y'),y-{\widehat{y}}\rangle +\gamma _y\Vert y-{\widehat{y}}\Vert ^2 \\&\ge (\gamma _y-\alpha _1)\Vert y-{\widehat{y}}\Vert ^2-\frac{L_y(x)^2}{4\alpha _1} \Vert y'-y\Vert ^2. \end{aligned}\end{aligned}$$

At the same time, using (16) and Young’s inequality for any $\alpha _2>0$,

$$\begin{aligned}&\Vert K_x(x',{\widehat{y}})-K_x(x',y')-K_{xy}(x',y')({\widehat{y}}-y')\Vert \le \frac{L_{xy}}{2}\Vert y'-{\widehat{y}}\Vert ^2 \\&\quad \le \frac{L_{xy}}{2}(1+\alpha _2)\Vert y-{\widehat{y}}\Vert ^2 +\frac{L_{xy}}{2}(1+\alpha _2^{-1})\Vert y'-y\Vert ^2. \end{aligned}$$

Therefore (15b) holds if we take $\theta _y\le 2\frac{\gamma _y-\alpha _1}{(1+\alpha _2)L_{xy}}$ and $\lambda _y=\frac{{\bar{L}}_y^2}{2\alpha _1}+(1+\alpha _2^{-1})L_{xy}\theta _y$. $\square$

Relaxations of the Three-Point Condition

In all the results of this paper, Assumption 3.2(iv) can be generalized to the following three-point condition similar to the one used in [11].

Assumption B.1

The functional $K(x,y)\in C^1(X\times Y)$ and there exists a neighborhood

$$\begin{aligned} {{\mathcal {U}}(\rho _x,\rho _y) :=(\mathbb {B}({\widehat{x}}, \rho _x) \cap {\mathcal {X}}_G) \times (\mathbb {B}({\widehat{y}}, \rho _y) \cap {\mathcal {Y}}_{F^*}),} \end{aligned}$$

(68)

for some $\rho _x,\rho _y>0$ such that for all $u',u \in {\mathcal {U}}(\rho _x,\rho _y)$, the following property holds:

• (iv*) (three-point condition) There exist $\theta _x,\theta _y > 0$, $\lambda _x,\lambda _y\ge 0$, $\xi _x,\xi _y\in {\mathbb {R}}$, and $p_x,p_y\in [1,2]$ such that

  $$\begin{aligned}&\langle K_x(x',{\widehat{y}})-K_x({\widehat{x}},{\widehat{y}}),x-{\widehat{x}}\rangle +\xi _x\Vert x-{\widehat{x}}\Vert ^2\nonumber \\&\quad \ge \theta _x\Vert K_y({\widehat{x}},y)-K_y(x,y)-K_{yx}(x,y)({\widehat{x}}-x)\Vert ^{p_x} -\frac{\lambda _x}{2}\Vert x-x'\Vert ^2, \end{aligned}$$

  (69a)
  $$\begin{aligned}&\langle K_y(x,y)-K_y(x,y')+K_y({\widehat{x}},{\widehat{y}})-K_y({\widehat{x}},y),y-{\widehat{y}}\rangle +\xi _y\Vert y-{\widehat{y}}\Vert ^2 \nonumber \\&\quad \ge \theta _y\Vert K_x(x',{\widehat{y}})-K_x(x',y')-K_{xy}(x',y')({\widehat{y}}-y')\Vert ^{p_y} -\frac{\lambda _y}{2}\Vert y-y'\Vert ^2. \end{aligned}$$

  (69b)

This assumption introduces $p_x$ and $p_y$ in [1, 2], while in Assumption 3.2 (iv) we had $p_x=p_y=1$. For instance, in [11, Appendix B] we verified Assumption B.1 with $p_x=2$ for the case $K(x,y)=\langle A(x),y\rangle$ for the reconstruction of the phase and amplitude of a complex number. This relaxation mainly affects the proof of Step 4 in Theorem 4.2, which now requires a few intermediate derivations.

Corollary B.1

The results of Theorem 4.2 continue to hold if Assumption 3.2 (iv) is replaced with Assumption B.1 (iv*) for some $p_x,p_y\in [1,2]$, where in case $p_y \in (1, 2]$, (24d) is replaced by

$$\begin{aligned} \gamma _G&\ge {{\tilde{\gamma }}}_G + \xi _x + \frac{p_y-1}{(\theta _yp_y^{p_y}\rho _x^{p_y-2}{\overline{\omega }}^{-1})^{\frac{1}{p_y-1}}}, \end{aligned}$$

(70a)

and in case $p_x \in (1, 2]$, (24e) is replaced by

$$\begin{aligned} \gamma _{F^*}&\ge {{\tilde{\gamma }}}_{F^*}+ \xi _y + \frac{p_x-1}{({\underline{\omega }}\theta _xp_x^{p_x} \rho _y^{p_x-2})^{\frac{1}{p_x-1}}}. \end{aligned}$$

(70b)

Proof

The beginning of the proof follows the exact same steps as in the proof of Theorem 4.2 up until (30). We now use Assumption B.1 (iv*) to further bound $D_x$ and $D_y$ similarly to (31) and (32). From (69a),

$$\begin{aligned} \begin{aligned} D_x&\ge \theta _x\Vert K_y({\widehat{x}},y^{i+1})-K_y(x^{i+1},y^{i+1})-K_{yx}(x^{i+1},y^{i+1})({\widehat{x}}-x^{i+1})\Vert ^{p_x} -\frac{\lambda _x}{2}\Vert x^{i+1}-x^i\Vert ^2 \\&-\Vert y^{i+1}-{\widehat{y}}\Vert \Vert K_y({\widehat{x}},y^{i+1})-K_y(x^{i+1},y^{i+1})-K_{yx}(x^{i+1},y^{i+1})({\widehat{x}}-x^{i+1})\Vert \omega _i^{-1}. \end{aligned} \end{aligned}$$

(71)

The following generalized Young’s inequality for any positive a, b, p and q such that $q^{-1}+p^{-1}=1$ allows for our choice of varying $p_x\in [1,2]$:

$$\begin{aligned} ab=\left( ab^{\frac{2-p}{p}}\right) b^{2\frac{p-1}{p}} \le \frac{1}{p}\left( ab^{\frac{2-p}{p}}\right) ^p+\frac{1}{q}b^{2\frac{p-1}{p}q} =\frac{1}{p}a^pb^{2-p}+\biggl (1-\frac{1}{p}\biggr )b^2. \end{aligned}$$

(72)

Applying this inequality with $p=p_x$,

$$\begin{aligned} a&:=(\zeta _x p_x)^{-1/2} \Vert K_y({\widehat{x}},y^{i+1})-K_y(x^{i+1},y^{i+1})-K_{yx}(x^{i+1},y^{i+1})({\widehat{x}}-x^{i+1})\Vert , \\ b&:=(\zeta _x p_x)^{1/2}\Vert y^{i+1}-{\widehat{y}}\Vert , \end{aligned}$$

for any $\zeta _x>0$ to the last term of (71), we arrive at the estimate

$$\begin{aligned} \begin{aligned} D_x&\ge \theta _x\Vert K_y({\widehat{x}},y^{i+1})-K_y(x^{i+1},y^{i+1})-K_{yx}(x^{i+1},y^{i+1})({\widehat{x}}-x^{i+1})\Vert ^{p_x} -\frac{\lambda _x}{2}\Vert x^{i+1}-x^i\Vert ^2 \\&\quad -\frac{\Vert y^{i+1}-{\widehat{y}}\Vert ^{2-p_x}}{p_x^{p_x}\omega _i\zeta _x^{p_x-1}} \Vert K_y({\widehat{x}},y^{i+1})-K_y(x^{i+1},y^{i+1})-K_{yx}(x^{i+1},y^{i+1})({\widehat{x}}-x^{i+1})\Vert ^{p_x} \\&\quad -\frac{p_x-1}{\omega _i}\zeta _x \Vert y^{i+1}-{\widehat{y}}\Vert ^2. \end{aligned} \end{aligned}$$

We now use $u^{i+1}\in {\mathcal {U}}(\rho _x,\rho _y)$ for some $\rho _x,\rho _y\ge 0$, and $\omega _i^{-1} \le {\underline{\omega }}^{-1}$ to obtain

$$\begin{aligned} \theta _x-\Vert y^{i+1}-{\widehat{y}}\Vert ^{2-p_x}(p_x^{p_x}\omega _i\zeta _x^{p_x-1})^{-1} \ge \theta _x-\rho _y^{2-p_x}(p_x^{p_x}{\underline{\omega }}\zeta _x^{p_x-1})^{-1}. \end{aligned}$$

(73)

If $p_x=1$, we use the assumed inequality $\theta _x \ge \rho _y{\underline{\omega }}^{-1}$ from (24e) to show that the right-hand side of (73) is non-negative for any $\zeta _x>0$. Otherwise we take $\zeta _x :=({\underline{\omega }}\theta _xp_x^{p_x}\rho _y^{p_x-2})^{1/(1-p_x)}$ to ensure the right-hand side of (73) is zero. In either case, $\theta _x-\rho _y^{2-p_x}(p_x^{p_x}{\underline{\omega }}\zeta _x^{p_x-1})^{-1}\ge 0$ and hence

$$\begin{aligned} D_x \ge -\frac{\lambda _x}{2}\Vert x^{i+1}-x^i\Vert ^2 -(p_x-1)\omega ^{-1}_i\zeta _x \Vert y^{i+1}-{\widehat{y}}\Vert ^2. \end{aligned}$$

(74)

Analogously, from (69b) and Cauchy’s inequality,

$$\begin{aligned} \begin{aligned} D_y&\ge \theta _y\Vert K_x(x^i,{\widehat{y}})-K_x(x^i,y^i)-K_{xy}(x^i,y^i)({\widehat{y}}-y^i)\Vert ^{p_y} -\frac{\lambda _y}{2}\Vert y^{i+1}-y^i\Vert ^2 \\&\quad -\omega _i\Vert x^{i+1}-{\widehat{x}}\Vert \Vert K_x(x^i,{\widehat{y}})-K_x(x^i,y^i)-K_{xy}(x^i,y^i)({\widehat{y}}-y^i)\Vert . \end{aligned} \end{aligned}$$

This has a structure similar to (71) with $\omega _i$ now as a multiplier. Hence, we apply a similar generalized Young’s inequality to the last term with any $\zeta _y>0$. Noting that $\omega _i\le {\overline{\omega }}$, we use the following bound similar to (73):

$$\begin{aligned} \theta _y-\Vert x^{i+1}-{\widehat{x}}\Vert ^{2-p_y}\omega _i(p_y^{p_y}\zeta _y^{p_y-1})^{-1} \ge \theta _y-\rho _x^{2-p_y}{\overline{\omega }}(p_y^{p_y}\zeta _y^{p_y-1})^{-1} \ge 0. \end{aligned}$$

The last inequality holds for any $\zeta _y>0$ if $p_y=1$ due to the assumed $\theta _y \ge {\overline{\omega }}\rho _x$ from (24d); otherwise, we set $\zeta _y :=(\theta _yp_y^{p_y}\rho _x^{p_y-2}{\overline{\omega }}^{-1})^{1/(1-p_y)}$. We then obtain that

$$\begin{aligned} D_y \ge -\frac{\lambda _y}{2}\Vert y^{i+1}-y^i\Vert ^2 -(p_y-1)\omega _i\zeta _y \Vert x^{i+1}-{\widehat{x}}\Vert ^2. \end{aligned}$$

(75)

Combining (30), (74), and (75), we can thus bound

$$\begin{aligned} \begin{aligned} D&= \eta _i D_x+\eta _{i+1}D_y+\eta _{i+1}D_\omega +\eta _i(\gamma _G-{{\tilde{\gamma }}}_G-\xi _x)\Vert x^{i+1}-{\widehat{x}}\Vert ^2 \\&\quad +\eta _{i+1}(\gamma _{F^*}-{{\tilde{\gamma }}}_{F^*}-\xi _y)\Vert y^{i+1}-{\widehat{y}}\Vert ^2 \\&\ge \eta _{i+1}(\gamma _{F^*}-{{\tilde{\gamma }}}_{F^*}-\xi _y-(p_x-1)\zeta _x) \Vert y^{i+1}-{\widehat{y}}\Vert ^2 -\eta _i\frac{\lambda _x}{2}\Vert x^{i+1}-x^i\Vert ^2 \\&\quad +\eta _{i}(\gamma _G-{{\tilde{\gamma }}}_G-\xi _x-(p_y-1)\zeta _y)\Vert x^{i+1}-{\widehat{x}}\Vert ^2 -\eta _{i+1}\frac{\lambda _y}{2}\Vert y^{i+1}-y^i\Vert ^2 \\&\quad -\eta _i \frac{L_{yx}}{2}(\omega _i+2)\rho _y\Vert x^{i+1}-x^i\Vert ^2 \\&\ge -\eta _i\frac{\lambda _x+L_{yx}(\omega _i+2)\rho _y}{2}\Vert x^{i+1}-x^i\Vert ^2 -\eta _{i+1}\frac{\lambda _y}{2}\Vert y^{i+1}-y^i\Vert ^2, \end{aligned} \end{aligned}$$

(76)

where in the final step, we have also used (70) and the selected $\zeta _x$ and $\zeta _y$ if $p_x>1$ or $p_y>1$ or both. Thus, we obtained exactly the same lower bound as in (33). We then continue along the rest of the proof of Theorem 4.2 to obtain the claim. $\square$

It is worth observing that when $p_x \in (1, 2]$ or $p_y\in (1,2]$, the inequalities (70) do not directly bound the respective $\rho _y$ or $\rho _x$. Hence, we do not need to initalize the corresponding variable locally, unlike when $p_x=1$ or $p_y=1$. On the other hand, sufficient strong convexity is required from the corresponding G and $F^*$.

We start with the lemma ensuring that the iterates stay in the initial neighborhood of the saddle point.

Corollary B.2

The results of Lemma 5.2 continue to hold if the corresponding conditions of Theorem 4.2 are replaced with those in Corollary B.1.

Proof

The proof repeats that of Lemma 5.2, applying Corollary B.1 instead of Theorem 4.2 in Step 2. $\square$

We next extend the results of Section 6 to arbitrary choices of both $p_x \in [1,2]$ and $p_y \in [1,2]$. This mainly consists of verifying (70a) when $p_y \ne 1$ and (70b) when $p_x \ne 1$. Note that it is possible to take $p_x=1$ and $p_y \ne 1$, or vice versa, as long as the corresponding conditions are satisfied.

Corollary B.3

The results of Theorem 6.1 continue to hold if Assumption 3.2 (iv) is replaced with Assumption B.1 (iv*) for some $p_x,p_y\in [1,2]$, where in case $p_y \in (1, 2]$, (46a) is replaced with

$$\begin{aligned} \xi _x&= \gamma _G - \frac{p_y-1}{(\theta _yp_y^{p_y}(2\rho _x)^{p_y-2})^{\frac{1}{p_y-1}}}, \end{aligned}$$

(77a)

and in case $p_x \in (1, 2]$, (46b) is replaced with

$$\begin{aligned} \xi _y&= \gamma _{F^*} - \frac{p_x-1}{(\theta _xp_x^{p_x}(2\rho _y)^{p_x-2})^{\frac{1}{p_x-1}}}. \end{aligned}$$

(77b)

Proof

Since conditions (77) are sufficient for (70) with ${\overline{\omega }}={\underline{\omega }}=1$ to hold, we can repeat the proof of Theorem 6.1 replacing the references to Theorem 4.2 by references to Corollary B.1 up until (50). If $p_x>1$, we now obtain a lower bound on $d_i^x$ by arguing as in (71)–(73) with ${\widehat{u}}$ replaced by ${\bar{u}}$. Specifically, using (16), Assumption B.1 (iv*) at ${\bar{u}}$, and the generalized Young’s inequality (72), we obtain for any $\zeta _x>0$ that

$$\begin{aligned} \begin{aligned} d_i^x&\le -\theta _x\Vert K_y({\bar{x}},y^{i+1})-K_y(x^{i+1},y^{i+1})-K_{yx}(x^{i+1},y^{i+1})({\bar{x}}-x^{i+1})\Vert ^{p_x} \\&\quad +\Vert y^{i+1}-{\bar{y}}\Vert \Vert K_y({\bar{x}},y^{i+1})-K_y(x^{i+1},y^{i+1})-K_{yx}(x^{i+1},y^{i+1})({\bar{x}}-x^{i+1})\Vert \\&\quad +\frac{\lambda _x}{2}\Vert x^{i+1}-x^i\Vert ^2 -\frac{p_y-1}{(\theta _yp_y^{p_y}(2\rho _x)^{p_y-2})^{\frac{1}{p_y-1}}}\Vert x^{i+1}-{\bar{x}}\Vert ^2 \\&\le \left( \frac{\Vert y^{i+1}-{\bar{y}}\Vert ^{2-p_x}}{p_x^{p_x}\zeta _x^{p_x-1}}-\theta _x\right) \Vert K_y({\bar{x}},y^{i+1})-K_y(x^{i+1},y^{i+1})-K_{yx}(x^{i+1},y^{i+1})({\bar{x}}-x^{i+1})\Vert ^{p_x} \\&\quad +(p_x-1)\zeta _x \Vert y^{i+1}-{\bar{y}}\Vert ^2 +\frac{\lambda _x}{2}\Vert x^{i+1}-x^i\Vert ^2 -\frac{p_y-1}{(\theta _yp_y^{p_y}(2\rho _x)^{p_y-2})^{\frac{1}{p_y-1}}}\Vert x^{i+1}-{\bar{x}}\Vert ^2. \end{aligned} \end{aligned}$$

Inserting $\zeta _x=(\theta _xp_x^{p_x}(2\rho _y)^{p_x-2})^{1/(1-p_x)}$ and $\Vert y^{i+1}-{\bar{y}}\Vert \le 2\rho _y$, we eliminate the first term on the right-hand side. Likewise, if $p_y>1$, similar steps applied to $d_i^y$ result in

$$\begin{aligned} d_i^y\le (p_y-1)\zeta _y \Vert x^{i+1}-{\bar{x}}\Vert ^2 +\frac{\lambda _y}{2}\Vert y^{i+1}-y^i\Vert ^2 -\frac{p_x-1}{(\theta _xp_x^{p_x}(2\rho _y)^{p_x-2})^{\frac{1}{p_x-1}}}\Vert y^{i+1}-{\bar{y}}\Vert ^2 \end{aligned}$$

for $\zeta _y=(\theta _yp_y^{p_y}(2\rho _x)^{p_y-2})^{1/(p_y-1)}$. Using $\Vert u^{i+1}-u^i\Vert \rightarrow 0$ and the selection of $\zeta _x$ and $\zeta _y$, we then obtain the desired estimate $\limsup _{i\rightarrow \infty }~q_i:=\limsup _{i\rightarrow \infty }~(d_i^x + d_i^y + O(\Vert u^{i+1}-u^i\Vert ))\le 0$. $\square$

Corollary B.4

The results of Theorem 6.3 continue to hold if Assumption 3.2 (iv) is replaced with Assumption B.1 (iv*) for some $p_x,p_y\in [1,2]$, where in case $p_y \in (1, 2]$, (51a) is replaced for some ${{\tilde{\gamma }}}_G > 0$ with

$$\begin{aligned} \xi _x&= \gamma _G - {{\tilde{\gamma }}}_G - \frac{p_y-1}{(\theta _yp_y^{p_y}(\rho _x)^{p_y-2})^{\frac{1}{p_y-1}}}, \end{aligned}$$

(78a)

and in case $p_x \in (1, 2]$, (51b) is replaced with

$$\begin{aligned} \xi _y&= \gamma _{F^*} - \frac{p_x-1}{(\theta _xp_x^{p_x}(\rho _y)^{p_x-2})^{\frac{1}{p_x-1}}}. \end{aligned}$$

(78b)

Proof

Conditions (78) are sufficient for (70) with ${\overline{\omega }}={\underline{\omega }}=1$ to hold; therefore, we can repeat the proof of Theorem 6.3 replacing the references to Theorem 4.2 by references to Corollary B.1. $\square$

Corollary B.5

The results of Theorem 6.4 continue to hold if Assumption 3.2 (iv) is replaced with Assumption B.1 (iv*) for some $p_x,p_y\in [1,2]$, where in case $p_y \in (1, 2]$, (54a) is replaced for some ${{\tilde{\gamma }}}_G>0$ with

$$\begin{aligned} \xi _x&= \gamma _G - {{\tilde{\gamma }}}_G - \frac{p_y-1}{(\theta _yp_y^{p_y}(\rho _x)^{p_y-2}\omega ^{-1})^{\frac{1}{p_y-1}}}, \end{aligned}$$

(79a)

and in case $p_x \in (1, 2]$, (54b) is replaced for some ${{\tilde{\gamma }}}_{F^*}>0$ with

$$\begin{aligned} \xi _y&= \gamma _{F^*} - {{\tilde{\gamma }}}_{F^*} - \frac{p_x-1}{(\omega \theta _xp_x^{p_x}(\rho _y)^{p_x-2})^{\frac{1}{p_x-1}}}. \end{aligned}$$

(79b)

Proof

Conditions (79) are sufficient for (70) with ${\overline{\omega }}={\underline{\omega }}=\omega$ to hold; therefore, we can repeat the proof of Theorem 6.4 replacing the references to Theorem 4.2 by references to Corollary B.1. $\square$

Corollary B.6

The results of Proposition 6.6 continue to hold if the corresponding conditions of Theorem 6.1, 6.3, or 6.4 are replaced with those in Corollary B.3, B.4, or B.5.

Proof

The proof repeats that of Proposition 6.6. $\square$

Verification of Conditions for Step Function Presentation and Potts Model

Throughout this section, we set $\rho (t) :=2t-t^2$ and $\kappa (x, y) :=\rho (\langle x,y\rangle )$ for $x, y \in {\mathbb {R}}^m$. Then $\rho '(t)=2(1-t)$ so that

$$\begin{aligned} \kappa _x(x,y)= & {} 2y(1-\langle y,x\rangle )\quad \text {and} \, \kappa _{xy}(x,y)=2(I-\langle y,x\rangle I-y \otimes x), \end{aligned}$$

(80a)

$$\begin{aligned} \kappa _y(x,y)= & {} 2x(1-\langle x,y\rangle )\quad \text {and} \, \kappa _{yx}(x,y)=2(I-\langle x,y\rangle I-x \otimes y), \end{aligned}$$

(80b)

where $a\otimes b\in {\mathbb {R}}^{n\times n}$ is the tensor product between two vectors a and b, producing a matrix of all the combinations of products between the entries.

The following lemma verifies Assumption 3.2 for $K=\kappa$.

Lemma C.1

Let $R_K>2$, and suppose ${\widehat{x}},{\widehat{y}}\in {\mathbb {R}}^m$ for $m \ge 1$ with

$$\begin{aligned} 0 \le \langle {\widehat{x}},{\widehat{y}}\rangle I + {\widehat{x}}\otimes {\widehat{y}}\le 2I. \end{aligned}$$

(81)

Then the function $K=\kappa$ defined above satisfies Assumption 3.2 for some $\theta _x, \theta _y >0$ and some $\rho _x,\rho _y>0$ dependent on $R_K$ with

$$\begin{aligned} L_x(y)&= 2|y|_2^2,&L_y(x)&= 2|x|_2^2,&L_{yx}&= 4(|{\widehat{y}}|_2+\rho _y), \end{aligned}$$

as well as the constants $\xi _x,\xi _y \in {\mathbb {R}}$, $\lambda _x,\lambda _y \ge 0$ satisfying $\lambda _x\xi _x > 2(\lambda _x+|{\widehat{y}}|_2^2)|{\widehat{y}}|_2^2$, $\xi _y > 0$, and $\lambda _y > |{\widehat{x}}|_2^2$.

Proof

First, Assumption 3.2 (i) holds everywhere since $K\in C^\infty ({\mathbb {R}}^m)$. To verify Assumption 3.2 (ii), we observe using (80) that

$$\begin{aligned} \kappa _x(x', y)-\kappa _x(x, y)= & {} 2(y \otimes y)(x-x'), \end{aligned}$$

(82a)

$$\begin{aligned} \kappa _{xy}(x, y')-\kappa _{xy}(x, y)= & {} 2\langle y-y',x\rangle I+2(y-y') \otimes x, \end{aligned}$$

(82b)

$$\begin{aligned} \kappa _y(x, y')-\kappa _y(x, y)= & {} 2(x \otimes x)(y-y'), \end{aligned}$$

(82c)

$$\begin{aligned} \kappa _{yx}(x', y)-\kappa _{yx}(x, y)= & {} 2\langle x-x',y\rangle I+2(x-x') \otimes y. \end{aligned}$$

(82d)

Hence $L_x$, $L_y$, and $L_{yx}$ are as claimed.

To verify Assumption 3.2 (iii), we first of all observe using (81) that

$$\begin{aligned} |\kappa _{xy}({\widehat{x}}, {\widehat{y}})|_2 = 2|I-\langle {\widehat{y}},{\widehat{x}}\rangle I-{\widehat{y}}\otimes {\widehat{x}}|_2 \le 2. \end{aligned}$$

Therefore ${\sup _{(x, y) \in \mathbb {B}({\widehat{x}}, \rho _x) \times \mathbb {B}({\widehat{y}}, \rho _y)} |\kappa _{xy}(x, y)|_2 \le R_K}$ for some ${\rho _x, \rho _y>0}$ dependent on $R_K>2$.

Finally, to verify Assumption 3.2 (iv), we start with (15a), i.e.,

$$\begin{aligned}&\langle \kappa _x(x',{\widehat{y}})-\kappa _x({\widehat{x}},{\widehat{y}}),x-{\widehat{x}}\rangle +\xi _x|x-{\widehat{x}}|_2^2 \\&\quad \ge \theta _x|\kappa _y({\widehat{x}},y)-\kappa _y(x,y)-\kappa _{yx}(x,y)({\widehat{x}}-x)|_2 -\frac{\lambda _x}{2}|x-x'|_2^2. \end{aligned}$$

Expanding the equation using (80), (82), and

$$\begin{aligned} \begin{aligned} \kappa _y({\widehat{x}},y)&-\kappa _y(x,y)-\kappa _{yx}(x,y)({\widehat{x}}-x) \\&=2{\widehat{x}}(1-\langle {\widehat{x}},y\rangle )-2x(1-\langle x,y\rangle ) -2(I-\langle x,y\rangle I-x \otimes y)({\widehat{x}}-x) \\ {}&=2[\langle x,y\rangle x-\langle {\widehat{x}},y\rangle {\widehat{x}}+(\langle x,y\rangle I+x \otimes y)({\widehat{x}}-x)] \\ {}&=2[\langle x-{\widehat{x}},y\rangle {\widehat{x}}+(x \otimes y)({\widehat{x}}-x)] \\ {}&=-2(({\widehat{x}}-x) \otimes y)({\widehat{x}}-x), \end{aligned} \end{aligned}$$

we require that

$$\begin{aligned} 2\langle {\widehat{x}}-x',x-{\widehat{x}}\rangle _{{\widehat{y}}\otimes {\widehat{y}}} +\xi _x|x-{\widehat{x}}|_2^2 \ge 2\theta _x|y|_2|x-{\widehat{x}}|_2^2 -\frac{\lambda _x}{2}|x-x'|_2^2. \end{aligned}$$

(83)

Taking any $\alpha >0$, this will hold by Cauchy’s and Young’s inequalities if $\xi _x \ge (2+\alpha )|{\widehat{y}}|_2^2 + 2\theta _x|y|_2$ and $\lambda _x/2 \ge \alpha ^{-1}|{\widehat{y}}|_2^2$. If $|{\widehat{y}}|_2=0$, clearly these hold for some $\alpha ,\theta _x>0$. Otherwise, solving $\alpha$ from the latter as an equality, i.e., taking $\alpha =2\lambda ^{-1}_x|{\widehat{y}}|_2^2$, the former holds if $\xi _x \ge 2(1+\lambda ^{-1}_x|{\widehat{y}}|_2^2)|{\widehat{y}}|_2^2 + 2\theta _x|y|_2$. If $\lambda _x\xi _x > 2(\lambda _x+|{\widehat{y}}|_2^2)|{\widehat{y}}|_2^2$, this holds for some $\theta _x,\rho _x,\rho _y>0$ in a neighborhood ${\mathbb {B}({\widehat{x}}, \rho _x) \times \mathbb {B}({\widehat{y}}, \rho _y)}$ of (${\widehat{x}}, {\widehat{y}})$.

It remains to verify (15b), i.e.,

$$\begin{aligned}&\langle \kappa _y(x,y)-\kappa _y(x,y')+\kappa _y({\widehat{x}},{\widehat{y}})-\kappa _y({\widehat{x}},y),y-{\widehat{y}}\rangle +\xi _y|y-{\widehat{y}}|_2^2 \\&\quad \ge \theta _y|\kappa _x(x',{\widehat{y}})-\kappa _x(x',y')-\kappa _{xy}(x',y')({\widehat{y}}-y')|_2 -\frac{\lambda _y}{2}|y-y'|_2^2. \end{aligned}$$

Again, using (80) and (82) we expand this as

$$\begin{aligned} 2\langle y'-y,y-{\widehat{y}}\rangle _{x \otimes x}+2|y-{\widehat{y}}|_{{\widehat{x}}\otimes {\widehat{x}}}^2 +\xi _y|y-{\widehat{y}}|_2^2 \ge 2\theta _y|x'|_2|y'-{\widehat{y}}|_2^2 -\frac{\lambda _y}{2}|y-y'|_2^2. \end{aligned}$$

Rearranging the $\theta _y$-term, we see that this holds if

$$\begin{aligned}&2\langle y'-y,y-{\widehat{y}}\rangle _{x \otimes x-2\theta _y|x'|_2I} +2|y-{\widehat{y}}|^2_{{\widehat{x}}\otimes {\widehat{x}}}+(\xi _y-2\theta _y)|x'|_2|y-{\widehat{y}}|_2^2 \\&\quad \ge \left( 2\theta _y|x'|_2-\frac{\lambda _y}{2}\right) |y'-y|_2^2. \end{aligned}$$

Rearranging and estimating the first term as

$$\begin{aligned} \begin{aligned} 2\langle y'-y,y-{\widehat{y}}\rangle _{x \otimes x-2\theta _y|x'|_2I}&= 2\langle y'-y,x\rangle \langle y-{\widehat{y}},x\rangle -4\theta _y|x'|_2\langle y'-y,y-{\widehat{y}}\rangle \\&\ge -2|y-{\widehat{y}}|^2_{x \otimes x}-\frac{1}{2}|y'-y|^2_{x \otimes x} -4\theta _y|x'|_2|y'-y|_2^2-\theta _y|x'|_2|y-{\widehat{y}}|_2^2 \end{aligned} \end{aligned}$$

and then using Young’s inequality on both parts, we obtain the condition

$$\begin{aligned} 2\left( |y-{\widehat{y}}|^2_{{\widehat{x}}\otimes {\widehat{x}}}-|y-{\widehat{y}}|^2_{x \otimes x}\right) +(\xi _y-3\theta _y)|x'|_2|y-{\widehat{y}}|_2^2 \ge \left( \frac{1}{2}|x|_2^2 + 6\theta _y|x'|_2-\frac{\lambda _y}{2}\right) |y'-y|_2^2. \end{aligned}$$

If $\xi _y > 0$ and $\lambda _y > |{\widehat{x}}|_2^2$, this holds for some $\theta _y,\rho _y,\rho _x>0$ in ${\mathbb {B}({\widehat{x}}, \rho _x) \times \mathbb {B}({\widehat{y}}, \rho _y)}$. $\square$

We comment on the condition (81) on the primal–dual solutions pair ${\widehat{x}}, {\widehat{y}}\in {\mathbb {R}}$. First, for $m=1$, this condition reduces to ${\widehat{x}}{\widehat{y}}\in [0, 1]$. This is necessarily satisfied in the case of the step function (where $f^*=\delta _{[0, \infty )}$) and in the case of the $\ell ^0$ function (where $f^*=0$) as in both cases, ${\widehat{x}}{\widehat{y}}\in \{0, 1\}$ by the dual optimality condition $\kappa _y({\widehat{x}}, {\widehat{y}}) \in \partial f^*({\widehat{y}})$. Furthermore, if we take $f^*_\gamma =\frac{\gamma }{2}|\,\varvec{\cdot }\,|_2^2$ for some $\gamma \ge 0$, then for any $m\ge 1$ the dual optimality condition reads $2{\widehat{x}}(1-\langle {\widehat{x}},{\widehat{y}}\rangle )=\gamma {\widehat{y}}$, i.e, ${\widehat{y}}=2{\widehat{x}}(\gamma +2|{\widehat{x}}|_2^2)^{-1}$, for which (81) is easily verified.

The following lemma shows that Assumption 3.2 remains valid if we include a linear operator in the primal component.

Lemma C.2

Let $K(x, y)={\tilde{K}}(Ax, y)$ for some ${A \in \mathbb {L}(X; Z)}$ and ${\tilde{K}} \in C^1(Z \times Y)$ on Hilbert spaces X, Y, Z. Suppose ${\tilde{K}}$ satisfies Assumption 3.2 at $({\widehat{z}}, {\widehat{y}}) :=(A {\widehat{x}}, {\widehat{y}})$. Mark the corresponding constants with a tilde: ${\tilde{L}}_z$, ${\tilde{R}}_K$, and so on. Then K satisfies Assumption 3.2 with $R_K :={\tilde{R}}_K \Vert A\Vert$; $\xi _x=\Vert A\Vert {{\tilde{\xi }}}_z$, $\xi _y={{\tilde{\xi }}}_y$; $\lambda _x=\Vert A\Vert {{\tilde{\lambda }}}_z$, $\lambda _y={{\tilde{\lambda }}}_y$; $\theta _x={{\tilde{\theta }}}_z$, $\theta _y={{\tilde{\theta }}}_y\Vert A\Vert ^{-1}$; $\rho _x=\Vert A\Vert ^{-1}{{\tilde{\rho }}}_x$, and $\rho _y={{\tilde{\rho }}}_y$ as well as

$$\begin{aligned} L_x(y)&=\Vert A\Vert ^2{\tilde{L}}_z(y),&L_y(x)&={\tilde{L}}_y(Ax),&L_{yx}&=\Vert A\Vert ^2{\tilde{L}}_{yz}. \end{aligned}$$

(84)

Proof

Observe first of all that by the chain rule,

$$\begin{aligned} K_x(x, y)&= A^* {\tilde{K}}_z(Ax, y),&K_y(x, y)&= {\tilde{K}}_y(Ax, y),&K_{xy}(x,y)&=A^*{\tilde{K}}_{zy}(Ax, y), \end{aligned}$$

and hence Assumption 3.2 (i) holds for K if it holds for ${\tilde{K}}$.

Let now Assumption 3.2 (ii) hold for ${\tilde{K}}$ with ${\tilde{L}}_x$, ${\tilde{L}}_y$, and ${\tilde{L}}_{yx}$. Observing that

$$\begin{aligned} {A\mathbb {B}({\widehat{x}}, \rho _x) \times \mathbb {B}({\widehat{y}}, \rho _y) \subset \mathbb {B}({\widehat{z}}, {{\tilde{\rho }}}_x) \times \mathbb {B}({\widehat{y}},{{\tilde{\rho }}}_y),} \end{aligned}$$

(85)

Assumption 3.2 (ii) thus also holds with the function of (84). Similarly in Assumption 3.2 (iii), we can take $R_K :={\tilde{R}}_K \Vert A\Vert$.

Finally, we expand Assumption 3.2 (iv) for K as

$$\begin{aligned} \begin{aligned}&\langle {\tilde{K}}_z(z',{\widehat{y}})-{\tilde{K}}_z({\widehat{z}},{\widehat{y}}),z-{\widehat{z}}\rangle +\xi _x\Vert x-{\widehat{x}}\Vert ^2 \\&\quad \ge \theta _x\Vert {\tilde{K}}_y({\widehat{z}},y)-{\tilde{K}}_y(z,y)-{\tilde{K}}_{yz}(z,y)({\widehat{z}}-z)\Vert -\frac{\lambda _x}{2}\Vert x-x'\Vert ^2 \end{aligned} \end{aligned}$$

and

$$\begin{aligned} \begin{aligned}&\langle {\tilde{K}}_y(z,y)-{\tilde{K}}_y(z,y') +{\tilde{K}}_y({\widehat{z}},{\widehat{y}})-{\tilde{K}}_y({\widehat{z}},y),y-{\widehat{y}}\rangle +\xi _y\Vert y-{\widehat{y}}\Vert ^2 \\&\quad \ge \theta _y\Vert A^*[{\tilde{K}}_z(z',{\widehat{y}}) -{\tilde{K}}_z(z',y')-{\tilde{K}}_{zy}(z',y')({\widehat{y}}-y')]\Vert -\frac{\lambda _y}{2}\Vert y-y'\Vert ^2, \end{aligned} \end{aligned}$$

where $z=Ax$, $z'=Ax'$, and ${\widehat{z}}=A{\widehat{x}}$. Since $\Vert z-z'\Vert \le \Vert A\Vert \Vert x-x'\Vert$, etc., this follows from Assumption 3.2 (iv) for ${\tilde{K}}$ with the constants as claimed. $\square$

Applying this lemma to ${\tilde{K}}(z, y)=\sum _{k=1}^n \kappa (z_k, y_k)$, we can thus lift the scalar estimates for $K=\kappa$ as in (80) to the corresponding estimates on $K(x, y) :=\sum _{k=1}^n \kappa ([D_h x]_k, y_k)$ as used in the Potts model example.