1 Introduction

1.1 Problem Statement and Motivation

In this paper we will consider the optimization problem

$$\begin{aligned} \begin{array}{ll} \min &{} f \left( x \right) , \\ \text {subject to} &{} Ax = b \end{array} \end{aligned}$$
(1.1)

where

$$\begin{aligned} {\left\{ \begin{array}{ll} \mathcal {X}, \mathcal {Y}\text { are real Hilbert spaces}; \\ f :\mathcal {X}\rightarrow \mathbb {R}\text { is a continuously differentiable convex function}; \\ A :\mathcal {X}\rightarrow \mathcal {Y}\text { is a continuous linear operator and } b \in \mathcal {Y}; \\ \text {the set } \mathbb {S} \text { of primal-dual optimal solutions of } (1.1) \text { is assumed to be nonempty}. \end{array}\right. } \end{aligned}$$
(1.2)

This model formulation underlies many important applications in various areas, such as image recovery [25], machine learning [20, 31], the energy dispatch of power grids [42, 43], distributed optimization [32, 44] and network optimization [40, 45].

In recent years, there has been a flurry of research on the relationship between continuous time dynamical systems and the numerical algorithms that arise from their discretizations. For the unconstrained optimization problem, it has been known that inertial systems with damped velocities enjoy good convergence properties. For a convex, smooth function \(f: \mathcal {X} \rightarrow \mathbb {R}\), Polyak is the first to consider the heavy ball with friction (HBF) dynamics [37, 38]

$$\begin{aligned} \ddot{x}(t) + \gamma \dot{x}(t) + \nabla f(x(t)) = 0 . \end{aligned}$$
(HBF)

Alvarez and Attouch continue the line of this study, focusing on inertial dynamics with a fixed viscous damping coefficient [2,3,4]. Later on, Cabot et al. [21, 22] consider the system that replaces \(\gamma \) with a time dependent damping coefficient \(\gamma \left( t \right) \). In [41], Su, Boyd, and Candès showed that it turns out one can achieve fast convergence rates by introducing a time dependent damping coefficient which vanishes in a controlled manner, neither too fast nor too slowly, as t goes to infinity

$$\begin{aligned} \ddot{x}(t) + \frac{\alpha }{t}\dot{x}(t) + \nabla f(x(t)) = 0. \end{aligned}$$
(AVD)

For \(\alpha \geqslant 3\), the authors showed that a solution \(x :\left[ t_{0}, +\infty \right) \rightarrow \mathcal {X}\) to (AVD) satisfies \(f(x(t)) - f(x_{*}) = \mathcal {O}\left( \frac{1}{t^{2}}\right) \) as \(t \rightarrow +\infty \). In fact, the choice \(\alpha = 3\) provides a continuous limit counterpart to Nesterov’s celebrated accelerated gradient algorithm [15, 34, 35]. Weak convergence of the trajectories to minimizers of f when \(\alpha > 3\) has been shown by Attouch et al. in [6] and May in [33], together with the improved rates of convergence \(f(x(t)) - f(x_{*}) = o\left( \frac{1}{t^{2}}\right) \) as \(t\rightarrow +\infty \). In the meantime, similar results for the discrete counterpart were also reported by Chambolle and Dossal in [23], and by Attouch and Peypouquet in [13].

In [7], Attouch, Chbani, and Riahi proposed an inertial proximal type algorithm, which results from a discretization of the time rescaled (AVD) system

$$\begin{aligned} \ddot{x}(t) + \frac{\alpha }{t}\dot{x}(t) + \delta (t) \nabla f(x(t)) = 0, \end{aligned}$$

where \(\delta :\left[ t_{0}, + \infty \right) \rightarrow \mathbb {R}_{+}\) is a time scaling function satisfying a certain growth condition, which enters the convergence statement by way of \(f \left( x \left( t \right) \right) - f \left( x_{*} \right) = \mathcal {O}\left( \frac{1}{t^{2} \delta \left( t \right) } \right) \) as \(t \rightarrow + \infty \). The resulting algorithm obtained by the authors is considerably simpler than the founding proximal point algorithm proposed by Güler in [26], while providing comparable convergence rates for the functional values.

In order to approach constrained optimization problems, Augmented Lagrangian Method (ALM) [39] (for linearly constrained problems) and Alternating Direction Method of Multipliers (ADMM) [20, 24] (for problems with separable objectives and block variables linearly coupled in the constraints) and some of their variants have been shown to enjoy substantial success. Continuous-time approaches for structured convex minimization problems formulated in the spirit of the full splitting paradigm have been recently addressed in [18] and, closely connected to our approach, in [10, 17, 27, 45], to which we will have a closer look in Subsection 2.2. The temporal discretization resulting from these dynamics gives rise to the numerical algorithm with fast convergence rates [28, 29] and with a convergence guarantee for the generated iterate [19], without additional assumptions such as strong convexity.

In this paper, we will investigate a second-order dynamical system with asymptotic vanishing damping and time rescaling term, which is associated with the optimization problem (1.1) and formulated in terms of its augmented Lagrangian. The case when the time rescaling term does not appear has been established in [17]. We show that by introducing this time rescaling function, we are able to derive faster convergence rates for the primal-dual gap, the feasibility measure, and the objective function value along the generated trajectories while still maintaining the asymptotic behaviour of the trajectories towards a primal-dual optimal solution. On the other hand, this work can also be viewed as an extension of the time rescaling technique derived in [7, 9] for the constrained case. To our knowledge, the trajectory convergence for dynamics with time scaling seems to be new in the constrained case.

1.2 Notations and a Preliminary Result

For both Hilbert spaces \(\mathcal {X}\) and \(\mathcal {Y}\), the Euclidean inner product and the associated norm will be denoted by \(\left\langle \cdot , \cdot \right\rangle \) and \(\left\Vert \cdot \right\Vert \), respectively. The Cartesian product \(\mathcal {X}\times \mathcal {Y}\) will be endowed with the inner product and the associated norm defined for \(\left( x, \lambda \right) , \left( z, \mu \right) \in \mathcal {X}\times \mathcal {Y}\) as

$$\begin{aligned} \left\langle \left( x, \lambda \right) , \left( z, \mu \right) \right\rangle = \left\langle x, z \right\rangle + \left\langle \lambda , \mu \right\rangle \qquad \text { and } \qquad \left\Vert \left( x, \lambda \right) \right\Vert = \sqrt{\left\Vert x \right\Vert ^{2} + \left\Vert \lambda \right\Vert ^{2}}, \end{aligned}$$

respectively.

Let \(f :\mathcal {X}\rightarrow \mathbb {R}\) be a continuously differentiable convex function such that \(\nabla f\) is \(\ell -\)Lipschitz continuous. For every \(x, y \in \mathcal {X}\) it holds (see [35, Theorem 2.1.5])

$$\begin{aligned} 0 \leqslant \dfrac{1}{2 \ell } \left\Vert \nabla f \left( x \right) - \nabla f \left( y \right) \right\Vert ^{2} \leqslant f \left( x \right) - f \left( y \right) - \left\langle \nabla f \left( y \right) , x - y \right\rangle \leqslant \dfrac{\ell }{2} \left\Vert x - y \right\Vert ^{2} . \end{aligned}$$
(1.3)

2 The Primal-Dual Dynamical Approach

2.1 Augmented Lagrangian Formulation

Consider the saddle point problem

$$\begin{aligned} \min _{x \in \mathcal {X}} \max _{\lambda \in \mathcal {Y}} \mathcal {L}\left( x, \lambda \right) \end{aligned}$$
(2.1)

associated to problem (1.1), where \(\mathcal {L}:\mathcal {X}\times \mathcal {Y}\rightarrow \mathbb {R}\) denotes the Lagrangian function

$$\begin{aligned} \mathcal {L}\left( x, \lambda \right) := f \left( x \right) + \left\langle \lambda , Ax - b \right\rangle . \end{aligned}$$

Under the assumptions (1.2), \(\mathcal {L}\) is convex with respect to \(x \in \mathcal {X}\) and affine with respect to \(\lambda \in \mathcal {Y}\). A pair \(\left( x_{*}, \lambda _{*} \right) \in \mathcal {X}\times \mathcal {Y}\) is said to be a saddle point of the Lagrangian function \(\mathcal {L}\) if for every \(\left( x, \lambda \right) \in \mathcal {X}\times \mathcal {Y}\)

$$\begin{aligned} \mathcal {L}\left( x_{*}, \lambda \right) \leqslant \mathcal {L}\left( x_{*}, \lambda _{*} \right) \leqslant \mathcal {L}\left( x, \lambda _{*} \right) . \end{aligned}$$
(2.2)

If \(\left( x_{*}, \lambda _{*} \right) \in \mathcal {X}\times \mathcal {Y}\) is a saddle point of \(\mathcal {L}\) then \(x_{*} \in \mathcal {X}\) is an optimal solution of (1.1), and \(\lambda _{*} \in \mathcal {Y}\) is an optimal solution of its Lagrange dual problem. If \(x_{*} \in \mathcal {X}\) is an optimal solution of (1.1) and a suitable constraint qualification is fulfilled, then there exists an optimal solution \(\lambda _{*} \in \mathcal {Y}\) of the Lagrange dual problem such that \(\left( x_{*}, \lambda _{*} \right) \in \mathcal {X}\times \mathcal {Y}\) is a saddle point of \(\mathcal {L}\). For details and insights into the topic of constraint qualifications for convex duality we refer to [14, 16].

The set of saddle points of \(\mathcal {L}\), called also primal-dual optimal solutions of (1.1), will be denoted by \(\mathbb {S}\) and, as stated in the assumptions, it will be assumed to be nonempty. The set of feasible points of (1.1) will be denoted by \(\mathbb {F}:= \left\{ x \in \mathcal {X}:Ax = b \right\} \) and the optimal objective value of (1.1) by \(f_{*}\).

The system of primal-dual optimality conditions for (1.1) reads

$$\begin{aligned} \left( x_{*}, \lambda _{*} \right) \in \mathbb {S}\Leftrightarrow {\left\{ \begin{array}{ll} \nabla _{x} \mathcal {L}\left( x_{*}, \lambda _{*} \right) &{} = 0 \\ \nabla _{\lambda } \mathcal {L}\left( x_{*}, \lambda _{*} \right) &{} = 0 \end{array}\right. } \Leftrightarrow {\left\{ \begin{array}{ll} \nabla f \left( x_{*} \right) + A^{*} \lambda _{*} &{} = 0 \\ Ax_{*} - b &{} = 0 \end{array}\right. }, \end{aligned}$$
(2.3)

where \(A^{*}: \mathcal {Y}\rightarrow \mathcal {X}\) denotes the adjoint operator of A.

For \(\beta \geqslant 0\), we consider also the augmented Lagrangian \(\mathcal {L}_{\beta }:\mathcal {X}\times \mathcal {Y}\rightarrow \mathbb {R}\) associated with (1.1)

$$\begin{aligned} \mathcal {L}_{\beta }\left( x, \lambda \right) := \mathcal {L}\left( x, \lambda \right) + \dfrac{\beta }{2} \left\Vert Ax - b \right\Vert ^{2} = f \left( x \right) + \left\langle \lambda , Ax - b \right\rangle + \dfrac{\beta }{2} \left\Vert Ax - b \right\Vert ^{2} . \end{aligned}$$
(2.4)

For every \((x, \lambda ) \in \mathbb {F}\times \mathcal {Y}\) it holds

$$\begin{aligned} f \left( x \right) = \mathcal {L}_{\beta }\left( x, \lambda \right) = \mathcal {L}\left( x, \lambda \right) . \end{aligned}$$
(2.5)

If \(\left( x_{*}, \lambda _{*} \right) \in \mathbb {S}\), then we have for every \(\left( x, \lambda \right) \in \mathcal {X}\times \mathcal {Y}\)

$$\begin{aligned} \mathcal {L}\left( x_{*}, \lambda \right) = \mathcal {L}_{\beta }\left( x_{*}, \lambda \right) \leqslant \mathcal {L}\left( x_{*}, \lambda _{*} \right) = \mathcal {L}_{\beta }\left( x_{*}, \lambda _{*} \right) \leqslant \mathcal {L}\left( x, \lambda _{*} \right) \leqslant \mathcal {L}_{\beta }\left( x, \lambda _{*} \right) . \end{aligned}$$

In addition, from (2.3) we have

$$\begin{aligned} \begin{aligned} \left( x_{*}, \lambda _{*} \right) \in \mathbb {S}&\Leftrightarrow {\left\{ \begin{array}{ll} \nabla f \left( x_{*} \right) + A^{*} \lambda _{*} &{} = 0 \\ Ax_{*} - b &{} = 0 \end{array}\right. } \,\Leftrightarrow \, {\left\{ \begin{array}{ll} \nabla f \left( x_{*} \right) + A^{*} \lambda _{*} + \beta A^{*}(Ax_{*} - b) &{}= 0 \\ Ax_{*} - b &{}= 0 \end{array}\right. } \\&\Leftrightarrow {\left\{ \begin{array}{ll} \nabla _{x} \mathcal {L}_{\beta }(x_{*}, \lambda _{*}) &{}= 0 \\ \nabla _{\lambda } \mathcal {L}_{\beta }(x_{*}, \lambda _{*}) &{}= 0. \end{array}\right. } \end{aligned} \end{aligned}$$

In other words, for any \(\beta \geqslant 0\) the sets of saddle points of \(\mathcal {L}\) and \(\mathcal {L}_{\beta }\) are identical.

2.2 The Primal-Dual Asymptotic Vanishing Damping Dynamical System with Time Rescaling

In this subsection we present the system under study, and we include a brief discussion regarding the existence and uniqueness of solutions.

The dynamical system which we associate to (1.1) and investigate in this paper reads

figure a

where \(t_0 >0\), \(\alpha > 0\), \(\theta > 0\), \(\delta :\left[ t_{0}, +\infty \right) \rightarrow \mathbb {R}\) is a nonnegative continuously differentiable function and \(\left( x_{0}, \lambda _{0} \right) , \bigl ( \dot{x}_{0}, \dot{\lambda }_{0} \bigr ) \in \mathcal {X}\times \mathcal {Y}\) are the initial conditions. Replacing the expressions of the partial gradients of \(\mathcal {L}_{\beta }\) into the system leads to the following formulation for (2.6):

The case (2.6) in which there is no time rescaling, i.e., when \(\delta (t) \equiv 1\), was studied by Zeng et al. in [45], and by Boţ and Nguyen in [17]. The system with more general damping, extrapolation and time rescaling coefficients was addressed by He et al. in [27, 30] and by Attouch et al. in [10]. We mention that extending the results in this paper to the multi-block case is possible. For further details, we refer the readers to [17, Sect. 2.4].

It is well known that the viscous damping term \(\frac{\alpha }{t}\) has a vital role in achieving fast convergence in unconstrained minimization [6, 8, 33]. The role of the extrapolation \(\theta t\) is to induce more flexibility in the dynamical system and in the associated discrete schemes, as it has been recently noticed in [10, 12, 27, 45]. The time scaling function \(\delta \left( \cdot \right) \) has the role to further improve the rates of convergence of the objective function value along the trajectory, as it was noticed in the context of unconstrained minimization problems in [7, 9, 11] and of linearly constrained minimization problems in [10, 30].

It is straightforward to show the existence of local solutions to (2.6), under the additional assumption that \(\nabla f\) is Lipschitz continuous on every bounded subset of \(\mathcal {X}\). First, notice that (2.6) can be rewritten as a first-order dynamical system. Indeed, \(\left( x, \lambda \right) :\left[ t_{0}, + \infty \right) \rightarrow \mathcal {X} \times \mathcal {Y}\) is a solution to (2.6) if and only if \((x, \lambda , y, \nu ) :\left[ t_{0}, + \infty \right) \rightarrow \mathcal {X} \times \mathcal {Y} \times \mathcal {X} \times \mathcal {Y}\) is a solution to

where \(F :\left[ t_{0}, + \infty \right) \times \mathcal {X} \times \mathcal {Y} \times \mathcal {X} \times \mathcal {Y} \rightarrow \mathcal {X} \times \mathcal {Y} \times \mathcal {X} \times \mathcal {Y}\) is given by

$$\begin{aligned}&F(t, z, \mu , w, \eta ) \\&\quad := \left( w, \eta , -\frac{\alpha }{t}w - \delta (t)\left[ \nabla f(z) + A^{*}\bigl ( \mu + \theta t \eta \bigr ) + \beta A^{*}\bigl ( Az - b\bigr )\right] ,\right. \\&\qquad \left. -\frac{\alpha }{t}\eta + \delta (t)\bigl [ A\bigl ( z + \theta t w\bigr ) - b\bigr ]\right) . \end{aligned}$$

where F is evidently continuous in t, and \(F(t, \cdot )\) is Lipschitz continuous on every bounded subset, provided that the same property holds for \(\nabla f\). We can then employ a theorem such as that by Cauchy-Lipschitz to obtain the existence of a unique solution to the previous system, and thus a unique solution to (2.6), defined on a maximal interval \([t_{0}, T_{\text {max}})\). To go further and show the existence and uniqueness of a global solution (that is, \(T_{\text {max}} = +\infty \)) we will need some energy estimates derived in the next section in a similar way as in [11, 17]. For this reason, the existence and uniqueness of a global solution is postponed to a later stage.

3 Faster Convergence Rates via Time Rescaling

In this section we will derive fast convergence rates for the primal-dual gap, the feasibility measure, and the objective function value along the trajectories generated by the dynamical system (2.6). We will make the following assumptions on the parameters \(\alpha \), \(\theta \), \(\beta \) and the function \(\delta \) throughout this section.

figure b

Besides the first three conditions that are known previously in [17], it is worth pointing out that we can deduce from the last one the following inequality for every \(t \geqslant t_{0}\):

$$\begin{aligned} \dfrac{t \dot{\delta } \left( t \right) }{\delta \left( t \right) } \leqslant \dfrac{1 - 2 \theta }{\theta } = \dfrac{1}{\theta } - 2 \leqslant \alpha - 3 . \end{aligned}$$
(3.2)

This gives a connection to the condition which appears in [7]. A few more comments regarding the function \(\delta \) will come later, after the convergence rates statements.

3.1 The Energy Function

Let \(\left( x, \lambda \right) :\left[ t_{0}, + \infty \right) \rightarrow \mathcal {X}\times \mathcal {Y}\) be a solution of (2.6). Let \((x_{*}, \lambda _{*}) \in \mathbb {S}\) be fixed, we define the energy function \(\mathcal {E}:\left[ t_{0}, + \infty \right) \rightarrow \mathbb {R}\)

$$\begin{aligned} \mathcal {E}\left( t \right)&:= \theta ^{2} t^{2} \delta (t) \left( \mathcal {L}_{\beta }\left( x \left( t \right) , \lambda _{*} \right) - \mathcal {L}_{\beta }\left( x_{*}, \lambda \left( t \right) \right) \right) + \dfrac{1}{2} \left\Vert v \left( t \right) \right\Vert ^{2} \nonumber \\&\quad + \dfrac{\xi }{2} \left\Vert \bigl ( x \left( t \right) , \lambda \left( t \right) \bigr ) - (x_{*}, \lambda _{*}) \right\Vert ^{2}, \end{aligned}$$
(3.3)

where

$$\begin{aligned} v \left( t \right)&:= \bigl ( x \left( t \right) , \lambda \left( t \right) \bigr ) - (x_{*}, \lambda _{*}) + \theta t \left( \dot{x} \left( t \right) , \dot{\lambda } \left( t \right) \right) , \end{aligned}$$
(3.4)
$$\begin{aligned} \xi&:= \alpha \theta - \theta - 1 \geqslant 0 . \end{aligned}$$
(3.5)

Notice that, according to (2.4) and (2.5), we have for every \(t \geqslant t_0\)

$$\begin{aligned} \mathcal {L}_{\beta }\left( x \left( t \right) , \lambda _{*} \right) - \mathcal {L}_{\beta }\left( x_{*}, \lambda \left( t \right) \right)&= \mathcal {L}\left( x \left( t \right) , \lambda _{*} \right) - \mathcal {L}\left( x_{*}, \lambda \left( t \right) \right) + \dfrac{\beta }{2} \left\Vert Ax \left( t \right) - b \right\Vert ^{2} \end{aligned}$$
(3.6)
$$\begin{aligned}&= \mathcal {L}\left( x \left( t \right) , \lambda _{*} \right) - f_{*} + \dfrac{\beta }{2} \left\Vert Ax \left( t \right) - b \right\Vert ^{2} \nonumber \\&= f \left( x \left( t \right) \right) - f_{*} \nonumber \\&\quad + \left\langle \lambda _{*}, Ax \left( t \right) - b \right\rangle + \dfrac{\beta }{2} \left\Vert Ax \left( t \right) - b \right\Vert ^{2} \geqslant 0, \end{aligned}$$
(3.7)

where \(f_{*}\) denotes the optimal objective value of (1.1). In addition, due to (3.7), we have

$$\begin{aligned} \mathcal {E}\left( t \right) \geqslant 0 \qquad \forall t \geqslant t_{0} . \end{aligned}$$
(3.8)

The construction of \(\mathcal {E}\) is inspired by [17]. However, one can notice that we only consider \(\mathcal {E}\) defined with respect to a fixed primal-dual solution \(\left( x_{*}, \lambda _{*} \right) \in \mathbb {S}\) rather than a family of energy functions, each defined with respect to a point \(\left( z, \mu \right) \in \mathbb {F} \times \mathcal {Y}\). This gives simpler proofs for some results when compared to those in [17].

Assumption 1 implies the nonnegativity of following quantity, which will appear many times in our analysis:

$$\begin{aligned} \sigma :\left[ t_{0}, + \infty \right) \rightarrow \mathbb {R}_{+}, \quad \sigma (t) := \frac{1 - 2\theta }{\theta }\delta (t) - t \dot{\delta }(t) . \end{aligned}$$
(3.9)

The following lemma gives us the decreasing property of the energy function. As a consequence of this lemma, we obtain some integrability results which will be needed later. The proofs are postponed to the Appendix.

Lemma 3.1

Let \(\left( x, \lambda \right) :\left[ t_{0}, + \infty \right) \rightarrow \mathcal {X}\times \mathcal {Y}\) be a solution of (2.6) and \((x_{*}, \lambda _{*}) \in \mathbb {S}\). For every \(t \geqslant t_{0}\) it holds

$$\begin{aligned} \dfrac{d}{dt} \mathcal {E}\left( t \right)&\leqslant - \theta ^{2} t \sigma (t) \left( \mathcal {L}_{\beta }\left( x \left( t \right) , \lambda _{*} \right) - \mathcal {L}_{\beta }\left( x_{*}, \lambda \left( t \right) \right) \right) - \dfrac{1}{2} \beta \theta t \delta \left( t \right) \left\Vert Ax \left( t \right) - b \right\Vert ^{2} \\&\quad - \xi \theta t \left\Vert \left( \dot{x} \left( t \right) , \dot{\lambda } \left( t \right) \right) \right\Vert ^{2} . \end{aligned}$$

Proof

See “Proof of Lemma 3.1” in Appendix B. \(\square \)

Theorem 3.2

Let \((x, \lambda ): \left[ t_{0}, +\infty \right) \rightarrow \mathcal {X} \times \mathcal {Y}\) be a solution of (2.6) and \((x_{*}, \lambda _{*}) \in \mathbb {S}\). The following statements are true

  1. (i)

    it holds

    $$\begin{aligned} \int _{t_{0}}^{+\infty } t \sigma (t)\Big [\mathcal {L}\left( x(t), \lambda _{*} \right) - \mathcal {L}(x_{*}, \lambda (t))\Big ]dt&\leqslant \mathcal {E}(t_{0}) \,< +\infty , \end{aligned}$$
    (3.10)
    $$\begin{aligned} \beta \int _{t_{0}}^{+\infty } t \delta (t) \left\Vert Ax \left( t \right) - b \right\Vert ^{2}dt&\leqslant \frac{2\mathcal {E}(t_{0})}{\theta } \,< +\infty , \end{aligned}$$
    (3.11)
    $$\begin{aligned} \xi \int _{t_{0}}^{+\infty } t \left\Vert \left( \dot{x} \left( t \right) , \dot{\lambda } \left( t \right) \right) \right\Vert ^{2}&\leqslant \frac{\mathcal {E}(t_{0})}{\theta } \, < +\infty ; \end{aligned}$$
    (3.12)
  2. (ii)

    if, in addition, \(\alpha > 3\) and \(\frac{1}{2} \geqslant \theta > \frac{1}{\alpha - 1}\), then the trajectory \((x(t), \lambda (t))_{t\geqslant t_{0}}\) is bounded and the convergence rate of its velocity is

    $$\begin{aligned} \left\Vert \left( \dot{x} \left( t \right) , \dot{\lambda } \left( t \right) \right) \right\Vert = \mathcal {O}\left( \frac{1}{t}\right) \quad \text {as} \quad t\rightarrow +\infty . \end{aligned}$$

Proof

See “Proof of Theorem 3.2” in Appendix B. \(\square \)

3.2 Fast Convergence Rates for the Primal-Dual Gap, the Feasibility Measure and the Objective Function Value

The following are the main convergence rates results of the paper.

Theorem 3.3

Let \((x, \lambda ): \left[ t_{0}, +\infty \right) \rightarrow \mathcal {X}\times \mathcal {Y}\) be a solution of (2.6) and \((x_{*}, \lambda _{*}) \in \mathbb {S}\). The following statements are true

  1. (i)

    for every \(t\geqslant t_{0}\) it holds

    $$\begin{aligned} 0 \leqslant \mathcal {L}\left( x(t), \lambda _{*} \right) - \mathcal {L}(x_{*}, \lambda (t)) \leqslant \frac{\mathcal {E}(t_{0})}{\theta ^{2} t^{2} \delta (t)}; \end{aligned}$$
    (3.13)
  2. (ii)

    for every \(t\geqslant t_{0}\) it holds

    $$\begin{aligned} \left\Vert Ax \left( t \right) - b \right\Vert \leqslant \frac{2 C_{1}}{t^{2} \delta (t)}, \end{aligned}$$
    (3.14)

    where

    $$\begin{aligned} C_{1}&:= \sup _{t\geqslant t_{0}} t \left\Vert \dot{\lambda } \left( t \right) \right\Vert + (\alpha - 1) \sup _{t\geqslant t_{0}} \left\Vert \lambda \left( t \right) - \lambda _{*} \right\Vert \\&\quad + t_{0}^{2} \delta (t_{0}) \left\Vert Ax(t_{0}) - b \right\Vert + t_{0}\left\Vert \dot{\lambda } \left( t_{0} \right) \right\Vert ; \end{aligned}$$
  3. (iii)

    for every \(t\geqslant t_{0}\) it holds

    $$\begin{aligned} \left|f \left( x \left( t \right) \right) - f_{*} \right|\leqslant \left( \frac{\mathcal {E}(t_{0})}{\theta ^{2}} + 2 C_{1} \left\Vert \lambda _{*} \right\Vert \right) \frac{1}{t^{2} \delta (t)}. \end{aligned}$$
    (3.15)

Proof

(i) We have already established that \(\mathcal {E}\) is nonincreasing on \(\left[ t_{0}, +\infty \right) \). Therefore, from the expression for \(\mathcal {E}\) and relation (3.6) we deduce

$$\begin{aligned} \theta ^{2} t^{2} \delta (t) \bigl [\mathcal {L}\left( x(t), \lambda _{*} \right) - \mathcal {L}(x_{*}, \lambda (t)) \bigr ] \leqslant \mathcal {E}(t_{0}) \quad \forall t\geqslant t_{0}, \end{aligned}$$
(3.16)

and the first claim follows.

(ii) From the second line of (2.6), for every \(t\geqslant t_{0}\) we have

$$\begin{aligned} t \ddot{\lambda }(t) + \alpha \dot{\lambda }(t) = t\delta (t) \Bigl (A\bigl (x(t) + \theta t \dot{x}(t)\bigr ) - b\Bigr ) = t\delta (t)\bigl ( Ax(t) - b\bigr ) + \theta t^{2} \delta (t) A\dot{x}(t). \end{aligned}$$
(3.17)

Fix \(t\geqslant t_{0}\). On the one hand, integration by parts yields

$$\begin{aligned} \begin{aligned} \int _{t_{0}}^{t}\bigl ( s \ddot{\lambda }(s) + \alpha \dot{\lambda }(s)\bigr )ds&= t \dot{\lambda }(t) - t_{0} \dot{\lambda }(t_{0}) - \int _{t_{0}}^{t}\dot{\lambda }(s)ds + \alpha \int _{t_{0}}^{t} \dot{\lambda }(s)ds \\&= t \dot{\lambda }(t) - t_{0} \dot{\lambda }(t_{0}) + (\alpha - 1)(\lambda (t) - \lambda (t_{0})). \end{aligned} \end{aligned}$$
(3.18)

On the other hand, again integrating by parts leads to

$$\begin{aligned} \begin{aligned} \int _{t_{0}}^{t}s^{2} \delta (s) A\dot{x}(s) ds&= t^{2} \delta (t) (Ax(t) - b) - t_{0}^{2} \delta (t_{0}) (Ax(t_{0}) - b) \\&- \int _{t_{0}}^{t}\bigl ( 2 s \delta (s) + s^{2} \dot{\delta }(s)\bigr ) (Ax(s) - b)ds. \end{aligned} \end{aligned}$$
(3.19)

Now, integrating (3.17) from \(t_{0}\) to t and using (3.18) and (3.19) gives us

$$\begin{aligned}&t \dot{\lambda }(t) - t_{0} \dot{\lambda }(t_{0}) + (\alpha - 1)(\lambda (t) - \lambda (t_{0})) \nonumber \\&\quad = \int _{t_{0}}^{t}s \delta (s) (Ax(s) - b) ds + \theta \int _{t_{0}}^{t} s^{2} \delta (s) A\dot{x}(s) ds \nonumber \\&\quad = t^{2} \delta (t) (Ax(t) - b) - t_{0}^{2} \delta (t_{0})(Ax(t_{0}) - b) \nonumber \\&\qquad + \int _{t_{0}}^{t} s \bigl [(1 - 2\theta )\delta (s) - \theta s \dot{\delta }(s)\bigr ] (Ax(s) - b) ds \nonumber \\&\quad = t^{2} \delta (t) (Ax(t) - b) - t_{0}^{2} \delta (t_{0})(Ax(t_{0}) - b)\nonumber \\&\qquad + \int _{t_{0}}^{t}\frac{(1 - 2\theta )\delta (s) - \theta s \dot{\delta }(s)}{s \delta (s)} s^{2} \delta (s) (Ax(s) - b) ds. \end{aligned}$$
(3.20)

It follows that, for every \(t \geqslant t_{0}\), we have

$$\begin{aligned} \left\| t^{2} \delta (t) (Ax(t) - b) + \int _{t_{0}}^{t}\frac{(1 - 2\theta )\delta (s) - \theta s \dot{\delta }(s)}{s \delta (s)} s^{2} \delta (s) (Ax(s) - b) ds \right\| \leqslant C_{1}, \end{aligned}$$
(3.21)

where

$$\begin{aligned} C_{1}= & {} \sup _{t\geqslant t_{0}} t\left\Vert \dot{\lambda } \left( t \right) \right\Vert + (\alpha - 1) \sup _{t\geqslant t_{0}}\Vert \lambda (t) - \lambda \left( t_{0} \right) \Vert + t_{0}^{2} \delta (t_{0}) \left\Vert Ax(t_{0}) - b \right\Vert \\{} & {} +\, t_{0}\left\Vert \dot{\lambda } \left( t_{0} \right) \right\Vert < + \infty , \end{aligned}$$

and this quantity is finite in light of (B.7) and (B.5). Now, we set

$$\begin{aligned} g(t):= t^{2} \delta (t) \left\Vert Ax \left( t \right) - b \right\Vert , \qquad a(t):= \frac{(1 - 2\theta )\delta (t) - \theta t \dot{\delta }(t)}{t \delta (t)} \qquad \forall t\geqslant t_{0} \end{aligned}$$

and we apply Lemma A.1 to deduce that

$$\begin{aligned} t^{2} \delta (t) \left\Vert Ax \left( t \right) - b \right\Vert \leqslant 2C_{1} \quad \forall t\geqslant t_{0}. \end{aligned}$$
(3.22)

(iii) For a fixed \(t\geqslant t_{0}\), we have

$$\begin{aligned} \mathcal {L}\left( x(t), \lambda _{*} \right) - \mathcal {L}(x_{*}, \lambda (t)) = f(x(t)) - f(x_{*}) + \langle \lambda _{*}, Ax(t) - b\rangle . \end{aligned}$$

Therefore, from using (3.22) and (3.16) we obtain, for every \(t\geqslant t_{0}\),

$$\begin{aligned} \left|f \left( x \left( t \right) \right) - f_{*} \right|&\leqslant \mathcal {L}\left( x(t), \lambda _{*} \right) - \mathcal {L}(x_{*}, \lambda (t)) + \left\Vert \lambda _{*} \right\Vert \left\Vert Ax \left( t \right) - b \right\Vert \\&\leqslant \left( \frac{\mathcal {E}(t_{0})}{\theta ^{2}} + 2C_{1} \left\Vert \lambda _{*} \right\Vert \right) \frac{1}{t^{2} \delta (t)}, \end{aligned}$$

which leads to the last statement. \(\square \)

Some comments regarding the previous proof and results are in order.

Remark 3.4

The proof we provided here is significantly shorter than the one derived in [17] thanks to Lemma A.1. This Lemma is the one used in [28] for showing the fast convergence to zero of the feasibility measure, although the authors study a different dynamical system. On the other hand, when \(\delta \left( t \right) \equiv 1\), the result in [17] is more robust than the one we obtain here, as it gives the \(\mathcal {O}\left( \frac{1}{t^{2}} \right) \) rates for the sum of primal-dual gap and feasibility measure, rather than each one individually. It also allows us to focus only on the energy function defined with respect to a primal-dual optimal solution \(\left( x_{*}, \lambda _{*} \right) \in \mathbb {S}\), rather than on an arbitrary feasible point \(\left( z, \mu \right) \in \mathbb {F} \times \mathcal {Y}\) as in [17].

Remark 3.5

Here are some remarks comparing our rates of convergence to those in [10, 30].

  • Primal-dual gap: According to (3.13), the following rate of convergence for the primal-dual is exhibited:

    $$\begin{aligned} \mathcal {L}\bigl (x(t), \lambda _{*}\bigr ) - \mathcal {L}\bigl (x_{*}, \lambda (t)\bigr ) = \mathcal {O}\left( \frac{1}{t^{2} \delta (t)}\right) \quad \text {as} \quad t\rightarrow +\infty , \end{aligned}$$

    which coincides with the findings of [10, 30].

  • Feasibility measure: According to (3.14), we have

    $$\begin{aligned} \left\Vert Ax \left( t \right) - b \right\Vert = \mathcal {O}\left( \frac{1}{t^{2} \delta (t)}\right) \quad \text {as} \quad t\rightarrow +\infty , \end{aligned}$$

    which improves the rate \(\mathcal {O}\left( \frac{1}{t \sqrt{\delta (t)}}\right) \) reported in [10, 30].

  • Functional values: Relation (3.15) tells us that

    $$\begin{aligned} \left|f \left( x \left( t \right) \right) - f_{*} \right|= \mathcal {O}\left( \frac{1}{t^{2} \delta (t)}\right) \quad \text {as} \quad t\rightarrow +\infty . \end{aligned}$$

    In [10], only the upper bound presents this order of convergence. The lower bound obtained is of order \(\mathcal {O}\left( \frac{1}{t \sqrt{\delta (t)}}\right) \) as \(t \rightarrow +\infty \). In [30], there are no comments on the rate attained by the functional values in the case of a general time rescaling parameter.

  • Further comparisons with  [30]: in [30, Theorem 2.16], unlike the preceding result [30, Theorem 2.15], the authors produce a rate of \(\mathcal {O}\left( \frac{1}{t^{1/\theta }}\right) \) as \(t\rightarrow +\infty \) for \(\Vert Ax(t) - b\Vert \) and \(|f(x(t)) - f(x_{*})|\), provided the time rescaling parameter is chosen to be \(\delta (t) = \delta _{0} t^{\frac{1}{\theta } - 2}\), for some \(\delta _{0} > 0\). This choice comes from the solution to the differential equation

    $$\begin{aligned} \frac{t \dot{\delta }(t)}{\delta (t)} = \frac{1 - 2\theta }{\theta } \quad \forall t\geqslant t_{0}, \end{aligned}$$

    and thus is covered by our results when the growth condition (3.1) holds with equality. The rates are consequently

    $$\begin{aligned} \mathcal {O}\left( \frac{1}{t^{2} \cdot \delta _{0} t^{\frac{1}{\theta } - 2}}\right) = \mathcal {O}\left( \frac{1}{t^{\frac{1}{\theta }}}\right) \quad \text {as} \quad t\rightarrow +\infty . \end{aligned}$$

    In this setting, if we wish to obtain fast convergence rates, we need to choose a small \(\theta \). In light of Assumption 1, where we have \(\frac{1}{2}\geqslant \theta \geqslant \frac{1}{\alpha - 1}\), this can be achieved by taking \(\alpha \) large enough. Such behaviour can also be seen in [10] and in the unconstrained case [7, 11].

4 Weak Convergence of the Trajectory to a Primal-Dual Solution

In this section we will show that the solutions to (2.6) weakly converge to an element of \(\mathbb {S}\). The fact that \(\delta \left( t \right) \) enters the convergence rate statement suggests that one can benefit from this time rescaling function when it is at least nondecreasing on \(\left[ t_{0}, +\infty \right) \). We are, in fact, going to need this condition when showing trajectory convergence.

figure c

Assumption 2 entails the existence of \(C_{2} > 0\) such that

$$\begin{aligned} \frac{t \dot{\delta }(t)}{\delta (t)} + C_{2} \leqslant \frac{1 - 2\theta }{\theta } \quad \forall t \geqslant t_{0}. \end{aligned}$$
(4.1)

and therefore it follows further from the nondecreasing property of \(\delta \) that

$$\begin{aligned} 0 < C_{2} \delta (t_{0}) \leqslant C_{2} \delta (t) \leqslant (1 - 2\theta )\delta (t) - \theta t \dot{\delta }(t) \quad \forall t\geqslant t_{0} . \end{aligned}$$
(4.2)

Moreover, from (4.1), for every \(t\geqslant t_{0}\), we have

$$\begin{aligned} 0 < C_{2} \leqslant \frac{1 - 2\theta }{\theta } - \frac{t \dot{\delta }(t)}{\delta (t)} = \frac{\sigma (t)}{\delta (t)}, \end{aligned}$$

which gives

$$\begin{aligned} \delta (t) \leqslant \frac{\sigma (t)}{C_{2}} \quad \forall t\geqslant t_{0}. \end{aligned}$$
(4.3)

We can now formally state the existence and uniqueness result of the trajectory. The proof follows the same argument as in [17, Theorem 4.1], therefore we omit the details.

Theorem 4.1

In the setting of Assumption 2, for every choice of initial conditions

$$\begin{aligned} x(t_{0}) = x_{0}, \quad \lambda (t_{0}) = \lambda _{0}, \quad \dot{x}(t_{0}) = \dot{x}_{0}, \quad \text {and} \quad \dot{\lambda }(t_{0}) = \dot{\lambda }_{0} \end{aligned}$$

the system (2.6) has a unique global twice continuously differentiable solution \((x, \lambda ): \left[ t_{0}, +\infty \right) \rightarrow \mathcal {X} \times \mathcal {Y}\).

The additional Lipschitz continuity condition of \(\nabla f\) and the fact that \(\delta \) is nondecreasing give rise to the following two essential integrability statements.

Proposition 4.2

Let \((x, \lambda ): \left[ t_{0}, +\infty \right) \rightarrow \mathcal {X} \times \mathcal {Y}\) be a solution of (2.6) and \((x_{*}, \lambda _{*}) \in \mathbb {S}\). Then it holds

$$\begin{aligned} \int _{t_{0}}^{+\infty }t\delta (t) \left\| \nabla f(x(t)) - \nabla f(x_{*})\right\| ^{2} dt < +\infty \end{aligned}$$
(4.4)

and

$$\begin{aligned} \int _{t_{0}}^{+\infty } t\delta (t) \left\Vert Ax \left( t \right) - b \right\Vert ^{2} dt < +\infty . \end{aligned}$$
(4.5)

Proof

See “Proof of Proposition 4.2” in Appendix B. \(\square \)

Now, for a given primal-dual solution \((x_{*}, \lambda _{*}) \in \mathbb {S}\), we define the following mappings on \(\left[ t_{0}, +\infty \right) \)

$$\begin{aligned} W(t)&:= \delta (t) \bigl [\mathcal {L}_{\beta }\left( x(t), \lambda _{*} \right) - \mathcal {L}_{\beta }(x_{*}, \lambda (t))\bigr ] + \frac{1}{2}\left\Vert \left( \dot{x} \left( t \right) , \dot{\lambda } \left( t \right) \right) \right\Vert ^{2} \geqslant 0, \end{aligned}$$
(4.6)
$$\begin{aligned} \varphi (t)&:= \frac{1}{2}\left\| \bigl (x(t), \lambda (t)\bigr ) - (x_{*}, \lambda _{*})\right\| ^{2} \geqslant 0. \end{aligned}$$
(4.7)

The following are three technical lemmas that we will need in this section. Lemma 4.4 guarantess that the first condition of Opial’s Lemma is met.

Lemma 4.3

Let \((x, \lambda ): \left[ t_{0}, +\infty \right) \rightarrow \mathcal {X} \times \mathcal {Y}\) a solution of (2.6) and \((x_{*}, \lambda _{*}) \in \mathbb {S}\). The following inequality holds for every \(t\geqslant t_{0}\):

$$\begin{aligned} \ddot{\varphi }(t) + \frac{\alpha }{t} \dot{\varphi }(t) + \theta t \dot{W}(t) + \frac{\delta (t)}{2\ell }\left\Vert \nabla f \left( x \left( t \right) \right) - \nabla f \left( x_{*} \right) \right\Vert ^{2} + \frac{\beta \delta (t)}{2}\left\Vert Ax \left( t \right) - b \right\Vert ^{2} \leqslant 0. \end{aligned}$$
(4.8)

Proof

See “Proof of Lemma 4.3” in Appendix B. \(\square \)

Lemma 4.4

Let \((x, \lambda ): \left[ t_{0}, +\infty \right) \rightarrow \mathcal {X} \times \mathcal {Y}\) be a solution to (2.6) and \((x_{*}, \lambda _{*}) \in \mathbb {S}\). Then the positive part \([\dot{\varphi }]_{+}\) of \(\dot{\varphi }\) belongs to \(\mathbb {L}^{1}\left[ t_{0}, +\infty \right) \) and the limit \(\lim _{t \rightarrow +\infty } \varphi (t)\) exists.

Proof

For any \(t\geqslant t_{0}\), we multiply (4.8) by t and drop the last two norm squared terms to obtain

$$\begin{aligned} t \ddot{\varphi }(t) + \alpha \dot{\varphi }(t) + \theta t^{2} \dot{W}(t) \leqslant 0. \end{aligned}$$

Recall from (4.6) that for every \(t\geqslant t_{0}\) we have

$$\begin{aligned} tW(t) = t\delta (t) \bigl [\mathcal {L}_{\beta }\left( x(t), \lambda _{*} \right) - \mathcal {L}_{\beta }(x_{*}, \lambda (t))\bigr ] + \frac{t}{2}\left\Vert \left( \dot{x} \left( t \right) , \dot{\lambda } \left( t \right) \right) \right\Vert ^{2}. \end{aligned}$$
(4.9)

On the one hand, according to (3.12), the second summand of the previous expression belongs to \(\mathbb {L}^{1}\left[ t_{0}, +\infty \right) \). On the other hand, using (4.3) and (3.10), we assert that

$$\begin{aligned}{} & {} \int _{t_{0}}^{+\infty } t \delta (t) \bigl [\mathcal {L}_{\beta }\left( x(t), \lambda _{*} \right) - \mathcal {L}_{\beta }(x_{*}, \lambda (t))\bigr ] dt\\{} & {} \quad \leqslant \frac{1}{C_{2}} \int _{t_{0}}^{+\infty } t \sigma (t) \bigl [\mathcal {L}_{\beta }\left( x(t), \lambda _{*} \right) - \mathcal {L}_{\beta }(x_{*}, \lambda (t))\bigr ] dt < +\infty . \end{aligned}$$

Hence, the first summand of (4.9) also belongs to \( \mathbb {L}^{1}\left[ t_{0}, +\infty \right) \), which implies that the mapping \(t \mapsto tW(t)\) belongs to \(\mathbb {L}^{1}\left[ t_{0}, +\infty \right) \) as well. For achieving the desired conclusion, we make use of Lemma A.4 with \(\phi := \varphi \) and \(w:= \theta W\). \(\square \)

Lemma 4.5

Let \((x, \lambda ): \left[ t_{0}, +\infty \right) \rightarrow \mathcal {X} \times \mathcal {Y}\) be a solution to (2.6) and \((x_{*}, \lambda _{*}) \in \mathbb {S}\). The following inequality holds for every \(t\geqslant t_{0}\)

$$\begin{aligned}&\frac{\alpha }{t \delta (t)} \frac{d}{dt}\left\Vert \left( \dot{x} \left( t \right) , \dot{\lambda } \left( t \right) \right) \right\Vert ^{2} + 2\left\langle \ddot{x}(t) + \frac{\alpha }{t}\dot{x}(t), A^{*}(\lambda (t) - \lambda _{*})\right\rangle \\&\quad +\theta \frac{d}{dt}\Bigl (t \delta (t) \left\| A^{*}(\lambda (t) - \lambda _{*})\right\| ^{2}\Bigr ) + \bigl ((1 - \theta )\delta (t) - \theta t \dot{\delta }(t)\bigr ) \left\| A^{*}(\lambda (t) - \lambda _{*})\right\| ^{2} \\&\quad \leqslant \delta (t) \Bigl [ 2\Vert \nabla f(x(t)) - \nabla f(x_{*})\Vert ^{2} + \left( 2\beta ^{2} \Vert A\Vert ^{2} + 1\right) \left\Vert Ax \left( t \right) - b \right\Vert ^{2}\Bigr ]. \end{aligned}$$

Proof

See “Proof of Lemma 4.5” in Appendix B. \(\square \)

Lemma 4.6

Let \((x, \lambda ): [t_{0}, +\infty ) \rightarrow \mathcal {X} \times \mathcal {Y}\) be a solution to (2.6) and \((x_{*}, \lambda _{*}) \in \mathbb {S}\). Then, for every \(t\geqslant t_{0}\) it holds

$$\begin{aligned}&\theta t^{\alpha + 1} \delta (t) \left\Vert A^{*} \left( \lambda \left( t \right) - \lambda _{*} \right) \right\Vert ^{2} \nonumber \\&\quad \leqslant -t^{\alpha } \dot{\varphi }(t) + \int _{t_{0}}^{t} s^{\alpha } V(s) ds \nonumber \\&\qquad + \int _{t_{0}}^{t} s^{\alpha } \Bigl [ \bigl (\theta (\alpha + 1) - 1\bigr )\delta (t) + \theta s \dot{\delta }(s)\Bigr ] \left\Vert A^{*}(\lambda (s) - \lambda _{*}) \right\Vert ^{2} ds \nonumber \\&\qquad -2 t^{\alpha } \bigl \langle \dot{x}(t), A^{*}(\lambda (t) - \lambda _{*})\bigr \rangle + C_{5}, \end{aligned}$$
(4.10)

where, for \(s\geqslant t_{0}\),

$$\begin{aligned} V(s)&:= \theta (\alpha + 1) W(s) + \left( \frac{\alpha (\alpha - 1)}{t_{0}^{2} \delta (_{0})} + \left\Vert A \right\Vert \right) \bigl \Vert \bigl ( \dot{x}(s), \dot{\lambda }(s)\bigr )\bigr \Vert ^{2} \\&\quad + C_{3} \delta (s) \Vert \nabla f(x(s)) - \nabla f(x_{*})\Vert ^{2} + C_{4} \delta (s) \Vert Ax(s) - b\Vert ^{2}, \end{aligned}$$

for certain nonnegative constants \(C_{3}, C_{4}\) and \(C_{5}\).

Proof

See “Proof of Lemma 4.6” in Appendix B. \(\square \)

The following proposition provides us with the main integrability result that will be used for verifying the second condition of Opial’s Lemma.

Proposition 4.7

Let \((x, \lambda ): \left[ t_{0}, +\infty \right) \rightarrow \mathcal {X} \times \mathcal {Y}\) be a solution to (2.6) and \((x_{*}, \lambda _{*}) \in \mathbb {S}\). Then it holds

$$\begin{aligned} \int _{t_{0}}^{+\infty } t \delta (t) \left\Vert A^{*} \left( \lambda \left( t \right) - \lambda _{*} \right) \right\Vert ^{2} dt < +\infty . \end{aligned}$$
(4.11)

Proof

We divide (4.10) by \(t^{\alpha }\), thus obtaining

$$\begin{aligned} \theta t \delta (t) \left\Vert A^{*} \left( \lambda \left( t \right) - \lambda _{*} \right) \right\Vert ^{2}&\leqslant - \dot{\varphi }(t) + \frac{1}{t^{\alpha }}\int _{t_{0}}^{t} s^{\alpha } V(s) ds \\&\quad + \frac{1}{t^{\alpha }}\int _{t_{0}}^{t} s^{\alpha } \Bigl [ \bigl (\theta (\alpha + 1) - 1\bigr )\delta (s) + \theta s \dot{\delta }(s)\Bigr ] \\&\quad \left\Vert A^{*}(\lambda (s) - \lambda _{*}) \right\Vert ^{2} ds \\&\quad -2 \bigl \langle \dot{x}(t), A^{*}(\lambda (t) - \lambda _{*})\bigr \rangle + \frac{C_{5}}{t^{\alpha }}. \end{aligned}$$

Now, we integrate this inequality from \(t_{0}\) to r. We get

$$\begin{aligned}&\theta \int _{t_{0}}^{r} t \delta (t) \left\Vert A^{*} \left( \lambda \left( t \right) - \lambda _{*} \right) \right\Vert ^{2} dt \nonumber \\&\quad \leqslant \ \varphi (t_{0}) - \varphi (r) + \int _{t_{0}}^{r}\frac{1}{t^{\alpha }} \left( \int _{t_{0}}^{t} s^{\alpha } V(s) ds\right) dt \nonumber \\&\qquad + \int _{t_{0}}^{r} \frac{1}{t^{\alpha }} \left( \int _{t_{0}}^{t} s^{\alpha } \Bigl [ \bigl (\theta (\alpha + 1) - 1\bigr )\delta (s) + \theta s \dot{\delta }(s)\Bigr ] \left\Vert A^{*}(\lambda (s) - \lambda _{*}) \right\Vert ^{2} ds\right) dt \nonumber \\&\qquad - 2 \int _{t_{0}}^{r} \bigl \langle A\dot{x}(t), \lambda (t) - \lambda _{*}\bigr \rangle dt + C_{5} \int _{t_{0}}^{r} t^{\alpha } dt. \end{aligned}$$
(4.12)

We now recall some important facts. First of all, we have

$$\begin{aligned} \int _{t_{0}}^{r}\frac{1}{t^{\alpha }} dt \leqslant \frac{1}{(\alpha - 1)t_{0}^{\alpha - 1}}. \end{aligned}$$
(4.13)

In addition, according to Lemma A.2, it holds

$$\begin{aligned} \int _{t_{0}}^{r} \frac{1}{t^{\alpha }} \left( \int _{t_{0}}^{t} s^{\alpha } V(s) ds\right) dt \leqslant \frac{1}{\alpha - 1}\int _{t_{0}}^{r}t V(t) dt, \end{aligned}$$
(4.14)

and

$$\begin{aligned}&\int _{t_{0}}^{r} \frac{1}{t^{\alpha }} \left( \int _{t_{0}}^{t} s^{\alpha } \Bigl [ \bigl (\theta (\alpha + 1) - 1\bigr )\delta (s) + \theta s \dot{\delta }(s)\Bigr ] \left\Vert A^{*}(\lambda (s) - \lambda _{*}) \right\Vert ^{2} ds\right) dt \nonumber \\&\quad \leqslant \frac{1}{\alpha - 1}\int _{t_{0}}^{r} \Bigl [ \bigl (\theta (\alpha + 1) - 1\bigr )\delta (t) + \theta t \dot{\delta }(t)\Bigr ] \left\Vert A^{*}(\lambda (t) - \lambda _{*}) \right\Vert ^{2} dt, \end{aligned}$$
(4.15)

respectively.

Finally, integrating by parts leads to

$$\begin{aligned}&- \int _{t_{0}}^{r} \bigl \langle A \dot{x}(t), \lambda (t) - \lambda _{*}\bigr \rangle dt \nonumber \\&\quad = \ - \bigl \langle Ax(r) - b, \lambda (r) - \lambda _{*}\bigr \rangle + \bigl \langle Ax(t_{0}) - b, \lambda (t_{0}) - \lambda _{*}\bigr \rangle + \int _{t_{0}}^{r} \bigl \langle Ax(t) - b, \dot{\lambda }(t)\bigr \rangle dt \nonumber \\&\quad \leqslant \Vert Ax(r) - b\Vert \Vert \lambda (r) - \lambda _{*}\Vert + \left\Vert Ax(t_{0}) - b \right\Vert \Vert \lambda (t_{0}) - \lambda _{*}\Vert + \int _{t_{0}}^{r} \bigl \langle Ax(t) - b, \dot{\lambda }(t)\bigr \rangle dt \nonumber \\&\quad \leqslant \sup _{t\geqslant t_{0}} \{\left\Vert Ax \left( t \right) - b \right\Vert \left\Vert \lambda \left( t \right) - \lambda _{*} \right\Vert \} + \left\Vert Ax(t_{0}) - b \right\Vert \Vert \lambda (t_{0} - \lambda _{*})\Vert \nonumber \\&\qquad + \frac{1}{2}\int _{t_{0}}^{r} \bigl ( \left\Vert Ax \left( t \right) - b \right\Vert ^{2} + \left\Vert \dot{\lambda } \left( t \right) \right\Vert ^{2}\bigr ) dt. \end{aligned}$$
(4.16)

The supremum term is finite due to the boundedness of the trajectory. Now, by using the nonnegativity of \(\varphi \) and the facts (4.13), (4.14), (4.15) and (4.16) on (4.12), we come to

$$\begin{aligned}&\dfrac{\theta }{\alpha - 1} \int _{t_{0}}^{r}t \sigma (t) \left\Vert A^{*} \left( \lambda \left( t \right) - \lambda _{*} \right) \right\Vert ^{2} dt \nonumber \\&\quad = \int _{t_{0}}^{r} \left[ \theta \delta (t) - \frac{\bigl (\theta (\alpha + 1) - 1\bigr )\delta (t) + \theta t \dot{\delta }(t)}{\alpha - 1}\right] t \left\Vert A^{*} \left( \lambda \left( t \right) - \lambda _{*} \right) \right\Vert ^{2} dt \nonumber \\&\quad \leqslant \frac{1}{\alpha - 1}\int _{t_{0}}^{r} t V(t) dt + \int _{t_{0}}^{r} t \left( \left\Vert Ax \left( t \right) - b \right\Vert ^{2} + \left\Vert \dot{\lambda } \left( t \right) \right\Vert ^{2} \right) dt + C_{6}, \end{aligned}$$
(4.17)

where

$$\begin{aligned} C_{6}&:= \varphi (t_{0}) + 2 \sup _{t\geqslant t_{0}} \left\{ \left\Vert Ax \left( t \right) - b \right\Vert \left\Vert \lambda \left( t \right) - \lambda _{*} \right\Vert \right\} \\&\quad + 2 \left\Vert Ax(t_{0}) - b \right\Vert \Vert \lambda (t_{0} - \lambda _{*})\Vert + \frac{C_{5}}{(\alpha - 1)t_{0}^{\alpha - 1}}. \end{aligned}$$

According to (3.11) and (3.12) in Theorem 3.2, as well as Lemma 4.4, we know that the mappings \(t \mapsto t V(t)\) and \(t \mapsto t \left( \left\Vert Ax \left( t \right) - b \right\Vert ^{2} + \left\Vert \dot{\lambda } \left( t \right) \right\Vert \right) \) belong to \(\mathbb {L}^{1}\left[ t_{0}, +\infty \right) \). Therefore, by taking the limit as \(r \rightarrow +\infty \) in (4.17) we obtain

$$\begin{aligned} \int _{t_{0}}^{+\infty } t \sigma (t) \left\Vert A^{*} \left( \lambda \left( t \right) - \lambda _{*} \right) \right\Vert ^{2} dt < +\infty . \end{aligned}$$

Again, from (4.3) we conclude that

$$\begin{aligned} \int _{t_{0}}^{+\infty } t \delta (t) \left\Vert A^{*} \left( \lambda \left( t \right) - \lambda _{*} \right) \right\Vert ^{2} dt \leqslant \frac{1}{C_{2}} \int _{t_{0}}^{+\infty } t \sigma (t) \left\Vert A^{*} \left( \lambda \left( t \right) - \lambda _{*} \right) \right\Vert ^{2} dt < +\infty , \end{aligned}$$

which completes the proof. \(\square \)

The following result is the final step towards the second condition of Opial’s Lemma.

Theorem 4.8

Let \((x, \lambda ): \left[ t_{0}, +\infty \right) \rightarrow \mathcal {X} \times \mathcal {Y}\) be a solution to (2.6) and \((x_{*}, \lambda _{*}) \in \mathbb {S}\). Then it holds

$$\begin{aligned}{} & {} \left\Vert \nabla f \left( x \left( t \right) \right) - \nabla f \left( x_{*} \right) \right\Vert = o \left( \frac{1}{\sqrt{t} \root 4 \of {\delta (t)}}\right) \text { and } \nonumber \\{} & {} \left\Vert A^{*} \left( \lambda \left( t \right) - \lambda _{*} \right) \right\Vert = o \left( \frac{1}{\sqrt{t} \root 4 \of {\delta (t)}}\right) \text { as } t \rightarrow +\infty . \end{aligned}$$
(4.18)

Consequently,

$$\begin{aligned} \left\Vert \nabla _{x} \mathcal {L}\bigl ( x \left( t \right) , \lambda \left( t \right) \bigr ) \right\Vert = \left\Vert \nabla f \left( x \left( t \right) \right) + A^{*} \lambda \left( t \right) \right\Vert = o\left( \frac{1}{\sqrt{t} \root 4 \of {\delta (t)}}\right) \quad \text {as} \quad t\rightarrow +\infty , \end{aligned}$$

while, as seen earlier,

$$\begin{aligned} \left\Vert \nabla _{\lambda } \mathcal {L}\bigl ( x \left( t \right) , \lambda \left( t \right) \bigr ) \right\Vert = \left\Vert Ax \left( t \right) - b \right\Vert = \mathcal {O}\left( \frac{1}{t^{2} \delta (t)}\right) \quad \text {as} \quad t\rightarrow +\infty . \end{aligned}$$

Proof

We first show the gradient rate. For \(t\geqslant t_{0}\), it holds

$$\begin{aligned}&\frac{d}{dt}\Bigl (t \sqrt{\delta (t)} \left\Vert \nabla f \left( x \left( t \right) \right) - \nabla f \left( x_{*} \right) \right\Vert ^{2}\Bigr )\nonumber \\&\quad = \left( \sqrt{\delta (t)} + \frac{t\dot{\delta }(t)}{2\sqrt{\delta (t)}}\right) \left\Vert \nabla f \left( x \left( t \right) \right) - \nabla f \left( x_{*} \right) \right\Vert ^{2} \nonumber \\&\quad + 2 t \sqrt{\delta (t)} \left\langle \nabla f(x(t)) - \nabla f(x_{*}), \frac{d}{dt}\nabla f(x(t))\right\rangle . \end{aligned}$$
(4.19)

On the one hand, by Assumption 2, we can write

$$\begin{aligned} \left( \sqrt{\delta (t)} + \frac{t\dot{\delta }(t)}{2\sqrt{\delta (t)}}\right)= & {} \left( \sqrt{\delta (t)} + \frac{\sqrt{\delta (t)}}{2} \cdot \frac{t\dot{\delta }(t)}{\delta (t)}\right) \leqslant \left( 1 + \frac{1 - 2\theta }{2\theta }\right) \sqrt{\delta (t)} \nonumber \\ {}= & {} \frac{1}{2\theta } \sqrt{\delta (t)}. \end{aligned}$$
(4.20)

Since \(\delta \) is nondecreasing, for \(t \geqslant t_{0}\) we have \(\sqrt{\delta (t)} \geqslant \sqrt{\delta (t_{0})} > 0\). Set \(t_{1}:= \max \left\{ t_{0}, \frac{1}{\sqrt{t_{0}}}\right\} \). Therefore, for \(t \geqslant t_{1}\) it holds

$$\begin{aligned} \frac{1}{\sqrt{\delta (t)}} \leqslant \frac{1}{\sqrt{\delta (t_{0})}} = t_{1} \leqslant t \end{aligned}$$

and thus

$$\begin{aligned} \sqrt{\delta (t)} \leqslant t \delta (t). \end{aligned}$$
(4.21)

On the other hand, for every \(t \geqslant t_{1}\) we deduce

$$\begin{aligned}&2 t \sqrt{\delta (t)} \left\langle \nabla f(x(t)) - \nabla f(x_{*}), \frac{d}{dt}\nabla f(x(t))\right\rangle \nonumber \\&\quad = 2 t \left\langle \sqrt{\delta (t)} \bigl [\nabla f(x(t)) - \nabla f(x_{*})\bigr ], \frac{d}{dt}\nabla f(x(t))\right\rangle \nonumber \\&\quad \leqslant t\delta (t) \left\Vert \nabla f \left( x \left( t \right) \right) - \nabla f \left( x_{*} \right) \right\Vert ^{2} + t\left\| \frac{d}{dt}\nabla f(x(t))\right\| ^{2} \nonumber \\&\quad \leqslant t\delta (t) \left\Vert \nabla f \left( x \left( t \right) \right) - \nabla f \left( x_{*} \right) \right\Vert ^{2} + \ell ^{2} t \bigl \Vert \dot{x}(t)\bigr \Vert ^{2} , \end{aligned}$$
(4.22)

where the last inequality is a consequence of the \(\ell \)-Lipschitz continuity of \(\nabla f\). By combining (4.20), (4.21) and (4.22), from (4.19) we assert that for every \(t\geqslant t_{1}\)

$$\begin{aligned}{} & {} \frac{d}{dt}\Bigl (t \sqrt{\delta (t)} \left\Vert \nabla f \left( x \left( t \right) \right) - \nabla f \left( x_{*} \right) \right\Vert ^{2}\Bigr ) \\{} & {} \quad \leqslant \left( 1 + \frac{1}{2\theta } \right) t \delta (t) \left\Vert \nabla f \left( x \left( t \right) \right) - \nabla f \left( x_{*} \right) \right\Vert ^{2} + \ell ^{2} t \bigl \Vert \dot{x}(t)\bigr \Vert ^{2}. \end{aligned}$$

The right hand side of the previous inequality belongs to \(\mathbb {L}^{1} [t_{1}, +\infty )\), according to (3.12) and (4.4). Since \(\delta \) is nondecreasing, for every \(t \geqslant t_{1}\) we have

$$\begin{aligned} \sqrt{\delta (t)} = \sqrt{\delta (t)} \cdot \frac{\sqrt{\delta (t)}}{\sqrt{\delta (t)}} \leqslant \frac{\delta (t)}{\sqrt{\delta (t_{0})}}, \end{aligned}$$

and thus

$$\begin{aligned}{} & {} \int _{t_{1}}^{+\infty } t \sqrt{\delta (t)} \left\Vert \nabla f \left( x \left( t \right) \right) - \nabla f \left( x_{*} \right) \right\Vert ^{2} dt \nonumber \\{} & {} \quad \leqslant \frac{1}{\sqrt{\delta (t_{0})}} \int _{t_{1}}^{+\infty } t \delta (t) \left\Vert \nabla f \left( x \left( t \right) \right) - \nabla f \left( x_{*} \right) \right\Vert ^{2} dt < +\infty . \end{aligned}$$
(4.23)

This means that the function being differentiated also belongs to \(\mathbb {L}^{1} [t_{1}, +\infty )\). Therefore, Lemma A.3 gives us

$$\begin{aligned} t \sqrt{\delta (t)} \left\Vert \nabla f \left( x \left( t \right) \right) - \nabla f \left( x_{*} \right) \right\Vert ^{2} \rightarrow 0 \quad \text {as} \quad t\rightarrow +\infty . \end{aligned}$$

Proceeding in the exact same way, for every \(t\geqslant t_{1}\) we have

$$\begin{aligned}&\frac{d}{dt} \Bigl ( t \sqrt{\delta (t)} \bigl \Vert A^{*}(\lambda (t) - \lambda _{*})\bigr \Vert ^{2}\Bigr ) \\&\quad = \left( \sqrt{\delta (t)} + \frac{t \dot{\delta }(t)}{2\sqrt{\delta (t)}}\right) \bigl \Vert A^{*}(\lambda (t) - \lambda _{*})\bigr \Vert ^{2} + 2 t \sqrt{\delta (t)} \bigl \langle A A^{*}(\lambda (t) - \lambda _{*}), \dot{\lambda }(t)\bigr \rangle \\&\quad \leqslant \left( \frac{1}{2\theta } + \Vert A\Vert ^{2} \right) t \delta (t) \bigl \Vert A^{*}(\lambda (t) - \lambda _{*})\bigr \Vert ^{2} + t \left\Vert \dot{\lambda } \left( t \right) \right\Vert ^{2}. \end{aligned}$$

According to (3.12) and (4.11), the right hand side of the previous inequality belongs to \(\mathbb {L}^{1}[t_{1}, +\infty )\). Arguing as in (4.23), we deduce that the function being differentiated also belongs to \(\mathbb {L}^{1}[t_{1}, +\infty )\). Again applying Lemma A.3, we come to

$$\begin{aligned} t \sqrt{\delta (t)} \bigl \Vert A^{*}(\lambda (t) - \lambda _{*})\bigr \Vert ^{2} \rightarrow 0 \quad \text {as} \quad t\rightarrow +\infty . \end{aligned}$$

Finally, recalling that \(A^{*}\lambda _{*} = -\nabla f(x_{*})\), we deduce from the triangle inequality that

$$\begin{aligned} \left\Vert \nabla _{x} \mathcal {L}\bigl ( x \left( t \right) , \lambda \left( t \right) \bigr ) \right\Vert&= \left\Vert \nabla f \left( x \left( t \right) \right) + A^{*} \lambda \left( t \right) \right\Vert \\&\leqslant \left\Vert \nabla f \left( x \left( t \right) \right) - \nabla f \left( x_{*} \right) \right\Vert + \bigl \Vert A^{*}(\lambda (t) - \lambda _{*})\bigr \Vert \nonumber \\&= o\left( \frac{1}{\sqrt{t} \root 4 \of {\delta (t)}}\right) \quad \text {as} \quad t\rightarrow +\infty , \end{aligned}$$

and the third claim follows. \(\square \)

Remark 4.9

The previous theorem also has its own interest. It tells us that the time rescaling parameter also plays a role in accelerating the rates of convergence for \(\left\Vert \nabla f \left( x \left( t \right) \right) - \nabla f \left( x_{*} \right) \right\Vert \) and \(\Vert A^{*}(\lambda (t) - \lambda _{*})\Vert \) as \(t\rightarrow +\infty \). Moreover, we deduce from (4.18) that the mapping \((x, \lambda ) \mapsto (\nabla f(x), A^{*}\lambda )\) is constant along \(\mathbb {S}\), as reported in [17, Proposition A.4].

We now come to the final step and show weak convergence of the trajectories of (2.6) to elements of \(\mathbb {S}\).

Theorem 4.10

Let \((x, \lambda ): \left[ t_{0}, +\infty \right) \rightarrow \mathcal {X} \times \mathcal {Y}\) be a solution to (2.6) and \(\left( x_{*}, \lambda _{*} \right) \in \mathbb {S}\). Then \(\bigl (x(t), \lambda (t)\bigr )\) converges weakly to a primal-dual solution of (1.1) as \(t\rightarrow +\infty \).

Proof

For proving this theorem, we make use of Opial’s Lemma (see Lemma A.5). Lemma 4.4 tells us that \(\lim _{t\rightarrow +\infty }\left\| \bigl (x(t), \lambda (t)\bigr ) - (x_{*}, \lambda _{*})\right\| \) exists for every \((x_{*}, \lambda _{*}) \in \mathbb {S}\), which proves condition (i) of Opial’s Lemma. Now let \(\bigl (\tilde{x}, \tilde{\lambda }\bigr )\) be any weak sequential cluster point of \(\bigl ( x(t), \lambda (t)\bigr )\) as \(t\rightarrow +\infty \), which means there exists a strictly increasing sequence \((t_{n})_{n\in \mathbb {N}} \subseteq [t_{0}, +\infty )\) such that

$$\begin{aligned} \bigl ( x(t_{n}), \lambda (t_{n})\bigr ) \rightharpoonup \bigl (\tilde{x}, \tilde{\lambda }\bigr ) \quad \text {as} \quad n \rightarrow +\infty . \end{aligned}$$

We want to show the remaining condition of Opial’s Lemma, which asks us to check that \(\bigl (\tilde{x}, \tilde{\lambda }\bigr ) \in \mathbb {S}\). In other words, we must show that

$$\begin{aligned} \mathcal {L}\bigl (\tilde{x}, \lambda \bigr ) \leqslant \mathcal {L}\bigl (\tilde{x}, \tilde{\lambda }\bigr ) \leqslant \mathcal {L}\bigl (x, \tilde{\lambda }\bigr ) \quad \forall (x, \lambda ) \in \mathcal {X} \times \mathcal {Y}. \end{aligned}$$
(4.24)

Let \((x, \lambda ) \in \mathcal {X} \times \mathcal {Y}\) and \((x_{*}, \lambda _{*}) \in \mathbb {S}\) be fixed. Notice that the functions

$$\begin{aligned} f(\cdot ) + \bigl \langle \tilde{\lambda }, A(\cdot ) - Ax\bigr \rangle :\mathcal {X} \rightarrow \mathbb {R} \quad \text {and} \quad \bigl \langle \cdot , b - Ax\bigr \rangle :\mathcal {Y} \rightarrow \mathbb {R} \end{aligned}$$

are convex and continuous, therefore they are lower semicontinuous. According to a known result (see, for example, [14, Theorem 9.1]), they are also weakly lower semicontinuous. Therefore, we can derive that

$$\begin{aligned} \mathcal {L}\bigl (\tilde{x}, \tilde{\lambda }\bigr ) - \mathcal {L}\bigl (x, \tilde{\lambda }\bigr )&= f\bigl (\tilde{x}\bigr ) + \bigl \langle \tilde{\lambda }, A \tilde{x} - Ax\bigr \rangle - f(x) \\&\leqslant \liminf _{n\rightarrow +\infty } \Bigl [f(x(t_{n})) + \bigl \langle \tilde{\lambda }, Ax_{n} - Ax\bigr \rangle \Bigr ] - f(x) \\&= f(x_{*}) + \bigl \langle \tilde{\lambda }, b - Ax\bigr \rangle - f(x) \\&\leqslant f(x_{*}) - f(x) + \liminf _{n\rightarrow +\infty } \bigr \langle A^{*} \lambda (t_{n}), x_{*} - x\bigr \rangle \\&= f(x_{*}) - \bigl ( f(x) + \langle \lambda _{*}, Ax - b\rangle \bigr ) \\&= \mathcal {L}(x_{*}, \lambda _{*}) - \mathcal {L}(x, \lambda _{*}) \leqslant 0, \end{aligned}$$

where in the second and third equalities we used the fact that, as \(n\rightarrow +\infty \), we have \(f(x(t_{n})) \rightarrow f(x_{*})\) and \(Ax(t_{n}) \rightarrow b\) (Theorem 3.3), and \(A^{*}\lambda _{n} \rightarrow A^{*} \lambda _{*}\) (Theorem 4.8). Similarly, the weak lower semicontinuity of the function \(\bigl \langle \lambda - \tilde{\lambda }, A(\cdot ) - b\bigr \rangle :\mathcal {X} \rightarrow \mathbb {R}\) yields

$$\begin{aligned} \mathcal {L}\bigl (\tilde{x}, \lambda \bigr ) - \mathcal {L}\bigl (\tilde{x}, \tilde{\lambda }\bigr ) = \bigl \langle \lambda - \tilde{\lambda }, A\tilde{x} - b\bigr \rangle \leqslant \liminf _{n\rightarrow +\infty } \bigl \langle \lambda - \tilde{\lambda }, Ax(t_{n}) - b\bigr \rangle = 0, \end{aligned}$$

We have thus showed (4.24) and the proof is concluded. \(\square \)

5 Numerical Experiments

We will illustate the theoretical results by two numerical examples, with \(\mathcal {X} = \mathbb {R}^{4}\) and \(\mathcal {Y} = \mathbb {R}^{2}\). We will address two minimization problems with linear constraints; one with a strongly convex objective function and another with a convex objective function which is not strongly convex. In both cases, the linear constraints are dictated by

$$\begin{aligned} A = \begin{bmatrix} 1 &{} -1 &{} -1 &{} 0 \\ 0 &{} 1 &{} 0 &{} -1 \end{bmatrix} \qquad \text {and} \qquad b = \begin{bmatrix} 0 \\ 0 \end{bmatrix}. \end{aligned}$$

Example 5.1

Consider the minimization problem

$$\begin{aligned} \begin{array}{rl} \min &{} f(x_{1}, x_{2}, x_{3}, x_{4}):= (x_{1} - 1)^{2} + (x_{2} - 1)^{2} + x_{3}^{2} + x_{4}^{2} \\ \text {subject to} &{} x_{1} - x_{2} - x_{3} = 0 \\ &{} x_{2} - x_{4} = 0. \end{array} \end{aligned}$$

The optimality conditions can be calculated and lead to the following primal-dual solution pair

$$\begin{aligned} x_{*} = \begin{bmatrix} 0.8 \\ 0.6 \\ 0.2 \\ 0.6 \end{bmatrix} \qquad \text {and} \qquad \lambda _{*} = \begin{bmatrix} 0.4 \\ 1.2 \end{bmatrix}. \end{aligned}$$

Example 5.2

Consider the minimization problem

$$\begin{aligned} \begin{array}{rl} \min &{} f(x_{1}, x_{2}, x_{3}, x_{4}):= \log \left( 1 + e^{-x_{1} - x_{2}}\right) + x_{3}^{2} + x_{4}^{2} \\ \text {subject to} &{} x_{1} - x_{2} - x_{3} = 0 \\ &{} x_{2} - x_{4} = 0. \end{array} \end{aligned}$$

This problem is similar to the regularized logistic regression frequently used in machine learning. We cannot explicitly calculate the optimality conditions as in the previous case; instead, we use the last solution in the numerical experiment as the approximate solution.

To comply with Assumption 2, we choose \(t_{0} > 0\), \(\alpha = 8\), \(\beta = 10\), \(\theta = \frac{1}{6}\), and we test four different choices for the rescaling parameter: \(\delta (t) = 1\) (i.e., the (PD-AVD) dynamics in [17, 45]), \(\delta (t) = t\), \(\delta (t) = t^{2}\) and \(\delta (t) = t^{3}\). In both examples, the initial conditions are

$$\begin{aligned} x(t_{0}) = \begin{bmatrix} 0.5 \\ 0.5 \\ 0.5 \\ 0.5 \end{bmatrix}, \quad \lambda (t_{0}) = \begin{bmatrix} 0.2 \\ 0.2 \end{bmatrix}, \quad \dot{x}(t_{0}) = \begin{bmatrix} 0.5 \\ 0.5 \\ 0.5 \\ 0.5 \end{bmatrix} \quad \text {and} \quad \dot{\lambda }(t_{0}) = \begin{bmatrix} 0.5 \\ 0.5 \end{bmatrix}. \end{aligned}$$

For each choice of \(\delta \), we plot, using a logarithmic scale, the primal-dual gap \(\mathcal {L}\bigl (x(t), \lambda _{*}\bigr ) - \mathcal {L}\bigl (x_{*}, \lambda (t)\bigr )\), the feasibility measure \(\left\Vert A x \left( t \right) - b \right\Vert \) and the functional values \(\left|f \left( x \left( t \right) \right) - f_{*} \right|\), to highlight the theoretical result in Theorem 3.3. We also illustrate the findings from Theorem 4.8, namely, we plot the quantities \(\left\Vert \nabla f \left( x \left( t \right) \right) - \nabla f \left( x_{*} \right) \right\Vert \) and \(\left\Vert A^{*} \left( \lambda \left( t \right) - \lambda _{*} \right) \right\Vert \), as well as the velocity \(\Vert ( \dot{x} (t), \dot{\lambda } (t) ) \Vert \).

Figures 1 and 2 display these plots for Examples 5.1 and 5.2, respectively. As predicted by the theory, choosing faster-growing time rescaling parameters yields better convergence rates. This is not the case for the velocities.

Fig. 1
figure 1

The function \(\delta \left( t \right) \) influences convergence behaviour in Example 5.1

Full size image
Fig. 2
figure 2

The function \(\delta \left( t \right) \) influences convergence behaviour in Example 5.2

Full size image

Next we use Example 5.2 to compare the convergence behaviour of our system (2.6) with the one where the asymptotically vanishing damping term is chosen to be \(\frac{\alpha }{t^{r}}\), for \(r \in [0,1]\). Notice that \(r = 1\) gives our system (2.6). When \(r = 0\), in the setting of [30, Theorem 2.2], we know that the primal-dual gap exhibits a convergence rate of \(\mathcal {O}\left( \frac{1}{t \delta (t)}\right) \) as \(t\rightarrow +\infty \). This is illustrated in Fig. 3, were we plotted the combinations \(\left( \delta (t) = t; r = 0 \right) \), \(\left( \delta (t) = t; r = 1 \right) \), \(\left( \delta (t) = t^{2}; r = 0 \right) \), and \(\left( \delta (t) = t^{2}; r = 1 \right) \). In particular, observe that the rate predicted by [30, Theorem 2.2] for the primal-dual gap for the case \(\left( \delta (t) = t^{2}; r = 0 \right) \) reads \(\mathcal {O}\left( \frac{1}{t\cdot t^{2}}\right) \), while the rate predicted by our Theorem 3.3 for the case \(\left( \delta (t) = t; r = 1 \right) \) reads \(\mathcal {O}\left( \frac{1}{t^{2}\cdot t}\right) \). It is no surprise then to see the combinations \(\left( \delta (t) = t^{2}; r = 0 \right) \) and \(\left( \delta (t) = t; r = 1 \right) \) exhibiting similar convergence behaviour in Fig. 3.

Fig. 3
figure 3

The function \(\delta (t)\), as well as the parameter r, influence convergence behaviour in Example 5.2

Full size image
Fig. 4
figure 4

The parameter r influences convergence behaviour in Example 5.2

Full size image

For better understanding, we run Example 5.2 once more to show the plots which result from fixing the time rescaling parameter \(\delta (t) = t\) and varying \(r \in \{0, 0.25, 0.5, 0.75, 1\}\). Notice how the convergence improves as r approaches 1. As \(t\rightarrow +\infty \), [30, Theorem 2.7] predicts convergence rates of \(\mathcal {O}\left( \frac{1}{t^{\tau } \delta (t)}\right) \) for the primal-dual gap and of \(\mathcal {O}\left( \frac{1}{t^{\tau /2}}\right) \) for the velocities, which is reflected in our plots.