Tikhonov regularization of a second order dynamical system with Hessian driven damping

We investigate the asymptotic properties of the trajectories generated by a second-order dynamical system with Hessian driven damping and a Tikhonov regularization term in connection with the minimization of a smooth convex function in Hilbert spaces. We obtain fast convergence results for the function values along the trajectories. The Tikhonov regularization term enables the derivation of strong convergence results of the trajectory to the minimizer of the objective function of minimum norm.


Introduction
The paper of Su et al. [20] was the starting point of intensive research of second order dynamical systems with an asymptotically vanishing damping term of the form where g : H −→ R is a convex and continuously Fréchet differentiable function defined on a real Hilbert space H fulfilling argming = ∅. The aim is to approach by the trajectories generated by this system the solution set of the optimization problem min x∈H g(x).
The convergence rate of the objective function along the trajectory is in case α > 3 of while in case α = 3 it is of where min g ∈ R denotes the minimal value of g. Also in view of this fact, system (1) is seen as a continuous version of the celebrated Nesterov accelerated gradient scheme (see [16]). In what concerns the asymptotic properties of the generated trajectories, weak convergence to a minimizer of g as the time goes to infinity has been proved by Attouch et al. [7] (see also [6]) for α > 3. Without any further geometrical assumption on g, the convergence of the trajectories in the case α ≤ 3 is still an open problem. Second order dynamical systems with a geometrical Hessian driven damping term have aroused the interest of the researchers, due to both their applications in optimization and mechanics and their natural relations to Newton and Levenberg-Marquardt iterative methods (see [2]). Furthermore, it has been observed for some classes of optimization problems that a geometrical damping term governed by the Hessian can induce a stabilization of the trajectories. In [11] the dynamical system with Hessian driven damping term where α ≥ 3 and β > 0, has been investigated in relation with the optimization problem (2). Fast convergence rates for the values and the gradient of the objective function along the trajectories are obtained and the weak convergence of the trajectories to a minimizer of g is shown. We would also like to mention that iterative schemes which result via (symplectic) discretizations of dynamical systems with Hessian driven damping terms have been recently formulated and investigated from the point of view of their convergence properties in [5,18,19]. Another development having as a starting point (1) is the investigation of dynamical systems involving a Tikhonov regularization term. Attouch, Chbani and Riahi investigated in this context in [8] the system where α ≥ 3 and : [t 0 , +∞) −→ [0, +∞). One of the main benefits of considering such a regularized dynamical system is that it generates trajectories which converge strongly to the minimum norm solution of (2). Besides that, in [8] it was proved that the fast convergence rate of the objective function values along the trajectories remains unaltered. For more insights into the role played by the Tikhonov regularization for optimization problems and, more general, for monotone inclusion problems, we refer the reader to [3,4,9,15]. This being said, it is natural to investigate a second order dynamical system which combines a Hessian driven damping and a Tikhonov regularization term and to examine if it inherits the properties of the dynamical systems (3) and (4). This is the aim of the manuscript, namely the analysis in the framework of the general assumption stated below of the dynamical system where α ≥ 3 and β ≥ 0, and u 0 , v 0 ∈ H. General assumption: • g : H −→ R is a convex and twice Fréchet differentiable function with Lipschitz continuous gradient on bounded sets and argming = ∅; • : [t 0 , +∞) −→ [0, +∞) is a nonincreasing function of class C 1 fulfilling lim t−→+∞ (t) = 0.
The fact that the starting time t 0 is taken as strictly greater than zero comes from the singularity of the damping coefficient α t . This is not a limitation of the generality of the proposed approach, since we will focus on the asymptotic behaviour of the generated trajectories. Notice that if H is finite-dimensional, then the Lipschitz continuity of ∇g on bounded sets follows from the continuity of ∇ 2 g.
To which extent the Tikhonov regularization does influence the convergence behaviour of the trajectories generated by (5) can be seen even when minimizing a one dimensional function. Consider the convex and twice continuously differentiable function It holds that argming = [−1, 1] and x * = 0 is its minimum norm solution. In the second column of Fig. 1 we can see the behaviour of the trajectories generated   Fig. 1 First column: the trajectories of the dynamical system with Tikhonov regularization (t) = t −γ are approaching the minimum norm solution x * = 0. Second column: the trajectories of the dynamical system without Tikhonov regularization the trajectory are approaching the optimal solution 1 by the dynamical system without Tikhonov regularization (which corresponds to the case when is identically 0) for β = 1 and α = 3 and, respectively, α = 4. In both cases the trajectories are approaching the optimal solution 1, which is a minimizer of g, however, not the minimum norm solution.
In the first column of Fig. 1 we can see the behaviour of the trajectories generated by the dynamical system with Tikhonov parametrizations of the form t → (t) = t −γ , for different values of γ ∈ (1, 2), which is in accordance to the conditions in Theorem 4.4, β = 1 and α = 3 and, respectively, α = 4. The trajectories are approaching the minimum norm solution x * = 0.
The organization of the paper is as follows. We start the analysis of the dynamical system (5) by proving the existence and uniqueness of a global C 2 -solution. In the third section we provide two different settings for the Tikhonov parametrization t → (t) in both of which g(x(t)) converges to min g, the minimal value of g, with a convergence rate of O 1 t 2 for α = 3 and of o 1 t 2 for α > 3. The proof relies on Lyapunov theory; the choice of the right energy functional plays a decisive role in this context. Weak convergence of the trajectory is also derived for α > 3. In the last section we focus on the proof of strong convergence to a minimum norm solution: firstly, in a general setting, for the ergodic trajectory, and, secondly, in a slightly restrictive setting, for the trajectory x(t) itself.

Existence and uniqueness
In this section we will prove the existence and uniqueness of a global C 2 -solution of the dynamical system (5). The proof of the existence and uniqueness theorem is based on the idea to reformulate (5) as a particular first order dynamical system in a suitably chosen product space (see also [11]).
Assume now that β > 0. We notice that x : [t 0 , +∞) −→ H is a solution of the dynamical system (5), that is We define F : and write (7) as Since ∇g is Lipschitz continuous on bounded sets and continuously differentiable, the local existence and uniqueness theorem (see [17, Theorems 46.2 and 46.3]) guarantees the existence of a unique solution (x, y) of (8) defined on a maximum intervall [t 0 , T max ), where t 0 < T max ≤ +∞. Furthermore, either T max = +∞ or lim t→T max x(t) + y(t) = +∞. We will prove that T max = +∞, which will imply that x is the unique global C 2 -solution of (5). Consider the energy functional (see [10]) By using (5) we get and, since is nonincreasing and ∇ 2 g(x(t)) is positive semidefinite, we obtain that Consequently, E is nonincreasing, hence From the fact that g is bounded from below we obtain thatẋ is bounded on [t 0 , T max ).
, there exists lim t−→T max x(t), which shows that x is bounded on [t 0 , T max ). Sinceẋ(t) + β∇g(x(t)) = y(t) for all t ∈ [t 0 , T max ) and ∇g is Lipschitz continuous on bounded sets, it yields that y is also bounded on [t 0 , T max ). Hence lim t→T max x(t) + y(t) cannot be +∞, thus T max = +∞, which completes the proof.

Asymptotic analysis
In this section we will show to which extent different assumptions we impose to the Tikhonov parametrization t → (t) influence the asymptotic behaviour of the trajectory x generated by the dynamical system (5). In particular, we are looking at the convergence of the function g along the trajectory and the weak convergence of the trajectory.
We recall that the asymptotic analysis of the system (5) is carried out in the framework of the general assumptions stated in the introduction.
We start with a result which provides a setting that guarantees the convergence of g(x(t)) to min g as t → +∞. Theorem 3.1 Let x be the unique global C 2 -solution of (5). Assume that one of the following conditions is fulfilled: t dt < +∞ and there exist a > 1 and t 1 ≥ t 0 such thaṫ If α ≥ 3, then lim t→+∞ g(x(t)) = min g.
Proof Let be x * ∈ argming and 2 ≤ b ≤ α − 1 be fixed. We introduce the following For every t ≥ t 0 it holdṡ Now, by using (5), we get for every t ≥ t 0 By taking x := x(t) and y := x * we get for every t ≥ t 0 From (10), (11) and (12) it follows that for every t ≥ t 0 it holdṡ At this point we treat the situations α > 3 and α = 3 separately.
The case α > 3 and 2 < b < α − 1. We will carry out the analysis by addressing the settings provided by the conditions (a) and (b) separately.
Condition (a) holds: Assuming that condition (a) holds, there exist a > 1 and Using that (13) leads to the following estimatė which holds for every t ≥ t 1 .
Since a > 1 and b > 2, we notice that for every t ≥ t 1 it holds On the other hand, we have that We define t 2 : According to (15), it holds for every t ≥ t 2 Condition (b) holds: Assuming now that condition (b) holds, there exist a > 0 and Further, the monotonicity of ∇g and the fact that ∇g(x * ) = 0 implies that (13) leads to the following estimatė ∇g(x(t)) 2 (18) for every t ≥ t 1 .
Since b > 2, we have that for every t ≥ t 1 it holds On the other hand, since holds for every t ≥ t 1 , it follows that We recall that We define t 2 : According to (18), it holds for every t ≥ t 2 From now on we will treat the two cases together. According to (16), in case (a), and to (20), in case (b), we obtaiṅ , in case (a), and On the other hand, The case α = 3 and b = 2. In this case the energy functional reads for every t ≥ t 0 . We will address again the settings provided by the conditions (a) and (b) separately.
Condition (a) holds: Relation (15) becomeṡ for every t ≥ t 3 . After multiplication with (t − β), it yields for every t ≥ t 3 . Repeating the above steps for the inequality (23) we obtain From now on we will treat the two cases together. According to (22), in case (a), and to (24), in case (b), we obtain , in case (a), and l := 2+aβ 2 and t 3 = max(t 1 , 4β) in case (b). By integrating the latter inequality on an interval [t 3 , T ], where T ≥ t 3 is arbitrarily chosen, we obtain On the other hand, for every t ≥ t 0 , hence, for every T ≥ max(β, t 3 ) = t 3 we get

Remark 3.2
One can easily notice that, in case β > 0, the fact that there exist a > 1 and t dt < +∞. The next theorem shows that, by strengthening the integrability condition +∞ t 0 (t) t dt < +∞ (which is actually required in both settings (a) and (b) of Theorem 3.1), a rate of O(1/t 2 ) ca be guaranteed for the convergence of g(x(t)) to min g. (5). Assume that (b) there exist a > 0 and t 1 ≥ t 0 such that

Theorem 3.3 Let x be the unique global C 2 -solution of
In addition, if α > 3, then the trajectory x is bounded and for every arbitrary x * ∈ argming.
Proof Let be x * argming and 2 ≤ b ≤ α − 1 fixed. We will use the energy functional introduced in the proof of the previous theorem and some of the estimate we derived for it. We will treat again the situations α > 3 and α = 3 separately. The case α > 3 and 2 < b < α − 1. As we already noticed in the proof of Theorem 3.1, according to (16), in case (a), and to (20), in case (b), we havė , in case (a), and l := b + aβ and t 2 = which is true for every t ≥ t 0 , leads to By integrating relation (16), in case (a), and relation (20), in case (b), on an interval [t 2 , s], where s ≥ t 3 is arbitrarily chosen, and by letting afterwards s converge to +∞, we obtain The boundedness of the trajectory and the condition on the Tikhonov parametrization guarantee that The case α = 3 and b = 2. As we already noticed in the proof of Theorem 3.1, according to (22), in case (a), and to (24), in case (b), we obtain , in case (a), and l = 2+aβ 2 and t 3 = max(t 1 , 4β) in case (b). Since is bounded from below, from Lemma A.2 it follows that the limit lim t−→+∞ t t−β E 2 (t) exists. Consequently, the limit lim t−→+∞ E 2 (t) also exists and t → E 2 (t) is bounded. This implies that there exist K > 0 and t ≥ t 0 such that The next result shows that the statements of Theorem 3.3 can be strengthened in case α > 3. (b) there exist a > 0 and t 1 ≥ t 0 such that

Theorem 3.4 Let x be the unique global C 2 -solution of (5). Assume that
Let be an arbitrary x * ∈ argming. If α > 3, then and the limits Proof Since α > 3 we can choose 2 < b < α − 1. From (10) and (11) we have thaṫ We will address the settings provided by the conditions (a) and (b) separately. Condition (a) holds: In this case we estimate −β (t)t 2 ∇g(x(t)), x(t) just as in (14) and from (25) we obtaiṅ We define t 2 := max β, t 1 , βa a−1 . By using condition (a), neglecting the nonpositive terms and afterwards integrating on the interval [t 2 , t], with arbitrary t ≥ t 2 , we obtain For every s ≥ t 2 , by the monotonicity of ∇g, we have ∇g(x(s)), By letting in (27) s converge to +∞ and by taking into account that, according to Theorem 3.3, Condition (b) holds: In this case we estimate −β (t)t 2 ∇g(x(t)), x(t) just as in (17) and from (25) we obtaiṅ We define t 2 := max (4β, t 1 ) . According to (19) we have that β 2 t − βt 2 + β (t)t 3 2a 1 ≤ 0 for every t ≥ t 2 . By using condition (b), neglecting the nonpositive terms and afterwards integrating on the interval [t 2 , t], with arbitrary t ≥ t 2 , we obtain From here, by using the similar arguments as for the case (a), we obtain (28).
According to Theorem 3.3, the limits also exists. For every t ≥ t 0 we define From (28) and the fact that k(t) has a limit whenever t −→ +∞, we obtain that (α − 1)q(t) + tq(t) has a limit when t −→ +∞. According to Lemma 4.6, q(t) has a limit when t −→ +∞. By using (28) again we obtain that the limit exists and, consequently, the limit also exists. On the other hand, we notice that for every t ≥ t 0 the energy functional can be written as Since the limits and notice that for sufficiently large t it holds According to Theorem 3.3 the right hand side of the above inequality is of class Hence, .
and the proof is complete.
Working in the hypotheses of Theorem 3.4 we can prove also the weak convergence of the trajectories generated by (5) to a minimizer of the objective function g. (5). Assume that (b) there exist a > 0 and t 1 ≥ t 0 such that (t) ≤ a t for every t ≥ t 1 .

If α > 3, then x(t) converges weakly to an element in argming as t −→ +∞.
Proof We will to apply the continuous version of the Opial Lemma (Lemma A.3) for S = argming. According to Theorem 3.4, the limit exists for every x * ∈ argming. Further, let x ∈ H be a weak sequential limit point of x(t). This means that there exists a sequence (t n ) n∈N ⊆ [t 0 , +∞) such that lim n−→∞ t n = +∞ and x(t n ) converges weakly to x as n −→ ∞. Since g is weakly lower semicontinuous, we have that On the other hand, according to Theorem 3.3, lim t−→+∞ g(x(t)) = min g, consequently one has g(x) ≤ min g, which shows that x ∈ argming.
The convergence of the trajectory is a consequence of Lemma A.3.
Remark 3. 6 We proved in this section that the convergence rate of o 1 t 2 for g(x(t)), the converge rate of o 1 t for ẋ(t) + β∇g(x(t)) and the weak convergence of the trajectory to a minimizer of g that have been obtained in [11] for the dynamical system with Hessian driven damping (3) are preserved when this system is enhanced with a Tikhonov regularization term. In addition, in the case when the Hessian driven damping term is removed, which is the case when β = 0, we recover the results provided in [8] for the dynamical system (4) with Tikhonov regularization term. In this setting, we have to assume in Theorem 3.1 just that +∞ t 0 (t) t dt < +∞, and in the theorems 3.3 -3.5 just that

Strong convergence to the minimum norm solution
In this section we will continue the investigations we did at the end of Section 3, by working in the same setting, on the behaviour of the trajectory of the dynamical system (5) by concentrating on strong convergence. In particular, we will provide conditions on the Tikhonov parametrization t → (t) which will guarantee that the trajectory converges to a minimum norm solution of g, which is the element of minimum norm of the nonempty convex closed set argming. We start with the following result. (5). For x * ∈ argming we introduce the function

Lemma 4.1 Let x be the unique global C 2 -solution of
If α > 0 and β ≥ 0, then In addition, Proof We consider the following energy functional By using (5) we have for every t ≥ t 0 From here, invoking the convexity of g, it followṡ for every t ≥ t 0 . Since is nonincreasing this leads further tȯ therefore the energy W is nonincreasing. Since W is bounded from bellow, there exists lim t−→+∞ W (t) ∈ R. Consequently, t → W (t) is bounded on [t 0 , +∞) from which, since g is bounded from bellow, we obtain that sup t≥t 0 ẋ(t) = K < +∞.
By integrating (34) on an interval [t 0 , t] for arbitrary t > t 0 it yields which, by letting t −→ +∞, leads to Further, for every t ≥ t 0 we have that For each > 0, we denote by x the unique solution of the strongly convex minimization problem In virtue of the Fermat rule, this is equivalent to It is well known that the Tikhonov approximation curve −→ x satisfies lim −→0 x = x * , where x * = argmin{ x : x ∈ argming} is the element of minimum norm of the nonempty convex closed set argming. Since ∇g is monotone, for every x , x * ≥ 0, which, by using the Cauchy-Schwarz inequality, implies x ≤ x * for every > 0.

Strong ergodic convergence
We will start by proving a strong ergodic convergence result for the trajectory of (5). (5). Assume that

Theorem 4.2 Let x be the unique global C 2 -solution of
Let x * = argmin{ x : x ∈ argming} be the element of minimum norm of the nonempty convex closed set argming. If α > 0, then Proof We introduce the function For every t ≥ t 0 we havë Further, for every t ≥ t 0 , the function g t : But ∇g t (x(t)) = ∇g(x(t)) + (t)x(t) and by using (5) we get Consequently, (36) becomes By using (35), the latter relation leads to for every t ≥ t 0 . For every t ≥ t 0 , let x (t) the unique solution of the strongly convex minimization problem Then for every t ≥ t 0 and taking into account (38) we get for every t ≥ t 0 . We havë and On the other hand, for every t ≥ t 0 , The last equality immediately implies that

Remark 4.3
The strong ergodic convergence obtained in [8] for the dynamical system (4) is extended to the dynamical system with Hessian driven damping and Tikhonov regularization term (5) under the same hypotheses concerning the Tikhonov parametrization t → (t).

Strong convergence
In order to prove strong convergence for the trajectory generated by the dynamical system (5) to an element of minimum norm of argming we have to strengthen the conditions on the Tikhonov parametrization. This is done in the following result.

Theorem 4.4
Let be α ≥ 3 and x the unique global C 2 -solution of (5). Assume that and that there exist a > 1 and t 1 ≥ t 0 such thaṫ In addition, assume that • in case α = 3: lim t−→+∞ t 2 (t) = +∞; • in case α > 3: there exists c > 0 such that t 2 (t) ≥ 2 3 α 1 3 α − 1 + βc 2 for t large enough. If x * = argmin{ x : x ∈ argming} is the element of minimum norm of the nonempty convex closed set argming, then In addition, In other words, x(t) ≥ x * for every t ≥ T . For p ≥ 0, we consider the energy functional The next step is to obtain an upper bound for t → E p b (t), and to this end we will evaluate its time derivative. For every t ≥ t 0 we have By using (5) we havë for every t ≥ t 0 . Further, for every t ≥ t 0 , which means that (51) becomes according to Theorem 3.1, we have lim t−→+∞ g(x(t)) = min g.
Consider x ∈ H a weak sequential cluster point of the trajectory x, which exists since the trajectory is bounded. This means that there exists a sequence (t n ) n∈N ⊆ [T , +∞) such that t n −→ +∞ and x(t n ) converges weakly to x as n −→ +∞.
Since g is weakly lower semicontinuous, it holds Since the norm is weakly lower semicontinuous, it holds which, by taking into account that x * is the unique element of minimum norm in argming, implies x = x * . This shows that the whole trajectory x converges weakly to x * . Thus, But by taking into account that x(t) x * as t −→ +∞, we obtain that the convergence is strong, that is Case III Assume that for every T ≥ t 0 there exists t ≥ T such that x * > x(t) and there exists s ≥ T such that x * ≤ x(s) .
By the continuity of x it follows that there exists a sequence (t n ) n∈N ⊆ [t 0 , +∞) such that t n −→ +∞ as n −→ +∞ and x(t n ) = x * for every n ∈ N.
We will show that x(t n ) −→ x * as n −→ +∞. To this end we consider x ∈ H a weak sequential cluster point of the sequence (x(t n )) n∈N . By repeating the arguments used in the previous case (notice that the sequence is bounded) it follows that (x(t n )) n∈N converges weakly to x * as n −→ +∞. Since x(t n ) −→ x * as n −→ +∞, it yields x(t n ) − x * −→ 0 as n −→ +∞. This shows that [8] for the dynamical system (4) to the dynamical system with Hessian driven damping and Tikhonov regularization term (5). One can notice that for the choice β = 0, which means that the Hessian driven damping is removed, the lower bound we impose for t → t 2 (t) in case α > 3 is less tight than the one considered in [8,Theorem 4.1] for the system (4). As we will see later, this lower bound influences the asymptotic behaviour of the trajectory.

Remark 4.5 Theorem 4.4 can be seen as an extension of a result given in
In case β > 0, in order to guarantee that This follows from Lemma A.1, by also taking into account that lim t−→+∞ (t)t α 3 +1 = +∞.
Combining the main results in the last two sections, one can see that if In the following, we would like to comment on the role on the condition in Theorem 4.4 which asks, in case α > 3, for the existence of a positive constant c such that t 2 (t) ≥ 2 3 α( 1 3 α −1+βc 2 ) for t large enough. To this end it is very helpful to visualize the trajectories generated by the dynamical system (5) in relation with the minimization of the function given in (6) for a fixed large value of α and Tikhonov parametrizations of the form t → (t) = t −γ , for different values of γ ∈ (1, 2). The trajectories in the plot in Fig. 2 have been generated for α = 200 and β = 1 and are all approaching the minimum norm solution x * = 0. The norm of the difference between the trajectory and the minimum norm solution is guaranteed to be bounded from above by a function which converges to zero, after the time point t is reached at which the inequality t 2 (t) ≥ 2 3 α( 1 3 α − 1 + βc 2 ) "starts" being fulfilled. For large α and the Tikhonov parametrizations considered in our experiment, the closer γ is to 1 is, the faster is this inequality fulfilled. This is reflected by the behaviour of the trajectories plotted in Fig. 2.
Finally, we would like to formulate some possible questions of future research related to the dynamical sytem (5): • In [7, Theorem 3.4] it has been proved for the dynamical system (1) that, when g is strongly convex, the rates of convergence of the function values and the tracjectory are both of O(t − 2 3 α ), thus they can be made arbitrarily fast by taking α large. It is natural to ask if similar rates of convergence can be obtained in a similar setting for the dynamical system (5) (see, also, [8,Section 5.4]).
• In the literature, in the context of dynamical systems, regularization terms have been considered not only in open-loop, but also in closed-loop form (see, for instance, [12]). It is an interesting question if one can obtain for the dynamical system (5) similar results if the Tikhonov regularization term is taken in closedloop form.
• A natural question is to formulate proper numerical algorithms via time discretization of (5), to investigate their theoretical convergence properties, and to validate them with numerical experiments.

Lemma A.3 Let S ⊆ H be a nonempty set and x : [t 0 , +∞) → H a given map such that:
(i) for every z ∈ S the limit lim t−→+∞ x(t) − z exists; (ii) every weak sequential limit point of x(t) belongs to the set S.
Then the trajectory x(t) converges weakly to an element in S as t → +∞.