# Douglas–Rachford splitting for nonconvex optimization with application to nonconvex feasibility problems

- 1.6k Downloads
- 13 Citations

## Abstract

We adapt the Douglas–Rachford (DR) splitting method to solve nonconvex feasibility problems by studying this method for a class of nonconvex optimization problem. While the convergence properties of the method for convex problems have been well studied, far less is known in the nonconvex setting. In this paper, for the direct adaptation of the method to minimize the sum of a proper closed function *g* and a smooth function *f* with a Lipschitz continuous gradient, we show that if the step-size parameter is smaller than a computable threshold and the sequence generated has a cluster point, then it gives a stationary point of the optimization problem. Convergence of the whole sequence and a local convergence rate are also established under the additional assumption that *f* and *g* are semi-algebraic. We also give simple sufficient conditions guaranteeing the boundedness of the sequence generated. We then apply our nonconvex DR splitting method to finding a point in the intersection of a closed convex set *C* and a general closed set *D* by minimizing the squared distance to *C* subject to *D*. We show that if either set is bounded and the step-size parameter is smaller than a computable threshold, then the sequence generated from the DR splitting method is actually bounded. Consequently, the sequence generated will have cluster points that are stationary for an optimization problem, and the whole sequence is convergent under an additional assumption that *C* and *D* are semi-algebraic. We achieve these results based on a new merit function constructed particularly for the DR splitting method. Our preliminary numerical results indicate that our DR splitting method usually outperforms the alternating projection method in finding a sparse solution of a linear system, in terms of both the solution quality and the number of iterations taken.

## Mathematics Subject Classification

90C06 90C26 90C90## Notes

### Acknowledgments

We would like to thank the two anonymous referees for their comments that helped improve the manuscript.

## References

- 1.Aragón, A.F., Borwein, J.M.: Global convergence of a non-convex Douglas–Rachford iteration. J. Global Optim.
**57**, 1–17 (2012)MathSciNetMATHGoogle Scholar - 2.Aragón, A.F., Borwein, J.M., Tam, M.K.: Douglas–Rachford feasibility methods for matrix completion problems. ANZIAM J.
**55**, 299–326 (2014)MathSciNetMATHGoogle Scholar - 3.Aragón, A.F., Borwein, J.M., Tam, M.K.: Recent results on Douglas–Rachford methods for combinatorial optimization problems. J. Optim. Theory Appl.
**163**, 1–30 (2014)MathSciNetCrossRefMATHGoogle Scholar - 4.Attouch, H., Bolte, J., Redont, P., Soubeyran, A.: Proximal alternating minimization and projection methods for nonconvex problems. An approach based on the Kurdyka–Lojasiewicz inequality. Math. Oper. Res.
**35**, 438–457 (2010)MathSciNetCrossRefMATHGoogle Scholar - 5.Attouch, H., Bolte, J., Svaiter, B.F.: Convergence of descent methods for semi-algebraic and tame problems: proximal algorithms, forward–backward splitting, and regularized Gauss–Seidel methods. Math. Program.
**137**, 91–129 (2013)MathSciNetCrossRefMATHGoogle Scholar - 6.Bauschke, H.H., Borwein, J.M.: On the convergence of von Neumann’s alternating projection algorithm for two sets. Set-Valued Anal.
**1**, 185–212 (1993)MathSciNetCrossRefMATHGoogle Scholar - 7.Bauschke, H.H., Borwein, J.M.: On projection algorithms for solving convex feasibility problems. SIAM Rev.
**38**, 367–426 (1996)MathSciNetCrossRefMATHGoogle Scholar - 8.Bauschke, H.H., Combettes, P.L.: Convex Analysis and Monotone Operator Theory in Hilbert Spaces. Springer, Berlin (2011)CrossRefMATHGoogle Scholar
- 9.Bauschke, H.H., Noll, D.: On the local convergence of the Douglas–Rachford algorithm. Arch. Math.
**102**, 589–600 (2014)Google Scholar - 10.Bolte, J., Daniilidis, A., Lewis, A.: The Łojasiewicz inequality for nonsmooth subanalytic functions with applications to subgradient dynamical systems. SIAM J. Optim.
**17**, 1205–1223 (2007)MathSciNetCrossRefMATHGoogle Scholar - 11.Bolte, J., Daniilidis, A., Lewis, A., Shiota, M.: Clarke subgradients of stratifiable functions. SIAM J. Optim.
**18**, 556–572 (2007)MathSciNetCrossRefMATHGoogle Scholar - 12.Borwein, J.M., Li, G., Yao, L.J.: Analysis of the convergence rate for the cyclic projection algorithm applied to basic semialgebraic convex sets. SIAM J. Optim.
**24**, 498–527 (2014)MathSciNetCrossRefMATHGoogle Scholar - 13.Combettes, P.L., Pesquet, J.-C.: A Douglas–Rachford splitting approach to nonsmooth convex variational signal recovery. IEEE J. Sel. Top. Signal Proces.
**1**, 564–574 (2007)CrossRefGoogle Scholar - 14.Douglas, J., Rachford, H.H.: On the numerical solution of heat conduction problems in two or three space variables. Trans. Am. Math. Soc.
**82**, 421–439 (1956)MathSciNetCrossRefMATHGoogle Scholar - 15.Eckstein, J., Bertsekas, D.P.: On the Douglas–Rachford splitting method and the proximal point algorithm for maximal monotone operators. Math. Program.
**55**, 293–318 (1992)MathSciNetCrossRefMATHGoogle Scholar - 16.Gandy, S., Recht, B., Yamada, I.: Tensor completion and low-\(n\)-rank tensor recovery via convex optimization. Inverse Probl.
**27**, 025010 (2011)MathSciNetCrossRefMATHGoogle Scholar - 17.Gong, P., Zhang, C., Lu, Z., Huang, J., Ye, J.: A general iterative shrinkage and thresholding algorithm for non-convex regularized optimization problems. In: The 30th International Conference on Machine Learning (ICML 2013)Google Scholar
- 18.He, B., Yuan, X.: On the \(O(1/n)\) convergence rate of the Douglas–Rachford alternating direction method. SIAM J. Numer. Anal.
**50**, 700–709 (2012)MathSciNetCrossRefMATHGoogle Scholar - 19.Hesse, R., Luke, D.R.: Nonconvex notions of regularity and convergence of fundamental algorithms for feasibility problems. SIAM J. Optim.
**23**, 2397–2419 (2013)MathSciNetCrossRefMATHGoogle Scholar - 20.Hesse, R., Luke, D.R., Neumann, P.: Alternating projections and Douglas–Rachford for sparse affine feasibility. IEEE Trans. Signal. Proces.
**62**, 4868–4881 (2014)MathSciNetCrossRefGoogle Scholar - 21.Lewis, A.S., Luke, D.R., Malick, J.: Local convergence for alternating and averaged nonconvex projections. Found. Comput. Math.
**9**, 485–513 (2009)MathSciNetCrossRefMATHGoogle Scholar - 22.Lewis, A.S., Malick, J.: Alternating projections on manifolds. Math. Oper. Res.
**33**, 216–234 (2008)MathSciNetCrossRefMATHGoogle Scholar - 23.Li, G., Mordukhovich, B.S., Pham, T.S.: New fractional error bounds for polynomial systems with applications to Hölderian stability in optimization and spectral theory of tensors. Math. Program.
**153**, 333–362 (2015)Google Scholar - 24.Lions, P.-L., Mercier, B.: Splitting algorithms for the sum of two nonlinear operators. SIAM J. Numer. Anal.
**16**, 964–979 (1979)MathSciNetCrossRefMATHGoogle Scholar - 25.Lu, Z., Zhang, Y.: Sparse approximation via penalty decomposition methods. SIAM J. Optim.
**23**, 2448–2478 (2013)MathSciNetCrossRefMATHGoogle Scholar - 26.Luke, D.R.: Finding best approximation pairs relative to a convex and a prox-regular set in Hilbert space. SIAM J. Optim.
**19**, 714–739 (2008)MathSciNetCrossRefMATHGoogle Scholar - 27.Patrinos, P., Stella, L., Bemporad, A.: Douglas–Rachford splitting: complexity estimates and accelerated variants. In: Proceedings of the 2014 IEEE 53rd Annual Conference on Decision and Control (CDC), pp. 4234–4239. IEEE, Los Angeles, CA (2014)Google Scholar
- 28.Phan, H.: Linear convergence of the Douglas–Rachford method for two closed sets. Optimization. arXiv:1401.6509
- 29.Poliquin, R.A., Rockafellar, R.T., Thibault, L.: Local differentiability of distance functions. Trans. Am. Math. Soc.
**352**, 5231–5249 (2000)MathSciNetCrossRefMATHGoogle Scholar - 30.Rockafellar, R.T., Wets, R.J.-B.: Variational Analysis. Springer, Berlin (1998)CrossRefMATHGoogle Scholar
- 31.Sun, D., Toh, K.-C., Yang, L.: A convergent 3-block semi-proximal alternating direction method of multipliers for conic programming with 4-type of constraints. SIAM J. Optim.
**25**, 882–915 (2015)Google Scholar - 32.Zeng, J., Lin, S., Wang, Y., Xu, Z.: \(L_{1/2}\) regularization: convergence of iterative half thresholding algorithm. IEEE Trans. Signal Process.
**62**, 2317–2329 (2014)MathSciNetCrossRefGoogle Scholar