Journal of Global Optimization

, Volume 74, Issue 1, pp 95–119 | Cite as

Asynchronous sequential inertial iterations for common fixed points problems with an application to linear systems

  • Howard HeatonEmail author
  • Yair Censor


The common fixed points problem requires finding a point in the intersection of fixed points sets of a finite collection of operators. Quickly solving problems of this sort is of great practical importance for engineering and scientific tasks (e.g., for computed tomography). Iterative methods for solving these problems often employ a Krasnosel’skiĭ–Mann type iteration. We present an asynchronous sequential inertial (ASI) algorithmic framework in a Hilbert space to solve common fixed points problems with a collection of nonexpansive operators. Our scheme allows use of out-of-date iterates when generating updates, thereby enabling processing nodes to work simultaneously and without synchronization. This method also includes inertial type extrapolation terms to increase the speed of convergence. In particular, we extend the application of the recent “ARock algorithm” (Peng et al. in SIAM J Sci Comput 38:A2851–A2879, 2016) in the context of convex feasibility problems. Convergence of the ASI algorithm is proven with no assumption on the distribution of delays, except that they be uniformly bounded. Discussion is provided along with a computational example showing the performance of the ASI algorithm applied in conjunction with a diagonally relaxed orthogonal projections (DROP) algorithm for estimating solutions to large linear systems.


Convex feasibility problem Asynchronous sequential iterations Nonexpansive operator Fixed point iteration Kaczmarz method DROP algorithm 



We thank Prof. Reinhard Schulte from Loma Linda Univeristy for valuable discussions and insights regarding the advantage of an asynchronous approach to large scale problems in proton CT (pCT) image reconstruction and Intensity-Modulated Proton Therapy (IMPT). We appreciate constructive comments received from Prof. Ernesto Gomez from California State University San Bernardino and Prof. Keith Schubert from Baylor University. We thank the anonymous reviewers for their constructive comments. We are indebted to Patrick Combettes, Jonathan Eckstein, Dan Gordon and Simeon Reich for valuable feedback on the earlier version of this work that was posted on arXiv. The first author’s work is supported by the National Science Foundation Graduate Research Fellowship under Grant No. DGE-1650604. Any opinion, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation. The second author’s work was supported by research Grant No. 2013003 of the United States-Israel Binational Science Foundation (BSF).


  1. 1.
    Aharoni, R., Censor, Y.: Block-iterative projection methods for parallel computation of solutions to convex feasibility problems. Linear Algebra Appl. 120, 165–175 (1989)MathSciNetCrossRefzbMATHGoogle Scholar
  2. 2.
    Alvarez, F., Attouch, H.: An inertial proximal method for maximal monotone operators via discretization of a nonlinear oscillator with damping. Set-Valued Anal. 9, 3–11 (2001)MathSciNetCrossRefzbMATHGoogle Scholar
  3. 3.
    Bailion, J.B., Bruck, R.E., Reich, S.: On the asymptotic behavior of nonexpansive mappings and semigroups in Banach spaces. Houston J. Math. 4, 1–9 (1978)MathSciNetGoogle Scholar
  4. 4.
    Bauschke, H., Borwein, J.: On projection algorithms for solving convex feasibility problems. SIAM Rev. 38, 367–426 (1996)MathSciNetCrossRefzbMATHGoogle Scholar
  5. 5.
    Bauschke, H., Combettes, P.: Convex Analysis and Monotone Operator Theory in Hilbert Spaces, 2nd edn. Springer, Berlin (2017)CrossRefzbMATHGoogle Scholar
  6. 6.
    Bauschke, H., Combettes, P., Kruk, S.: Extrapolation algorithm for affine-convex feasibility problems. Numer. Algorithms 41, 239–274 (2006)MathSciNetCrossRefzbMATHGoogle Scholar
  7. 7.
    Bauschke, H., Iorio, F., Koch, V.: The method of cyclic intrepid projections: convergence analysis and numerical experiments. In: Wakayama, M., Anderssen, R., Cheng, J., Fukumoto, Y., McKibbin, R., Polthier, K., Takagi, T., Toh, K.C. (eds.) The Impact of Applications on Mathematics, pp. 187–200. Springer Japan, Tokyo (2014)Google Scholar
  8. 8.
    Bertsekas, D., Tsitsiklis, J.: Parallel and Distributed Computation: Numerical Methods. Prentice Hall, Englewood Cliffs (1989)zbMATHGoogle Scholar
  9. 9.
    Cegielski, A.: Iterative Methods for Fixed Point Problems in Hilbert Spaces. Lecture Notes in Mathematics, vol. 2057. Springer, Berlin (2012)zbMATHGoogle Scholar
  10. 10.
    Censor, Y.: An automatic relaxation method for solving interval linear inequalities. J. Math. Anal. Appl. 106, 19–25 (1985)MathSciNetCrossRefzbMATHGoogle Scholar
  11. 11.
    Censor, Y.: Parallel application of block-iterative methods in medical imaging and radiation therapy. Math. Program. 42, 307–325 (1988)MathSciNetCrossRefzbMATHGoogle Scholar
  12. 12.
    Censor, Y., Altschuler, M., Powlis, W.: On the use of Cimmino’s simultaneous projections method for computing a solution of the inverse problem in radiation therapy treatment planning. Inverse Probl. 4, 607 (1988)MathSciNetCrossRefzbMATHGoogle Scholar
  13. 13.
    Censor, Y., Chen, W., Combettes, P., Davidi, R., Herman, G.: On the effectiveness of projection methods for convex feasibility problems with linear inequality constraints. Comput. Optim. Appl. 51, 1065–1088 (2012)MathSciNetCrossRefzbMATHGoogle Scholar
  14. 14.
    Censor, Y., Elfving, T.: Block-iterative algorithms with diagonally scaled oblique projections for the linear feasibility problem. SIAM J. Matrix Anal. Appl. 24, 40–58 (2002)MathSciNetCrossRefzbMATHGoogle Scholar
  15. 15.
    Censor, Y., Elfving, T., Herman, G.: Averaging strings of sequential iterations for convex feasibility problems. In: Butnariu, D., Censor, Y., Reich, S. (eds.) Inherently Parallel Algorithms in Feasibility and Optimization and Their Applications, pp. 101–113. Elsevier, Amsterdam (2001)CrossRefGoogle Scholar
  16. 16.
    Censor, Y., Elfving, T., Herman, G., Nikazad, T.: On diagonally relaxed orthogonal projection methods. SIAM J. Sci. Comput. 30, 473–504 (2008)MathSciNetCrossRefzbMATHGoogle Scholar
  17. 17.
    Censor, Y., Gordon, D., Gordon, R.: BICAV: a block-iterative parallel algorithm for sparse systems with pixel-related weighting. IEEE Trans. Med. Imaging 20, 1050–1060 (2001)CrossRefGoogle Scholar
  18. 18.
    Censor, Y., Gordon, D., Gordon, R.: Component averaging: an efficient iterative parallel algorithm for large and sparse unstructured problems. Parallel Comput. 27, 777–808 (2001)MathSciNetCrossRefzbMATHGoogle Scholar
  19. 19.
    Censor, Y., Mansour, R.: Convergence analysis of processes with valiant projection operators in hilbert space. J. Optim. Theory Appl. 176, 35–56 (2018)MathSciNetCrossRefzbMATHGoogle Scholar
  20. 20.
    Censor, Y., Reich, S.: Iterations of paracontractions and firmaly nonexpansive operators with applications to feasibility and optimization. Optimization 37, 323–339 (1996)MathSciNetCrossRefzbMATHGoogle Scholar
  21. 21.
    Censor, Y., Zenios, S.: Parallel Optimization: Theory, Algorithm, and Applications. Oxford University Press, New York (1997)zbMATHGoogle Scholar
  22. 22.
    Chazan, D., Miranker, W.: Chaotic relaxation. Linear Algebra Appl. 2, 199–222 (1969)MathSciNetCrossRefzbMATHGoogle Scholar
  23. 23.
    Cimmino, G.: Cacolo approssimato per soluzioni dei sistemi di equazioni lineari. La Ricerca Scientifica XVI, Series II, Anno IX 1, 326–333 (1938)zbMATHGoogle Scholar
  24. 24.
    Combettes, P.: The Convex Feasibility Problem in Image Recovery, vol. 95. Academic Press, New York (1996)Google Scholar
  25. 25.
    Combettes, P., Eckstein, J.: Asynchronous block-iterative primal–dual decomposition methods for monotone inclusions. Math. Program. 168, 645–672 (2018)MathSciNetCrossRefzbMATHGoogle Scholar
  26. 26.
    Combettes, P., Glaudin, L.: Quasi-nonexpansive iterations on the affine hull of orbits: from Mann’s mean value algorithm to inertial methods. SIAM J. Optim. 27, 2356–2380 (2017)MathSciNetCrossRefzbMATHGoogle Scholar
  27. 27.
    Dong, Q.L., Huang, J.Z., Li, X.H., Cho, Y.J., Rassias, T.M.: MiKM: multi-step inertial Krasnoselski–Mann algorithm and its applications. J. Global Optim. (2018).
  28. 28.
    Eckstein, J.: A simplified form of block-iterative operator splitting and an asynchronous algorithm resembling the multi-block alternating direction method of multipliers. J. Optim. Theory Appl. 173, 155–182 (2017)MathSciNetCrossRefzbMATHGoogle Scholar
  29. 29.
    Elsner, L., Koltracht, I., Neumann, M.: Convergence of sequential and asynchronous nonlinear paracontractions. Numer. Math. 62, 305–319 (1992)MathSciNetCrossRefzbMATHGoogle Scholar
  30. 30.
    Frommer, A., Szyld, D.: On asynchronous iterations. J. Comput. Appl. Math. 123, 201–216 (2000)MathSciNetCrossRefzbMATHGoogle Scholar
  31. 31.
    Geer, D.: Chip makers turn to multicore processors. Computer 38, 11–13 (2005)CrossRefGoogle Scholar
  32. 32.
    Gordon, D., Gordon, R.: Component-averaged row projections: a robust, block-parallel scheme for sparse linear systems. SIAM J. Sci. Comput. 27, 1092–1117 (2005)MathSciNetCrossRefzbMATHGoogle Scholar
  33. 33.
    Gordon, R., Bender, R., Herman, G.: Algebraic reconstruction techniques (art) for three-dimensional electron microscopy and x-ray photography. J. Theor. Biol. 29, 471–481 (1970)CrossRefGoogle Scholar
  34. 34.
    Hannah, R., Yin, W.: On unbounded delays in asynchronous parallel fixed-point algorithms. J. Sci. Comput. 76, 299–326 (2018)MathSciNetCrossRefzbMATHGoogle Scholar
  35. 35.
    Hansen, P.C., Saxild-Hansen, M.: AIR Tools-A MATLAB package of algebraic iterative reconstruction methods. J. Comput. Appl. Math. 236, 2167–2178 (2012)MathSciNetCrossRefzbMATHGoogle Scholar
  36. 36.
    Herman, G.: A relaxation method for reconstructing objects from noisy x-rays. Math. Program. 8, 1–19 (1975)MathSciNetCrossRefzbMATHGoogle Scholar
  37. 37.
    Herman, G.T.: Fundamentals of Computerized Tomography, 2nd edn. Springer, London (2009)CrossRefzbMATHGoogle Scholar
  38. 38.
    Johnstone, P.R., Eckstein, J.: Convergence Rates for Projective Splitting. arXiv preprint arXiv:1806.03920 (2018)
  39. 39.
    Johnstone, P.R., Eckstein, J.: Projective Splitting with Forward Steps: Asynchronous and Block-Iterative Operator Splitting. arXiv preprint arXiv:1803.07043 (2018)
  40. 40.
    Kaczmarz, S.: Angenäherte Auflösung von Systemen linearer Gleichungen. Bull. Int. Acad. Pol. Sci. Lett. A35, 355–357 (1937)zbMATHGoogle Scholar
  41. 41.
    Kazimierz, G., Reich, S.: Uniform Convexity, Hyperbolic Geometry, and Nonexpansive Mappings. Marcel Dekker, New York (1984)zbMATHGoogle Scholar
  42. 42.
    Kiwiel, K.: Block-iterative surrogate projection methods for convex feasibility problems. Linear Algebra Appl. 215, 225–259 (1995)MathSciNetCrossRefzbMATHGoogle Scholar
  43. 43.
    Krasnosel’skii, M.: Two remarks on the method of successive approximations. Uspekhi Mat. Nauk 10, 123–127 (1955). (in Russian)MathSciNetGoogle Scholar
  44. 44.
    Lorenz, D., Pock, T.: An inertial forward-backward algorithm for monotone inclusions. J. Math. Imaging Vis. 51, 311–325 (2015)MathSciNetCrossRefzbMATHGoogle Scholar
  45. 45.
    Maingé, P.E.: Convergence theorems for inertial KM-type algorithms. J. Comput. Appl. Math. 219, 223–236 (2008)MathSciNetCrossRefzbMATHGoogle Scholar
  46. 46.
    Mann, W.: Mean value methods in iteration. Proc. Am. Math. Soc. 4, 506–510 (1953)MathSciNetCrossRefzbMATHGoogle Scholar
  47. 47.
    Moudafi, A., Oliny, M.: Convergence of a splitting inertial proximal method for monotone operators. J. Comput. Appl. Math. 155, 447–454 (2003)MathSciNetCrossRefzbMATHGoogle Scholar
  48. 48.
    Peng, Z., Xu, Y., Yan, M., Yin, W.: ARock: an algorithmic framework for asynchronous parallel coordinate updates. SIAM J. Sci. Comput. 38, A2851–A2879 (2016)MathSciNetCrossRefzbMATHGoogle Scholar
  49. 49.
    Polyak, B.: Some methods of speeding up the convergence of iteration methods. USSR Comput. Math. Math. Phys. 4, 1–17 (1964)CrossRefGoogle Scholar
  50. 50.
    Reich, S.: Weak convergence theorems for nonexpansive mappings in Banach spaces. J. Math. Anal. Appl. 67, 274–276 (1979)MathSciNetCrossRefzbMATHGoogle Scholar
  51. 51.
    Reich, S.: A note on the mean ergodic theorem for nonlinear semigroups. J. Math. Anal. Appl. 91, 547–551 (1983)MathSciNetCrossRefzbMATHGoogle Scholar
  52. 52.
    Reich, S., Zalas, R.: A modular string averaging procedure for solving the common fixed point problem for quasi-nonexpansive mappings in Hilbert space. Numer. Algorithms 72, 297–323 (2016)MathSciNetCrossRefzbMATHGoogle Scholar
  53. 53.
    Shepp, L., Logan, B.: The Fourier reconstruction of a head section. IEEE Trans. Nucl. Sci. 21, 21–43 (1974)CrossRefGoogle Scholar
  54. 54.
    Strikwerda, J.: A convergence theorem for chaotic asynchronous relaxation. Linear Algebra Appl. 253, 15–24 (1997)MathSciNetCrossRefzbMATHGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2019

Authors and Affiliations

  1. 1.Department of MathematicsUniversity of California Los AngelesLos AngelesUSA
  2. 2.Department of MathematicsUniversity of HaifaHaifaIsrael

Personalised recommendations