Convergence properties of Gauss-Newton iterative algorithms in nonlinear image restoration

  • M. E. Zervakis
  • A. N. Venetsanopoulos


Nonlinear image restoration is a complicated problem that is receiving increasing attention. Since every image formation system involves a built-in nonlinearity, nonlinear image restoration finds applications in a wide variety of research areas. Iterative algorithms have been well established in the corresponding linear restoration problem. In this paper, a generalized analysis regarding the convergence properties of nonlinear iterative algorithms is introduced. Moreover, the applications of the iterative Gauss-Newton (GN) algorithm in nonlinear image restoration are considered. The convergence properties of a general class of nonlinear iterative algorithms are rigorously studied through the Global Convergence Theorem (GCT). The derivation of the convergence properties is based on the eigen-analysis, rather than on the norm analysis. This approach offers a global picture of the evolution and the convergence properties of an iterative algorithm. Moreover, the generalized convergence-analysis introduced may be interpreted as a link towards the integration of minimization and projection algorithms. The iterative GN algorithm for the solution of the least-squares optimization problem is introduced. The computational complexity of this algorithm is enormous, making its implementation very difficult in practical applications. Structural modifications are introduced, which drastically reduce the computational complexity while preserving the convergence rate of the GN algorithm. With the structural modifications, the GN algorithm becomes particularly useful in nonlinear optimization problems. The convergence properties of the algorithms introduced are readily derived, on the basis of the generalized analysis through the GCT. The application of these algorithms on practical problems, is demonstrated through several examples.


Computational Complexity Generalize Analysis Nonlinear Optimization Iterative Algorithm Structural Modification 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. H.C. Andrews and B.R. Hunt,Digital Image Restoration, Prentice-Hall, Englewood Cliffs, NJ: 1977.Google Scholar
  2. H.J. Trussell, “A Priori Knowledge in Algebraic Reconstruction Methods,” inAdvances in Computer Vision and Image Processing, vol.1, (T.S. Huang ed.) JAI Press Inc., Greenwich, Conneticut, 1984.Google Scholar
  3. M. Bertero, T.A. Poggio, and V. Torre, “Ill-Posed Problems in Early Vision,”IEEE Proceedings, vol. 76, no. 8, 1988.Google Scholar
  4. B.R. Hunt, “The Application of Constrained Least Squares Estimation to Image Restoration by Digital Computer,”IEEE Trans. on Computers, vol. C-22, no. 9, 1973.Google Scholar
  5. B.R. Hunt, “Bayesian Methods in Nonlinear Digital Image Restoration,”IEEE Trans. on Computers, vol. C-26, no. 3, 1977.Google Scholar
  6. D.T. Kuan, A.A. Sawchuk, T.C. Strand, and P. Chavel, “Adaptive Noise Smoothing Filter for Images with Signal Dependent Noise,”IEEE Trans. on Pattern Analysis and Machine Intelligence, vol. PAMI-7, no. 2, 1985.Google Scholar
  7. J.S. Lee, “Speckle Analysis and Smoothing of Synthetic Aperture Radar Images,”Computer Graphics and Image Processing,” vol. 17, pp. 24–32, 1981.Google Scholar
  8. R. Bernstein, “Adaptive Nonlinear Filters for Simultaneous Removal of Different Kinds of Noise in Images,”IEEE Trans. on Circuits and Systems, vol. CAS-34, no. 11, 1987.Google Scholar
  9. V.S. Frost, J.A. Stiles, K.S. Shanmugam, J.C. Holtzman, and S.A. Smith, “An Adaptive Filter for Smoothing Noisy Radar Images,”Proceedings of the IEEE, vol. 69, no. 1, 1981.Google Scholar
  10. S. German, and D. Geman, “Stochastic Relaxation, Gibbs Distributions, and the Bayesian Restoration of Images,”IEEE Trans. on Pattern Analysis and Machine Intelligence, vol. PAMI-6, no. 6, 1984.Google Scholar
  11. H.J. Trussell and B.R. Hunt, “Improved Methods of Maximum a Posteriori Restoration,”IEEE Trans. on Computers, vol. C-27, no. 1, 1979.Google Scholar
  12. M.E. Zervakis and A.N. Venetsanopoulos, “A Generalized Adaptive Model for Nonlinear Image Restoration,”Proc. of Int'l Symposium on Circuits and Systems ISCAS' 89, Portland, Oregon, May 9–11, 1989, July 1991.Google Scholar
  13. M.E. Zervakis and A.N. Venetsanopoulos, “Iterative Least Squares Estimators in Nonlinear Image Restoration,”IEEE Trans. on Acoustics, Speech, and Signal Processing, in Press, April 1992.Google Scholar
  14. D.G. Luenberger,Introduction to Linear and Nonlinear Programming, Addison-Wesley: Reading, MA 1973.Google Scholar
  15. R.W. Schafer, R.M. Mersereau, and M.A. Richards, “Constrainted Iterative Restoration Algorithms,”Proceedings of the IEEE, vol. 69, no. 4, 1981.Google Scholar
  16. V.T. Tom, T.F. Quatieri, M.H. Hayes, and J.H. McClellan, “Convergence of Iterative Nonexpansive Signal Reconstruction Algorithms,”IEEE Trans. on Acoustics, Speech, and Signal Processing, vol. ASSP-29, no. 5, 1981.Google Scholar
  17. J.A. Gadzow, “Signal Enhancement—A Composite Property Mapping Algorithm,”IEEE Trans. on Acoustics, Speech, and Signal Processing, vol. ASSP-36, no. 1, 1988.Google Scholar
  18. H.J. Trussell, “Convergence Criteria for Iterative Restoration Methods,”IEEE Trans. on Acoustics, Speech, and Signal Processing, vol. ASSP-31, no. 1, 1983.Google Scholar
  19. H.J. Trussell and M.R. Civanlar, “The Feasible Solution in Image Restoration,”IEEE Trans. on Acoustics, Speech, and Signal Processing, vol. ASSP-32, no. 2, 1984.Google Scholar
  20. M.E. Zervakis, “Nonlinear Image Restoration Techniques,” Ph.D. Thesis, University of Toronto, 1989.Google Scholar
  21. R.L. Lagendijk, J. Biemond, and D.E. Boekee, “Regularized Iterative Image Restoration with Ringing Reduction,”IEEE Trans. on Acoustics, Speech, and Signal Processing, vol. ASSP-36, no. 12, 1988.Google Scholar
  22. A. Katsaggelos, “Iterative Image Restoration Algorithms,”Optical Engineering, vol. 28, no. 7, pp. 735–748, 1989.Google Scholar
  23. D.C. Youla, “Mathematical Theory of Image Restoration by the Method of Convex Projections,” inImage Recovery: Theory and Application, (H. Stark, ed.), Academic Press, Orlando, FL, 1987.Google Scholar
  24. R. Gallant,Nonlinear Statistical Models J. Wiley: New York, 1987.Google Scholar
  25. G.H. Golub, and C.F. Van Loan,Matrix Computations Johns Hopkins, Baltimore, MD, 1983.Google Scholar
  26. M.E. Zervakis and A.N. Venetsanopoulos, “Iterative algorithms with Fast Convergence Rates in Nonlinear Image Restoration,”Proc. of the 1991 SPIE/SPSE Symp. on Electronic Imaging Science and Technology, San Jose, CA, Feb. 1991.Google Scholar

Copyright information

© Kluwer Academic Publishers 1991

Authors and Affiliations

  • M. E. Zervakis
    • 1
  • A. N. Venetsanopoulos
    • 2
  1. 1.Department of Computer EngineeringUniversity of MinnesotaDuluth
  2. 2.Department of Electrical EngineeringUniversity of TorontoTorontoCanada

Personalised recommendations