Abstract
The success of all currently available regularization techniques relies heavily on the proper choice of the regularization parameter. Although many regularization parameter selection methods (RPSMs) have been proposed, very few of them are used in engineering practice. This is due to the fact that theoretically justified methods often require unrealistic assumptions, while empirical methods do not guarantee a good regularization parameter for any set of data. Among the methods that found their way into engineering applications, the most common are Morozov’s Discrepancy Principle (abbreviated as MDP) [morozov84, phillips62], Mallows’ CL [mallows73], generalized cross validation (abbreviated as GCV) [wahba90], and the L-curve method [hansen98]. A high sensitivity of CL and MDP to an underestimation of the noise level has limited their application to cases in which the noise level can be estimated with high fidelity [hansen98]. On the other hand, noise-estimate-free GCV occasionally fails, presumably due to the presence of correlated noise [wahba90]. The L-curve method is widely used; however, this method is nonconvergent [leonov97, vogel96]. An example of image restoration using different values of regularization parameters is shown in Figs. 2.1, 2.2, 2.3, 2.4, and 2.5. The Matlab code for this example was provided by Dr. Curt Vogel of Montana State University in a personal communication. The original image is presented in Fig. 2.1, and the observed blurred image is in Fig. 2.2.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
H. Akaike, Information theory and an extension of the maximum likelihood principle, in 2nd International Symposium on Information Theory, ed. by B.N. Petrov, F. Csaki (Akademiai Kiado, Budapest, 1973), pp. 267–281
H. Bozdogan, Model selection and Akaike’s information criterion (AIC): the general theory and its analytical extensions. Psychometrika 52(3), 345–370 (1987)
H. Bozdogan, ICOMP: a new model selection criterion, in Classification and Related Methods of Data Analysis, ed. by Hans H. Bock (Elsevier Science Publishers B.V. (North-Holland), Amsterdam, 1988), pp. 599–608
H. Bozdogan, A new informational complexity criterion for model selection: the general theory and its applications, in Information Theoretic Models and Inference (INFORMS), Washington D.C., 5–8 May 1996
H. Bozdogan, Informational complexity criteria for regression models, in Information Theory and Statistics Section on Bayesian Stat. Science, ASA Annual Meeting, Chicago, IL, 4–8 August 1996
H.W. Engl, M. Hanke, A. Neubauer, Regularization of Inverse Problems (Kluwer Academic, Dordrecht, 2000)
G.H. Golub, U. von Matt, Tikhonov regularization for large scale problems. Technical report SCCM-97-03, Stanford University, 1997
P.C. Hansen, D.P. O’Leary, The use of the L-curve in the regularization of discrete ill-posed problems. SIAM J. Sci. Comput. 14, 1487–1503 (1993)
P.C. Hansen, Regularization tools: a Matlab package for analysis and solution of discrete ill-posed problems. Numer. Algorithms 6, 1–35 (1994)
P.C. Hansen, Rank-Deficient and Discrete Ill-Posed Problems. SIAM Monographs on Mathematical Modeling and Computation (SIAM, Philadelphia, 1998)
P.J. Huber, Robust Statistics (Wiley, New York, 1981)
S. Konishi, G. Kitagawa, Generalized information criteria in model selection. Biometrika 83(4), 875–890 (1996)
S. Kullback, R.A. Leibler, On information and sufficiency. Ann. Math. Stat. 22, 79–86 (1951)
A.S. Leonov, A.G. Yagola, The L-curve method always introduces a non removable systematic error. Mosc. Univ. Phys. Bull. 52(6), 20–23 (1997)
C.L. Mallows, Some comments on CP. Technometrics 15(4), 661–675 (1973)
V.A. Morozov, Methods for Solving Incorrectly Posed Problems (Springer, New York, 1984)
D.L. Phillips, A technique for the numerical solution of certain integral equations of the first kind. JACM 9, 84–97 (1962)
Y. Sakamoto, Akaike Information Criterion Statistics (KTK Scientific publishers, Tokyo, 1986)
R. Shibata, Statistical aspects of model selection, in From Data to Model, ed. by J.C. Willems (Springer, New York, 1989), pp. 215–240
A.M. Urmanov, A.V. Gribok, H. Bozdogan, J.W. Hines, R.E. Uhrig, Information complexity-based regularization parameter selection for solution of ill-conditioned inverse problems. Inverse Prob. 18, L1–L9 (2002)
M.H. van Emden, An analysis of complexity, in Mathematical Centre Tracts, vol. 35 (Mathematisch Centrum, Amsterdam, 1971)
C.R. Vogel, Non-convergence of the L-curve regularization parameter selection method. Inverse Prob. 12, 535–547 (1996)
C.R. Vogel, Computational Methods for Inverse Problems. SIAM, Frontiers in Applied Mathematics Series, vol 23 (SIAM, Philadelphia, 2002)
G. Wahba, Spline Models for Observational Data (Society for Industrial and Applied Mathematics, Philadelphia, PA, 1990)
Author information
Authors and Affiliations
Rights and permissions
Copyright information
© 2016 Springer International Publishing Switzerland
About this chapter
Cite this chapter
Abidi, M.A., Gribok, A.V., Paik, J. (2016). Selection of the Regularization Parameter. In: Optimization Techniques in Computer Vision. Advances in Computer Vision and Pattern Recognition. Springer, Cham. https://doi.org/10.1007/978-3-319-46364-3_2
Download citation
DOI: https://doi.org/10.1007/978-3-319-46364-3_2
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-46363-6
Online ISBN: 978-3-319-46364-3
eBook Packages: Computer ScienceComputer Science (R0)