Notes on the convergence of a Levenberg-Marquardt method with singular scaling matrices for non-zero residue problems
DOI:
https://doi.org/10.5540/03.2025.011.01.0446Keywords:
Levenberg-Marquardt, Singular Scaling Matrix, Convergence analysis, Non-zero residueAbstract
This work addresses the local and global convergence analysis of a variant of the Levenberg-Marquardt method (LMM), designed for non-linear least-squares problems with non-zero residue. Such variant, called LMM with singular scaling (LMMSS), allows the so-called LMM scaling matrix to be singular, which can be useful in certain applications. In order to handle the non-zero residue while preserving local convergence, a judicious choice of the LMM parameter is made based on the gradient linearization error which is dictated by non-linearity and residue size. The main contributions of this work are related to local and global convergence of LMMSS in this setting. More specifically, we demonstrate that the sequence of directions \(d_k\) generated by LMMSS is gradient-related and then prove that limit points of a line-search version of LMMSS are stationary for the least-squares function. Moreover, the local analysis is further demonstrated under an error bound condition on the gradient and for different hypotheses on the linearization error.
Downloads
References
R. Behling, D. S. Gonçalves, and S. A. Santos. “Local convergence analysis of the Levenberg–Marquardt framework for nonzero-residue nonlinear least-squares problems under an error bound condition”. In: Journal of Optimization Theory and Applications 183.3 (2019), pp. 1099–1122.
E. H. Bergou, Y. Diouane, and V. Kungurtsev. “Convergence and Complexity Analysis of a Levenberg–Marquardt Algorithm for Inverse Problems”. In: Journal of Optimization Theory and Applications 185 (2020), pp. 927–944. doi: https://doi.org/10.1007/s10957-020-01666-1.
D. P. Bertsekas. Nonlinear Programming. Second. Athena Scientific, 1999.
E. Boos, D.S. Gonçalves, and F.S.V. Bazán. “Levenberg-Marquardt method with singular scaling and applications”. In: Applied Mathematics and Computation 474 (2024), p. 128688. doi: https://doi.org/10.1016/j.amc.2024.128688.
J. E. Dennis Jr. “Nonlinear least squares and equations”. In: The State of the Art in Numerical Analysis. Ed. by D. Jacobs. London: Academic Press, 1977, pp. 269–312.
J. E. Dennis Jr and R. B. Schnabel. Numerical Methods for Unconstrained Optimization and Nonlinear Equations. Philadelphia: SIAM, 1996.
R. Filippozzi, E. Boos, D.S. Gonçalves, and F.S.V. Bazán. “Convergence analysis of Levenberg-Marquardt method with Singular Scaling for nonzero residue nonlinear least-squares problems”. In: arxiv (2024). url: https://arxiv.org/abs/2408.10370.
P. C. Hansen. Rank-Deficient and Discrete Ill-Posed Problems. Philadelphia: SIAM, 1998.
K. Levenberg. “A Method for the Solution of Certain Non-Linear Problems in Least Squares”. In: Quarterly of Applied Mathematics 2 (1944), pp. 164–168.
D. Marquardt. “An Algorithm for Least-Squares Estimation of Nonlinear Parameters”. In: SIAM Journal on Applied Mathematics 11.2 (1963), pp. 431–441.
C. F. Van Loan. “Generalizing the Singular Value Decomposition”. In: SIAM Journal on Numerical Analysis 13.1 (1976), pp. 76–83.
N. Yamashita and M. Fukushima. “On the rate of convergence of the Levenberg-Marquardt method”. In: Topics in Numerical Analysis. Springer, 2001, pp. 239–249.
