Abstract
Random monotone operators are stochastic versions of maximal monotone operators which play an important role in stochastic nonsmooth optimization. Several stochastic nonsmooth optimization algorithms have been shown to converge to a zero of a mean operator defined as the expectation, in the sense of the Aumann integral, of a random monotone operator.
In this note, we prove a strong law of large numbers for random monotone operators where the limit is the mean operator. We apply this result to the empirical risk minimization problem appearing in machine learning. We show that if the empirical risk minimizers converge as the number of data points goes to infinity, then they converge to an expected risk minimizer.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Data Availability
Data sharing not applicable to this article as no datasets were generated or analysed during the current study.
References
Atchade, Y.F., Fort, G., Moulines, E.: On stochastic proximal gradient algorithms. arXiv e-prints (2014). arXiv:1402.2365
Attouch, H.: Familles d’opérateurs maximaux monotones et mesurabilité. Ann. Mat. Pura Appl. 120(1), 35–111 (1979)
Attouch, H.: Variational Convergence for Functions and Operators, vol. 1. Pitman Advanced Publishing Program (1984)
Aumann, R.J.: Integrals of set-valued functions. J. Math. Anal. Appl. 12, 1–12 (1965)
Azé, D., Attouch, H., Wets, R.J-B.: Convergence of convex-concave saddle functions: applications to convex programming and mechanics. Ann. Inst. Henri Poincaré, Anal. Non Linéaire 5, 537–572 (1988)
Bauschke, H.H., Combettes, P.L.: Convex Analysis and Monotone Operator Theory in Hilbert Spaces, vol. 408. Springer, Berlin (2011)
Bianchi, P.: Ergodic convergence of a stochastic proximal point algorithm. SIAM J. Optim. 26(4), 2235–2260 (2016)
Bianchi, P., Hachem, W.: Dynamical behavior of a stochastic forward-backward algorithm using random monotone operators. J. Optim. Theory Appl. 171(1), 90–120 (2016)
Bianchi, P., Hachem, W., Salim, A.: A constant step forward-backward algorithm involving random maximal monotone operators. J. Convex Anal. 26(2), 397–436 (2019)
Bottou, L., Bousquet, O.: The tradeoffs of large scale learning. In: NIPS, pp. 161–168 (2008)
Brézis, H.: Opérateurs Maximaux Monotones et Semi-Groupes de Contractions dans les Espaces de Hilbert. North-Holland Mathematics Studies. Elsevier, Burlington (1973)
Castaing, C., Raynaud de Fitte, P.: Law of large numbers and ergodic theorem for convex weak star compact valued Gelfand-integrable mappings. In: Advances in Mathematical Economics, vol. 17, pp. 1–37. Springer, Berlin (2013)
Condat, L., Kitahara, D., Contreras, A., Hirabayashi, A.: Proximal splitting algorithms for convex optimization: a tour of recent advances, with new twists. SIAM Rev. 65(2), 375–435 (2023)
Dal Maso, G.: An Introduction to \(\Gamma \)-Convergence, vol. 8. Springer, Berlin (2012)
Damek, D., Yin, W.: A three-operator splitting scheme and its optimization applications. Set-Valued Var. Anal. 25(4), 829–858 (2017)
Gorbunov, E., Hanzely, F., Richtárik, P.: A unified theory of SGD: variance reduction, sampling, quantization and coordinate descent. In: International Conference on Artificial Intelligence and Statistics, pp. 680–690. PMLR (2020)
Le Gall, J.-F.: Uniqueness and universality of the Brownian map. Ann. Probab. 41(4), 2880–2960 (2013)
Ledoux, M., Talagrand, M.: Probability in Banach Spaces: Isoperimetry and Processes. Springer, Berlin (2013)
Marčenko, V.A., Pastur, L.A.: Distribution of eigenvalues for some sets of random matrices. Math. USSR Sb. 1(4), 457 (1967)
Minty, G.J.: Monotone (nonlinear) operators in Hilbert space. Duke Math. J. 29(3), 341–346 (1962)
Phelps, R.R.: Convex Functions, Monotone Operators and Differentiability, vol. 1364. Springer, Berlin (2009)
Rockafellar, R.T., Wets, R.J-B.: On the interchange of subdifferentiation and conditional expectations for convex functionals. Stochastics 7(3), 173–182 (1982)
Shapiro, A., Xu, H.: Uniform laws of large numbers for set-valued mappings and subdifferentials of random functions. J. Math. Anal. Appl. 325(2), 1390–1399 (2007)
Taylor, R.L., Inoue, H.: Laws of Large Numbers for Random Sets, pp. 347–360. Springer, New York (1997)
Terán, P.: On a uniform law of large numbers for random sets and subdifferentials of random functions. Stat. Probab. Lett. 78(1), 42–49 (2008)
Zvi, A., Hart, S.: Law of large numbers for random sets and allocation processes. Math. Oper. Res. 6(4), 485–492 (1981)
Zvi, A., Vitale, R.A.: A strong law of large numbers for random compact sets. Ann. Probab. 3(5), 879–882 (1975)
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Competing Interests
The author declares no competing interests.
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Salim, A. A Strong Law of Large Numbers for Random Monotone Operators. Set-Valued Var. Anal 31, 38 (2023). https://doi.org/10.1007/s11228-023-00701-3
Received:
Accepted:
Published:
DOI: https://doi.org/10.1007/s11228-023-00701-3