Algebraic Analysis for Singular Statistical Estimation

Abstract

This paper clarifies learning efficiency of a non-regular parametric model such as a neural network whose true parameter set is an analytic variety with singular points. By using Sato’s b-function we rigorously prove that the free energy or the Bayesian stochastic complexity is asymptotically equal to λ₁ log n − (m₁ − 1) log log n+constant, where λ₁ is a rational number, m₁ is a natural number, and n is the number of training samples. Also we show an algorithm to calculate λ₁ and m₁ based on the resolution of singularity. In regular models, 2λ₁ is equal to the number of parameters and m₁ = 1, whereas in non-regular models such as neural networks, 2λ₁ is smaller than the number of parameters and m₁ ≥ 1.