| |
Abstract:
This paper mathematically clarifies the asymptotic form of
the free energy or the stochastic complexity for non-regular and
over-realizable learning machines. For regular learning machines,
it is well known that the asymptotic form of the free energy is
F(n)= (d/2)log n, where d is the dimension of the parameter space
and n is the number of training samples. However, layered
statistical models such as neural networks are not regular models,
because the set of true parameters is not one point but an analytic
set with singularities if the model contains the true distribution.
For such non-regular and non-identifiable learning machines, we
prove that F(n) = a log n + (b-1)loglog n, where a is a rational
number and b is a natural number which are determined from the zero
point of Sato-Bernstein polynomial and its multiplicity. We also
show that a and b can be calculated by using resolution of
singularities in algebraic geometry.
|
