Monthly
288 pp. per issue
6 x 9, illustrated
ISSN
0899-7667
E-ISSN
1530-888X
2014 Impact factor:
2.21

Neural Computation

February 1, 2002, Vol. 14, No. 2, Pages 421-435
(doi: 10.1162/08997660252741185)
© 2001 Massachusetts Institute of Technology
Stochastic Trapping in a Solvable Model of On-Line Independent Component Analysis
Article PDF (232.5 KB)
Abstract

Previous analytical studies of on-line independent component analysis (ICA) learning rules have focused on asymptotic stability and efficiency. In practice, the transient stages of learning are often more significant in determining the success of an algorithm. This is demonstrated here with an analysis of a Hebbian ICA algorithm, which can find a small number of nongaussian components given data composed of a linear mixture of independent source signals. An idealized data model is considered in which the sources comprise a number of nongaussian and gaussian sources, and a solution to the dynamics is obtained in the limit where the number of gaussian sources is infinite. Previous stability results are confirmed by expanding around optimal fixed points, where a closed-form solution to the learning dynamics is obtained. However, stochastic effects are shown to stabilize otherwise unstable suboptimal fixed points. Conditions required to destabilize one such fixed point are obtained for the case of a single nongaussian component, indicating that the initial learning rate η required to escape successfully is very low (η = O(N−2) where N is the data dimension), resulting in very slow learning typically requiring O(N3) iterations. Simulations confirm that this picture holds for a finite system.