Monthly
288 pp. per issue
6 x 9, illustrated
ISSN
0899-7667
E-ISSN
1530-888X
2014 Impact factor:
2.21

Neural Computation

November 1992, Vol. 4, No. 6, Pages 839-853
(doi: 10.1162/neco.1992.4.6.839)
© 1992 Massachusetts Institute of Technology
Maximum Entropy and Learning Theory
Article PDF (613.93 KB)
Abstract

We derive the learning theory recently reported by Tishby, Levin, and Solla (TLS) directly from the principle of maximum entropy instead of statistical mechanics. The theory generally applies to any problem of modeling data. We analyze an elementary example for which we find the predictions consistent with intuition and conventional statistical results and we numerically examine the more realistic problem of training a competitive net to learn a one-dimensional probability density from samples. The TLS theory is useful for predicting average training behavior.