| |
Abstract:
We describe an algorithm for learning an optimal set of basis
functions for modeling data with sparse structure. A principal
challenge in learning the basis functions is presented by the fact
that when the basis set is overcomplete, the posterior probability
distribution over the coefficients for each data point is difficult
to integrate (which is a necessary step in the learning procedure).
Previous attempts at approximating the posterior typically do not
properly capture its full volume, and as the prior over the
coefficients is pushed to capture higher degrees of sparseness
(i.e., probability more tightly peaked at zero), the problems
associated with these approximations become exacerbated. Here, we
address this problem by using a mixture-of-Gaussians prior on the
coefficients. The prior is formed from a linear combination of two
or more Gaussian distributions: one Gaussian captures the peak at
zero while the others capture the spread over the non-zero
coefficient values. We show that when the prior is in such a form,
there exist efficient methods for learning the basis functions as
well as parameters of the prior. The performance of the algorithm
is demonstrated on natural images, showing similar results to those
obtained with other sparse coding algorithms. Importantly, though,
since the parameters of the prior are adapted to the data, no
assumption about sparse structure in the images need be made a
priori, rather it is learned from the data
|
