Restricted Boltzmann Machine for real-valued data - gaussian linear units (glu) -

Question

I want my Restricted Boltzmann Machine to learn a new representation of real-valued data (see: Hinton - 2010 - A Practical Guide to Training RBMs). I'm struggling with an implementation of Gaussian linear units.

With Gaussian linear units in the visible layer the energy changes to E(v,h)= ∑ (v-a)²/2σ - ∑ bh - ∑v/σ h w. Now I don't know how to change the Contrastive Divergence Learning Algorithm. The visible units won't be sampled any more as they are linear. I use the expectation (mean-fied activation) p(v_i=1|h)= a +∑hw + N(0,1) as their state. The associations are left unchangend ( pos: data*p(h=1|v)' neg: p(v=1|h)*p(h=1|v)' ). But this only leads to random noise when I want to reconstruct the data. The error rate will stop improving around 50%.

Finally I want to use Gaussian linear units in both layers. How will I get the states of the hidden units then? I suggest by using the mean-field activation p(h_i=1|v)= b +∑vw + N(0,1) but I'm not sure.

Concoct

Answer 1

You could take a look at the gaussian RBM that Hinton himself has provided Please find it here. http://www.cs.toronto.edu/~hinton/code/rbmhidlinear.m

Savoyard

Note that link-only answers are discouraged, SO answers should be the end-point of a search for a solution (vs. yet another stopover of references, which tend to get stale over time). Please consider adding a stand-alone synopsis here, keeping the link as a reference. – Blamable

Useful example. It's slightly different than what OP wanted - the hidden layer is linear-gaussian and the visible is bernoulli and sigma is not learned. – Bullwhip

Answer 2

I have been working on a similar project implementing an RBM with c++ and matlab mexfunction. i have found from the implementation of Professor Hinton (in matlab) that the the binary activation for visible units uses the simoid function. ie : vs = sigmoid(bsxfun(@plus, hs*obj.W2', obj.b)); But when implementing the gaussian visible units RBM you simply have to sample the visible units without using the simoid. ie: vs = bsxfun(@plus, h0*obj.W2', obj.b); or better look at the Professor Hinton implementation (see: http://www.cs.toronto.edu/~hinton/code/rbmhidlinear.m)

Affliction