PyBrain: When creating network from ground up how and where do you create a bias?

Question

Following the PyBrain documentation, Building Networks with Modules and Connections, I'm building a neural network piecewise (in contrast to using the buildNetwork shortcut). I'm constructing a simple 3-layer (input, hidden, output) neural network. How do I properly add a bias unit?

I'm guessing I construct a BiasUnit module as in:

b = BiasUnit(name='bias')
network.addModule(b)

Is this the right way? Do I have to create FullConnection object? If so, what should I be connecting?

Lowerclassman

As much as I love python, I've switched to using the C-based fanntool which blows PyBrain out of the water in terms of performance. – Lowerclassman

Answer 1

Realized PyBrain is open source and I have the source code sitting in my Python directory. I opened the C:\Python27\Lib\site-packages\pybrain\tools\shortcuts.py file. Inside this file I located the buildNetwork function and saw how it adds BiasUnit's. The relevant code is here:

...
n = Network()
# linear input layer
n.addInputModule(LinearLayer(layers[0], name='in'))
# output layer of type 'outclass'
n.addOutputModule(opt['outclass'](layers[-1], name='out'))
if opt['bias']:
    # add bias module and connection to out module, if desired
    n.addModule(BiasUnit(name='bias'))
    if opt['outputbias']:
        n.addConnection(FullConnection(n['bias'], n['out']))
# arbitrary number of hidden layers of type 'hiddenclass'
for i, num in enumerate(layers[1:-1]):
    layername = 'hidden%i' % i
    n.addModule(opt['hiddenclass'](num, name=layername))
    if opt['bias']:
        # also connect all the layers with the bias
        n.addConnection(FullConnection(n['bias'], n[layername]))
# connections between hidden layers
...

Basically it looks like it creates a single BiasUnit and connects it to each hidden layer and optionally to the output layer as well.

Lowerclassman

Good detective work. Note the buildNetwork is just a shortcut and in the API documentation they discuss building a network (look at the docs for Network) – Hollenbeck

Answer 2

Here you have a simple example:

n = RecurrentNetwork()
n.addModule(TanhLayer(hsize, name = 'h'))
n.addModule(BiasUnit(name = 'bias'))
n.addOutputModule(LinearLayer(1, name = 'out'))
n.addConnection(FullConnection(n['bias'], n['h']))
n.addConnection(FullConnection(n['h'], n['out']))
n.sortModules()

Note that the BiasUnit is connected to TanhLayer effectively making the h layer a layer with bias.

Peerage