Theano HiddenLayer Activation Function

Question

Is there anyway to use Rectified Linear Unit (ReLU) as the activation function of the hidden layer instead of tanh() or sigmoid() in Theano? The implementation of the hidden layer is as follows and as far as I have searched on the internet ReLU is not implemented inside the Theano.

class HiddenLayer(object):
  def __init__(self, rng, input, n_in, n_out, W=None, b=None, activation=T.tanh):
    pass

Airwaves

Answer 1

relu is easy to do in Theano:

switch(x<0, 0, x)

To use it in your case make a python function that will implement relu and pass it to activation:

def relu(x):
    return theano.tensor.switch(x<0, 0, x)
HiddenLayer(..., activation=relu)

Some people use this implementation: x * (x > 0)

UPDATE: Newer Theano version have theano.tensor.nnet.relu(x) available.

Frohman

How is the non differentiability at zero dealt with here? – Heathcote

@Frohman I just installed Theano on my laptop. The library does not include nnet.relu. However, I can use nnet.relu on a desktop machine on which I installed Theano a few days back. What could be the reason? – Fetid

@Amir, this is because they don't have the same Theano version. The one without relu use the last released Theano version 0.7, the one with relu use the development version (it is stable and we recommand people to use it): deeplearning.net/software/theano/… – Frohman

Answer 2

UPDATE: Latest version of theano has native support of ReLU: T.nnet.relu, which should be preferred over custom solutions.

I decided to compare the speed of solutions, since it is very important for NNs. Compared speed of function itself and it's gradient, in first case switch is preferred, the gradient is faster for x * (x>0). All the computed gradients are correct.

def relu1(x):
    return T.switch(x<0, 0, x)

def relu2(x):
    return T.maximum(x, 0)

def relu3(x):
    return x * (x > 0)


z = numpy.random.normal(size=[1000, 1000])
for f in [relu1, relu2, relu3]:
    x = theano.tensor.matrix()
    fun = theano.function([x], f(x))
    %timeit fun(z)
    assert numpy.all(fun(z) == numpy.where(z > 0, z, 0))

Output: (time to compute ReLU function)
>100 loops, best of 3: 3.09 ms per loop
>100 loops, best of 3: 8.47 ms per loop
>100 loops, best of 3: 7.87 ms per loop

for f in [relu1, relu2, relu3]:
    x = theano.tensor.matrix()
    fun = theano.function([x], theano.grad(T.sum(f(x)), x))
    %timeit fun(z)
    assert numpy.all(fun(z) == (z > 0)

Output: time to compute gradient 
>100 loops, best of 3: 8.3 ms per loop
>100 loops, best of 3: 7.46 ms per loop
>100 loops, best of 3: 5.74 ms per loop

Finally, let's compare to how gradient should be computed (the fastest way)

x = theano.tensor.matrix()
fun = theano.function([x], x > 0)
%timeit fun(z)
Output:
>100 loops, best of 3: 2.77 ms per loop

So theano generates inoptimal code for gradient. IMHO, switch version today should be preferred.

Canter

Is this from here? Note that when you care about the GPU speed, T.maximum is the fastest. See also here. – Mollescent

@Albert, no, I decided to compare the versions I found here (unfortunately I don't have GPU, so these are CPU results). Thanks for the first link! – Canter

Some follow up discussion about the speed is here. – Mollescent

It seems, that relu is absent in theano.tensor.nnet According to deeplearning.net/software/theano/library/tensor/nnet/… It should be present in 0.7 But I have pip3 show theano --- Name: Theano Version: 0.7.0 Location: /Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages and In [5]: theano.tensor.nnet.relu ----> 1 theano.tensor.nnet.relu AttributeError: 'module' object has no attribute 'relu' Has anybody faced with this? – Spondylitis

I've just got clue about it. If install theano using pip(even if install from scratch and use -I option), 0.7 stable version would be installed. But for now(at the moment I'm writing this comment) relu is absent there. But if install bleeding-edge version: pip3 install --upgrade --no-deps git+git://github.com/Theano/Theano.git Then relu appears in theano.tensor.nnet – Spondylitis

Answer 3

I think it is more precise to write it in this way:

x * (x > 0.) + 0. * (x < 0.)

Akers

0. * (x < 0.) will get optimized away. So the executed formula will be x * (x > 0.) – Frohman

Answer 4

I wrote it like this:

lambda x: T.maximum(0,x)

or:

lambda x: x * (x > 0)

Eger

Answer 5

The function is very simple in Python:

def relu(input):
    output = max(input, 0)
    return(output)

Photomap