How to implement the derivative of Leaky Relu in python?

Question

How would I implement the derivative of Leaky ReLU in Python without using Tensorflow?

Is there a better way than this? I want the function to return a numpy array

def dlrelu(x, alpha=.01):
     # return alpha if x < 0 else 1

     return np.array ([1 if i >= 0 else alpha for i in x])

Thanks in advance for the help

Tintoretto

What is x? Is it an array already? – Ecosystem

Where is TensorFlow in your code? – Margenemargent

@Margenemargent "without using Tensorflow" – Ecosystem

@Ecosystem Oops! Sorry, my bad! – Margenemargent

Yes, x is n numpy array! @Ecosystem – Summerville

Answer 1

The method you use works, but strictly speaking you are computing the derivative with respect to the loss, or lower layer, so it might be wise to also pass the value from lower layer to compute the derivative (dl/dx).

Anyway, you can avoid using the loop which is more efficient for large x. This is one way to do it:

def dlrelu(x, alpha=0.01):
  dx = np.ones_like(x)
  dx[x < 0] = alpha
  return dx

If you passed the error from lower layer, it looks like this:

def dlrelu(dl, x, alpha=0.01):
  """ dl and x have same shape. """
  dx = np.ones_like(x)
  dx[x < 0] = alpha
  return dx*dl

Cordy

Thanks a lot. I´m doing this multiplication "dx*dl" outside of the defined function... – Summerville

Answer 2

A simple convenience function:

def leakyReLU_deriv(x, alpha=0.01):
    return np.where(x>0, 1, alpha) 

Oriel