Make a custom loss function in keras

Question

Hi I have been trying to make a custom loss function in keras for dice_error_coefficient. It has its implementations in tensorboard and I tried using the same function in keras with tensorflow but it keeps returning a NoneType when I used model.train_on_batch or model.fit where as it gives proper values when used in metrics in the model. Can please someone help me out with what should i do? I have tried following libraries like Keras-FCN by ahundt where he has used custom loss functions but none of it seems to work. The target and output in the code are y_true and y_pred respectively as used in the losses.py file in keras.

def dice_hard_coe(target, output, threshold=0.5, axis=[1,2], smooth=1e-5):
    """References
    -----------
    - `Wiki-Dice <https://en.wikipedia.org/wiki/Sørensen–Dice_coefficient>`_
    """

    output = tf.cast(output > threshold, dtype=tf.float32)
    target = tf.cast(target > threshold, dtype=tf.float32)
    inse = tf.reduce_sum(tf.multiply(output, target), axis=axis)
    l = tf.reduce_sum(output, axis=axis)
    r = tf.reduce_sum(target, axis=axis)
    hard_dice = (2. * inse + smooth) / (l + r + smooth)
    hard_dice = tf.reduce_mean(hard_dice)
    return hard_dice

Fatty

Answer 1

In addition, you can extend an existing loss function by inheriting from it. For example masking the BinaryCrossEntropy:

class MaskedBinaryCrossentropy(tf.keras.losses.BinaryCrossentropy):
    def call(self, y_true, y_pred):
        mask = y_true != -1
        y_true = y_true[mask]
        y_pred = y_pred[mask]
        return super().call(y_true, y_pred)

A good starting point is the custom log guide: https://www.tensorflow.org/guide/keras/train_and_evaluate#custom_losses

Organization

Answer 2

There are two steps in implementing a parameterized custom loss function in Keras. First, writing a method for the coefficient/metric. Second, writing a wrapper function to format things the way Keras needs them to be.

1. It's actually quite a bit cleaner to use the Keras backend instead of tensorflow directly for simple custom loss functions like DICE. Here's an example of the coefficient implemented that way:

   import keras.backend as K
   def dice_coef(y_true, y_pred, smooth, thresh):
       y_pred = y_pred > thresh
       y_true_f = K.flatten(y_true)
       y_pred_f = K.flatten(y_pred)
       intersection = K.sum(y_true_f * y_pred_f)
   
       return (2. * intersection + smooth) / (K.sum(y_true_f) + K.sum(y_pred_f) + smooth)
   

2. Now for the tricky part. Keras loss functions must only take (y_true, y_pred) as parameters. So we need a separate function that returns another function.

   def dice_loss(smooth, thresh):
     def dice(y_true, y_pred)
       return -dice_coef(y_true, y_pred, smooth, thresh)
     return dice
   

Finally, you can use it as follows in Keras compile.

# build model 
model = my_model()
# get the loss function
model_dice = dice_loss(smooth=1e-5, thresh=0.5)
# compile model
model.compile(loss=model_dice)

Retired

'def dice(y_true, y_pred): return -dice_coef(y_true, y_pred, 1e-5, 0.5) def dice_coef(y_true, y_pred, smooth, thresh): y_pred = K.cast(y_pred > thresh ,dtype=tf.float32) y_true = K.cast(y_true > thresh, dtype=tf.float32) y_true_f = K.flatten(y_true) y_pred_f = K.flatten(y_pred) intersection = K.sum(y_true_f * y_pred_f) return (2. * intersection + smooth) / (K.sum(y_true_f) + K.sum(y_pred_f) + smooth) Final_Model.compile(optimizer=opt, loss=dice,metrics=['acc'])' this gives me an error 'Tried to convert 'x' to a tensor and failed. Error: None values not supported.' – Fatty

are you calling an instance of dice before passing it to your model? dice_fn = dice(smooth=1e-5, thresh=0.5) Final_Model.compile(optimizer=opt, loss=dice_fn) – Retired

It worked just the way you said. The problem is the output cannot be casted to 1,0 that way. So it was returning a None Value. On removing that part it worked perfectly. Thanks for you help. – Fatty

Hi, can anyone here help in my question: #53651190 Thanks, – Joliejoliet

Can someone confirm if this works? ypred and ytrue are Tensors, so ypred = ypred > ytrue will fail. Any computation on ypred and ytrue fails in my case. – Joey

Why the negative return of the dice_coef function? – Harod

There's no need for a "tricky part", if you can choose the argument order of dice_coeff. With functools.partial you can pass a subset of the arguments, but only the initial arguments. Hence, y_true, y_pred would have to be the last arguments of dice_coeff – Dyad

how can you call dice function without giving y_true and y_pred in dice_loss function? – Navicular

@T.Nair Also, how does this work when we multiple losses for a single output? – Navicular

in my case I want to use the input to compute the loss I have used this method and give the top function the input layer, but it's give an error, symbolic error trying to convert numpy or kerastensor to tensor – Troglodyte

@Harod I believe it's a negative because it's a minimization function – Rahmann

Answer 3

According to the documentation, you can use a custom loss function like this:

Any callable with the signature loss_fn(y_true, y_pred) that returns an array of losses (one of sample in the input batch) can be passed to compile() as a loss. Note that sample weighting is automatically supported for any such loss.

As a simple example:

def my_loss_fn(y_true, y_pred):
    squared_difference = tf.square(y_true - y_pred)
    return tf.reduce_mean(squared_difference, axis=-1)  # Note the `axis=-1`

model.compile(optimizer='adam', loss=my_loss_fn)

Complete example:

import tensorflow as tf
import numpy as np

def my_loss_fn(y_true, y_pred):
    squared_difference = tf.square(y_true - y_pred)
    return tf.reduce_mean(squared_difference, axis=-1)  # Note the `axis=-1`

model = tf.keras.Sequential([
    tf.keras.layers.Dense(8, activation='relu'),
    tf.keras.layers.Dense(16, activation='relu'),
    tf.keras.layers.Dense(1)])

model.compile(optimizer='adam', loss=my_loss_fn)

x = np.random.rand(1000)
y = x**2

history = model.fit(x, y, epochs=10)

Tieratierce