Keras Implementation of Customized Loss Function that need internal layer output as label

Question

in keras, I want to customize my loss function which not only takes (y_true, y_pred) as input but also need to use the output from the internal layer of the network as the label for an output layer.This picture shows the Network Layout

Here, the internal output is xn, which is a 1D feature vector. in the upper right corner, the output is xn', which is the prediction of xn. In other words, xn is the label for xn'.

While [Ax, Ay] is traditionally known as y_true, and [Ax',Ay'] is y_pred.

I want to combine these two loss components into one and train the network jointly.

Any ideas or thoughts are much appreciated!

Lemuellemuela

Answer 1

I have figured out a way out, in case anyone is searching for the same, I posted here (based on the network given in this post):

The idea is to define the customized loss function and use it as the output of the network. (Notation: A is the true label of variable A, and A' is the predicted value of variable A)

def customized_loss(args):
    #A is from the training data
    #S is the internal state
    A, A', S, S' = args 
    #customize your own loss components
    loss1 = K.mean(K.square(A - A'), axis=-1)
    loss2 = K.mean(K.square(S - S'), axis=-1)
    #adjust the weight between loss components
    return 0.5 * loss1 + 0.5 * loss2

 def model():
     #define other inputs
     A = Input(...) # define input A
     #construct your model 
     cnn_model = Sequential()
     ...
     # get true internal state
     S = cnn_model(prev_layer_output0)
     # get predicted internal state output
     S' = Dense(...)(prev_layer_output1)
     # get predicted A output
     A' = Dense(...)(prev_layer_output2)
     # customized loss function
     loss_out = Lambda(customized_loss, output_shape=(1,), name='joint_loss')([A, A', S, S'])
     model = Model(input=[...], output=[loss_out])
     return model

  def train():
      m = model()
      opt = 'adam'
      model.compile(loss={'joint_loss': lambda y_true, y_pred:y_pred}, optimizer = opt)
      # train the model 
      ....

Lemuellemuela

I know this has been quiet for a while now but can you explain what A and A' are? A' is the prediction of A is it not? A should be the actual expected result so why is it defined as an input instead of a numpy array that takes on values? – Coward

why is the loss_out is given as the output of the model – Apprehend

Answer 2

First of all you should be using the Functional API. Then you should define the network output as the output plus the result from the internal layer, merge them into a single output (by concatenating), and then make a custom loss function that then splits the merged output into two parts and does the loss computations on its own.

Something like:

def customLoss(y_true, y_pred):
    #loss here
    internalLayer = Convolution2D()(inputs) #or other layers
    internalModel = Model(input=inputs, output=internalLayer)
    tmpOut = Dense(...)(internalModel)
    mergedOut = merge([tmpOut, mergedOut], mode = "concat", axis = -1)
    fullModel = Model(input=inputs, output=mergedOut)

    fullModel.compile(loss = customLoss, optimizer = "whatever")

Interinsurance

Thanks for your reply! I think my main concern is how to form my loss function. Yes, we can combine both output into one, but when calculating the loss for internal output (xn'), the customLoss needs access to the model itself in order to get the label for the internal output (xn). xn is not the training data but some transformation of the training data processed by the model. – Lemuellemuela

@LiJuekun can't you just put the "internal label" into the y value that you pass to fit? – Interinsurance

seems there an indentation problem in the code snippet? – Lavatory

Answer 3

I have my reservations regarding this implementation. The loss computed at the merged layer is propagated back to both the merged branches. Generally you would like to propagate it through just one layer.

Araby