How to interpret increase in both loss and accuracy

Question

I have run deep learning models(CNN's) using tensorflow. Many times during the epoch, i have observed that both loss and accuracy have increased, or both have decreased. My understanding was that both are always inversely related. What could be scenario where both increase or decrease simultaneously.

Criticism

Are you referring to validation set loss and accuracy or training set loss and accuracy. – Aperture

Answer 1

The loss decreases as the training process goes on, except for some fluctuation introduced by the mini-batch gradient descent and/or regularization techniques like dropout (that introduces random noise).

If the loss decreases, the training process is going well.

The (validation I suppose) accuracy, instead, it's a measure of how good the predictions of your model are.

If the model is learning, the accuracy increases. If the model is overfitting, instead, the accuracy stops to increase and can even start to decrease.

If the loss decreases and the accuracy decreases, your model is overfitting.

If the loss increases and the accuracy increase too is because your regularization techniques are working well and you're fighting the overfitting problem. This is true only if the loss, then, starts to decrease whilst the accuracy continues to increase. Otherwise, if the loss keep growing your model is diverging and you should look for the cause (usually you're using a too high learning rate value).

Holothurian

My experience is with an increasing val_loss and a decreasing val_acc in the early steps of the training, i assume the model isn't improving at all. Are there guidelines (or current streams) for discerning data-set symptoms from network architecture problems? – Rossiter

AFAIK there's not a comprehensive list of tips to follow to "debug" the training phase. But this can be an idea for my next blog post, though – Holothurian

this explanation doesn't match my understanding of accuracy. See the answer by @nimi, or this blog article: jussihuotari.com/2018/01/17/… – Elsey

Answer 2

I think the top-rated answer is incorrect.

I will assume you are talking about cross-entropy loss, which can be thought of as a measure of 'surprise'.

Loss and accuracy increasing/decreasing simultaneously on the training data tells you nothing about whether your model is overfitting. This can only be determined by comparing loss/accuracy on the validation vs. training data.

If loss and accuracy are both decreasing, it means your model is becoming more confident on its correct predictions, or less confident on its incorrect predictions, or both, hence decreased loss. However, it is also making more incorrect predictions overall, hence the drop in accuracy. Vice versa if both are increasing. That is all we can say.

Kristopher

Really helpful and intuitive explanation. A more detailed explanation may be found here if someone wants to get a deeper insight of this problem: stats.stackexchange.com/questions/282160/… – Sideslip

Answer 3

I'd like to add a possible option here for all those who struggle with a model training right now.

If your validation data is a bit dirty, you might experience that in the beginning of the training the validation loss is low as well as the accuracy, and the more you train your network, the accuracy increases with the loss side by side. The reason why it happens, because it finds the possible outliers of your dirty data and gets a super high loss there. Therefore, your accuracy will grow as it guesses more data right, but the loss grows with it.

Garnishee

Answer 4

This is just what I think based on the math behind the loss and the accuracy,

Note :-

I expect your data is categorical

Your models output :-

[0.1,0.9,0.9009,0.8] (used to calculate loss)

Maxed output :-

[0,0,1,0] (used to calculate acc )

Expected output :-

[0,1,0,0]

Lets clarify what loss and acc calculates :

Loss :- The overall error of y and ypred

Acc :- Just if y and maxed(ypred) is equal

So in a overall our model almost nailed it , resulting in a low loss

But in maxed output no overall is seen its just that they should completely match ,

If they completely match :-

1

else:

0

Thus resulting in a low accuracy too

Try to check mae of the model

remove regularization

check if your are using correct loss

Lachrymal

Answer 5

You should check your class index (both train and valid) in training process. It might be sorted in different ways. I have this problem in colab.

Sundown

Does correcting the class index solve the problem? – Stare