What function defines accuracy in Keras when the loss is mean squared error (MSE)?

Question

How is Accuracy defined when the loss function is mean square error? Is it mean absolute percentage error?

The model I use has output activation linear and is compiled with loss= mean_squared_error

model.add(Dense(1))
model.add(Activation('linear'))  # number

model.compile(loss='mean_squared_error', optimizer='adam', metrics=['accuracy'])

and the output looks like this:

Epoch 99/100
1000/1000 [==============================] - 687s 687ms/step - loss: 0.0463 - acc: 0.9689 - val_loss: 3.7303 - val_acc: 0.3250
Epoch 100/100
1000/1000 [==============================] - 688s 688ms/step - loss: 0.0424 - acc: 0.9740 - val_loss: 3.4221 - val_acc: 0.3701

So what does e.g. val_acc: 0.3250 mean? Mean_squared_error should be a scalar not a percentage - shouldnt it? So is val_acc - mean squared error, or mean percentage error or another function?

From definition of MSE on wikipedia:https://en.wikipedia.org/wiki/Mean_squared_error

The MSE is a measure of the quality of an estimator—it is always non-negative, and values closer to zero are better.

Does that mean a value of val_acc: 0.0 is better than val_acc: 0.325?

edit: more examples of the output of accuracy metric when I train - where the accuracy is increase as I train more. While the loss function - mse should decrease. Is Accuracy well defined for mse - and how is it defined in Keras?

lAllocator: After 14014 get requests, put_count=14032 evicted_count=1000 eviction_rate=0.0712657 and unsatisfied allocation rate=0.071714
1000/1000 [==============================] - 453s 453ms/step - loss: 17.4875 - acc: 0.1443 - val_loss: 98.0973 - val_acc: 0.0333
Epoch 2/100
1000/1000 [==============================] - 443s 443ms/step - loss: 6.6793 - acc: 0.1973 - val_loss: 11.9101 - val_acc: 0.1500
Epoch 3/100
1000/1000 [==============================] - 444s 444ms/step - loss: 6.3867 - acc: 0.1980 - val_loss: 6.8647 - val_acc: 0.1667
Epoch 4/100
1000/1000 [==============================] - 445s 445ms/step - loss: 5.4062 - acc: 0.2255 - val_loss: 5.6029 - val_acc: 0.1600
Epoch 5/100
783/1000 [======================>.......] - ETA: 1:36 - loss: 5.0148 - acc: 0.2306

Raguelragweed

I think you are massively confused. Accuracy is not defined for regression problems, the mean squared error you see its not a percentage, that is the actual value you get, it might be less than one. – Suffer

Yes Matias, I am confused. Because the accuracy for my regression problem - is behaving like a percentage. It starts of on the first epoch of training as acc: 0.0 - and increases up to acc: 0.99. If acc actually was mse - then I would have gotten the best result when starting the first epoch right? Mse of 0.0 is better then 0.99 - thats why i dont think acc is mse but percentage. But how is that percentage defined? is it mean absolute percentage error? – Raguelragweed

Accuracy is just the fraction of correctly classified examples, which is always the fraction where label == prediction is true. For regression this makes no sense as the chance that the predicted value is exactly the same as the label is very small, but seems that your model can do this with big accuracy. The accuracy in Keras is never mean absolute error, but just as I mentioned previously. – Suffer

Since I dont have a classification problem - but a regression problem - what is Keras then outputting when I ask for the accuracy metric? Because it still looks like a sensible output - while accuracy as percentage for loss function mean squared error is not sensible? It should be a scalar where values closer to 0 is better. The opposite of what Im seen from accuracy when I train. – Raguelragweed

Again, the answer is the same, the accuracy in Keras does not change if its regression or classification, its always fraction where label == predicted. It is behaving correctly, your interpretation of it is what is wrong. – Suffer

So for two floating points to be equal such that label == predicted then they have to be equal to a high decimal precision. Do you know if the equal constrained is relaxed when the label is floating points? – Raguelragweed

No, its not, that's why accuracy makes no sense for regression. – Suffer

This is relevant here: "Integer Data: Categorical or Continuous? A: It really depends on context. If the integer variable has some inherent ordering to it, for example it could be colours where lower numbers represent "darker shades" and higher numbers represented "lighter shades", then treating it as a continuous variable is almost certainly preferable. Not only would it make more sense, but you're eliminating some 200 variables from your model which is a huge bonus" stats.stackexchange.com/questions/261396/… – Raguelragweed

I don't see the relevance, this does not change that accuracy is meaningless for regression. If you use custom metrics, do not call them "accuracy". – Suffer

"We obtained high prediction accuracy for ... sea age (86.99%), but lower accuracy for river age (63.20%)" ref: sciencedirect.com/science/article/pii/S1574954121001138 From peer-reviewed publication where I use regression on age - and accuracy. – Raguelragweed

"Accuracy of the regression tasks was calculated by rounding the prediction to the nearest integer age and comparing it with the ground truth." This is not what Keras does when computing accuracy, and not the standard meaning for accuracy in the ML literature. That this is part of a peer-review publication does not mean it is correct :) – Suffer

Answer 1

There are at least two separate issues with your question.

The first one should be clear by now from the comments by Dr. Snoopy and the other answer: accuracy is meaningless in a regression problem, such as yours; see also the comment by patyork in this Keras thread. For good or bad, the fact is that Keras will not "protect" you or any other user from putting not-meaningful requests in your code, i.e. you will not get any error, or even a warning, that you are attempting something that does not make sense, such as requesting the accuracy in a regression setting.

Having clarified that, the other issue is:

Since Keras does indeed return an "accuracy", even in a regression setting, what exactly is it and how is it calculated?

To shed some light here, let's revert to a public dataset (since you do not provide any details about your data), namely the Boston house price dataset (saved locally as housing.csv), and run a simple experiment as follows:

import numpy as np
import pandas
import keras

from keras.models import Sequential
from keras.layers import Dense

# load dataset
dataframe = pandas.read_csv("housing.csv", delim_whitespace=True, header=None)
dataset = dataframe.values
# split into input (X) and output (Y) variables
X = dataset[:,0:13]
Y = dataset[:,13]

model = Sequential()
model.add(Dense(13, input_dim=13, kernel_initializer='normal', activation='relu'))
model.add(Dense(1, kernel_initializer='normal'))
# Compile model asking for accuracy, too:
model.compile(loss='mean_squared_error', optimizer='adam', metrics=['accuracy'])

model.fit(X, Y,
     batch_size=5,
     epochs=100,
     verbose=1)

As in your case, the model fitting history (not shown here) shows a decreasing loss, and an accuracy roughly increasing. Let's evaluate now the model performance in the same training set, using the appropriate Keras built-in function:

score = model.evaluate(X, Y, verbose=0)
score
# [16.863721372581754, 0.013833992168483997]

The exact contents of the score array depend on what exactly we have requested during model compilation; in our case here, the first element is the loss (MSE), and the second one is the "accuracy".

At this point, let us have a look at the definition of Keras binary_accuracy in the metrics.py file:

def binary_accuracy(y_true, y_pred):
    return K.mean(K.equal(y_true, K.round(y_pred)), axis=-1)

So, after Keras has generated the predictions y_pred, it first rounds them, and then checks to see how many of them are equal to the true labels y_true, before getting the mean.

Let's replicate this operation using plain Python & Numpy code in our case, where the true labels are Y:

y_pred = model.predict(X)
l = len(Y)
acc = sum([np.round(y_pred[i])==Y[i] for i in range(l)])/l
acc
# array([0.01383399])

Well, bingo! This is actually the same value returned by score[1] above...

To make a long story short: since you (erroneously) request metrics=['accuracy'] in your model compilation, Keras will do its best to satisfy you, and will return some "accuracy" indeed, calculated as shown above, despite this being completely meaningless in your setting.

---

There are quite a few settings where Keras, under the hood, performs rather meaningless operations without giving any hint or warning to the user; two of them I have happened to encounter are:

• Giving meaningless results when, in a multi-class setting, one happens to request loss='binary_crossentropy' (instead of categorical_crossentropy) with metrics=['accuracy'] - see my answers in Keras binary_crossentropy vs categorical_crossentropy performance? and Why is binary_crossentropy more accurate than categorical_crossentropy for multiclass classification in Keras?

• Disabling completely Dropout, in the extreme case when one requests a dropout rate of 1.0 - see my answer in Dropout behavior in Keras with rate=1 (dropping all input units) not as expected

Randellrandene

Answer 2

The loss function (Mean Square Error in this case) is used to indicate how far your predictions deviate from the target values. In the training phase, the weights are updated based on this quantity. If you are dealing with a classification problem, it is quite common to define an additional metric called accuracy. It monitors in how many cases the correct class was predicted. This is expressed as a percentage value. Consequently, a value of 0.0 means no correct decision and 1.0 only correct decisons. While your network is training, the loss is decreasing and usually the accuracy increases.

Note, that in contrast to loss, the accuracy is usally not used to update the parameters of your network. It helps to monitor the learning progress and the current performane of the network.

Epigene

BGraf, as you can see from the code in my post - i dont have a classification problem, but a regression problem. And my loss is mse - and the output from Keras training ive asked to be accuracy. But what does accuracy mean when the problem is a regression problem, and not a classification problem? – Raguelragweed

in case of a regression problem there is no reason for calculating an accuracy metric since there are no "wrong" or "right" decisions to measure. If you are predicting continous targets then the accuracy metric in keras has no relevant meaning. You should use a quantity like mean square error to evaluate the performance of your network. – Epigene

Yes, it does! Im working on predicting age of fish based on images of otoliths, and MSE loss function is good because that imposes a total-ordering on the predictions. E.g. an otolith of age 3 is more similar to age 2 or 3 then say age 7 or 8. While in order to compare the performance of the network with human experts accuracy is more useful, as that is how experts evaluates themselfs. Maybe its an unusual use-case but definitly a reason to calculate accuracy. – Raguelragweed

@EndreMoen What you are computing is not accuracy as it is understood in Machine Learning, you are computing some custom metric that is not really accuracy, you are getting confused by this fact. – Suffer

Answer 3

@desertnaut has said it very clearly.

Consider the following two pieces of code

compile code

binary_accuracy code

def binary_accuracy(y_true, y_pred):
    return K.mean(K.equal(y_true, K.round(y_pred)), axis=-1)

Your labels should be integer，Because keras does not round y_true, and you get high accuracy.......

Dewar