When using RMSE loss in TensorFlow I receive very small loss values smalerl than 1 [closed]

Question

Hello I have a network that produces logits / outputs like this:

logits = tf.placeholder(tf.float32, [None, 128, 64, 64]) // outputs
y = tf.placeholder(tf.float32, [None, 128, 64, 64]) // ground_truth, targets

--> y ground truth values are downscaled from [0, 255] to [0, 1] in order to increase perforce as I have read it is better to use range [0, 1]

Now I want to calculate the RMSE / EuclideanLoss like this:

loss = tf.reduce_mean(tf.square(logits - y))

or

loss = tf.sqrt(tf.reduce_mean(tf.square(tf.subtract(y, logits))))

not sure which one is better.

When doing so my loss values start at roughly 1. and then quickly go down to 2.5e-4. When I use the EuclideanLoss in Caffe for the same network my loss values starts from roughly 1000 and goes down to 200. Am I doing anything wrong in Tensorflow or why are the loss values that small? I can't really track the loss values in tensorboard since they are so small. Can anyone help me?

Ursuline

It looks like you're computing the output between two placeholders and not doing any treatment to logits before using it in the loss, is that true? Could you show a bit more of the code? – Shorten

Oh no! the placeholders are just for illustration! My last layer looks roughly like this: logits = tf.nn.conv2d(inputs, weights, [1,strides,strides,1], padding='VALID', data_format='NHWC') @FlorentinHennecker – Ursuline

Answer 1

The first loss you propose is the better one (the second introduces an unnecessary sqrt).

Values equal to or smaller than 1 seem to be the only possible values since the range of values you provide is within 0 and 1 - so the biggest possible error is 1.

If you have trouble visualising the loss in tensorboard, try showing the graphs using a log scale (one of the two buttons under the graphs)

Shorten

Problem ist my error starts with a value greater than 1 which confuses me... Plus my values get so small that the value is zero since the values are like value.e-10. – Ursuline

You might want to show your code then – Shorten