Tensorflow: How to convert NaNs to a number?

Question

I'm trying to calculate the entropy of the weights while training my graph and to use it for regularization. This of course involves w*tf.log(w), and as my weights are changing some of them are bound to get into a region which results in NaNs being returned.

Ideally I would include a line in my graph setup:

w[tf.is_nan(w)] = <number>

but tensorflow doesn't support assigning like that. I could of course create an operation, but that wouldn't work because I need for it to happen during the execution of the entire graph. I can't wait for the graph to execute and then 'fix' my weights, is has to be part of the graph execution.

I haven't been able to find an equivalent to np.nan_to_num in the docs.

Anybody have an idea?

(For obvious reasons, adding an epsilon doesn't work)

Demello

Try to avoid NaNs in the first place. – Furor

Answer 1

I think you need to use tf.select.

w = tf.select(tf.is_nan(w), tf.ones_like(w) * NUMBER, w); #if w is nan use 1 * NUMBER else use element in w

---

Update: TensorFlow 1.0 has deprecated tf.select in favor of Numpy compatible tf.where.

Smirk

w = tf.where(tf.is_nan(w), tf.ones_like(w) * NUMBER, w) # version with tf.where - to copy faster and avoid confusion – Elviselvish

For Tensorflow 2.0+ you need to use tf.math.is_nan. This code removes nans that where created by the padding portion of categorically_cross_entropy loss function: loss_value = tf.where(tf.math.is_nan(loss_value), tf.zeros_like(loss_value), loss_value) – Rudyrudyard

Answer 2

I use this to convert a value to 0 if it's NaN:

value_not_nan = tf.dtypes.cast(tf.math.logical_not(tf.math.is_nan(value)), dtype=tf.float32)
tf.math.multiply_no_nan(value, value_not_nan)

Acidulant

Answer 3

You cannot convert nan to a number (such as you cannot convert infinite to a number).

The nan resulkts most probably from the w*tf.log(w) once w is (or contains) zero(s). You might add 1e-6 first so that no division by zero occurs.

Submit

In the original question he said he can't add an epsilon so I don't think adding 1e-6 will work. – Smirk