TensorFlow: How can I inspect gradients and weights in eager execution?

Question

I am using TensorFlow 1.12 in eager execution, and I want to inspect the values of my gradients and my weights at different points during training for debugging purposes. This answer uses TensorBoard to get nice graphs of weight and gradient distribution over epochs, which is what I would like. However, when I use Keras' TensorBoard callback, I get this:

WARNING:tensorflow:Weight and gradient histograms not supported for eagerexecution, setting `histogram_freq` to `0`.

In other words, this is not compatible with eager execution. Is there any other way to print gradients and/or weigths? Most non-TensorBoard answers seem to rely on graph-based execution.

Estey

Answer 1

In eager execution, you can directly print the weights. As for the gradients, you can use tf.GradientTape to get the gradients of the loss function with respect to some weights. Here is an example showing how to print gradients and weights:

import tensorflow as tf

tf.enable_eager_execution()

x = tf.ones(shape=(4, 3))
y = tf.ones(shape=(4, 1))
dense = tf.layers.Dense(1)

# Print gradients
with tf.GradientTape() as t:
    h = dense(x)
    loss = tf.losses.mean_squared_error(y, h)
gradients = t.gradient(loss, dense.kernel)
print('Gradients: ', gradients)

# Print weights
weights = dense.get_weights()
print('Weights: ', weights)

Caporal

Thanks for your help! Can this method be used in conjunction with tf.fit(), in order to print gradients over multiple epochs? Since that function abstracts away to loss taking and weights updating, I wouldn´t know how to get (the kernel of), say, my input layer and the loss "in the same place" and to feed those to the t.gradient function. – Estey

@EmielBoss do you mean tf.keras.Model.fit? If that's the case you could either (1) not use tf.keras.Model.fit implement your custom training algorithm or (2) implement a custom keras callback (wherein you print the weights/gradients, e.g. in the method on_epoch_end) and pass it to the argument callbacks of tf.keras.Model.fit. – Caporal