tensorflow theano.tensor.set_subtensor equivalent

Question

I am implementing an operation in keras, such that it can work on both theano and tensorflow backend. Suppose the input of the operation is:

array([[ 0,  1,  2],
       [ 3,  4,  5],
       [ 6,  7,  8],
       [ 9, 10, 11]], dtype=int64)

then its output should be:

array([[ 0,  1,  2,  3,  4,  5],
       [ 3,  4,  5,  0,  1,  2],
       [ 6,  7,  8,  9,  10, 11],
       [ 9, 10, 11,  6,   7, 8]], dtype=int64)

My codes are as follows:

from keras import backend as K
def pairreshape(x,target_dim,input_shape):
    x1, x2 = x[0::2,], x[1::2,]
    x1_concate = K.concatenate((x1,x2), axis=target_dim)
    x2_concate = K.concatenate((x2,x1), axis=target_dim)
    if K.image_dim_ordering() == 'th':
        import theano.tensor as T
        x_new = T.repeat(x,2,axis=target_dim)
        x_new = T.set_subtensor(x_new[0::2], x1_concate)
        x_new = T.set_subtensor(x_new[1::2], x2_concate)
    elif K.image_dim_ordering() == 'tf':
        import tensorflow as tf
        repeats = [1] * len(input_shape)
        repeats[target_dim] = 2
        x_new = tf.tile(x, repeats)
        x_new[0::2] = x1_concate #TypeError: 'Tensor' object does not support item assignment
        x_new[1::2] = x2_concate #TypeError: 'Tensor' object does not support item assignment

I have successfully implemented it by theano, but I can not figure out how to assign a tensor by tensorflow. The last two lines of tensor assignment in tensorflow will report error. Is there a T.set_subtensor equivalence in tensorflow? or can you please recommend a better implementation of the operation? Thanks.

Sining

Answer 1

TensorFlow tensors are read-only. In order to modify things you need to use variables and .assign (= can not be overriden in Python)

tensor = tf.Variable(tf.ones((3,3)))
sess.run(tf.initialize_all_variables())
sess.run(tensor[1:, 1:].assign(2*tensor[1:,1:]))
print(tensor.eval())

Output

[[ 1.  1.  1.]
 [ 1.  2.  2.]
 [ 1.  2.  2.]]

Clymer

Hi Yaroslav. Thanks for your answer. I am afraid that I am not clear in my question. The input x are training samples in deep neural network, and as far as I known it should be represented as a tensor. So is there a solution for it? Thanks. – Sining

Since Tensors are read-only, the only way to do set_subtensor is to first copy it into a variable, and then modify that variable – Clymer

you can do tensor_copy.assign(tensor) do do the copy, and add a control dependency to make sure this gets executed before your set_subtensor analog – Clymer

Answer 2

It took a lot of searching, but the closest functions to theano.tensor.set_subtensor are gather, gather_nd, scatter, and scatter_nd. If you are trying to perform sparse updates to a variable, the other answers could work. But if you're trying to dynamically create a tensor from indexing another tensor, those are the functions to use.

The point of these functions is to be able to dynamically create a tensor (not a variable) from something else. My use-case is I am generating a flat tensor and I am trying to reshape it into various triangular matrices.

gather is what you use if you're trying to create a smaller matrix from a large sparse matrix. scatter is what you use if you're trying to embed a smaller matrix into a large zero matrix.

Some combination of gather, scatter and addition and multiplication can recreate theano.tensor.set_subtensor.

https://www.tensorflow.org/api_docs/python/tf/scatter_nd https://www.tensorflow.org/api_guides/python/array_ops#Slicing_and_Joining

You could also use a very complicated set of slicing and concatenation, but gather and scatter would be preferred.

Cheers

Cankerous