RuntimeError: "exp" not implemented for 'torch.LongTensor'

Question

I am following this tutorial: http://nlp.seas.harvard.edu/2018/04/03/attention.html to implement the Transformer model from the "Attention Is All You Need" paper.

However I am getting the following error : RuntimeError: "exp" not implemented for 'torch.LongTensor'

This is the line, in the PositionalEnconding class, that is causing the error:

div_term = torch.exp(torch.arange(0, d_model, 2) * -(math.log(10000.0) / d_model))

When it is being constructed here:

pe = PositionalEncoding(20, 0)

Any ideas?? I've already tried converting this to perhaps a Tensor Float type, but this has not worked.

I've even downloaded the whole notebook with accompanying files and the error seems to persist in the original tutorial.

Any ideas what may be causing this error?

Thanks!

Jeminah

What is d_model in the code? – Olsewski

@AkhileshPandey d_model is one of the parameters to initialise the PositionalEncoding class: def __init__(self, d_model, dropout, max_len=5000): Also the entire notebook can be found here: github.com/harvardnlp/annotated-transformer/blob/master/… – Jeminah

After going through the code I found that at one point the value of d_model was 20. Using this value I find that the line works fine for me – Olsewski

Try converting it to some other type like a= a.type(torch.float36) then call torch.exp(a) – Olsewski

I ran the code and it works fine for me. Just see if you have copied correctly. – Olsewski

@Jeminah Hey man, I'm trying to run this tutorial and I had this same problem. After applying the fix, I had a bunch of other similar problems, which I fixed by calling float() on some of the tensors and converting them to FloatTensors. But now I'm getting this error: Floating point exception: 8 and python quits. I was wondering if you had the same problem. If you didn't what did you do to fix those other problems with the float type? – Gyratory

Answer 1

I happened to follow this tutorial too.

For me I just got the torch.arange to generate float type tensor

from

position = torch.arange(0, max_len).unsqueeze(1)
div_term = torch.exp(torch.arange(0, d_model, 2) * -(math.log(10000.0) / d_model))

to

position = torch.arange(0., max_len).unsqueeze(1)
div_term = torch.exp(torch.arange(0., d_model, 2) * -(math.log(10000.0) / d_model))

Just a simple fix. But now it works for me. It is possible that the torch exp and sin previously support LongTensor but not anymore (not very sure about it).

Ist

Thank you very much and welcome to Stack overflow ! :) – Jeminah

Answer 2

It seems like torch.arange returns a LongTensor, try torch.arange(0.0, d_model, 2) to force torch to return a FloatTensor instead.

Available

Hi, thanks for the response ! But that doesn't seem to work :( – Jeminah

@Jeminah are you sure this line produces this error? can you verify (in debug) that the dtype of torch.arage(0, d_model, 2) is indeed float and not long? – Available

Answer 3

The suggestion given by @shai worked for me. I modified the init method of PositionalEncoding by using 0.0 in two spots:

position = torch.arange(0.0, max_len).unsqueeze(1)
div_term = torch.exp(torch.arange(0.0, d_model, 2) * -(math.log(10000.0) / d_model))

Gentleman

For me also (same tutorial; code copied to *.py script): needed to swap 0, to 0.0, as indicated above. :-) – Exacting

Answer 4

For me, installing pytorch == 1.7.1 solved the problem.

Sliver

Answer 5

Like Rubens said, in the higher version of Pytorch, you don't need to worry about this stuff. I can easily run it on my desktop's 1.8.0 Pytorch, but failed to go through it in my server's 1.2.0 Pytorch. There is something incompatible between different versions.

Angelikaangelina