RuntimeError: "nll_loss_forward_reduce_cuda_kernel_2d_index" not implemented for 'Int': Pytorch

Question

So, I was trying to code a chatbot using Pytorch following this tutorial.

Code: (Minimal, Reproducible one)

tags = []
for intent in intents['intents']:
    tag = intent['tag']
    tags.append(tag)

tags = sorted(set(tags))

X_train = []
X_train = np.array(X_train)

class ChatDataset(Dataset):
    def __init__(self):
        self.n_sample = len(X_train)
        self.x_data = X_train

#Hyperparameter
batch_size = 8
hidden_size = 47
output_size = len(tags)
input_size = len(X_train[0])
learning_rate = 0.001
num_epochs = 1000


dataset = ChatDataset()
train_loader = DataLoader(dataset=dataset, batch_size=batch_size, shuffle=True, num_workers=0)

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu') # using gpu
model = NeuralNet(input_size, hidden_size, output_size).to(device)

# loss and optimizer
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)

for epoch in range(num_epochs):
    for (words, labels) in train_loader:
        words = words.to(device)
        labels = labels.to(device)

        #forward
        outputs = model(words)
        loss = criterion(outputs, labels) #the line where it is showing the problem

        #backward and optimizer step
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

    if (epoch +1) % 100 == 0:
        print(f'epoch {epoch+1}/{num_epochs}, loss={loss.item():.4f}')

print(f'final loss, loss={loss.item():.4f}')

Full Code (if needed)

I am getting this error while trying to get the loss function.

RuntimeError: "nll_loss_forward_reduce_cuda_kernel_2d_index" not implemented for 'Int'

Traceback:

Traceback (most recent call last): File "train.py", line 91, in <module> loss = criterion(outputs, labels) File "C:\Users\PC\anaconda3\lib\site-packages\torch\nn\modules\module.py", line 1102, in _call_impl return forward_call(*input, **kwargs) File "C:\Users\PC\anaconda3\lib\site-packages\torch\nn\modules\loss.py", line 1150, in forward return F.cross_entropy(input, target, weight=self.weight, File "C:\Users\PC\anaconda3\lib\site-packages\torch\nn\functional.py", line 2846, in cross_entropy return torch._C._nn.cross_entropy_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index, label_smoothing) RuntimeError: "nll_loss_forward_reduce_cuda_kernel_2d_index" not implemented for 'Int'

But looking into the tutorial, it seems to work perfectly there whereas it is not in my case.

What to do now?

Thanks.

Reservoir

Answer 1

In my case, I solved this problem by converting the type of targets to torch.LongTensor before storing the data into the GPU as follows:

for inputs, targets in data_loader:
    targets = targets.type(torch.LongTensor)   # casting to long
    inputs, targets = inputs.to(device), targets.to(device)
    ...
    ...
 
    loss = self.criterion(output, targets)

Spacecraft

Answer 2

I guess you followed Python Engineer's tutorial on YouTube (I did too and met with the same problems !). @Phoenix 's solution worked for me. All I needed to do was cast the label (he calls it target) like this :

for epoch in range(num_epochs):
    for (words, labels) in train_loader:
        words = words.to(device)
        labels = labels.type(torch.LongTensor) # <---- Here (casting)
        labels = labels.to(device)
        
        #forward
        outputs = model(words)
        loss = criterion(outputs, labels)
        
        #backward and optimizer step
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

    if (epoch + 1) % 100 == 0:
        print(f'epoch{epoch+1}/{num_epochs}, loss={loss.item():.4f}')

It worked and the evolution of the loss was printed in the terminal. Thank you @Phoenix !

P.S. : here is the link to the series of videos I got this code from : Python Engineer's video (this is part 4 of 4)

Lotti

This works - check the type when you debug with x.type() – Carduaceous

Answer 3

I had the same problem, my issue was that I was doing a binary classification problem and set the output size of the model to 1 instead of 2, so the model was returning a float (in my case) instead of a tensor of floats.

Check if you have set the right output_size

Tourneur

Answer 4

Just verify what your model is returning,it should be float type i.e your outputs variable Else change it to type float
I think you have returned int type in forward method

Parse

In my case, the custom target i was comparing the model output to was of type float. y = y.to(torch.int64) fixed it. – Bevatron

Answer 5

In my case, using torch.autocase took care of this error when using the criterion:

with torch.autocast('cuda'):
    loss = self.criterion(out, torch.tensor(labels).cuda())

Alba

Answer 6

In Multi-Class classification problem, you have to pay attention to dimension of y (labels), in nn.CrossEntropyLoss() loss function does not work with float type it expects int type.

So you can do:

loss = self.criterion(out, labels.type(torch.int64))

Weinert