I encounter this weird error when building a simple NN in Pytorch. I dont understand this error and why this consern Long and Float datatype in backward function. Anyone encounter this before? Than...

I'm trying to calculate MSELoss when mask is used. Suppose that I have tensor with batch_size of 2: [2, 33, 1] as my target, and another input tensor with the same shape. Since sequence length migh...

I am about making backpropagation on a neural network that uses ReLU. In a previous project of mine, I did it on a network that was using Sigmoid activation function, but now I'm a little bit confu...

If I have something like: model = Model(inputs = input, outputs = [y1,y2]) l1 = 0.5 l2 = 0.3 model.compile(loss = [loss1,loss2], loss_weights = [l1,l2], ...) what does Keras do with the losse...

I'm trying to implement a neural network architecture in Haskell, and use it on MNIST. I'm using the hmatrix package for linear algebra. My training framework is built using the pipes package. My c...

I am trying to compute a loss on the jacobian of the network (i.e. to perform double backprop), and I get the following error: RuntimeError: one of the variables needed for gradient computation has...

I am currently implementing a CNN in plain numpy and have a brief question regarding a special case of the backpropagation for a max-pool layer: While it is clear that the gradient with respect to...

TLDR I have been trying to fit a simple neural network on MNIST, and it works for a small debugging setup, but when I bring it over to a subset of MNIST, it trains super fast and the gradient is cl...

If I use a stateful RNN in Keras for processing a sequence of length N divided into N parts (each time step is processed individually), how is backpropagation handled? Does it only affect the las...

When coding PyTorch in torch.nn.utils I see two functions, clip_grad_norm and clip_grad_norm_. I want to know the difference so I went to check the documentation but when I searched I only found th...

I use the Pytorch. In the computation, I move some data and operators A in the GPU. In the middle step, I move the data and operators B to CPU and continue the forward. My question is that: My oper...

I am learning Artificial Neural Network (ANN) recently and have got a code working and running in Python for the same based on mini-batch training. I followed the book of Michael Nilson's Neural Ne...

Suppose I have my custom loss function and I want to fit the solution of some differential equation with help of my neural network. So in each forward pass, I am calculating the output of my neural...

I'm training a XOR neural network via back-propagation using stochastic gradient descent. The weights of the neural network are initialized to random values between -0.5 and 0.5. The neural network...

I am exploring PyTorch, and I do not understand the output of the following example: # Initialize x, y and z to values 4, -3 and 5 x = torch.tensor(4., requires_grad = True) y = torch.tensor(-3., r...

Can you please tell me the difference between Stochastic Gradient Descent (SGD) and back-propagation?

If I understood correctly, to perform TBPTT in keras we have to split our sequences into smaller parts of k timesteps. To re-use the state of our LSTM accross all the parts of the sequence we have ...

I get an error using gradient visualization with transfer learning in TF 2.0. The gradient visualization works on a model that does not use transfer learning. When I run my code I get the error: ...

For a neural networks library I implemented some activation functions and loss functions and their derivatives. They can be combined arbitrarily and the derivative at the output layers just becomes...

How to detect source of vanishing gradients in pytorch? By vanishing gradients, I mean then the training loss doesn't go down below some value, even on limited sets of data. I am trying to train so...

I am trying to work out question 26 from this exam paper (the exam is from 2002, not one I'm getting marked on!) This is the exact question: The answer is B. Could someone point out where I'm goin...

I recently study PyTorch and backward(). I understood how to use it, but when I try: x = Variable(2*torch.ones(2, 2), requires_grad=True) x.backward(x) print(x.grad) I expect: tensor([[1., 1.], [...

Given the following tensor (which is the result of a network [note the grad_fn]): tensor([121., 241., 125., 1., 108., 238., 125., 121., 13., 117., 121., 229., 161., 13., 0., 202., 161., 121., 121...

I have made a small script in Python to solve various Gym environments with policy gradients. import gym, os import numpy as np #create environment env = gym.make('Cartpole-v0') env.reset() s_size ...

I am trying to implement neural network with RELU. input layer -> 1 hidden layer -> relu -> output layer -> softmax layer Above is the architecture of my neural network. I am confused about back...