Training Resnet deep neural network from scratch

Question

I need to gain some knowledge about deep neural networks.

For a 'ResNet' very deep neural network, we can use transfer learning to train a model. But Resnet has been trained over the ImageNet dataset. So their pre-trained weights can be used to train the model with another dataset. (for an example training a model for lung cancer detection with CT lung images)

I feels that this approach will be not accurate as the pre-trained weights has been completely trained over other objects but not with medical data.

Instead of transfer learning, is it possible to train the resnet from scratch? (but the available number of images to train the resnet is around 1500) . Is it something possible to do with a normal computer.

Can someone please share your valuable ideas with me

Astto

Answer 1

is it possible to train the resnet from scratch?

Yes, it is possible, but the amount of time one needs to get to good accuracy greatly depends on the data. For instance, training original ResNet-50 on a NVIDIA M40 GPU took 14 days (10^18 single precision ops). The most expensive operation in CNN is the convolution in the early layers.

ImageNet contains 14m 226x226x3 images. Since your dataset is ~10000x smaller, each epoch will take ~10000x less ops. On top of that, if you pass gray-scale instead of RGB images, the first convolution will take 3x less ops. Likewise spatial image size affects the training time as well. Training on smaller images can also increase the batch size, which usually speeds things up due to vectorization.

All in all, I estimate that a machine with a single consumer GPU, such as 1080 or 1080ti, can train ~100 epochs of ResNet-50 model in a day. Obviously, training on a 2-GPU machine would be even faster. If that is what you mean by a normal computer, the answer is yes.

But since your dataset is very small, there's a big chance of overfitting. This looks like the biggest issue that your approach faces.

Stenography