Pytorch: nn.Dropout vs. F.dropout

Question

There are two ways to perform dropout:

• torch.nn.Dropout
• torch.nn.functional.Dropout

I ask:

• Is there a difference between them?
• When should I use one over the other?

I don't see any performance difference when I switched them around.

Know

Answer 1

The technical differences have already been shown in the other answer. However the main difference is that nn.Dropout is a torch Module itself which bears some convenience:

A short example for illustration of some differences:

import torch
import torch.nn as nn

class Model1(nn.Module):
    # Model 1 using functional dropout
    def __init__(self, p=0.0):
        super().__init__()
        self.p = p

    def forward(self, inputs):
        return nn.functional.dropout(inputs, p=self.p, training=True)

class Model2(nn.Module):
    # Model 2 using dropout module
    def __init__(self, p=0.0):
        super().__init__()
        self.drop_layer = nn.Dropout(p=p)

    def forward(self, inputs):
        return self.drop_layer(inputs)
model1 = Model1(p=0.5) # functional dropout 
model2 = Model2(p=0.5) # dropout module

# creating inputs
inputs = torch.rand(10)
# forwarding inputs in train mode
print('Normal (train) model:')
print('Model 1', model1(inputs))
print('Model 2', model2(inputs))
print()

# switching to eval mode
model1.eval()
model2.eval()

# forwarding inputs in evaluation mode
print('Evaluation mode:')
print('Model 1', model1(inputs))
print('Model 2', model2(inputs))
# show model summary
print('Print summary:')
print(model1)
print(model2)

Output:

Normal (train) model:
Model 1 tensor([ 1.5040,  0.0000,  0.0000,  0.8563,  0.0000,  0.0000,  1.5951,
         0.0000,  0.0000,  0.0946])
Model 2 tensor([ 0.0000,  0.3713,  1.9303,  0.0000,  0.0000,  0.3574,  0.0000,
         1.1273,  1.5818,  0.0946])

Evaluation mode:
Model 1 tensor([ 0.0000,  0.3713,  0.0000,  0.0000,  0.0000,  0.0000,  0.0000,
         0.0000,  0.0000,  0.0000])
Model 2 tensor([ 0.7520,  0.1857,  0.9651,  0.4281,  0.7883,  0.1787,  0.7975,
         0.5636,  0.7909,  0.0473])
Print summary:
Model1()
Model2(
  (drop_layer): Dropout(p=0.5)
)

So which should I use?

Both are completely equivalent in terms of applying dropout and even though the differences in usage are not that big, there are some reasons to favour the nn.Dropout over nn.functional.dropout:

Dropout is designed to be only applied during training, so when doing predictions or evaluation of the model you want dropout to be turned off.

The dropout module nn.Dropout conveniently handles this and shuts dropout off as soon as your model enters evaluation mode, while the functional dropout does not care about the evaluation / prediction mode.

Even though you can set functional dropout to training=False to turn it off, it is still not such a convenient solution like with nn.Dropout.

Also the drop rate is stored in the module, so you don't have to save it in an extra variable. In larger networks you might want to create different dropout layers with different drop rates - here nn.Dropout may increase readability and can bear also some convenience when using the layers multiple times.

Finally, all modules which are assigned to your model are registered in your model. So you model class keeps track of them, that is why you can just turn off the dropout module by calling eval(). When using the functional dropout your model is not aware of it, thus it won't appear in any summary.

Echoism

Thank you for the "SO which should I use?" - that was the part I was missing! I normally just do F.dropout(x,training = self.training) for handling the train/eval difference of it. So to summarize: its a matter of personal preferences? – Know

@Jakob Yes, exactly! - nn.Dropout just intends to provide a slightly higher level API to the functional dropout that can be used in a layer style. However, there is no real difference in behaviour if you use it as you described. – Echoism

When I have more than one layer where I want to apply dropout, should I instantiate one nn.Dropout object for each layer or can I safely reuse it? In general: How do I know which layers can be reused and which not? – Cellaret

I actually have one in each layer (apart from the input and output of course) as standard. – Know

On the other hand, I find the functional API more convenient when dropout is not fixed, compared to changing the dropout rate of the Dropout "layer". – Kinser

Answer 2

If you look at the source code of nn.Dropout and Functional.Dropout, you can see Functional is an interface and nn module implement functions with respect to this interface.
Look at the implementations in nn class:

from .. import functional as F
class Dropout(_DropoutNd):
    def forward(self, input):
        return F.dropout(input, self.p, self.training, self.inplace)

class Dropout2d(_DropoutNd):
    def forward(self, input):
        return F.dropout2d(input, self.p, self.training, self.inplace)

And so on.

Implementation of Functional class:

def dropout(input, p=0.5, training=False, inplace=False):
    return _functions.dropout.Dropout.apply(input, p, training, inplace)

def dropout2d(input, p=0.5, training=False, inplace=False):
    return _functions.dropout.FeatureDropout.apply(input, p, training, inplace)

look at the example below to understand:

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(1, 10, kernel_size=5)
        self.conv2 = nn.Conv2d(10, 20, kernel_size=5)
        self.conv2_drop = nn.Dropout2d()
        self.fc1 = nn.Linear(320, 50)
        self.fc2 = nn.Linear(50, 10)

    def forward(self, x):
        x = F.relu(F.max_pool2d(self.conv1(x), 2))
        x = F.relu(F.max_pool2d(self.conv2_drop(self.conv2(x)), 2))
        x = x.view(-1, 320)
        x = F.relu(self.fc1(x))
        x = F.dropout(x, training=self.training)
        x = self.fc2(x)
        return F.log_softmax(x)

There is a F.dropout in forward() function and a nn.Dropout in __init__() function. Now this is the explanation:

In PyTorch you define your Models as subclasses of torch.nn.Module.

In the init function, you are supposed to initialize the layers you want to use. Unlike keras, Pytorch goes more low level and you have to specify the sizes of your network so that everything matches.

In the forward method, you specify the connections of your layers. This means that you will use the layers you already initialized, in order to re-use the same layer for each forward pass of data you make.

torch.nn.Functional contains some useful functions like activation functions a convolution operations you can use. However, these are not full layers so if you want to specify a layer of any kind you should use torch.nn.Module.

You would use the torch.nn.Functional conv operations to define a custom layer for example with a convolution operation, but not to define a standard convolution layer.

Blunt

But what should be used when? Does that make a difference? – Know

But the dropout itself does not have any parameters/weights. So why would you add them as a layer? I kinda struggle to see when F.dropout(x) is superior to nn.Dropout (or vice versa). To me they do exactly the same. For instance: what are the difference (appart from one being a function and the other a module) of the F.droput(x) and F.relu(F.max_pool2d(self.conv2_drop(self.conv2(x)), 2))? Could you not replace the latter with F.relu(F.max_pool2d(F.dropout(self.conv2(x)), 2)) – Know

To edit above: Why would you add them in the initial function/use them that way? – Know

You can see this post also: discuss.pytorch.org/t/… – Blunt

@Know You're right, Dropout does not have any parameters/weights. However, when pytorch model is in evaluation mode, it does NOT apply Dropout; dropout should only be applied in training for best effects. If you use F.dropout(x) then you have to remember to specify if you are in training or evaluation mode, whereas if you use nn.Dropout(x) then it will do it automatically for you. – Hg

That is correct thats why I always use the parameter "train=self.train", that handles the train/evaluation. – Know

Answer 3

Check torch.nn.functional's implementation:

 if p < 0. or p > 1.:
        raise ValueError("dropout probability has to be between 0 and 1, "
                         "but got {}".format(p))
    return (_VF.dropout_(input, p, training)
            if inplace
            else _VF.dropout(input, p, training))

Check: torch.nn.dropout's implementation:

def forward(self, input):
        return F.dropout(input, self.p, self.training, self.inplace)

So: their internal operation is the same. Interfaces are different. As for _VF, I guess that's some C/C++ code.

Airedale