Why isn't DropOut used in Unsupervised Learning?

Question

All or nearly all of the papers using dropout are using it for supervised learning. It seems that it could just as easily be used to regularize deep autoencoders, RBMs and DBNs. So why isn't dropout used in unsupervised learning?

Feltie

Look at denoising autoencoder, adding random noise is similar to dropout. – Aqueduct

Yep, that's exactly the same idea. They randomly select inputs and set them to 0. See: deeplearning.net/tutorial/dA.html – Intuition

I have the same question. I think denoising AE only apply dropout on intput layer. My question is why isn't dropout applied to hidden layers to prevent overfitting? – Yacov

Answer 1

Dropout is used in unsupervised learning. For example:

Shuangfei Zhai, Zhongfei Zhang: Dropout Training of Matrix Factorization and Autoencoder for Link Prediction in Sparse Graphs (arxiv, 14 Dec 2015)

Barthelemy

Answer 2

Labeled data is relatively scarce, and that's why supervised learning often benefits from strong regularization, like DropOut.

On the other hand, unlabeled data is usually plentiful, and that's why DropOut is typically not needed, and may be detrimental (as it reduces the model capacity).

Even gigantic models like GPT-3 (175e9 parameters) are still underfitting after being updated on 300e9 tokens.

Feltie