Batch normalization instead of input normalization

Question

Can I use batch normalization layer right after input layer and not normalize my data? May I expect to get similar effect/performance?

In keras functional it would be something like this:

x = Input (...)
x = Batchnorm(...)(x)
...

Arose

Answer 1

You can do it. But the nice thing about batchnorm, in addition to activation distribution stabilization, is that the mean and std deviation are likely migrate as the network learns.

Effectively, setting the batchnorm right after the input layer is a fancy data pre-processing step. It helps, sometimes a lot (e.g. in linear regression). But it's easier and more efficient to compute the mean and variance of the whole training sample once, than learn it per-batch. Note that batchnorm isn't free in terms of performance and you shouldn't abuse it.

---

Holton

what about standardizing the entire data in advance (e.g. through standardscaler) and using batchnormalization as well? – Clearance

Does the BN layer need to be added always AFTER the Inputlayer, If one wants to standardize inputs? What happens if I add the BN before the Inputlayer? – Infinite

@Infinite Check out kaggle.com/ryanholbrook/…, you can add BN as Input and it acts as sklearn.StandardScaler() – Impearl

You should keep in mind that the mean and std for the train and val sets should be calculated separately. – Alasteir

@КонстантинПисаный Are you sure? I would think one would want to use the train mean/std for val, or perhaps an updated mean/std that includes both train and val samples. Just thinking in terms of single-sample batches having zero standard deviation - it makes sense why you can't have these for training, but it seems odd to me that one should need multiple samples just to validate. – Rebekahrebekkah

Answer 2

Yes this is possible, and I have used it very successfully for vision models. There are some pros and cons of this approach though, the main advantages being:

1. You can’t forget the normalization step when integrating the model in production since it’s part of the model itself (this happens more often than you think).
2. The normalization is Data augmentation aware this way.

The main drawbacks are:

1. Added runtime cost in case you had normalized inputs already available.

I’ve also written about this subject in detail here: Replace Manual Normalization with Batch Normalization in Vision AI Models. https://towardsdatascience.com/replace-manual-normalization-with-batch-normalization-in-vision-ai-models-e7782e82193c

Lallans