Neural Networks normalizing output data

Question

I have a training data for NN along with expected outputs. Each input is 10 dimensional vector and has 1 expected output.I have normalised the training data using Gaussian but I don't know how to normalise the outputs since it only has single dimension. Any ideas?

Example:

Raw Input Vector:-128.91, 71.076, -100.75,4.2475, -98.811, 77.219, 4.4096, -15.382, -6.1477, -361.18

Normalised Input Vector: -0.6049, 1.0412, -0.3731, 0.4912, -0.3571, 1.0918, 0.4925, 0.3296, 0.4056, -2.5168

The raw expected output for the above input is 1183.6 but I don't know how to normalise that. Should I normalise the expected output as part of the input vector?

Klos

if you feed the nn with Normalised Input Vector whats the output? – Holcombe

What are you trying to do with the network? Is this a regression problem? Regression problems you don't normally normalize the outputs. The training data you provide for a regression problem, the expected output should be within the range you're expecting. – Aniseed

you have te reverse the normalization for the output – Holcombe

@Aniseed i see. so only input is normalised and output stays the same. then the normalised input is trained to produce the raw output, is that right? – Klos

That's correct. After training, when you use the network to perform predictions with test inputs, you must normalize those inputs in the same fashion as you did in training... With exactly the same parameters. – Aniseed

@Aniseed got it. thanks a lot. – Klos

No problem!... Even though I didn't directly answer your question. – Aniseed

@Aniseed if you copy paste your comment as an answer, I'll accept it. – Klos

Sure! Thanks a bunch. – Aniseed

Let us continue this discussion in chat. – Klos

Answer 1

From the looks of your problem, you are trying to implement some sort of regression algorithm. For regression problems you don't normally normalize the outputs. For the training data you provide for a regression system, the expected output should be within the range you're expecting, or simply whatever data you have for the expected outputs.

Therefore, you can normalize the training inputs to allow the training to go faster, but you typically don't normalize the target outputs. When it comes to testing time or providing new inputs, make sure you normalize the data in the same way that you did during training. Specifically, use exactly the same parameters for normalization during training for any test inputs into the network.

Aniseed

what if the output is a multiregression values? let's say 2 values must be predicted. for example latitude and longitude for position prediction, would also not normalizing be necessary? – Astonishment

@Astonishment If you have multiple outputs you want to regress on, then normalizing can help. It depends on the range of each dimension. See: datascience.stackexchange.com/questions/24214/…. You typically do not do so for single dimensional outputs, but for multidimensional data it can depend. My suggestion for you is to try both and see which works better. – Aniseed

Thanks for the answer, what does "regress on then normalizing" mean in this context? the two outputs I have are longitude and latitude ( the range are -180, 180] and [-90, 90] respectively. I think it should be normalized since the range is not the same and the values can have huge difference between them let's say longitude is -180 and latitude is 90! In contrast if I'm trying to track a bus for example that would be always in Germany then the values of the longitude and latitude would be closer to each other, thus it's ok to not normalize the target values. Am I right? – Astonishment

@Astonishment Yeah, the dynamic range of the two values are quite close to each other. You should be fine with not normalizing the two. "regress on" means to predict. – Aniseed

Answer 2

One important remark is that you normalized elements of a single input vector. Having one-dimensional output space, you could not normalize the output. The correct way is, indeed, to take a complete batch of training data, say N input (and output) vectors, and normalize each dimension (variable) individually (using N samples). Thus, for one-dimensional output, you will have N samples for normalization. In this way, the vector space of your input will not be distorted. The normalization of the output dimension is usually required when the scale-space of output variables significantly different. After training, you should use the same set normalization parameters (e.g., for zscore it is "mean" and "std") as you obtain from the training data. In this case, you will put new (unseen) data into the same scale space as you in training.

Colonize