XOR problem solvable with 2x2x1 neural network without bias?

Question

Is a Neural network with 2 input nodes, 2 hidden nodes and an output supposed to be able to solve the XOR problem provided there is no bias? Or can it get stuck?

Terricolous

There are more limitations to not having a bias, NOT isn't possible for example. – Boatsman

you may be interested in this answer #38561682 – Derision

Answer 1

Leave the bias in. It doesn't see the values of your inputs.

In terms of a one-to-one analogy, I like to think of the bias as the offsetting c-value in the straight line equation: y = mx + c; it adds an independent degree of freedom to your system that is not influenced by the inputs to your network.

Emerson

+1. would you please consider answer my question here, I think its related again on bias maybe I think the answer is wrong... datascience.stackexchange.com/questions/86000/… – Jaquelinejaquelyn

Answer 2

If I remember correctly it's not possible to have XOR without a bias.

Heteroclite

I have a (I hope working) neural network class that seems to be able to solve it with 2 inputs 3 hidden neurons and 1 output, but can't with 2 hidden neurons. Is it enough when calculating net = w1x1 + w2x2 to just add 1 to the equation, net = w1x1 + w2x2 + 1, or that is still not enough? do i have to have the 3rd weight? Thanks – Terricolous

I think that in your example with 3 hidden neurons one of the neurons acts like a bias, that's why it works. I can't think of an example that would work with 2 hidden nodes. What's the reason not to use a bias? – Boatsman

Answer 3

I have built a neural network without bias and a 2x2x1 architecture solves XOR in 280 epochs. Am new to this, so didn't know either way, but it works, so it is possible.

Regards,

Pothook

I have such network as well but I set weights manually. Bo now learning back propagation and gradient descent and all minimising the cost function requires weights and biases so not sure how to apply gradient descent if there are only weights? – Obed

Answer 4

Yes, you can if you use an activation function like Relu (f(x) =max(0,x))

Example of weights of such network are:

Layer1: [[-1, 1], [1, -1]]
Layer2: [[1], [1]]

For the first (hidden) layer:

• If the input is [0,0], both nodes will have an activation of 0: ReLU(-1*0 + 1*0) = 0, ReLU(1*0 + -1*0) = 0
• If the input is [1,0], one node will have activation of 0 ReLU(-1*1 + 1*0) = 0 and the other activation of 1 ReLU(1*1 + -1*0) = 1
• If the input is [0,1], one node will have activation of 1 ReLu(-1*0 + 1*1) = 1 and the other activation of 0 ReLU(1*0 + -1*1) = 0
• If the input is [1,1], both nodes will have an activation of 0: ReLU(-1*1 + 1*1 = 0) = 0, ReLU(1*1 + -1*1 = 0) = 0

For the second (output) layer: Since the weights are [[1], [1]] (and there can be no negative activations from previous layer due to ReLU), the layer simply acts as a summation of activations in layer 1

• If the input is [0,0], the sum of activations in the previous layer is 0
• If the input is [1,0], the sum of activations in the previous layer is 1
• If the input is [0,1], the sum of activations in the previous layer is 1
• If the input is [1,1], the sum of activations in the previous layer is 0

While this method coincidentally works in the example above, it is limited to using zero (0) label for False examples of the XOR problem. If, for example, we used ones for False examples and twos for True examples, this approach would not work anymore.

Cirone