What's the point of the threshold in a perceptron?

Question

I'm having trouble seeing what the threshold actually does in a single-layer perceptron. The data is usually separated no matter what the value of the threshold is. It seems a lower threshold divides the data more equally; is this what it is used for?

Tilly

related question: https://mcmap.net/q/54483/-perceptron-learning-algorithm-not-converging-to-0 – Strand

Answer 1

Actually, you'll just set threshold when you aren't using bias. Otherwise, the threshold is 0.

Remember that, a single neuron divides your input space with a hyperplane. Ok?

Now imagine a neuron with 2 inputs X=[x1, x2], 2 weights W=[w1, w2] and threshold TH. The equation shows how this neuron works:

x1.w1 + x2.w2 = TH

this is equals to:

x1.w1 + x2.w2 - 1.TH = 0

I.e., this is your hyperplane equation that will divides the input space.

Notice that, this neuron just work if you set manually the threshold. The solution is change TH to another weight, so:

x1.w1 + x2.w2 - 1.w0 = 0

Where the term 1.w0 is your BIAS. Now you still can draw a plane in your input space without set manually a threshold (i.e, threshold is always 0). But, in case you set the threshold to another value, the weights will just adapt themselves to adjust equation, i.e., weights (INCLUDING BIAS) absorves the threshold effects.

Dentition

Thanks for clarifying, that makes a lot of sense! I think it's essentially what I said only clearer. – Tilly

@Dentition Are you aware if there are any effects on feature importance based on a learned threshold? – Phyllome

Answer 2

The sum of the products of the weights and the inputs is calculated in each node, and if the value is above some threshold (typically 0) the neuron fires and takes the activated value (typically 1); otherwise it takes the deactivated value (typically -1). Neurons with this kind of activation function are also called Artificial neurons or linear threshold units.

Saccharine

I understand this, but what is the point of changing the threshold value? The training set allows for the separation of points anyways. Is the threshold value arbitrary? – Tilly

Well, you can "train" the threshold so it become dynamically and not suggestive. At the beginning you can set it almost randomly or to zero. – Saccharine

Oh, I see. So essentially, changing the threshold is equivalent to changing the bias, but negatively (since they are on opposite sides of the equation)? – Tilly

Changing bias resonates for me. So essentially you are shifting the separating line? How about the slope of that line? – Drye

So when we use a threshold other than 0 in training, we also use the same threshold for classifying new unseen sample, right? For example, if we choose our threshold to be 10 then we use the same threshold of 10 to decide which class an unseen sample belongs to, right? – Synthesize

Answer 3

I think I understand now, with help from Daok. I just wanted to add information for other people to find.

The equation for the separator for a single-layer perceptron is

Σw_jx_j+bias=threshold

This means that if the input is higher than the threshold, or

Σw_jx_j+bias > threshold, it gets classified into one category, and if

Σw_jx_j+bias < threshold, it get classified into the other.

The bias and the threshold really serve the same purpose, to translate the line (see Role of Bias in Neural Networks). Being on opposite sides of the equation, though, they are "negatively proportional".

For example, if the bias was 0 and the threshold 0.5, this would be equivalent to a bias of -0.5 and a threshold of 0.

Tilly

This video helped me quite a lot: youtu.be/dXuNAkHsos4. I interpret it to be saying that in addition to rotating the weight vector based on its angle from the vector of input data, we also want to make a binary determination of whether the weight vector should be flipped 180 degrees into a mirror quadrant on the opposite side of the decision boundary. Thus thresholding helps us determine the sign of angle we are rotating by. – Drye