where f() is the activation function, w
where x It is one of the simplest of the neuron models, which is only occasionally used in practice because the most of phenomena in the surrounding world have nonlinear characteristics. As the example we can mean the biological neurons. Neuron could be biased; it means that it has additional input with constant value. The weight of that input is modified during the learning process like the other neuron weights. Generally we assume the bias input equal to one, in this case the neuron mathematical equation could be written as follows:
where f() is the activation function, w
In these two figures we can see that the bias enables moving the activation threshold along the x-axis. When the bias is negative the movement is made to the right side, and when the bias is positive it the movement is made to the left side. Conclusion is that the biased neuron should learn even such input vectors that non biased neuron is not able to learn. We come to the conclusion that the additional weight cost us more calculations but it improves neuron properties. Normalization does not have any sense for single input neuron, because every normalized point could have only three different values –1, 0 or 1. Lets see normalization for biased single dimensional neuron. Realization of normalization of input vectors (the bias is the input equal to 1) and weights cause movement of all points to the edge of the circle with unitary radius. Result of that operation is shown in the figure below.
According to the sign of bias normalization all points are moved to the adequate part of the circle,
for positive bias, to the upper part of the circle and for negative bias, to the bottom one. Increase of the
dimension cause that we can simply draw the line to separate points of different neuron responses. This straight
line passes through the center of coordinate system and its gradient depends on w
Normalization of the input vectors causes that all of them are moved to the edge of a unitary radius circle, with one exception of point (0,0), which stay on its place. Now we might think about how the bias works in the dual input neuron. At first lets take a look to the activation function only. As we know from the previous chapter, bias input is responsible for moving of activation function toward the straight line. In the two dimensional case bias moves the activate function towards the direction that is perpendicular to the line given by the equation: Examples of activation function for neuron with and without a bias are shown in the figure below.
Taking into account neuron with bias addition of an extra weight cause move of input vectors
from the two to the three dimensional space. All of points are situated on the sphere; however for the positive bias
on top half of sphere, and for negative bias on a bottom sphere part. This is result of the input vectors normalization,
namely the third coordinate is constant and that causes points separation for negative
and positive bias. The (0,0) point changes to (0,0,1) “the highest” point of sphere or to (0,0,-1), “the lowest” point of sphere.
In these figures we can see that the points are chosen in such a way that for non biased neuron we cannot drive straight line through the center of coordinate system that separates different values of neuron response. Neuron responses for each point was marked by circles colored dependently on the neuron response value. Conclusion is that the non-biased neuron cannot make correct classification of the selected points, so the neuron cannot be leart it. However for the biased neuron the points are moved to the edge of a sphere and because of that they might be separated by plane that passes through the center of coordinates system. Conclusion is that the biased neuron can separate these points.
References:Ryszard Tadeusiewcz "Sieci neuronowe", Kraków 1992 Andrzej Kos, Wykład "Sieci neuronowe i sztuczna inteligencja w elektronice", 2004/2005 |
|||||||||

mgr inż. Adam Gołda Krzysztof Ziaja, Piotr Miernikowski Electronics Department AGH |