Neural networks 2

Learning in neural networks

• Gradual change in weights as learning
• Unsupervised learning of weights
• Supervised (error-driven) learning of weights
  • change in a weight depends on
    • The error for the output unit: the difference between the target value and the activation of that unit
    • The activation of the input unit
    • A learning rate (η)
  • The delta rule for updating the weight from unit i to unit j in a network with linear output units:
    w^new_ji = w^old_ji + η (t_j - x_j) x_i
    or
    Δw_ji = η (t_j - x_j) x_i
• A network that learns.

Example: AND

• Learning rate: 0.5
• Targets: 1 for input [1, 1]; 0 or negative for other inputs
• Initial weights

  0	1	bias
  0.0	0.0	0.0

• Input pattern: 0 0 1, output: 0, target: 0; error: 0; no weight changes
• Input pattern: 1 0 1, output: 0, target: 0; error: 0; no weight changes
• Input pattern: 1 1 1, output: 0, target: 1; error: 1

  0	1	bias
  0.5	0.5	0.5

• Input pattern: 0 1 1, output: 1, target: 0; error: -1

  0	1	bias
  0.5	0.0	0.0

• Input pattern: 1 0 1, output: .5, target: 0; error: -0.5

  0	1	bias
  0.25	0.0	-0.25

• Input pattern: 0 0 1, output: -0.25, error: 0
• Input pattern: 1 1 1, output: 0, target: 1; error: 1

  0	1	bias
  0.75	0.75	0.25

• Input pattern: 0 0 1, output: 0.25, target: 0; error: -0.25

  0	1	bias
  0.75	0.75	0.125

• Input pattern: 1 0 1, output: 0.875, target: 0; error: -0.875

  0	1	bias
  0.31	0.75	-0.32

An example of a set of weights

• You want a network that can take inputs representing objects in a scene and count them. Say each object is represented as the activation of a single unit at its center of mass, and the output numbers are represented using the "thermometer" encoding described here. Say the output units are binary threshold units which turn on when their input is greater than 0.
• Since there are no distinctions among the input units, the weights out of each input unit should be the same.
• The output unit representing one should turn on only when one or more input units are on. Therefore the weights into it from the input units should be positive, say, +1.
• The second output unit should turn on only when two or more input units are on. Again all of the weights from the input units should be positive, say, +1. But now we have to prevent the unit from turning on when only one unit is on. With the weight from the bias unit set to -1.5, this will happen.
• Similarly, with weights of +1 from all of the input units to all of the output units, the bias weight into the third unit could be -2.5, into the fourth unit -3.5, etc.

Activation functions

• The simplest activation function is the identity function: the activation is just the input.
• Another possibility is a threshold function which converts inputs above some threshold to a maximum value (usually 1.0) and inputs below the threshold to a minimum values (usually -1.0 or 0.0).
• A third possibility is a "soft" threshold function which smooths out the region near the threshold. The following function has a minimum of 0.0 and a maximum of 1.0 (note that these are never reached):
  f(I) = 1 / (1 + e^-I)
  (I is the input to the unit.)

Supervised learning in neural networks

• The delta rule for supervised learning (when the activation function of the output unit is the identity function)
  Δw_ji = η (t_j - x_j) x_i
  (The xs represent activations, t represents a target, and η is a learning rate between 0.0 and 1.0.)
• A formal way to derive the delta (least mean squares) rule
  • Gradient descent learning: learning by moving in the direction which looks locally to be the best
  • For supervised neural network learning, the best direction to move in "weight space": for each weight, how the global error changes with respect to that weight
  • A global error function: for each pattern the sum of the errors over all of the output units
    E = ∑_j ½ (t_j - x_j)²
  • We want to move in "weight space" in a direction which is opposite that of the slope of the error function with respect to each weight because this will move us toward a region with a lower error. The size of the move should be proportional to the magnitude of the slope.
    1. To find the slope, we take the partial derivative of the error with respect to the weight. But the only element in the sum of error terms that depends on the weight is the one for the output unit where that weight ends (j in what follows).
       ∂E/∂w_ji = ∂ [½ (t_j - x_j)²] / ∂w_ji
    2. Using the chain rule, we can decompose this derivative into two that are easier to calculate:
       (∂[½ (t_j - x_j)²]/∂x_j) (∂x_j/∂w_ji)
    3. The first derivative is easy to figure; it's just
       -(t_j - x_j)
    4. The second derivative can be decomposed using the chain rule again if we remember that the activation of unit j is a function of the input to the unit, I_j, which is in turn a function of the weights into the unit.
       ∂x_j/∂w_ji = (∂x_j/∂I_j) (∂I_j/∂w_ji)
    5. Since the activation of an output unit is the activation function f applied to the input I, the first derivative on the right-hand side of (4) is just
       f'(I_j)
       that is, the derivative of whatever the activation function is at the value of the current input to unit j.
    6. The second derivative on the right-hand side of (4) can be derived as follows:
       ∂I_j/∂w_ji = ∂(∑_kx_kw_jk)/∂w_ji = x_i
       because none of the other weights or input activations depend on w_ji.
    7. Putting all of the parts together, we get
       ∂E/∂w_ji = -(t_j - x_j) f'(I_j) x_i
    8. Remember that we want the weight change to be proportional to the negative of the derivative with respect to the weight. So with a learning rate to control the step size for weight changes, we get the more general delta (least mean squares) learning rule
       Δw_ji = η (t_j - x_j) f'(I_j) x_i

© 2004. Indiana University and Michael Gasser. www.indiana.edu/~gasser/Q530/Notes/nn2.html 13 Oct 2004