Feedforward Networks I

Supervised Learning

Feedforward Networks

Perceptrons (Rosenblatt)

• Pattern classification problems: given an input pattern, decide whether it belongs to a particular category or not (compute a binary function of the input)
• Linearly separable classification problems: it is possible to separate the regions inside and outside the category in the input space with a line (or hyperplane), the decision surface
• Pattern classification in a perceptron
  • Inputs binary (0,1 or -1,1) or real-valued (between 0 and 1 or -1 and -1)
  • Connection weights are real-valued, positive or negative
  • Bias or threshold, which can be viewed as the weight on a connection from an input unit which always has activation of 1
  • Output units are linear threshold units
    Activation rule:
    • Net input to output unit is the weighted sum of the inputs (the dot product of the input and weight vectors)
    • If net input is greater than 0, it is turned on (a=1); otherwise it is turned off (a=0)
  • Learning
    • Finding a decision surface for the given pattern classification problem
    • Error: for all patterns which are misclassified, the sum of the deviations of the outputs from 0
    • Gradient descent (a form of hill climbing): minimize the error by moving weights gradually along the gradient, the direction in which error gets smaller
    • Gradient (partial derivative of the error with respect to the weights): for each misclassified pattern, the input vector if the output is too low, the negative of the input vector if the input is too high
    • Perceptron learning rule
      • If a pattern is correctly classified, don't change the weights.
      • If a pattern is incorrectly classified as a positive example, subtract the input vector from the weight vector.
      • If a pattern is incorrectly classified as a negative example, add the input vector to the weight vector.
      • (Usually the weight changes are multiplied by a learning rate less than 1.)
    • Rosenblatt showed that if there is a set of weights in a perceptron which solves a pattern classification problem, the perceptron learning rule will find it.