Symmetric Recurrent Networks

Hopfield Networks

• The associative memory problem:
  Store a set of p patterns in such a way that when presented with a new pattern, the network responds by producing whichever one of the stored patterns most closely resembles the new pattern.
• Discrete Hopfield net architecture
  • Potentially completely recurrent
  • Symmetric weights
  • Units are binary, with activations of -1 or +1
• Processing in Hopfield nets
  • Units in the network are repeatedly updated until the network settles
  • Hopfield activation (unit update) rule (with no threshold):
    
    (sgn(x) = 1 if x is positive, -1 if x is negative)
  • Synchronous and asynchronous update; different updating procedures may lead to different attractors
• Gradient descent (a form of hill climbing) in Hopfield nets
  • Energy of network:
    
  • Each unit update should move the network towards a state of lower energy
  • Standard update rule minimizes energy
    Assuming no self-coupling terms or thresholds, for a given updated unit i, either its activation is unchanged, or it is negated
    
    Then the difference between the energy after and before the update of unit i is
    
    But this term is negative, so, for asynchronous updates, the energy always either remains the same or decreases.
• "Learning" in Hopfield nets
  • For one pattern m, we get stability if, for all i:
    
    This is true if
    
  • Storing p memories in a Hopfield net of N units:
    
  • Hebbian learning: weight on the connection joining two units is proportional to the correlation between their activations
  • Capacity of a network is limited, roughly proportional to N for random patterns

Boltzmann Machines

Connectionism: Connectionist vs. Symbolic Models

Connectionism: Feedforward Networks I