Time and Simple Recurrent Networks

Sequences and Sequence Processing

• Prediction and sequence span, the number of previous events that must be known to predict the next
• Sequence recognition and sequence generation
• The need for context (short-term memory)

Approaches to Time and Short-Term Memory

• Mapping time onto space: time-specific input units
  • Drawbacks
    • No hardware sharing and consequent failure to generalize across individual events
    • Must know in advance length of longest sequence
• Mapping relative time onto space: STM as a sliding window
  
  • Implementations
    
    • Relative-time-specific units
    • Time-delay connections
  • Drawbacks
    • Rigid limit on width of context (though not with flexible delays)
    • Must know in advance width of longest required context (though not with trainable delays)
• Decaying STM
  
  • One sort of implementation: recurrent connections on hidden units (e.g., simple recurrent network / Elman net)
• Continuously running networks: the update cycle of the network bears no relation to the input patterns.

Connectionism: Structure

Last updated: 1 April 1996
URL: http://www.indiana.edu/~gasser/Q351/time.html
Comments: gasser@salsa.indiana.edu
Copyright 1996, The Trustees of Indiana University