Representing Structure in Connectionist Networks

Why Is Structure Hard for Neural Nets?

Symbols correspond to patterns of activation across groups of (hardware) units
Two different symbols of the same general type correspond to two different patterns of activation across the same units
Symbol structures in symbolic models are built up through concatenation (cons); there is no way to concatenate patterns of activation in neural networks
A simple, but inadequate, approach
- Separate units for separate parts or features
- Role-specific groups: distributed filler representations in localized role groups
- Problems
  - What and how many parts?
  - Embedding?
  - Crosstalk: how to tell which features go with which?
    - Conjunctive coding: units represent conjunctions of features
    - Phase synchrony binding: each unit has a phase angle in addition to an activation; units whose phase angles are synchronized represent "the same thing"

Representing fixed-valence trees
Encoder network: input groups represent branches, output (length of one input group) a distributed representation of a subtree
Decoder network: input represents a distributed subtree, output the branches of the subtree, test for terminals
Training
- Encoder and decoder trained simultaneously
- Network auto-associates input branches with output branches via distributed subtree representation
- All subtrees are trained
- Moving targets: as the weights change, the hidden-layer subtree representations change, so the training set itself changes
Performance
- Generalization to novel trees
- Using tree representations as inputs and/or targets in other networks

Encoding (binding): 2 item vectors (e.g., role, filler) -> memory trace
Decoding (unbinding): memory trace and single item (cue) -> item associated with cue
Composition: combining associations in a single trace
Capacity: number of associations that can be represented
Tensor product approach (Smolensky)
- Encoding (binding): tensor product (generalized outer product)
- Decoding (unbinding): inner product of cue and trace
- Composition: tensor addition
- Size of trace increases exponentially with the depth of the structure being represented
Holographic reduced representations (Plate)
- Encoding (binding): circular convolution
- Decoding (unbinding): circular correlation
- Composition: vector addition
- Size of trace is constant as depth of structure increases
- For correlation to decode convolution, elements of each vector must be independently distributed with mean zero and variance 1/n
- Convolution trace stores pairs only with enough information to discriminate item from others; a cleanup memory (e.g., a Hopfield net) is required if an accurate output is needed
- (Some problems: types and tokens, microfeature representation)

Last updated: 27 March 1997
URL: http://www.indiana.edu/~gasser/Q351/cx-structure.html
Comments: gasser@salsa.indiana.edu
Copyright 1997, The Trustees of Indiana University