Global Geometric Gradualism

 

From Local Nonlinearity to Global Geometric Gradualism

A different way to picture what happens inside a neural network



When people imagine what happens inside a neural network, they often picture a succession of nonlinear functions, each performing its own little piece of mathematical gymnastics on the data. These functions bend, reshape, clip and otherwise manipulate the results, with the overall intelligence emerging from their coordinated activity.

But there is another, rather different way to understand the process.

Instead of imagining countless little nonlinear operations doing complicated things to the data, we can view a neural network as progressively transforming the geometry of its data through a sequence of relatively simple linear mappings.

This is the idea of Geometric Gradualism.

Complexity through accumulation

A linear mapping is geometrically straightforward. It can rotate, reflect, stretch, compress, shear or collapse dimensions. These are familiar operations from linear algebra.

Individually, they are not particularly complicated. But when transformations are accumulated through many layers, the resulting geometric changes can become remarkably sophisticated.

The important distinction is that the complexity need not reside in each individual transformation. It can emerge from the accumulated effect of many transformations.

In CCSLM (Context-Selected Linear Mapping) networks, the applicable linear mapping can change according to context. This introduces a powerful additional dimension: the network can select different geometric transformations depending on the situation.

The result is globally complex behaviour constructed from comparatively simple local operations.

Even ReLU networks fit this picture

This perspective is not limited to CCSLM architectures.

ReLU networks provide a particularly interesting example. Although ReLU is conventionally described as a nonlinear activation function, geometrically it simply clips negative coordinates to zero.

For any particular activation pattern, a ReLU network behaves as a linear mapping, or an affine mapping when biases are included.

Different activation patterns correspond to different effective mappings. The network's nonlinear behaviour emerges from switching between these mappings.

Thus, even a conventional ReLU network can be understood as a system of context-dependent geometric transformations.

A change in perspective

The conventional mental picture places considerable emphasis on nonlinear functions individually manipulating the data.

Geometric Gradualism shifts the emphasis towards the accumulated geometric effects of successive transformations.

This is more than a change in terminology. It opens up a different way of investigating neural networks:

  • What geometric transformation is taking place at each stage?

  • How do successive transformations accumulate?

  • How does context determine which transformation is applied?

  • How can linear algebra help us understand the resulting global behaviour?

These questions bring neural network analysis into much closer contact with the established tools of linear algebra.

The bigger idea

Perhaps the most surprising possibility is that sophisticated intelligence does not necessarily require sophisticated individual data transformations.

It may be possible to achieve remarkable complexity through the intelligent selection and accumulation of relatively simple transformations.

Rather than imagining nonlinear functions doing increasingly elaborate things to data, we can imagine geometry gradually evolving through a sequence of context-dependent linear mappings.

That is the paradigm shift: from imagined coordinated local nonlinearity to global geometric gradualism.

Comments

Popular posts from this blog

Centralizing Neural Networks on Linear Algebra

The nonlinear defense

Sinbad and the Seven Seas of Linear Algebra