The Priory Dispute III

 

The Priory Dispute III

The Abbot's Emergency Meeting

A confidential gathering of the higher orders.

Setting: The Chapter House. Midnight. All senior monks have been summoned. Two people have specifically NOT been invited: Terence Tao and Jürgen Schmidhuber.

The Abbot enters, looking deeply troubled.

The Abbot: Brothers, I fear we have a serious problem. For decades we have imagined neural networks as collections of advanced nonlinear functions, each performing its own little piece of geometric gymnastics on the data.

But have we stopped to consider what happens when we compose all these elaborate transformations? We could end up with hopeless geometric chaos! At the very least, how would anything generalise?

Brother Anselm: Surely the training process sorts it all out?

The Abbot: That is not an explanation. That is a prayer!

Brother Matthias: Perhaps we have been looking at the wrong thing. Consider linear mappings. They perform relatively simple geometric operations: rotations, reflections, stretching, compression and shearing. Compose them and we still have a coherent linear transformation.

Now introduce context-dependent switching, and we can obtain remarkably sophisticated global behaviour without requiring every individual operation to be complicated.

Brother Anselm: You mean CCSLM?

Brother Matthias: Indeed. And even ReLU networks can be understood in terms of successive effective linear mappings, selected by their activation patterns.

The Abbot: So perhaps the intelligence isn't in a collection of little nonlinear functions performing acrobatics. Perhaps it is in the organised selection and accumulation of geometric transformations!

Brother Benedict: A rather different way of looking at neural networks.

The Abbot: A rather disturbing way, actually. We may have been looking at the wrong end of the telescope for decades!

The monks exchange nervous glances.

Brother Anselm: We should publish this immediately.

The Abbot: ABSOLUTELY NOT!

Everyone jumps.

The Abbot: Have you completely lost your senses? Do you realise what would happen if Schmidhuber or Tao heard about this meeting?

Brother Matthias: They might be interested in the idea?

The Abbot: Interested? Schmidhuber would claim he invented geometric gradualism in the previous century. Tao would produce a forty-page proof that the underlying mathematics was already known to Euclid. Then they would argue over who had misunderstood whom!

He leans forward and lowers his voice.

The Abbot: Brothers, I implore you. Not a word of this meeting must reach Schmidhuber or Tao. Not a whisper. Not a footnote. Not even an innocent-looking matrix.

Brother Benedict: And what shall we call our research programme?

The Abbot: Geometric Gradualism.

Brother Anselm: Rather catchy.

The Abbot: Yes. And if anyone asks, we are studying medieval architecture.

The meeting is adjourned.


Meanwhile, outside the Chapter House...

Two figures are standing in the shadows.

Tao: Did you hear something about geometric gradualism?

Schmidhuber: Yes. I believe it was my idea.

Tao: Interesting. I believe I can prove otherwise.

The monastery bells begin ringing.

The Abbot, from inside: OH, DEAR HEAVENS!

Comments

Popular posts from this blog

Centralizing Neural Networks on Linear Algebra

The nonlinear defense

Sinbad and the Seven Seas of Linear Algebra