How Neural Circuitry is Laid Down

In thinking about hidden layers, it’s important to distinguish between the routine efficiency and power of a good network, once that network has been set up, and the difficult issue of how to set it up in the first place. That difference is reflected in the difference between playing the piano (or, say, riding a bicycle, or swimming) once you’ve learned (easy) and learning to do it in the first place (hard). Understanding exactly how new hidden layers get laid down in neural circuitry is a great unsolved problem of science. I’m tempted to say it’s the greatest unsolved problem.

Liberated from its origin in neural networks, the concept of hidden layers becomes a versatile metaphor with genuine explanatory power. For example, in my work in physics I’ve noticed many times the impact of inventing names for things. When Murray Gell-Mann invented “quarks,” he was giving a name to a paradoxical pattern of facts. Once that pattern was recognized, physicists faced the challenge of refining it into something mathematically precise and consistent, but identifying the problem was the crucial step toward solving it! Similarly, when I invented “anyons,” for theoretical particles existing in only two dimensions, I knew I had put my finger on a coherent set of ideas, but I hardly anticipated how wonderfully those ideas would evolve and be embodied in reality. In cases like this, names create new nodes in hidden layers of thought.

Frank Wilczek describes one of the great questions of science, how the difficult taks of learning something leads to the learned easy of later doing it.