Neural Networks and Brain Scaling

Seed idea. Sparked by Children of Time (1.2) — Portia’s ~60k neurons vs a human’s ~100B.

The hook

The book frames intelligence as a function of scale. That’s essentially a scaling law. Worth exploring how capability tracks “brain size” (neurons / parameters).

Threads to grow

• Miniature scaling-law demo: capability vs neuron/param count. Github Repo

• Where do diminishing returns kick in? Where do new abilities emerge?

• Biological vs artificial neuron counts — what the comparison does and doesn’t capture.

• Link to Genetic Algorithms and Benevolent Viruses via neuroevolution.

What the research says

Digested from a first reading pass — the grounding for the experiment below.

1. Capability really does scale as a smooth power law. Kaplan et al. (2020, OpenAI) trained Transformers across a wide range of sizes and found test loss falls as a power law in three quantities — model parameters (N), dataset size (D), and compute (C). Not random, not jagged — a straight line on a log-log plot, predictable enough to extrapolate. They also found bigger models are more sample-efficient: they learn more per example. (Caveat: they counted only non-embedding parameters, since embeddings scale differently.) This is the core claim the note is chasing, and it’s real.

2. Raw size isn’t the whole story — the size/data balance is. Hoffmann et al. (2022, DeepMind — “Chinchilla”) trained 400+ models and showed Kaplan’s recipe under-weighted data. For a fixed compute budget, you should scale parameters and training data together (~20 tokens per parameter). Their 70B “Chinchilla” beat the 280B “Gopher” — a smaller model, trained better, won. First concrete dent in “bigger = smarter”: how you feed and wire the network matters as much as its neuron count. (Directly echoes the note’s “architecture beats raw count” thread.)

3. “Emergence” might be partly a measurement artifact. The dramatic “ability suddenly appears past a size threshold” story is contested. Schaeffer et al. (2023, Stanford — “Are Emergent Abilities a Mirage?”) argue the sharp jumps are often created by discontinuous metrics (exact-match, all-or-nothing scoring). Switch to a continuous metric and the same data shows smooth, gradual improvement — 25 of 29 metrics they tested showed no emergence. This changes how we build the POC: we should measure capability with a continuous metric (loss, or per-digit confidence) and a harsh discrete one (exact accuracy), and watch whether an “emergent jump” appears only under the harsh metric. That would let us reproduce the illusion on purpose — a far more interesting result than just a curve.

4. The biology rhymes with this. Herculano-Houzel’s “brain soup” neuron counts established that cortical neuron number tracks behavioral complexity across species — the human brain is essentially a scaled-up primate brain, no special sauce, just more neurons. But spiders are the counterexample that proves the point: Portia plans detours and primes on prey-specific cues with a few hundred thousand neurons — punching far above its count through specialized wiring. Scale sets the ceiling; architecture decides how close you get to it.

Throughline: all four sources agree on the same shape — scale gives you a predictable, diminishing-returns curve; architecture and training decide where you land on it. That’s exactly the pair of things the experiment below should show.

Sources: Kaplan et al. 2020 · Hoffmann et al. 2022 (Chinchilla) · Schaeffer et al. 2023 · Herculano-Houzel / “86 billion neurons” · Portia cognition review (Nelson 2025)

Side Project / Prompt Seed

If I hand this file to Claude: let’s build something. Start by digging up relevant research, then prototype a proof of concept in code.

Premise: capability as a function of “brain size.” Build a miniature scaling-law experiment.

What to build (POC): train a series of small MLPs at increasing width/depth on one fixed task (MNIST or a synthetic function). Sweep param/neuron count — e.g. start near Portia’s ~60k and scale up. Plot capability (accuracy/loss) vs size. Look for the diminishing-returns curve and any sharp “emergent” jumps.

Research to pull first:

• Neural scaling laws (Kaplan et al. 2020; Hoffmann/Chinchilla 2022).
• Emergent abilities of large models — and the “are they real or a metric artifact?” debate.
• Biological neuron counts across species vs. behavioral complexity.

Stretch: vary architecture (not just size) at fixed param budget — show capability isn’t only raw count. Bridges to Emergent and Collective Intelligence.