The neural net gets 14 numbers each move: whether something deadly is one step
ahead, right or left; which way it's facing; which way the nearest apple is; and how much
open room lies each way. That's all — it never sees the board. It starts by moving at
random and every crash and every apple nudges it a little. Give it about 300 games before
judging it. Its progress is saved in this browser.
A nudge toward the apple. Eating scores +10 and dying scores −10, but those are
rare events early on, and an agent that only hears about them takes a long time to hear
anything at all. So each move also carries a small bonus for closing the distance to the
nearest apple and a matching penalty for drifting away. This is only used while
the neural net is training — the other three players never learn, so they never see it.
It does not make the agent smarter, and the honest framing matters. In every run
tried, the nudge got the agent scoring inside its first fifty games, where the plain
version needed a couple of hundred before it did anything at all. After that the
advantage stops being reliable: the two trade places, and no consistent winner could be
measured by the end of training. So the nudge buys a faster start, not a better player.
It is here so that the first few minutes of watching something learn from zero are worth
watching, which is the whole point of the page. (Deliberately no precise figures here —
the runs behind this were few enough to show the early gap clearly but too few to put a
firm number on it, and a page about being careful with measurements shouldn't overstate
its own.)
Why this particular form. The obvious version — plus one for getting closer,
minus one for getting further — can quietly change which strategy is optimal, so the
agent may end up excellent at a subtly different game. The version used here is
potential-based: the bonus is γ·P(next) − P(current), where P is minus the distance
to the apple. Ng, Harada and Russell proved in 1999 that this form leaves the best
strategy untouched no matter what P you pick, because the bonuses cancel out across a
whole game. It needs P to be zero at the moment the game ends, which is easy to forget
and is exactly what voids the guarantee people cite it for.
Why it never fills the board. Filling all 100 squares takes about 2,400 moves
without a single fatal mistake, and it can't see its own body's shape — sealing yourself
into a pocket is a mistake you make twenty moves before you die. Nothing in those 14
numbers contains that. More training doesn't teach a fact the input doesn't carry.
Search doesn't learn at all. Every move it finds the shortest route to an apple,
imagines eating it, and asks "could I still reach my own tail from there?" If not, that
route is a trap and it's thrown away. With no safe route it stalls by following its tail.
The mint line is the route it committed to; amber means it's stalling.
Net + search is the two together, and it's how real game-playing systems are built:
the network judges, and a cheap check filters out one clearly-unsafe class of move. Same trained brain, measured over 25 games —
36 apples alone, 85 with the veto. Mint is what the net wanted, amber is what it
was allowed to do.
What that check can and cannot promise. It simulates exactly one move, then asks
whether a path still exists from the head to the tail. That is a cheap proxy for "not
trapped", not a proof of safety: a snake can pass this test and still be walking into a
trap that only closes twenty moves later, because nothing here searches those twenty
moves. It removes one clearly-fatal class of mistake, which turns out to be worth about
fifty apples — but "verified safe" would be the wrong words for it.
Walls change the board and, with it, which player still works. Search
re-plans every move so it copes for free (93 apples without walls, 73 with). The net
keeps its 14 senses and just needs retraining. The perfect cycle stops existing
entirely — you cannot build a loop through every square when squares are missing —
so that option switches itself off. Each combination of settings keeps its own saved
brain, so training on walls doesn't overwrite the one you trained without them.
Whole board vision swaps the 14 hand-picked senses for all 100 squares as three
layers — blocked, apples, head — 304 numbers instead of 14. In principle this is the fix
for its blindness: the information about self-trapping is finally in the input. In
practice it is dramatically worse, and that result is the interesting part.
Measured: 14 senses reach 28 apples by game 300 and 39 by game 1,500. Whole board
reaches 1.4 by game 1,000 and 4.7 by game 5,000. A higher learning rate didn't help.
It is learning — just roughly ten times slower.
Why is worth sitting with. The 14 senses are already relative to the snake:
"danger to your right" means the same thing anywhere on the board. The 304 board numbers
are absolute positions, so the net has to learn what a wall means at square 14, and then
learn it all over again at square 15, and again at square 16. Nothing tells it those are
the same lesson. That property — the same pattern meaning the same thing wherever it
appears — is exactly what convolutional networks are built to provide, and it is why
anything that reads a grid or an image uses them instead of a plain network like this
one. Leave it on TURBO for a long while and it does climb.
Two snakes puts a rival on the board that heads for the nearest apple it can
safely reach. It counts as a wall that moves, so the danger and room senses pick it up
with no changes at all. This is where learning earns its keep: the rival is a rule change
that search handles, the cycle cannot survive, and the net can simply retrain against.
The perfect cycle follows one fixed loop through every square, so it cannot hit
itself and always wins. It also ignores the apples completely and takes ~2,500 moves. Put
obstacles on the board and it stops existing entirely — you can't build a loop through
every square when squares are missing.