Games
Playable versions of things I write about. No scores to chase — the point is watching a learning algorithm work on you in real time.
Both games below learn the same way: one tabular Q-learning update per move,
with $\alpha = 0.2$ and $\gamma = 0.9$. Each game keeps two tables and trains both on every move you make: a personal one in your browser's localStorage, and a shared one in Firebase that every visitor contributes to. The toggle only changes which table the agent consults when choosing its move — so the shared agent keeps improving whether or not you're the one playing it.
Rock, Paper, Scissors
The agent has no idea what rock-paper-scissors is. All it sees is a state — your last three moves — and three buttons it can press. It learns purely from the reward: +1 when it beats you, −1 when you beat it, 0 for a tie.
Humans are famously bad at being random, so if you have any habit at all, it will find it.
Pick a move to start.
State (your last 3 moves): --- · Record: 0W / 0L / 0T
Rounds trained on: 0 by you · — by everyone
What the agent currently thinks each move is worth from here:
The state is your last three moves, so 64 states counting the padded openings — small enough that the whole shared table is mirrored into your browser and updates live as other people play.
Tic-Tac-Toe
Same algorithm, much bigger problem. The state is the whole board, the legal actions are whichever squares are still empty, and the reward only arrives at the end of the game — +1 for a win, −1 for a loss, 0 for a draw. Everything in between scores zero, so the agent has to propagate credit backwards through the $\gamma \max_{a'} Q(s', a')$ term to work out which early move was responsible.
That takes tens of thousands of games — far more than rock-paper-scissors, even though the algorithm is identical. A fresh agent plays like a toddler, so hit Train to run 50,000 games offline in about half a second, or switch to the shared table that every visitor has been training.
You are X. Take a square.
Record: 0W / 0L / 0T · States seen by this table: 0
Games trained on: 0 by you · — by everyone (offline training not counted)
Values appear once the agent has made its first move.
Tic-tac-toe reaches a few thousand states, so mirroring the whole shared table would mean downloading a lot of Q-values nobody will visit. Instead each board position is fetched from Firebase on demand, the first time it comes up.
Offline training uses a minimax sparring partner for most games and a random one for the rest. That's not cheating — the agent never consults minimax, it only has to survive it. Training purely against a random opponent produces an agent that has never once had to block a real threat, and it collapses the moment a person builds one. It also writes to your personal table only; synthetic games have no business inflating the shared one, which is meant to represent how real people actually play.
Note that O moving second cannot force a win against good play, so the best possible outcome for a fully-trained agent is a draw every time.
Source: qlearner.js · rps-qlearning.js · tictactoe-qlearning.js