Renforce

Seven ways to watch something learn.

Most explanations of reinforcement learning are diagrams. These are products — a desk, a town, a poker table, a market with two bots in it — and each one makes a claim out loud and ships a command that tries to break it. Where the mathematics has a closed-form answer, that is what the claim is checked against, rather than a baseline.

7projects on one floor

0of them running

0new dependencies to build them

Gradiente

Planned

Backpropagation you can step through.

A scalar autodiff engine, the computation graph it builds drawn node by node, and a finite difference sitting next to every gradient it claims.

ProvesEvery analytic gradient matches a central difference to 1e-6, including at ReLU's kink.

Corelibs/renforce/grad/value.ts

In-memory host

Regrette

Planned

You awarded the cheapest bid. Here is what that cost you.

Epsilon-greedy, UCB1, Thompson sampling and EXP3 run against the same job stream, with cumulative regret drawn against the bound it is not allowed to cross.

ProvesAn adversary drives UCB1 to linear regret while EXP3 stays sublinear.

Corelibs/renforce/regrette/policies/

Persisted runs

Falaise

Planned

Eight shopkeepers. Nobody told them where the cliff is.

Value iteration, policy iteration, SARSA and Q-learning painted onto a town map — the value function as a heat layer, the greedy policy as arrows, both moving as the sweeps run.

ProvesQ-learning's path hugs the cliff and SARSA's does not, and the Bellman operator contracts at gamma.

Corelibs/renforce/falaise/mdp/bellman.ts

In-memory host

Kuhne

Planned

A game small enough that we already know the right answer.

Counterfactual regret minimisation on Kuhn poker, checked against the closed-form Nash equilibrium rather than against a baseline, then run again on Leduc.

ProvesThe game value converges to exactly -1/18, and the second player's equilibrium comes out in thirds.

Corelibs/renforce/kuhne/cfr/regret-matching.ts

Committed artifact

Entente

Planned

Two pricing bots and no phone call.

Q-learning agents setting prices in a repeated Bertrand market, converging above competitive and defending it with a punishment scheme nobody wrote down.

ProvesA forced undercut triggers punishment and a measured recovery — visible in the price series.

Corelibs/renforce/entente/market/bertrand.ts

Second process

Goodharte

Planned

It learns what you like, then it learns to cheat you.

A Bradley-Terry reward model fitted to your own pairwise preferences, a KL-penalised policy optimised against it, and the moment proxy reward keeps climbing while the real thing turns over.

ProvesThe KL-constrained optimum matches pi_ref * exp(r/beta) / Z, checked numerically.

Corelibs/renforce/goodharte/reward/bradley-terry.ts

Persisted runs

Contrefait

Planned

You never ran the new policy. Answer anyway.

Off-policy evaluation over logged decisions — importance sampling, weighted importance sampling and doubly robust — each with an honest interval and a refusal when overlap is too thin.

ProvesDoubly robust stays unbiased with the outcome model broken, and again with the propensities broken instead.

Corelibs/renforce/contrefait/estimators/

Persisted runs

The floor, which is built

Four of the seven need gradients and all seven need to say how sure they are, so those two things exist before any of the projects do. pnpm renforce:prove checks them and exits non-zero if any claim breaks.

Scalar autodiff

libs/renforce/grad/

One object and one closure per operation, so the graph can be drawn and stepped through. Slow on purpose — around ten thousand forward-and-backward passes a second, which is a screen's worth and not a training run's.

Estimators and intervals

libs/renforce/stats/

Welford's online variance, a seeded percentile bootstrap, and Kish's effective sample size. Every claim any of the seven makes is some version of run it two hundred times and show where the estimate landed.

The seam

libs/renforce/deps.ts

The one file allowed to import from the repository around it. Everything else borrows through it, which is what makes lifting this suite into its own repository a single edit rather than a sweep.

Seven ways to watch something learn.