Recursive Language Models

1 · The pipeline

One closed loop. The policy acts in an environment; the environment produces evidence; a filter turns evidence into a preference signal; an optimiser turns that signal into the next set of weights, which is the policy that acts next round.

Every arrow is a control point. The policy acting is where data can leave. The arena runs untrusted code written by something trying to score well. The filter is the objective — whoever writes it decides what the model becomes. And the update ships a new model. Which is the argument for an external boundary: the thing being controlled keeps changing, so the control cannot live inside it.

2 · Two recursions

Runtime recursion — weights frozen

The model re-enters its own output. A fixed-point iteration on sequence space:

y⁰ = π_θ(· | x) c^i = π_θ(· | x, y^i) critique y^{i+1} = π_θ(· | x, y^i, c^i) revision This converges only if the revision operator contracts with respect to some quality metric. For a language model critiquing itself, it generally does not: unverified self-correction drifts, oscillates, or converges confidently on a wrong answer. The grounding signal is what makes it a contraction — which is why the diagram puts an execution arena at the far end and not a second model.

Training recursion — weights move

Round index k. The training distribution is a function of the current parameters:

D_k ~ π_θk ⊗ E sample trajectories D̃_k = Φ(D_k) filter / score θ_{k+1} = θ_k − η ∇_θ L(θ; D̃_k) update This is a self-consuming loop. Round k's data comes from a policy that round k−1 moved, so errors compound rather than cancel. The loop has its own dynamics independent of whether any individual rollout was good.

3 · The objective

The diagram names DPO, so the filter's job is to emit preference pairs (y_w, y_l) over a shared prompt. Direct Preference Optimization drops the separate reward model of RLHF and optimises the pairs directly:

L_DPO(θ; θ_ref) = − E_(x, y_w, y_l) ~ D [ log σ( β·log( π_θ(y_w|x) / π_ref(y_w|x) ) − β·log( π_θ(y_l|x) / π_ref(y_l|x) ) ) ] Where it comes from: the best policy under a KL-constrained reward is π*(y|x) ∝ π_ref(y|x)·exp(r(x,y)/β). Invert that and the reward is a log-ratio, at which point the pairwise likelihood collapses into the line above — no separate reward model needed.

The choice that bites

π_ref is the frozen anchor, and which round you freeze is a policy decision with real consequences. Anchoring to θ₀ bounds total drift across every round. Re-anchoring each round to θ_k bounds only per-round drift — and lets the model walk arbitrarily far given enough rounds. β is the leash: as β → 0 the constraint vanishes and the policy is free to collapse onto whatever maximises the proxy.

LoRA is what makes a round cheap enough to close the loop at all. Instead of updating the full weight matrix, learn a low-rank correction:

W' = W + (α / r)·B·A B ∈ ℝ^(d×r), A ∈ ℝ^(r×d), r ≪ d Trainable parameters fall by three to four orders of magnitude, base weights stay shared across rounds, and a round becomes an adapter swap rather than a redeploy. It also gives you the rollback story: θ_{k+1} is a file you can delete.

4 · Where the recursion is legitimate

The loop is sound exactly where verification is cheaper than generation. That asymmetry is what makes Φ trustworthy, and its absence is what makes a self-improvement pipeline a rumour mill.

Domain	Verifier	Asymmetry
Code	Test suite, type checker, compiler	Strong — execution is ground truth
Formal maths	A mechanical proof checker	Strong — mechanically decidable
Retrieval	Citation resolves and entails	Moderate
Open-ended prose	Another model	None — this is where collapse lives

Coding agents are the canonical RLM domain for exactly this reason, and it is why the demonstrator below decides with a test suite rather than a judge model.

Live · free · no account

5 · Run the loop

The pipeline above, running in your browser. The policy proposes a program, the arena runs it against real tests, the score is the fraction that passed, and the failures become the feedback for the next attempt. Solve a task and it emits the preference pair the training half would train on.

nessco · rlm demonstrator

Two things worth watching. On fizzbuzz, attempt 1 makes the fix it says it is making and gets worse — 0.67 down to 0.00. Self-correction does not only ever improve, and without the arena nothing catches that. On collatz, attempt 0 never returns and the watchdog kills the thread. Code that will not stop cannot be asked to stop from inside itself, which is why the arena has to be a separate thread.

6 · Failure modes

Model collapse

Each round samples from the last round's output, so rare cases disappear first and the distribution narrows toward whatever the model already does most. It compounds, and fresh real data is the only way back. Only accepting runs that actually executed and passed helps — it keeps the data tied to something outside the model — but it does not stop the narrowing.

Goodhart and reward hacking

The filter is a proxy for what you want, and training finds the places where the proxy and the real goal come apart, because that is where the cheap wins are. A test-suite filter selects for passing the tests — which includes hard-coding the expected output and catching the assertion. Every proxy works this way. The only question is how fast the policy finds the gap.

Distribution shift

D_k ~ π_θk means each round trains on data drawn from a policy the previous round moved. The KL anchor is the only thing bounding the walk, and β sets how tight that bound is.

Unbounded runtime recursion

A depth cap is a correctness property, not a cost control. Without one — and without checking that each attempt actually improved — a loop will re-derive the same wrong answer until it runs out of context.

For developers

7 · How the system actually works

Everything below is the code running on this page — three modules, one message protocol, four invariants. Read it if you want to plug a real model in, add a task, or lift the arena into your own project.

Module map

File	Role in the diagram	Depends on
`assets/js/rlm-policy.js`	Base Policy π_θk, plus the task corpus and the Trajectory Filter's pair construction	nothing
`assets/js/rlm-worker.js`	Target Environment — the Execution Arena	nothing (runs in a Worker; no imports)
`assets/js/rlm-terminal.js`	The driver: closes the loop, bounds it, renders it	the other two
`test/rlm-policy.test.js`	Executes every candidate and pins the reward it earns	`rlm-policy.js`

The policy module has no dependencies and no DOM access on purpose, so it runs the same under Node — which is how the tests execute every candidate outside a browser.

The arena protocol

One request, one response, matched on runId. The driver owns the timeout; the worker never sees it.

// driver → arena { runId: number, source: string, // the candidate program, as text entry: string, // the function name to call, e.g. "twoSum" tests: [{ name, args: any[], expect: any }] } // arena → driver { runId: number, ok: boolean, // every test passed fatal: string | null, // syntax error, or entry not defined results: [{ name, pass, detail }] } On failure, detail is the real call and the real return value — twoSum([3,2,4], 6) → [2,4], want [1,2]. That string is the feedback the next attempt gets, so it has to carry something usable.

Loop state

One array. Each entry is what the arena reported for one attempt, and it is everything the policy learns about its own past:

history: [{ rung: number, reward: number, failing: string[] }] A timeout and a syntax error both record reward: 0 with every test listed as failing. An attempt that could not be judged is a failure, not a gap — dropping it would let a hanging candidate look like it was never tried.

Invariants

Guarantee	Enforced by
The loop always terminates	`MAX_DEPTH = 4` in the driver, and `propose()` returning `null` when the ladder is exhausted
A single candidate cannot hang the page	`RUN_TIMEOUT_MS = 1200` watchdog, then `worker.terminate()` and respawn
A stale result cannot be attributed to a later attempt	`runId` check, plus a `settled` latch so watchdog and message race exactly once
Two loops cannot share one worker	the input is disabled for the whole run and re-enabled in a `finally`

Plugging in a real model

One function. The driver awaits propose() even though the stub is synchronous, so a network-backed policy drops straight in:

// rlm-policy.js — replace the stub with this shape export async function propose(task, history) { const last = history[history.length - 1]; const response = await fetch('/api/rlm/propose', { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ prompt: task.prompt, entry: task.entry, attempt: history.length, critique: last ? last.failing : [] // the grounded signal }) }); const { source } = await response.json(); return { rung: history.length, source, note: 'sampled from π_θk', claims: [] }; } The driver reads four fields: rung, source, and optionally note and claims. Return null to stop early. Nothing else changes — the arena, the scoring and the depth bound do not care where the program came from.

What changes when the policy is real

Three things stop being free. The arena becomes a genuine sandbox problem — a Web Worker is isolation from the page, not from a determined adversary, and model-written code needs a container with no network and a filesystem you are willing to lose. Sampling replaces the ladder, so MAX_DEPTH becomes a cost control as well as a termination guarantee. And the filter needs a floor: a pair is only worth training on if y_w beats y_l by a margin that is not noise.

Adding a task

Append to TASKS. The tests check that the last rung passes everything, that no earlier one accidentally does, and that every claims entry names a real test — so a task that is wrong about itself breaks the build instead of misleading a visitor.

{ id: 'reverse', title: 'Reverse words', entry: 'reverseWords', prompt: 'Reverse the order of words, collapsing runs of spaces.', tests: [{ name: 'collapses spaces', args: ['a b'], expect: 'b a' }], ladder: [ { note: 'splits on a single space, so runs leave empty tokens', claims: [], // what the policy believes it fixes source: 'function reverseWords(s) { return s.split(" ").reverse().join(" "); }' } ] } A rung that never terminates has to be listed in NON_TERMINATING in the test file — the suite runs every other rung directly, and nothing there can stop an infinite loop.

Running it locally

python3 -m http.server 8000 # the page is static; no build step node --test "test/*.test.js" "api/_lib/*.test.js" # 19 tests cd sdk/js && node --test "test/*.test.js" # 30 tests Pass the glob, not the directory — node --test test/ resolves the bare directory as a module path and fails. The worker is loaded by relative path, so the page has to be served over HTTP; opening the file directly gives you the prose and a dead terminal.

Recursive Language Models.