Motivation

Reinforcement learning for coding agents optimizes long-horizon behavior over repositories, tools and execution environments. A single rollout is not one answer: it is repeated model calls, repository inspection, tool use, code edits, dependency installs and test runs, ending in one sparse executable reward.

That trajectory is produced by a native agent harness — SWE-agent, Claude Code, OpenHands, OpenCode — which owns prompt construction, the tool set, context management and execution state. The behavior being optimized is therefore the behavior induced by that control flow. Change the harness and you have changed the optimization target, whether or not you meant to.

Lego-RL exists because the three things that most often break in an agentic RL run are not the algorithm.

1. The trajectory the trainer sees is not the one that was sampled

Harnesses rewrite their own history. They compact context, drop rejected actions, re-render tool results and rebuild prompts between turns. Reconstruct the trajectory afterwards by re-tokenizing a rendered conversation and it will differ from the exact token sequence the policy actually produced — which makes trainer-side log-probability recomputation unfaithful, in a way that shows up as a plausible-looking loss curve rather than as an error.

Sparse mixture-of-experts policies add a second mismatch: if rollout-time expert routing is not reproduced during the update, the probabilities the trainer computes belong to a different network than the one that generated the tokens.

Lego-RL captures token ids, masks and log-probabilities at generation time, inside the serving path, through an in-process proxy, and replays rollout-time routing during training. On our reference runs that holds the rollout–training probability correlation above 0.99; run validation treats a drop below it as a first-step failure, not a curiosity.

2. The reward is a proxy, or the environment quietly ate it

Similarity to a reference patch, a model judge, or a heuristic over the diff can all be satisfied without fixing anything. Lego-RL instead runs each task's own test suite in a fresh sandbox and takes its exit status, which turns reward hacking from a theoretical worry into an auditable one: a task whose verifier passes without the problem being solved is a bug in the task. See Reward Hacking.

The other half of the problem is not the reward function but the execution layer. Sandbox failures, dependency errors, verifier misconfiguration and timeouts discard expensive trajectories or, worse, score them as failures of the model. Lego-RL treats environment failure as a distinct outcome: termination-aware filtering drops broken trials out of the loss instead of letting them pollute the gradient, and image caching and stage-wise defenses keep the verifier's answer about the model.

3. When it does go wrong, the evidence is scattered

Asynchronous pipelines propagate failures across stages and fragment the evidence across workers, so a symptom observed in the loss originates several layers away — a parser mismatch, an unreachable registry, a val set whose images were never built. Lego-RL answers this with run validation before any GPU is committed, a live dashboard over metrics and per-trial trajectories, and a symptom-indexed Failure Playbook.

Why not adapt the agent to the framework

The usual alternative is to reimplement the harness against a framework's rollout interface: a simplified loop, a fixed tool set, overridden initialization, custom termination logic to extract a reward. It works, and it optimizes a scaffold nobody runs in production.

Lego-RL drives unmodified harnesses. Supporting a new one is a thin adapter that launches the agent, points it at the inference service and returns the interaction data; everything after that — sandboxing, verification, rollout, optimization, observability — is shared. The control flow that produced a rollout is the control flow your users get.

Next

• Introduction — what Lego-RL is, and its architecture
• Installation — build the venv and load the agent plugin
• Running the Pipeline — the ordered procedure, end to end