Running the Pipeline

Training a coding agent with Lego-RL is four stages run in order, each one a config file passed to a runner. This page is what each stage consumes, what it must produce before the next is worth launching, and where a finished run leaves its evidence.

task pool ─▶ [ rollout generation ] ─▶ [ training ] ─▶ [ merge ] ─▶ [ evaluation ]
              filtered index           checkpoints     servable      held-out score

The full closed loop has five phases — data preparation, run validation, the training run, the live UI, and the human review that decides what changes before the next run. The four stages on this page are the operator's path through it.

Two ways to run, one runner

Every stage can be driven either through the agent plugin or from a shell. The skills are not a second implementation: they shell out to the same runner and the same scripts/lib/preflight.sh a manual launch uses.

The shape of every stage

All three runners share one lifecycle:

Before anything below, you need the venv and the plugin from Installation, and the four inputs Lego-RL cannot invent — checkpoint, train index, val index, kubeconfig — from Configuration.

Stage 1: Build the task pool

A run samples rows from a task index built entirely offline. Start from Data Preparation, which turns a raw issue dataset into task directories, images and a parquet pointer table.

The output of this stage is a path, not a dataset in the trainer's memory: TRAIN_FILES and VAL_FILES are thin tables pointing at Harbor task directories.

Stage 2: Measure the pool

Runs the agent over the pool with the policy frozen and scores every attempt with the same verifier training uses, then keeps the instances whose pass rate is strictly between 0 and 1.

cp scripts/infer/_template.env scripts/infer/configs/my_infer.env

bash scripts/infer/infer.sh --dry-run scripts/infer/configs/my_infer.env
nohup setsid bash scripts/infer/infer.sh scripts/infer/configs/my_infer.env \
  > logs/infer_$(date +%m%d).out 2>&1 &

A training run accepts an unfiltered index, so this stage is optional — and expensive to skip. Under group-relative advantages an instance solved in every attempt or in none has zero variance and buys nothing but wall-clock time; on one raw 14,415-instance shard, roughly 90% of the pool was in that category.

Stage 3: Train

Each step samples instances from the index, runs the agent against them in sandboxes, grades the result with each task's own test suite, and takes a gradient step on the resulting trajectories.

cp scripts/train/_template.env scripts/train/configs/my_run.env

PREFLIGHT_ONLY=1 bash scripts/train/train.sh scripts/train/configs/my_run.env
nohup setsid bash scripts/train/train.sh scripts/train/configs/my_run.env \
  > logs/train_$(date +%m%d).out 2>&1 &

For a multi-node run, start the same config on every node; the node whose local IP matches the Ray head becomes rank 0 automatically. /rl:run prints the per-node command and never SSHes anywhere.

Stage 4: Merge and score

A training checkpoint is FSDP/veomni shards, not a servable model. Merge one before scoring it, then evaluate the merged directory against a held-out set that does not overlap the training index.

python -m verl.model_merger merge \
    --backend fsdp \
    --local_dir checkpoints/<project>/<exp>/global_step_30/actor \
    --target_dir /models/checkpoints/global_step_30_hf

cp scripts/eval/_template.env scripts/eval/configs/my_eval.env
bash scripts/eval/eval.sh --dry-run scripts/eval/configs/my_eval.env
nohup setsid bash scripts/eval/eval.sh scripts/eval/configs/my_eval.env \
  > logs/eval_$(date +%m%d).out 2>&1 &

Score the base model with the same config and only MODEL_PATH changed. Without that baseline a number has nothing to be compared against, and scoring noise has been mistaken for a training gain before.

How the stages hand off

Nothing is passed implicitly between stages: each handoff is a path one config writes and the next config reads. A broken edge is a validation failure at the next launch, not a mid-run surprise.

Sharded infer runs are the one place this needs a manual step: union the per-node OUTPUT_INDEX files into one parquet before training reads them (how).

What to hold each stage to

Each stage has one condition that decides whether the next one is worth its GPU hours. Check it before moving on, not after.

Running a slice

Partial pipelines are normal. Each stage reads paths and writes paths, so any prefix or suffix of the chain is a valid run on its own:

• Stage 2 alone when you want a trajectory corpus or a difficulty profile of a model you are not training.
• Stages 3–4 when the filtered index already exists.
• Stage 4 alone to score a checkpoint someone else produced, or to establish a base-model baseline before any training starts.

While it runs

/rl:status

Read-only diagnosis of a run in flight: which run is alive, how far it has got, and whether its numbers are sound. It matches the metrics against the failure signatures this stack keeps producing — R3 misalignment, frozen model, gradient starvation, env avalanche, val fake-zero, no-tool-call collapse, deadlock — and says so when none of them fit rather than forcing a match.

When a number goes wrong: read the evidence, form one hypothesis, change the smallest thing responsible, and rerun only the stages that change affects. A filtered index does not have to be rebuilt because a training config was wrong. Start from the Failure Playbook.

After a run

Every stage writes under the repo root, named by PROJECT_NAME / EXP_NAME:

logs/<exp>.log                                    step metric lines, then the final tally
logs/<exp>_vllm.log                               serving / throughput log
harbor_trials/<project>/<exp>/step_*/<session>/   per-trial trajectories
checkpoints/<project>/<exp>/global_step_N/        actor shards, every TRAINER_SAVE_FREQ

The log and the trials directory answer what happened: which step the run reached, what the agent actually did on a given task, why a trial scored zero. Every launch also prints its fully resolved configuration block into the log before starting, so a result stays explainable after the config file has moved on.

The dashboard answers whether the result is any good:

/rl:dashboard

Reward, KL, entropy, response-length and MFU curves, plus a browser over the per-trial trajectory JSON. It reads logs/ and harbor_trials/ directly with no separate database, so it is as current as the files on disk and works on a finished run too. See Monitoring and Results & Artifacts.

What reproducible means here

Exact trajectories, rewards and scores will not repeat bit for bit. Sampling, sandbox scheduling, package mirrors, container startup, GPU kernels and retries all move underneath a run, and an agent's recovery from a failure depends on the failures it happens to see. Two runs of the same config are two samples, not one result twice.

What Lego-RL does make reproducible is procedural and structural. The run config is a single file and the only source of truth for a run; the resolved configuration is printed in full before anything expensive starts; the proxy captures the exact tokens and log-probabilities the agent produced, so what the trainer saw is recoverable from the trajectories rather than re-derived; and each stage reads and writes explicit paths, so one stage can be rerun without disturbing the rest.

Quote a number with the config, the index and the checkpoint that produced it. Compare two runs by diffing their configuration blocks, not by remembering what was changed.

Next