Demo Run

scripts/train/examples/demo.env trains a policy on real repository tasks at 1/16th the default scale: 8 prompts × 4 responses = 32 trials/step instead of 64 × 8 = 512. Everything else about it is a production run — the same sandboxes, agent, verifier and gradient path.

A plumbing check, not a learning signal

At 32 trials/step the gradients are noisy and reward groups degenerate. The demo proves the path works end to end. Read nothing into its reward curve.

Prerequisites

These are the pieces Lego-RL cannot create for you:

8× GPU (A100/H100-class). On this single-node config vLLM serves the policy and verl runs the trainer on the same host.
A reachable Kubernetes cluster, which is what demo.env selects (BACKEND=k8s). Each trial runs as a pod. BACKEND=docker drives a single machine's daemon instead — see Sandbox Backends.
The four values from Configuration: checkpoint, train index, val index, kubeconfig.
An image source. demo.env sets HARBOR_K8S_INLINE_BUILD=true, so pods build each task image on first use and no registry is needed — slow, but it works on a bare cluster. An unreachable image source is the most common cause of a first run that produces nothing but env_setup_failed.

Launch

From the repository root, in Claude Code:

/rl:run scripts/train/configs/demo.env

That one command is the whole demo. It will:

Refuse if a run is already in flight on this host, rather than starting a second one that fights the first for GPUs.
Run the full validation — the structure checks plus the ones --structure-only skipped: the checkpoint, both indexes and the kubeconfig must now exist. Any ✗ FATAL aborts. There is no force flag.
Print the resolved parameters and wait for you to confirm — data, scale, estimated hours per step, where the log will land, whether the dashboard can see it.
Launch in the background, because these runs outlive a shell session.
Report the real exp_name, the PIDs, the log paths and the first-step gates worth watching.

Start Claude Code from the repository root

.claude/settings.json registers the plugin as a local marketplace, so the /rl:* commands only appear in a session started at the repo root. If they are missing, see Setup FAQ.

Without the plugin

The same run by hand — validation and the nohup discipline are then yours to remember:

PREFLIGHT_ONLY=1 bash scripts/train/train.sh scripts/train/configs/demo.env

nohup setsid bash scripts/train/train.sh scripts/train/configs/demo.env \
  > logs/train_$(date +%m%d).out 2>&1 &

What happens next

The runner brings up Ray, boots vLLM, starts the in-process proxy, and drives the GRPO loop. On a 30B MoE at TP=4, vLLM CUDA-graph capture takes 10–20 minutes before the first replica registers. Nothing is wrong during that window.

Watch the first step

The first step decides whether the rest of the run is worth its GPU hours.

/rl:status

Or read the same signals by hand:

Signal	Healthy	If it is wrong
`router_replay/pearson` (MoE only)	≈ 0.999	rollout and training disagree on expert routing; the gradients are noise
`actor/lr`	the configured value	a decaying schedule with `total_training_steps=-1` collapses to `0`
`critic/rewards/mean`	non-zero within a step or two	an all-zero reward is almost always infrastructure, not the model

tail -F logs/demo.log

Monitor

/rl:dashboard

Serves the dashboard on http://<host>:8090, after checking that the port is free and that it is pointed at the directory this run actually writes to. By hand: bash webui/start_dashboard.sh.

Output

demo.env sets TRAINER_SAVE_FREQ=5, so the first checkpoint lands at step 5 rather than step 10:

logs/demo.log                                     step metric lines
harbor_trials/lego-rl-demo/demo/step_*/<session>/ per-trial trajectories
checkpoints/lego-rl-demo/demo/global_step_N/      actor shards

Results & Artifacts explains each.

Scaling it up

Once the demo completes a few steps, the production values are the ones demo.env overrode:

TRAIN_BSZ=64
TRAIN_MINI_BSZ=64
N_RESP=8
TRAINER_VAL_BEFORE_TRAIN=True    # baseline before the first update

HARBOR_K8S_INLINE_BUILD=false    # stop rebuilding every task image per trial
HARBOR_OPENSWE_IMAGE_REGISTRY=<registry>   # prebuilt images instead

Swapping inline build for a registry is the change that matters most at scale: per-trial dependency installs do not survive hundreds of instances.

The Configuration Reference covers the rest — other backends, scaffolds, async mode and multi-node topologies.

Goal	Page
Go from a task pool to a scored checkpoint	Running the Pipeline
Every setting the demo did not need	Configuration Reference
Score a checkpoint on a held-out set	Evaluation
Sample the agent without training, to measure task difficulty	Rollout Generation
Build a task set from a raw issue dataset	Data Preparation
One host is no longer enough	Scaling Up
Reading the curves and trajectories	Monitoring
Something went wrong	Failure Playbook · Setup FAQ
What a trial, rollout and reward actually are	Core Concepts