Training

Trains a policy on real repository tasks. Each step samples instances from a task index, runs the agent against them in sandboxes, grades the result with each task's own test suite, and takes a gradient step on the resulting trajectories.

task index ─▶ [ training ] ─▶ checkpoints ─▶ evaluation

Prerequisites

• A train and a val task index (parquet pointer tables). Difficulty-filtered indexes train faster; see Rollout Generation.
• Sandbox images for those tasks, in a registry the cluster can pull from.
• A policy checkpoint on local disk, typically SFT-trained.
• A site.env describing the cluster, and the venv from Getting Started.
• 8× GPU per node. Multi-node needs a reachable Ray head; see Scaling Up.

Setup

A run config starts as one template and replaces its CHANGEME values:

cp scripts/train/_template.env scripts/train/configs/my_run.env

The config does not pick a template file per axis — it sets four values, and the runner composes the matching modules from scripts/templates/ at launch:

Identity and topology

# scripts/train/configs/my_run.env
PROJECT_NAME=my-project
EXP_NAME=first-run
SCAFFOLD=ohsdk                      # cc | ohsdk | oc | terminus
BACKEND=k8s                         # k8s | docker
TRAINING_MODE=async                 # sync colocates rollout and training
MODEL_ENGINE=veomni                 # veomni for MoE, fsdp for dense

NNODES=3                            # must equal the two below
N_NODES_TRAIN=2
N_NODES_ROLLOUT=1
SP_SIZE=8                           # must divide the training world

PROJECT_NAME and EXP_NAME name every artifact the run writes, so pick them before launching rather than renaming directories afterwards.

Model and data

MODEL_PATH=/models/Qwen3.5-35B-A3B
TOOL_CALL_PARSER=qwen3_coder             # must match the model's chat template

TRAIN_FILES=/data/harbor_indexes/train.parquet
VAL_FILES=/data/harbor_indexes/val.parquet

TOOL_CALL_PARSER mismatched against the model's chat template is the single most expensive typo here: the agent emits tool calls the server cannot parse, every trajectory ends early, and reward stays near zero for reasons that look like the model's fault. Validation checks this pair.

Batch and context

TRAIN_BSZ=64                        # prompts per step
N_RESP=8                            # responses per prompt → 512 trials/step
MAX_PROMPT=30000
MAX_RESP=170000
TRAINER_SAVE_FREQ=10
TRAINER_TEST_FREQ=10

Everything else takes a default. Configuration lists every value a run reads and which layer each one belongs to.

Check

PREFLIGHT_ONLY=1 bash scripts/train/train.sh scripts/train/configs/my_run.env
# or:  /rl:check scripts/train/configs/my_run.env

This resolves the config, prints the run configuration block, and runs every assertion in Run Validation. It launches nothing and stops on any ✗ FATAL. To see the fully expanded launch command instead:

DRY_RUN=1 bash scripts/train/train.sh scripts/train/configs/my_run.env

/rl:check runs the same assertions and adds what a config-only script cannot judge: whether a run is already in flight, whether those GPUs are yours.

Run

nohup setsid bash scripts/train/train.sh scripts/train/configs/my_run.env \
  > logs/train_$(date +%m%d).out 2>&1 &
# or:  /rl:run scripts/train/configs/my_run.env

A run lasts days. Launch it under tmux or nohup setsid so it survives a shell disconnect. For multi-node runs, start the same config on every node; the node whose local IP matches the Ray head becomes rank 0 automatically. If a previous run left Ray or vLLM processes behind, clear them first with bash scripts/cleanup_before_run.sh.

The runner brings up Ray, boots vLLM, starts the in-process proxy, and drives the PPO/GRPO/GSPO loop. On a 30B MoE at TP=4, vLLM CUDA-graph capture takes 10–20 minutes before the first replica registers.

Rollout dominates: timing_s/gen is 70–80% of the step in both. Adding rollout nodes, not training nodes, is what shortens a step.

First step

The first step decides whether the rest of the run is worth its GPU hours. Three numbers, in the order they appear:

tail -F logs/<exp_name>.log
# or:  /rl:status

Output

logs/<exp>.log                                    step metric lines
logs/<exp>_vllm.log                               throughput-only log
harbor_trials/<project>/<exp>/step_*/<session>/   per-trial trajectories
checkpoints/<project>/<exp>/global_step_N/        actor shards, every TRAINER_SAVE_FREQ

A checkpoint is FSDP/veomni shards, not a servable model. Merge one to HuggingFace format before scoring it:

python -m verl.model_merger merge \
    --backend fsdp \
    --local_dir checkpoints/<project>/<exp>/global_step_30/actor \
    --target_dir /models/checkpoints/global_step_30_hf

That directory is the MODEL_PATH of an Evaluation run. Results & Artifacts documents every stream in full.

Dashboard

bash webui/start_dashboard.sh          # http://<host>:8090 (+ public tunnel)
# or:  /rl:dashboard

Reward, KL, entropy, response-length and MFU curves, plus a browser over the per-trial trajectory JSON. It reads logs/ and harbor_trials/ directly, so it works on a finished run too. See Monitoring.

Cleanup

bash scripts/cleanup_before_run.sh     # ray + vLLM + litellm workers, orphan pods, /tmp

Safe to re-run, and safe when nothing is up. Run it after any run that did not exit cleanly: orphaned litellm workers keep holding port 8002, and the next launch's health probe is happy to talk to a proxy whose vLLM backends are gone.

Checkpoints stay where verl wrote them. On a finished run, deleting the optimizer shards and keeping model_world_size_* frees roughly two thirds of the space — the merger only reads the model shards.

Parameters