Evaluation

Scores a fixed checkpoint on a held-out Harbor dataset. Serving is a plain vLLM server with real weights (load_format=auto) plus harbor run. This is deliberately not the verl val_only path, whose colocated dummy-weight initialization can mis-map fused MoE weights (details).

merged checkpoint ─▶ [ evaluation ] ─▶ per-task rewards + trajectories

Prerequisites

• A checkpoint in HuggingFace format. A training checkpoint must be merged first — see Training → Output.
• A held-out dataset: a directory of Harbor task directories, or a registry name. It must not overlap the training index.
• Sandbox images for those tasks, pullable from the cluster. Val images are a common gap: the official SWE-bench images are not the ones a training run built (details).
• Enough GPUs for one vLLM server at GEN_TP.

Setup

cp scripts/eval/_template.env scripts/eval/configs/my_eval.env

BACKEND and SCAFFOLD are the two axes an eval config picks; the runner composes the rest from scripts/templates/.

Identity and checkpoint

# scripts/eval/configs/my_eval.env
PROJECT_NAME=eval-qwen36-27b
EXP_NAME=step30
SCAFFOLD=ohsdk
BACKEND=k8s
MODEL_PATH=/models/checkpoints/global_step_30_hf

Score the base model with the same config and only MODEL_PATH changed. Without that baseline a number has nothing to be compared against, and scoring noise has been mistaken for a training gain before.

Dataset

# exactly one of the two
DATASET_PATH=/data/harbor_swe_val_tasks_official
# DATASET_NAME=swebench-verified
# N_TASKS=500

Sampling and context

EVAL_TEMPERATURE=1.0
MAX_INPUT_TOKENS=200000
MAX_OUTPUT_TOKENS=32768
HARBOR_AGENT_MAX_TIMEOUT_SEC=7200   # 2 h per task; what every shipped template sets

A per-task timeout that is too tight does not lower the score honestly — it truncates the slow half of the tasks and reports them as failures.

Serving

GEN_TP=8
EVAL_ENABLE_EXPERT_PARALLEL=false   # MoE: true, dense: false
EVAL_VLLM_EXTRA_ARGS="--enforce-eager --enable-prefix-caching --dtype bfloat16"
N_CONCURRENT=16

Check

bash scripts/eval/eval.sh --dry-run scripts/eval/configs/my_eval.env
# or:  /rl:check scripts/eval/configs/my_eval.env

Resolves the config, prints the resolved run block and the launch command, and exits. PREFLIGHT_ONLY and its assertion suite are wired into the train runner only; eval and infer validate through --dry-run.

Run

nohup setsid bash scripts/eval/eval.sh scripts/eval/configs/my_eval.env \
  > logs/eval_$(date +%m%d).out 2>&1 &
# or:  /rl:run scripts/eval/configs/my_eval.env

Cost intuition. Wall clock is tasks ÷ N_CONCURRENT rounds, each bounded by HARBOR_AGENT_MAX_TIMEOUT_SEC, plus 10–20 minutes of vLLM startup. A full long-context held-out set at N_CONCURRENT=16 is an overnight job — one such run on our reference cluster took ~11.5 h.

tail -F logs/<exp_name>.log   # vLLM boot → per-task progress → final tally
# === DONE: solved=163 / scored=489  (harbor_trials/<exp_name>) ===

Output

logs/<exp>.log                              progress and the final tally
logs/<exp>_vllm.log                         serving log
harbor_trials/<exp>/<task>/                 per-task trajectory + reward.txt

find harbor_trials/<exp_name> -name reward.txt | wc -l          # tasks scored
find harbor_trials/<exp_name> -name reward.txt -exec cat {} + \
  | grep -c '^1$'                                               # tasks solved

scored is the denominator that matters: tasks whose sandbox never came up are not failures of the model, and a score computed over the full dataset instead of the scored subset understates the checkpoint. Check the two counts against each other before quoting a number.

Dashboard

An eval run writes no step metrics, so it does not appear on the board on its own. To read it next to the training curves, inject the finished run as a val point; Monitoring covers the layout.

Cleanup

bash scripts/cleanup_before_run.sh

An eval holds a full vLLM server. Free it before the next run rather than discovering the GPUs are still busy at launch time.

Parameters