Low Val Scores That Are Not the Model

Four infrastructure causes of a low or zero val score, how to tell them apart, and how to re-score a finished run without re-running the agent

A val score that is zero or "suspiciously low" is more often broken infrastructure than a broken policy. Four independent causes produce the same headline number, and each has a distinct fingerprint. Triage first, then fix; and when in doubt, re-grade offline. It is cheap and it settles the argument.

Triage: read `num_turns` before anything else

Look at val-aux/num_turns/{min,max,mean} for the failing step:

Fingerprint	Cause	Go to
`max = 0`: every trial zero turns	val images could not be pulled at all; the env never started	§3
`min ≥ 1`, but far fewer trials landed than the val set size	coverage shortfall: throughput / timeouts	§1
trajectories look normal, the verifier ran, yet `reward = 0`	grading infrastructure, not the model	§2
numbers plausible but disagree with an offline number	grading noise: re-grade	§4

Make sure you are looking at val trials

Distinguish val from train trials by task path: harbor_swe_val_tasks_official is val; openswe_filtered / swerebench_filtered are train.

Training gradients are not affected by any of this

Training tasks use a self-contained verifier (echo 1/0 > reward.txt) that touches neither uv nor the network, and env failures are dropped by trajectory_filter rather than fed to the update as reward=0. So these bugs pollute the val metric while the gradients stay clean, which is exactly how you end up with train reward climbing while val looks flat or depressed.

§1 Coverage shortfall (only ~170/500 scored)

Issue. The val set has 500 tasks, but only ~150–180 trials land per validation pass, and val-core divides by that smaller number. This reproduces on every model; it is a fully-async behavior, not a regression.

Reason. Validation dispatches all 500 with one asyncio.gather and no overall budget. A single rollout node cannot push 500 × ~58k-token trajectories through the ~7000 s validate window when each trial carries a 2400 s agent timeout, max_retries=1, and a 1200 s pod-startup allowance. Trials that miss the window return _make_empty_output (ENV_SETUP_FAILED) and are filtered out. The underlying driver is the very long responses.

Fix. Set the val-specific timeout and retry variables. They leave the training path alone. Unset, each falls back to its training value (retries 2 / startup 1200 s / timeout 2400 s / deadline 6000 s):

HARBOR_VAL_MAX_RETRIES=2
HARBOR_VAL_AGENT_MAX_TIMEOUT_SEC=4800
K8S_VAL_POD_STARTUP_TIMEOUT=1800
K8S_VAL_POD_ACTIVE_DEADLINE_SECONDS=9000

This is why validation warns about it

These exports were originally hand-written into one launch script, and every new script that forgot to copy them silently regressed to ~170/500. Validation now warns when HARBOR_VAL_AGENT_MAX_TIMEOUT_SEC is unset. A real fix is shorter responses (58k → ~15k) or more rollout capacity; these settings only buy headroom.

§2 No egress breaks the grader (official SWE-bench only)

Issue. Trajectories look normal and the verifier runs, yet a perfectly good patch scores reward = 0.

Reason. On an egress-less cluster, the official val set's grading path fails in two independent places:

uv missing from the image. The val task's tests/test.sh grades with uv run parser.py, and the Dockerfile installs uv via RUN curl -LsSf https://astral.sh/uv/... | sh with no ENV PATH. Without egress the curl fetches nothing and curl | sh exits 0 anyway (sh reads an empty script), so the image "builds successfully" without uv → uv: command not found. This hits ~27–30% of pods and a different subset each run. The intersection of two runs' broken sets was only 40/136, proving it is runtime-flaky, not a permanently bad image.
The parser itself reaches the network. Even with uv present, parser.py's make_test_spec() downloads test dependencies from raw.githubusercontent.com → NameResolutionError → exit 1 → reward.txt = 0.

Fix. On the image side: COPY the uv binary into the image and set ENV PATH=/root/.local/bin:$PATH; pre-cache the swebench dependencies at build time instead of downloading at runtime; add set -o pipefail so a broken curl | sh can never pass silently again.

§3 The whole val image set is unpullable

Issue. num_turns is 0 across the board, verifier_result: null, reward 0, and the training half of the same run is completely healthy.

Reason. Official val tasks carry docker_image=docker.io/swebench/sweb.eval.x86_64.* in their parquet. On a cluster with no Docker Hub egress, whose local registry only holds the training images, and with INLINE_BUILD=false + SKIP_DOCKERFILE=true, the val pod fails to pull → container not found("main") → zero turns.

Fix. Mirror the val images into the shared registry (one designated writer node; everyone else read-only) and point harbor at it:

export HARBOR_NYDUS_MIRROR=<registry-host>:5001   # rewrites docker.io/swebench/* → local mirror

The rewrite happens in _maybe_rewrite_to_nydus_mirror. The run must be restarted to pick up the new export. See also Sandbox Backends.

§4 Offline regrade

Not a failure mode but the tool that settles them. For any finished run you can re-derive the score from the archived pytest output, bypassing every failure above. No agent re-run, no image rebuild.

On a node with PyPI access: pip install swebench==4.0.3 datasets==2.16.1 'fastcore<1.11'.
For each trial, read verifier/test-stdout.txt (the raw pytest output, archived at run time).
Look up FAIL_TO_PASS / PASS_TO_PASS from harbor_swe_val_tasks_official/<instance_id>/tests/config.json.
Wrap the output in START/END_TEST_OUTPUT markers, then reproduce the official grading with get_logs_eval → get_eval_tests_report → get_resolution_status.

Reliability. Agreement with harbor's own record is 94.5%; every one of the 19 disagreements was a harbor fake zero, confirmed by hand.

What this found: read before trusting a val curve

Re-grading a Qwen3.6-27B run moved step 5 from 0.451 to 0.687 and step 30 from 0.541 to 0.697. True val was ≈0.67–0.70 and essentially flat; the reported "RL improvement from 0.45 to 0.56" was mostly decreasing grading noise, not capability gain. If a val curve is the evidence for a claim, regrade it first.

Full write-up: troubleshooting/harbor/val-false-zero-diagnosis-and-offline-regrade.md · related: troubleshooting/harbor/val-failures-pythonsafepath-test-sqlite-20260717.md