Failure Playbook
The failures that recur when operating Lego-RL, each as issue, reason and fix
The failures that recur when operating Lego-RL, grouped by the stage they show up in. Every entry below names the issue and its reason, and links to the page that carries the fix. Each of those pages is written the same way: Issue, Reason, Fix.
Most config-level problems never get this far
PREFLIGHT_ONLY=1 catches the mistakes that would otherwise cost a multi-hour run:
parser mismatch, topology that doesn't add up, R3 without the new verl tree, no
reachable image source. Run it first; see Run Validation.
Startup
| Issue | Reason | Fix |
|---|---|---|
response_mask must contain at least one valid token on step 0 | the tool-call parser does not match the model's output format | Set HARBOR_TOOL_PARSER |
import torch costs ~9 s on every launch | the venv sits on a network filesystem | Copy the venv to local NVMe |
every trial fails env_setup, trajectories empty | agent_name and SCAFFOLD disagree | Run Validation §6 |
pods end in ImagePullBackOff on a fresh cluster | no image source the cluster can reach | Set a registry or INLINE_BUILD |
Training runtime
| Issue | Reason | Fix |
|---|---|---|
every reward is 0.0 though the grader runs | the task build clones from the internet, inside a pod with no egress | Pre-stage the repo and mount it |
| pod startup creeps 30 s → 100 s+; throughput decays | cgroup/memcg state leaks under rollout pod churn | Reboot, then cut tmpfs emptyDirs |
| the first validation hangs forever; no traceback | do_validate blocks the actor's event loop | Make it async, or drain first |
reward flat while grad_norm is a fraction of baseline | lr=0, gradient starvation, or sequence-TIS collapse | training_env/fully-async-veomni-reward-flat-three-causes.md |
val and eval scores
| Issue | Reason | Fix |
|---|---|---|
num_turns is 0 across the whole val pass | the val images cannot be pulled at all | Mirror them into the shared registry |
| only ~170 of 500 val tasks get scored | trials miss the validate window and are filtered out | Set the val-specific timeouts |
| offline eval disagrees with the reported val number | grading noise, not capability | Regrade offline from archived output |
standalone val_only eval is garbled, even for the base model | colocated vLLM's dummy→real sync mis-maps fused MoE experts | Serve with plain vLLM instead |
Cluster and sandbox
| Issue | Reason | Fix |
|---|---|---|
| remote Docker will not start | stale FlashInfer cache, env vars missing from Ray's runtime env, a shadowed docker SDK, or no compose plugin | Four pitfalls, in order |
| nvme at 100% util with no pods being created | the Kyverno reports-controller crash-loops listing EphemeralReports | Scale it to zero |
pods stuck ContainerCreating; uv installs storm the disk | every pod writes its uv cache to overlayfs | Put the uv cache on tmpfs |
On the K8s backend, check the health baseline before diagnosing anything else. Most rollout stalls trace to one of its five indicators.
Deeper write-ups
The pages above are the short version of specific incidents. The raw investigations
live in the repository under troubleshooting/<subsystem>/ — training_env/,
harbor/, k8s/, nydus/, docker/, data_collection/, webui/ — dated,
specific to the run that hit the problem, and occasionally superseded. They are
working notes, not maintained documentation; what generalized is on these pages.
Adding one: file it under the subsystem that owned the bug, name it
<symptom>-<date>.md, and lead with issue → reason → fix. If it recurs across
runs or clusters, promote it to a page here.