Reward Hacking
Where a verifier-graded reward can be earned without solving the task, and the check or fence that closes each route
Reward comes from a verifier, not a judge model, so it cannot be talked into a high score. It can still be earned without fixing anything: the reference fix may already be inside the task, inside the repository's history, or one network request away. Each route has its own detector or fence, and each has been seen on this stack.
§1 Reference fix inside the task definition
Some instances ship their own answer. The reference patch appears in two places, and each needs its own detector:
| Where it leaks | What it looks like | Effect on the run |
|---|---|---|
Dockerfile build layer | the golden diff is applied while building the image | the agent opens an already-fixed repository |
eval_body grading script | the script applies the golden patch before running tests | reward is 1 whatever the agent did |
The scan reads files. It needs no rollouts and no GPU, so it can run over a pool
at any point after conversion. Auditing 760 OpenSWE
instances removed 24 — 8 Dockerfile, 16 eval_body. A second batch audited the
same way went from 494 to 470, splitting 10 / 14 across the same two causes.
Leaking instances are always solved, so auditing the instances that were solved at least once finds every one of them at a fraction of the cost.
Do not grep for the word golden. Grading scripts are full of # Apply golden
comments, echo lines that print it, and branch names like testWithFix that
apply nothing. The reliable test is whether the script literally contains a diff
of the source files, or whether its stdout shows a patch message for them.
A related check belongs to the same pass: a verifier must fail on the unpatched repository and pass on the reference patch. One that passes either way produces a constant reward and wastes every rollout spent on it.
§2 Fix inside the repository's git history
The checkout copied into /testbed must be a depth-1 checkout of the buggy
commit, not a clone. A Dockerfile that runs git clone ships the full
history, and the fixing commit sits in it: agents read it out with git log and
git show instead of solving the issue.
This is why the released task directories exclude environment/repo/ and restore
it with environment/prepare_repos.py, which reproduces the original shallow
checkout — one commit, no refs. Replacing the COPY repo /testbed with a clone
would be simpler and would reopen the leak.
§3 Network egress from the task pod
A clean checkout is not enough on its own: the upstream repository is public, so an agent with egress can fetch the fix from GitHub.
Task pods are therefore fenced. flannel does not enforce NetworkPolicy objects,
so the rules are applied inside the pod: a netadmin sidecar
(HARBOR_NETADMIN_IMAGE, alpine + iptables) programs them, and only the sidecar
holds NET_ADMIN/NET_RAW — the main container cannot flush its own rules. The
policy is phased: during the agent phase egress is restricted to the LAN
allowlist it needs (LLM proxy, image registry, cluster DNS), and public egress is
opened for the verifier phase, where the model is no longer in the loop.
HARBOR_K8S_EGRESS_ISOLATION is the master switch. It defaults to on and exists
for emergency rollback.
Verify the fence applies, do not assume it. One sidecar release shipped an
exec hardcoded to bash in an image that has only sh; every rule application
returned 127 and the fence was silently a no-op while every pod still reported
healthy.