Data Preparation
Four stages from a raw issue dataset to the filtered task index a run samples from
A training run samples rows from a task index that is built entirely before the run starts. Four stages produce it. The first three turn a raw issue dataset into tasks a sandbox can run and a verifier can grade; the last one removes the instances that train nothing, which on a raw pool is most of them.
Data Selection Process
| Stage | Input | Output |
|---|---|---|
| 1. Task Instances | raw issue records from a source dataset | one task directory per instance |
| 2. Sandbox Images | each instance's Dockerfile | one image per instance, in a registry the cluster can pull |
| 3. Task Index | the task directories | a parquet table, one pointer row per task |
| 4. Difficulty Filtering | an index, plus a policy to measure it with | the rows whose pass rate is strictly between 0 and 1 |
The example below follows one OpenSWE instance,
astropy__astropy-15883, through all four.
1. Task Instances
Converts one record of the source dataset into a self-contained directory: a
Dockerfile that builds the repository at the buggy commit, an instruction.md
holding the issue text, and a verifier that runs the hidden tests. This is the
unit a sandbox runs and a verifier grades — the trainer never reads it.
Example — record astropy__astropy-15883 of
GAIR/OpenSWE becomes
harbor/astropy__astropy-15883/ in
SWE-Lego/SWE-Lego-RL-2699:
task.toml, instruction.md, environment/Dockerfile, tests/, solution/.
2. Sandbox Images
Builds each instance's environment once, offline, and pushes it to a registry every sandbox node can pull from. Without this, every trial rebuilds its image inside the pod.
Example — astropy__astropy-15883 is served as
<registry>/openswe-astropy__astropy-15883:latest; the repository checkout it
copies into /testbed is restored by environment/prepare_repos.py in the
dataset above.
3. Task Index
Writes one row per task directory into a parquet table. The row carries a path
and an instance_id, no task content, so an index costs kilobytes per thousand
tasks and every later stage is a dataframe operation.
Example — the row layout, and index/ in the
dataset above.
4. Difficulty Filtering
Measures per-instance difficulty by running the agent over the pool with the policy frozen, then keeps the band that produces gradient. Under group-relative advantages, an instance solved in every attempt or in none has zero variance and contributes nothing but wall-clock time.
Example — of one 14,415-instance OpenSWE shard, 760 were solved at least once in 4 attempts; roughly 90% of that shard would have trained nothing. Measurement and band selection.
Data Selection Example
OpenSWE, from the raw dataset to the pool released on Hugging Face.
Bar width is proportional to instance count. The last row is not a subset of the row above it: it is the union of several inference shards, each difficulty-filtered on its own.
| Dataset | What it holds |
|---|---|
| GAIR/OpenSWE | the raw issue records the conversion reads |
| SWE-Lego/SWE-Lego-RL-2699 | the 2,699 converted task directories, their source records, the index, and prepare_repos.py to restore each repository checkout |
The released directories exclude environment/repo/, the shallow checkout each
Dockerfile copies into /testbed. Restore it with
environment/prepare_repos.py; do not replace the COPY with a git clone
(why).
A pool also has to be checked for instances that contain the reference fix, and for repositories that carry it in their git history: Reward Hacking.