Agent Loop Workers

AgentLoopWorkers are the verl processes that run the trials. The trainer hands each step's batch of tasks to a pool of workers; each worker drives one scaffold through one task, start to finish, and returns the captured rollout.

The trial lifecycle

For each trial a worker:

1. Launches the configured scaffold inside a fresh Environment sandbox.
2. Points the scaffold at the In-Process Proxy (overriding its base URL with a unique per-session URL).
3. Lets the agent run multi-turn (reason, edit, run tests) until it submits or hits the turn/timeout limit.
4. Collects the verifier's reward and the per-token trajectory for the training update.

The core of the loop (src/verl_patch/agent_loop/builtin_cc_agent_loop.py, simplified):

class BuiltinCCAgentLoop(AgentLoopBase):
    async def run(self, sampling_params, **kwargs) -> AgentLoopOutput:
        task_path = kwargs["extra_info"]["harbor_task_path"]

        # in-process proxy: one singleton per worker; a fresh session per attempt
        proxy = await _ensure_proxy_started(self)

        # sandboxed trial: env setup → agent execution → verifier (reward 0/1)
        reward, trial_reason, session_meta = await self._run_harbor_trial(
            task_path=task_path, sampling_params=sampling_params, proxy=proxy)

        # token source of truth = the proxy trajectory, never a re-tokenized
        # transcript; robust to crash/timeout trials (pre-failure tokens kept)
        ids  = session_meta["traj_acc_ids"]
        mask = session_meta["traj_response_mask"]
        lps  = session_meta["traj_response_logprobs"]
        return self._make_output(ids, mask, lps, reward, trial_reason)

Worker concurrency

NUM_WORKERS sets how many trials run in parallel (16 at cold start; raise for steady state). Most of a single trial's wall-clock is spent outside the GPU, provisioning the sandbox, running commands and waiting on tests, so one trial alone leaves the inference servers mostly idle. Running many trials concurrently is the main lever for keeping them busy, and in synchronous mode the trainer with them.

How far you can raise it depends on which side saturates first, and that shifts with the deployment. The ceiling can be sandbox-side (pod churn, image pulls, node I/O and memory; see Environment) or rollouter-side (KV-cache pressure and replica throughput. Raise NUM_WORKERS against whichever resource limits your run. Do not assume the sandbox is the bottleneck.

Agent-loop classes

The worker's per-scaffold logic lives in an agent-loop class:

• BuiltinCCAgentLoop: Claude Code (Anthropic protocol).
• BuiltinSWEAgentLoop: OpenHands, OpenCode and Terminus (OpenAI protocol).

Four scaffolds (Claude Code, OpenHands, OpenCode, Terminus) are validated end to end and strictly adapted. Any other scaffold that speaks the OpenAI or Anthropic API can be integrated the same way: implement an agent-loop adapter, drop it under src/harbor_patch/agents/, and select it with HARBOR_AGENT_IMPORT_PATH:

HARBOR_AGENT_IMPORT_PATH=harbor_patch.agents.my_scaffold:MyScaffold

Every adapter funnels through the same proxy, so a new scaffold gets trajectory capture, load balancing, and R3 routing replay for free.