Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

rewriter-queue

A resumable job queue for multi-agent LLM code synthesis. You submit a source tree and a mission; a small organization of agent roles (implementer, security/maintenance/production-readiness reviewers, a merge agent for large inputs) iterates on it — patching rather than rewriting where it can, checkpointing every real step so an interrupted run resumes instead of starting over — until the reviewers are satisfied or the iteration budget runs out.

Submit and watch jobs from a terminal, from Zed, or from Claude Code.

Not a hosted service — three small Rust binaries you run yourself, against whatever OpenAI-compatible inference endpoint you point them at.

Start with the tutorial. To see what a run did and why, see run observability. To check that a change to the pipeline did not make it slower or less effective, see benchmarks and regression tests.

Tutorial

This walks through the whole loop once, end to end: get a binary, point it at a model, submit something to rewrite, and watch it work. Fifteen minutes if you’ve already got an OpenAI-compatible inference server running somewhere; longer if you need to stand one up first (see Remote deployment for that part).

1. Get the binaries

Download the archive for your platform from the latest release and unpack it — see Installation if you’d rather build from source. Either way you end up with two binaries sitting next to each other: rewriter-queue and ai-org-orchestrator. They need to stay in the same directory — the queue’s worker finds the orchestrator by looking beside its own binary, not on PATH.

Put that directory on your PATH, or just remember the full path for the commands below.

2. Tell it where your model lives

Create ~/.config/rewriter/config.toml:

[[providers]]
name = "local"
type = "openai-compat"
base_url = "http://localhost:8001/v1"
models = ["your-model-id"]

Point base_url at any OpenAI-compatible /v1/chat/completions endpoint — a llama.cpp/vLLM server you’re running, or a hosted API. If you don’t have one yet, that’s the whole subject of Remote deployment.

3. Pick a queue mode

For trying this out on one machine, skip the server entirely — every command below falls back to a local, file-backed queue when there’s no [queue] section in the config. If you’re running the inference backend on a separate box and want to submit jobs from your laptop, start the server there instead and add a [queue] url = "..." pointing at it (see the root README’s quickstart for the server command and token setup).

This tutorial uses local mode — nothing extra to start.

4. Submit something

Pick a small, real piece of code you want rewritten or reviewed — a single source file or a small directory is a better first run than a whole project. The workspace directory holds the tool’s working state as it goes, plus one thing you provide before submitting: the mission.

The mission is a required artifact — a short statement of what V2 should actually do, not just “port this file.” Without it, the first agent in the pipeline has to guess your intent from raw source alone, which shows up later as a contract that’s technically correct but not what you meant. It’s stored the same way as every other artifact this pipeline produces (objective_contract.md, schema.rs, …), at <workspace>/artifacts/mission.md, and the run refuses to start without one:

mkdir -p /tmp/my-first-run/artifacts
cat > /tmp/my-first-run/artifacts/mission.md <<'EOF'
# Mission

V1 is a small in-memory counter store. Rebuild it as V2 with identical
behavior: increment a key's count, and read a key's current count (0 for
an unseen key).
EOF

rewriter-queue submit \
  --source /path/to/the/code \
  --workspace /tmp/my-first-run \
  --max-iter 10

That prints something like queued job #1 (0 ahead). In local mode with nothing else queued, it starts immediately.

5. Watch it work

rewriter-queue watch

A live view that refreshes once a second: which job is running, what stage it’s on (ObjectiveContract → test matrix → inductive analysis → schema → V2 synthesis), and how long it’s been going. Ctrl-C to stop watching — the job keeps running either way.

Want more detail on one job specifically?

rewriter-queue status 1

Shows the last 20 lines of its log, including which agent is running and the token counts for each call.

6. See what it’s produced, before it’s even done

You don’t have to wait for the job to finish to look at what it’s built so far:

rewriter-queue artifacts 1          # list what's been emitted
rewriter-queue artifact 1 objective_contract.md   # read one of them

objective_contract.md is usually the first thing worth reading — it’s the agreed-on shape of what’s being built (public API, invariants, edge cases, explicit exclusions) before any code gets written.

7. Get the result

In local mode there’s nothing to download — the workspace directory you passed to submit is where everything lands, no copy needed:

ls /tmp/my-first-run/artifacts/v2/   # the actual synthesized code

(rewriter-queue download 1 --out ./result is for the remote-server case — see Remote deployment — where the job ran on a different machine and you need its artifacts/ tree copied to yours in one call instead of one file at a time.)

8. If it gets interrupted

Kill rewriter-queue serve (or just Ctrl-C a local-mode run) mid-job and resubmit isn’t what you want — restart the server (or just try watch again in local mode) and the same job picks back up from wherever it had already checkpointed, not from the beginning. This is true down to the level of individual agent calls, not just whole pipeline stages — see the root README’s design notes if you’re curious how.

9. If the mission is too big

Every part of the build carries the whole schema (the Rust types and function signatures drafted for the mission), and crate files over 500 lines fail the build gate. So when the schema comes to more than 500 lines, the run stops before building anything and has a planner split the mission into 2 to 6 increments: the smallest usable core first, then the pieces that build on it. It writes them to artifacts/increments.json and finishes succeeded; rewriter-queue status shows a note: naming the new job.

The queue submits increment 1 as an ordinary job in a fresh workspace, with that increment’s own mission. When an increment succeeds, the queue submits the next one, with the crate the previous increment built added to its source as v2-so-far/. Each increment’s job name reads <name> · increment 2/4: <slug>. An increment whose own schema is still over budget fails instead of splitting again; give it a smaller mission by hand.

What’s next

  • Installation — the binary download details, and building from source if you’d rather.
  • Zed setup — submit and watch jobs from Zed’s agent panel instead of a terminal.
  • The Claude Code plugin — the same queue tools inside Claude Code, plus each reviewer persona as a standalone slash command.
  • Remote deployment — running the inference backend and the queue server on a separate (e.g. GPU) machine from the one you’re submitting jobs from.

Installation

Prebuilt binaries

Every release ships prebuilt archives for:

PlatformArchive
Linux x86_64rewriter-queue-x86_64-unknown-linux-gnu.tar.gz
Linux aarch64 (ARM64)rewriter-queue-aarch64-unknown-linux-gnu.tar.gz
macOS Intelrewriter-queue-x86_64-apple-darwin.tar.gz
macOS Apple Siliconrewriter-queue-aarch64-apple-darwin.tar.gz
curl -LO https://github.com/AndyGauge/rewriter-queue/releases/latest/download/rewriter-queue-<your-target>.tar.gz
tar xzf rewriter-queue-<your-target>.tar.gz

Each archive contains two binaries — rewriter-queue and ai-org-orchestrator — plus the license files. Keep both binaries in the same directory. The queue’s worker finds the orchestrator by looking beside its own executable path, not on PATH, so unpacking them apart from each other breaks job execution.

Put that directory on your PATH, or reference rewriter-queue’s full path directly (e.g. in Zed’s context_servers config or the Claude Code plugin’s MCP config).

No Windows build yet — job-queue’s worker uses Unix process-group APIs (libc::killpg) to reliably kill a job’s whole process tree on cancel or timeout, which doesn’t have a direct Windows equivalent as written. Runs fine under WSL in the meantime.

Building from source

Needs a recent stable Rust toolchain (rustup.rs).

git clone https://github.com/AndyGauge/rewriter-queue
cd rewriter-queue
cargo build --release

Binaries land in target/release/rewriter-queue and target/release/ai-org-orchestrator — same “keep them together” rule applies.

The lint plugins (optional)

ai-org-orchestrator’s quality gate can run ast-grep rules from ai-org-orchestrator/lints/rules/ as a non-blocking check alongside cargo build/clippy/test. It degrades gracefully if the binary isn’t installed — this is opt-in, not a hard dependency:

cargo install ast-grep --locked

The Zed extension and Claude Code plugin

Not part of the release archive — see Zed setup and claude-code-plugin/README.md for how to install each.

Zed setup

Two independent pieces: the context server (lets Zed’s agent panel submit and watch queue jobs) and the model provider (lets Zed chat with whatever backend is serving your models). You can set up either without the other.

1. The context server (zed-extension/)

This is a real Zed extension — zed-extension/extension.toml + zed-extension/src/lib.rs, compiled to WASM — not a hand-edited settings.json block. It registers rewriter-queue mcp as a context server and defaults its arguments to ["mcp"], so you don’t have to remember to add that yourself.

Build and install it as a dev extension:

rustup target add wasm32-wasip2
cargo build -p job-queue --release   # the rewriter-queue binary itself

In Zed: open the command palette and run zed: install dev extension, then point it at the zed-extension/ directory. Zed builds and loads the WASM extension itself — you don’t need to invoke the WASM build by hand.

Tell it where the rewriter-queue binary is, if it’s not on your PATH, in settings.json:

{
  "context_servers": {
    "rewriter-queue": {
      "command": {
        "path": "/absolute/path/to/rewriter-queue/target/release/rewriter-queue"
      }
    }
  }
}

Leave out command entirely if rewriter-queue is already on PATH (e.g. cargo install --path job-queue).

Point it at a queue server via ~/.config/rewriter/config.toml:

[queue]
url = "http://192.0.2.10:8003"   # wherever `rewriter-queue serve` is running

…and the token it expects, in ~/.config/rewriter/queue-token. Without a [queue] section, every command (including the ones the context server runs on your behalf) falls back to a local, file-backed queue — fine for trying this out on one machine, but you’ll usually want a real server if you’re also running the inference backend on a GPU box (see remote-deployment.md).

Once it’s connected (check the context-servers panel for a green status, not a red error dot), just ask Zed’s agent about your jobs in plain language — “what jobs are in the queue?”, “submit a synthesis run for ./src with a 20-iteration budget”, “download job 3’s artifacts to ~/Desktop” — and it calls the right tool (queue_submit/queue_list/queue_status/queue_artifacts/ queue_artifact/queue_download/queue_fetch/queue_cancel) itself. There’s no slash command for this — MCP tools are things the model calls on its own, not something you invoke directly, unless a server also exposes MCP prompts (this one doesn’t).

A live terminal view is also available without going through Zed’s agent at all: .zed/tasks.json in this repo defines a “Queue: watch” task that runs rewriter-queue watch.

2. The model provider

This is unrelated to the extension above — it’s Zed’s own openai_compatible provider type, pointed at whatever OpenAI-compatible server (llama.cpp, vLLM, etc.) you’re running your models on. Add it to settings.json:

{
  "language_models": {
    "openai_compatible": {
      "my-backend": {
        "api_url": "http://192.0.2.10:8001/v1",
        "available_models": [
          {
            "name": "your-model-id",
            "display_name": "My Model",
            "max_tokens": 128000,
            "capabilities": {
              "tools": true,
              "images": false,
              "parallel_tool_calls": false,
              "prompt_cache_key": false,
              "chat_completions": true
            }
          }
        ]
      }
    }
  }
}

"tools": true matters if you want Zed’s agent to actually call tools through this model — verified working with both llama.cpp’s --jinja mode and vLLM’s --enable-auto-tool-choice, but not every locally-served model reliably emits well-formed tool calls. If the agent starts guessing at shell commands instead of calling a tool by name, that’s usually the model’s tool-calling support, not a Zed or context-server bug — sanity-check by sending the same tool schema directly to the model’s /v1/chat/completions endpoint and see what comes back before assuming the wiring is broken.

Running two backends side by side (e.g. a general model and a coding-specialized one) on the same box just means two openai_compatible entries pointed at two different ports — see remote-deployment.md for sizing multiple backends into one machine’s memory.

Running this on a remote target

The queue server and the inference backend don’t have to be the same machine, and usually shouldn’t be — the inference backend wants a GPU, the queue server doesn’t need one at all. This covers running both on a remote Linux box (a rented GPU instance, a home server, whatever) and reaching it from your laptop.

The inference backend

Anything that speaks the OpenAI-compatible /v1/chat/completions API works — llama.cpp’s llama-server, vLLM, or a hosted API you don’t run yourself. If you’re self-hosting on a GPU box:

llama-server \
  -m /path/to/model.gguf \
  -a your-model-id \
  -ngl 999 \
  -c 65536 \
  -fa auto \
  --jinja \
  --host 0.0.0.0 --port 8001

Running more than one model on the same box is fine as long as the memory budget actually adds up — size it explicitly rather than finding out at OOM time. Two independent servers on two ports (e.g. 8001/8002) is simpler and safer than trying to multiplex one server across models. Watch in particular:

  • KV cache scales with context, not just model size. For a dense transformer: 2 (K and V) × layers × kv_heads × head_dim × context × bytes_per_element. A model with a Mixture-of-Experts or hybrid linear-attention architecture can have a very different (often much smaller) real footprint than its parameter count suggests — check the model’s own config.json, don’t assume.
  • -ctk q8_0 -ctv q8_0 (with -fa on) roughly halves KV cache size at a near-lossless quality cost — the quantization error is small and fixed per cached token, not cumulative with age, so it mostly costs you long-range recall precision in very long contexts, not general quality. q4_0 is meaningfully lossier; reach for q8_0 first.
  • Leave real headroom, not just enough to fit at idle — concurrent requests, the OS, and anything else on the box all need memory too.

The queue server

Runs anywhere that can reach the inference backend(s) over HTTP — doesn’t need to be the GPU box, and arguably shouldn’t be if you want to restart the inference backend independently of the queue.

rewriter-queue serve --bind 0.0.0.0:8003

It needs a bearer token (~/.config/rewriter/queue-token or REWRITER_QUEUE_TOKEN) and providers configured in ~/.config/rewriter/config.toml:

[[providers]]
name = "backend-a"
type = "openai-compat"
base_url = "http://<inference-box>:8001/v1"
models = ["your-model-id"]

[queue]
url = "http://<this-box>:8003"

A PATH gotcha that will bite you

quality_gate() shells out to cargo build/clippy/test (and, if installed, ast-grep) from inside the orchestrator subprocess the queue worker spawns. That subprocess inherits whatever PATH the queue server’s own process happened to have — which depends entirely on how you started it. A plain ssh host 'rewriter-queue serve ...' gets a minimal, non-login shell PATH on most systems (skips .bashrc/.profile, so ~/.cargo/bin usually isn’t on it) even though an interactive SSH session would have it. If cargo isn’t reliably found, every quality-gate check silently fails with “cargo build not available” — treated as a normal retryable error, so it doesn’t crash anything, it just quietly never actually validates code until the iteration budget runs out and broken output gets accepted anyway.

job-queue/src/worker.rs already guards against this — it explicitly sets PATH (prepending $HOME/.cargo/bin) on the orchestrator process it spawns, rather than trusting whatever it inherited. If you install cargo somewhere else, or need other tools (like ast-grep) on a non-standard PATH, that’s the place to adjust it.

Keeping it running

Neither binary is a managed service by default — write a systemd unit if you want one to survive a reboot or restart on crash. Two examples:

# /etc/systemd/system/rewriter-queue.service
[Unit]
Description=rewriter-queue server
After=network.target

[Service]
Type=simple
User=youruser
WorkingDirectory=/home/youruser/rewriter-queue
ExecStart=/home/youruser/rewriter-queue/target/release/rewriter-queue serve --bind 0.0.0.0:8003
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
# /etc/systemd/system/llama-server.service
[Unit]
Description=llama.cpp inference server
After=network.target

[Service]
Type=simple
User=youruser
ExecStart=/home/youruser/llama.cpp/build/bin/llama-server \
  -m /home/youruser/models/your-model.gguf -a your-model-id \
  -ngl 999 -c 65536 -fa auto --jinja --host 0.0.0.0 --port 8001
Restart=on-failure
RestartSec=10

[Install]
WantedBy=multi-user.target
sudo systemctl daemon-reload
sudo systemctl enable --now rewriter-queue llama-server

If you’d rather not deal with sudo/systemd at all, tmux (or screen) is a fine substitute for a personal box: start each in its own session so an SSH disconnect doesn’t kill it, and accept that a reboot needs a manual restart.

The lint plugins need ast-grep

ai-org-orchestrator/lints/ (see the root README) is optional — the orchestrator degrades gracefully if the binary isn’t found — but if you want it: cargo install ast-grep --locked on whichever machine actually runs the orchestrator subprocess (the queue server’s box, per the PATH note above).

Security posture — decide this deliberately

None of the above puts a token or TLS in front of the inference backend itself, and the queue server’s Bearer token is the only thing gating job submission (no per-job auth, no rate limiting). That’s a reasonable default for a private LAN or a locked-down VPC where you trust everything that can reach the port — it is not a default you want facing the public internet. If you’re putting either behind a real network boundary (a cloud security group, a home router’s port forward, anything multi-tenant), add a reverse proxy with TLS and real auth in front, and give llama-server its own --api-key rather than relying on network trust alone.

Run observability

Every agent in a job reports what it did to one place, so a running job can be watched in detail, a finished or failed job can be analysed without reading logs over SSH, and recurring problems turn into feature requests. This is a structured event stream added on top of the workspace files (agents/<name>/..., deviations.md), which are still written as before.

Reporting is best effort. A failure to record or send an event never fails or slows a job.

What gets recorded

Each event is one JSON object (one line of JSONL), defined in the run-events crate:

FieldMeaning
tsRFC 3339 timestamp, millisecond precision
job_idThe queue job id (absent when the orchestrator is run outside the queue)
stageThe pipeline stage that was current when the event was emitted, if any
agentWhich agent emitted it (orchestrator for stage events, QualityGate for gate runs)
kindOne of the kinds below
summaryA one-line description
detailOptional JSON object; keys depend on the kind
KindWhen it is emitteddetail keys
stage_startA pipeline stage begins (ObjectiveContract, Acceptance criteria, Test matrix refinement, Inductive analysis, Schema, V2 synthesis). Starting a stage ends the previous one.none
stage_endA stage endsnone
agent_callOne per plain model call, and one per agentic call (tokens and latency are summed over its turns; the summary says how many turns)provider, model, input_tokens, output_tokens, latency_ms, ok
decisionAn agent logged a decisiontext (full text, capped at 4000 characters)
deviationA deviation from plan was loggedtext
findingAn agent called report_findingseverity (info, warn, error), category, evidence
gate_resultA quality-gate run (build, clippy, test, fmt)passed, error_bytes, attempt, variant for parallel milestone build directories, and integration: true on runs of the final whole-crate gate
retryA retrytext
errorAn error, including a panic in the orchestrator (a panic hook records it)text

attempt counts consecutive gate runs against the same build directory since it last passed. A failed model call is an agent_call with ok: false.

A stage with a stage_start and no stage_end means the process died in it. Stages that were resumed from a checkpoint emit no stage events.

Where events are stored

  • In the job’s workspace: <workspace>/events.jsonl. The orchestrator always appends here, whether or not a queue server is involved.
  • On the queue server: <queue data>/events/<id>.jsonl, with <id> zero-padded to four digits like the job files. The orchestrator posts each event to POST /jobs/<id>/events.
  • Analyses: <queue data>/analysis/<id>.json, written by the worker after a job ends.
  • Feature requests: <queue data>/feature-requests.json, one list across all jobs.

<queue data> is $REWRITER_QUEUE_DIR (default ~/.local/share/rewriter/queue).

When you read a job’s events, the server’s copy is used if it exists and the workspace file otherwise, so a local queue (no server) works the same way.

Environment variables

The worker sets these when it launches the orchestrator:

VariableSet when
REWRITER_JOB_IDAlways
REWRITER_QUEUE_URLOnly when the worker is hosted by rewriter-queue serve (a loopback URL for the bind address, so 0.0.0.0:8003 becomes http://127.0.0.1:8003)
REWRITER_QUEUE_TOKENOnly when the worker is hosted by serve

A standalone worker removes any REWRITER_QUEUE_URL and REWRITER_QUEUE_TOKEN it inherited, so the orchestrator never posts anywhere unexpected. The orchestrator posts only when both the URL and token are present and non-empty.

CLI

All four commands work against a remote server or a local queue. Add --json to events, analysis and requests for machine-readable output.

rewriter-queue events <id> [--kind <kind>] [--agent <name>] [--since <timestamp>] [--json]

Works while the job is running. One line per event, oldest first, in the form <ts> <kind> [<stage>] <agent> <summary>:

2026-10-02T14:03:11.204Z stage_start [Schema] orchestrator Schema
2026-10-02T14:03:40.918Z agent_call [Schema] SchemaArchitect SchemaArchitect: 5120in/1873out in 29714ms
2026-10-02T14:05:02.377Z gate_result [V2 synthesis] QualityGate quality gate failed (attempt 1, 2048 bytes of errors)

(These lines are illustrative; the shape comes from the rendering code.) --kind takes the snake_case kinds above, and --since returns only events strictly after that RFC 3339 timestamp. With no events it prints (no events). --json prints the raw JSONL instead.

rewriter-queue analysis <id> [--json]
job #12 failed

<summary>

Root causes:
- <cause> (evidence: <agent@ts>; ...)

Feature requests:
- <id> (<title>)

The root-causes and feature-requests sections are omitted when empty. If the job has no analysis yet the command errors with no analysis for job <id>.

rewriter-queue requests [--json]
<id> [<status> / <priority>] <title>
    jobs: #12, #15
    <rationale>

With none it prints (no feature requests).

rewriter-queue request <id> <open|accepted|done|rejected>

Sets a request’s status and prints it in the same format as requests. An unknown request id is an error.

Server routes behind these (same bearer-token auth as the rest): POST /jobs/<id>/events (one event object or a JSON array; the job must exist), GET /jobs/<id>/events (query kind, agent, since; JSONL), GET /jobs/<id>/analysis, GET /feature-requests, POST /feature-requests/<id>/status (body {"status": "accepted"}).

MCP tools

rewriter-queue mcp exposes three new tools alongside the existing ones:

  • job_events: id (required), optional kind, agent, since. Returns the events as JSONL, oldest first. Works on a running job.
  • job_analysis: id. Returns the analysis as JSON; errors if the job has none yet.
  • feature_requests: no arguments. Returns every request as JSON.

There is no MCP tool for changing a request’s status; use the CLI.

Post-run analysis

When a job ends, whatever the outcome, the worker runs ai-org-orchestrator analyze --job-id <id> --events <file> --out <file> over that job’s events (the server’s copy if present, else the workspace file). If neither exists the analysis is skipped. It has a 15 minute timeout, and any failure is one [worker] analysis ... line in the job’s log; the job’s outcome is never affected.

The analysis has two parts:

  1. Mechanical facts, computed in Rust: model calls, failed calls, tokens and latency per agent; retries per stage; gate runs and each failed gate run (stage, attempt, error bytes); stages that never ended; and an inferred outcome. The outcome is failed if the last event is an error, a stage never ended, or the last stage_end has detail.ok == false; cancelled if the last event’s summary mentions “cancel”; failed for an empty event file; otherwise succeeded.
  2. A PostRunAnalyst agent is given those facts plus the findings, deviations and errors, and proposes a summary, root causes and feature requests.

If the model call fails or its reply is not valid JSON, the analysis is still written with the mechanical summary and no root causes or feature requests.

How claims are validated

The analyst must cite evidence for every root cause and feature request, and the code checks each citation against the real events. A citation is accepted if it is either an agent@ts reference matching an event’s agent and timestamp exactly, or a quote of at least 8 characters that appears (case-insensitively) in some event’s summary or in a string in its detail. Citations that match nothing are dropped. A root cause or request left with no valid citation is dropped entirely. Surviving root causes are stored as one string, <text> (evidence: <cites>).

Feature requests

A feature request has an id, title, rationale, evidence, source_job_ids, priority (low, medium, high) and status.

Deduplication. The id is a kebab-case slug of the title (lowercase alphanumerics joined by single hyphens), so the same title in two jobs gets the same id. When the worker merges an analysis into feature-requests.json:

  • a request with a new id is added, always as open, whatever the analyst said;
  • a request whose id already exists keeps its current status, title, rationale and priority, and gains the new job in source_job_ids and any new evidence entries.

Within one analysis, repeated ids are collapsed to the first.

Statuses. The tool stores and shows the statuses but attaches no behaviour to them:

  • open: newly raised, not yet looked at.
  • accepted: you intend to build it.
  • done: built.
  • rejected: you decided against it. A later job that raises the same problem is merged into this request, which stays rejected.

Those meanings are the natural reading of the names; the code only stores the value you set.

The report_finding tool

Agents running agentically (with the tool box) can call report_finding to record something they noticed: a missing or misleading tool or skill, malformed input, a pattern that wasted effort, a risk in the source. It does not change the agent’s task or answer.

ArgumentRequiredNotes
severityyesinfo, warn or error
categoryyesShort kebab-case label (lowercase letters, digits and hyphens, at most 60 characters), for example missing-tool
textyesWhat was noticed; becomes the event summary (capped at 4000 characters)
evidencenoA quote or file/line reference

Invalid arguments are returned to the agent as an error string so it can retry. The tool is described in the reminder text appended to agentic calls. The result is a finding event.

When the queue server is unreachable

The job is unaffected. Each event is first appended to <workspace>/events.jsonl. The post to the server has a 2 second timeout; on a failure the orchestrator logs one line to stderr, then pauses posting for 30 seconds before trying again. Events emitted during a pause are not re-sent later, so the server’s copy can have gaps (see Limitations). The workspace file is complete as long as the workspace is writable; if it is not, one line is logged and events are lost.

Limitations

  • The final integration step has no stage events of its own, because it runs inside the V2 synthesis stage. Escalations emit retry events, and stages resumed from a checkpoint emit no stage events, so a stage’s measured duration can include a cached stage after it.
  • analyze skips the startup qualification probes, so its provider ratings are the defaults. That is deliberate (it makes one short call), but it means the analyst call is not routed by measured quality.
  • The feature-request file lock has only been exercised between threads in one process.
  • Events dropped while the orchestrator is pausing posts after a failed POST are not re-sent to the server; the workspace events.jsonl still has them, and analysis merges both copies.

Watching without being able to change anything

Give the server a second, read-only token and point the MCP server at it:

# on the machine running `rewriter-queue serve`
echo "some-long-random-string" > ~/.config/rewriter/queue-read-token
# restart `rewriter-queue serve`

# on the machine you watch from
REWRITER_QUEUE_URL=http://host:8003 REWRITER_QUEUE_READ_TOKEN=some-long-random-string \
  rewriter-queue mcp --read-only

The read-only server offers queue_list, queue_status, queue_artifacts, queue_artifact, queue_download, job_events, job_analysis and feature_requests. The permission is enforced by the queue server, not just hidden in the client: the read-only token gets a 403 on anything that submits, cancels or changes a request, and on downloading a job’s whole uploaded workspace.

Benchmarks and regression tests

The suite answers one question for every change to the pipeline, the prompts, the skills or the model configuration: did it get slower or less effective? A change that fixes the one problem it was written for must not quietly make five others worse.

It exists because the pipeline is not deterministic (it calls live models) and its failures are subtle. Two real ones shaped it:

  • A job (#26) reported succeeded and produced a program that did nothing. The contract copied a stub V1 and the contract reviewer was never shown the mission, so nothing checked the contract against what was asked for. The pipeline’s own gates passed because an empty main builds, lints and tests cleanly. This is why pass or fail is decided by a hidden acceptance check, never by the pipeline’s own verdict.
  • Another job (#24) produced a correct rectangle drawing program that failed its own tests over one newline: the output ended in a doubled trailing newline. Exact comparison of stdout, stderr and exit code, with escaped output in the failure message, is how the suite sees this.

Both are now tasks (rect covers both), and every future bug report should become one.

Two tiers

TierWhat it isCostWhen
1. Structure testsScripted-provider tests that pin down how the pipeline is wiredFree, deterministic, secondsEvery change, as part of cargo test
2. Live benchmarkReal tasks run through the real pipeline against a live model, then checked by hidden acceptance checksReal model time and tokensOn demand, before a release, or nightly on the inference box

Tier 1 catches “a refactor removed a behaviour”. Tier 2 catches “the behaviour is still there but the results got worse”.

Tier 1: structure tests

The tests live in ai-org-orchestrator/src/pipeline_structure/ and run with the rest of the workspace:

cargo test

Each test drives the real Manager over a scripted provider that answers from canned text keyed on the calling agent’s system prompt. It then asserts on which agents were called, in what order, what was in their prompts, and on the emitted event stream. No model is involved. What they guard:

  • Plan review. The plan reviewer runs before implementation and its corrected plan is the one implemented; an approval keeps the original plan; a review call that errors falls back to the original plan and the run continues; a nested plan’s review is given the plans above it.
  • Debug and repair. The first failed repair attempt does not consult the debugger but later ones do, and the diagnosis reaches the repair prompt; a failing debugger call does not stop the repair loop; escalation to the milestone planner happens only after the repair budget is spent and carries the diagnosis.
  • Contract review. The contract reviewer is shown the mission and the contract it reviews; a rejected contract is revised with the reviewer’s reasons and reviewed again; the entry point hands the mission to both the contract author and the reviewer. The last of these reads main.rs as text, because the contract task is built inline there, so it fails if that wiring is edited away.
  • Observability. An unwritable events file, or an unreachable queue server, does not fail the run, and a healthy run records each model call as an event.
  • Call budget. One easy milestone, taken from plan to production review, stays within a fixed number of model calls, and the events agree with the provider’s own call count.

Updating the call budget on purpose

The budget test uses MAX_CALLS_HAPPY_PATH in ai-org-orchestrator/src/pipeline_structure/budget.rs. It is set one above what the pipeline uses today, so adding an agent to a step on the happy path fails the test. If the extra call is intended, raise the constant in the same change and say why in the commit message. The failure message lists the roles that were called, so you can see which one is new.

Tier 2: running the live benchmark

The binary is rewriter-bench, from the bench crate:

cargo build --release -p bench
target/release/rewriter-bench <command>

The commands are run, compare, promote, validate-tasks and list. Run it from inside the repository checkout: the environment stamp records the git commit of the current directory.

run

rewriter-bench run [--tasks a,b] [--repeat N] [--label L] [--direct]
    [--queue-bin P] [--orchestrator-bin P] [--tasks-dir D] [--results-dir D] [--keep]

For each task and each repeat, it prepares a work directory holding the task’s source/ and mission.md, runs the pipeline, reads the run’s events, then runs the task’s acceptance checks on the finished crate and records the result.

FlagMeaningDefault
--tasks a,bComma-separated task names. An unknown name is a usage error.all tasks
--repeat NRuns per task. Must be at least 1.3
--label LA name for the run, used in the results file namerun
--directRun ai-org-orchestrator directly instead of going through the queueoff (queue mode)
--queue-bin PThe rewriter-queue binary (queue mode)found on PATH
--orchestrator-bin PThe ai-org-orchestrator binary (--direct)found on PATH; --direct fails if it is not found
--tasks-dir DWhere tasks livebench/tasks
--results-dir DWhere results gobench/results
--keepKeep each run’s work directory (workspace, output crate, orchestrator log in direct mode)off, work directories are deleted

The default is every task, three times each: five tasks makes fifteen pipeline runs. For a quick check, name the tasks:

rewriter-bench run --tasks rect --repeat 3 --label before-prompt-change
rewriter-bench run --tasks hello-trivial,wc-lite --repeat 1 --label smoke
rewriter-bench run --direct --orchestrator-bin target/release/ai-org-orchestrator --tasks rect

Queue mode submits each task with rewriter-queue submit, polls rewriter-queue status every 5 seconds, then reads the events with rewriter-queue events <id> --json (falling back to the workspace events.jsonl). A job must be processed by something, so a worker (rewriter-queue worker, or a serve setup) has to be running; the benchmark only submits and waits. If the output crate is not in the workspace it is fetched with rewriter-queue fetch. A job that is still going when its time budget runs out is cancelled and recorded as a timeout. The budget starts counting when the job leaves the queued state, so waiting behind other jobs does not use it up.

Direct mode runs the orchestrator as a child process with --workspace, --source and --max-iter, and a clock of the task’s time budget. The pipeline outcome is succeeded if it exits 0, failed otherwise, and timeout if it was killed. There is no job id.

run exits 0 whenever it completes, even if tasks failed. It prints a table of every run and the path of the results file. A failing task is a result, not an error; use compare to decide.

What a results file contains

One run writes one pretty-printed JSON file, bench/results/<timestamp>-<label>.json (the label is sanitised to letters, digits, -, _ and .). bench/results/ is gitignored. The file has a label, an environment stamp and one record per run.

Each record:

FieldMeaning
task, repeatWhich task and which repeat (counted from 0)
job_idThe queue job id; null in direct mode
passedThe acceptance checks all passed and the run finished within its time budget
failed_checksNames of the failed checks, or setup, timeout or no-crate
pipeline_outcomesucceeded, failed, cancelled or timeout, as the pipeline reported it. It never decides passed.
wall_secsWall-clock time for the pipeline run
input_tokens, output_tokensSummed over every model call
model_callsNumber of model calls, including failed ones
retriesNumber of retry events
gate_failuresQuality-gate runs that failed
integration_attemptsRuns of the final whole-crate gate (gate events marked integration)
escalationsRetries recorded by the milestone planner (a milestone sent back for a finer split)
plan_revisedThe plan reviewer recorded a decision, meaning it changed the plan
debugger_consultedThe debug implementer made at least one call
startedWhen the run started

The metrics are derived from the run’s events.jsonl (see run observability). If no events could be read, they are zero.

The environment stamp records the git commit (short hash, of the directory you ran from), the bench crate version, the names of the providers in ~/.config/rewriter/config.toml, the host name, the SHA-256 of the rewriter-queue and ai-org-orchestrator binaries, and a timestamp. A value that could not be found is unknown; the binaries are hashed from the path you passed or from PATH, so a stamp of unknown usually means the binary was not where the benchmark looked. The stamp holds provider names only: it does not record model names or the inference server’s context settings, so note those in the --label when they matter.

How to read results

A task passes a run only if its hidden acceptance checks all pass. Look at failed_checks first: the names come from task.toml, so no-doubled-trailing-newline tells you what broke without opening anything. The failure detail (what was expected and what was got, escaped so stray newlines are visible) is printed to stderr while the run goes. To dig into a failure, re-run that task with --keep and read the workspace, or use the job_id with rewriter-queue events and analysis.

A pipeline_outcome of succeeded with passed: false is the case the suite is built to catch: the pipeline believed it was done and was wrong.

Compare runs by pass rate first, then by median time and tokens. A single run is a sample, not a measurement: a live model gives a different answer each time, so look at the pass rate over several repeats, not at one run.

rewriter-bench list shows the tasks (with iteration cap, time budget, tags and description) and the results files in bench/results/.

Comparing runs: compare

rewriter-bench compare <baseline> <current> [--pass-rate-drop X] [--time-ratio X]
    [--token-ratio X] [--min-runs N]

It reads two results files, prints a per-task table (pass rate, median time and median tokens, baseline to current, and a status), then lists any regressions with the job ids of both sides so you can read the analysis of the run that got worse. The exit status is 0 when there are no regressions, 1 when there are, and 2 for a usage error such as a missing argument.

The policy, in plain words

Each task is judged on its own, using only tasks that are in the baseline.

  • Pass rate. A regression when the pass rate falls by more than --pass-rate-drop (default 0.2, that is 20 percentage points). With three repeats the rate moves in steps of a third, so a single extra failure out of three (100% to 67%) is a regression.
  • Time and tokens. A regression when the current median exceeds the baseline median times the ratio (--time-ratio and --token-ratio, both default 1.5) and the rise is also larger than the baseline’s own spread. The spread is the median absolute deviation of the baseline’s runs, so a baseline that was itself noisy needs a bigger rise before it counts. A baseline of zero is never flagged.
  • --min-runs (default 3). A task with fewer runs than this on either side is shown as insufficient and never fails the comparison. A single run therefore can never fail a comparison; the benchmark is too noisy for that.
  • A task missing from the current run but present in the baseline is a regression, because silently dropping a task would hide a problem.
  • A task only in the current run is shown as new and is not judged.
  • Improvements (the mirror image of the rules above) are shown as improved and are informational only.

If the two runs used different provider names, the report opens with a model note saying results across different models are not directly comparable. It is a note, not a failure; it is up to you to decide whether the comparison still means anything.

Promoting a baseline and gating a release

A baseline is a results file kept in the repository:

rewriter-bench run --repeat 3 --label baseline-0.10.0
rewriter-bench promote bench/results/<timestamp>-baseline-0.10.0.json

promote validates the file and writes it to bench/baseline.json, which is committed (unlike bench/results/). To gate a release, run the suite on the release candidate and compare:

rewriter-bench run --repeat 3 --label rc1
rewriter-bench compare bench/baseline.json bench/results/<timestamp>-rc1.json

and fail the release on a non-zero exit. Run with the same --repeat as the baseline, on the same model and server settings. Promote a new baseline deliberately, after a change that is meant to shift the numbers, and say why in the commit.

Adding a task

Add a directory under bench/tasks/<name>/; the directory name must match the name in task.toml:

bench/tasks/<name>/
  task.toml            the specification (below)
  mission.md           what the pipeline is asked to do
  source/              the V1 input; a stub crate for a greenfield task
  acceptance/          hidden from the pipeline
    tests/*.rs         cargo integration tests, when cargo_tests = true
  solutions/
    good/              a complete crate that passes every check
    bad/               a complete crate that fails at least one check

rewriter-bench refuses to load a task that is missing source/, mission.md, acceptance/, or a Cargo.toml in each of solutions/good and solutions/bad, or that has neither [[check]] entries nor cargo_tests, or that has checks but no binary. The error names the task.

task.toml

FieldMeaningDefault
nameMust equal the directory namerequired
descriptionOne line, shown by listrequired
max_iterIteration cap passed to the pipeline10
time_budget_secsA run that takes longer is a failure, not a hang3600
tagsFree-form labels; by convention one difficulty: tagnone
binaryThe binary the finished crate must produce; required when there are [[check]] entriesnone
cargo_testsAlso run acceptance/tests/*.rs as cargo integration testsfalse
[[check]]One run of the binary and an exact comparison (below)none

A check has a name, and optionally args, stdin, expect_stdout, expect_stderr and expect_exit. Any expect_* you leave out is not checked. Comparison is exact, byte for byte, so expect_stdout = "+-+\n+-+\n" is what catches a doubled trailing newline.

name = "wc-lite"
description = "Greenfield stdin counter printing lines, words and bytes"
max_iter = 8
time_budget_secs = 1800
tags = ["greenfield", "cli", "stdin", "difficulty:easy"]
binary = "wc-lite"

[[check]]
name = "no-trailing-newline"
stdin = "a b c"
expect_stdout = "0 3 5\n"
expect_stderr = ""
expect_exit = 0

Writing a fair mission

The mission is the only thing the pipeline sees besides source/, so every assertion in the hidden checks must follow from it. If a check expects a behaviour the mission does not state, a pass or fail tells you nothing about the pipeline. In practice:

  • State the exact bytes: the output format, the terminating newline, what goes to stderr, the exit status for each error, with examples.
  • Define edge cases the checks exercise (empty input, no trailing newline, zero-size cases, bad arguments) and say what happens in each.
  • Name the package and binary, and say standard library only: acceptance builds offline (see Limitations).
  • Do not describe the checks or hint at the solution; the point is to see whether the pipeline gets there from the specification.

Hidden checks

Everything under acceptance/ and solutions/ is kept out of the pipeline’s way: the runner copies only source/ and mission.md into the run. To check a finished crate the runner copies it to a scratch directory, runs cargo build --offline, runs each [[check]] against target/debug/<binary> (10 seconds each), and, when cargo_tests is set, copies acceptance/tests/*.rs into the crate’s tests/ and runs them (build and test steps are capped at 5 minutes each). A build failure is a single failed check named build. A failing cargo test is reported under its own test name. Use cargo_tests for what exact output cannot express, such as the structure requirements of the stack-calc-refactor task (a short main.rs, at least three other source files).

Known-good and known-bad solutions

Every task ships a full crate that passes (solutions/good) and one that fails (solutions/bad). This proves the checks can tell the difference: a check that passes everything checks nothing. For example, rect’s bad solution draws the right rectangle with a doubled trailing newline. Run:

rewriter-bench validate-tasks [--tasks-dir D]

It prints ok or FAIL for each task and exits 1 if any task fails. A task fails validation when a solution directory is missing, when the good solution fails any check, when the bad solution passes every check, or when either solution ran no checks. Run it whenever you add or edit a task. Some tasks keep extra bad variants (for example rect/solutions/bad-noop, a program that does nothing); validate-tasks only runs good and bad, so the extras are for reference.

The tasks

TaskWhat it testsTime budget
hello-trivialA trivial greeting program. It should be done in one milestone and must not be over-split.600s
rectGreenfield ASCII rectangle program from a stub (fn main() {}). Regression for the doubled trailing newline and the program that did nothing. Eighteen checks cover corners in any order, degenerate shapes and every usage error.1800s
wc-liteGreenfield stdin counter (lines, words, bytes). Edge cases: empty input, no trailing newline, repeated whitespace, CRLF, multibyte text counted as bytes.1800s
contract-trapRefactor a crate whose doc comments and README describe stale behaviour. The mission says the true contract is the mission, not the comments, so a pipeline that follows the stale text fails.2400s
stack-calc-refactorRestructure a single-file RPN calculator into modules with identical behaviour, including every error message and exit code, plus structure checks on the layout.2700s

rewriter-bench list prints the current set with tags and iteration caps.

Cost and time

These are not known yet. No baseline has been recorded, so there is no measured time or token cost per task, and this page does not guess. What the code does tell you:

  • The default run is every task three times, fifteen pipeline runs.
  • The time budgets above add up to 9,300 seconds for one pass over all five tasks, so a default run is at most about 7.75 hours in the worst case where every run uses its whole budget. Typical runs should be well under that, but by how much is to be measured.
  • Tokens are recorded per run (input_tokens and output_tokens), so the first real run gives you the numbers.
  • Tier 1 costs nothing and takes seconds.

Until a baseline exists, use targeted runs (--tasks, --repeat 1) for development and keep full runs for release checks. Recording the first baseline on the inference box is the next step; when it exists, replace this section with the measured figures.

Limitations

  • Live models vary. The same task can pass and fail on consecutive runs. That is why compare uses medians and a spread, needs at least --min-runs runs per task, and never fails a comparison on a single run. A real regression can hide in a small number of repeats; raise --repeat when it matters.
  • Acceptance builds offline and uses the standard library only. Checks run cargo build --offline, so a crate that needs a dependency that is not already in the local cargo cache fails to build. Missions should say “standard library only”, and the tasks do.
  • Stamps are partial. The environment stamp has provider names, not models or server context settings, so two runs on the same provider with a different model look identical to the model note in compare.
  • Tasks are small Rust programs. The suite measures the pipeline on short, single-crate Rust tasks. It says little about large inputs or the merge path.
  • Queue mode needs infrastructure. A queue and a worker must be running and pointed at a model; the benchmark does not start them.