rewriter-queue
A resumable job queue for multi-agent LLM code synthesis. You submit a source tree and a mission; a small organization of agent roles (implementer, security/maintenance/production-readiness reviewers, a merge agent for large inputs) iterates on it — patching rather than rewriting where it can, checkpointing every real step so an interrupted run resumes instead of starting over — until the reviewers are satisfied or the iteration budget runs out.
Submit and watch jobs from a terminal, from Zed, or from Claude Code.
Not a hosted service — three small Rust binaries you run yourself, against whatever OpenAI-compatible inference endpoint you point them at.
Start with the tutorial. To see what a run did and why, see run observability. To check that a change to the pipeline did not make it slower or less effective, see benchmarks and regression tests.
Tutorial
This walks through the whole loop once, end to end: get a binary, point it at a model, submit something to rewrite, and watch it work. Fifteen minutes if you’ve already got an OpenAI-compatible inference server running somewhere; longer if you need to stand one up first (see Remote deployment for that part).
1. Get the binaries
Download the archive for your platform from the
latest release
and unpack it — see Installation if you’d rather build
from source. Either way you end up with two binaries sitting next to each
other: rewriter-queue and ai-org-orchestrator. They need to stay in the
same directory — the queue’s worker finds the orchestrator by looking beside
its own binary, not on PATH.
Put that directory on your PATH, or just remember the full path for the
commands below.
2. Tell it where your model lives
Create ~/.config/rewriter/config.toml:
[[providers]]
name = "local"
type = "openai-compat"
base_url = "http://localhost:8001/v1"
models = ["your-model-id"]
Point base_url at any OpenAI-compatible /v1/chat/completions endpoint —
a llama.cpp/vLLM server you’re running, or a hosted API. If you don’t
have one yet, that’s the whole subject of
Remote deployment.
3. Pick a queue mode
For trying this out on one machine, skip the server entirely — every
command below falls back to a local, file-backed queue when there’s no
[queue] section in the config. If you’re running the inference backend on
a separate box and want to submit jobs from your laptop, start the server
there instead and add a [queue] url = "..." pointing at it (see the root
README’s quickstart for the server command and token setup).
This tutorial uses local mode — nothing extra to start.
4. Submit something
Pick a small, real piece of code you want rewritten or reviewed — a single source file or a small directory is a better first run than a whole project. The workspace directory holds the tool’s working state as it goes, plus one thing you provide before submitting: the mission.
The mission is a required artifact — a short statement of what V2 should
actually do, not just “port this file.” Without it, the first agent in the
pipeline has to guess your intent from raw source alone, which shows up
later as a contract that’s technically correct but not what you meant. It’s
stored the same way as every other artifact this pipeline produces
(objective_contract.md, schema.rs, …), at <workspace>/artifacts/mission.md,
and the run refuses to start without one:
mkdir -p /tmp/my-first-run/artifacts
cat > /tmp/my-first-run/artifacts/mission.md <<'EOF'
# Mission
V1 is a small in-memory counter store. Rebuild it as V2 with identical
behavior: increment a key's count, and read a key's current count (0 for
an unseen key).
EOF
rewriter-queue submit \
--source /path/to/the/code \
--workspace /tmp/my-first-run \
--max-iter 10
That prints something like queued job #1 (0 ahead). In local mode with
nothing else queued, it starts immediately.
5. Watch it work
rewriter-queue watch
A live view that refreshes once a second: which job is running, what stage
it’s on (ObjectiveContract → test matrix → inductive analysis → schema →
V2 synthesis), and how long it’s been going. Ctrl-C to stop watching —
the job keeps running either way.
Want more detail on one job specifically?
rewriter-queue status 1
Shows the last 20 lines of its log, including which agent is running and the token counts for each call.
6. See what it’s produced, before it’s even done
You don’t have to wait for the job to finish to look at what it’s built so far:
rewriter-queue artifacts 1 # list what's been emitted
rewriter-queue artifact 1 objective_contract.md # read one of them
objective_contract.md is usually the first thing worth reading — it’s the
agreed-on shape of what’s being built (public API, invariants, edge cases,
explicit exclusions) before any code gets written.
7. Get the result
In local mode there’s nothing to download — the workspace directory you
passed to submit is where everything lands, no copy needed:
ls /tmp/my-first-run/artifacts/v2/ # the actual synthesized code
(rewriter-queue download 1 --out ./result is for the remote-server case —
see Remote deployment — where the job ran on a
different machine and you need its artifacts/ tree copied to yours in one
call instead of one file at a time.)
8. If it gets interrupted
Kill rewriter-queue serve (or just Ctrl-C a local-mode run) mid-job and
resubmit isn’t what you want — restart the server (or just try watch
again in local mode) and the same job picks back up from wherever it had
already checkpointed, not from the beginning. This is true down to the
level of individual agent calls, not just whole pipeline stages — see the
root README’s design notes if you’re curious how.
9. If the mission is too big
Every part of the build carries the whole schema (the Rust types and function signatures
drafted for the mission), and crate files over 500 lines fail the build gate. So when the
schema comes to more than 500 lines, the run stops before building anything and has a
planner split the mission into 2 to 6 increments: the smallest usable core first, then the
pieces that build on it. It writes them to artifacts/increments.json and finishes
succeeded; rewriter-queue status shows a note: naming the new job.
The queue submits increment 1 as an ordinary job in a fresh workspace, with that
increment’s own mission. When an increment succeeds, the queue submits the next one, with
the crate the previous increment built added to its source as v2-so-far/. Each increment’s
job name reads <name> · increment 2/4: <slug>. An increment whose own schema is still over
budget fails instead of splitting again; give it a smaller mission by hand.
What’s next
- Installation — the binary download details, and building from source if you’d rather.
- Zed setup — submit and watch jobs from Zed’s agent panel instead of a terminal.
- The Claude Code plugin — the same queue tools inside Claude Code, plus each reviewer persona as a standalone slash command.
- Remote deployment — running the inference backend and the queue server on a separate (e.g. GPU) machine from the one you’re submitting jobs from.
Installation
Prebuilt binaries
Every release ships prebuilt archives for:
| Platform | Archive |
|---|---|
| Linux x86_64 | rewriter-queue-x86_64-unknown-linux-gnu.tar.gz |
| Linux aarch64 (ARM64) | rewriter-queue-aarch64-unknown-linux-gnu.tar.gz |
| macOS Intel | rewriter-queue-x86_64-apple-darwin.tar.gz |
| macOS Apple Silicon | rewriter-queue-aarch64-apple-darwin.tar.gz |
curl -LO https://github.com/AndyGauge/rewriter-queue/releases/latest/download/rewriter-queue-<your-target>.tar.gz
tar xzf rewriter-queue-<your-target>.tar.gz
Each archive contains two binaries — rewriter-queue and
ai-org-orchestrator — plus the license files. Keep both binaries in the
same directory. The queue’s worker finds the orchestrator by looking
beside its own executable path, not on PATH, so unpacking them apart from
each other breaks job execution.
Put that directory on your PATH, or reference rewriter-queue’s full
path directly (e.g. in Zed’s context_servers config or the Claude Code
plugin’s MCP config).
No Windows build yet — job-queue’s worker uses Unix process-group APIs
(libc::killpg) to reliably kill a job’s whole process tree on cancel or
timeout, which doesn’t have a direct Windows equivalent as written. Runs
fine under WSL in the meantime.
Building from source
Needs a recent stable Rust toolchain (rustup.rs).
git clone https://github.com/AndyGauge/rewriter-queue
cd rewriter-queue
cargo build --release
Binaries land in target/release/rewriter-queue and
target/release/ai-org-orchestrator — same “keep them together” rule
applies.
The lint plugins (optional)
ai-org-orchestrator’s quality gate can run ast-grep
rules from ai-org-orchestrator/lints/rules/ as a non-blocking check
alongside cargo build/clippy/test. It degrades gracefully if the
binary isn’t installed — this is opt-in, not a hard dependency:
cargo install ast-grep --locked
The Zed extension and Claude Code plugin
Not part of the release archive — see Zed setup and
claude-code-plugin/README.md
for how to install each.
Zed setup
Two independent pieces: the context server (lets Zed’s agent panel submit and watch queue jobs) and the model provider (lets Zed chat with whatever backend is serving your models). You can set up either without the other.
1. The context server (zed-extension/)
This is a real Zed extension — zed-extension/extension.toml +
zed-extension/src/lib.rs, compiled to WASM — not a hand-edited
settings.json block. It registers rewriter-queue mcp as a context
server and defaults its arguments to ["mcp"], so you don’t have to
remember to add that yourself.
Build and install it as a dev extension:
rustup target add wasm32-wasip2
cargo build -p job-queue --release # the rewriter-queue binary itself
In Zed: open the command palette and run zed: install dev extension,
then point it at the zed-extension/ directory. Zed builds and loads the
WASM extension itself — you don’t need to invoke the WASM build by hand.
Tell it where the rewriter-queue binary is, if it’s not on your
PATH, in settings.json:
{
"context_servers": {
"rewriter-queue": {
"command": {
"path": "/absolute/path/to/rewriter-queue/target/release/rewriter-queue"
}
}
}
}
Leave out command entirely if rewriter-queue is already on PATH
(e.g. cargo install --path job-queue).
Point it at a queue server via ~/.config/rewriter/config.toml:
[queue]
url = "http://192.0.2.10:8003" # wherever `rewriter-queue serve` is running
…and the token it expects, in ~/.config/rewriter/queue-token. Without a
[queue] section, every command (including the ones the context server
runs on your behalf) falls back to a local, file-backed queue — fine for
trying this out on one machine, but you’ll usually want a real server if
you’re also running the inference backend on a GPU box (see
remote-deployment.md).
Once it’s connected (check the context-servers panel for a green status,
not a red error dot), just ask Zed’s agent about your jobs in plain
language — “what jobs are in the queue?”, “submit a synthesis run for
./src with a 20-iteration budget”, “download job 3’s artifacts to
~/Desktop” — and it calls the right tool
(queue_submit/queue_list/queue_status/queue_artifacts/
queue_artifact/queue_download/queue_fetch/queue_cancel) itself.
There’s no slash command for this — MCP tools are things the model calls on
its own, not something you invoke directly, unless a server also exposes
MCP prompts (this one doesn’t).
A live terminal view is also available without going through Zed’s agent
at all: .zed/tasks.json in this repo defines a “Queue: watch” task
that runs rewriter-queue watch.
2. The model provider
This is unrelated to the extension above — it’s Zed’s own
openai_compatible provider type, pointed at whatever OpenAI-compatible
server (llama.cpp, vLLM, etc.) you’re running your models on. Add it to
settings.json:
{
"language_models": {
"openai_compatible": {
"my-backend": {
"api_url": "http://192.0.2.10:8001/v1",
"available_models": [
{
"name": "your-model-id",
"display_name": "My Model",
"max_tokens": 128000,
"capabilities": {
"tools": true,
"images": false,
"parallel_tool_calls": false,
"prompt_cache_key": false,
"chat_completions": true
}
}
]
}
}
}
}
"tools": true matters if you want Zed’s agent to actually call tools
through this model — verified working with both llama.cpp’s --jinja
mode and vLLM’s --enable-auto-tool-choice, but not every locally-served
model reliably emits well-formed tool calls. If the agent starts guessing
at shell commands instead of calling a tool by name, that’s usually the
model’s tool-calling support, not a Zed or context-server bug — sanity-check
by sending the same tool schema directly to the model’s /v1/chat/completions
endpoint and see what comes back before assuming the wiring is broken.
Running two backends side by side (e.g. a general model and a
coding-specialized one) on the same box just means two openai_compatible
entries pointed at two different ports — see
remote-deployment.md for sizing multiple backends
into one machine’s memory.
Running this on a remote target
The queue server and the inference backend don’t have to be the same machine, and usually shouldn’t be — the inference backend wants a GPU, the queue server doesn’t need one at all. This covers running both on a remote Linux box (a rented GPU instance, a home server, whatever) and reaching it from your laptop.
The inference backend
Anything that speaks the OpenAI-compatible /v1/chat/completions API
works — llama.cpp’s llama-server, vLLM, or a hosted API you don’t run
yourself. If you’re self-hosting on a GPU box:
llama-server \
-m /path/to/model.gguf \
-a your-model-id \
-ngl 999 \
-c 65536 \
-fa auto \
--jinja \
--host 0.0.0.0 --port 8001
Running more than one model on the same box is fine as long as the
memory budget actually adds up — size it explicitly rather than finding out
at OOM time. Two independent servers on two ports (e.g. 8001/8002) is
simpler and safer than trying to multiplex one server across models. Watch
in particular:
- KV cache scales with context, not just model size. For a dense
transformer:
2 (K and V) × layers × kv_heads × head_dim × context × bytes_per_element. A model with a Mixture-of-Experts or hybrid linear-attention architecture can have a very different (often much smaller) real footprint than its parameter count suggests — check the model’s ownconfig.json, don’t assume. -ctk q8_0 -ctv q8_0(with-fa on) roughly halves KV cache size at a near-lossless quality cost — the quantization error is small and fixed per cached token, not cumulative with age, so it mostly costs you long-range recall precision in very long contexts, not general quality.q4_0is meaningfully lossier; reach forq8_0first.- Leave real headroom, not just enough to fit at idle — concurrent requests, the OS, and anything else on the box all need memory too.
The queue server
Runs anywhere that can reach the inference backend(s) over HTTP — doesn’t need to be the GPU box, and arguably shouldn’t be if you want to restart the inference backend independently of the queue.
rewriter-queue serve --bind 0.0.0.0:8003
It needs a bearer token (~/.config/rewriter/queue-token or
REWRITER_QUEUE_TOKEN) and providers configured in
~/.config/rewriter/config.toml:
[[providers]]
name = "backend-a"
type = "openai-compat"
base_url = "http://<inference-box>:8001/v1"
models = ["your-model-id"]
[queue]
url = "http://<this-box>:8003"
A PATH gotcha that will bite you
quality_gate() shells out to cargo build/clippy/test (and, if
installed, ast-grep) from inside the orchestrator subprocess the queue
worker spawns. That subprocess inherits whatever PATH the queue server’s
own process happened to have — which depends entirely on how you started
it. A plain ssh host 'rewriter-queue serve ...' gets a minimal, non-login
shell PATH on most systems (skips .bashrc/.profile, so ~/.cargo/bin
usually isn’t on it) even though an interactive SSH session would have it.
If cargo isn’t reliably found, every quality-gate check silently fails with
“cargo build not available” — treated as a normal retryable error, so it
doesn’t crash anything, it just quietly never actually validates code
until the iteration budget runs out and broken output gets accepted anyway.
job-queue/src/worker.rs already guards against this — it explicitly sets
PATH (prepending $HOME/.cargo/bin) on the orchestrator process it
spawns, rather than trusting whatever it inherited. If you install cargo
somewhere else, or need other tools (like ast-grep) on a non-standard
PATH, that’s the place to adjust it.
Keeping it running
Neither binary is a managed service by default — write a systemd unit if you want one to survive a reboot or restart on crash. Two examples:
# /etc/systemd/system/rewriter-queue.service
[Unit]
Description=rewriter-queue server
After=network.target
[Service]
Type=simple
User=youruser
WorkingDirectory=/home/youruser/rewriter-queue
ExecStart=/home/youruser/rewriter-queue/target/release/rewriter-queue serve --bind 0.0.0.0:8003
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.target
# /etc/systemd/system/llama-server.service
[Unit]
Description=llama.cpp inference server
After=network.target
[Service]
Type=simple
User=youruser
ExecStart=/home/youruser/llama.cpp/build/bin/llama-server \
-m /home/youruser/models/your-model.gguf -a your-model-id \
-ngl 999 -c 65536 -fa auto --jinja --host 0.0.0.0 --port 8001
Restart=on-failure
RestartSec=10
[Install]
WantedBy=multi-user.target
sudo systemctl daemon-reload
sudo systemctl enable --now rewriter-queue llama-server
If you’d rather not deal with sudo/systemd at all, tmux (or screen)
is a fine substitute for a personal box: start each in its own session so
an SSH disconnect doesn’t kill it, and accept that a reboot needs a manual
restart.
The lint plugins need ast-grep
ai-org-orchestrator/lints/ (see the root README) is optional — the
orchestrator degrades gracefully if the binary isn’t found — but if you
want it: cargo install ast-grep --locked on whichever machine actually
runs the orchestrator subprocess (the queue server’s box, per the PATH
note above).
Security posture — decide this deliberately
None of the above puts a token or TLS in front of the inference backend
itself, and the queue server’s Bearer token is the only thing gating job
submission (no per-job auth, no rate limiting). That’s a reasonable
default for a private LAN or a locked-down VPC where you trust everything
that can reach the port — it is not a default you want facing the public
internet. If you’re putting either behind a real network boundary (a cloud
security group, a home router’s port forward, anything multi-tenant), add
a reverse proxy with TLS and real auth in front, and give llama-server
its own --api-key rather than relying on network trust alone.
Run observability
Every agent in a job reports what it did to one place, so a running job can be watched in
detail, a finished or failed job can be analysed without reading logs over SSH, and recurring
problems turn into feature requests. This is a structured event stream added on top of the
workspace files (agents/<name>/..., deviations.md), which are still written as before.
Reporting is best effort. A failure to record or send an event never fails or slows a job.
What gets recorded
Each event is one JSON object (one line of JSONL), defined in the run-events crate:
| Field | Meaning |
|---|---|
ts | RFC 3339 timestamp, millisecond precision |
job_id | The queue job id (absent when the orchestrator is run outside the queue) |
stage | The pipeline stage that was current when the event was emitted, if any |
agent | Which agent emitted it (orchestrator for stage events, QualityGate for gate runs) |
kind | One of the kinds below |
summary | A one-line description |
detail | Optional JSON object; keys depend on the kind |
| Kind | When it is emitted | detail keys |
|---|---|---|
stage_start | A pipeline stage begins (ObjectiveContract, Acceptance criteria, Test matrix refinement, Inductive analysis, Schema, V2 synthesis). Starting a stage ends the previous one. | none |
stage_end | A stage ends | none |
agent_call | One per plain model call, and one per agentic call (tokens and latency are summed over its turns; the summary says how many turns) | provider, model, input_tokens, output_tokens, latency_ms, ok |
decision | An agent logged a decision | text (full text, capped at 4000 characters) |
deviation | A deviation from plan was logged | text |
finding | An agent called report_finding | severity (info, warn, error), category, evidence |
gate_result | A quality-gate run (build, clippy, test, fmt) | passed, error_bytes, attempt, variant for parallel milestone build directories, and integration: true on runs of the final whole-crate gate |
retry | A retry | text |
error | An error, including a panic in the orchestrator (a panic hook records it) | text |
attempt counts consecutive gate runs against the same build directory since it last passed.
A failed model call is an agent_call with ok: false.
A stage with a stage_start and no stage_end means the process died in it. Stages that were
resumed from a checkpoint emit no stage events.
Where events are stored
- In the job’s workspace:
<workspace>/events.jsonl. The orchestrator always appends here, whether or not a queue server is involved. - On the queue server:
<queue data>/events/<id>.jsonl, with<id>zero-padded to four digits like the job files. The orchestrator posts each event toPOST /jobs/<id>/events. - Analyses:
<queue data>/analysis/<id>.json, written by the worker after a job ends. - Feature requests:
<queue data>/feature-requests.json, one list across all jobs.
<queue data> is $REWRITER_QUEUE_DIR (default ~/.local/share/rewriter/queue).
When you read a job’s events, the server’s copy is used if it exists and the workspace file otherwise, so a local queue (no server) works the same way.
Environment variables
The worker sets these when it launches the orchestrator:
| Variable | Set when |
|---|---|
REWRITER_JOB_ID | Always |
REWRITER_QUEUE_URL | Only when the worker is hosted by rewriter-queue serve (a loopback URL for the bind address, so 0.0.0.0:8003 becomes http://127.0.0.1:8003) |
REWRITER_QUEUE_TOKEN | Only when the worker is hosted by serve |
A standalone worker removes any REWRITER_QUEUE_URL and REWRITER_QUEUE_TOKEN it inherited, so
the orchestrator never posts anywhere unexpected. The orchestrator posts only when both the URL
and token are present and non-empty.
CLI
All four commands work against a remote server or a local queue. Add --json to events,
analysis and requests for machine-readable output.
rewriter-queue events <id> [--kind <kind>] [--agent <name>] [--since <timestamp>] [--json]
Works while the job is running. One line per event, oldest first, in the form
<ts> <kind> [<stage>] <agent> <summary>:
2026-10-02T14:03:11.204Z stage_start [Schema] orchestrator Schema
2026-10-02T14:03:40.918Z agent_call [Schema] SchemaArchitect SchemaArchitect: 5120in/1873out in 29714ms
2026-10-02T14:05:02.377Z gate_result [V2 synthesis] QualityGate quality gate failed (attempt 1, 2048 bytes of errors)
(These lines are illustrative; the shape comes from the rendering code.) --kind takes the
snake_case kinds above, and --since returns only events strictly after that RFC 3339 timestamp.
With no events it prints (no events). --json prints the raw JSONL instead.
rewriter-queue analysis <id> [--json]
job #12 failed
<summary>
Root causes:
- <cause> (evidence: <agent@ts>; ...)
Feature requests:
- <id> (<title>)
The root-causes and feature-requests sections are omitted when empty. If the job has no analysis
yet the command errors with no analysis for job <id>.
rewriter-queue requests [--json]
<id> [<status> / <priority>] <title>
jobs: #12, #15
<rationale>
With none it prints (no feature requests).
rewriter-queue request <id> <open|accepted|done|rejected>
Sets a request’s status and prints it in the same format as requests. An unknown request id is
an error.
Server routes behind these (same bearer-token auth as the rest): POST /jobs/<id>/events (one
event object or a JSON array; the job must exist), GET /jobs/<id>/events (query kind,
agent, since; JSONL), GET /jobs/<id>/analysis, GET /feature-requests,
POST /feature-requests/<id>/status (body {"status": "accepted"}).
MCP tools
rewriter-queue mcp exposes three new tools alongside the existing ones:
job_events:id(required), optionalkind,agent,since. Returns the events as JSONL, oldest first. Works on a running job.job_analysis:id. Returns the analysis as JSON; errors if the job has none yet.feature_requests: no arguments. Returns every request as JSON.
There is no MCP tool for changing a request’s status; use the CLI.
Post-run analysis
When a job ends, whatever the outcome, the worker runs
ai-org-orchestrator analyze --job-id <id> --events <file> --out <file> over that job’s
events (the server’s copy if present, else the workspace file). If neither exists the analysis
is skipped. It has a 15 minute timeout, and any failure is one [worker] analysis ... line in
the job’s log; the job’s outcome is never affected.
The analysis has two parts:
- Mechanical facts, computed in Rust: model calls, failed calls, tokens and latency per
agent; retries per stage; gate runs and each failed gate run (stage, attempt, error bytes);
stages that never ended; and an inferred outcome. The outcome is
failedif the last event is anerror, a stage never ended, or the laststage_endhasdetail.ok == false;cancelledif the last event’s summary mentions “cancel”;failedfor an empty event file; otherwisesucceeded. - A
PostRunAnalystagent is given those facts plus the findings, deviations and errors, and proposes a summary, root causes and feature requests.
If the model call fails or its reply is not valid JSON, the analysis is still written with the mechanical summary and no root causes or feature requests.
How claims are validated
The analyst must cite evidence for every root cause and feature request, and the code checks
each citation against the real events. A citation is accepted if it is either an agent@ts
reference matching an event’s agent and timestamp exactly, or a quote of at least 8 characters
that appears (case-insensitively) in some event’s summary or in a string in its detail.
Citations that match nothing are dropped. A root cause or request left with no valid citation
is dropped entirely. Surviving root causes are stored as one string,
<text> (evidence: <cites>).
Feature requests
A feature request has an id, title, rationale, evidence, source_job_ids, priority
(low, medium, high) and status.
Deduplication. The id is a kebab-case slug of the title (lowercase alphanumerics joined by
single hyphens), so the same title in two jobs gets the same id. When the worker merges an
analysis into feature-requests.json:
- a request with a new
idis added, always asopen, whatever the analyst said; - a request whose
idalready exists keeps its currentstatus,title,rationaleandpriority, and gains the new job insource_job_idsand any newevidenceentries.
Within one analysis, repeated ids are collapsed to the first.
Statuses. The tool stores and shows the statuses but attaches no behaviour to them:
open: newly raised, not yet looked at.accepted: you intend to build it.done: built.rejected: you decided against it. A later job that raises the same problem is merged into this request, which staysrejected.
Those meanings are the natural reading of the names; the code only stores the value you set.
The report_finding tool
Agents running agentically (with the tool box) can call report_finding to record something
they noticed: a missing or misleading tool or skill, malformed input, a pattern that wasted
effort, a risk in the source. It does not change the agent’s task or answer.
| Argument | Required | Notes |
|---|---|---|
severity | yes | info, warn or error |
category | yes | Short kebab-case label (lowercase letters, digits and hyphens, at most 60 characters), for example missing-tool |
text | yes | What was noticed; becomes the event summary (capped at 4000 characters) |
evidence | no | A quote or file/line reference |
Invalid arguments are returned to the agent as an error string so it can retry. The tool is
described in the reminder text appended to agentic calls. The result is a finding event.
When the queue server is unreachable
The job is unaffected. Each event is first appended to <workspace>/events.jsonl. The post to
the server has a 2 second timeout; on a failure the orchestrator logs one line to stderr, then
pauses posting for 30 seconds before trying again. Events emitted during a pause are not
re-sent later, so the server’s copy can have gaps (see Limitations). The workspace file is
complete as long as the workspace is writable; if it is not, one line is logged and events are
lost.
Limitations
- The final integration step has no stage events of its own, because it runs inside the
V2 synthesisstage. Escalations emitretryevents, and stages resumed from a checkpoint emit no stage events, so a stage’s measured duration can include a cached stage after it. analyzeskips the startup qualification probes, so its provider ratings are the defaults. That is deliberate (it makes one short call), but it means the analyst call is not routed by measured quality.- The feature-request file lock has only been exercised between threads in one process.
- Events dropped while the orchestrator is pausing posts after a failed POST are not re-sent to
the server; the workspace
events.jsonlstill has them, and analysis merges both copies.
Watching without being able to change anything
Give the server a second, read-only token and point the MCP server at it:
# on the machine running `rewriter-queue serve`
echo "some-long-random-string" > ~/.config/rewriter/queue-read-token
# restart `rewriter-queue serve`
# on the machine you watch from
REWRITER_QUEUE_URL=http://host:8003 REWRITER_QUEUE_READ_TOKEN=some-long-random-string \
rewriter-queue mcp --read-only
The read-only server offers queue_list, queue_status, queue_artifacts, queue_artifact,
queue_download, job_events, job_analysis and feature_requests. The permission is enforced
by the queue server, not just hidden in the client: the read-only token gets a 403 on anything
that submits, cancels or changes a request, and on downloading a job’s whole uploaded workspace.
Benchmarks and regression tests
The suite answers one question for every change to the pipeline, the prompts, the skills or the model configuration: did it get slower or less effective? A change that fixes the one problem it was written for must not quietly make five others worse.
It exists because the pipeline is not deterministic (it calls live models) and its failures are subtle. Two real ones shaped it:
- A job (#26) reported
succeededand produced a program that did nothing. The contract copied a stub V1 and the contract reviewer was never shown the mission, so nothing checked the contract against what was asked for. The pipeline’s own gates passed because an emptymainbuilds, lints and tests cleanly. This is why pass or fail is decided by a hidden acceptance check, never by the pipeline’s own verdict. - Another job (#24) produced a correct rectangle drawing program that failed its own tests over one newline: the output ended in a doubled trailing newline. Exact comparison of stdout, stderr and exit code, with escaped output in the failure message, is how the suite sees this.
Both are now tasks (rect covers both), and every future bug report should become one.
Two tiers
| Tier | What it is | Cost | When |
|---|---|---|---|
| 1. Structure tests | Scripted-provider tests that pin down how the pipeline is wired | Free, deterministic, seconds | Every change, as part of cargo test |
| 2. Live benchmark | Real tasks run through the real pipeline against a live model, then checked by hidden acceptance checks | Real model time and tokens | On demand, before a release, or nightly on the inference box |
Tier 1 catches “a refactor removed a behaviour”. Tier 2 catches “the behaviour is still there but the results got worse”.
Tier 1: structure tests
The tests live in ai-org-orchestrator/src/pipeline_structure/ and run with the rest of the
workspace:
cargo test
Each test drives the real Manager over a scripted provider that answers from canned text keyed on
the calling agent’s system prompt. It then asserts on which agents were called, in what order, what
was in their prompts, and on the emitted event stream. No model is involved. What they guard:
- Plan review. The plan reviewer runs before implementation and its corrected plan is the one implemented; an approval keeps the original plan; a review call that errors falls back to the original plan and the run continues; a nested plan’s review is given the plans above it.
- Debug and repair. The first failed repair attempt does not consult the debugger but later ones do, and the diagnosis reaches the repair prompt; a failing debugger call does not stop the repair loop; escalation to the milestone planner happens only after the repair budget is spent and carries the diagnosis.
- Contract review. The contract reviewer is shown the mission and the contract it reviews; a
rejected contract is revised with the reviewer’s reasons and reviewed again; the entry point
hands the mission to both the contract author and the reviewer. The last of these reads
main.rsas text, because the contract task is built inline there, so it fails if that wiring is edited away. - Observability. An unwritable events file, or an unreachable queue server, does not fail the run, and a healthy run records each model call as an event.
- Call budget. One easy milestone, taken from plan to production review, stays within a fixed number of model calls, and the events agree with the provider’s own call count.
Updating the call budget on purpose
The budget test uses MAX_CALLS_HAPPY_PATH in
ai-org-orchestrator/src/pipeline_structure/budget.rs. It is set one above what the pipeline uses
today, so adding an agent to a step on the happy path fails the test. If the extra call is
intended, raise the constant in the same change and say why in the commit message. The failure
message lists the roles that were called, so you can see which one is new.
Tier 2: running the live benchmark
The binary is rewriter-bench, from the bench crate:
cargo build --release -p bench
target/release/rewriter-bench <command>
The commands are run, compare, promote, validate-tasks and list. Run it from inside the
repository checkout: the environment stamp records the git commit of the current directory.
run
rewriter-bench run [--tasks a,b] [--repeat N] [--label L] [--direct]
[--queue-bin P] [--orchestrator-bin P] [--tasks-dir D] [--results-dir D] [--keep]
For each task and each repeat, it prepares a work directory holding the task’s source/ and
mission.md, runs the pipeline, reads the run’s events, then runs the task’s acceptance checks on
the finished crate and records the result.
| Flag | Meaning | Default |
|---|---|---|
--tasks a,b | Comma-separated task names. An unknown name is a usage error. | all tasks |
--repeat N | Runs per task. Must be at least 1. | 3 |
--label L | A name for the run, used in the results file name | run |
--direct | Run ai-org-orchestrator directly instead of going through the queue | off (queue mode) |
--queue-bin P | The rewriter-queue binary (queue mode) | found on PATH |
--orchestrator-bin P | The ai-org-orchestrator binary (--direct) | found on PATH; --direct fails if it is not found |
--tasks-dir D | Where tasks live | bench/tasks |
--results-dir D | Where results go | bench/results |
--keep | Keep each run’s work directory (workspace, output crate, orchestrator log in direct mode) | off, work directories are deleted |
The default is every task, three times each: five tasks makes fifteen pipeline runs. For a quick check, name the tasks:
rewriter-bench run --tasks rect --repeat 3 --label before-prompt-change
rewriter-bench run --tasks hello-trivial,wc-lite --repeat 1 --label smoke
rewriter-bench run --direct --orchestrator-bin target/release/ai-org-orchestrator --tasks rect
Queue mode submits each task with rewriter-queue submit, polls rewriter-queue status every
5 seconds, then reads the events with rewriter-queue events <id> --json (falling back to the
workspace events.jsonl). A job must be processed by something, so a worker (rewriter-queue worker, or a serve setup) has to be running; the benchmark only submits and waits. If the output
crate is not in the workspace it is fetched with rewriter-queue fetch. A job that is still going
when its time budget runs out is cancelled and recorded as a timeout. The budget starts counting
when the job leaves the queued state, so waiting behind other jobs does not use it up.
Direct mode runs the orchestrator as a child process with --workspace, --source and
--max-iter, and a clock of the task’s time budget. The pipeline outcome is succeeded if it
exits 0, failed otherwise, and timeout if it was killed. There is no job id.
run exits 0 whenever it completes, even if tasks failed. It prints a table of every run and the
path of the results file. A failing task is a result, not an error; use compare to decide.
What a results file contains
One run writes one pretty-printed JSON file, bench/results/<timestamp>-<label>.json (the label
is sanitised to letters, digits, -, _ and .). bench/results/ is gitignored. The file has a
label, an environment stamp and one record per run.
Each record:
| Field | Meaning |
|---|---|
task, repeat | Which task and which repeat (counted from 0) |
job_id | The queue job id; null in direct mode |
passed | The acceptance checks all passed and the run finished within its time budget |
failed_checks | Names of the failed checks, or setup, timeout or no-crate |
pipeline_outcome | succeeded, failed, cancelled or timeout, as the pipeline reported it. It never decides passed. |
wall_secs | Wall-clock time for the pipeline run |
input_tokens, output_tokens | Summed over every model call |
model_calls | Number of model calls, including failed ones |
retries | Number of retry events |
gate_failures | Quality-gate runs that failed |
integration_attempts | Runs of the final whole-crate gate (gate events marked integration) |
escalations | Retries recorded by the milestone planner (a milestone sent back for a finer split) |
plan_revised | The plan reviewer recorded a decision, meaning it changed the plan |
debugger_consulted | The debug implementer made at least one call |
started | When the run started |
The metrics are derived from the run’s events.jsonl (see run observability).
If no events could be read, they are zero.
The environment stamp records the git commit (short hash, of the directory you ran from), the
bench crate version, the names of the providers in ~/.config/rewriter/config.toml, the host
name, the SHA-256 of the rewriter-queue and ai-org-orchestrator binaries, and a timestamp. A
value that could not be found is unknown; the binaries are hashed from the path you passed or
from PATH, so a stamp of unknown usually means the binary was not where the benchmark looked.
The stamp holds provider names only: it does not record model names or the inference server’s
context settings, so note those in the --label when they matter.
How to read results
A task passes a run only if its hidden acceptance checks all pass. Look at failed_checks first:
the names come from task.toml, so no-doubled-trailing-newline tells you what broke without
opening anything. The failure detail (what was expected and what was got, escaped so stray newlines
are visible) is printed to stderr while the run goes. To dig into a failure, re-run that task with
--keep and read the workspace, or use the job_id with rewriter-queue events and analysis.
A pipeline_outcome of succeeded with passed: false is the case the suite is built to catch:
the pipeline believed it was done and was wrong.
Compare runs by pass rate first, then by median time and tokens. A single run is a sample, not a measurement: a live model gives a different answer each time, so look at the pass rate over several repeats, not at one run.
rewriter-bench list shows the tasks (with iteration cap, time budget, tags and description) and
the results files in bench/results/.
Comparing runs: compare
rewriter-bench compare <baseline> <current> [--pass-rate-drop X] [--time-ratio X]
[--token-ratio X] [--min-runs N]
It reads two results files, prints a per-task table (pass rate, median time and median tokens, baseline to current, and a status), then lists any regressions with the job ids of both sides so you can read the analysis of the run that got worse. The exit status is 0 when there are no regressions, 1 when there are, and 2 for a usage error such as a missing argument.
The policy, in plain words
Each task is judged on its own, using only tasks that are in the baseline.
- Pass rate. A regression when the pass rate falls by more than
--pass-rate-drop(default 0.2, that is 20 percentage points). With three repeats the rate moves in steps of a third, so a single extra failure out of three (100% to 67%) is a regression. - Time and tokens. A regression when the current median exceeds the baseline median times the
ratio (
--time-ratioand--token-ratio, both default 1.5) and the rise is also larger than the baseline’s own spread. The spread is the median absolute deviation of the baseline’s runs, so a baseline that was itself noisy needs a bigger rise before it counts. A baseline of zero is never flagged. --min-runs(default 3). A task with fewer runs than this on either side is shown asinsufficientand never fails the comparison. A single run therefore can never fail a comparison; the benchmark is too noisy for that.- A task missing from the current run but present in the baseline is a regression, because silently dropping a task would hide a problem.
- A task only in the current run is shown as
newand is not judged. - Improvements (the mirror image of the rules above) are shown as
improvedand are informational only.
If the two runs used different provider names, the report opens with a model note saying results across different models are not directly comparable. It is a note, not a failure; it is up to you to decide whether the comparison still means anything.
Promoting a baseline and gating a release
A baseline is a results file kept in the repository:
rewriter-bench run --repeat 3 --label baseline-0.10.0
rewriter-bench promote bench/results/<timestamp>-baseline-0.10.0.json
promote validates the file and writes it to bench/baseline.json, which is committed (unlike
bench/results/). To gate a release, run the suite on the release candidate and compare:
rewriter-bench run --repeat 3 --label rc1
rewriter-bench compare bench/baseline.json bench/results/<timestamp>-rc1.json
and fail the release on a non-zero exit. Run with the same --repeat as the baseline, on the same
model and server settings. Promote a new baseline deliberately, after a change that is meant to
shift the numbers, and say why in the commit.
Adding a task
Add a directory under bench/tasks/<name>/; the directory name must match the name in
task.toml:
bench/tasks/<name>/
task.toml the specification (below)
mission.md what the pipeline is asked to do
source/ the V1 input; a stub crate for a greenfield task
acceptance/ hidden from the pipeline
tests/*.rs cargo integration tests, when cargo_tests = true
solutions/
good/ a complete crate that passes every check
bad/ a complete crate that fails at least one check
rewriter-bench refuses to load a task that is missing source/, mission.md, acceptance/, or
a Cargo.toml in each of solutions/good and solutions/bad, or that has neither [[check]]
entries nor cargo_tests, or that has checks but no binary. The error names the task.
task.toml
| Field | Meaning | Default |
|---|---|---|
name | Must equal the directory name | required |
description | One line, shown by list | required |
max_iter | Iteration cap passed to the pipeline | 10 |
time_budget_secs | A run that takes longer is a failure, not a hang | 3600 |
tags | Free-form labels; by convention one difficulty: tag | none |
binary | The binary the finished crate must produce; required when there are [[check]] entries | none |
cargo_tests | Also run acceptance/tests/*.rs as cargo integration tests | false |
[[check]] | One run of the binary and an exact comparison (below) | none |
A check has a name, and optionally args, stdin, expect_stdout, expect_stderr and
expect_exit. Any expect_* you leave out is not checked. Comparison is exact, byte for byte, so
expect_stdout = "+-+\n+-+\n" is what catches a doubled trailing newline.
name = "wc-lite"
description = "Greenfield stdin counter printing lines, words and bytes"
max_iter = 8
time_budget_secs = 1800
tags = ["greenfield", "cli", "stdin", "difficulty:easy"]
binary = "wc-lite"
[[check]]
name = "no-trailing-newline"
stdin = "a b c"
expect_stdout = "0 3 5\n"
expect_stderr = ""
expect_exit = 0
Writing a fair mission
The mission is the only thing the pipeline sees besides source/, so every assertion in the hidden
checks must follow from it. If a check expects a behaviour the mission does not state, a pass or
fail tells you nothing about the pipeline. In practice:
- State the exact bytes: the output format, the terminating newline, what goes to stderr, the exit status for each error, with examples.
- Define edge cases the checks exercise (empty input, no trailing newline, zero-size cases, bad arguments) and say what happens in each.
- Name the package and binary, and say standard library only: acceptance builds offline (see Limitations).
- Do not describe the checks or hint at the solution; the point is to see whether the pipeline gets there from the specification.
Hidden checks
Everything under acceptance/ and solutions/ is kept out of the pipeline’s way: the runner
copies only source/ and mission.md into the run. To check a finished crate the runner copies it
to a scratch directory, runs cargo build --offline, runs each [[check]] against
target/debug/<binary> (10 seconds each), and, when cargo_tests is set, copies
acceptance/tests/*.rs into the crate’s tests/ and runs them (build and test steps are capped at
5 minutes each). A build failure is a single failed check named build. A failing cargo test is
reported under its own test name. Use cargo_tests for what exact output cannot express, such as
the structure requirements of the stack-calc-refactor task (a short main.rs, at least three
other source files).
Known-good and known-bad solutions
Every task ships a full crate that passes (solutions/good) and one that fails
(solutions/bad). This proves the checks can tell the difference: a check that passes everything
checks nothing. For example, rect’s bad solution draws the right rectangle with a doubled
trailing newline. Run:
rewriter-bench validate-tasks [--tasks-dir D]
It prints ok or FAIL for each task and exits 1 if any task fails. A task fails validation when
a solution directory is missing, when the good solution fails any check, when the bad solution
passes every check, or when either solution ran no checks. Run it whenever you add or edit a
task. Some tasks keep extra bad variants (for example rect/solutions/bad-noop, a program that
does nothing); validate-tasks only runs good and bad, so the extras are for reference.
The tasks
| Task | What it tests | Time budget |
|---|---|---|
hello-trivial | A trivial greeting program. It should be done in one milestone and must not be over-split. | 600s |
rect | Greenfield ASCII rectangle program from a stub (fn main() {}). Regression for the doubled trailing newline and the program that did nothing. Eighteen checks cover corners in any order, degenerate shapes and every usage error. | 1800s |
wc-lite | Greenfield stdin counter (lines, words, bytes). Edge cases: empty input, no trailing newline, repeated whitespace, CRLF, multibyte text counted as bytes. | 1800s |
contract-trap | Refactor a crate whose doc comments and README describe stale behaviour. The mission says the true contract is the mission, not the comments, so a pipeline that follows the stale text fails. | 2400s |
stack-calc-refactor | Restructure a single-file RPN calculator into modules with identical behaviour, including every error message and exit code, plus structure checks on the layout. | 2700s |
rewriter-bench list prints the current set with tags and iteration caps.
Cost and time
These are not known yet. No baseline has been recorded, so there is no measured time or token cost per task, and this page does not guess. What the code does tell you:
- The default run is every task three times, fifteen pipeline runs.
- The time budgets above add up to 9,300 seconds for one pass over all five tasks, so a default run is at most about 7.75 hours in the worst case where every run uses its whole budget. Typical runs should be well under that, but by how much is to be measured.
- Tokens are recorded per run (
input_tokensandoutput_tokens), so the first real run gives you the numbers. - Tier 1 costs nothing and takes seconds.
Until a baseline exists, use targeted runs (--tasks, --repeat 1) for development and keep full
runs for release checks. Recording the first baseline on the inference box is the next step; when it
exists, replace this section with the measured figures.
Limitations
- Live models vary. The same task can pass and fail on consecutive runs. That is why
compareuses medians and a spread, needs at least--min-runsruns per task, and never fails a comparison on a single run. A real regression can hide in a small number of repeats; raise--repeatwhen it matters. - Acceptance builds offline and uses the standard library only. Checks run
cargo build --offline, so a crate that needs a dependency that is not already in the local cargo cache fails to build. Missions should say “standard library only”, and the tasks do. - Stamps are partial. The environment stamp has provider names, not models or server context
settings, so two runs on the same provider with a different model look identical to the model
note in
compare. - Tasks are small Rust programs. The suite measures the pipeline on short, single-crate Rust tasks. It says little about large inputs or the merge path.
- Queue mode needs infrastructure. A queue and a worker must be running and pointed at a model; the benchmark does not start them.