plow is open source: a compiler that emits packet streams

The thing we kept not explaining

We’ve published four posts about plow — cold start, long-context decode on MI350X, Blackwell decode, Kimi-K3 day-zero bring-up — and every one of them assumed you already knew what a packet was, or a wave-class segment, or why a counter would be more interesting than a stream.

The code is now Apache-2.0 on GitHub — the compiler, the runtime, the CUDA and HSA interpreters, and the Lean proofs. Along with it ships a model bring-up agent harness: seven gated stages, one LLM-agent prompt each, which is how we take an architecture from a config.json to a served deployment.

So this is the post that backfills the rest: what the architecture actually is, that it works on both vendors and scales across GPUs and across models, what it buys, and what it costs. It carries no benchmark tables on purpose — those belong with their methodology, in the performance posts and in the repo.

---

The dispatch tax

Decoding one token from a 12B model is not a compute problem. At batch 1 you read every weight once, do a trivial amount of arithmetic per byte, and write a single row. The GPU is bandwidth-starved by construction — the arithmetic is nearly free.

So what fills the time? Everything that isn’t arithmetic. A graph executor walks an operator list on the CPU. For each operator it looks up a kernel, sets up arguments, launches, and — because the next operator consumes the last one’s output — synchronizes. A Gemma-4-12B decode step is a few hundred operators. Multiply by launch overhead and per-op synchronization and you get a fixed cost per token that has nothing to do with the model.

CUDA graphs exist to attack exactly this, and they work — which is why vLLM captures them. But a captured graph is still a graph of separate kernels with separate launches; you’ve amortized the CPU-side replay cost, not removed the per-op boundary. And you pay for the capture on every fresh process, which is the cold-start tax we wrote about separately.

That fixed, model-independent cost per token is what this architecture exists to attack.

---

Compile the model, don’t interpret the graph

plow splits the problem in two, and the split is the whole idea.

plowc is an ahead-of-time compiler. It takes a Hugging Face checkpoint and lowers it: operator graph → e-graph rewriting → tile dependency graph → interval-conflict scheduler → a packet stream. A packet is a fixed-size instruction describing one unit of device work — which operator body, which tiles, which compute units, which counters to wait on and which to signal.

The output is a static artifact. Not a plan that gets re-derived at request time; a .pkt file you build once, ship, and mmap. Every scheduling decision, tile assignment, and CU placement is already made and already on disk.

This is why boot is fast — there is no compile on the critical path — and it’s also why the runtime can be so small.

---

One kernel that never exits

plowrt launches one cooperative kernel and leaves it resident. It does not exit between operators. It does not exit between tokens.

Inside, the persistent interpreter pulls packets from a queue and executes their bodies at warp granularity — a warp on NVIDIA, a wave on AMD, which is where the name comes from: a Packet Language for On-device Workers, where a worker is one or the other. Ordering is not enforced by returning to the host. It’s enforced by counters in device memory: a packet waits on a counter reaching a threshold, does its work, and increments the counters its consumers are waiting on.

The consequences follow directly:

• No per-op launch. Operators are packets, not kernels.
• No host round-trip for ordering. Producer/consumer edges are counter waits.
• Dependencies are as fine as the data. Because the compiler knows the tile graph, a consumer can wait on the specific producers it needs rather than a whole-device barrier.

That last one is the part that surprises people, and we have the counter-example to prove it: fusing the q/k/v projections into one packet — strictly fewer packets, strictly fewer gates — measured slower. Fusing coarsened a fine per-head producer map into a whole-device wait, and the gates it deleted had cost nothing, because the packets it merged were already starting within microseconds of each other on disjoint compute units.

The win isn’t “fewer kernels.” It’s that a static schedule can express dependencies a dynamic dispatcher cannot afford to compute.

---

The same architecture, two vendors, eight GPUs

An execution model is only interesting if it survives contact with hardware it wasn’t designed on. This one has been ported four times.

The four shipping interpreters — one compiler, one packet ABI, one counter protocol behind all of them:

ISA	Vendor	Parts	Interpreter
sm_90a (Hopper)	NVIDIA	H100, GH200	nvidia/interp_sm90a.cu
sm_120 (Blackwell)	NVIDIA	RTX 5090, RTX PRO 6000	nvidia/interp_sm120.cu
gfx942 (CDNA3)	AMD	MI300X	amd/interp.hip
gfx950 (CDNA4)	AMD	MI350X, MI355X	amd/interp.hip — same file

What is not duplicated is the point. The compiler, the tile graph, the scheduler, the packet ABI, the counter protocol, and the cost model are one implementation. Only the interpreter body and the instruction primitives are per-vendor. On the AMD side even that is shared: CDNA3 and CDNA4 run the same runtime/amd/*.h, with amd_arch.h owning the instruction differences, hwspec::IsaLevel::geometry owning budget and shape, and a drift-guard test that fails the build when the device header and the host geometry disagree. That was a decision, not an accident, and the chapter records the threshold at which it should be reopened.

Warp granularity is what makes this portable: NVIDIA’s 32-lane warp and AMD’s 64-lane wave are the same abstraction to a packet body, which is why the compiler emits for a worker and never has to care which one it lands on.

It scales across GPUs

Tensor parallelism is wired and is the default strategy. The compiler runs a pre-pass that picks the strategy, partitions the tile graph across units, and adds cross-unit counter scopes — so the same counter protocol that orders two packets on one GPU orders two packets on different GPUs, over NVLink or xGMI. This is routinely exercised up to eight GPUs: the MI350X post sweeps a 31B from one GPU to eight, and Kimi-K3 was brought up across eight.

The honest boundaries: data-, pipeline-, and expert-parallel are selectable but not implemented — plowc rejects them with an explicit error rather than silently falling back to TP. We evaluated a mixed tensor/pipeline split for one model and rejected it on measurement: the seam we expected to save turned out to cost more than it saved. Inter-node is designed but not built.

It runs several compiled models at once

This is the result we’re proudest of, and it falls out of compiling ahead of time. Point plowrt serve at several --assets directories and it serves them all from one process, on one GPU, behind one OpenAI-compatible endpoint.

The reason it works is that a compiled artifact is measurable before it runs. The planner reads each blob’s header — weight tensors, KV arena at the compiled context, activation and table bytes — and computes that model’s exact VRAM footprint without touching the GPU. Overhead is measured against the driver at first load and cached, so the planner self-calibrates. A JIT stack cannot do this: it doesn’t know what it’s going to build until it builds it.

From there it’s bookkeeping the compiler has already made possible. The VRAM-fitting subset stays resident; a request for a non-resident model evicts LRU engines until the planner says the target fits, then loads it. Concurrent requests for the same target coalesce onto one switch. If eviction still can’t make it fit, the request is shed with a 503 and a Retry-After rather than an out-of-memory crash. Resident models each keep their own dispatcher and share a single context, so their launches are stream-ordered against each other and no global launch lock is needed.

The boundary, stated plainly: this is switching, not full multi-tenancy. There is no weight paging and no shared arenas — a model is either resident or it isn’t.

---

What the design buys, and what it costs

This post is about the architecture, so it deliberately carries no benchmark tables. Numbers belong with their methodology, their hardware, and their caveats — they live in the performance posts and, in raw form with every refutation attached, in perf-data/ in the repo. What follows is the shape of the thing, which is what you actually need to decide whether the design is interesting.

Where a fixed plan pays. Decode latency stays close to flat as context grows. A dynamic scheduler re-derives its work every step and that per-step cost grows with sequence state; a compiled plan has already paid for it. The effect is strongest on models whose attention is mostly sliding-window, where only the full-attention layers scale at all. This is an architectural property rather than a tuning result — it shows up on Blackwell and on both CDNA generations.

Where a fixed plan doesn’t. Short-context, batch-1 decode is not our region and we don’t have a story that makes it one. The per-packet machinery — gating, signalling, protocol — is a floor you pay whether the context is 1k or 64k, so the shorter the context, the larger that fixed cost looms relative to the work.

Prefill. This has been the weak side for the whole project and it is where most recent effort has gone. It is much better than it was, still behind on a single stream, and better than the alternative under concurrency, where a compiled admission path holds steady while a dynamic queue builds. The auto-tuner has never been pointed at the prefill path at all, which is the single largest known lever in the codebase.

Serving. plowrt’s muxer does continuous batching — arrivals land in idle slots without waiting on the current tick, with admission, preemption, graceful drain and per-slot streaming — and multi-step decode is on by default, producing several tokens per tick with the step count scaled inversely to batch size.

---

Checked, not just tested

The counter protocol is the part of this design that can fail in ways tests don’t reach: a missed signal shows up as a race under load on hardware you don’t own, months later.

So it’s proved. lean-plow/ carries 185 theorems across 29 Lean 4 files covering the counter protocol, rewrite-rule soundness, tile-partition completeness, memory disjointness, and cost-model monotonicity — plus a plow_verify CLI that checks a compiled artifact against them. On top of that, 997 workspace tests, and kernel bodies gated against CPU f64 oracles for token-identity.

That combination — proofs for the protocol, oracles for the kernels, gates for the end-to-end tokens — is what makes it possible to bring up a 93-layer hybrid MoE in a day without guessing.

---

Adding a model is a pipeline, not a pull request

The part of this release we’d most like people to take is not the kernels. It’s docs/bringup/ — the distilled form of how every architecture in the tree (Gemma, Llama, Qwen, DeepSeek, GLM, Kimi-K3, Nemotron) was actually brought up.

Seven stages, each with a blocking gate:

Stage	Gate
1. Operator IR	Graph builds, shapes infer, the serde-free IR still compiles
2. egglog rewrite	Saturation runs, intended fusions fire, every fired rule is in soundRules
3. Lean verify	lake build clean, checkpoints A–G certify, no vacuous proofs
4. Kernel tuning	Hot kernels at the measured roofline, winners read back from tunedb
5. Single-block sweep	Block matches the oracle reference; block latency at target
6. Runtime optimization	TTFT/TPOT at target concurrency, memory fits, no shed-request artifacts
7. Perf campaign	Correctness battery passes and perf targets met, written up in perf-data/

Each stage also ships a matching agent prompt — a self-contained brief telling an LLM coding agent what to read, what to change, what to run, and the gate it has to clear. The ground rules are enforced by the prompts themselves: an agent that cannot pass its gate stops and reports rather than proceeding or weakening the gate; sorry fails stage 3; every A/B runs both arms in the same GPU lease, one lever at a time.

That last set of rules isn’t idealism. It’s LESSONS.md — ten ways this campaign produced confident wrong answers, and the discipline that now prevents each. A gate that gets carried forward red turns a one-stage defect into a multi-stage bisection, and we have the write-ups to prove it.

The Kimi-K3 day-zero bring-up is what this pipeline looks like when it’s run under time pressure.

---

Where the bodies are buried

Two things we’d rather you hear from us.

The e-graph rewriter is advisory. crates/rewrite runs egglog equality saturation and it works, but no rewrite it finds reaches a GPU. Both plowc paths discard the fused graph, and crates/devgen — the emitter every shipped asset comes out of — doesn’t depend on it. The fusions actually in a packet stream are hand-written. This isn’t a bug on the way to a win, either: when we measured it, deleting packets bought almost nothing and the un-fused arm ran faster.

We retract our own results, including ones that flattered us. Three so far. A comparison against another engine improved sharply once we noticed the baseline was wrong rather than our code getting faster — it had been recorded with prefix caching left on, so we threw it out and re-ran with one client driving both engines in one session. A win of ours evaporated once we measured our own serving stack and compared server against server rather than our kernel against their server. And an entire sweep was retracted after an emitter bug meant the objects had been staging rows past the end of shared memory — it passed every gate, because the gate ran at batch 1 and the broken configurations did not.

All three are written up in perf-data/ with the reasoning, next to the results they invalidated. If you find a fourth, we’d like the issue.

---

Run it

Five GPUs across four recipes are exercised end-to-end and documented in the README: RTX 5090 and RTX PRO 6000 Blackwell (sm_120), MI350X and MI355X (gfx950), and MI300X (gfx942). Gemma-4 12B/31B dense and 26B-A4B MoE are the walkthrough paths; Qwen3 and Llama-3.1 also emit. GLM-5.2 and Kimi-K3 have builders in crates/nn-graph/src/models/ and the campaign data above, but no first-run recipe yet — treat them as bring-up in progress rather than supported.

git clone https://github.com/infervisor/plow && cd plow
nix develop
cargo build --release -p plowc
cargo build --release -p plowrt --features cuda,hsa

The dev shell provides the CUDA and ROCm toolchains, so interpreter and kernel builds need nothing installed on the host. plowrt doesn’t link CUDA or HIP either — cuda/hsa dlopen the drivers at runtime, so one binary runs on NVIDIA, on AMD, or falls back to a (correct, very slow) CPU backend.

Then pick your recipe. The compiler takes a checkpoint directory and emits a bundle; the runtime loads it and serves an OpenAI-compatible API:

# compile — checkpoint to packet stream, MI355X shown
plowc --hf-dir ~/models/gemma-4-12B-it \
  --arch gfx950 --gpu mi355x --max-ctx 131072 \
  --out ~/plow-assets/gemma4-12b

# serve — OpenAI-compatible API on 8080
plowrt serve --assets ~/plow-assets/gemma4-12b --port 8080

The one step folded away above is the per-target interpreter object, built into the same assets directory by scripts/build_gfx950.sh and its sm120 / gfx942 siblings. An emit is not portable across SM counts — emitting for a 188-SM RTX PRO 6000 and running it on a 170-SM 5090 mis-schedules the work, so the flags in the README’s support table are load-bearing.

The architecture is documented properly in docs/arch/ — 15 chapters covering the compiler pipeline, tile graph, scheduler, packet ABI, counter system, runtime, cost model, formal verification, multi-GPU, prefill chunking, and where the AMD architectures diverge. Those chapters are the authority; this post is the tour.

---

Where this goes

plow is not finished, and we are not treating this release as a trophy drop. We are committed to developing it into a production release in the open — same repository, same license, same habit of publishing the refutations next to the wins. The roadmap is not a secret: pointing the auto-tuner at prefill, wide-rung decode objects routed to the GEMM family, data- and pipeline-parallel, inter-node, and the model coverage the bring-up playbook is designed to make routine.

Contributors are welcome, and so is disagreement. The most useful thing you can send us is a measurement that contradicts one of ours — this codebase has retracted its own numbers three times already, in both directions, and each retraction came from someone re-running a thing properly. Concretely:

• Prefill is the biggest open lever, and the tuner has never been pointed at it.
• Wide-rung decode. Routing the dense projections to the MFMA GEMM family at large batch is a well-specified piece of work with a clear payoff.
• New hardware. If the persistent-interpreter model holds on a fifth ISA, that is worth knowing. The porting surface is smaller than it looks — the compiler, packet ABI, and counter protocol come along unchanged.
• New models. docs/bringup/ exists so this does not require us.

Issues and pull requests, or CONTRIBUTING.md for the build and review conventions. If you’d rather just tell us we’re wrong about something, that works too.