Engineering Blog
Deep dives into GPU observability, inference infrastructure, and accelerator tooling.
Kimi-K3 day-zero bring-up on 8× MI355X
A 93-layer MLA + KDA hybrid MoE, from “cannot load” to a correct answer
What day-zero support for a 93-layer MLA+KDA hybrid with 896 experts looks like from inside a compiled runtime — the interpreter, counter dataflow, the Lean-gated compiler, the wave/warp switch, tensor parallelism, per-XCD queues and the counter hierarchy they unlock, the correctness bugs a near-right model hides, an end-to-end accuracy check through the served endpoint, and a measured 28.9 ms/token decode taken apart against its bandwidth and protocol-chain floors.
Read article →Long-context decode on 8× AMD MI350X
Per-token latency vs context: plow grows 1.52× where vLLM grows up to 2.9×
From 1k to 128k context, vLLM’s per-token latency grows up to 2.9× — plow’s grows 1.52×. Where the fixed execution plan pays off, crossovers included.
Read article →Gemma-4 decode on RTX PRO 6000 Blackwell
A methodology-first, correctness-gated comparison with vLLM
Batch-1 decode on an RTX PRO 6000 Blackwell: plow fp8 beats vLLM fp8 on Gemma-4-26B MoE from 16k to 128k context — methodology, roofline analysis, and the not-yet-tuned paths included.
Read article →Plow: the compiled inference runtime
Why we moved optimization and orchestration out of the request path
The case for moving orchestration and optimization out of the request path — and a clear separation between measured gains, architectural properties, and open evaluation work.
Read article →InferSight: cross-vendor GPU observability
Production-grade GPU telemetry and monitoring for heterogeneous AI infrastructure
A unified, vendor-agnostic platform — built in Rust — that brings first-class GPU telemetry and interactive monitoring into a single, extensible toolkit.
Read article →