PRODUCT · IN DEVELOPMENT

A purpose-built GPU OS
for real-time inference

Not a general-purpose scheduler — an operating system for inference fleets

Models and latency budgets are the primitives. Compiled plowrt plans are placed across mixed NVIDIA and AMD fleets ahead of time, so at serve time the fleet has nothing left to decide.

General-purpose schedulers weren't built for token deadlines

Kubernetes was built to schedule containers; Slurm to schedule batch jobs. To both, a GPU is a countable device and a model is opaque cargo. A production inference fleet serves many models on many GPUs under tight latency budgets — and real-time workloads such as voice make the budgets explicit: a reply that streams late is a reply that failed. Running that through a general-purpose layer that assigns work as it arrives recovers utilization, but it reintroduces at fleet scale exactly what host orchestration causes inside a single GPU: decisions on the critical path, queueing variance, and tail latency that inherits every scheduling accident.

Static assignment avoids the jitter but wastes capacity; dynamic scheduling recovers capacity but sells the tail. Real-time serving needs a third option: an operating system for the fleet, built purely around inference.

A GPU OS: models as processes, budgets as the contract

The Orchestrator treats the fleet the way an operating system treats a machine — except its processes are models. Plow compiles a model into a fixed execution plan a GPU runs without consulting the host; the Orchestrator is being built to admit those plans against declared latency budgets, place them across GPUs, nodes, and vendors deliberately, ahead of time, and supervise them with measured telemetry — so that at serve time the fleet, like the GPU, has nothing left to decide.

A GPU OS where the model is the first-class unit

declare a model and its latency budget; the OS admits it, places plowrt plans onto GPU and CPU nodes ahead of time, and supervises them — where Kubernetes schedules containers and Slurm schedules batch jobs, this schedules models

A purpose-built GPU OS for real-time inference Model specs with latency budgets enter a GPU operating system control plane, which admits, places, and tracks them across NVIDIA and AMD GPU nodes and a CPU node, each running a plowrt agent. Placement happens ahead of time; telemetry feeds back through InferSight. MODELS — THE FIRST-CLASS UNIT MODEL SPEC voice-asr plan · plowrt-ready SLO p99 · 120 ms MODEL SPEC chat-llm plan · plowrt-ready SLO p99 · 250 ms MODEL SPEC avatar-tts plan · plowrt-ready SLO p99 · 90 ms orchestrator a GPU OS for realtime inference admission fit or reject against budgets placement models → GPUs, ahead of time fleet state measured, not guessed FLEET — MIXED VENDORS, ONE PLANE node 0 · NVIDIA plowrt agent asr gpu0 asr gpu1 llm gpu2 llm gpu3 node 1 · AMD plowrt agent llm gpu0 tts gpu1 tts gpu2 headroom gpu3 node 2 · CPU — pre/post plowrt agent tokenize vad stream io telemetry — InferSight at serve time: nothing left to decide

It is designed around four commitments:

  • SLO-aware placement. Models are packed onto GPUs against declared latency budgets, not best-effort queues — predictability is the contract, and admission control enforces it.
  • Mixed fleets, one plane. NVIDIA and AMD nodes managed from a single control plane, because plow's plans already run natively on both.
  • On-prem first. Bare metal, private cloud, sovereign and air-gapped deployments — the same Rust-first, no-Python-serving-stack posture as the engine.
  • Measured feedback. Fleet telemetry flows through InferSight; placement decisions are calibrated by measurement, in the same spirit as the engine's auto-tuner.