plow vs vLLM

plow separates model compilation from serving and can host several model bundles in one Rust process. Each bundle has its own checkpoint, settings and model name. vLLM offers broad model support and a large serving ecosystem. The practical choice depends on your models, deployment needs and measured workload.

Architecture and deployment

Sources: vLLM’s multi-model FAQ, compilation cache and sleep mode; plow’s runtime deployment contract and model manager.

The multiple-base-model row describes standard vLLM serving. vLLM also supports dynamic LoRA adapters, and its Gaudi plugin offers sequential model swaps.

Auto objective follows demand

--objective auto is plow’s default serving policy. It selects latency- or throughput-oriented rules from live decode width, queued requests and KV-memory pressure:

• Light traffic: latency-oriented decode grouping and prefill packing.
• Rising demand: switch to throughput rules when a queue forms, the decode batch widens or the KV budget fills.
• Traffic settles: return to latency rules after a two-second calm window, avoiding repeated mode changes around a threshold.

The policy adjusts supported decode and prefill mechanisms within the compiled plan. Backend and packet capabilities still determine which mechanisms can run. You can pin --objective latency or --objective throughput; /metrics exposes the active class and switch count as plowrt_serve_mode and plowrt_serve_mode_switches_total.

This is an adaptive scheduling policy, not a measured speedup by itself. See the serving-objective reference and policy implementation.

Multiple models, one process

Repeat --assets to register separate compiled bundles. Each can use a different checkpoint and served name. This example assumes compatible bundles at the paths shown, each with its weights and tokenizer in a checkpoint/ subdirectory:

plowrt serve --objective auto --bind 127.0.0.1 --port 8080 \
  --socket /tmp/plowrt.sock \
  --assets /models/model-a,slug=model-a \
  --assets /models/model-b,slug=model-b

Send "model": "model-a" or "model": "model-b" in the inference request. Each model has its own dispatcher and engine state. Models can stay resident together when memory permits; they still share compute capacity. The runtime provides scheduling between resident models, including round-robin turns. Co-residency does not guarantee simultaneous kernel execution or unchanged latency.

Load, unload and inspect

The control API manages residency without restarting the server:

• Load: make a registered model resident, or register a new bundle. Explicit loads do not evict other models unless evict: true is requested.
• Unload: stop admission, drain in-flight work up to a deadline, then release the model’s engine. The default deadline is 30 seconds; remaining work is preempted when it expires.
• Reload: an explicitly unloaded model stays unavailable until loaded again. Its name remains registered unless deregister: true was requested.
• Inspect: /v1/models/status reports resident and serving state, device placement and supported control capabilities. /v1/models lists registered models, including unloaded ones.

These operations use the owner-only Unix socket or the authenticated control listener, separate from the public inference listener. With the example server above:

curl --unix-socket /tmp/plowrt.sock http://localhost/v1/models/status

curl --unix-socket /tmp/plowrt.sock http://localhost/v1/models/unload \
  -H 'Content-Type: application/json' \
  -d '{"model":"model-b","drain_timeout_ms":30000}'

curl --unix-socket /tmp/plowrt.sock http://localhost/v1/models/load \
  -H 'Content-Type: application/json' \
  -d '{"model":"model-b"}'

The CLI command plowrt load downloads assets into the local store. The control API’s load makes a model resident in a running server. CPU and Metal support explicit load/unload but do not support automatic eviction. See the control API and CPU/Metal implementation.

Other reasons to evaluate plow

• Language and speech together. Supported LLM, speech recognition and speech synthesis bundles share the server’s API, model listing and metrics. Availability depends on the model and backend. See the deployment contract.
• Check bundles before serving. The loader checks packet/object pairing and runtime requirements. Bundles with checkpoint pins also check shard sizes and tensor headers before loading weights. These compatibility checks complement numerical validation. See the bundle checks.
• Separate downloads from startup. serve --model resolves locally prepared assets. Fetching a build is a separate step, so a provisioned server can start without contacting the registry. See model distribution.

Selected measured comparisons

These results compare plow with vLLM on specific models and configurations. They do not measure multi-model switching or establish a universal latency advantage. The board includes wins, ties and losses; each result links to settings and quality checks.

Choose for your workload

Check available plow builds against your model, hardware and precision. Then test the same prompts, output lengths, cache settings and offered load on both engines. Include memory use, cold and warm startup, and first-request latency after a model reload.

OpenAI-compatible clients can reuse the base URL pattern, but compatibility is endpoint- and option-specific. Validate the request fields, tool calls and streaming behavior your application uses. plow’s API compatibility notes describe its supported surface and gaps.