Scheduling inference onto SRAM

What was measured, and what wasn’t. Every Xeon number below was measured by us on one Intel Xeon 6975P-C in an AWS bare-metal instance (c8i.metal-48xl class, single socket). Groq, d-Matrix, Cerebras and NVIDIA figures are the vendors’ own published numbers, cited as reference designs; we did not benchmark those systems. Xeon results come in three kinds, and every table says which. Stage: one real decoder layer on one socket, every weight in on-chip cache. Serving: a whole model served end to end, which on one socket reads its weights from DRAM. Projection: measured stages placed on a multi-socket plan with assumed link costs. Nothing has run on more than one socket yet.

The question

The fastest inference systems of the last few years share one idea: keep the weights in SRAM next to the compute, and move tokens instead of weights. Groq spreads a model over many 230 MB chips and schedules every transfer at compile time. Cerebras puts 44 GB of SRAM on a wafer. d-Matrix pairs 2 GB of SRAM per card with LPDDR for capacity. HBM GPUs instead stream every weight from off-chip memory for every token.

What makes those systems work is as much software as silicon. A compiler has to decide which weights live in which SRAM, when each byte moves, and how compute waits for data, without a scheduler in the request path. This post studies that software problem on hardware anyone can rent.

An Intel Xeon 6975P-C (Granite Rapids) has 96 cores. Each core has a private 2 MiB L2, 192 MiB per socket, and the cores share a 480 MiB L3. Every core also has an AMX matrix unit: 512 BF16 MACs per cycle, which we measured at 3.27 TFLOP/s per core and 294 TFLOP/s over 90 cores. That is about 672 MiB of on-chip SRAM per socket. The research question: what does software have to do to schedule inference onto that SRAM, and how far does it get? This post is the single-socket half of the answer, from a campaign in early October 2026.

---

The software stack

Treating a cache as SRAM is a scheduling problem at every layer, from the compiler down to a kernel driver. Here is the stack we ended with, and the measurement behind each decision. The rest of the post goes through it.

Six design rules came out of it:

1. Residency is a placement problem, not a locking problem. Size each core’s slice to fit, back it with 2 MiB pages, and first-touch it on the core that owns it. Hardware LRU then keeps it resident.
2. Plan with measured constants. The planner’s cost model is built from microbenchmarks on the target. It predicted an E2B layer at 44 µs (batch 1) and 209 µs (batch 16) against 47 and 213 measured.
3. The exchange count is the budget. Once weights are on chip, cross-core synchronisation sets the time per layer, not math.
4. Batch to feed the matrix unit. AMX only pays once there are enough rows per tile.
5. Stream the KV; don’t try to hold it. Long-context attention is a DRAM-bandwidth problem.
6. Separate the workers from everything else. Reserve cores for the OS, and give the runtime’s host threads somewhere to run.

---

The memory ladder, measured

We pinned one thread to each of 90 isolated cores and streamed a per-core buffer, sized to land in each tier, for 2 s per repetition.

• L2 is 23× DRAM; L3 is only 2×. The prize is the private L2. L3 is a large but modest tier: a loop over up to ~9 MiB per core (~800 MiB per socket) still streams at L3 speed. That is more than nominal L2 + L3, because the non-inclusive L3 keeps part of the stream.
• Page size decides L2 residency. With 4 KiB pages, physical page colouring maps a 1.5 MiB buffer unevenly over L2’s 2,048 sets. Conflict misses start at 1.0 MiB on some cores, and the mean falls 25%. 2 MiB pages cover the sets uniformly, and every core then holds 1.5 MiB at full rate, the slowest within 0.2% of the fastest.
• Per core, a cache line per cycle is the ceiling. A Xeon core moves data in 64-byte (512-bit) cache lines, and the L2 returns at most about one line per cycle to the core. We measured 171 GB/s per core for AVX-512 reads and 139 GB/s for AMX tile loads: about 48 bytes per cycle at the clocks those loads sustain (~3.5 and 2.9 GHz). GPUs work with 128-byte (1,024-bit) cache lines, split into 32-byte sectors, and each SM’s L1 / shared memory serves 128 bytes per cycle. By that arithmetic an H100’s 132 SMs have roughly 30 TB/s of L1 bandwidth, about twice the socket’s, from wider pipes per unit. The Xeon’s 15.4 TB/s comes from 90 narrow pipes, so the software has to keep all 90 busy at once, and every exchange between them is paid in 64-byte lines.

Clocks held under load: AVX-512 at 3.5–3.8 GHz and AMX at 2.9–3.2 GHz, for 10 s at 570–602 W, with no throttling.

---

Reference points: published SRAM designs

These systems were not measured by us. The chart places their published capacity and bandwidth next to the socket we did measure, for scale.

On paper a socket holds about 3× a Groq chip’s SRAM at about a fifth of its published bandwidth. It has no deterministic chip-to-chip fabric, but it does have a full DRAM tier under the SRAM, and it is already in every datacentre. The question here is not whether a CPU matches an LPU. It is whether the SRAM style of scheduling carries over to hardware that is already deployed.

---

Isolating the socket: 2 cores per NUMA node for everything else

An SRAM schedule assumes nothing else touches the cores or their caches. The box runs in SNC3 mode: three NUMA nodes, one per compute die, 32 cores each. We isolated it without a reboot:

• 2 housekeeping cores per SNC node (cores 0–1, 32–33, 64–65 and their SMT siblings) take everything that isn’t inference. 161 of 165 device IRQs were moved there, irqbalance stopped. The kernel workqueue cpumask and the system services’ cgroups are confined to them.
• 90 inference cores, 30 per node, run one pinned worker each.
• Deep C-states are held off with a zero-latency /dev/cpu_dma_latency request, because C6 flushes L2. THP is in madvise mode, NUMA balancing is off.
• What remains on an inference core is the 100 Hz timer tick (nohz_full needs a reboot) and about 2 other interrupts per second.

One finding mattered more than any tuning flag. Confining the runtime to exactly the worker CPUs cost 30% of decode time. Its engine and I/O threads then time-slice against spinning workers on every token. E2B TPOT was 13.83 ms that way. Giving those host threads the inference cores’ SMT siblings brought it to 10.47 ms. A schedule that owns every core still needs somewhere for the host to run.

---

Making cache behave like SRAM: pseudo-locking

A cache is not a scratchpad. Any line can be evicted by whatever the core, its SMT sibling or the kernel touches next. Intel’s Cache Allocation Technology (CAT) gives software a handle on this. Each core runs in a class of service (CLOS), and each CLOS has a bitmask of the cache ways it may allocate into. Pseudo-locking builds on CAT:

1. Reserve some ways that no running CLOS may allocate into.
2. Fill a buffer into exactly those ways from a dedicated fill CLOS, with prefetchers off so nothing else gets pulled in.
3. Move everything else back to classes that exclude those ways.

The lines can still be read, but replacement can no longer evict them. In effect the buffer is SRAM mapped at an ordinary virtual address.

Linux kernel support

Mainline Linux has supported this since 4.19 (Reinette Chatre’s pseudo-locking series), through the resctrl filesystem that manages CAT, CDP and memory bandwidth allocation:

• You create a resource group, write pseudo-locksetup to its mode file, then write a schemata line for the region.
• The kernel allocates contiguous memory and loads it into the reserved ways from a core in the right cache domain, with hardware prefetchers off. It exposes the region as a character device under /dev/pseudo_lock/ for an application to mmap().
• It restricts deep C-states, because power management can shrink or power off cache.
• A debugfs trigger, pseudo_lock_measure, reports L2 or L3 residency of the region.

The documentation is candid about the limits. Placement is not guaranteed. WBINVD and CLFLUSH still evict “locked” lines. The sanity checks run only at mmap() time.

Three things keep it from working on this box:

• Model gating. The kernel enables pseudo-locking only on CPUs whose prefetcher-disable bits it has validated: Broadwell-X and Goldmont / Goldmont Plus. Every other model is refused, including Sapphire, Emerald and Granite Rapids, whose CAT hardware is present (cat_l2 cat_l3 cdp_l2 cdp_l3).
• Kernel config. The Amazon Linux 6.18 kernel on this instance is built without CONFIG_X86_CPU_RESCTRL, so there is no resctrl filesystem at all.
• The kernel owns the memory. resctrl hands you memory it allocated. A runtime wants the opposite: lock buffers it already placed, node-local, huge-page-backed, holding the exact weight slice each core uses.

Linux 6.16 moved the filesystem half of resctrl to fs/resctrl, so Arm’s MPAM can share it. Cache partitioning is becoming a cross-architecture kernel interface, but pseudo-locking is still gated per CPU model.

What we built: pseudo_lock_sram

So we wrote an out-of-tree driver: runtime/cpu/driver in plow, Apache-2.0. The same directory carries a patch that adds the three Rapids generations to resctrl’s model list, for anyone who can rebuild and reboot their kernel. At load, the driver partitions L2 and L3 with CAT. It then locks memory that the runtime owns, either into one core’s private L2 or into one SNC node’s L3 slices. Everything goes through ioctls on /dev/pseudo_lock:

• lock: pin, fill on the owner core in the lock CLOS with prefetchers off, measure;
• measure: classify every line by calibrated L1/L2, L3 or DRAM latency, read from the owner;
• reload and unlock;
• worker: run a CPU in the worker CLOS.

Locks are released when the file descriptor closes, so a crashed server cannot leak pinned cache; we checked with SIGKILL mid-run. Masks come from CPUID leaf 0x10. Only 7/8 of the reserved ways are lockable, the rest left as set-conflict headroom: 12 L2 ways give 1,344 KiB per core and 14 give 1,568 KiB. plowrt drives it through memory/sram.rs (PLOW_CPU_SRAM).

An earlier version measured about 98% DRAM. Each of these pieces fixed part of that:

• Non-inclusive L3. A read hit moves the line into the reader’s L2, and on eviction the line refills L3 through the reader’s CLOS. Readers in CLOS 0 therefore “unlock” what they read: 122.5 MiB locked per node fell to 30–47% held. Workers now run in a worker CLOS, and the fill CLDEMOTEs each line.
• SNC. A line is cached only in its home node’s slices. An L3 lock has to be per node, filled from a CPU on that node, from pages on that node.
• C6 flushes L2. The driver holds a zero-latency CPU QoS request while it is loaded.

The surprise: we didn’t need the lock

Each of the 90 cores streamed its own BF16 weight slice every ~10 µs, under four kinds of interference: none, the activation exchange, a KV stream from DRAM, and KV from LLC with DRAM streamers on the housekeeping cores.

With 2 MiB pages and plain LRU, the weights already stay resident. A slice re-read every few microseconds is never the eviction victim. With no interference the lock holds 99.99%, but under DRAM traffic a few locked lines still leave L2. The likely cause is snoop-filter back-invalidations, which we counted (38.7 K/s) but could not attribute per line. A lost line refills through CLOS 0 into the four unlocked ways, competes with the KV stream there, and is evicted again on every step. The lock turns a one-off loss into a miss on every step.

The same held at every scale we tried:

• Batched layer stages: the lock costs 1–26% for E2B and 15–53% at batch 16 for 12B, 26B and 31B slices, because KV and activations thrash the 2–4 free ways.
• Whole-model serving: locking 209 MiB of layer tensors into L3, plus worker scratch into L2, made decode 4.8% slower.

The lock’s one win so far is narrow: holding each worker’s packed AMX scratch in L2 cuts 31B prefill by 9% (a candidate recipe, not yet qualified). We run unlocked, and keep the lock as an instrument: if a slice holds under the lock, the slice fits.

---

A real layer in L2: synchronisation, not math

Next we took a real BF16 Gemma-4 decoder layer from the checkpoint and split its rows over the 90 cores. Each core keeps its rows of all nine matrices in its own L2. An E2B layer is 69–83 MiB, or 0.77–0.92 MiB per core, so it fits. An E4B layer, at 1.97–2.25 MiB per core, does not, and runs as an L2/L3 split.

Every run is checked against an FP32 reference from the Hugging Face model at 17 op boundaries. The bar is error at most 1.5× a BF16 reference at every boundary, with output cosine ≥ 0.99999. All 840 runs across the layer, sync and long-context phases passed, bit-identical across worker orderings and 200,000-step runs.

A row-partitioned GEMV needs the whole previous vector on every core. A decode layer therefore has seven data-dependent all-to-all exchanges: q/k/v, attention partials, attention output, o, the FFN activation, down, and the per-layer input. Each vector is only 0.5–20 KiB. Every exchange is bound by latency, not bandwidth.

• Per-die replicas with non-temporal stores remove ~83% of the exchange cost: 147.8 → 51.8 µs per layer. Plain stores leave lines in the producer’s L2, where every remote reader has to snoop them.
• Dropping the barriers doesn’t help. Flag-in-data, where readers poll for a step tag in each line, removes 7 of 8 barriers and is still no faster. The cost is the dependency itself: the slowest producer plus one cross-die line transfer.
• There is a floor. The best full-socket exchange we measured is 1.81 µs, so seven of them cost at least 12.7 µs per layer. A die-local exchange is 0.58 µs, but one die’s L2 cannot hold a layer. L2 residency needs the whole socket, and the whole socket costs at least 12 µs of exchange per layer. On this part the two goals conflict.

---

Feeding AMX: where the matrix unit falls short

AMX is the reason a CPU stage can be fast at all, and it is also where the stage is hardest to feed.

• One token fills 1/16 of a tile. TDPBF16PS multiplies a 16-row A tile by a weight tile. At batch 1 only one A row carries data, so AMX is no faster than AVX-512: 46.8 vs 50.6 µs for an E2B layer.
• The feed is narrower than the compute. Tile loads stream 139 GB/s per core, below the 171 GB/s AVX-512 reads get from the same L2, and AMX runs at a lower clock: 2.9–3.2 GHz against 3.5–3.8 GHz for AVX-512. With BF16 weights each weight byte feeds B FLOPs at batch B. Keeping 3.27 TFLOP/s busy from 139 GB/s therefore needs about 24 rows per tile from L2, and about 450 from DRAM. At B ≤ 16 even L2 cannot saturate the TMUL.
• 16-row granularity breaks balance. Weights split in 16-row tiles. A 1,536-row matrix is 96 tiles over 90 cores, so 6 cores do double work: max/mean bytes per core 1.59. A 26B MoE expert has 704 gate/up rows, 44 tiles over 90 cores: max/mean 1.70, which no longer fits L2. Balancing the leftover rows on AVX-512 was slower at every batch above 1.
• Layout costs. Weights must be pre-packed into VNNI tiles at load, and activations repacked on every call. Skipping the activation pack, a timing-only experiment, bounds that cost at 18% of an E2B batch-32 decode step (30.1 → 24.7 ms).
• AMX is more fragile next to streams. After touching 64–256 KiB of KV, the next AMX weight pass runs 9–13% slower even with the weights still resident; AVX-512 GEMV loses ≤ 1%. Beside long-context KV streams, AMX GEMV time grows 16–29%. We have not found the mechanism.
• BF16 inputs bound accuracy. TMUL takes BF16 operands, so attention’s softmax probabilities have to be rounded to BF16 for the PV product. On 26B-A4B that flipped near-tie MoE routing decisions and failed our FP32 gate (KL 0.118–0.128 vs a 0.112 limit). Splitting P to keep it FP32-accurate through the TMUL fixed it.

Batching is what pays:

* E4B TP2 rows fail the strict gate on single-element boundaries (traced to one BF16 rounding); the layer output passes.

Long context breaks residency. Decode attention past ~8K tokens is DRAM-bound at 500–626 GB/s, 80–97% of peak. At 32K the KV stream pushes 29% of the weight slice out of L2 on every step, and from 64K the whole slice is refetched each step. The best policy we measured is tiled online-softmax attention, 256 KiB tiles, with software prefetch 8 rows ahead. Explicit L3 staging copies cost more than they saved.

---

What runs today: one socket, whole models

A whole model doesn’t fit in one socket’s cache: E2B reads 4.28 GiB of weights per token. So single-socket serving is DRAM-bound and runs with no pseudo-lock. The stage work still paid off. Its synchronisation lessons went into plowrt’s CPU engine as combining-tree completion counters, 7–10% faster decode at c1, alongside AMX FLASH_PREFILL, a weight-stationary GEMM partition and packed prefill.

Infervisor wins every row, including TTFT P99 (0.36–0.84×) and peak memory (0.27–0.77×). E2B and E4B ran on the binary with combining counters; 12B and 26B-A4B on the one before it. E2B decode at c1 is 9.3 ms per token and reads 3,964 MiB of DRAM per token: about 445 GB/s against a measured 612–669 GB/s. That ceiling is why a model has to be spread over many sockets’ SRAM to go faster.

---

Future scope: rack-scale Xeon with a static schedule over RDMA

Groq’s central idea is less about SRAM than about taking the scheduler out of the request path. Its compiler decides which chip computes what, when, and when each byte moves between chips, over direct chip-to-chip links. d-Matrix scales out with a purpose-built NIC (JetStream) beside its Corsair cards. Cerebras keeps the model on one wafer, or a few. plow already works this way on GPUs: it compiles a model into a static packet stream, and on-device counters resolve dependencies. The open question is whether the same contract can span sockets and servers.

The design that falls out of the single-socket measurements:

• The unit is one socket. 90 isolated cores, 6 housekeeping, a weight slice of ≤ 1.6 MiB per core in L2, unlocked, running a microbatch of up to 16 rows on AMX.
• The compiler decides the partition. plowc stage-plan reads the checkpoint shapes and packs units in execution order. A dense layer too large for one socket becomes a tensor-parallel group of the smallest socket count that fits: 12B at TP4, 31B at TP8. Each such layer needs two all-reduces.
• The all-reduce sizes the fabric, not the hop. At batch 16 a 31B TP8 rank moves 588 KiB per all-reduce, which in a 10 µs budget is 60 GB/s per socket. So TP groups stay inside a server, on UPI. Pipeline hops carry only B × hidden × 4 bytes, 6–21 KiB at B=1, and can cross servers.
• Hops are one-sided RDMA writes. Each stage pre-registers a landing buffer per hop. The producer issues an RDMA WRITE of tagged cache lines, and the consumer polls the tag: the flag-in-data protocol we validated across dies, stretched across NICs. No kernel, no RPC, no runtime scheduler. Intel DDIO lands inbound DMA in a small set of L3 I/O ways, which our driver keeps out of the lock by default (l3_io_ways=0), so incoming activations land on chip.
• The schedule is fixed at compile time.

A CPU isn’t cycle-deterministic the way an LPU is. The 100 Hz tick still runs on isolated cores, and we measured rare 0.3 ms outliers outside p99. So either each slot carries measured jitter as slack, or the schedule is dataflow-triggered: a fixed order with a ready flag per slot, the way plow’s counters already work on GPUs. Measuring that tradeoff is part of the next step.

For dense models the projection trades sockets for latency at equal or better per-socket throughput. 12B goes from 45 ms to 4.1 ms per token at B=1, with 167 tok/s per socket at B=16 against 152 on one socket at c32. For MoE, holding the experts only in L2 spends 390 of 437 sockets on experts that sit idle most of the step; spilling them into L3 at three sockets per layer gives 250 tok/s per socket on 137. With slower links (hop 10 µs, all-reduce 25 µs), B=1 TPOT rises 4–41% and B=16 throughput drops 3–7%.

What has to be measured before any of this is a result:

1. The hop and the all-reduce: over UPI and across a 400G RDMA fabric, at B=1 and B=16.
2. Two chained stages streaming tokens, including a split FFN, against a chained reference.
3. Sequence-parallel attention for full-attention layers at long context. A 128K full-attention stage takes 670–1,250 µs against ~60–80 µs elsewhere, and tensor parallelism replicates the KV heads.
4. MoE routing balance over real traffic, with experts assigned to sockets by load.
5. INT8 weights, which halve the bytes per core and the number of sockets.

---

What didn’t work

• Pseudo-locking weights. A net loss everywhere except one 31B prefill buffer. Locked lines leak under DRAM traffic and then miss on every step; the root cause is unconfirmed.
• A ≤ 4 µs synchronisation target. Infeasible on a full socket: the floor is at least 12.7 µs per layer.
• AMX at batch 1 and beside long KV streams. Equal to AVX-512 at batch 1, 16–29% slower beside long KV, and 9–13% slower after a KV phase for reasons we haven’t found.
• Balancing AMX’s 16-row tiles with an AVX-512 tail. Slower at every batch above 1.
• PREFETCHNTA and explicit L3 staging. NTA lines still allocate in L2 on this part. Staging copies cost more than they save.
• L2-only MoE expert placement. 437 sockets to serve what 137 serve with L2 + L3.

---

Code and sources

Code (Apache-2.0):

• Pseudo-lock driver, pseudo_lock_sram with its benchmark and the resctrl model-list patch
• Stage benchmarks and campaign drivers and phase reports P0–P6
• Serving comparison vs vLLM 0.30 CPU

References:

• Linux kernel, resctrl: Cache Pseudo-Locking, and the model gate in pseudo_lock.c
• Groq, GroqCard Accelerator product brief (230 MB SRAM per chip, up to 80 TB/s on-die)
• d-Matrix, Corsair Product Brief
• Cerebras, WSE-3 chip page and Hot Chips 2024 slides
• NVIDIA, H100 specifications
• Earlier posts: the plow architecture article and plow is open source