Part of the space heater series — Friday Morning Space Heaters, the breakeven posts, Does AI Agree With Itself? — and a follow-on to Qwen3.8-27B on Three Boxes.

Back in January I ended a post about quantization with a throwaway line: “These NVIDIA space heaters have been great this winter, but I’m beginning to wonder how they’re going to work out in the spring. Oh well, a problem for a few months from now.”

It is now the middle of August. It is a few months from now.

The answer, which I could have gotten any time since May by walking down the stairs with a thermometer, is that the basement has been sitting at 80°F and touching 85°F on heavy LLM days. Not dangerous. But it’s the room the whole lab lives in, and I spent four months not measuring it, because in January the heat was the feature.

TL;DR

The basement has no HVAC zone of its own. The thermostat that governs it reads a sensor one floor up, so GPU heat is never counted where it’s made — it’s 80°F down there right now.

Measuring that turned up something better. On an otherwise unloaded RTX 5090, sustained prefill pulls 570W, pinned against the card’s 575W limit. Decode on the same card and model pulls 277W — under half, though both report 99% “utilization.” Batching decode four ways only reached 315W.

The wrinkle is architectural: a sparse MoE with 3B active parameters is the weakest heat load in the building, while prefill lights up the whole card regardless.

So prefill-heavy work is winter work, and TeraContext spec takeoff — 100k-token windows, thousands of calls, short answers — is the best furnace we own. It’s also why the rack isn’t in Ashburn: at 109 W/U my density is low enough that colo bills me for floor space, not power — $900 to $1,100 a month against ~$50 metered at 15% duty.

The room has no vote

Here’s the actual defect, and it isn’t the GPUs.

The basement doesn’t have its own HVAC zone. It’s conditioned as an afterthought off the first-floor zone, and that zone’s thermostat reads a sensor in a hallway one floor up. I can put 2,400W of Blackwell into that room and the thermostat’s opinion of the house does not change by a tenth of a degree. It’s a furnace with no feedback path.

In winter that’s tolerable, and it’s most of why the January posts worked. Heat leaks up through the floor into the family room, which is where the breakeven post described the rack as “positioned strategically under the family room.” That was not strategy. That’s where the electrical panel is. But it worked: my wife got a warm family room, I got benchmark data, and the oil company got nothing.

In summer the same physics runs backward. The heat still leaks upstairs, where the first-floor zone fights it with air conditioning using a sensor that can’t see the source, while the basement climbs because nothing in the control system is accountable for it. Two seasons, one missing sensor, opposite outcomes.

What the card actually draws

One card was quiet this afternoon, so I sampled power and temperature while driving it through different load shapes. I’ve been benchmarking this rack for eight months and had somehow never once written down watts.

There are two 5090s in the basement and they aren’t the same card. One is a 600W board inside the Ryzen box. The other — the one I measured — is a 575W card in its own enclosure, attached to a host rather than installed in it, with its limit fixed at 575W floor and ceiling both. The other two GPUs on that host were parked at 0%, so nothing on that machine is contaminating the numbers. The 600W card in the Ryzen box was busy with unrelated work the whole time, which is exactly why I measured the other one.

That last row is the one to sit with. Prefill didn’t just approach the card’s rating — it sat about five watts under it, with samples peaking at 572W against a 575W wall. During a long prefill this card isn’t choosing how much power to draw — it’s being held back by the limiter. Decode never got past half.

That’s 2.06× the power of decode on the same card running the same weights: a 293W spread, about a slow cooker on high, hiding behind two rows that both report 99% utilization. That utilization figure is close to useless here. It reports whether a kernel is resident, not whether the transistors are doing anything expensive.

The mechanism is the same one that made the three-boxes post end up being about prefill. Decode emits one token at a time and spends its life waiting on memory — stream the weights, do a trivial amount of arithmetic, repeat. It’s bandwidth-bound, and a bandwidth-bound kernel leaves the tensor cores standing around drawing almost nothing. Prefill eats the entire prompt as one enormous matrix multiply. That’s FLOPS, that’s the tensor cores lit up, and that’s the only workload I’ve measured that makes a 575W card actually ask for 575W.

The corollary I hadn’t considered until I had both rows side by side: you cannot heat a house with chat. Every evening I’ve described as “firing up the space heaters” while asking a model who caught the Immaculate Reception, I was running the card at 48% of nameplate and calling it heating.

Sparse models make bad furnaces

The model on that card is a 35B mixture-of-experts with roughly 3B active parameters, and that detail turns out to matter more than I expected.

MoE decode is the least thermally productive thing a GPU can do. The router picks a couple of experts, so a given token touches only a small slice of the weights — less memory traffic than a dense model of the same nominal size, and still no meaningful math. That’s exactly why it decodes so fast at 254 tok/s, which is wonderful for chat and useless for heating. Sparse activation is a computational bargain, and heating is the one application where you don’t want a bargain.

Prefill doesn’t get that discount. Push 100,000 tokens through at once and the batch is large enough that essentially every expert gets work. The sparsity that saves you on decode buys you nothing on ingest.

Which sets up a rule I like:

Dense models are better space heaters than sparse ones. Prefill is a better space heater than decode. And the two effects stack.

The dense 27B on the other box lands where that predicts: its decode sits meaningfully higher, because a dense model streams every weight for every token, while its prefill ends up in the same near-nameplate territory. That box was serving other traffic while I sampled it, so I’m treating those figures as directional and re-running them when it’s quiet. The ordering is solid; I don’t yet trust the third digit.

Batching is the other lever, and it’s a weak one. Four decode streams instead of one added just 38W. It should help, and it does — not enough to matter. You cannot batch your way to a warm house.

Watts are watts

A GPU is a resistive heater with a side effect. Essentially all the electrical energy going in comes out as heat in that room; the tokens are free. The conversion is arithmetic at 3.412 BTU/hr per watt.

Adding up the enforced limits, the two Blackwell boxes carry 1,775W of GPU nameplate — a 600W 5090, the 575W external 5090, a 300W PRO 6000 Max-Q, and a 300W 5070 Ti. Add CPUs, drives, and power supply losses and the room’s ceiling lands right around the 2,400W I’ve been quoting since January, which is a nicer coincidence than I deserve:

The rack at full tilt is about a 1.6× space heater, which retroactively justifies eight months of calling it one. That same rack on decode-only work is more like 0.8×.

I want to be careful about the efficiency claim, because it’s the part people get wrong in both directions — including me, on the first draft of this post.

A GPU is electric resistance heat, and at $0.15/kWh that’s about $44 per million BTU. Against the heat pumps this is a rout: they deliver the same BTU for roughly $15 when they’re behaving, and around $22 when it’s cold enough that the coefficient of performance falls off. No argument there.

Oil is where I was wrong. I’d been carrying $3.50 a gallon around in my head, which lands oil in the high $20s and makes the GPUs look like three times the cost of running the furnace. Oil is not $3.50 anymore. It’s been $4.30–$4.50 a gallon in Virginia this year. A gallon of No. 2 holds 138,500 BTU, and a boiler at 86% AFUE delivers about 119,000 of them:

So the GPUs are about 15–20% worse than the oil burner, not three times worse. And during the March spike, oil crossed $46 per million BTU and the space heaters were briefly the cheaper heat in the house, which is not a sentence I expected to write when I started this post. Strictly as a heater the GPUs still lose — but only to the heat pumps by much, and to oil by a margin thin enough to disappear on a bad week for crude.

That comparison is a trap, though, and it’s the same trap as January. I’m not buying heat. I’m buying compute I was going to buy anyway, and the heat falls out for free — marginal cost zero, because the electricity was already committed to the thing I actually wanted. The only real decision is when to spend it, which is where a 293W gap turns into a plan.

Prefill is winter work

If prefill converts twice as much electricity into heat as decode, then the workload mix isn’t only a throughput question. It’s a seasonal one.

And the most prefill-heavy thing in the building is the thing we’re actually trying to ship. TeraContext takes a commercial RFP — a 500 to 2,000 page specification book plus the drawing set — classifies pages against a work breakdown structure, and bundles them into trade packages with a citation back to the paragraph it used. That means 100,000+ token windows, thousands of calls per job, and short outputs. As the three-boxes post worked out, roughly 95% of one of those turns is prefill: the model spends its time reading Division 03 and almost no time talking.

So construction spec takeoff is, thermally, the best workload we have. It’s the one job that holds a card at the power limit for hours instead of dropping to 277W between tokens, and it produces something a client is paying for.

The scheduling rule for this winter, which is the actual conclusion here:

• Batch prefill-heavy work into the cold months and the cold hours. Spec takeoff, corpus ingest, embedding and reranking backfills, long-context evals. Anything where the prompt is the work. Queue it overnight in January and the family room is warm by morning.
• Push decode-heavy work to summer. Chat, coding assistance, the debate platform, short-answer benchmarks. Same tokens either way, half the heat, in a room that’s already at 80°F.
• Prefer the dense model in winter and the sparse one in summer, when quality permits. I did not expect the model zoo to acquire a seasonal dimension.
• Long-context sweeps are a December luxury. A 176k-token window is the hottest thing I can run. That’s not an August experiment.

None of this changes a benchmark number. It just means the calendar gets a say in queue order, which it never had before.

Thirty minutes from the data center capital of the world

Which raises the obvious question. I live near Ashburn, Virginia. There is more data center capacity within a half-hour drive of my house than in most countries. Why is any of this in my basement?

Because of how colocation is priced, and because my deployment is precisely the shape colo prices badly.

Start with power. GPU rack rental varies, but all in you’re looking at roughly $300 per kW per month billed on peak demand. My rack peaks at 2.4 kW, so that’s $720 a month whether I use it or not.

Except $720 is the floor, not the bill. Providers charge on space or power, whichever is greater, and 2.4 kW simply isn’t much power. My gear needs about 22U — and consumer tower cases don’t rack, so each one eats four or five U sitting on a shelf. Spread 2.4 kW across 22U and the density is 109 watts per U, in a market where cabinets are routinely provisioned at 5 to 10 kW and up. I’d be renting rack space I have no way to fill with power, which means the space charge binds, not the power charge. Half-cabinet space in Ashburn runs about $861 to $1,100 a month right now, and a full 48U cabinet is $1,500 to $1,875. Cross-connects, bandwidth, and remote hands stack on top.

So the honest colo number for my footprint is $900 to $1,100 a month, not $720. Low density is the trap: below roughly 3 kW a cabinet you stop buying electricity and start buying floor.

Meanwhile my workload isn’t steady. It’s bursty and irregular, maybe 15% duty cycle across a month — a weekend of benchmarks, a takeoff run, then days where the cards idle at 9W. Metered at $0.15/kWh that’s about $50 a month. Colo prices power like a subscription on the peak; the utility prices it like a meter on the consumption.

That’s roughly $11,000 a year, which is a couple of GPUs. And none of it includes the reset button: when something wedges I either pay a tech for remote hands or drive thirty minutes each way. Downstairs is thirty seconds. The rest of what colo sells, I already have — redundant internet, redundant power with UPS, and Cloudflare in front as the CDN.

Then there’s the part this whole post is about. Four months of the year the waste heat is worth something. No colocation contract will ever credit me for that. In Ashburn I’d be paying a premium to have someone else’s chillers throw away the exact BTUs I want in my family room in January.

None of which survives scale, and the reason isn’t money. Run it flat out at 100% duty cycle and the basement is still cheaper — about $259 a month against $900-plus. Home electricity wins on the meter at every duty cycle I can model. Colo doesn’t beat my basement on price and it never will.

What beats my basement is the ability to get rid of heat, and that’s the wall I’m actually up against. The room is out of thermal headroom. Put the current rack in cooling terms: 2.4 kW is 8,200 BTU/hr, which is 0.68 tons of heat rejection. The dedicated cooling capacity currently serving that room is zero tons. That gap is the entire reason it sat at 85°F all summer, and it’s why the honest answer to “can you add another card” is no.

And to be clear about which constraint is actually binding, it isn’t the panel. That 2.4 kW is 20 amps spread across three 15-amp circuits. Derated to the 80% you’re allowed to hold continuously, those three circuits carry 4,320W, so I’m running at about 56% of my electrical capacity with roughly 1,900W of headroom — call it another big card and its host. I could plug more hardware in tomorrow and the breakers would be perfectly happy. The room would cook. The wiring has room and the air does not, and that’s the whole problem in one sentence.

The arithmetic past that is unforgiving. A 5 kW rack wants about 1.4 tons of dedicated cooling; 10 kW wants nearly 3 tons — a real mechanical system, ducted and drained and condensate-pumped. At that point the panel does become an issue too, since 10 kW is 83 amps and a subpanel. That is not a shopping trip. It’s a major construction project on a house that was never meant to be a data hall, and I’d pay for it twice: once to build it, then again every July to run it.

And once the answer to “how do we get more compute” is “renovate the house,” the answer is really lease commercial space. Not a colocation rack — paying for floor space I can’t fill with power is still terrible value for a bursty load, and I already own the redundant power, redundant internet, and CDN that colo is mostly selling. Commercial space is different: I’d be buying square footage with cooling engineered for the load and power still on a meter, and a loading dock instead of a basement stairwell. On which point — try getting a full 42U cabinet, call it seven feet on casters, down a straight run of basement stairs. There’s no corner to snag on. It’s just that tilting seven feet of steel back onto a staircase runs the top of it into the ceiling, and once it’s leaning it very much wants to get to the bottom faster than you do. The workaround is building the rack in place from parts, which is its own kind of penance. That’s a different blog post.

So the honest version: the basement wins right now precisely because the load is irregular and the heat has somewhere useful to go. Both of those stop being true the moment this is production. The thing that makes it a good furnace is the same thing that caps its size — and the space heater era has an expiration date measured in tons of cooling, not dollars per month.

What has to happen before November

1. Put a sensor in the basement. This is the whole problem. I already run dcgm-exporter, Prometheus, and Grafana. I checked what’s actually in there: GPU die temperature, GPU memory temperature, motherboard hwmon, thermal zones. There is no metric for the room. I can tell you a 5090 hit 68°C during a 100k-token prefill at two-second resolution, and the best I have for the space that card is heating is “80, I think.”

2. Get the heat where the vote is. A real zone for the basement, or a transfer path — a return near the rack, a boosted duct into the first-floor return. In winter I want that air upstairs and in summer I want it gone, which is the same duct with opposite intent. The external 5090 is the interesting case: a standalone enclosure putting 570W into one concentrated spot rather than burying it in a chassis, which makes it the easiest heat source in the room to duct on purpose.

3. Wire the furnace interlock. In January my wife asked me to set up the oil furnace to come on when the heat pumps can’t keep up. I said I’d look into it tomorrow. That was seven months ago. Once the room is instrumented, the interesting version becomes possible: let GPU heat count as the first stage. Heat pumps, then GPUs, then oil.

4. Set a summer ceiling. 85°F is fine for silicon and less fine for the drives, the power supplies, and the UPS batteries. Past a threshold, the long-context sweeps wait for October.

So

The joke for eight months has been that these are space heaters that happen to do inference. The measurement says something sharper: they’re space heaters with a high setting and a low setting, the high setting is the one that reads thousand-page specification books, and I have been running the low setting on purpose every time I sat down to chat with them.

At dinner I told my wife the basement had been hitting 85 degrees this summer.

“I know,” she said. “That’s why I stopped putting the laundry down there in July.”

She had known since roughly the first week of June, and had not mentioned it, on the theory that I would eventually measure something and find out on my own. Which is exactly what happened.

“Good news is it should be great in January.”

“Did you ever set up the furnace?”

“…I’ll look into it tomorrow.”

Power and thermal figures sampled August 17, 2026 on the externally-enclosed RTX 5090 — 575W enforced limit, default and maximum both — running Qwen3.6-35B-A3B-NVFP4 under vLLM at 262k context, FP8 KV cache, vision enabled, with the other two GPUs on that host at 0%. nvidia-smi polled at 2-second intervals; the first quarter of each load window is discarded as ramp-up. Decode is streamed with ignore_eos; prefill uses fresh random-word prompts at ~98k tokens to defeat the prefix cache. Dense-model comparisons are directional only — that box was serving other traffic at 99% utilization while I sampled it, and will be re-run when it’s quiet. Heat conversion at 3.412 BTU/hr per watt; electricity at $0.15/kWh. Basement temperature is a handheld reading, which is the entire point of item 1.

Follows the $1,000 eBay series — 200 Billion Parameters for $1,000, The $1,000 Box, One Month On, The Junk Drawer Upgrade — and the two 5070 Ti writeup.

TL;DR

Qwen3.8-27B was getting all the hype this weekend — same as Meta’s Glimmer last weekend, and DeepSeek-V4-Flash the weekend before that — so I ran it on the three lab boxes. Those are the Xeon 2x P5000 (Pascal) — the ~$1,000 eBay special — Ultra9 with two RTX 5070 Tis, and the Ryzen box with one RTX 5090. Thirty-two gigs of VRAM on each, just arranged differently: two 180 W 16 GB Pascal GPUs, two 300W 16 GB Consumer Blackwell GPUs, and one 600W 32 GB Blackwell GPU.

If you only look at generation, the upgrade is boring. Pascal does 23.4 tok/s, the 5070 Tis do 51.8, the 5090 does 57.3. Call it 2×, maybe 2.5×. All three are faster than most people read. For “Who caught the Immaculate Reception?” I’d still tell someone to buy the eBay box. That’s the equivalent of walking over to the trailer to pull one page from the drawing set.

Prefill is what actually changed. Unique ~16k-token prompts (so the prefix cache couldn’t cheat) went at 257 tok/s on the Xeon 2x P5000, 2,185 on the 5070 Tis, and 9,204 on the 5090 — about 36 times the Pascal pair.

That’s the number TeraContext cares about. Construction specs are 500 to 2,000 pages, the model sees 100k+ tokens at a time, and a real job is thousands of those calls — takeoff on the whole project manual, not one detail. At 257 tok/s, one 100k window is six and a half minutes and a thousand of them is four and a half days. The 5090 does the thousand in about three hours. I like the cheap box. We just don’t have four days.

The three boxes, one model

The last few posts were about getting 200B-class MoEs running at all. --cpu-moe, pinning 108 GB of experts in RAM, the junk-drawer DDR4 — that was “can I run this.” This one is just: the internet spent the weekend on Qwen3.8, so I put the same weights on the three boxes and timed them.

• Xeon 2x P5000 (Pascal). Dual E5-2698 v4, two $250 Quadro P5000s from 2016 (16 GB each, 32 GB together), the ~$1,000 build. It’s stuck on a CUDA 12.8 llama.cpp image because CUDA 13 dropped Pascal, which means it will never run the vLLM we use on the Blackwell boxes. The weights here are an INT4 GGUF — only Blackwell has native NVFP4, so Pascal doesn’t get the Unsloth NVFP4 file the other two boxes are running. MTP is on. That’s why a 27B here is at 23 tok/s instead of the ~12 we saw in July before MTP.
• Ultra9, two RTX 5070 Tis. The consumer desktop from the boardwalk posts. 16 GB per card, 32 GB together, no NVLink, vLLM tensor-parallel, same Unsloth NVFP4 weights. MTP is off.
• Ryzen, one RTX 5090. The ~$5,000 box from the first post. One 32 GB card, vLLM 0.27.1, 180k context, vision left on because that’s how we actually serve it. MTP is off here too.

The RTX boxes aren’t leaving MTP on the table for fun. Vision, MTP, and a long context window don’t all fit in 32 GB. We kept vision and the long window, so MTP had to go. The Xeon isn’t doing vision, so it can spend that VRAM on MTP. Fair to know when you look at the 23 vs 52 vs 57 decode numbers: the cheap box is the only one drafting.

One request at a time, thinking off. Decode is 400 tokens streamed with ignore_eos, timed from the first output token. Prefill used a fresh random-word prompt each time, because the first C=1 run this week looked amazing until I realized I was measuring prefix cache. Cached prefill comes back in the 8k–22k tok/s range and tells you nothing about a new document.

Decode is 2.2× and 2.5×. Prefill is 8.5× and 36×. Generation is mostly memory bandwidth; prefill is FLOPS and tensor cores, and a 2016 P5000 has basically none of the latter. Seeing it on one model just makes the gap harder to talk around.

“Who caught the Immaculate Reception?”

A normal chat turn is almost all decode. Twenty tokens of question, maybe 150 tokens of answer.

On the Xeon 2x P5000 the prompt is gone in a tenth of a second. The answer is 150 ÷ 23.4, about 6.4 seconds. Same answer on the 5090 is 150 ÷ 57.3, about 2.6 seconds. You can tell them apart if you’re watching the cursor. If you’re just trying to get an answer, both are faster than I read.

That’s the walk to the trailer. You need one detail out of the project manual, you grab the binder, you flip to it, you go back to work. It is not even worth booting the computer for that one page. For a coding assistant, or to ask who caught the Immaculate Reception, the original claim still holds: the $1,000 eBay special is a lot of model for the money. Spending five times as much to talk 2.5× faster is a luxury, not a requirement.

A thousand-page spec is a different job

TeraContext is the thing we actually have to ship. A commercial RFP shows up as a 500–2,000 page specification book and a drawing set. The software classifies pages against a work breakdown structure, bundles them into trade packages, and has to point at the paragraph it used. That is not a trip to the trailer for one detail. That is takeoff on the whole project manual — usually 100,000+ tokens a call, thousands of times on one job — because a spec is a few thousand sections plus addenda plus the “does this also apply to the electrical package” checks.

Almost all of the wait is prefill. The classification that comes out the other side is short. Using the unique-prefill rates above, one 100,000-token call:

Xeon 2x P5000: 100,000 ÷ 257 = 389 seconds = 6 minutes 29 seconds 2× RTX 5070 Ti: 100,000 ÷ 2,185 = 46 seconds RTX 5090: 100,000 ÷ 9,204 = 11 seconds

The generated tokens on that call barely show up. A hundred tokens of “cast-in-place concrete, WBS 3.2” is four seconds on Pascal and under two on the 5090. Something like 95% of the turn is just eating the window.

A thousand calls is a light day on a real set. Some jobs are more.

I sat with that 4-day number for a minute because it sounds like I dropped a zero. 389 seconds times 1,000 is 108 hours. That’s four and a half days of the dual-Xeon box doing nothing but ingest. The 5090 is done in three hours. The 5070 Tis take a long workday, or two if you run 2,000 calls.

You can move one load of spoils with a wheelbarrow. You cannot clean a basement excavation that way and still make the pour. Estimators do not have from Friday’s addendum until the following Tuesday for the GPUs to finish reading.

Two things make this slightly less ugly, and not enough of either. The 100k times assume the 16k rate holds. It doesn’t — longer prompts get slower. At 180k we measured 1,415 tok/s on the 5070 Tis and 3,268 on the 5090. I didn’t run 180k unique on Pascal this time; at the 32k rate of 226 tok/s that window is already thirteen minutes. Prefix cache helps when the next call starts the same way, but TeraContext’s windows walk through the book. They overlap some. Not enough. You still pay for most of the page.

What I think the extra money is for

People ask if the 5090 is “worth it” next to the eBay box. For this model I think that’s the wrong framing. You’re not buying a faster walk to the trailer. You’re buying the difference between flipping one page and taking off the book.

The $1,000 machine is still the right buy if you want to talk to a 27B, or keep running the 200B MoEs that don’t fit anywhere else. MTP already doubled decode on the 27B in July, and that’s only on this box — the RTX machines can’t stack it on top of vision and 180k. I would not spend $4,000 to shave four seconds off “who caught the Immaculate Reception.” The wheelbarrow is the right tool for one load.

The ~$5,000 Ryzen box is what you buy if the prompt is the work. Same 27B, only a little quicker on the way out, 36× quicker on the way in. The 5090 isn’t even that far ahead of the two 5070 Tis on decode (57 vs 52 tok/s), but it’s about 4× on a 16k unique prefill and still 2.3× at 180k. For long documents I’d take the one 32 GB card over the pair of 16 GB cards, and either over the P5000s. There isn’t a llama.cpp flag that closes a 36× compute gap.

If you already have the 5070 Ti desktop and your prompts are tens of thousands of tokens, use it. I wouldn’t buy it for chat, and I wouldn’t make it the only box for a 180k construction window if the 5090 is sitting there.

This doesn’t undo the first post. The Xeon 2x P5000 is still why I can run DeepSeek-V4-Flash at all. That’s a different weekend’s hype and a different reason to own the box. This weekend the question was just Qwen3.8 on the hardware we already have. When they all run the same weights, prefill is the number I actually care about.

So

If you want a local chatbot, get the eBay special. Walk to the trailer, pull the page, go back to work. It will tell you Franco Harris caught the Immaculate Reception in Three Rivers Stadium in 1972 and you can feel clever about the receipt.

If you want takeoff on Division 03 a thousand times before Friday’s bid, it will still be on page 400 on Tuesday. The 5090 is done Monday afternoon. We don’t have until Tuesday.

A follow-on to The $1,000 Box, One Month On: A Model Zoo and Multi-Token Prediction, which followed 200 Billion Parameters for $1,000: Running 4-Bit Quants on eBay Hardware.

Follow-up: Qwen3.8-27B on Three Boxes: 2.5× Decode, 36× Prefill — same 27B on the Xeon 2x P5000, two 5070 Tis, and a 5090. Decode is 2–2.5×. Unique prefill is 36×.

TL;DR

Cleaning out the junk drawer, I found 64 GB of DDR4 I’d paid $50 for before the rampacolypse. The sensible move was listing it on Facebook Marketplace for some astronomical number like $600. Instead I plugged it into the eBay Xeon box from the last two posts, taking it from 128 GB to 192 GB — the first hardware change in the entire series.

Then I handed the box to Claude Opus and told it to go find the wins: pick the quants, run the benchmarks, keep the ones that pay. I stayed out of the way except for a handful of calls where I overrode it. That division of labor turns out to matter to the story, because four of the five quant upgrades made models slower, one by 23%, and the only outright win was an upgrade Opus recommended skipping and I ran anyway. Along the way a piece of folklore in my own notes — that i-quants dequantize slower than K-quants on CPU — failed head-on twice and had to be thrown out.

The useful result is two rules that actually predict what a quant change costs on a CPU-offloaded box, and they are not the same rule:

• Prefill tracks bytes added. +1.1 GB is free. +25 to +48 GB costs 15–20%.
• Decode tracks expert bits-per-weight, not quant format. Fewer bits per weight, less RAM traffic per token, faster decode — whatever the letters in the filename say.

Higher quant ≠ better. On this box it’s usually quality bought with speed, and you should know the exchange rate before you spend 40 GB of download on it.

And the other thing more memory appears to buy — huge context windows — turns out to be mostly theoretical at these prefill rates. Filling DeepSeek’s context to 128K means 35 minutes before the first token appears. Point a coding agent at it with a 64K context and a 20-turn loop takes 6h 14m, against under two minutes for a smaller model on a consumer Blackwell card. I raised zero context windows this round, and the section near the end does that arithmetic in full.

The junk drawer

The box in this series has never had a hardware change. That was sort of the point — two posts about squeezing 200B-class models out of $1,000 of 2016-vintage eBay parts, and all the gains came from software: --cpu-moe, layer offload tuning, split modes, and multi-token prediction. Same dual E5-2698 v4s, same pair of $250 Quadro P5000s, same 128 GB of DDR4.

Then I found four 16 GB sticks in a drawer. Fifty dollars, bought well before DRAM prices went vertical. Current Marketplace comps for the same kit are genuinely absurd — I could have flipped it for something in the neighborhood of $600 and felt clever.

I put it in the server instead.

192 GB total, 188 usable. In practice that lifted the ceiling on how much model I can pin in RAM from about 105 GB to about 165 GB, because these models are mlock‘d — pinned into anonymous memory so the kernel can’t page an expert out to swap mid-generation. That pin is the whole trick from the first post, and its size is the constraint on how good a quant you can run.

So: new headroom, five models sitting at compromise quants chosen when the ceiling was 105 GB. Obvious move, right? Upgrade them all.

How this actually got run

Worth being explicit about the working arrangement, because it shaped the result.

I didn’t do this optimization pass by hand. I gave Claude Opus the box — the repo, the router, shell access, the benchmark scripts — and a standing instruction: figure out which models can move to better quants now that there’s headroom, measure everything, keep what pays and revert what doesn’t. It picked the tiers, probed the files, ran the A/Bs, and wrote the results into the repo’s notes as it went.

My interventions were few and mostly consisted of pushing for more than it wanted to do:

• It proposed a middle tier for DeepSeek; I asked why we weren’t going straight to the top 4-bit tier. (That one blew up spectacularly. Read on.)
• I extended the same top-tier question to two more models.
• It wanted to delete a rejected 144 GB quant; I told it to archive it to a spare array instead.
• It recommended skipping the Nemotron upgrade as pointless. I told it to run the benchmark anyway.
• On hy3, it laid out a speed-versus-quality trade and asked which I wanted. I took the quality side, eyes open, and told it to skip the second perplexity run to save half an hour.

That last category is the one I’d point at for anyone doing this. The model was rigorous about measuring — arguably more rigorous than I would have been, since it caught its own bad reasoning twice — but it was consistently conservative about what was worth measuring. The single biggest throughput gain of the exercise came from overruling a well-argued “don’t bother.”

The obvious move was wrong four times out of five

Here’s what actually happened. Every one of these is a same-build A/B — old quant and new quant measured back to back on the same binary, on a settled box, with the old files kept on disk until the comparison was done.

Four of six swaps made the model slower. One made it dramatically slower. The single upgrade that improved throughput was the one Opus had recommended skipping on the grounds that it barely changed anything.

Let me take the interesting failures in order, because each one broke a different assumption.

Failure 1: the MXFP4 trap (−23%)

DeepSeek-V4-Flash was running at UD-IQ3_XXS — a 3-bit quant — purely because the 4-bit version wouldn’t fit in the old 128 GB pin ceiling. This was the upgrade the RAM was for.

Opus initially queued a middle tier. I asked why we weren’t just going to UD-Q4_K_XL, the top 4-bit tier, since 144 GB now fit comfortably. No good reason surfaced, so we killed the first download and pulled the big one.

Decode fell from 6.94 to 5.33 tok/s. Minus 23%. Prefill fell 17%. The memory pin verified clean, swap was idle, the box wasn’t contended. It was just slower.

The reason is a genuinely counterintuitive property of this particular model, and it’s worth internalizing because more models are shipping this way. DeepSeek-V4-Flash is natively MXFP4 — a 4-bit floating-point format the model was quantization-aware-trained in. So the tiers don’t work the way you assume. A higher tier doesn’t replace MXFP4 with something better; it preserves more of it:

Pascal has no FP4 tensor cores. Every MXFP4 expert matmul on this box is a software dequantization loop. So the “best” quant of this model is the one that hands my nine-year-old GPUs the largest possible pile of arithmetic they have no silicon for. The shipped 3-bit quant is the only tier in the entire lineup with zero MXFP4 in it — which makes it not a compromise but very possibly the optimal choice for this hardware, and I’d been treating it as a placeholder for a year.

On a Blackwell card the ranking inverts completely. Same file, same flag, opposite answer.

Opus wanted to delete the 144 GB after the revert. I told it to archive the files to a spare array instead — if a future llama.cpp build lands a fast Pascal MXFP4 path, the test is a cp away rather than another overnight download.

The cheap lesson, and it was cheap for a reason: the load log printed ftype: MXFP4 MoE while the model was coming up. Opus read that line and quoted it back to me as confirmation the right file had loaded — correct, and completely missing what it implied about throughput on a GPU with no FP4 units. The information that would have cancelled the download arrived before the download finished, and we both walked past it.

Failure 2: folklore that survived because nobody tested it

Both MiniMax and Step3.7 were on IQ4_XS. Having seen the DeepSeek result, I asked whether the same top tier made sense for these two. Opus said yes — and expected a speed win on top of the quality win, citing a rule sitting in my own repo notes: i-quants dequantize slower than K-quants on CPU. These were textbook i-quant → K-quant swaps. Should be free speed.

MiniMax: decode flat (−1.2%), prefill −16%. Step3.7: decode −6.1%, prefill −14.4%.

Tested head-on, twice, the rule lost twice. And when Opus went back to find where it came from, it turned out to rest on two incidental observations from unrelated work — never once a controlled comparison. It had been sitting in my notes for months looking like knowledge because nothing had contradicted it, mostly because nothing had asked it a direct question. I’d written it; the model inherited it and dutifully repeated it right up until the bench said no.

Both models stayed on the new quant anyway — I told it to keep them and delete the old files. That’s a deliberate trade: buying output quality with prefill throughput, with the price now known. The point isn’t that the upgrade was wrong, it’s that we’d have made the same call for the wrong reason and filed a fictitious speed win next to it.

Failure 3: a rule invented, written into the docs, and retracted inside a day

Hy3 is the 295B monster, and the slowest thing in the rack — 21B active parameters per token means the heaviest expert matmul here by a wide margin. It was running Q2_K, a 2-bit quant, and my question was reasonable enough: with 64 GB more RAM, can we do better than 2-bit?

I also asked Opus to measure perplexity on the current quant first, so any quality claim about the replacement had a real baseline instead of vibes.

Step one, Q2_K → IQ3_XXS (+8 GB): decode fell 28%. An 8 GB increase should be nearly free by the prefill rule — and prefill was indeed nearly free, −3.7%. But decode cratered.

Opus had an explanation ready. This was a K-quant → i-quant swap, the reverse direction from MiniMax and Step3.7, and it lost badly. So the folklore wasn’t dead, just directional: i-quant dequantization is genuinely slower on CPU, and this test had isolated it. That went into the repo notes as the finding.

Step two, Q3_K_M — pure K-quant experts, no i-quant kernel involved anywhere: 4.12 tok/s.

IQ3_XXS had measured 4.16.

A dead heat. Whatever slowed hy3 down had nothing to do with quant format, because the format changed completely and the number didn’t move. What both quants share is more bits per expert weight than Q2_K had.

That’s the real mechanism, and it should have been obvious from the first post’s own framing: cpu-moe decode is memory-bandwidth-bound. Every token streams the active experts out of DDR4. Fewer bits per weight means fewer bytes crossing the bus per token means more tokens per second. Format determines quality at a given size; bit-width determines speed. Q2_K is fast here because it is small, full stop.

To its credit, Opus caught this itself and retracted the rule in the repo notes without being prompted — it flagged the contradiction the moment the Q3_K_M number came in. But that’s a save, not a clean play. The rule shouldn’t have been written after one test.

What makes it worth writing up rather than quietly fixing: two different explanations predicted the identical result for the first test. “i-quants are slow” and “more bits are slow” both forecast a decode loss for Q2_K → IQ3_XXS. One datapoint could not tell them apart. The wrong one got picked and recorded as established. It took a second test the two hypotheses disagreed about — a pure K-quant at higher bit-width — to separate them.

When two mechanisms fit your one datapoint, you don’t have a finding. You have a coin flip you haven’t noticed you’re calling.

Hy3 stayed on Q3_K_M, and that one’s on me. Opus laid the trade out straight — quality up, −29% decode and −20% prefill — and asked which I wanted. I took quality on a model I use for hard single-shot questions rather than volume, and told it to skip the second perplexity run to save half an hour of box time. It’s down to about 4.1 tok/s, below reading speed and a genuinely different experience from the 5.8 it used to do. The Q2_K files went to the spare array, restorable in a cp if I decide that was a bad call.

The one that won, which the model told me not to do

Nemotron-3-Super was on Q4_K_M. The Q4_K_XL was only 1.1 GB bigger, and Opus probed the tensor tables before downloading anything: of 80 expert tensors, exactly three differed. Two moved Q8_1 → Q8_0, one moved Q4_K → Q5_K.

Its recommendation was to skip it, and the argument was good. Three tensors out of eighty, essentially no added precision, some prefill cost for the extra bytes — nothing to gain on either axis. On the evidence presented, that’s the right call.

I told it to run the benchmark anyway. Partly because 1.1 GB is a cheap experiment, mostly because by that point three confident predictions had already been wrong and I’d stopped trusting the reasoning ahead of the measurement.

Decode: 8.07 → 8.70 tok/s. +7.8%. Prefill flat. The only throughput win of the entire exercise, from the swap that had been argued down to nothing.

The best read is that the two Q8_1 → Q8_0 tensors hit a better-optimized CPU dequantization path — a kernel-quality difference, not a precision difference. Which means the analysis was fine as far as it went and simply answered a different question than the one that mattered: a tensor-count delta tells you about quality, not about speed. Three tensors can’t add meaningful precision. Three tensors can absolutely land on a faster code path.

A +7.8% gain from three changed tensors is exactly the shape of a measurement artifact, so it got the full treatment before going in the table: re-run the new quant (8.69, 8.70 — reproduced), then put the old quant back and re-benchmark it under the same conditions (8.07, matching its 8.12 from before the swap). That’s the single best reason to keep the old files until the comparison is finished — without them the honest control doesn’t exist and you’re comparing against a number from a different week. Opus did this unprompted, which is the flip side of the conservatism: rigorous once measuring, just reluctant to start.

There’s also a mundane failure mode worth flagging if you automate this. Midway through the Nemotron run, another process on the network hit the router and evicted the model — the box serves one model at a time, so anything else using it can wipe out your benchmark. I spotted the number looked off and told it to retest. A fully unattended loop would have recorded that as data.

The two rules, and why they’re separate

Six measurements, one coherent picture. The costs are independent and you have to price them separately:

Prefill ≈ bytes added. Prefill streams the entire weight set through the CPU to digest your prompt, so it pays the size increase in full and linearly.

Decode ≈ expert bits-per-weight. Decode touches only the active experts per token and is bandwidth-bound, so it tracks bits, not gigabytes and not format. A big quant jump that keeps bits-per-weight similar can leave decode roughly flat (MiniMax, −1.2%) while still costing 16% of prefill. A jump that raises bits-per-weight hits decode hard (hy3, −29%).

Practically, for anyone running MoE models with experts in system RAM:

1. A large quant jump is quality bought with prefill. Budget ~15% and decide if you want it. Nobody’s stopping you — just don’t expect it to be free.
2. A near-same-size tier bump is cheap and might win outright. Nemotron gained 7.8% for 1.1 GB. These are the ones worth trying.
3. Read the expert tensor types, never the tier name. “Q4_K_XL” told us nothing useful about any of these five models. What mattered was inside the file.
4. Keep the old quant until the A/B is finished. Non-negotiable. It’s the only honest control you’ll get.

The trick that would have saved us a 144 GB download

Rule 3 sounds like it requires downloading the thing you’re evaluating, which defeats the purpose. It doesn’t.

GGUF files put their tensor table near the front, and Hugging Face serves HTTP range requests. So you can pull the first ~40 MB of part 2 of a split model, parse the tensor metadata out of it, and read exactly which quantization type every expert tensor uses — in seconds, without moving the other 144 GB.

That’s how the MXFP4 percentages in the DeepSeek table got measured. Retroactively, unfortunately — Opus wrote the probe after the disappointing benchmark, to explain the result. Run first, it would have said “100% MXFP4 experts” and the whole 144 GB detour wouldn’t have happened.

It became standard procedure immediately after, and has already paid for itself twice: it’s what produced the tensor-level detail for every model in this post, and when a promising new 152 GB model came up at the end of the session, the same probe found its architecture isn’t in any released llama.cpp build yet (the upstream PR is still an open draft). That download never started.

What 192 GB actually bought

Not what I expected. Two of five models are genuinely better — Nemotron is faster and slightly higher precision, and MiniMax and Step3.7 are running quants I couldn’t fit before, at a prefill cost I can now quote exactly. DeepSeek is back on the quant it started with, but for the first time I know why that’s the right answer rather than the affordable one. The context windows, as covered above, stayed exactly where they were.

The real return was on the measurement discipline. Every table cell in my notes that we’d been treating as a baseline had quietly drifted with upstream builds — deepseek 7.12 → 6.94, minimax 8.05 → 7.46, step3.7 8.54 → 8.36, nemotron 8.60 → 8.12. Not one was a valid control anymore. Meanwhile a perplexity measurement from a build a month older reproduced to within noise (6.6996 → 6.7031), because perplexity measures weights and speed measures kernels, and only one of those changes when you rebuild.

Box state alone swung one decode number 14% — same model, same config, measured while a previous 137 GB memory pin was still unwinding, then again ninety seconds later. Both readings internally consistent across three runs. “Three tight runs” is not evidence of a clean measurement; it’s evidence of a stable wrong answer.

There was also a stretch spent chasing a load average of 1.3 during a perplexity run, which looked absurd on a 40-core box — I flagged it as obviously broken. It wasn’t: 45 of 46 threads were correctly asleep because that workload is GPU-bound. The suspicious number was the right number. Worth remembering that “this looks wrong” is a hypothesis too, and mine was the one that failed that time.

The context length you can hold but cannot use

There’s a second thing more RAM buys, and it’s the one I’d been most excited about: context. Freeing memory means you can allocate a bigger KV cache, and several of these models advertise 256K or even 1M token windows. The temptation is to crank every ctx-size to its trained maximum because you finally can.

Don’t. Run the arithmetic first.

Prefill — digesting the prompt before a single token comes out — is the slow phase on this box, and unlike decode, multi-token prediction does nothing for it. DeepSeek-V4-Flash prefills at a measured 62.4 tok/s. So:

131,072 tokens ÷ 62.4 tok/s = 2,101 seconds = 35 minutes

Thirty-five minutes of a dual-Xeon box at full tilt before the model emits its first character. The answer itself is the fast part: 500 tokens at 6.94 tok/s is another 72 seconds. Fill DeepSeek’s context to 128K and you wait ~36 minutes for a reply.

Here’s the full table at the prefill rates measured in this post:

Hy3 at its full 256K window is a two-hour wait for the first token. And note the cruel interaction with everything above: the Q3_K_M upgrade cost 20% of prefill, which on a 256K prompt is roughly 25 minutes of pure penalty added to a job that was already unreasonable.

The question that actually settles it

Let me put this in the terms I care about, because I don’t run these models in a vacuum — I run agents. So: just how useful is the increased intelligence I’d get from switching to DeepSeek-V4-Flash for my OpenClaw or Hermes agent with a 64K context, if the turn time became 18½ minutes instead of the 5.7 seconds I get from Qwen3.6-35B-A3B on a single RTX 5090 — 17,500 tok/s prefill and 250 tok/s generation?

That comparison is deliberately not a datacenter card. It’s a 35B MoE with 3B active parameters on one consumer RTX 5090 — fully resident in 32 GB of VRAM, no CPU offload anywhere in the picture. Same 4-bit quantization philosophy as everything else in this series, just small enough to fit.

It is not cheap, mind you. That’s the ~$3,200 card in the ~$5,000 Ryzen box from the original post — five times what the Xeon cost. It’s been sitting there the whole series as the fast-but-expensive option. Hold that thought, it’s the actual decision at the end of this.

The arithmetic:

DeepSeek-V4-Flash on the Xeon box: 65,536 ÷ 62.4 tok/s = 1,050 s = 17m 30s Qwen3.6-35B-A3B on one RTX 5090: 65,536 ÷ 17,500 tok/s = 3.7 s

That’s a 280× gap on prefill alone. Now add a 500-token response. Qwen generates at 250 tok/s on the Blackwell card — 2 seconds. DeepSeek does 6.94 tok/s — 72 seconds, and that’s with every decode optimization from the last two posts already applied.

Note what the decode column does to the argument. Decode is the axis I’ve spent two posts optimizing — MTP, layer offload, split modes, quant selection — and it’s only a 36× gap. Prefill is 280×. The thing I can tune is not the thing that dominates, and on a 64K agent context prefill is 94% of the turn.

Now run an agent, which is the whole point. Agents don’t take one turn; they take dozens, each re-reading an accumulating context of tool output, file contents, and prior reasoning. A modest 20-turn loop at 64K context:

Under two minutes against six and a quarter hours. That is not a slower agent, it’s a different category of object — one you supervise interactively, and one you launch before bed and read about in the morning.

So is DeepSeek-V4-Flash smarter than Qwen3.6-35B-A3B? On a lot of tasks, honestly, yes — 284B total against 35B, 13B active against 3B, and you can feel it on hard reasoning. It does not matter. No plausible intelligence gain survives a 195× turn-time penalty in an agentic loop, because agents convert latency into wall-clock multiplicatively. A slightly worse answer in 6 seconds beats a better answer in 19 minutes every time you’re going to ask 20 more questions after it. The only regime where the big model wins is the one-shot hard question you’re willing to wait on — and notice that’s precisely not an agent workload.

And the mechanism behind that 195× is the same trick this whole series is built on. --cpu-moe is what makes 284B parameters possible on cheap hardware, and it is exactly what makes them unusable in a loop. Experts streaming out of DDR4 is a brilliant answer to “can I run this at all” and a terrible answer to “can I run this forty times in a row.” The 5090 wins here for one reason: it never touches system RAM.

Which brings back the price tag. At ~$5,000 for the Ryzen build against ~$1,000 for the Xeon, this is emphatically not “cheap hardware loses to expensive hardware.” It’s two genuinely different purchases that happen to share a name:

• $1,000 buys capability. 200B-class models you otherwise could not run at all, at conversational speed, one question at a time. Nothing that fits in the 5090’s 32 GB touches DeepSeek-V4-Flash on a hard problem — that model doesn’t fit on that card at any quantization, so this isn’t a speed tradeoff, it’s the difference between having the option and not.
• $5,000 buys throughput. A smaller model that answers in seconds, over and over, without complaint. That is what an agent actually consumes.

5× the money for a model with 1/8th the parameters, and for agentic work it is the correct buy. That sentence should feel wrong and doesn’t, which is the whole lesson: you are not buying intelligence per dollar, you are buying the shape of the latency curve your workload rides on.

Neither box is a mistake. They answer different questions, and the trap is assuming money spent on one buys you the other. I spent $50 expecting to improve my agent stack and improved something else entirely — a fine outcome, just not the one I’d budgeted for.

Model quality is what you shop for when latency is roughly comparable. When latency differs by two orders of magnitude, latency is the only spec that matters — and prefill speed, not decode speed and not parameter count, is what sets it for anything that reads before it writes.

So what is a big context window for, then?

It’s a ceiling, not a workload. The value is that a long conversation or a document-plus-history session doesn’t hit a wall and truncate. You get there incrementally — a few thousand tokens per turn, each turn’s prefill measured in seconds — and llama.cpp caches the prefix, so turn 40 only re-processes what turn 40 added. That prefix cache is what makes local long-context usable at all, and it’s also exactly what an agent defeats: agents rewrite their context, so the cache misses and you pay full prefill again. Accumulating to 100K over an afternoon of chat is completely fine. Arriving at 100K in one paste, or churning it every turn, is not.

The practical rule I’ve landed on: on this box, prompts should be under ~10K tokens for interactive work (about 2½ minutes of prefill on the slow models, tolerable) and long context should be reached by conversation, not by upload. Anything where you genuinely need to dump 100K tokens of fresh material and get an answer — full-repo analysis, a long PDF, bulk document extraction — is the exact workload this hardware is worst at, and it’s worth being honest that a cloud API or a Blackwell card will do it in seconds. Agentic workloads sit squarely in the bad category and belong on the fast GPU regardless of how much smarter the big model is.

This is also why I declined to raise any context windows during this session, despite now having the memory to do it. The KV cache would fit. The wait wouldn’t. More RAM bought better weights, not usable context — and it took doing the division to see that those are separate purchases.

What I’d tell someone handing a box to a model

The autonomy worked, with a shape to it. Opus was better than I would have been at the parts that are tedious and unglamorous: capturing fresh controls instead of reusing stale ones, re-running surprising results, reloading an old quant to get an honest A/B, retracting its own rule when a later test contradicted it, writing all of it down as it went. That’s most of the actual work, and it’s exactly the part humans skip at 11pm.

What it needed from me was appetite. Left alone it converged on the defensible choice — the middle tier, the skipped benchmark, the well-argued “nothing to gain here.” Three of my five interventions were just run it anyway, and one of those produced the only speed win of the entire exercise. The other two were about optionality: archive the rejected quant instead of deleting it, keep the old files until the comparison is done.

Which sounds like a knock and isn’t, quite. A model that only proposes well-justified experiments is a good property when it’s driving your production box unattended. It just means the human’s job shifts from doing the work to deciding which unjustified experiments are cheap enough to run anyway. On a $1,000 box with a spare 1.5 TB array, that bar is low.

The thesis from the first post still holds and got a little sharper. Frontier-scale local inference isn’t a hardware problem, it’s a patience-and-tuning problem — and this round the tuning mostly consisted of discovering that four confident predictions, several of them from rules in my own notes, were wrong. The box serves 48 models today on the same $1,000 of eBay parts, plus $50 of DDR4 I’d forgotten I owned.

Which, at current Marketplace prices, means the upgrade carried an opportunity cost of about $600. I regret nothing.

A follow-on to 200 Billion Parameters for $1,000: Running 4-Bit Quants on eBay Hardware.

Follow-up: The Junk Drawer Upgrade: What 64 GB of Forgotten DDR4 Taught Me About Quantization — $50 of found DDR4, and four of five quant upgrades made the models slower.

TL;DR

When we first wired up the $1,000 Xeon box — dual 2016-vintage E5-2698 v4s, 128 GB of DDR4, and a pair of $250 Quadro P5000s — it ran three big MoE models at faster-than-you-read speeds. Nothing in that hardware has changed. Today the same ~$1000 box serves 44 models, including four 200B-plus giants the original post never mentioned: DeepSeek-V4-Flash (284B), Tencent’s Hunyuan Hy3 (295B), MiniMax-M2.7 (230B), and Step3.7-Flash (198B). The bigger story is multi-token prediction (MTP) — letting a model draft its own next few tokens and verify them in a single pass. On our flagship Qwen3.5-122B it lifted decode from 11 to 17 tok/s (+52%); on dense 27B models it nearly doubled throughput, and a coder model hit 58 tok/s at 96–99% draft acceptance. The catch: MTP only pays when acceptance is high. On a cheap MoE it’s a net slowdown. Measure acceptance, not architecture.

The zoo grew — same box, no new hardware

The original piece made a narrow claim: for about $1,000 of eBay parts you can run a handful of 4-bit, 200B-class MoE models locally, trading concurrency and prefill speed for access. A month of nights-and-weekends later, that same chassis is a router in front of 44 distinct model presets — dense chat models, OCR pipelines, embedding and rerank endpoints, and a shelf of frontier-scale MoEs loaded on demand.

The heavyweights are the headline. All of these decode on the same 40 cores, with their expert tensors parked in system RAM (--cpu-moe) while attention and the KV cache ride the two 16 GB Pascal cards:

Note the pattern the original post predicted and this table confirms: active parameters, not total, set decode speed. Hy3’s 21B-active experts make it the slowest thing in the rack despite not being the largest. DeepSeek-V4-Flash’s 13B-active MLA gets it to 7.4 — and a single upstream patch (fused hyper-connection ops, llama.cpp #25585) bumped it there from 5.2, a free +42% for the cost of a rebuild.

MTP: the model drafts, then checks its own work

The interesting lever since the last post is multi-token prediction, a flavor of speculative decoding. Instead of a separate draft model, the newer Qwen3.6, Hy3, and Qwen3.5-122B GGUFs bundle a tiny “NextN” head inside the file. That head proposes the next few tokens; the full model verifies them in one forward pass. Every accepted guess is a token you got almost for free. Because verification is batched and our cpu-moe models are memory-bandwidth-bound, that extra check costs almost nothing — the win is close to pure.

The numbers on our slow box:

• Qwen3.5-122B, the flagship: 11.1 → 16.9 tok/s (+52%), with 90–96% draft acceptance. This is the single biggest quality-of-life change since launch — the original “about 10 tok/s, faster than you read” is now comfortably past 15.
• Dense 27B models: qwen3.6:27b went 12.7 → 24.0 (~1.9×) at 95.7% accept; the coder variant, 13.2 → 24.6.
• A 3B-active coder MoE: qwopus3.6:35b-coder reaches 58 tok/s with MTP on, riding 96–99% acceptance on actual code.

The catch: acceptance decides, not architecture

MTP is not free money. It wins only when the draft head is usually right — high acceptance amortizes the verification pass. When acceptance is low, you pay for guesses you throw away.

We learned this twice. First, turning MTP on for Gemma4-26B — a cheap 3B-active MoE — was a net slowdown, 45 → 33 tok/s: the model already decodes cheaply, so draft overhead swamped the gain. Second, Hy3 shipped at ~44% acceptance and sat exactly at break-even (4.96 tok/s, tied with the non-MTP control). The fix wasn’t more drafting — it was less. Setting spec-draft-p-min = 0.75, so the head only drafts tokens it’s confident about, pushed acceptance to 83–87% and nudged decode to 5.12. The coder’s 96–99% is why a low-active MoE that “should” lose with MTP wins big instead: on code, the next token is nearly always predictable.

So the rule from a month of benchmarking is blunt: measure acceptance for your workload before trusting MTP. Coding and QA love it; creative writing can actually regress. Architecture is a weak predictor.

Prefill is still the tax

None of this fixes the original post’s honest caveat: prefill on old CPUs is slow, and cpu-moe makes it slower. The 200B-class models still ingest context at 40–80 tok/s, so a long document is a coffee break. MTP helps decode, not prefill — the draft head is tiny and adds nothing to the prompt pass. If your use is single-user, conversational, and decode-bound, the box has quietly gotten 50–90% faster on the same silicon. If you feed it 40K-token prompts all day, the answer is the same as it was: buy newer cores, or wait.

The thesis holds, and got cheaper per token: frontier-scale local inference isn’t a hardware problem. It’s a patience-and-tuning problem — and the tuning keeps paying off.

Two 5070 Tis, one 35B model — Part 3 of 3: the CPU vision sidecar, the OOM that freed memory, and the production decision

TL;DR

Qwen3.6 is multimodal: it carries a BF16 vision transformer (ViT) that, under tensor parallelism, gets replicated on both 16 GB cards and OOMs image requests. Our fix was to evict the ViT from the GPU entirely: a CPU vision sidecar. A small FastAPI proxy intercepts image requests, runs the vision tower on the CPU with plain transformers, and forwards precomputed image_embeds to vLLM, which we tell to skip loading its own ViT via --enable-mm-embeds. It passed a 7/7 exact-parity check (including three identical misreads, proving the CPU tower is functionally the same model), and it did something nothing else could: it made NVFP4 run under tensor parallelism with working images for the first time. It also taught us a nasty lesson: freeing VRAM can crash a hybrid model, because the profiler hands the freed memory to the KV cache and starves the GDN activation scratch. And after all that, we chose not to ship it. Here’s why all of that is true at once.

The problem the ViT creates

Recap from Part 1: under tensor parallelism the vision tower is replicated per card, so an image request inflates memory symmetrically on both GPUs and OOMs them together. Under pipeline parallelism the ViT lives on rank 0 only, which is why PP can serve images and TP can’t. On 16 GB cards already 85%+ full of weights, the resident ViT is the margin between working and dead.

But notice what that implies: the ViT is in the way. It occupies VRAM on the critical card, it blocks TP, and, for a MoE language model, it’s a comparatively small, embarrassingly CPU-friendly piece of compute that runs once per image, not once per token. What if it just… weren’t on the GPU?

The sidecar architecture

vLLM has a feature built for exactly this: --enable-mm-embeds. It lets a client send precomputed multimodal embeddings instead of raw pixels. Pair it with --limit-mm-per-prompt '{"image":0,"video":0}' and vLLM stubs out its vision tower entirely: the ViT weights are never loaded. The GPU gets that memory back.

Something still has to turn pixels into embeddings. That’s the sidecar:

client ──image_url──▶  vision-sidecar (:8006)  ──image_embeds──▶  vLLM (:8005)
                        │  FastAPI proxy
                        │  transformers Qwen3_5MoeVisionModel
                        │  CPU, 16 threads, BF16
                        └─ rewrites the chat request in flight

The proxy is a drop-in shim in front of vLLM. It watches chat completions for image_url content parts; when it sees one, it downloads the image, runs the CPU vision tower, serializes the resulting embedding tensor, and rewrites that content part into an image_embeds part before passing the request along. Text-only requests pass straight through untouched. To the client it’s just an OpenAI-compatible endpoint; to vLLM it’s a text-only model receiving embeddings.

The sidecar loads only the vision weights: it reads the model’s safetensors index and pulls just the model.visual.* shards into a transformers Qwen3_5MoeVisionModel. One subtlety worth calling out: the visual weights live on different shards in different checkpoints (shard 9/10 in the AutoRound build, shard 1/3 in NVFP4), so the loader auto-detects them from the weight index rather than hardcoding a shard number. That one change is what let the same sidecar serve both builds.

The wire-format gotchas

Getting the embeddings into vLLM correctly took two specific fixes that are easy to get wrong and produce opaque errors.

The embedding payload. Each rewritten content part looks like:

{"type": "image_embeds",
 "image_embeds": {"image_embeds": "<b64 torch.save of [n_tokens, 2048] bf16>",
                  "image_grid_thw": "<b64 torch.save of the grid shape>"}}

The grid_thw shape trap. The grid-shape tensor must be a per-item 1-D tensor of shape [3], not [1, 3]. vLLM’s chat_utils._merge_embeds stacks the parts and adds the batch dimension itself; hand it [1, 3] and you get a 400 error (size_per_item should be a 1-D tensor) with nothing pointing you at the extra dimension. Squeeze it to [3] and it works. Hours can disappear into that one.

“Does the embedding ever go back to the GPU?”

A fair question, and the answer is yes, and it clarifies exactly what the sidecar does and doesn’t move off the GPU.

Only the pixel → embedding computation happens on the CPU. The resulting embedding tensor is sent to vLLM, and vLLM copies it onto the GPU (pe_tensor.to(self.device) in the model runner) so it can be spliced into the token stream and attended over by the language model, which is entirely on the GPU. The sidecar doesn’t move the language model’s work to the CPU, which would be catastrophic. It moves only the one-shot vision encode, whose output is a modest [n_tokens, 2048] tensor that rides back to the GPU cheaply. The heavy, per-token transformer stays where it belongs.

Proving it’s the same model

Offloading a model component to a different framework (transformers on CPU vs vLLM’s own ViT on GPU) invites the fear that you’ve subtly changed the model’s behavior. We ran a same-boot A/B parity probe: identical images through the native GPU ViT and through the CPU sidecar, comparing the model’s actual text output.

7/7 exact match. The most convincing evidence wasn’t the correct reads. It was the incorrect ones. On three hard 120-pixel words the model misread the text, and the CPU sidecar produced the exact same three misreads (PERFECT/TPR/CORE) as the native GPU tower. Two independent implementations agreeing on the right answer is reassuring; two implementations making the same mistakes is proof they’re computing the same function. Needle-in-a-haystack retrieval held at 8/8 up to 192K context with concurrent image traffic. The CPU tower is the GPU tower.

The cost: CPU encoding runs about 3.2 seconds per megapixel, capped at ~1.1 MP (MAX_PIXELS) to bound latency. Sixteen encoder threads was the sweet spot on this hybrid P/E-core CPU; 24 threads was slower, because the extra threads land on efficiency cores and drag the batch.

The OOM that freeing memory caused

This is where it got strange. We unloaded the ViT expecting free VRAM and a happier engine. Instead the first 32K-token prefill OOM’d in a GDN kernel (chunk_fwd_o) and killed the engine.

The cause is a subtlety of hybrid models plus how vLLM sizes its cache. When the ViT stopped loading, ~1.5 GiB freed up on the critical card. vLLM’s memory profiler saw that free space and grew the KV cache pool to absorb it — from 196,608 tokens all the way to 481,689. But the GDN linear-attention layers need a chunk of activation scratch at prefill time (chunk_gated_delta_rule / chunk_fwd_o) that the profiler’s steady-state snapshot doesn’t fully account for. The ballooned KV pool ate the headroom that scratch needed, and the first real long prefill ran the card out of memory.

The instinct, “just cap the KV cache with --kv-cache-memory-bytes,” makes it worse on this hybrid model. That flag budgets only the paged KV; the mamba/GDN state cache lands on top of the budget, and we watched a card drop to 21 MiB free and OOM during cudagraph warmup. The only knob that behaves correctly is --gpu-memory-utilization, because it caps total usage and leaves the GDN scratch its implicit slack. Dropping util from 0.92 to 0.85 fixed it: a 267,386-token pool (still 1.36× the original) with ~2.7 GiB genuinely free for GDN scratch.

The same trap, worse, hit NVFP4 under TP: util 0.92 with the ViT unloaded ballooned the pool to 699,347 tokens and left 31 MiB free → cudagraph-capture OOM. Util 0.85 fixed it there too, though the margin stayed thin (176 MiB free on a near-ceiling prefill), which is why 0.82–0.83 would be the safer production number for that build.

Lesson: on a GDN/mamba hybrid, freeing VRAM is not automatically safe. The profiler will spend it on KV cache and starve the linear-attention scratch. Util is the only headroom knob that respects the hybrid.

The payoff: NVFP4 under tensor parallelism

The best result the sidecar bought us: with the ViT off the GPU, we ran the NVFP4/b12x model under TP=2 with working images — something that had never worked before, because the replicated ViT always OOM’d it (Part 1). On the dev552 pin (Part 2), text stayed coherent under sharding, images worked, b12x stayed engaged, and the KV pool hit 489,739 tokens (2.72×) with prefill ~7,700 tok/s and warm decode ~166 tok/s. The vision-tower blocker that made TP a non-starter for the multimodal model was simply gone.

Why we shipped none of it

And then production went back to AutoRound INT4, PP=2, native in-vLLM ViT — the sidecar switched off. That’s not a walk-back; it’s the workload talking.

• The production client sends large images — ~16K vision tokens each. The CPU sidecar caps at ~1.1 MP to keep latency sane, which would downscale the client’s images; uncapped, a single large image takes ~50 s to encode on CPU. For this client, native GPU encoding is both faster and lossless.
• PP runs the ViT natively on rank 0 with no OOM anyway. The problem the sidecar solves — ViT replication — is a TP problem. Production is PP. There’s nothing to fix.
• The workload is prefill-heavy, and PP’s prefill is 1.9× TP’s (Part 1). The sidecar’s whole reason to exist is to unlock TP’s bigger KV cache, which this workload doesn’t need.
• AutoRound rides the latest nightly; NVFP4 is frozen on dev552 until the upstream sharding regression is fixed (Part 2). Staying on AutoRound keeps us current.

So the sidecar, the NVFP4-under-TP result, and the 256K-context experiments it enabled are all validated capabilities on the shelf, not the running config. They’re the answer if the workload changes — text-heavy, high-concurrency, very-long-context, or the day #47365 gets fixed. Building them taught us more about this model’s memory behavior than any amount of reading would have. But the production box runs the boring, proven thing: INT4, pipeline-parallel, vision tower on the GPU where the client’s big images want it, 196,608 tokens of context, both cards busy, clean boots.

The best engineering outcome of a three-week investigation was knowing precisely why to keep doing what we were already doing — and having a validated escape hatch built for the day that stops being true.

A closing note

I should say who actually wrote this thing. The series opened with four developers and a toddler all running out of tokens on the same afternoon. I wasn’t one of them; I still had room left on my Fable plan, the same shiny new model that had drained everyone else’s quota by lunchtime.

So Fable wrote the sidecar. It read through the vLLM internals, wrote the encoder, hit the [1,3]-vs-[3] wall and got past it, and worked out why unloading the ViT was blowing up the GDN memory profiler. I aimed it at the problem, argued with it, and made the calls about what to ship. But the code is the model’s.

Which is a little funny, given the point of the whole project. The reason you build your own inference box is so you’re not at the mercy of someone else’s quota, and the only reason this particular box got built is that, that afternoon, I hadn’t hit mine. The escape hatch got built with the exact thing everyone else had just run out of. My granddaughter, for the record, was still at Zelky’s.

This concludes the 3-part series. Part 1 covered pipeline vs tensor parallelism on no-P2P consumer Blackwell; Part 2 covered bisecting vLLM nightlies and the SM120 FP4 crash fix; Part 3 covered the CPU vision sidecar and the production decision.

Two 5070 Tis, one 35B model — Part 2 of 3: bisecting vLLM and diagnosing SM120

TL;DR

Native FP4 (NVFP4) is the format Blackwell was built for, and Qwen3.6-35B ships an NVFP4 checkpoint. Getting it to run on two consumer RTX 5070 Ti cards took two separate battles. First, a crash: on SM120 the FP4 GEMM throws an “illegal instruction” unless you feed FlashInfer three specific environment variables that select the 120f code path, undocumented tribal knowledge without which the model won’t load. Second, a correctness regression: the :nightly vLLM image produced pure garbage (empty completions and <|box_end|> spam) under our sharded config. We bisected nine daily nightly builds to pin the regression to a 49-commit window, ruled out the obvious suspects, and froze production on the last good pin. Along the way we root-caused a recurring ~17-second engine freeze to GDN kernel JIT recompilation. This is the debugging story.

Why bother with NVFP4 at all

The production model (Part 1) is Intel AutoRound INT4. It works, it’s stable, it stays on the latest nightly. So why chase a second, fussier build?

Because NVFP4 is native to the hardware. Blackwell’s tensor cores have first-class FP4 support; nvidia/Qwen3.6-35B-A3B-NVFP4 is a mixed-precision checkpoint (W4A16 NVFP4 experts + FP8 attention/KV) designed to hit the flashinfer_b12x MoE kernel, a native-FP4 path that on paper should be the fastest way to run this model on this silicon. If it lived up to that, it’d be the better production build. Finding out whether it did meant getting it stable first. That’s where the trouble started.

Battle 1: the SM120 “illegal instruction”

The very first boot didn’t produce wrong answers. It crashed, hard, with an illegal-instruction fault inside the FP4 GEMM. This is the kind of failure that eats an afternoon because it looks like a broken build when it’s actually a missing incantation.

Consumer Blackwell (SM120) comes in two ISA flavors that FlashInfer distinguishes as 120a and 120f. The 120a variant lacks the TMA warp-specialized grouped GEMM that the FP4 path wants; route the FP4 GEMM through it and you get an illegal instruction. The fix is to force FlashInfer onto the 120f code path with three environment variables, all required together:

- FLASHINFER_CUDA_ARCH_LIST=12.0f
- FLASHINFER_FORCE_SM=120f
- FLASHINFER_DISABLE_VERSION_CHECK=1

That last one matters because the pinned FlashInfer build’s version string doesn’t match what vLLM expects, and the check aborts otherwise. None of this is in a getting-started guide; it’s assembled from issue threads and one working reference deployment. Miss any of the three and the model either crashes or silently refuses the native path.

And selecting the native path is itself a choice you have to make explicitly:

- --moe-backend=flashinfer_b12x

Leaving it on auto lets vLLM pick the flashinfer_cutlass FP4 MoE path, which is broken on SM120. Two more traps sit right next to it: do not set VLLM_USE_FLASHINFER_MOE_FP4=1 (routes to the broken cutlass MoE), and do not force --attention-backend flashinfer (breaks SM120 hybrid models, vllm#40677; leave attention on auto, which resolves to FlashInfer anyway). The single most important post-boot check is one grep:

docker logs qwen36-moe-nvfp4-b12x 2>&1 | grep -Ei "FLASHINFER_B12X|NvFp4 MoE"
# WANT: "Using 'FLASHINFER_B12X' NvFp4 MoE backend"

There’s a benign red herring here: a marlin.py warning that says “GPU does not have native support for FP4.” That’s only the dense linear layers falling back to Marlin; the experts, which are the bulk of the compute, still run on b12x. If you see the b12x line, you’re good; if you don’t, don’t ship it.

With the three env vars and the forced backend, the crash was gone and the model loaded. Then it started lying.

Battle 2: the garbage-output regression

The vllm/vllm-openai:nightly image at the time (build 0.23.1rc1.dev672, dated 2026-07-01) loaded cleanly, engaged b12x, passed its health check, and produced garbage: empty completions, or streams of <|box_end|> special-token spam. Same checkpoint, same config, incoherent output.

The catch: this only happened under sharding. The reference single-card deployment on a 32 GB RTX 5090 ran the same model fine. Something in the interaction between a recent vLLM change and our multi-GPU split was corrupting the model’s output.

When you have a build that works and a build that doesn’t and a pile of daily images in between, you stop theorizing and start bisecting.

The bisect

vLLM publishes a nightly image essentially every day. We pulled the run of dailies spanning 2026-06-22 through 2026-07-01, nine builds, and ran the identical sharded NVFP4 config against each, checking a fixed prompt for coherent output (“What is the capital of France?” → must say Paris; a 17×23 arithmetic check; special-token sanity).

The window collapsed fast:

The regression landed between dev552 and dev601: 49 commits. Everything from dev601 onward, including the :nightly tag everyone pulls by default, was broken for our config.

Ruling out the obvious suspects

Forty-nine commits is a lot, and the tempting move is to blame the biggest, scariest change in the range. Two candidates stood out:

• The FlashInfer 0.6.13 bump. Given that all our SM120 pain lives in FlashInfer, a version bump inside the bad window looked like an obvious culprit. We tested around it. It wasn’t the regression.
• A mamba/GDN align-prefix-cache change. Qwen3.6 is a hybrid model with mamba-style GDN state; a change touching prefix-cache alignment for exactly those layers is a plausible way to corrupt sharded output. Also not it. Ruled out.

We could not pin the regression to a single commit in the window before the practical answer took over: we don’t need the exact commit, we need a working pin. dev552 (2026-06-29) is coherent, sharded, and, importantly, newer than the dev245 build the reference 5090 deployment runs. So dev552 became the production pin for NVFP4: the newest build that’s proven-good under our config.

image: vllm/vllm-openai@sha256:7feb2a09304e3b2d38e224a100316e84fe3205faa7605060609e2c02179cbca6
# 0.23.1rc1.dev552 — newest nightly confirmed clean under sharded NVFP4/b12x

Pinning by digest, not by the :nightly tag, is the whole lesson. The upstream regression (tracked as #47365) remains unfixed, which means NVFP4 is frozen on this pin: it can’t take newer nightlies until upstream fixes it. That freeze is a real cost, and it’s one of the reasons the other build (AutoRound, which happily rides the latest nightly) stayed in production. More on that trade-off in Part 3.

The fp8 KV detour

A smaller battle inside the big one: what dtype to use for the KV cache. NVFP4 is FP4 for the weights, but the KV cache is a separate decision, and the checkpoint is calibrated for fp8 KV. The obvious move on Blackwell is to reach for nvfp4 KV to save even more memory, but that path is hard-gated to SM100/B200 in vLLM and raises a requires sm100f error on our SM120 consumer cards. It simply doesn’t exist for us.

We briefly suspected fp8 KV itself as the source of the garbage output and tried disabling it, a dead end. The reference 5090 deployment runs fp8 KV correctly with the same b12x backend, which is proof it’s not the bug. So --kv-cache-dtype=fp8 stayed, matching the working reference and the checkpoint’s calibration. The lesson recurs throughout this project: when a known-good reference deployment exists, match it first and change one thing at a time. Most of the “obvious” culprits are things the reference already proves innocent.

Cudagraphs, enforce-eager, and capture sizing

There was also a performance knob we nearly left money on the table over. Early debugging runs used --enforce-eager to take the compiler and cudagraph capture out of the picture while chasing the garbage-output bug — a reasonable move when you’re isolating a correctness problem. But eager mode is slow, and once the dev552 pin fixed the correctness issue, eager had to go: compilation and cudagraph capture were restored, matching how the reference deployment runs.

Cudagraph capture on SM120 has its own footgun. The reference 5090 deployment sets --max-cudagraph-capture-size=64 to dodge an SM120 cudagraph-replay crash at larger capture sizes. We took a different route: a PIECEWISE capture set explicitly bounded to the sequence sizes we actually run —

- '--compilation-config={"cudagraph_mode":"PIECEWISE","cudagraph_capture_sizes":[1,2,4,8]}'

Because max-num-seqs is 8, capturing only sizes [1,2,4,8] keeps every captured graph far under the size that triggers the replay crash, so the standalone max-cudagraph-capture-size flag isn’t needed (and would in fact conflict — vLLM requires it to equal the max of the capture sizes). Bounding the capture set to the real concurrency ceiling is both safer and lighter on capture time than a heavy long-context sweep.

The 17-second freeze

One more gremlin surfaced repeatedly during benchmarking, and it’s worth documenting because it will fool your measurements. Intermittently, the engine would simply freeze for 8 to 17 seconds mid-serving, then resume as if nothing happened. No error, no OOM.

The trigger is a request-shape transition: a large prefill followed by a small decode-shaped request (or vice versa). Qwen3.6’s GDN linear-attention layers use a CuteDSL kernel that is JIT-compiled per problem shape. When the shape changes sharply, the kernel recompiles — and that recompilation blocks the engine (vllm#47458). On the NVFP4 build the effect was pathological: decode throughput measured cold, right after a big prefill, read ~113 tok/s; the same decode measured warm, after the kernel was cached, read ~171 tok/s.

The practical consequences:

1. Benchmarks must be warm. The first small-shape request after big prefills eats the stall. We switched to best-of-N warm timings to get numbers that reflect steady state rather than a compile.
2. Decode feels erratic to a user hitting the server with mixed request sizes, especially on NVFP4. It’s not a throughput problem, it’s a latency spike from a compiler running on the critical path.

There’s no clean fix short of pre-warming every shape you expect or an upstream change to cache across shapes. Knowing the cause at least turns a terrifying “the engine randomly hangs” into a bounded, explainable “the GDN kernel is recompiling.”

Where NVFP4 landed

After both battles, NVFP4/b12x runs correctly on the two 5070 Ti cards: native FP4 experts, FP8 attention and KV, coherent output, images working. Its prefill under TP came in a touch faster than the INT4 build’s TP (~7,700 vs ~6,800 tok/s), and it earned a genuine first in this project — working image requests under tensor parallelism — but only after we solved the vision-tower problem in Part 3.

The catch is everything around the pin: the flaky TP boot from Part 1, the frozen dev552 nightly, an undocumented three-variable crash fix, and a thin memory margin. NVFP4 is a validated capability, not the shipped default. Understanding exactly why it stayed on the bench — and how moving the vision encoder to the CPU changed what’s even possible — is Part 3.

Next — Part 3, “Evicting the Vision Encoder to the CPU”: how a FastAPI sidecar freed VRAM, unlocked NVFP4 under tensor parallelism, and taught us why “more KV cache” nearly killed the engine.

Two 5070 Tis, one 35B model — Part 1 of 3: the parallelism problem (pipeline vs tensor parallel on consumer Blackwell)

TL;DR

We wanted to serve Qwen3.6-35B-A3B, a 35-billion-parameter mixture-of-experts model, on a desktop with two RTX 5070 Ti cards, 16 GB each. The weights don’t fit on one card, so the model has to be split. The obvious choice, tensor parallelism (TP), turned out to be the wrong one on this hardware: these consumer Blackwell cards have no GPU-to-GPU P2P link, so every layer’s all-reduce crawls over plain PCIe. Pipeline parallelism (PP) splits the model by whole layers, hands off activations only at stage boundaries, and ran prefill 1.9× faster than TP (~13K vs ~6.8K tok/s). TP won on one axis only (a ~2.5× larger KV cache), but its boot was flaky and its decode was no better. PP became our production split. This post covers why each split behaves the way it does on no-P2P consumer silicon, with the benchmark numbers that settled it.

Why run this locally at all

Fable 5 shipped, again, and my phone would not stop buzzing. Three developer friends, three separate texts over the course of an afternoon, all some version of the same complaint: out of tokens. Their Claude Max plans had hit the wall on launch day, which is what happens to anything good the moment everyone piles onto it at once.

Then my three-year-old granddaughter walked into the room, planted herself in the doorway, and announced, with total conviction, that she was out of tokens. For a second I thought the outage had achieved sentience. She meant Zelky’s, the arcade down in Rehoboth Beach, where a fistful of tokens buys a fixed amount of whac-a-mole and a few doomed passes at the impossible-to-win stuffed-animal claw, and then, abruptly, does not. Same phrase, same disappointment, entirely different economy. And the only reason she was at the arcade in the first place: her father, the fourth developer of the day, had run out of Claude tokens too, so an afternoon that was supposed to be spent shipping code became an afternoon of whac-a-mole and the claw instead.

The coincidence stuck with me, because a token is a token: a metered unit of something you want more of than you’re given. When the good model ships and everyone shows up, the shared meter runs dry, for a developer on a Max plan and a three-year-old at a claw machine alike. The way out is to own the machine that mints them. Two consumer GPUs and a decent open-weight 35B model, and the meter is yours: no per-request billing, no launch-day rate limit, no afternoon of “out of tokens” texts. The arcade doesn’t run out when you own the arcade.

That’s the motivation. The rest of this series is what it took to get a genuinely useful open-weight model (not frontier-class, but more than good enough for real work) running on hardware you can buy at Walmart. It wasn’t free either; the currency was three weeks of debugging instead of dollars. Here’s how it went.

The box

The machine is deliberately unglamorous: two NVIDIA RTX 5070 Ti GPUs, Blackwell architecture, compute capability 12.0 (SM120), 16 GB of VRAM each, hanging off a consumer Intel Core Ultra 9 on a consumer motherboard. Not an H100 with NVLink. Not a workstation board with two full-width slots wired for P2P. This is a gaming desktop pressed into service as an inference server, and its interconnect topology shows it.

Look at how the two cards are actually attached. The board has exactly one PCIe 5.0 x16 slot, and that’s where card 0 lives, with a proper fat pipe to the CPU. There’s nowhere on the board to put a second card at anything like that width. So card 1 is hung off an Oculink cable running PCIe 4.0 x4. Do the arithmetic on that link: PCIe 4.0 x4 is ~8 GB/s in each direction, roughly an eighth of the x16 5.0 slot the other card enjoys, and a rounding error next to the ~450 GB/s of an NVLink bridge. The two GPUs aren’t just missing a fast link between them; the second card’s link to the rest of the machine is a drinking straw.

And it gets worse for the thing TP needs most: on this consumer platform the GPUs cannot DMA directly into each other’s memory. There is no P2P path. Anything one card needs from the other doesn’t even get the straw directly. It rides up card 1’s PCIe 4.0 x4 Oculink link to the CPU’s root complex and back down card 0’s link, a full host bounce, on top of the width mismatch.

That combination — no P2P, and an asymmetric x16 / x4 topology — ends up dictating almost every architectural decision in this series. Any strategy that chats constantly between the cards is paying tolls at the slowest link on the board.

The model is Qwen3.6-35B-A3B: 35B total parameters, but a mixture-of-experts design that activates only ~3B per token (the “A3B”). It’s also a hybrid model: it interleaves Gated Delta Net (GDN) linear-attention layers with full-attention layers, which matters enormously later. In INT4 (Intel AutoRound) the weights are ~13.5 GB; in native FP4 (NVFP4) they’re ~22 GB. Either way, one 16 GB card can’t hold the whole thing plus a usable KV cache plus activation scratch. It must be split.

Two ways to split a model

There are two standard ways to shard a transformer across GPUs:

Tensor parallelism (TP) slices every weight matrix across cards: each GPU holds half of every layer’s columns/rows. To compute a single layer you must combine partial results from both cards with an all-reduce. That’s a collective communication on every layer, every forward pass. TP is the darling of datacenter deployments precisely because NVLink makes those all-reduces nearly free.

Pipeline parallelism (PP) slices the model by depth: card 0 holds the first N layers, card 1 holds the rest. Activations cross the PCIe bus exactly once per micro-batch, at the single stage boundary. No per-layer collective. The cost is “pipeline bubble”: while card 1 works on the back half, card 0 could be idle unless you keep multiple micro-batches in flight.

On NVLink hardware, TP usually wins. The received wisdom is “use TP within a node, PP across nodes.” We are inside a node, so TP should win here too, right?

It didn’t, for three reasons that all trace back to that interconnect.

Why TP loses on no-P2P consumer cards

Three separate failure modes stacked up against TP on this box.

1. The all-reduce tax over a x4 host bounce. With no P2P, every per-layer all-reduce is a round trip through host memory, and its throughput is gated by the slowest link in the path, which on this board is card 1’s PCIe 4.0 x4 Oculink straw (~8 GB/s). Qwen3.6 has dozens of layers; at TP=2 you pay two of those collectives (one for attention, one for the MLP) per layer, per token batch, each one squeezing through that x4 pipe and bouncing off the CPU. On NVLink that’s tens of microseconds; over an x4 host bounce it’s an order of magnitude worse, and it lands directly on the prefill critical path. This is the dominant reason TP prefill came in at roughly half PP’s throughput. PP, by contrast, crosses that slow link once per micro-batch at the single stage boundary instead of twice per layer. It’s the one strategy that respects the drinking straw.

2. Marlin’s minimum tile width. The FP4/INT4 dense linear layers fall back to the Marlin GEMM kernel, which has a hard min_thread_n = 64 — the output dimension of a sharded matmul can’t go below 64 columns. TP=2 slices those dense layers in half; several of them drop under Marlin’s floor and the model simply fails to load. PP never slices a matrix — it moves whole layers — so nothing ever falls under the kernel’s minimum. (This bit us specifically on the NVFP4 build, covered in Part 2.)

3. The vision tower gets replicated. Qwen3.6 is multimodal — it ships a BF16 vision transformer (ViT) for image inputs. Under TP the ViT is replicated on every card, so an image request inflates memory symmetrically on both GPUs and OOMs them together. Under PP the ViT runs on rank 0 only. On 16 GB cards already ~85% full of weights, that replicated ViT is the difference between “images work” and “engine dies.” (Part 3 is entirely about attacking this problem from the other side — evicting the ViT from the GPU altogether.)

None of these three is fatal alone. Together they make TP the wrong default on this hardware.

The one thing TP is genuinely better at

TP isn’t strictly worse. It has a real, measurable advantage: KV cache capacity.

Because TP shards the weights and the per-layer activation working set across both cards, each GPU carries a lighter fixed load, and the memory the profiler can hand to the paged KV cache roughly doubles. On the GDN hybrid layers, TP also halves the linear-attention state each card must hold. Concretely, at the same context length:

So if your workload is text-only, high-concurrency, very-long-context, the regime where you’re starved for KV pages and don’t care about prefill latency, TP’s 2.5× bigger cache could be the deciding factor. That’s a real niche. It just isn’t our niche.

The flaky-boot problem

TP had one more strike that doesn’t show up in a throughput table: it wouldn’t boot reliably. Roughly two out of every three cold starts hung silently in NCCL rendezvous — the two ranks never completed their handshake, no error, just a process sitting forever at initialization. A retry loop and --force-recreate got it up eventually, but “eventually, after two silent hangs” is not a property you want in a service that’s supposed to restart: unless-stopped. PP boots cleanly, first try, essentially every time.

We never fully root-caused the rendezvous hangs — on a two-GPU single-host setup they’re most likely a shared-memory/IPC timing issue in the collective bootstrap, aggravated by the no-P2P topology. For a production decision it didn’t matter: unreliable boot was disqualifying on its own.

Reading the throughput numbers

The prefill gap is worth staring at, because it’s bigger than most people expect. ~13K vs ~6.8K tok/s isn’t a rounding difference — it’s TP paying the PCIe all-reduce tax on a workload that is dominated by prefill. When you feed a long prompt, the model runs one big parallel forward over all prompt tokens; that’s exactly where TP’s per-layer collectives pile up and PP’s single boundary hand-off shines.

Decode (token-by-token generation) is a different story — it’s memory-bandwidth-bound per card and the communication is a smaller fraction of the step, so PP and TP land close. But PP was ≥ TP at every concurrency level we measured, so there was no decode-side reason to prefer TP either.

One measurement caveat that cost us real debugging time: the GDN hybrid layers JIT-compile a CuteDSL kernel that recompiles whenever the request shape changes sharply (a big prefill followed by a small decode). The first small request after a large one can eat an ~8–17 second engine stall, and if you’re not careful you’ll measure that stall as “decode throughput” and get ~113 tok/s when the warm number is ~171. We learned to run best-of-N warm timings. We dig into that stall in Part 2, because on the NVFP4 build it got much worse.

The KV-cache paradox: why 16 concurrent sequences fit in a pool that “can’t hold them”

There’s a number that looks alarming until you understand vLLM’s scheduler. At 196,608 tokens of context and 16 concurrent sequences, a naive reading says you need 16 × 196,608 ≈ 3.1M tokens of KV cache to guarantee every sequence its full context. Our pool is 196,608 tokens — one sequence’s worth. By that arithmetic the server should fall over the moment two long requests arrive.

It doesn’t, because vLLM’s V1 scheduler is admit-and-queue, not admit-and-preempt. It doesn’t promise every admitted sequence its maximum context up front; it pages KV in as tokens are actually generated, and when the pool gets tight it queues waiting sequences rather than evicting running ones. Under a deliberate 16-way stress test we watched the scheduler settle at ~7 running, 11 waiting, KV at 88.8% utilization, and zero preemptions. The structural preemption trigger (the one that would thrash) turned out to be unreachable for this configuration. Real prompts don’t all demand full context simultaneously; the pool is sized for the working set, not the theoretical worst case.

This matters for the PP-vs-TP decision because it defuses TP’s one advantage. TP’s 2.5× bigger pool sounds decisive only if you believe you need pool ≈ seqs × context. You don’t. Once the scheduler is doing its job, a 196,608-token pool comfortably serves 16 concurrent sequences at long context — so PP’s smaller pool stops being a liability, and its prefill speed and clean boot carry the decision unopposed.

What “prefill-heavy” actually means for the split

One more piece of context makes the PP choice concrete rather than abstract. Our workload is prefill-heavy: long prompts (documents, long chat histories, images that expand into thousands of vision tokens) relative to the number of tokens generated back. That’s the regime where the forward pass over the prompt dominates wall-clock, and it’s exactly the regime where TP’s per-layer all-reduce tax is most punishing and PP’s single boundary hand-off is cheapest.

If the workload were the opposite — short prompts, long generations, dozens of concurrent streams all starved for KV pages — the calculus would shift toward TP’s bigger pool and away from PP’s prefill edge. We didn’t have that workload. But because we built and validated the TP path anyway (Part 3 shows how), switching is a config change, not a re-architecture, if the traffic ever inverts.

Why this ordering of evidence matters

The takeaway isn’t “PP good, TP bad.” It’s that the right parallelism strategy is a property of your interconnect and your workload, not a universal ranking. On an NVLink DGX, TP=2 for this model would likely win outright. On two consumer cards with no P2P, serving a prefill-heavy multimodal workload, PP wins on throughput, boots reliably, and keeps the vision tower on a single card. TP keeps exactly one trophy, KV capacity, that we didn’t need.

So production runs PP=2: 196,608-token context, the vision tower native on rank 0, both cards ~85–92% utilized, clean boots. That’s the split.

But settling PP vs TP was the easy half. The hard half was getting the native-FP4 build to produce correct text at all — which meant bisecting nine days of vLLM nightlies to find the one that didn’t output garbage, and diagnosing an SM120 kernel crash that only exists on consumer Blackwell. That’s Part 2.

Next — Part 2, “The Nightly From Hell”: bisecting vLLM to find a working NVFP4 build, and the SM120 FP4 crash nobody documents.

Follow-up: The $1,000 Box, One Month On: A Model Zoo and Multi-Token Prediction — the same box now serves 44 models, and multi-token prediction lifted the flagship 122B decode 52%.

TL;DR You can run frontier-size mixture-of-experts (MoE) models—such as Qwen3.5-122B, MiniMax-M2.7 (~229B), and DeepSeek-V4-Flash (284B)—on a modest GPU backed by standard system RAM using llama.cpp’s --cpu-moe flag. This works because these models only activate a fraction of their parameters per token (around 10–13B), allowing the massive “expert” weights to live in cheap system RAM while the GPU handles attention and the KV cache.

While a $12.5K RTX PRO 6000 setup runs Qwen3.5-122B at a blistering 128 tokens per second (tok/s) for a single user—and an aggregate 780 tok/s for a concurrency of 8 users—a $1,000 eBay-scavenged Xeon workstation with 2016-era Pascal cards runs the same model at ~10 tok/s. It’s slower, yes, but still faster than most people read—and entirely usable for a single person. You sacrifice raw concurrency, not capability.

The prevailing advice for running massive MoE models locally usually demands a massive budget: drop $25,000 on two RTX PRO 6000 Blackwells, buy two DGX Sparks (which run about $4,699 each), or pick up two Strix Halos with 128GB of unified memory (priced around $3,999 each). The logic assumes that because 4-bit model weights take up 70–160GB, you need a matching mountain of ultra-fast VRAM.

You don’t.

Over the past two weeks, I’ve been running Qwen3.5-122B-A10B, MiniMax-M2.7 (~229B), and DeepSeek-V4-Flash (284B) on setups that look nothing like a datacenter. One is a machine with a single RTX 5090, and the other is a ~$1,000 workstation built from eBay parts featuring a pair of $250 Quadro P5000s from 2016. The secret lies in a llama.cpp feature called --cpu-moe, which exploits a massive architectural asymmetry that standard hardware advice ignores.

The Fact That Changes the Math

Running a dense 122B model on a CPU would be agonizing. Every generated token requires touching all 122 billion parameters. Because CPU memory bandwidth (a few hundred GB/s on DDR4/DDR5) is an order of magnitude slower than a GPU, you would wait seconds for a single token.

But new MoE models aren’t dense. Consider the active parameters:

• Qwen3.5-122B-A10B: 122B total, 10B active per token.
• MiniMax-M2.7: 229B total, ~10B active per token.
• DeepSeek-V4-Flash: 284B total, 13B active per token.

This asymmetry changes everything. The bulk of the weights—tens of gigabytes of “experts”—sit idle for any given token. Using the --cpu-moe (or -cmoe) flag stores these mostly dormant experts in affordable system RAM. Only the components that touch every token—the attention layers, dense projections, and KV cache—remain on the GPU.

The result is a clean division of labor: the modest GPU handles the compute-intensive attention math, while cheap system RAM holds the massive pile of sleeping experts.

The Two Budget Boxes

• The Ryzen Box: A more conventional build pairing an RTX 5090 (32GB, Blackwell) with an AMD Ryzen 9 9950X (16 cores / 32 threads) and 192GB of DDR5 RAM. The total build costs about $5,000, which includes the ~$3,200 GPU.
• The Xeon Box: Built purely from older eBay parts, featuring dual Intel Xeon E5-2698 v4 CPUs (2016 Broadwell, 40 physical cores total), 128GB of DDR4 RAM, and two Quadro P5000s (16GB VRAM each). The GPUs were $250 each, the CPUs were $100 for the pair, and the chassis/board/RAM cost $425. Total cost: ~$1,000 for 32GB of aggregate VRAM and enough system memory to host a 108GB model.

(Note: These prices are roughly a year old, sourced before recent DRAM spikes. Replicating the $1,000 box today might cost closer to $1,400–$1,800 due to memory pricing.)

The Performance Numbers

Here is the single-stream decode throughput for Qwen3.5-122B-A10B, comparing our budget boxes against a “money-no-object” $12.5K RTX PRO 6000 reference machine:

The $12.5K setup (a $10K card and a $2.5K box) is instantly responsive because it holds all 73GB of quantized weights in VRAM. But a $1,000 box with nine-year-old GPUs running a 122-billion-parameter model at 10 tokens per second is highly usable for interactive chat or coding assistance.

The Ryzen box is roughly twice as fast as the Xeon for two reasons—and the first one is easy to get backwards. With --cpu-moe, the experts that dominate decode are streamed from system RAM, not from the GPU. The Ryzen’s DDR5 is considerably faster than the Xeon’s 2016-era DDR4, so the host memory feeding those experts is the real difference. (The 5090’s own GDDR7 VRAM is faster still, but the offloaded experts never live there—only the attention layers and KV cache do.) Second, the modern 5090 chews through the attention math far quicker than a pair of Pascal P5000s.

The Trade-offs of CPU Offloading

This approach isn’t free. Here are the very real bottlenecks:

• Decode is CPU-bandwidth-bound: The speed limit (10–23 tok/s) is dictated by how fast your RAM feeds the active experts to the CPU cores. Relying on hyperthreading can actually hurt performance; sticking to physical cores yields better results.
• Prefill is compute-bound and agonizingly slow: Digesting a long prompt requires raw FLOPS and TOPS, which old CPUs severely lack.
• Context is surprisingly cheap: Because the massive experts live in RAM, your GPU effortlessly handles the KV cache. The RTX 5090 fits Qwen3.5-122B’s entire 256K context into just 12GB of VRAM.
• You must pin the experts in RAM: Using --no-mmap and --mlock together prevents the kernel from paging files out to swap memory mid-generation.

The Asymmetry: Prefill vs. Generation

An LLM request has two phases with entirely different bottlenecks. Generation is memory-bandwidth-bound, but prefill (digesting the prompt) is compute-bound.

While the dual Xeon’s 150 GB/s memory bandwidth is about 10% of the RTX PRO 6000’s 1.8 TB/s, the compute gap is astronomical. The Blackwell card delivers 125 TFLOPS of FP32 and 4,000 AI TOPS via tensor cores, whereas the 2016 Xeons manage only ~2.8 TFLOPS of FP32 with zero tensor cores.

The rule of thumb: decode speed barely moves with prompt length, but prefill cost explodes. A 44.5K token prefill for DeepSeek-V4-Flash took roughly 10 minutes on the Ryzen box. Extrapolating a 256K-token prompt implies waiting over an hour just for the first token. Pick your hardware by your expected prompt length, not just your model size.

Concurrency: What the $12.5K Box Actually Buys

When serving Qwen3.5-122B via vLLM on the RTX PRO 6000, a single user gets 128 tok/s, but pushing a concurrency of 8 users yields a massive 780 tok/s aggregate throughput.

CPU-offloading cannot match this. On the 5090, pushing multiple requests causes aggregate throughput to rise sub-linearly, while the per-request rate collapses from 23 tok/s to roughly 8.5 tok/s as the CPU experts saturate. If you need to serve a team, buy the big card.

Important Caveats & Setup

• Old GPUs mean old CUDA: The P5000s are Pascal architecture, meaning they are pinned to a CUDA-12.8 llama.cpp image since CUDA 13 dropped Pascal support.
• DeepSeek-V4-Flash: This architecture requires a community fork to run properly, so verified numbers are only available for the Ryzen box (hitting 11–12 tok/s).
• Heat and Power Draw: Old servers are power hungry. The small server room holding the boxes for these tests rose to 83°F when I started running these evals—just a bit too warm for the office. Expect about 800W of power draw on the dual Xeons, and pushing 1000W+ on the RTX 5090 setup.

The Flags You Need:

• --cpu-moe (or --n-cpu-moe N to shift layers manually).
• --no-mmap and --mlock to pin weights in anonymous RAM. On the Ryzen box, MiniMax’s 131GB pins cleanly into the 192GB of RAM; on the Xeon, an IQ4_XS quant’s ~108GB pin leaves about ~20GB free on the 128GB box.
• --flash-attn on alongside a quantized KV cache (like q8_0) to save VRAM.
• --threads set strictly to physical cores (plus --numa distribute for dual-socket setups).
• --parallel 1 to dedicate the entire context window to a single user.

Conclusion

This was never about speed; it’s about access. You are giving up the headroom for concurrency and the instant first token. But you are gaining the ability to run 122-billion, 229-billion, or 284-billion-parameter models on hardware that costs as much as a used laptop.

Last month I published benchmark results comparing five LLMs on LegalBench, a suite of 161 legal reasoning tasks. The 27B Qwen3.5 model won at 0.7936, beating a 120B reasoning model by 6 points. The headline was that bigger isn’t better for legal work.

Since then, two things happened. First, I added two more models to the lineup: Qwen3.6-35B (which dropped yesterday) and Qwen3.5-122B (which I simply didn’t evaluate in round one). Second, I found a bug in my benchmarking code that was quietly suppressing scores on one category of tasks. Fixing it changes the leaderboard.

The Additional Models

Both are MoE architectures from the Qwen team:

Both AWQ 4-bit quantizations were produced by cyankiwi. The 3.6-35B ran on my RTX 5090; the 122B ran on an RTX 6000 Pro. Both served via vLLM in no-think mode, same prompts as the original benchmark.

The Bug I Missed

When I pulled the raw outputs for the MAUD category — 34 tasks on M&A agreement interpretation that use A/B/C/D/E multiple-choice — I noticed something weird. Qwen3.6-35B had scored 0.012 on maud_fiduciary_exception_board_determination_trigger_(no_shop). That’s below random chance for a binary question.

A look at the generations explained it: the model was answering "Option B" while the gold label was "B". My extract_answer() function was returning the full string "Option B", which never matched "B" in the grader.

Worse, on some tasks the model answered "Yes" when the question was A/B multiple choice. The “disproportionate impact modifier” prompts read like yes/no questions, and the model took the bait.

This was present in every model’s results to varying degrees:

The 122B got clean letters on everything — the issue was specific to how smaller models handled the MAUD prompt format. Still, a benchmark bug is a benchmark bug.

The Fix

Two changes to run_legalbench.py:

1. Output extraction — added a regex to strip "Option X" prefix: ^Option\s+([A-Z])\b → \1
2. System prompt — added an explicit instruction: “If the question offers lettered answer choices (A, B, C, …), reply with ONLY the letter — never ‘Yes’ or ‘No’, never ‘Option X’, just the letter.”

I re-ran the 20 problematic MAUD tasks for Qwen3.6-35B with both fixes in place. The results were dramatic:

Every one of the 20 tasks improved. No regressions. Qwen3.6-35B’s overall score went from 0.7483 to 0.7982 — a +5.0 point jump from an extraction fix alone.

I didn’t re-run the fix on the other models. Their rankings in the original post stand, but be aware that Nemotron and the older 35B are underreported. If I re-ran Nemotron with the fix, I’d expect it to gain 5-8 points and climb out of last place.

Updated Leaderboard

The top three models are separated by less than one point. The 122B edges out the 3.6-35B by 0.0008 — statistical noise.

What the 122B Buys You

The 122B has 3.5x more total parameters than the 3.6-35B and runs with 3.3x more active parameters per token. For a one-point gain over the 3.6-35B, is it worth it?

Looking at head-to-head on the 34 MAUD tasks (where the 122B should theoretically benefit most from its extra capacity):

Essentially a tie. The 122B wins on tasks that require memorized legal domain knowledge (accuracy_of_target_capitalization_rw: 0.755 vs 0.399). The 3.6-35B wins where the MAUD fix saved it (fiduciary_exception_board_determination_trigger: 0.964 vs 0.494).

Outside MAUD, both models perform similarly on contract NLI, CUAD clause detection, and privacy policy tasks — in the 0.90s range for most of them.

Verdict: The 122B gives you minimal gains — like the 3rd decimal point. It takes up ~3x the memory and runs at about half the speed. The real “gain” was that it followed instructions and answered without the word “Option” prefix. A better system prompt fixed that on the 3.6-35B. So the original verdict stands: moving from smaller models that fit on consumer GPU cards like the 5090 to workstation-class models did not offer a noticeable improvement on this benchmark.

Qwen3.6-35B vs Qwen3.5-35B

The most interesting comparison is between the two 35B MoE models. Same parameter count, same active params, same quantization. Just a generation apart:

A clean 3.7-point improvement at fixed parameter count. That’s the “raw model quality” delta between 3.5 and 3.6 — separate from any quantization or architectural choice.

Does the Original Blog’s Conclusion Still Hold?

The original post argued that smaller, well-quantized local models can beat a 120B reasoning model on legal work. That conclusion is stronger now, not weaker:

• The 27B Qwen3.5 (dense, 16GB VRAM) still beats gpt-oss-120b by 6 points.
• The 3.6-35B (MoE, 20GB VRAM) beats gpt-oss-120b by 7 points.
• Even the 9B (single GPU) beats gpt-oss-120b by 3 points.

The 122B scoring 0.7990 is notable — it’s the first local model to cross 0.79 — but it’s not enough to change the fundamental story. Parameter count continues to be a bad predictor of legal reasoning ability relative to model generation and training data.

And the MAUD bug is a reminder: benchmarks measure your whole pipeline, not just the model. A small string in an extraction function can cost 5 points.

What’s Next

The most interesting finding here is the generational jump from Qwen3.5-35B to Qwen3.6-35B: +3.7 points at fixed parameter count, active parameter count, and quantization. That’s a clean measurement of how much the 3.5 → 3.6 update is worth on legal reasoning.

And Qwen3.6-35B dropped yesterday. There’s no 3.6-122B yet, only the 3.5-122B I tested here. If the same 3.7-point generational improvement carries over to the larger MoE, the eventual Qwen3.6-122B could push past 0.83 on this benchmark. I’ll re-run as soon as it’s released.

Zooming out: on this legal benchmark, local Qwen models are consistently strong against other local open-weight options. The 27B, 9B, 35B, new 3.6-35B, and 122B all outperform gpt-oss-120b. That’s not a knock on OpenAI’s open-weight model — it’s a real legal benchmark, and these Qwen models are very good at it.

Note: “We” throughout this post refers to Jim Smith working alongside Claude Code.

This morning my 87-year-old mother texted me that TJ Maxx had tried to charge her 6% sales tax on a clothing purchase in Pennsylvania. Clothing isn’t taxed in PA. She caught it at the register and saved herself the 6%. A few hours later, sitting in front of some vLLM benchmarks, I caught something that rhymed: an 8% loss in token generation speed that everyone on the moving nightly tag has been quietly paying. Same instinct, different register tape.

TL;DR

We noticed our Qwen3.5-35B-A3B serving container on an RTX PRO 6000 Blackwell Max-Q was decoding at 183 tok/s while a sibling container on an RTX 5090 hit 225 tok/s. We chalked it up to the 5090’s higher memory bandwidth — until a controlled test exposed the real story: the regression wasn’t the GPU, it was the vLLM nightly image. Running the exact same configuration on the PRO 6000 with a pinned mar23 nightly bumped decode from 183 → 198.6 tok/s, an 8% jump from changing only the image tag.

We bisected through 315 vLLM commits between the two image builds, narrowed to three suspects, and found the culprit: PR #38152, an 8-line revert of dual-stream execution for Qwen3 and Qwen3.5 input projections. The PR is unusually candid — it deliberately gave back hot-path decode throughput to fix a 4x cold-compile-time regression, with a TODO to re-enable once PyTorch 2.11 and #38123 land.

Three options if you’re hit by this: pin the older image, overlay the file to restore the parallel-stream branch, or wait for the proper fix. This post walks through the bisection and the tradeoff.

The setup

We run two vLLM containers side by side on a workstation with two Blackwell GPUs: an RTX PRO 6000 Blackwell Max-Q (97 GB, SM 12.0) and an RTX 5090 (32 GB, same SM). Both serve the same 35B-parameter hybrid MoE model, cyankiwi/Qwen3.5-35B-A3B-AWQ-4bit, through the vllm/vllm-openai:cu130-nightly image. One container stays pinned to a known-good digest; the other rides the moving nightly tag so we can test patches against the latest Triton, FlashInfer, and vLLM changes.

The pinned sibling on the 5090 has been humming along at around 225 tokens per second of single-stream decode. The latest-nightly container on the PRO 6000 was posting 183 tok/s. We had been telling ourselves this was just the 5090’s higher memory bandwidth showing up — decode is memory-bound, and the 5090’s ~1.8 TB/s beats the PRO 6000 Max-Q’s ~1.6 TB/s. A ~13% gap felt about right.

It wasn’t right. The difference wasn’t the GPU. It was the nightly.

The apples-to-apples test

The trick was to break the comparison into two steps.

Step 1: move the sibling’s exact configuration to the PRO 6000. We copied the sibling’s docker-compose verbatim, changed nothing except CUDA_VISIBLE_DEVICES=0 and the port, and pointed the mount at the same pinned digest (vllm/vllm-openai@sha256:923cbdaf..., which maps to cu130-nightly-mar23). Single-stream decode on the PRO 6000 jumped from 183 to 198.6 tok/s. That closed most of the gap to the 5090’s 225 tok/s, which is now just the memory-bandwidth story we had originally told ourselves, at a believable ~12%.

Step 2: change only the image, not the configuration. Same container, same GPU, same PR #37700 chunk_o overlay, same --gpu-memory-utilization 0.5, same --max-num-seqs 8. Only the image tag changed: mar23 → latest cu130-nightly.

Roughly 8% slower on single-stream decode, widening at higher concurrency. Not a config artifact. Not a hardware story. The nightly got slower.

Narrowing the window

Image versions revealed two vLLM commits at the endpoints:

• mar23: vllm 0.18.1rc1.dev32+g1f0d21064
• current: vllm 0.18.2rc1.dev54+g73f48ce55

Torch and Triton were identical (2.10.0+cu130 and 3.6.0 respectively). FlashInfer moved 0.6.6 → 0.6.7, but its kernels are not on the GDN decode path for this model. Between those two vLLM commits: 315 changes.

We filtered git log down to paths that actually touch the Qwen3.5-35B-A3B-AWQ decode hot loop — fused MoE, FLA kernels, the Qwen3.5/Qwen3-Next model files, the GPU model runner, and the sampler. That dropped the candidate list to roughly 80 commits. Three stood out immediately:

• a8eab8f30 “Extract GatedDeltaNetAttention into shared layer for Qwen3Next and Qwen3.5” — a 2,000-line refactor.
• b779eb336 “Sync upstream BT=chunk_size fix for GDN chunk_fwd_kernel_o, simplify warmup to single pass” — touches the exact kernel we’d been trying to tune.
• 9704a5c31 “Disable dual stream execution of input projection for Qwen3.”

The last one was an 8-line diff in vllm/model_executor/models/qwen3_5.py. That was the culprit.

The culprit: PR #38152

Before the change, the GDN block’s input projection looked like this:

mixed_qkvz, ba = torch.ops.vllm.gdn_in_proj(
    hidden_states,
    sum(self.in_proj_qkvz.output_sizes) // self.tp_size,
    sum(self.in_proj_ba.output_sizes) // self.tp_size,
    self.prefix,
)

After:

mixed_qkvz, _ = self.in_proj_qkvz(hidden_states)
ba, _ = self.in_proj_ba(hidden_states)

The removed custom op wasn’t just a convenience wrapper. It dispatched the two projection GEMMs on two parallel CUDA streams, overlapping them. The replacement runs them sequentially on the default stream. Every GDN layer, every decode step, pays twice the launch latency and loses the overlap.

PR #38152’s own description is unusually candid about the reason:

Currently dual stream execution requires custom ops that pass the layer_name as a string. This will regress cold compile times by ~4x. So this PR temporarily reverts dual stream optimization in Qwen3 and Qwen3.5 models.

TODO: Re-enable dual stream after #38123 and upgrade to Pytorch 2.11.

So this wasn’t a correctness fix or a cleanup. It was a deliberate runtime-perf regression accepted to fix a compile-time regression. That is a reasonable call when the alternative is four-times-longer cold starts in CI — but the tradeoff lands on every Qwen3/Qwen3.5 user served from a nightly between the revert and whatever future torch 2.11 + #38123 combo re-enables the optimization.

Why it matters more on this model than you’d think

Qwen3.5-35B-A3B is a hybrid. Not every layer is GDN; many are vanilla attention over an MoE. But the GDN blocks are on the critical path of every decode step, and the input projection runs once per block per token. Removing the parallel-stream dispatch means dozens of additional serial waits per generated token. At 183 tok/s versus 198 tok/s, a decode step is roughly 5.0 ms versus 5.5 ms — the 0.5 ms delta is entirely credible as stream-serialization overhead across a hybrid model’s GDN layers.

The other two suspect commits probably aren’t innocent — b779eb336’s “simplify warmup to single pass” could be choosing worse autotuner configs, and the GDN refactor might have introduced a small per-step Python cost — but they are orders of magnitude smaller contributors than the dual-stream revert.

What to do about it

Three options, cheapest first.

Pin the image. If you aren’t chasing a specific new feature, vllm/vllm-openai@sha256:923cbdaf… (cu130-nightly-mar23) gives the faster decode path today. The sibling container already does this and it’s why it’s faster.

Overlay the revert-of-the-revert. vLLM’s model files are pure Python. A five-line file overlay that restores the torch.ops.vllm.gdn_in_proj branch in qwen3_5.py (and its qwen3_next.py twin) gives a recent nightly the old speed back, at the cost of the compile-time regression that motivated #38152 — perfectly fine for a long-running serving deployment, painful for CI.

Wait. #38152’s TODO is real. Once vLLM’s plumbing (#38123) and PyTorch 2.11’s compile infrastructure land, dual-stream execution should come back without the compile-time penalty. That’s the clean fix and the one you want if you don’t need the speed back this week.

Takeaways

Two lessons worth writing down.

First, nightly “slower than it was last month” is not always a phantom. Docker image tags are moving targets and two builds can legitimately diverge by 8% on the exact same GPU, driver, and command line. Pin something you trust and diff against it.

Second, performance regressions often look like tradeoffs, not bugs. #38152 is honest about what it is — a cold-start fix that gives back some hot-path throughput. The signal isn’t “somebody broke decode.” The signal is “somebody accepted a tax on decode to pay a bigger bill elsewhere.” Finding these requires reading the PR body, not just the diff.

The code is the same. The speed is different. The difference is a decision.