infermaxx: can stock vLLM replace my patched deployment? A single-RTX-3090 field note

Follows on from Putting an RTX 3090 in an HP ProLiant ML350 Gen9. Living document; numbers update as the campaign runs. Last updated 2026-08-13.

In the last post I got a second-hand RTX 3090 running in a 2016-era ProLiant ML350 Gen9, capped at 275 W because that’s what the PSU budget allows. This post is about what it was for.

The 3090-in-a-ProLiant is a research box. My actual workhorse is a dual-3090 machine that serves some production traffic, and it runs a community fork of vLLM carrying 119 patches, 45 of which apply at boot. It’s the kind of setup where every upstream release is a merge project, so you quietly stop upgrading. The question I wanted answered on hardware I could freely abuse: does current stock vLLM make the patch set unnecessary? (This post gets as far as “very probably, pending the dual-card test”.) Plus the practical questions behind it: what’s the fastest way to serve Qwen3.6-27B on one 24 GB card, how many agent sessions fit, and what do you give up to get there.

Over about two days the research box ran an experiment campaign I called infermaxx, driven by an agent working in half-hour ticks under three rules that mattered more than any individual result: register every prediction and its refutation line before running anything, log “no movement” rather than inventing a number, and record the clocks during each run, because the governor lies.

§Executive summary

The headline numbers, all measured on the same card at 275 W:

And the answers to the questions the box exists to answer:

1. Retiring the patch set looks plausible, on a single card. Stock vLLM 0.27.1 beat the fully-patched reference configuration’s published numbers (their 55.0/70.5 with 27 patches; 64.98/71.31 with zero). The narrative win is clear; the ~1% code edge is within cross-rig noise. Two caveats before I over-conclude: the reference’s own no-patches control (54.7/68.2) was already about equal to their patched config, so the patches were roughly neutral by their own data. And my production box is dual-3090 with tensor parallelism, which a single-card result doesn’t cover. The dual-card validation is the outstanding test, not a formality.
2. Concurrency works on vLLM and is disqualified on the llama.cpp fork. Not on speed, though it anti-scales there too, but on correctness: under concurrent sessions the fork leaked one conversation’s contents into another’s answers. vLLM passed the same trap-laden test clean at 4 and 8 sessions.
3. The real constraint is memory, not speed. 18 GB of weights leave about 2.4 GB for conversation state: 65,536 tokens shared across every concurrent session, and nearly half my measured agent sessions peak above that on their own.
4. TurboQuant 3-bit KV breaks that constraint: 163,840 tokens (2.50x) for about 3% of decode speed and no measurable quality change on my gate. But it does not compose with speculative decoding. That combination produced the fastest apparent throughput of the entire campaign, 102.66 t/s, and every word of it was garbage. Only the quality gate knew.
5. The most dangerous bugs were in my own measuring instruments. Twenty defects in the test tooling over the campaign, against a handful of real system defects (the fork’s cross-session leak, its output-changing speculation, the TurboQuant-plus-MTP babble). Every system defect only became visible once the instrument in front of it was fixed.

Caveats that belong up here rather than buried: nothing in this post has served real traffic; TurboQuant at 8 streams sits at about 93% of the fp8 aggregate; and the dual-3090 production test, the point of all this, hasn’t run. The rest of the post is the chronological version. Twelve of my own claims got retracted along the way, and the retractions are the most instructive part.

§Day 0: chartering against a number that turned out to be wrong

The campaign chartered against a published bar: a well-known single-3090 recipe repo for this model (the club-3090 project) reporting “~63/69 TPS”. My goal said beat it, don’t just reproduce it.

An adversarial audit, run before the experiment DAG started, found the first problem: “63/69 TPS at 200K” is not a 200K-context number. Their benchmark uses ~15-token prompts; 200K is configured capacity, not processed context. That became retraction 7 of an eventual twelve, and it set the campaign’s recurring theme: a number without its measurement conditions isn’t a number. I ended up calling this failure mode regime collapse: collapsing engine, device, checkpoint, context and concurrency into one portable-looking figure.

The CPU baseline made the motivation concrete: ~11 tokens/second decode, and a p95 prefill of 507 seconds. Eight and a half minutes before the first token. My agent loops were unusable on it.

§The llama.cpp era: fast, then disqualified

First GPU attempt: the repo’s pinned CUDA-13 container against my driver. It printed CPU: using device CPU, then planned 19,794 MiB of GPU allocations anyway, and segfaulted. The crash was lucky. Had it quietly served on CPU, every later “GPU” number would have inherited a CPU denominator. Every GPU runner since then asserts CUDA liveness before measuring anything.

Building the ik_llama fork from source (with the concurrency fix #2260 provably in its ancestry) produced the first real results, on the 35B-A3B MoE variant. I’m being deliberate about naming the model, because conflating models is the campaign’s named failure mode. The MoE numbers: 41.2x the CPU at my working session depth, and a clean endurance run to 51,802 accumulated tokens with zero VRAM creep.

Then two levers died in one evening, both on the MoE first:

Concurrency. With multiple parallel server slots (-np 4/8, the fork’s concurrency flag) throughput anti-scaled to 0.49x/0.46x of serial. Worse: at around 9,400 tokens of depth, sessions started answering with other sessions’ planted secrets. Codes that appear nowhere in their own conversation. The decisive control: the same eight conversations interleaved through a one-slot server pass clean far deeper, so the variable was -np itself. The corruption transfers to the dense 27B too; its own gate caught slot 0 answering with slot 3’s code at -np 4. Concurrency on the fork was rejected on correctness, which is much worse than being rejected on speed.

Speculative decoding (MTP), on the MoE. A 13% regression despite a 74.6% draft-acceptance rate, and it changed the model’s outputs (0/3 byte-identical, where off-vs-off reproduces 3/3). A speedup that changes the answers is not a speedup. MTP on the dense model never ran on the fork; by the time it was next in the queue, the fork was out of the running.

Only after those verdicts did the dense 27B, the model this is actually about, get its own baseline: 34.41 t/s (a later arm nudged to 36.48 with f16 KV). So the fork’s best safe configuration for it was one slot, no speculation, 34.41 t/s. Respectable. But I was benchmarking against the wrong bar, and the correction came from an unexpected direction.

§Two retractions that changed everything

The power cap wasn’t costing anything. The campaign confidently attributed the gap-to-published to the 275 W cap. I didn’t buy it; my money was on the second-hand card needing a repaste, if anything. An independent same-model, same-card power sweep settled it in favour of neither theory being needed: 275 W and 350 W within noise, because the memory clock sits at 9501 MHz at every power level and decode is memory-bound. The note from that night: “Being AT a cap is not being SLOWED BY it.” At the time we concluded the card was fine. Slightly faster than the reference card, even.

Postscript, two days later: I was right about the paste after all. An ops investigation of a screaming blower found the die hotspot at 102-103 C, which is 25 degrees above the 78 C edge temperature that’s all nvidia-smi shows, plus a cumulative 38 minutes of thermal throttle accumulated during the campaign’s compute soaks. Degraded paste, healthy pads (VRAM junction 82 C), invisible to every sensor we were watching. Two lessons: the edge temp is not the story, and the throttle cuts the generous way for this post. Every number here was measured on a thermally degraded card, so the wins under-report the hardware. A repaste (PTM7950) and a re-baseline are queued. The power-cap conclusion itself still stands; that sweep’s memory clock never moved.

The published bar was never a llama.cpp number. Reading the club-3090 repo primary-source-first: it contains no llama.cpp benchmark at all. Every figure is vLLM. Their no-speculation control is 32.4/32.6, meaning my fork build at 34.41 was already about 5% ahead. I’d been reporting us 45% behind for about 69 minutes before the primary source corrected it. (It felt like a day.) I had compared my no-MTP number against their MTP-enabled one. Retraction 12.

Which reframed everything: the entire apparent gap was vLLM plus MTP. vLLM had been the plan from the campaign’s first message; the fork detour happened because the bar was misattributed to ik_llama. Now the engine the workhorse actually runs came back on stage.

§vLLM: four environment faults, then it beat everything

Getting vLLM 0.27.1 to start cost four environment faults, none of them the model:

1. Python 3.10 vs flashinfer’s tuple[..., array.array[int]] annotation, which needs 3.11+.
2. Two “obvious” workarounds, uninstalling flashinfer and forcing a different attention backend, both tested and both refuted: vLLM imports it unconditionally.
3. Ubuntu 22.04’s python3.11 package is secretly 3.11.0rc1, a release candidate missing stdlib functions. A release candidate, under a stable package name, in universe.
4. ninja missing; then present in the venv but invisible to the engine’s subprocess PATH (my bug).

A uv-managed CPython 3.12.13 fixed all of it. Then, first clean run:

Stock vLLM 0.27.1, no patches: 44.89/43.44. That’s 1.39x/1.33x the reference repo’s control and 1.30x my own fork build, with the caveat that checkpoint format and KV dtype changed alongside the engine. The likely reason is mundane: their recipe pins vLLM eight minor versions back, and 0.27.1, released the day before I tested, has this in its patch notes: “Qwen3-Next. GDN linear attention fusion. OOM.” That’s their “explicitly unfixable on one card” cliff, fixed upstream per the patch notes (I haven’t pushed past ~50K on their config to verify it myself). Being current mattered as much as engine choice, though I’ll note I inferred that from the patch notes; the clean test, old versus new vLLM on the same checkpoint and command, never ran.

MTP on vLLM: 64.98/71.31 (1.45x/1.64x), at 73-75% acceptance. And the outputs don’t change, verified on an 80-character head (the fork check used a full-text hash and was stronger). That’s the opposite verdict from the fork’s MTP, though not a controlled comparison: the fork’s rejection was on the MoE at draft depth 1-2, this is the dense model at depth 3. It beats the 27-patch config’s narrative number and ties its code number.

Concurrency reverses too: 6.74x aggregate scaling at eight streams (298.89 t/s), where the fork anti-scaled, and the same trap-laden correctness gate that caught the fork’s cross-session leak passes clean at 4 and 8 slots. But MTP inverts under batching. It wins 1.45x at one stream and loses 2.1x at eight. Speculation spends compute to save memory bandwidth; continuous batching already amortises the weight reads. There is no single optimal configuration. Profile by load.

§The wall: 65,536 tokens of working memory

With speed solved, vLLM’s own startup line delivered the actual constraint:

Available KV cache memory: 2.33 GiB → GPU KV cache size: 65,536 tokens

About 18 GB of int4 weights leave about 2.4 GB of conversation memory, shared across every concurrent session. Tokenising real agent transcripts with the model’s own tokenizer gave peak context of p50 49,867, p90 105,794, max 106,767. That’s 7 of the 15 sessions I could measure exceeding the entire pool on their own (eval-run transcripts, which run heavier than production). Free levers (utilisation 0.98, artefact shuffling) top out around 79K. For a while the roadmap said “produce a 3-bit weight quant”, a real project, since no such artefact exists to download.

Three smaller findings from this stretch that generalise:

• Size on peak, not average. Per-call averages badly under-count what a KV pool must hold.
• Pick artefacts by byte count, never by name. A back-of-envelope model projected a plausible-sounding AWQ repo at ~27K tokens, 2.4x worse, purely because its files are 1.35 GiB bigger. (The model was later refuted, but the direction of the lesson survived it.)
• Then don’t trust disk bytes either. Two int4 artefacts differing by 0.27 GB on disk consume an identical 17.93 GiB on the card; the loader repacks. Only vLLM’s own “consumed memory” line is evidence.

§The KV dtype I never actually chose

Asked why the config used fp8_e5m2 for the KV cache, I checked my own reasoning and found there wasn’t any: I had inherited it from the reference repo’s compose file. The capacity of e4m3 and e5m2 is identical (65,536 both, measured); the difference is numerics. e4m3 spends its bits on mantissa, e5m2 on exponent range that KV values never use. A published measurement (arXiv 2606.09864, Qwen-2.5-7B on a safety benchmark) put e5m2 at 4.3x the answer-flip rate of e4m3. Different model and task, so I treated it as a warning rather than a Qwen3.6 result, and changed the recommendation to fp8_e4m3. Cost of the swap: zero without speculation, measured twice, with one MTP code-shape oddity still open (below).

§The soak that “failed”, and the bug that was mine

While collecting results one morning (several runs had executed the previous day but their verdicts were never read, a distinction this campaign now respects) I found an endurance run whose verdict JSON I’d never opened: FAIL, retention loss. Concurrent sessions apparently forgetting their own planted secrets at around 21K tokens. For a few hours the campaign’s deep-concurrency story looked broken.

Then the free check: every failing recall reply had emitted exactly 200 tokens, the instrument’s own reply cap, and every “forgotten” secret was a strict suffix of the planted list. Real memory loss takes the oldest or random items; a token budget cuts an in-order enumeration at the tail. The model thinks out loud before answering, 200 tokens fits two secrets but not five, and my test scored budget exhaustion as amnesia. Instrument defect #16. The test now gives recall turns their own budget and refuses to issue a verdict on any reply that hit its cap. A verdict must not fire when the instrument couldn’t have seen the answer.

This is the pattern the campaign kept falling into. Twenty defects in the test tooling by the end: a gate that passed vacuously, canaries that couldn’t distinguish leak from typo, a fetch verifier comparing bytes against a rounded display string, an answer key where I’d computed 987654 mod 321 wrong myself. The systems under test had real defects too, and the fork’s cross-session leak is as real as bugs get. But every one of them was only findable through an instrument that had itself been debugged first. The instruments are where the bugs concentrate, because the instruments are the code nobody reviews adversarially.

§TurboQuant: 2.5x the context for 3% of the speed

The reference repo’s advice said don’t use TurboQuant KV quantisation; it fought their speculative decoding. But that advice was written against vLLM 0.19, and the fleet profile runs speculation off anyway. Current vLLM ships four TurboQuant modes as flags. So: a ladder, one server load per rung, pool size read from vLLM’s own startup line (all rungs on the GPTQ artefact at the 40960-ctx/4-seq shape, utilisation 0.96 unless stated):

Only the bottom rungs clear every session line; everything above fp8 in the table still fails a real session. (The p90 peak, 105,794, sits almost exactly on the heaviest-session line and is omitted for legibility.)

163,840 tokens of pool covers every session peak I’ve measured, including the 131,072 hard ceiling my heaviest loop is configured for. That started as a startup-line figure; it’s now backed by a real serve. At the long-context shape (--max-model-len 131072, one sequence, where the pool actually reads 207,530) the box served a genuine 105,697-token prompt with five secrets planted between 17% and 83% depth, and recalled all five, on 3-bit KV. The 3-bit weight requant project died right there, replaced by a command-line flag.

The number long sessions actually feel came out of the same runs: decode falls from ~44 t/s shallow to 17.4 at 63K context to 12.4 at 106K. Attention over deep KV is the tax, and it applies whatever your KV format. Prefill holds 815-940 t/s, so a cold 106K prompt costs about 130 seconds before the first token. Quote decode numbers with their depth or don’t quote them.

The catches, measured rather than assumed: decode cost about 3% (43.88 vs 45.29). Quality on my 24-task exact-match gate: identical scores, and zero genuinely wrong answers on either arm. The misses were the model thinking past the gate’s token budget (same tasks on both configs), plus one case where my answer key demanded 50 and the model correctly answered 50.0 to a prompt that said “as a decimal”. Instrument defect #18.

Then the composition question: TurboQuant with speculative decoding, the one the old advice forbade. It serves. And then it produced the single most seductive number of the whole campaign, 102.66 tokens/second, the fastest anything had gone. The quality gate scored it 0/24. Every reply degenerates into token loops (</think></think></think>, of of of of, 's's's's) and the speed is the symptom: repetition is trivially predictable, so the speculative drafter’s acceptance rate soars and throughput inflates. Forced-length benchmarks are blind to this by construction; only a natural-generation quality check could see it. The reference project’s warning about this combination, which came from their own failure reports and deserves credit for being the reason I tested it carefully, was right about the combination and wrong about the failure mode on current vLLM. It doesn’t crash. It babbles, fast.

Pick the rightmost point on throughput alone and you deploy the one at quality zero.

The isolation is clean, which is the part I’m most confident in: on the same 24-task gate, no-levers scores 18/24, TurboQuant alone 18/24, MTP alone 17/24, all with zero genuinely wrong answers, and the two together score 0/24. (The four arms span two artefacts and two serving shapes; the gate and prompts are constant.) Each lever is fine. The composition is broken.

One more surprise en route: the artefact gates speculation. One int4 checkpoint’s draft head loads dense (2.37 GiB) and OOMs on a full card at any shape, while the other artefact’s fits. The two checkpoint sets differ by 1.5% on disk, and one of them silently costs you the entire MTP lever.

§The soaks, finally

After the instrument was fixed, the deep soaks all three profiles depend on finally ran validly: the unquantised control clean to 36,153 tokens; TurboQuant 3-bit with four concurrent sessions to 36,148 tokens each (88 turns, twenty planted canaries, zero forgotten, zero leaked between sessions); and the MTP profile clean to 36,152. One more tooling defect on the way there (#20: my runner’s own 15-minute server timeout killed the first attempt silently, and the control had cleared the same wall by about sixty seconds), which by this point in the post should surprise nobody.

§The three working configurations

This is the part headed for the dual-3090 workhorse. All three run on stock vLLM 0.27.1 (CPython 3.12+; the 3.10 and 3.11-rc traps above are real), no patches of any kind.

Profile A, one agent, lowest latency. The AutoRound checkpoint, because the other artefact’s draft head doesn’t fit:

vllm serve Lorbus/Qwen3.6-27B-int4-AutoRound \
  --max-model-len 32768 --max-num-seqs 1 \
  --gpu-memory-utilization 0.96 --kv-cache-dtype fp8_e4m3 \
  --speculative-config '{"method":"qwen3_next_mtp","num_speculative_tokens":3}'

Profile B, one agent, maximum context. No speculation: the combination babbles, see above:

vllm serve groxaxo/Qwen3.6-27B-GPTQ-Pro-4Bit \
  --max-model-len 131072 --max-num-seqs 1 \
  --gpu-memory-utilization 0.96 --kv-cache-dtype turboquant_3bit_nc

Profile C, fleet throughput, eight streams:

vllm serve groxaxo/Qwen3.6-27B-GPTQ-Pro-4Bit \
  --max-model-len 40960 --max-num-seqs 8 \
  --gpu-memory-utilization 0.96 --kv-cache-dtype turboquant_3bit_nc

Still open: the card needs a repaste (hotspot postscript above) and a re-baseline before any cross-card comparison. One dtype oddity remains: the e4m3 arm measured 10.6% lower code-shape MTP throughput than the earlier e5m2 arm while narrative was unchanged, and the matched pair needed to settle it hasn’t run. And nothing here has served real traffic yet.

§What I’d tell you to steal

1. Register predictions with refutation lines before running. Half this campaign’s value came from bands that missed. A miss with a pre-registered band is information; a miss you re-explain afterwards is nothing.
2. Suspect your instruments first. Twenty tooling defects against a handful of system ones is not a subtle ratio, and the system ones were only visible through repaired instruments.
3. Benchmark speed and quality in the same harness. The fastest configuration of this campaign was broken, and no throughput number could have told me. A forced-length benchmark cannot see degeneration; twenty-four boring exact-match questions caught it immediately.
4. Being current beats being clever. A stock release that was one day old outperformed the older patched reference’s published numbers, and most of the patch set existed to fix things upstream had since fixed.
5. A number without its regime is not a number. Engine, model, artefact, dtype, serving shape, concurrency, clocks during the run. Every time I dropped one, a retraction followed, including in the first draft of this post, where a fact-checking pass caught me quoting MoE-measured verdicts in the dense model’s story.
6. Quantise the KV cache before you requantise the weights. The context lever hiding in --kv-cache-dtype was worth 2.5x, and it replaced an entire produce-a-new-weight-quant project with a flag.

I’m pulling the probes, the soak harness, the quality gate and the process itself into a small open source repo, so the next model (there’s always a next model) gets the same treatment with less ceremony. Link here when it’s up.

§Credits

• The club-3090 project did the original single-card work on this model, and their failure reports are the reason I knew which combinations to test carefully. Beating their numbers with stock software was only possible because they mapped the terrain first, on an engine eight versions older.
• vLLM, whose 0.27.1 release quietly fixed the thing everyone was patching around, and whose startup logs are honest enough to build a capacity model on.
• ik_llama.cpp and llama.cpp, which carried this project through its CPU era.
• Jean F. Brito’s 3090 power-limit sweep, the independent data that settled the power-cap question and saved me from a wrong conclusion.
• uv, for making “just use a real Python” a one-liner after apt handed me a release candidate.
• The authors of arXiv 2606.09864, for measuring what fp8 KV formats actually do to answers, so I didn’t have to choose numerics by folklore.