Can Qwen3.8-27B run well in vLLM on one RTX 3090, the day it ships?

Fourth in a series. Follows the DeepSeek-V4-Flash single-3090 note. Living document; numbers update as the campaign runs. Last updated 2026-08-16. A revision history is at the bottom, and it is worth reading: this note has retracted one headline and reversed one verdict since rev 1.

Qwen3.8-27B released on 14 August. It is a dense 27B with native vision, a 262K context window, and a hybrid attention stack: 48 of its 64 layers are Gated DeltaNet linear attention, and only 16 carry ordinary KV-cache attention. On release day I found no ready-made way to run it in vLLM on a 24 GB card. The official quants were FP8 (28 GB of weights) and NVFP4 (Blackwell GPUs only), and in two release-day X searches I found no Ampere-compatible INT4 artifact. Those searches were not archived, so I do not treat them as proof of priority; read “nobody had one” as “I could not find one.”

The hardware is a single RTX 3090 in a shared research box, and the question behind the question is the same one driving this whole series: how much of a current frontier-adjacent model can one consumer card actually deliver, measured honestly. The method rules are unchanged from the earlier notes: register acceptance bands and refutation lines before a run, log quiet periods as quiet, and record clocks and thermal counters during the measurement, not around it.

Within roughly six and a half hours of the repository appearing, the checkpoint was serving in vLLM with a 24-task exact-match gate passed. Over the following two days the same card produced a quality-verified quant, a large fleet-throughput gain from a bigger serving shape plus KV-cache quantization, and a 1.7 to 1.9x single-stream speedup from the model’s built-in speculative decoding, which I had first declared impossible on this hardware. This note covers the method, the measurements, one streaming-field defect that inflated a headline by 44%, and one wrong verdict that a challenge reversed.

§The numbers

Two denominators appear below and they are never mixed: wall is all server-counted generated tokens (reasoning included) divided by total request wall time; phase decode is the post-first-token generation rate as the probe computes it. Single RTX 3090 at a 275 W power cap. Engine is vLLM 0.27.1 stable except where a row names the nightly or a patch.

The campaign chartered three bars against what the previous-generation model achieved on this same card, and all three now have answers:

1. Single stream at 50 tok/s or better via the built-in speculative decoding: cleared, at 72.63 narrative and 79.27 code, 1.73x and 1.90x over the matched speculation-off control, on a patched nightly (details and caveats below).
2. Fleet aggregate at the 281 tok/s the previous model reached: not reached, closed honestly at 263.57/266.35 clean. A one-lever ladder at the large-pool shape showed 3-bit KV beats an 8/4-bit split (248) beats 4-bit (240) and that the remaining gap tracks the model’s own per-token cost, its 248K-entry vocabulary head being the prime suspect.
3. Perplexity within 4% of bf16: cleared at +2.98%.

§Caveats that belong up here rather than at the bottom

Everything below is one card, one box, one artifact, mostly single sessions. The quality gate is 24 short exact-match tasks, which establishes “did not grossly fail” and nothing stronger; the stronger, but still narrow, quality evidence is the paired Wikitext-2 perplexity comparison. Vision is entirely unvalidated post-quantization. The MTP speedup requires a one-line patch to the serving engine until the underlying bug is fixed upstream, so it is a working recipe, not the certified serving configuration, and its gate score (22/24 in both arms) sits a task or two below the certified stable serve’s 24/24 thinking and 23/24 instruct on this small gate. The fleet comparison against the bf16-KV row is a configuration comparison, not an isolated KV-quantization effect: the serving shape, utilization, and thermal state changed along with the KV dtype. And the calibration provenance has two gaps I would close next time: the exact calibration token total was not recorded, and neither the dataset nor the source model download was revision-pinned.

§The quant

I quantized the official BF16 safetensors with llm-compressor 0.12.0 GPTQ: W4A16, INT4 weights, symmetric, group size 128, calibrated on the first 128 rows of a public calibration dataset (the llm-compressor ecosystem’s standard calibration set), each row truncated to at most 2,048 tokens. GPTQ is a calibration-based method: it streams real activations through each layer and compensates quantization error against the resulting Hessian, the same family of idea as imatrix calibration in the GGUF world. The recipe excluded lm_head, the vision tower, and the MTP head from quantization. I downloaded the 18 source shards with size verification only, no revision pin, no content hashes; a gap the audit flagged and the pipeline now needs.

One provenance episode: the checkpoint I first measured had no MTP draft head (a re-graft helper failed on an unexpected save layout), and I grafted the 15 bf16 MTP tensors in afterward. That made the layout intended for upload not byte-identical to the layout that was tested, so it stayed uncertified until a sha-pinned re-serve of the exact upload layout repeated the gate (24/24) and landed the wall rate in band. The artifact is now certified, with content hashes recorded: what would ship is what was measured.

The perplexity comparison is the quality evidence the gate cannot provide. Both artifacts were scored on the same Wikitext-2 test input through the same 40-thread CPU path (the quant decompressed to dense weights on load, so the arithmetic path matches): 32,752 next-token targets across sixteen 2,048-token input windows, about 6.4 hours per arm. The quant was worse in 13 of 16 windows by small margins and better in none by much, and the delta came to +2.98%, inside the pre-registered 4% band.

On thermals: the bf16-KV fleet session added 18.75 seconds to the cumulative software-slowdown counter in a few minutes, and the single-stream session immediately after it added another 11.04 seconds, while later sessions on a cooled card added zero. The brackets establish that throttling happened during sustained load; they do not isolate whether the power cap, ambient, or accumulated heat was the trigger. Rev 1 claimed a cause the instruments don’t support.

§The inflated headline

The probe first reported 52.64 tok/s for thinking mode, computed as (completion_tokens - 1) / (wall - TTFT). That crossed both its registered acceptance band (top 48) and the explicit audit trigger (above 50), which forced an audit before banking. The audit was arithmetic: the same tokens over total wall gave 36.53, a 44% inflation, and the observed TTFT of 8.4 s against a 79-token prompt is not a prefill time on any GPU.

The mechanism: a thinking model streams its reasoning phase separately from its answer. The probe had been fixed once to watch reasoning_content, one engine’s field name. This engine’s parser streams reasoning. The probe was blind again, counted all tokens but timed only the post-thinking window, and produced a rate inflated by exactly the hidden fraction. Any phase-windowed “decode” figure published for a thinking model deserves this suspicion; wall-basis was immune to this particular defect (it is not immune to server-side token miscounting, failures, or queuing; no metric is).

After fixing the probe I re-ran with a detector band registered before the rerun, but anchored on the already-known wall value, so it was a gross sanity check against recurrence, not independent validation. It passed (37.13 phase vs 36.72 wall). The instruct-mode run was the cleaner control all along: with no thinking phase, phase decode and wall agree within 1.5%.

§The verdict I had to take back

MTP is the part I got most wrong, and the reversal is the most useful thing in this post. Qwen3.8 ships a draft head for speculative decoding baked into the checkpoint: a small predictor guesses the next few tokens and the main model verifies them in one pass. Five serve attempts failed with an invariant out-of-memory error: 2.37 GiB requested, every time, at every context size, sequence count, utilization, and even with CUDA graphs disabled. 2.37 GiB is exactly 248,320 vocabulary x 5,120 hidden x 2 bytes: a full bf16 copy of the language-model head, allocated for the draft instead of shared with the target. I wrote it up as structurally blocked on 24 GB and closed the arm.

Postscript, seven hours later, under the original wrong verdict where it belongs. A challenge arrived: “did you search how others are doing this?” I had not, not after the failure. The search took ten minutes and overturned the verdict. People were running this model’s MTP on single 24 GB cards, and vLLM’s loader is explicitly designed to share the target’s head with the draft (it logs the sharing when it works). My OOM was a bug I was triggering, not a wall I was hitting.

The isolation, on one checkpoint and one engine build: loaded through the text-only Qwen3_5ForCausalLM path (plus a one-line patch mapping the unused vision tower’s weights to nothing, which the mapper class supports natively), the loader logs “Sharing target model lm_head weights with the draft model” and the 2.37 GiB copy vanishes. Loaded through the multimodal Qwen3_5ForConditionalGeneration path, it silently allocates the duplicate. Same checkpoint, same engine, two load paths, one bug; a two-command reproduction is drafted for upstream.

Measured on the working path, single stream, wall basis, three reps each: 72.63 tok/s narrative and 79.27 code with speculation on, 41.93 and 41.83 with it off on the otherwise identical serve. That is 1.73x and 1.90x from one lever, at an engine-reported mean acceptance length of ~2.7, with the quality gate scoring identically (22/24) in both arms and clean thermal brackets. The code figure crossed my pre-registered too-good line (above 78) and banked only after the matched control isolated the lever. One scope note on the control: against the older stable-engine baseline (41.7), the whole nightly-plus-text-only-plus-patch bundle was net-neutral (41.9), which bounds the bundle’s own contribution near zero but does not isolate the load path from the other changes; the speculation-on/off comparison is the controlled one. When a speedup spans multiple changes, the control is the difference between a claim and a number.

§Fleet throughput: the KV lever

At eight concurrent streams, moving from the small bf16-KV shape (8K context, utilization 0.90, thermally flagged, 136.14/133.68 aggregate) to a large-pool shape with a 3-bit quantized KV cache (40,960 max-model-len, utilization 0.96, clean bracket) roughly doubled aggregate throughput to 263.57/266.35. That is a configuration comparison, several things moved at once. The controlled evidence is the one-lever ladder run entirely at the large shape: 3-bit KV (263 to 266) beats an 8/4-bit asymmetric split (248) beats 4-bit (240), and raising the engine’s sequence cap from 8 to 16 while still offering eight streams moved aggregate by noise. The ladder’s reading: at this concurrency the fleet attention path is KV-bandwidth-bound, so the hardest KV compression wins, and it is worth testing in engines that offer one. Only a quarter of this architecture’s layers hold KV at all, which is why the pool stretches so far. The 3-bit KV dtype here is a local engine patch, not upstream vLLM, so this section has no copyable recipe.

§What broke on the way

Ten blockers between release and first serve, a mix of dependency and API drift with defects in my own fetch and orchestration logic: a resume-through-CDN-redirect that appended past a complete file; a job triggered by a sentinel file’s existence when its content said PARTIAL; truncated downloads silently accepted by single-shot curl; a version pin above a library’s supported ceiling; an API check probing an internal that had become lazy.

Then: a dependency wiped because the venv is recreated per run and I’d installed it out-of-band; a tokenizer/processor mutual exclusion; a dataset name resolved against an internal registry instead of the hub; a missing build binary, twice, because a venv install doesn’t reach a subprocess PATH; an OOM from default reservation arithmetic. Two more arrived during the perplexity run: a derived script written without its execute bit whose launcher still reported success (the failure was asynchronous), and packed compressed-tensors weights that the plain transformers forward pass cannot run until you ask for decompression on load. The fixes are committed, but I won’t claim they can’t recur; the fetcher still uses mutable branch URLs and size-only validation.

§Working configurations

The certified serving configuration, stable vLLM 0.27.1, both modes gated:

vllm serve ./qwen38-w4a16 \
  --max-model-len 16384 --max-num-seqs 4 \
  --gpu-memory-utilization 0.90 \
  --reasoning-parser qwen3
# instruct mode: add --default-chat-template-kwargs '{"enable_thinking": false}'

The MTP recipe. Honest scope: this is not yet copyable from the block alone. It needs vLLM nightly 0.27.2rc1.dev108 (or later), a text-only copy of the checkpoint’s config (architectures set to Qwen3_5ForCausalLM, re:mtp.* appended to quantization_config.ignore), and, until the upstream fix lands, a one-line edit to the model file’s weight mapper: add "model.visual.": None beside the existing "model.language_model.": "model." entry in Qwen3_5ForCausalLMBase.hf_to_vllm_mapper, which tells the loader to skip the vision tower. The quant itself will be linked here once uploaded.

vllm serve ./qwen38-w4a16-textonly \
  --max-model-len 8192 --max-num-seqs 2 \
  --gpu-memory-utilization 0.92 \
  --speculative-config '{"method": "mtp", "num_speculative_tokens": 2}'

Still open: vision post-quantization (never validated), the roughly 14% wall-basis gap between thinking and instruct modes on identical accounting (unexplained, logged), and everything past 5.1K tokens of real depth. At the 40,960/8 3-bit-KV startup shape the engine reported roughly 327K tokens of aggregate KV-pool capacity; that is allocation capacity, not a served-depth result. My longest measured prompt was 5,113 tokens, and nobody has measured quality or speed near the native 262K limit on this card. That experiment is designed and waiting.

§What I’d tell you to steal

1. Keep two denominators and label both. Tokens-over-total-wall as the anchor, phase decode as the diagnostic. When they disagree on a non-thinking run, suspect the instrument, not the hardware.
2. When an engine adds a reasoning parser, re-audit your first-token detector by field name. content, reasoning_content, and reasoning have all appeared as the first-token-bearing field across engines in these incidents; missing the live one moves your detected first token past the reasoning phase and materially inflates phase-decode rates (44% in this post’s case).
3. Register a too-good line before you run. Mine is the only reason 52.64 died in minutes instead of becoming the post, and the reason 79.27 got a control before it got banked.
4. When you declare something impossible, search for people doing it. My structural-impossibility verdict had a byte-exact mechanism and five failed attempts behind it, and it was still wrong. Ten minutes of searching found the working configurations that proved the wall was a bug.
5. A speedup that spans multiple changes needs a control that isolates one. The speculation-off serve on the identical load path is what turned “2x faster, probably the draft head” into “1.73x/1.90x, attributed.”
6. Trust the server’s token counts over your client-side estimates. My “7.9K-token” depth probe was a 5,113-token depth probe; the server knew, I didn’t ask.
7. The artifact you test must be the artifact you ship, byte-for-byte. Or you re-certify the shipped layout. Post-measurement “improvements” to a checkpoint un-certify it.
8. Sentinel files gate on content, never existence, and launchers verify by process, never by exit code. Both bit me in one campaign; the second one hid a dead job behind a success message for an hour.

§Credits

The two community benchmark reports that reversed the MTP verdict: a 4090 run of this model with the draft head enabled (their measurement: 1.9x at short context, tapering with depth) and a 3090 llama.cpp run at +29% with acceptance 0.565. Their existence was the falsifying evidence; I found them only when challenged to look. The config of a community AWQ-MTP checkpoint provided the working metadata reference that narrowed the bug hunt. Unsloth’s day-zero documentation collected the sampling parameters and gotchas in one place. And vLLM and llm-compressor are the stack that let a single consumer card quantize and serve a released-yesterday multimodal hybrid at all; the bug report going their way is written with gratitude.

§Revision history, for the record

Rev 2 (15 August): an adversarial audit of this note against the raw evidence produced two rates relabeled from wall to phase decode; depth re-stated from an estimated 7.9K to the server-counted 5,113 tokens (prefill 1,565 to 1,013 tok/s); fp16 KV corrected to bf16; “eleven hours” corrected to about six and a half; a thermal causal claim withdrawn; a “self-validating” claim downgraded to a sanity check; and a priority claim downgraded to “not found in unarchived searches.” The measurements that survived unchanged: both gate scores, every wall rate, the fleet aggregate, the thermal deltas, and the 0.968 ratio.

Rev 2.2 (15 August): the MTP verdict reversed, with measured numbers, on evidence a reader’s challenge forced me to find.

Rev 2.4 (16 August): a second dual audit (two independent reviewers against the raw JSONs) caught the fleet headline quoting a thermally flagged session’s numbers as the clean pair, a pool-size mislabel (40,960 is the context limit, not the KV pool), an over-attributed load-path claim, and an unsupported inflation-range claim. All corrected above. If you publish numbers, this is the argument for letting something adversarial read your evidence before strangers do; three times now, in my case.