How does a 320B model run on one RTX 3090? GLM-5.3-Flash, weight by weight

Part of a series on one HP ProLiant. Follows Putting an RTX 3090 in an HP ProLiant ML350 Gen9 and infermaxx: can stock vLLM replace my patched deployment?. Living document; numbers update as the campaign runs. Last updated 2026-10-15: my patches are now public, and there’s a section on reproducing this.

I posted that GLM-5.3-Flash runs at 15 tokens/second on one 24 GB RTX 3090 and the RAM of an eight-year-old dual-Xeon server. A reply said I had it wrong: you cannot fit 758B parameters into about 26 GB of VRAM without “obnoxious accuracy penalties”, that works out to “less than 0.25bpp”, and anything under 4 bits “results in catastrophic stupidity” (the thread).

He’s right that squeezing hundreds of billions of parameters entirely into 24 GB would take absurd compression, and right that quantization costs quality. My claim is narrower. The model is the Flash variant, 320B total with 18B active per token (Z.ai’s model card). Most of its weights sit in system RAM, not VRAM. They average about 3.1 bits per parameter, not 0.25, and the bit width varies a lot by tensor. Whether that quant is good enough for my work is a question I still have to answer with my own tests. This post is about where the model lives and how fast it runs.

Why bother at all: the box was sitting there as a cold spare, and what I want from it is offline work on things I won’t send to an API, like reversing firmware and testing our own systems. Fifteen tokens a second is faster than I read, which makes it worth testing for that job.

The hardware results below are from this machine alone, with no other GPUs involved. For every number I’ve tried to say whether it is my measurement, read from the checkpoint, someone else’s published benchmark, or my estimate.

§Executive summary

The box: HP ProLiant ML350 Gen9, two Xeon E5-2699 v4 (Broadwell, 22 cores each, from 2016), 512 GB of DDR4-2400 (503 GiB visible to the OS), one RTX 3090 on PCIe 3.0 x16. In the configuration below the serving process uses about 225 GB of that RAM. The model: turboderp/GLM-5.3-Flash-exl3, the 3.05 bits-per-weight branch, 125 GB on disk. The serving design is 0xSero’s glm53-flash-offload (exllamav3 plus a CPU expert tier), which I run natively and tuned for this older hardware.

These are milestones from a ten-day campaign, measured with different procedures, not a controlled before-and-after comparison. The last row is one stream, a short prose prompt, up to 1,200 output tokens, greedy decoding, thinking off.

Short answers:

1. Storage: 121 GB of language-model weights at 3.1 bits per parameter on average. The routed experts (97% of the parameters) are 3-bit and live in RAM; everything a token touches every time is 4 to 16 bits and lives in VRAM.
2. Speed: about 15 tok/s single stream. Each token needs only 8 of 288 experts per layer, and the CPU computes most of the ones the GPU doesn’t have, where they already sit.
3. Quality: unmeasured by me. turboderp’s published numbers for this exact quant put it close to a GGUF 4-bit i-quant (IQ4_XS) on this model. Details and limits below.

Caveats that belong up here:

• Single stream only. I have only benchmarked one conversation at a time.
• Prose only, for this number. On an older version of the benchmark, code prompts ran about 7 to 8% slower than prose (13.1 to 13.8 vs 14.3 to 15.0 tok/s).
• The server is configured for 131,072 tokens of context, but these windows used a short prompt. I have not measured decode speed with a long occupied history, and prefill is slow: a 2,017-token prompt took 17 s (119 tok/s).
• fast mode changes the numerics relative to stock exllamav3 on the same quantized weights. A panel that forces every cold expert through the CPU path agreed with stock on the top token 98.65% of the time (KL 0.0035, renormalized over the reference’s top 20 tokens, 2,154 positions). That panel exercises the CPU kernel’s multi-token arithmetic. Single-stream decode uses a different integer path, so the panel is a consistency check, not an end-to-end quality measurement. fast mode is also not bit-deterministic from run to run. exact mode matched stock exactly on the same panel.
• I have not measured this quant against the original model on my own tasks.

§Where the weights live

The checkpoint ships a manifest (quantization_config.json) listing the quantized tensors and their bit widths. Adding up the language model’s tensors by class, plus the few small ones the manifest leaves out:

The rest of the 125 GB download is the multi-token-prediction layer, shipped as a separate 2.9 GB file (about 7 B parameters) and not used here, and a 0.6 GB vision encoder. Those account for the gap to Z.ai’s 320B count.

Two things fall out of that table.

First, 3.1 bits per parameter, not 0.25. A 122 GB model has to live somewhere with that much room, and here that’s the DDR4. In fast mode the experts are in RAM twice: once pinned so the GPU can read them across PCIe, and once rearranged for the CPU kernel, 229 GB together. The 3090 holds the part of the model every token touches (about 5.8 GB plus the embedding table), the KV cache and attention state, and a cache of 1,376 hot experts (13 GB, about 11% of all 12,096).

Second, the precision isn’t uniform. The routed experts are 97% of the parameters and get 3 bits. Each token selects 8 distinct experts out of 288 in each MoE layer, so any one expert does a small share of the work. Everything a token passes through every time (attention, the shared expert, the dense layers, the output head) gets 4 to 6 bits, and the router and embeddings stay at 16. turboderp chose that split when making the quant; I didn’t change it.

§What happens to one token

Per decoded token the model runs 45 layers. The non-expert work (attention, shared expert, routing, the head) runs on the GPU from VRAM: about 13.7 ms per token in one profiled window, 13 to 16 ms across runs. Its 8 ms of matrix-vector work runs at roughly 70 to 75% of the 3090’s spec-sheet bandwidth.

In each of the 42 MoE layers the router then picks 8 experts. In the four 60-second windows, those 8 split up like this on average:

The GPU and CPU work on the same layer at the same time, and the GPU adds the CPU’s partial result before moving on. A cost model decides per layer which misses go where, so both sides finish at about the same time.

§Three memory paths and the CPU bottleneck

Decode on a model like this mostly means reading weights. Per token that’s about 5.8 GB of always-used weights plus 3.17 GB of routed experts, and the box has three paths to memory at very different speeds:

As a bandwidth-only bound: if all 9 GB a token uses had to cross PCIe, the ceiling would be about one token per second before any other overhead. That’s the intuition behind “you can’t run this on 24 GB”, and for that design it’s right. This one keeps the always-used weights in VRAM and avoids moving cold experts to the GPU at all. It computes them on the CPU, where they already sit.

exact mode shows what the PCIe path alone gets you. There the CPU computes nothing and every uncached expert crosses the bus: about 1.63 GB per token at ~7.5 GB/s, a ceiling near 4.6 tok/s. It measured 4.13.

The CPU side surprised me. I expected DDR4 bandwidth to be the limit, and it isn’t. The first measurement, on 20 cores, worked out to about 16 GB/s of expert weights, far below what one socket can stream. Today, on 42 cores, each job costs about 0.2 ms plus 0.245 ms per 9.44 MB expert; in the 15.2 windows that’s 3.3 experts in 1.04 ms, about 30 GB/s, still well under the 136 GB/s available. Decoding a trellis takes arithmetic for every weight, and a 2016 core without AVX-512 does it slowly. The kernel timings and scheduling experiments point to CPU computation and work distribution as the limits here, which is why most of the tuning went into CPU scheduling rather than memory placement.

§What 3-bit EXL3 means

EXL3 uses Hadamard transforms and trellis coding rather than a simple uniform scalar code. GGUF, the other common format, covers several quantization families. The classic Q4_0 and Q4_K round each weight onto a uniform grid with a scale per block of 32 weights (Q4_K_M works out to about 4.8 to 4.9 bits per weight on real models), while its newer IQ formats use lattice codebooks and non-uniform levels. So the useful comparison is between specific quants at stated bit budgets, not between “3 bits” and “4 bits”.

In the exllamav3 README’s words, EXL3 is “a streamlined variant of QTIP from Cornell RelaxML”. Both of its main ideas are visible in this checkpoint:

1. Incoherence processing. Each weight matrix is wrapped in random sign flips and a Hadamard transform before quantizing. That spreads outliers across many weights, so no single large value wrecks its neighbours. Every quantized matrix in the file carries two fp16 sign vectors for this, suh on the input side and svh on the output side.
2. Trellis coding. Weights are encoded in 16x16 tiles, and the 3 bits per weight are not a level index. They are steps along a path through a trellis with a 16-bit state, and each weight’s value is computed from the current state by a cheap hash (the mul1 codebook named in the config). The encoder picks the path for a whole tile at once, so it behaves like vector quantization in 256 dimensions. You can read the bit width off the tensor shapes: a routed expert’s trellis tensor holds 48 int16 per tile, 768 bits for 256 weights, exactly 3 bits each. The output head’s holds 96, so 6.

Why that helps, from the QTIP paper: on Gaussian data at 2 bits per value, the best scalar quantizer reaches a mean squared error of 0.118 and QTIP’s trellis 0.069, against a theoretical floor of 0.063. On Llama-2-70B without fine-tuning, Wikitext2 perplexity goes from 3.12 at FP16 to 3.16 at 4 bits and 3.27 at 3 bits. The same paper also reports a serious zero-shot failure on Llama 3 70B, traced to one tensor, which is a good reason to test the model you actually run.

For this model, turboderp published mean KL divergence against the unquantized model on a GLM-5.3-Flash trace of about 74k tokens (model card):

The bit widths are read off the chart’s axis and exclude the embeddings and output head; the KL values are printed on it. On this trace, EXL3 3.05 has about half the divergence of the GGUF quant at a similar layer bit budget. It’s close to UD-IQ4_XS, which spends about 0.8 more bits per weight, but slightly above it.

Three limits on reading that:

• Q4_K_M wasn’t measured on this model. In separate tests on other models, Q4_K_M at about 4.8 bits had 2.4x to 3.5x lower KL than EXL3 at 3.0 (Gemma 4 12B, Qwen3-30B-A3B). So I can’t claim this quant matches Q4_K_M, and it probably doesn’t.
• KL against the reference measures how far the output distribution moves, not task accuracy. It also measures the quant alone, not my CPU offload on top of it, and the two can’t simply be added.
• Z.ai releases the model with most weights stored as FP8, so the “unquantized” reference is presumably those FP8 weights, itself a compressed release.

On “anything under 4 bits”: the evidence above says the bit count alone doesn’t decide it. There’s also published work showing MoE expert weights holding up at low bit widths. In the MoQE paper’s translation MoE (5.3B parameters, 32 experts), BLEU stayed “within -0.3 down to 3-bit”, while a dense model “loses most of capability” at 2 bits (arXiv 2310.02410). Those are other models and other tasks. They show why a universal four-bit cutoff is unreliable; they don’t establish the quality of this checkpoint.

The cost of trellis coding is arithmetic: every weight needs a few operations to decode. A GPU absorbs that easily. A Broadwell core doesn’t, which is the CPU bottleneck from the previous section.

§How it got here

All dates 2026.

• Oct 2, stock. Native install, exact mode, quality panel identical to stock exllamav3: 4.13 tok/s. The same design does 12.57 on its author’s EPYC with PCIe 4.0; this box has half the PCIe bandwidth and half the memory channels per socket, at a lower speed.
• Oct 2, the CPU joins. fast mode on one socket gave 7.98 tok/s. It was compute-bound at ~16 GB/s, so both sockets came next: 12.3.
• Oct 2 to 10, tuning. A refit of the scheduler’s cost model (+5%), a second CPU-friendly copy of the experts (+3%), finer CPU work units, and decode lookahead ported from glm-flash-lite (+3.3%) took it to 15.2 on the standard benchmark. Things that didn’t work: hyperthreads (−9%), a Broadwell-specific compiler target (0%), and multi-token prediction (−20% with one draft token, −56% with three). Draft tokens share almost no experts, so each draft costs nearly a full token’s expert reads.
• A correction. I first credited 2 MB pages with a 22% prefill gain. That came from comparing against a run from a different session; a same-session control cut it to 4 to 5%, and decode was a tie throughout. I now design every comparison against that mistake.
• Oct 11 to 12. Measurement got better. Same prompt, greedy decoding and interleaved windows took per-window noise from about ±5% to 1.3%. That’s how the 15.19 average (14.99 to 15.50) in the summary was measured.

§Working configuration

Upstream’s design and launch settings are in the glm53-flash-offload README; the project has since been renamed glm-flash-lite. The single-GPU profile measured above, as environment settings. Lines marked “patch” are my changes (or ports) on top of upstream, published in the broadwell-pcie3 branch of my fork. Its BROADWELL.md has the full environment, including the base settings upstream’s launcher sets:

GLM53_MODE=fast            # CPU tier on: compute cold experts in RAM instead of reading them over PCIe
GLM53_CT_CPUS=2-43         # 42 physical cores across both sockets; cores 0-1 left free
GLM53_CT_SWZ=1             # second copy of the experts laid out for the CPU kernel (+114 GB RAM, +3%)
GLM53_CT_B=0.31            # scheduler cost model: CPU time per expert, refit from measurements
GLM53_CT_TZC=1.0           # cost of a GPU zero-copy read, same units
GLM53_CT_SPLIT1=2          # patch: gate and up projections as separate CPU work units
GLM53_CT_SOCKQ=1           # patch: per-socket work queues
GLM53_CT_P0BLK=1           # patch: blocked first phase in the CPU kernel
GLM53_ZC_THP=1             # patch: 2 MB pages for the pinned expert arena (prefill +4-5%, decode tie)
GLM53_LA=1                 # patch (ported from glm-flash-lite): decode lookahead, +3.3%
GLM53_CT_SWIGLU_CLAMP=1    # patch: clamp the activation as the model config specifies (correctness)
GLM53_EC_BG=1              # patch: refill the VRAM expert cache in the background
GLM53_CT_FUSE=1 GLM53_CT_FUSE_MAXN=1 GLM53_CT_PF_LEAN=1   # patch: CPU job overhead trims (<1%)

§Reproducing this

What you’d need, beyond the config above:

• The code. Upstream’s settings alone (the first five lines of the config block) measured about 13.1 to 13.3 tok/s on this box. The rest needs the broadwell-pcie3 branch, which is upstream commit 1966a4f plus my patches, not rebased onto current glm-flash-lite. Upstream ignores the patch settings silently, so setting them on stock code does nothing.
• Hardware. A 24 GB GPU (built and tested on an RTX 3090, CUDA arch 8.6) and an x86 CPU with AVX2, FMA and F16C. The model is the 3.05bpw branch of turboderp/GLM-5.3-Flash-exl3, not main. turboderp’s card asks for exllamav3 1.4.5 or newer.
• RAM. About 225 GB for the serving process with the second expert copy (GLM53_CT_SWZ=1). With GLM53_CT_SWZ=0 it needs about 115 GB less and ran about 3% slower here.
• A raised memlock limit. About 115 GB of RAM gets pinned for the GPU, so set ulimit -l unlimited or Docker’s --ulimit memlock=-1.
• Your own core list. GLM53_CT_CPUS=2-43 is this box’s physical cores minus 0 and 1. List physical cores only; adding hyperthread siblings was 9% slower here.
• Your own cost model. GLM53_CT_B and GLM53_CT_TZC are the CPU’s and the PCIe path’s time per expert, and they decide which misses the CPU takes. Mine fit 42 Broadwell cores on PCIe 3.0. Refit them so the CPU job time and the GPU’s wait on it, both reported by /stats, come out about equal.
• Patience. Loading takes about six and a half minutes, and speed climbs for a while as the VRAM expert cache warms up.
• Different numbers. Upstream’s author gets about 28 tok/s on an EPYC with eight memory channels and PCIe 4.0. Your CPU, memory channels and PCIe generation decide where you land.

§What I’d tell you to steal

1. Compute where the weights already are. exact mode moved cold experts to the GPU and topped out at 4.13 tok/s on PCIe 3.0. Computing them in RAM on old CPUs nearly quadrupled it.
2. Validate the numeric path you actually serve. My quality panel exercises the CPU kernel’s multi-token arithmetic, but single-stream decode takes a different integer path. A passing panel told me less than I first assumed.
3. Don’t trust a gain measured across sessions. My 22% hugepage prefill win was 4 to 5% in a same-session control.
4. Make the noise small before chasing small wins. Same prompt, greedy decoding and interleaved windows cut per-window noise from about 5% to 1.3%, which is what separates a real +1% from nothing.
5. Fit the scheduler to measurements, not guesses. The CPU cost per expert was double my estimate on day one; refitting the cost model was worth 5%.

§Still open

• My own quality measurement of this quant against the original model, on coding and firmware-analysis tasks with objective pass/fail outcomes. Separately, the extra error from the CPU offload on the real decode path.
• Decode speed and time-to-first-token with long occupied histories, not just a configured 131k capacity.
• Concurrency. One stream at a time; the CPU tier is already the critical path for one.
• Whether the 4.05 bpw quant (half the KL) is worth it here. Each expert gets a third bigger: about 38 GB more per copy, and with two copies ~76 GB more RAM, on a box I also use for other models. I expect it to change throughput; I haven’t measured the direction or size.

§Credits

• 0xSero for the offload design (GPU expert cache plus a CPU expert tier) that all of this builds on. My work was porting it natively and tuning it for Broadwell and PCIe 3.0.
• turboderp for exllamav3, EXL3 and this quant, and for publishing per-format KL measurements on the model card, which is the only reason I can say anything about its quality yet.
• The QTIP authors (Tseng, Sun, Hou, De Sa) for the trellis method EXL3 builds on.
• Z.ai for releasing GLM-5.3-Flash with open weights.