Benchmarks · Measured on our own rig
Measured tokens per second on dual RTX 4090s
Updated 22 July 2026 · 36 benchmark runs on record
Almost every "tokens per second" figure you will find online is either a vendor claim, a single cherry-picked run, or someone's impression. These are measured on one rig, with the same harness every time, and every result carries the exact configuration that produced it.
The rig: two modded RTX 4090s with 48GB each (96GB total), vLLM in Docker under WSL2. The harness records five things per run - single-stream decode, throughput across four concurrent requests, prefill rate, and time to first token on a 16k prompt both cold and repeated - and appends a JSON line with the full config to a results file. Nothing here is estimated.
The numbers
| Model | Quant | GPUs | Decode tok/s | 4 concurrent | Prefill tok/s | TTFT 16k cold / warm |
|---|---|---|---|---|---|---|
| LFM2.5-8B-A1B | native | 1 | 176.3 | 265.9 | 14,912 | 1.2s / 0.05s |
| Qwen3.6-35B-A3B | AWQ4 | 1 | 133.9 | 398.2 | 12,715 | 1.45s / 0.22s |
| Qwen3-Coder-Next 80B | AWQ8 | 2 | 118.4 | 314.8 | 6,229 | 2.65s / 0.14s |
| Qwen3.6-35B-A3B | FP8 | 1 | 87.5 | 301.2 | 10,130 | 1.83s / 0.2s |
| Qwen3.6-27B (+MTP) | AWQ4 | 1 | 60.9 | 134.1 | - | - |
| Ornith-1.0-9B | native | 1 | 54.6 | 195 | 8,305 | 2.23s / 0.12s |
| Qwen3-VL-32B | AWQ | 1 | 45 | 171.2 | 1,884 | 8.76s / 0.07s |
| Gemma 4 31B | QAT W4A16 | 1 | 42 | 134.1 | 2,093 | 9.99s / 0.13s |
| Qwen3.6-27B | FP8 | 1 | 36 | 99.2 | 3,290 | 5.62s / 0.86s |
| gemma-4-12B-coder | BF16 | 1 | 34.1 | 114.1 | 5,329 | 3.92s / 0.08s |
| Gemma 4 31B | FP8-dynamic | 1 | 17.8 | 80.9 | 3,224 | 6.49s / 0.11s |
| Qwen3-VL-32B | FP8 | 1 | 16.3 | 72.8 | 2,677 | 6.17s / 0.1s |
Best run per model and configuration. Decode is single-stream on a short prompt; concurrent is aggregate across four parallel requests.
Architecture beats size, and it is not close
An 8B model tops this table at 176 tokens per second. An 80B model sits third at 118. A dense 27B manages 61 at its best and 36 at stock settings. If parameter count drove speed, that ordering would be impossible.
What actually drives it is how many parameters are active per token. The models at the top are mixture-of-experts designs: the 35B has roughly 3B active, the 80B coder similar. They hold a lot of knowledge in memory but only wake a fraction of it for any given token. A dense 27B wakes all 27 billion, every time.
The buying implication is worth stating plainly, because it changes what you should shop for: you are not buying VRAM to fit the biggest model, you are buying VRAM to fit the biggest MoE model. Those are different shopping lists, and the second one gets you far more capability per pound.
Quantisation is the biggest free lever most people never pull
Same model, same card, same everything except the quantisation format:
| Model | Format A | Format B | Speed-up |
|---|---|---|---|
| Qwen3-VL-32B | AWQ 45 | FP8 16.3 | 2.76x |
| Gemma 4 31B | QAT W4A16 42 | FP8-dynamic 17.8 | 2.36x |
| Qwen3.6-27B | AWQ4 +MTP 60.9 | FP8 36 | 1.69x |
| Qwen3.6-35B-A3B | AWQ4 133.9 | FP8 87.5 | 1.53x |
Four models tested, four wins for the 4-bit AWQ or W4A16 build over FP8, ranging from 1.53x to 2.76x. On Ada-generation cards this is consistent enough to treat as a rule: if a good AWQ or QAT quantisation of your model exists, start there. It costs nothing, it frees VRAM as a side effect, and the quality difference at 4-bit is small enough that we stopped worrying about it months ago.
The 80B on two cards
This is the run we were most curious about. Qwen3-Coder-Next is an 80B model that cannot fit on a single 48GB card at 8-bit, so it has to be split across both. Split across two 4090s over PCIe, with peer-to-peer disabled because it hangs on consumer boards, it still returns 118.4 tokens per second, 314.8 across four concurrent requests, and prefills at 6,229 tokens per second.
The detail that surprised us: long context costs nothing. Decode with a full 16k prompt loaded measured 122.6 tok/s against 118.4 on a short prompt - the same number within run-to-run variance. An 80B coding model at conversational speed, holding a large context, on two consumer graphics cards.
Prefix caching is close to free money
Time to first token on an identical 16k prompt, cold versus repeated: the 12B coder goes from 3.92s to 0.08s. The 80B goes from 2.65s to 0.14s. The 35B from 1.45s to 0.22s.
If your workload repeats a system prompt or a set of tool definitions on every request - which describes most agent and coding setups - enabling prefix caching will do more for the way the thing feels than changing model will. It is one flag.
Power capping the card costs nothing at the top
Same model, same settings, only the board power limit changed: at 370W the 35B returned 133.1 tok/s, and at 330W it returned 133.9. Forty watts for nothing, and the second figure is marginally the faster of the two, which tells you the difference is noise rather than a real cost.
That matters for anyone leaving a machine inferencing around the clock. We have not yet found where the curve actually bends - the sweep below 330W is the next test on the list.
Method, and what we have not proven
One harness, one rig, every config recorded alongside its result. Runs that contradict expectations are kept rather than dropped, and where a result is contaminated it is labelled rather than quietly rounded off.
Two things we deliberately are not claiming yet:
- Whether a second GPU helps a model that already fits on one. We have a run that appears to show a dramatic slowdown when splitting a 35B across both cards, but that run also carried a flag which disables CUDA graphs, and that flag alone is a large penalty. Until the clean comparison is run - same model, same flags, one card versus two - we cannot honestly separate the two effects, so the number stays out of the table.
- Anything about other hardware. These figures describe two 4090s. They do not tell you what a 5090, a Mac Studio or an RTX PRO 6000 will do, and we will not pretend otherwise. Those come as we get hands on the cards.
If you want the same numbers for your own machine, the harness is not complicated: fixed prompts, a warm-up call, then decode, concurrency, prefill and cold/warm TTFT, appending the config with the result every time. The discipline matters more than the tooling.