The short answer on a 24 GB card
On a 24 GB card, a 4-bit 30B-class build such as the 16.47 GiB 30B MoE Q4_K_XL fits in VRAM. The RTX 4090 official page prints 24 GB GDDR6X, a 384-bit memory interface, 16384 CUDA cores and a 450 W TGP, with “Starting at $1599.00” (NVIDIA). Hardware-Corner’s March 2026 matrix prints the RTX 4090 at 24 GB / 1,008 GB/s and the RTX 3090 at 24 GB / 986 GB/s (Hardware Corner GPU index).
The practical answer is not “which model fits” but “how much conversation fits with it.” On the RTX 4090, a 32B Q4_K_XL build at 18.64 GiB has printed numbers only through 16K context; a 30B MoE Q4_K_XL at 16.47 GiB runs through 57K; a 14B Q4_K_XL at 8.53 GiB through 86K; an 8B Q4_K_XL at 4.79 GiB through 131K (Hardware Corner RTX 4090 benchmarks).
So the rule is: pick the largest 4-bit model whose printed context ceiling covers your real conversations, then check the measured speed at that context, not at 4K.
Six terms, defined once
VRAM is the memory on the graphics card itself. It holds model weights, the KV cache and runtime overhead; when it runs out, the model spills to system memory or fails with an out-of-memory error.
Quantization is the compression scheme used for weights. Q4_K_M, Q4_K_XL and AWQ INT4 are different 4-bit formats; Q8_0 and FP8 are higher-precision ones. The label matters: the same 32B model prints at 19.9 GB in Q4_K_M and 34.8 GB in Q8_0 (Qwen2.5-32B GGUF).
KV cache holds attention keys and values for tokens already in context, so it grows with context length: Gigagpu prints FP8 KV @ 16k at 4.0 GB and FP8 KV @ 8k at 2.0 GB for a 32B model (Gigagpu 32B benchmark).
Context length is how many tokens the model sees at once — your prompt plus the conversation so far — measured in K, where 4K means roughly four thousand tokens.
Tokens per second (t/s) is generation speed, how fast the model writes text; prompt processing is a separate speed, how fast it reads your prompt.
MoE means mixture of experts: a 30B MoE model activates only part of its parameters per token, which is why the 30B MoE Q4_K_XL row prints 195.78 t/s at 4K while the dense 32B row prints 38.89 t/s (Hardware Corner RTX 4090 benchmarks).
The VRAM budget, as an equation
weights + KV cache + activations and runtime overhead = peak VRAM
Gigagpu’s printed component table makes that concrete for a 32B model on an RTX 4090. For AWQ INT4: 17.9 GB of weights + 4.0 GB of FP8 KV @ 16k + 1.4 GB of activations and CUDA scratch + 0.6 GB reserved by the vLLM scheduler = the printed ~23.9 GB peak at 16k context, b=1 (Gigagpu). The same table prints GGUF Q4_K_M at ~28.5 GB peak and FP8 (does not fit) at ~38 GB.
Computed against 24 GB: GGUF Q4_K_M at 28.5 GB is 4.5 GB over; GPTQ INT4 at 24.3 GB is 0.3 GB over, which the source labels “(overflow)”. The weights can fit while the full runtime does not (Gigagpu).
What fits: the 24 GB comparison table
For Hugging Face rows, “fits” means the file size alone fits; no runtime VRAM figure for that build exists in the sources. The two Qwen 2.5 32B Q4_K numbers also differ: Hugging Face prints a 19.9 GB file and Gigagpu prints 19.1 GB for its weights row, with no explanation in either source (HF, Gigagpu).
| Model | Quantization (verbatim) | Printed size | Fits 24 GB? | Measured context ceiling | Measured t/s | Source |
|---|---|---|---|---|---|---|
| 8B benchmark model | Q4_K_XL | 4.79 GiB | Yes | 131K | 131.00 at 4K | Hardware Corner |
| 14B benchmark model | Q4_K_XL | 8.53 GiB | Yes | 86K | 82.82 at 4K | Hardware Corner |
| 30B MoE benchmark model | Q4_K_XL | 16.47 GiB | Yes | 57K | 195.78 at 4K | Hardware Corner |
| 32B benchmark model | Q4_K_XL | 18.64 GiB | Yes, through 16K | 16K | 38.89 at 4K | Hardware Corner |
| Qwen3.8 27B | Q4_K Small | 16.68 GiB | Yes | 64k tested | 46.16 at 4k | Hardware Corner |
| Qwen 2.5 32B (Gigagpu) | AWQ INT4 | 17.9 GB weights | Yes at b=1, 16k peak ~23.9 GB | 16k (OOM at 32,768) | 65 at b=1 | Gigagpu |
| Qwen2.5-32B-Instruct | Q4_K_M | 19.9 GB | File size fits; runtime not established | Not established | Not measured | bartowski GGUF |
| Qwen2.5-32B-Instruct | Q8_0 | 34.8 GB | No | Not established | Not measured | bartowski GGUF |
| Llama-3.3-70B-Instruct | IQ1_M | 16.8 GB | File size fits; runtime not established | Not established | Not measured | bartowski GGUF |
| Llama-3.3-70B-Instruct | Q4_K_M | 42.5 GB | No | Not established | Not measured | bartowski GGUF |
Measured speed on the RTX 4090 and RTX 3090
Token generation on the RTX 4090
The November 6, 2025 benchmark page prints this token-generation table, measured with ./llama-bench -m /home/allanw/model-name -fa 1 -d 4098,8196,32768,45062,139247 -p 2048 -n 128 -ngl 99 (Hardware Corner):
| Model | Size (GiB) | 4K | 8K | 16K | 32K | 45K | 57K | 65K | 86K | 131K |
|---|---|---|---|---|---|---|---|---|---|---|
8B Q4_K_XL | 4.79 | 131.00 | 119.36 | 96.12 | 77.42 | 66.18 | 62.81 | 53.07 | 43.01 | 32.27 |
14B Q4_K_XL | 8.53 | 82.82 | 77.46 | 63.21 | 55.49 | 48.11 | 42.99 | 38.73 | 32.87 | – |
30B MoE Q4_K_XL | 16.47 | 195.78 | 171.94 | 130.11 | 102.57 | 92.12 | 74.64 | – | – | – |
32B Q4_K_XL | 18.64 | 38.89 | 36.94 | 33.83 | – | – | – | – | – | – |
The March 2026 matrix prints its own token-generation grid for the same card (Hardware Corner RTX 4090 matrix):
| Model | 4k Ctx | 16k Ctx | 32k Ctx | 64k Ctx | 128k Ctx | 256k Ctx |
|---|---|---|---|---|---|---|
| Qwen3 8B (Q4_K) | 141.3 | 108.0 | 82.3 | 56.1 | 33.8 | — |
| Qwen3 14B (Q4_K) | 84.4 | 69.8 | 55.4 | 38.8 | — | — |
| gpt-oss 20B (MXFP4) | 190.6 | 163.3 | 140.7 | 104.0 | 70.3 | — |
| Qwen3 30B A3B (Q4_K) | 207.3 | 139.2 | 105.1 | 68.2 | — | — |
| Qwen3 32B (Q4_K) | 39.6 | 34.4 | — | — | — | — |
Prompt processing on the RTX 4090
The November 2025 page prints prompt-processing speed separately (Hardware Corner) — how fast the card reads a long prompt, and why a long document can take tens of seconds before the first output token:
| Model | Size (GiB) | 4K | 8K | 16K | 32K | 45K | 57K | 65K | 86K | 131K |
|---|---|---|---|---|---|---|---|---|---|---|
8B Q4_K_XL | 4.79 | 9121 | 7907 | 5697 | 4224 | 3410 | 2967 | 2614 | 1951 | 1451 |
14B Q4_K_XL | 8.53 | 5358 | 4621 | 3355 | 2511 | 2124 | 1667 | 1545 | 1190 | – |
30B MoE Q4_K_XL | 16.47 | 6266 | 5415 | 3802 | 2726 | 2290 | 1718 | – | – | – |
32B Q4_K_XL | 18.64 | 2393 | 2093 | 1685 | – | – | – | – | – | – |
The printed command line lists five -d values (4098,8196,32768,45062,139247) while the tables show nine context columns; the source does not reconcile them (Hardware Corner).
The RTX 3090: same memory, slower card
The March 2026 RTX 3090 matrix prints token generation with CUDA, -fa 1 tags on several rows (Hardware Corner RTX 3090 matrix):
| Model | 4k Ctx | 16k Ctx | 32k Ctx | 64k Ctx | 128k Ctx | 256k Ctx |
|---|---|---|---|---|---|---|
| Qwen3 8B (Q4_K) CUDA, -fa 1 | 115.3 | 87.5 | 67.9 | 46.6 | 28.1 | — |
| Qwen3 14B (Q4_K) CUDA, -fa 1 | 70.0 | 52.1 | 38.6 | 25.4 | — | — |
| gpt-oss 20B (MXFP4) CUDA, -fa 1 | 147.5 | 128.5 | 112.6 | 89.6 | 62.2 | — |
| Gemma4 26B (Q4_K) | 119.4 | 115.0 | 107.5 | 98.9 | 83.0 | 64.4 |
| Qwen3.5 27B (Q4_K) | 33.5 | 32.3 | 31.0 | 28.8 | — | — |
| Qwen3 30B A3B (Q4_K) CUDA, -fa 1 | 153.6 | 113.8 | 87.2 | 58.3 | — | — |
| Gemma4 31B (Q4_K) | 34.7 | 33.5 | 31.4 | — | — | — |
| Qwen3 32B (Q4_K) CUDA, -fa 1 | 35.1 | 30.3 | — | — | — | — |
| Qwen3.5 35B (MXFP4) | 111.2 | 107.1 | 101.2 | 93.1 | 79.4 | — |
The 3090 page header prints Gen (14B 4-bit) 52.1 t/s and PP (14B 4-bit) 1,679 t/s; those match the 16k table values, not the header’s “4k” label, and the page does not explain the mismatch (Hardware Corner RTX 3090 matrix).
How speed falls as context grows
For the 32B Q4_K_XL row, token generation goes from 38.89 t/s at 4K to 33.83 t/s at 16K. Computed from those two printed inputs: (38.89 − 33.83) / 38.89 = 13.0% drop (Hardware Corner).
Source disagreements stay visible: the November 2025 page prints 82.82 t/s at 4K for 14B Q4_K_XL while the March 2026 matrix prints 84.4 t/s at 4k for Qwen3 14B (Q4_K), and the matrix page header prints Gen (14B 4-bit) 78.7 t/s against its own table’s 84.4 at 4k and 69.8 at 16k (detail, matrix).
Context is the real ceiling on 24 GB
The Qwen3.8 27B page states it directly: “The model itself fits on a 24 GB GPU, but the usable context length depends on the additional memory required by the KV cache and runtime overhead” (Hardware Corner Qwen3.8 27B tests). The measured curve for Q4_K Small at 16.68 GiB shows why:
| Context length | Measured VRAM usage |
|---|---|
| 4k | 18 GB |
| 8k | 18 GB |
| 16k | 19 GB |
| 32k | 20 GB |
| 64k | 22 GB |
| 128k | 26 GB |
| 256k | 34 GB |
Source: Hardware Corner Qwen3.8 27B tests.
Computed against 24 GB: at 128k the printed 26 GB is 2 GB over; at 256k the printed 34 GB is 10 GB over. The same page marks 128k and 256k as not practical on 24 GB and gives 64k as the practical target for both the RTX 3090 and RTX 4090.
The 32B class shows the same wall: Gigagpu prints 16,384 prompt tokens → 23.9 GB peak, and 32,768 prompt tokens → OOM, with the note “Activations push past 24 GB” (Gigagpu).
There are two documented levers. In llama.cpp’s server, -c, --ctx-size N sets the prompt context and -ctk, --cache-type-k TYPE / -ctv, --cache-type-v TYPE set the KV cache data types (env LLAMA_ARG_CACHE_TYPE_K / LLAMA_ARG_CACHE_TYPE_V) (llama.cpp server README). In Ollama, the default context is 4096 tokens, overridden with OLLAMA_CONTEXT_LENGTH (Ollama FAQ). Raising context without watching VRAM is how a working setup becomes an out-of-memory error.
What does not fit, and what nobody measured
A 70B model is the clearest “it depends on the file”: Llama-3.3-70B-Instruct IQ1_M is 16.8 GB and fits on file size alone, while Q4_K_M is 42.5 GB and does not (bartowski GGUF). Fit is not usability, though: the RTX 4090 matrix FAQ answers its own “Can the RTX 4090 run 70B models?” with “No, it can’t run 70B models at any quantizatio” (4090 matrix), while the RTX 3090 page says a 70B can run with CPU offloading but that “generation speed will usually drop to single-digit tokens per second” (3090 matrix). A 32B model in FP8 is out too: the printed weights are 32.0 GB, 8.0 GB over 24 GB before any KV cache, and the same page prints a ~38 GB peak at 16k for that build (Gigagpu).
One line needs correcting rather than repeating. The RTX 4090 matrix page prints “Handles gpt-oss 120B model in MXFP4 quantization at full128K context” (Hardware Corner RTX 4090 matrix), but Ollama lists gpt-oss:120b at 65GB with a 128K context (Ollama gpt-oss tags) — roughly 41 GB over 24 GB on file size alone, so that line cannot describe a single 24 GB card.
24 GB vs 16 GB vs 32 GB
The tier question is capacity first, bandwidth second. The printed anchors are RTX 3090 24 GB / 986 GB/s, RTX 3090 Ti 24 GB / 1,008 GB/s, RTX 4090 24 GB / 1,008 GB/s, RTX 5080 16 GB / 960 GB/s and RTX 5090 32 GB / 1,792 GB/s (Hardware Corner GPU index). Labels disagree across sources: GDDR6X on the NVIDIA and detail pages, “24 GB GDDR6” on the matrix header, and 1.01 TB/s against 1,008 GB/s (NVIDIA, detail, matrix).
A 16 GB card can be fast and still be the wrong tier for 32B-class 4-bit work. Gigagpu’s cross-GPU table prints RTX 5080 16GB | 16 GB | does not fit | 0 | — for the 32B AWQ INT4 build that costs “23.5 GB (tight)” on an RTX 4090 at b=4 (Gigagpu).
A 32 GB card buys context, and speed only on some workloads. The Gigagpu table prints RTX 5090 32GB | 32 GB | 105 | 8 | 360 against RTX 4090 24GB | 24 GB | 65 | 4 | 180 for the same 32B AWQ workload at b=1 and b=4 aggregate (Gigagpu). But the Qwen3.8 27B page measures the RTX 4090 faster than the RTX 5090 at 64k — 38.41 against 26.22 t/s — so more VRAM is not automatically more speed (Hardware Corner).
On price, the printed average market value is $1,000 for the RTX 3090 against $2,200 for the RTX 4090, with Price/GB VRAM of $41.67 against $91.67 and Price/(t/s) with 14B @ 16k of $19.18 against $27.95 (3090 matrix, 4090 matrix). NVIDIA still lists the 4090 at “Starting at $1599.00” (NVIDIA), and the November 2025 page puts retail near $2500 and second-hand closer to $2100 (detail). No price is printed for the 3090 Ti, 5080, 5090, 5070 Ti or RX 7900 XTX.
The bottom line: what to run and what to buy
For speed at long context, run gpt-oss 20B (MXFP4): the March 2026 matrix prints 190.6 t/s at 4k and 70.3 t/s at 128k on the RTX 4090 (Hardware Corner RTX 4090 matrix). For the fast middle, run Qwen3 30B A3B (Q4_K), at 207.3 t/s at 4k and 68.2 t/s at 64k, or the 30B MoE Q4_K_XL build at 195.78 t/s at 4K and 74.64 t/s at 57K (matrix, detail).
For the densest model that fits at short context, run a 32B Q4_K_XL build: 38.89 t/s at 4K and 33.83 t/s at 16K, which suits chat and short documents rather than long ones. Context ceilings are model-specific, not card-specific: Qwen3.8 27B needs about 26 GB at 128k on this class of card, while Meta’s Muse Glimmer 30B holds 256k in about 20 GB and stays faster at every comparable length (45.30 against 40.31 t/s at 4k on the RTX 3090) (Hardware Corner).
On the card question, Hardware-Corner’s RTX 3090 verdict is that it is “the most cost-effective second-hand entry point for anyone serious about local LLM work” (Hardware Corner RTX 3090 matrix). Its RTX 4090 verdict adds that “the practical difference in daily use wasn’t massive”, that both cards handle 34B-parameter models at 4-bit, and that it still points “toward a second-hand 3090 as the smarter buy for value-focused users” (Hardware Corner RTX 4090 matrix). The printed economics back that up: $1,000 average market value and $19.18 per t/s for the 3090 against $2,200 and $27.95 for the 4090.
That is the buying verdict: 16 GB rules out the 32B Q4/AWQ class, 24 GB is the value tier for 4-bit 30B-class work, and 32 GB is what you buy when your chosen model’s own VRAM curve crosses 24 GB before your context target does. The 4090 is faster, but the 3090 has the same capacity, and capacity plus model choice decides what fits.
FAQ
Does a 70B fit on 24 GB?
Only on file size, for the smallest builds: IQ1_M is 16.8 GB, Q4_K_M is 42.5 GB (bartowski GGUF). The RTX 4090 matrix FAQ says flatly “No, it can’t run 70B models at any quantizatio”, and the RTX 3090 page expects CPU offloading with “generation speed will usually drop to single-digit tokens per second” (4090 matrix, 3090 matrix).
Can I run gpt-oss 120B on 24 GB?
No. Ollama lists gpt-oss:120b at 65GB, which is about 41 GB over 24 GB before any KV cache (Ollama). The matrix page’s “Handles gpt-oss 120B model in MXFP4 quantization at full128K context” line therefore does not describe a single 24 GB card (Hardware Corner).
How do I raise the context window in Ollama?
Ollama uses 4096 tokens by default; override it with OLLAMA_CONTEXT_LENGTH=8192 ollama serve (Ollama FAQ). Watch VRAM as you raise it: the Qwen3.8 27B curve goes from 18 GB at 4k to 26 GB at 128k (Hardware Corner).
What about AMD?
The Radeon RX 7900 XTX is a 24 GB GDDR6 card (AMD), but no AMD throughput, bandwidth or price figure exists in the sources, so no AMD performance claim is made here.
How this guide was built
This is documentation of published benchmarks and vendor pages, not hands-on testing; every figure links inline to its source. Benchmark vintages are November 6, 2025 (RTX 4090 detail), March 2026 (RTX 4090 matrix, RTX 3090 matrix) and August 17, 2026 (Qwen3.8 27B tests). Where sources disagree, both figures are printed rather than averaged.
Back to all posts