Your 8GB Card Can Run a 35B Model. llama.cpp Won't Turn On the Part That Makes It Fast
llama.cpp now picks the GPU layers itself. The two settings it leaves to you decide whether a 35B model is usable on 8GB, and one ships switched off.
Almost every guide that tells you how to run a 35B model on 8GB VRAM opens the same way: pass -ngl 99, push as many layers onto the card as will fit, and hope. On a current build that advice is worse than useless. llama.cpp’s own documentation gives -ngl a default of auto and describes --fit, which is on by default, as adjusting the arguments you leave unset. Typing a number is how you opt out of the sizing pass that would have worked it out for you.
Two settings genuinely are yours. One has been there since August 2025 and is the reason a 35-billion-parameter model runs on a laptop card at all. The other arrived on 7 October, is the reason it can run quickly, and ships switched off.
What llama.cpp already decides for you
The relevant default is -fit, documented as “whether to adjust unset arguments to fit in device memory”, with a default of on. With it enabled — which is to say, unless you turn it off — llama.cpp measures your device memory and resolves the arguments you did not set. That includes the number of GPU layers and the context size. Two companion flags bound its choices: -fitt keeps a margin free on each device, 1024 MiB by default, and -fitc sets the smallest context it is allowed to pick, 4096 by default.
It is worth being fair about how far this goes, because it is further than most people assume. Reading common/fit.cpp, the pass does not only count whole layers. When one more layer nearly fits, it splits that boundary layer by fraction, and one of those fractions is described in the source as “everything but sparse MoE weights” — the attention of the layer goes to the GPU while its experts stay behind. llama.cpp is already doing a one-layer, automatic version of the thing the guides tell you to do by hand.
What it does not do is the two things below. The fitting pass reads the placement overrides you supply; it never invents them, and it never enables the expert cache. Those remain manual, and on a small card they are where the result is decided.
Why a 35B model fits on 8GB of VRAM at all
The arithmetic is worth doing once, because it explains when the technique applies and when it will not.
Take Qwen3.6-35B-A3B, the model most of this year’s small-card guides are written around. Its published config says 40 hidden layers, 256 experts, 8 experts used per token, a hidden size of 2048 and an expert intermediate size of 512. Each expert is three matrices — gate, up and down. So the routed expert weights come to 40 × 256 × 3 × 2048 × 512, which is 32.2 billion parameters out of a model of roughly 35 billion. Around 92% of this model is experts.
And per token the router picks 8 of the 256. That is about a billion parameters of expert weight actually touched for any given token, out of thirty-two billion sitting there.
That ratio is the entire opportunity. The 8% that runs on every single token — attention, the shared expert, the embeddings, the output head — is small enough to live on a modest card. The 92% that is mostly idle can live in system RAM and be read as needed. Splitting a model by layer, the old way, cuts across that grain and drags attention onto the CPU with the experts. Splitting it by tensor type does not.
Treat the 92% as derived rather than measured, because it is: it comes from the parameter counts in Qwen’s config, and quantisation is not uniform across tensors. Unsloth’s dynamic quants deliberately keep some tensors at higher precision, so the byte split in a given file will not match the parameter split exactly. Your own figure comes out of llama.cpp’s load log, not out of my sum.
The file sizes are concrete, though. Unsloth’s GGUF repository for this model lists the Q4_K_M build at 20.61 GiB, the IQ4_XS at 16.51 GiB and a Q2_K_XL at 11.45 GiB. Hold on to those numbers; they come back at the end.
The flag that is still yours to set
Two forms, both quoted from llama.cpp’s own argument reference. -cmoe, --cpu-moe will “keep all Mixture of Experts (MoE) weights in the CPU”. -ncmoe, --n-cpu-moe N will “keep the Mixture of Experts (MoE) weights of the first N layers in the CPU”. The all-or-nothing version is the blunt one; the counted version lets you leave some layers’ experts on the card and push the rest off.
For a dense model there is a sibling, -ncffn, --n-cpu-ffn N, which does the same job for ordinary feed-forward weights.
What nobody can give you is the right N. Searching for one on 10 November returned guides recommending confidently different values for the same card, and they are not all lying: each is describing its own machine, at its own quant and context length. The number depends on how much VRAM the rest of your configuration has already spent, which is a property of your setup rather than of the model. A borrowed N is the commonest way this goes wrong.
The one nobody has switched on
On 7 October, llama.cpp merged a GPU cache for experts that live in host memory. The idea is small and good: if the experts are in RAM, keep a pool of the hot ones in VRAM, serve the hits from the card, and only upload the misses.
--moe-cache-mib ships at default 0, disabled, while -ngl defaults to auto.The flag is --moe-cache-mib N, and the documented default is “0, disabled”. A second flag, --moe-cache-cpu-share F, decides what share of cache misses the CPU computes itself instead of uploading the expert to the GPU; its default is 0.00. Neither is on for anybody who has not gone looking.
For sizing, the merge request that introduced it says plainly: “In experiments it was found 10% of the total expert size is good.”
The project published its own before-and-after, and it is worth quoting with the hardware attached, because the hardware is most of the story. On an RTX 4090, serving a 93.7 GiB model with 65.4 GiB of experts from a 16-core EPYC 7742 over PCIe 4.0 x16, generation went from 25.0 tokens per second with the automatic fit to 40.7 with -cmoe --moe-cache-mib 11000 — 1.62×, at a 77% cache hit rate. On an RTX 5090 the same comparison ran 30.8 to 67.8, a 2.20× improvement at an 89% hit rate.
Do not carry those multipliers to your laptop. They describe two of the fastest consumer cards made, fed by a server CPU across a full-width PCIe 4.0 link, on a file more than four times the size of the one in this article. What transfers is the mechanism, not the number.
The mechanism does transfer, and the arithmetic is friendlier than you would guess. On the derived ratio above, ten percent of the expert weight in a 20.61 GiB Q4_K_M file lands somewhere around two gigabytes. With -cmoe sending the experts to RAM, the resident part of the model on the card is the small 8%, so a two-gigabyte cache is a plausible ask on an 8GB card rather than a fantasy. This is the rare feature that is more interesting on a small card than a large one, because on a large one you would simply have kept the experts in VRAM.
Why it speeds up generation and not prompt processing
This one will bite you if nobody says it, so here it is from the source rather than from a benchmark. In src/llama-moe-cache.cpp the cache carries a hard ceiling, max_batch = 32, and the lookup returns nothing above it. The comment explains why: “large batches use most experts of a layer, so they gain little from the cache and would evict the experts used in generation.”
Generating a reply processes a token or two at a time, which is exactly the regime the cache is built for. Reading a long prompt processes hundreds of tokens in a batch, which uses most of the experts anyway and would flush the cache on the way through. So the cache makes the model type faster. It does not make it read faster, and a benchmark that measures prompt processing will show you nothing. That is a design decision rather than a bug, and knowing it saves an evening.
Doing it, then measuring it
Four steps, in order.
First, check your build has it. The feature is weeks old and your package manager’s version may not. Run llama-server --help | grep moe-cache. If that prints nothing, the rest of this section does not apply yet and you need a newer build.
Second, look before you change anything. Start the server with -fitp on and let llama.cpp print what it thinks it needs and what it has decided. That estimate is your baseline, and it is where you read the real expert-versus-everything-else split for your file rather than trusting a ratio derived from a config.
Third, move the experts off the card with -cmoe, and leave -ngl alone. You are not fighting the fitting pass; you are telling it something it cannot work out for itself, and then letting it place what remains.
Fourth — and this is the step that makes the difference — stop guessing the cache size. On 7 November the project merged a tuning script for exactly this, because, in the author’s words, the two cache parameters are now “free” parameters. You put it in front of your normal command:
python3 scripts/moe-cache-tune.py -- build/bin/llama-server -m model.gguf -cmoe -t 8 -c 32768
It reads the model once so no run is penalised by disk, then makes six calibration passes across two cache sizes and three CPU-share values, printing measured tokens per second and hit rate for each. Then it reports what it learned about your machine — upload bandwidth, how fast the CPU reads experts, the per-token cost of each part — and prints a recommendation line. The example in the merge request ends: Recommended: --moe-cache-mib 24000 --moe-cache-cpu-share 0, 83.0 t/s.
Ten minutes of measurement on your own hardware beats any number you can copy, including the ones above.
If you would rather not build llama.cpp
Most people running a local model are not compiling one, and the two obvious apps differ here.
LM Studio has had the first half since 12 August 2025, when release 0.3.23 added “an advanced model load setting to place all MoE expert weights onto the CPU or GPU (default)”, recommended with the words “Try this for low VRAM machines”, and described by LM Studio as “utilizing the same underlying technology as llama.cpp’s --n-cpu-moe”. It now has the second half too: the changelog entry for Bionic 1.1.10 on 29 October 2026 reads “Experimental MoE expert GPU cache setting for llama.cpp MoE models” — three weeks after the upstream merge, and still labelled experimental. The runtime under it is versioned separately, so check which llama.cpp pack your install is on; Bionic 1.2.0 on 5 November lists 2.67.0.
Ollama does not expose either one. It places layers itself and gives you no equivalent control, which is a defensible product decision and an annoying one if you have read this far. It is not standing still on the underlying problem — the v0.40.16 release on 9 November reports Qwen3.8 Flash Next using about 27 GiB less memory on Apple silicon, and bumps its bundled llama.cpp — but there is no knob. If that is the trade you want, the full setup guide for running local models with Ollama still stands; just do not go looking for a flag that is not there.
What actually stops you
Here is the part the VRAM-focused guides bury, and the reason those file sizes mattered.
The experts have to live somewhere, and with -cmoe that somewhere is system RAM. Running the 20.61 GiB Q4_K_M build means having roughly twenty gigabytes of RAM free for it, on top of your operating system, your browser and whatever else is open. The 16.51 GiB IQ4_XS needs about sixteen and a half.
So the binding constraint is not the graphics card. A machine with an 8GB GPU and 32GB of RAM can do this. A machine with a much better GPU and 16GB of RAM cannot do it at all. If you are costing an upgrade to run bigger models locally, the honest answer this month is that you should buy RAM. That inversion is the most useful thing on this page, and almost never the first thing people check.
Three errors turned up in the first page of search results when I checked on 10 November.
The first is a version number. Several write-ups refer to “llama.cpp v0.7.0” and attach release notes to it. llama.cpp does not publish semantic versions; its releases are build tags, and b12239 went out on 10 November. A guide citing a version that does not exist has not looked at the project.
The second is a flag that belongs to someone else. A third-party fork implements a similar idea under --moe-hot-cache, and one well-ranked page documents that fork’s interface as though it were upstream. The upstream flags are the ones in the capture above. If --help does not list yours, you are reading about a different program.
The third is the borrowed -ncmoe value, which is the same mistake as the borrowed cache size and has the same fix.
One last thing, which is less a warning than a date stamp. Between 6 and 8 November the expert cache took five more merges: cache-replacement policy, draft-model behaviour, how CPU misses are chosen, how the CPU part is synchronised. This is a feature being actively built, not a settled one. Pin the build you tuned against, and re-run the tuning script after you update, because the thing you measured may not be the thing you are now running.
None of which argues against trying it. It argues for measuring rather than copying — the same posture that pays off with a 1.7B model claiming to replace a 9B one, or with speech to text on hardware too old to deserve it. The useful question was never how big a model you can load. It is which part of it has to be fast and where the rest can sit. Answer that and an 8GB card stops being the limit people treat it as, while what a self-hosted coding model really costs at datacentre scale stays a different question entirely.
Sources
Every figure in this article traces to one of these. Dates are when the page was read.
- llama.cpp server README — command-line argumentsggml-org/llama.cppAccessed Nov 10, 2026Official docs
- llama : add a GPU cache for MoE experts kept in host memory (PR #29887)ggml-org/llama.cppAccessed Nov 10, 2026Official docs
- llama_moe_cache: run missed experts on CPU, add tuning script (PR #30966)ggml-org/llama.cppAccessed Nov 10, 2026Official docs
- src/llama-moe-cache.cpp (master)ggml-org/llama.cppAccessed Nov 10, 2026Official docs
- common/fit.cpp (master)ggml-org/llama.cppAccessed Nov 10, 2026Official docs
- llama.cpp releasesggml-org/llama.cppAccessed Nov 10, 2026Official docs
- Qwen3.6-35B-A3B model configQwen (Alibaba) on Hugging FaceAccessed Nov 10, 2026Official docs
- unsloth/Qwen3.6-35B-A3B-GGUF file listingUnsloth on Hugging FaceAccessed Nov 10, 2026Official docs
- LM Studio 0.3.23 release notesLM StudioAccessed Nov 10, 2026Official docs
- LM Studio changelogLM StudioAccessed Nov 10, 2026Official docs
- Ollama release v0.40.16OllamaAccessed Nov 10, 2026Official docs
Written by
Full-stack developer who builds and runs AI automation systems in production. Runs local LLMs on personal hardware, builds N8N pipelines that actually ship, and deploys on Cloudflare Pages. Every guide on Pipeline Monk is tested on real consumer hardware — a Ryzen 7 5800HS with 16GB RAM and a GTX 1650. If it works on that, it works.