A Portable LLM on a USB Stick: What Actually Works

Every guide to running AI from a USB drive points at an archived project and misses the one tool built for exactly this. Here is the working setup.

Flat illustration of a memory stick shape in front of a large rounded square on a cream background
9 min read
1,872 words

A portable LLM on a USB stick is a genuinely good idea — a chatbot in your pocket that works on a machine with no internet, no account and no install, which is exactly the situation you are in when you are standing over a computer that will not boot. Search for how to build one and you will find half a dozen guides that all describe the same approach, all link the same project, and all miss the tool that was built for this.

The portable LLM tool every USB guide misses

It is called llamafile, it comes from Mozilla, and the premise is one sentence: a model and everything needed to run it, packaged as a single executable file.

It works by combining llama.cpp with Cosmopolitan Libc, which is what lets one file run across operating systems without being rebuilt for each. The project describes its own goal as collapsing all the complexity of LLMs down to a single-file executable, and that is not marketing — there is no runtime to install, no Python, no package manager, no model directory to configure. One file.

Using it is three steps on macOS, Linux or BSD:

curl -LO <llamafile-url>
chmod +x <the-file>
./<the-file>

On Windows you rename it to end in .exe and double-click. That is the whole procedure, and it is why this is the right answer for a drive you might plug into a stranger’s machine: nothing is installed, nothing is left behind, and nothing needs administrator rights to unpack itself into Program Files.

The project is Apache 2.0, with its changes to llama.cpp and whisper.cpp under MIT, and it sits at 26,500 stars.

The 4GB wall on Windows

There is one constraint that shapes everything else about this build, and it is worth knowing before you choose a model.

On Windows, only executables under 4GB will run. A llamafile larger than that simply will not start there. Since the file contains the model, that puts a hard ceiling on how big a model your single-file approach can carry — and it is a Windows limit, not a llamafile one, so there is no version that fixes it.

The documented workaround is to split the two apart: use the llamafile binary on its own and point it at external GGUF weights sitting next to it on the drive. You lose the single-file elegance and keep the portability, which for a rescue drive is the right trade. Two files on a stick is still a setup that needs no installation.

In practice the 4GB ceiling is less painful than it sounds, because the models worth carrying for this job are small anyway. A machine that will not boot does not need a frontier model; it needs something that knows what a boot sector is.

Choosing what fits

The README’s own quick-start example is a Qwen3.5 0.8B llamafile, which tells you the scale this is designed around: sub-billion to a few billion parameters, comfortably inside the Windows ceiling, and fast enough on a CPU to hold a conversation rather than a correspondence.

That is a good default for the rescue-drive job. A model of a few billion parameters at four-bit quantisation generally lands in the low gigabytes — check the actual file size before you commit to it — which leaves room inside a 4GB budget and loads quickly off the stick. Going bigger buys knowledge you mostly do not need in this situation and costs load time you feel every single launch.

If you do want a larger model, that is the moment to switch to the binary-plus-GGUF arrangement rather than fighting the single-file limit.

The project you will be sent to is archived

Here is the part that costs people an afternoon. The guides that rank for this topic nearly all point at the same repository, and it has not been maintained since the summer.

GitHub repository header showing a banner reading that the repository was archived by the owner on Jul 4, 2026 and is now read-only, beside a Public archive badge
USB-Uncensored-LLM on GitHub, captured 3 Nov 2026: archived since July, and still the project most guides send people to.

“This repository was archived by the owner on Jul 4, 2026. It is now read-only.” Its own README adds that the project is deprecated and no longer actively maintained. It still has 2,100 stars and 509 forks, which is exactly why it keeps getting cited — popularity metrics do not expire when maintenance does.

To be fair to it, the idea was sound: a portable setup running open-weight models from a USB drive, SSD or local disk, across Windows, macOS, Linux and Android through Termux, with a chat interface and saved history. The owner has not abandoned the concept either — the README points at a successor, Uncensored-Local-Studio, which it says now receives all development and bug fixes. If you have been following a guide that told you to clone the old one, that is where to go instead.

The broader lesson is the one worth carrying: in a niche with no publisher coverage, the top search results are frozen at whenever they were written, and the thing they recommend can be a year out of date without a single page updating.

Carrying Ollama instead

If you already run Ollama and want your existing models on the drive rather than a separate tool, that works too, with one caveat stated up front: this moves the models, not the installation. Ollama still has to be installed on the machine you plug into. It is a way to carry 40GB of weights between your own computers, not a way to run AI on a borrowed one.

The mechanism is the OLLAMA_MODELS environment variable, and the FAQ documents it per platform. The defaults it overrides are ~/.ollama/models on macOS, /usr/share/ollama/.ollama/models on Linux and C:\Users\%username%\.ollama\models on Windows.

Setting it is not the same on each:

  • macOS: launchctl setenv OLLAMA_MODELS "<path>", then restart the Ollama app.
  • Linux: systemctl edit ollama.service, add Environment="OLLAMA_MODELS=<path>" under [Service], then systemctl daemon-reload and systemctl restart ollama.
  • Windows: quit Ollama, add OLLAMA_MODELS as a user environment variable, start it again.

On Linux with the standard installer there is an extra step people miss: the ollama user needs read and write access to the new directory, and the FAQ suggests sudo chown -R ollama:ollama <directory>. On a removable drive whose mount options control ownership, that is the step that will fight you.

Making it genuinely offline

Offline is two separate things, and only one of them is a setting.

The first is having the weights already on the disk, which is just a matter of pulling them before you unplug. Ollama fetches models from the internet, so a drive you have not pre-loaded is a drive that does nothing useful in a dead-network situation.

The second is stopping the tool reaching out when it does have a connection. Ollama’s FAQ documents OLLAMA_NO_CLOUD=1, or disable_ollama_cloud in ~/.ollama/server.json, which runs it in local-only mode and disables cloud models and web search. Worth setting on a rescue drive so a half-working network cannot quietly route your question somewhere else.

llamafile sidesteps the question entirely by having nothing to call home to — the model is inside the file you are executing.

What to actually put on the stick

For a drive whose job is helping with a broken machine, the build is: a llamafile under 4GB, or the llamafile binary plus a GGUF beside it if you want something larger, and a copy of your own notes in plain text so you can paste context in.

Add Whisperfile if voice is useful to you — it is the same single-file idea applied to speech recognition, built on whisper.cpp, handling transcription and translation. On a laptop with a broken display but working audio, that combination is more useful than it sounds, and it needs the same modest resources as any local speech recognition setup.

Use a fast drive. This is the one piece of advice every hobby guide gets right, and the reason is physical rather than clever: the model has to be read off the stick into memory before anything happens, and on a slow USB 2.0 drive that read is the whole experience.

Test it before you need it

A rescue drive that has never been tested is a rescue drive that does not work, and this one has more ways to fail silently than most.

Plug it into a machine that is not the one you built it on, ideally running a different operating system, and run it end to end: launch the file, ask it something, read the answer. You are checking four things at once — that the executable bit survived the copy, that the drive’s filesystem did not mangle it, that the machine will run an unsigned binary from removable media, and that the model loads in whatever memory that machine has free.

Do it once more from a user account without administrator rights, because that is the account you will be borrowing when this matters.

Then write the date on the stick in marker. Models improve, llamafile’s build system has already changed once — the project notes that versions from 0.10.0 onward use a new one — and a drive you assembled eighteen months ago is carrying decisions you no longer remember making.

Where this falls down

The model runs in the borrowed machine’s RAM, not on your stick. The stick is storage. If you plug a 7B model into a laptop with 4GB of memory, the drive’s capacity is irrelevant — this is the same arithmetic that makes small models the interesting ones for hardware you do not control.

Locked-down machines may not let you execute from removable media at all. Corporate policy, SmartScreen and Gatekeeper all have opinions about an unsigned executable arriving on a USB drive, and that is a reasonable thing for them to have opinions about.

A troubleshooting chatbot is working from memory, not from the patient. It cannot read the dead machine’s logs unless you copy them across by hand, and it does not know what changed yesterday. It is a knowledgeable colleague who has never seen this computer — useful, and not a diagnostic tool.

The filesystem matters more than you would guess. A drive formatted for maximum compatibility is usually FAT32, which cannot hold a file over 4GB at all and does not carry a Unix executable bit — so a llamafile copied to it will need chmod +x again on every machine, and a binary-plus-GGUF split may be forced on you by the filesystem rather than by Windows. exFAT lifts the size limit and reads on Windows and macOS out of the box, and on Linux wherever exFAT support is installed, which makes it the sensible choice for this one job.

Nothing here is a backup. A drive carrying a 4GB executable and nothing else is a drive not carrying your recovery image, and on the day you need one of them it is unlikely to be the chatbot.

Sources

Every figure in this article traces to one of these. Dates are when the page was read.

  1. Mozilla-Ocho/llamafile
    MozillaAccessed Nov 3, 2026Official docs
  2. FAQ
    OllamaAccessed Nov 3, 2026Official docs
  3. techjarves/USB-Uncensored-LLM
    GitHubAccessed Nov 3, 2026Official docs

Written by

Kai Devlin AI Automation Engineer

Full-stack developer who builds and runs AI automation systems in production. Runs local LLMs on personal hardware, builds N8N pipelines that actually ship, and deploys on Cloudflare Pages. Every guide on Pipeline Monk is tested on real consumer hardware — a Ryzen 7 5800HS with 16GB RAM and a GTX 1650. If it works on that, it works.