What the Heck is a Local Model?

What a local model is

ChatGPT, Claude and Gemini are cloud models. You type, your words travel over the internet to a data center, a rack of GPUs works out the answer, and it travels back. You're borrowing someone else's computer.

A local model is the same idea running on your own machine. The model is a file on your disk. Your laptop's chip does the work. Turn off the wifi and it keeps answering.

This is possible because several AI labs publish their models for anyone to download. Meta has Llama. Alibaba has Qwen. Google has Gemma. OpenAI has gpt-oss. These are called open-weight models, and they're free.

How it works

A model is a very large file of numbers. During training, a lab adjusts billions of those numbers until the file gets good at one task: given some text, predict what comes next. Training costs the lab millions of dollars. Once it's done, a decent laptop can run the result.

Running it looks like this. Your laptop loads the file into memory. You type a prompt. The chip pushes your words through all of those numbers and out comes the most likely next piece of a word. That piece is added to the text, and the whole thing runs again. And again. Every word you see on screen is one full pass through the model.

Those pieces are called tokens. A token is roughly three quarters of a word, so 100 tokens is about 75 words.

The software that does the loading and running is called a runner. The one I use is Ollama. It downloads models, loads them, and gives you a chat. It works a bit like a music player, with models as the songs.

The numbers, decoded

Every model comes with a name like qwen3:8b and a few specs. Six terms cover most of what you'll see.

Parameters: the B

Parameters are the numbers inside the file. The B means billion. qwen3:8b has 8.2 billion of them. More parameters usually means a smarter model. It also means a bigger file and slower replies.

A rough feel for sizes: 1 to 4B handles rewriting, summaries and simple questions. 7 to 14B is a decent everyday assistant. 20 to 30B copes with longer documents and harder reasoning. The big cloud models are far larger again.

Quantization: the Q

Each parameter is stored as a number with a certain precision. Full precision uses 16 bits per number. Quantization rounds those numbers down to fewer bits, most commonly 4. The file shrinks to about a quarter of its size and the model gets slightly less sharp.

It's like saving a photo as a high-quality JPEG. Smaller file, and you mostly can't tell. Labels like Q4_K_M mean 4-bit and are the sensible default. A 4-bit version of a bigger model usually beats a full-precision version of a smaller one.

Memory

The whole model has to fit in your computer's memory (RAM) while it runs. At 4-bit, a model needs roughly 0.6GB per billion parameters for the file alone.

Then it needs working space for the conversation, and that grows with the context window (next section). On my machine qwen3:8b is a 5.2GB file. With room for 4,096 tokens it used 5.1GB of memory. At 16,384 tokens, 7.0GB. At 32,768 tokens, 9.4GB. Same model, almost double the memory.

qwen3:8b memory in use, by context window

Measured with ollama ps on a MacBook Air M4, 32GB

Show data

Context	Memory
2,048 tokens	4.8GB
4,096 tokens	5.1GB
8,192 tokens	5.8GB
16,384 tokens	7.0GB
32,768 tokens	9.4GB

Newer models are much leaner here. Ollama reported 4.1GB for qwen3.5:4b at every context size from 2,048 to 32,768 tokens. Its design needs far less working space, though it also runs on a different engine that reserves memory differently, so treat that one as a floor.

Context window

The context window is how much text the model can hold in mind at once: your conversation, any document you pasted, and its own replies. On my Mac, Ollama gave both models 16,384 tokens by default, about 12,000 words. The default changes between Ollama versions and machines. A bigger window lets you work with longer documents, and costs more memory.

Tokens per second

This is how fast the model writes. Around 10 tokens per second feels like someone typing quickly. Around 20 is faster than you can comfortably read. Here's the same reply at real speed, from my MacBook Air M4 (32GB) and from published tests on other chips:

The two Qwen lanes are mine: Ollama on my Air, 4-bit models, timed after the model was already loaded. Ollama ran qwen3.5:4b on Apple's MLX engine and qwen3:8b on llama.cpp, so treat the gap between those two as rough. The Llama 2 7B lanes come from llama.cpp's public Apple Silicon benchmark, the same 4-bit model run on each chip, which makes them the cleanest chip-to-chip comparison there is. The gpt-oss:20b lane is Micro Center's test. Your numbers will move with context size, whatever else is open, and the Ollama version.

Seconds to write a 300-word reply

About 400 tokens, divided by each setup's tokens per second. Lower is faster

Measured on my AirPublished benchmark

Show data

Model and machine	Tokens/sec	400 tokens
gpt-oss:120b, M2 Ultra desktop	85.6 (llama.cpp)	4.7s
qwen3.5:4b, my M4 Air	27.8 (measured)	14.4s
Llama 2 7B, base M4	24.1 (llama.cpp)	16.6s
qwen3:8b, my M4 Air	19.6 (measured)	20.4s
gpt-oss:20b, M4 Air 24GB	13.0 (Micro Center)	30.8s

The fastest bar is the biggest model. gpt-oss:120b on an M2 Ultra desktop Mac, from llama.cpp's own benchmark file, writes the reply in under 5 seconds. Two things explain it: that chip moves memory at 800GB per second, more than six times my Air, and gpt-oss only reads a small slice of itself for each token.

Speed has a simple ceiling. Every token means reading the whole model file from memory once, so speed is capped at memory bandwidth divided by file size. The base M4 chip reads memory at 120GB per second. 120 divided by 5.2GB gives qwen3:8b a ceiling of about 23 tokens per second. I measured 19 to 21. For the 4.0GB qwen3.5:4b the ceiling is 30, and I measured 28. The same math explains why bigger chips are faster: an M4 Pro moves 273GB per second and an M4 Max up to 546GB per second, so the same model should run roughly 2 to 4 times faster on them.

Speed ceiling on a base M4, and where real models land

Tokens per second against model file size

Ceiling: 120GB/s ÷ file sizeMeasured on my AirMeasured by Micro Center

Show data

Model	File	Ceiling	Measured
qwen3.5:4b	4.0GB	30 tok/s	27.8 tok/s
qwen3:8b	5.2GB	23 tok/s	19.6 tok/s
gpt-oss:20b	13.8GB	8.7 tok/s	13 tok/s (24GB Air)

The one model above the line is gpt-oss:20b. It's a mixture-of-experts model: only a slice of the file is read for each token, so it beats the simple ceiling. That's the main trick behind big models that still run fast on laptops.

Thinking mode

Some models, Qwen3 included, can think before answering. They write out their reasoning first, then the reply. It helps with logic and maths, and it's slow. With thinking on, a simple riddle took qwen3:8b 70 seconds on my Air, because it wrote 1,313 tokens of reasoning before a one-line answer. qwen3.5:4b took 34 seconds for the same riddle. For rewriting and quick questions, turn it off. The setup section shows how.

What your laptop can run

Memory decides what you can run. Leave roughly a third of it for your operating system and browser, and the rest is your model budget. Download sizes below are from the Ollama library and can differ slightly by machine.

A note on two of those. qwen3:30b is a mixture-of-experts model: it holds 30B parameters but only uses about 3B for each token, so it answers far faster than its size suggests. gpt-oss:20b is OpenAI's open model, also mixture-of-experts. I haven't run it locally on my Air, but Micro Center measured about 13 tokens per second on a MacBook Air M4 with 24GB.

On a Mac, the chip and graphics share one pool of memory, which is why Macs are unusually good at this. On a Windows PC, look at the memory on your graphics card (VRAM). System RAM matters much less. A model that fits entirely on an NVIDIA card with 12GB or more runs very fast. Without a graphics card it still runs on the processor, just slower.

To check whether a model really fits, run ollama ps while it's loaded. The PROCESSOR column should say 100% GPU. If it shows a split like 40%/60% CPU/GPU, the model is spilling out of fast memory and you should drop a size.

Pick a model for the job

Text chat is only the start. There are open models for coding, reading images, making images, speech and video. These are the ones I'd try first as of October 2026, all from the lab that made them. New releases land every week, so sort the Ollama library by newest once in a while.

Text, code and images. The e4b in gemma4:e4b means "effective 4B": it runs about as fast as a 4B model and also understands images and audio. For coding, qwen3-coder:30b is a mixture-of-experts model, so it stays quick despite its size. Both coding picks need a 32GB machine. On 16GB, qwen3.5:9b is a capable coding helper. For images, drag a photo or screenshot into the Ollama chat window, or put its file path after your question in the terminal.

Making images. Ollama added image generation in January 2026, currently on Macs only. Run ollama run x/flux2-klein:4b "a fox reading a newspaper" and the picture is saved to the folder you ran it from. The 4B version is free for any use. The 9B version and some other image models are non-commercial only, so check the licence if it's for work.

Voice. phonon is a tiny speech model that runs inside Ollama on Apple Silicon and uses about 120MB of memory. Kokoro is the popular open alternative. Going the other way, Whisper large-v3-turbo from OpenAI transcribes speech to text and is small enough for any laptop. whisper.cpp runs it from the same team behind llama.cpp.

Video is the exception. Wan 2.2's 5B model makes 720p clips and is described by its makers as running on a consumer graphics card like an RTX 4090. That is a 24GB desktop card. On a laptop, video generation is slow enough that it isn't worth it yet.

One safety habit for Hugging Face: download from the lab's own account (Qwen, Black Forest Labs, Wan-AI, OpenAI). Popular models get re-uploaded by strangers, sometimes with changes you can't see.

Why bother

Cloud models are better, and I still use them every day. Some work shouldn't leave your machine, though, and a lot of it doesn't need the best model available.

For you

1. 1
   Privacy. Journals, health questions, finances, personal documents. Nothing you type is sent anywhere.
2. 2
   Save your tokens. Rewrites, summaries and quick lookups run free on your laptop. Your paid cloud quota goes to the hard problems.
3. 3
   Offline. On a plane or on bad hotel wifi, it works exactly the same.
4. 4
   Fewer guardrails. You pick a model whose behaviour suits you instead of living with someone else's refusal policy. Where to find these is covered in the setup section.
5. 5
   It doesn't change under you. Cloud models get updated, retired and repriced overnight. A downloaded model behaves the same next year.

For teams and companies

1. 1
   Data stays in-house. Client records, contracts and source code never reach a third-party server. That matters in finance, health and law, and anywhere data-residency rules apply.
2. 2
   Internal workflows stay private. Your tools and agents run without showing an outside AI provider how your business works.
3. 3
   Predictable cost. You pay for hardware once. Per-token bills keep growing as more people and automations use them.
4. 4
   No lock-in. Open models can be swapped, tuned on your own data, and kept running whatever happens to a vendor.

The catch is quality. A model that fits on a laptop is noticeably weaker than the best cloud models at hard reasoning, long code and niche facts, and it can be wrong with full confidence. Check anything that matters. I use both: local for private, simple and high-volume work, cloud for problems that need the strongest model.

Try it now

This takes about ten minutes, most of it the download. The commands are the same on Mac, Windows and Linux.

1. 1
   Install Ollama. Download it from ollama.com/download and open it once. It runs quietly in the background after that.
2. 2
   Open a terminal. The terminal is a window where you type commands instead of clicking. On a Mac, press Cmd + Space and type Terminal. On Windows, open the Start menu and type PowerShell.
3. 3
   Run your first model. Type this and press Enter. It downloads about 4GB the first time, then opens a chat.

   ollama run qwen3.5:4b

4. 4
   Talk to it. Type a question and press Enter. To leave the chat, type /bye.
5. 5
   Turn thinking off for everyday tasks. Faster answers, no visible reasoning.

   ollama run qwen3.5:4b --think=false

6. 6
   Learn three housekeeping commands. ollama list shows what you have downloaded. ollama ps shows what's loaded and how much memory it uses. ollama rm qwen3.5:4b deletes a model you no longer need.
7. 7
   Step up a size. Use the table above. If the 4B model felt fast, try the 9B. Stop when replies slow down or ollama ps stops showing 100% GPU.

Where to find models

The Ollama library at ollama.com/library is the easiest start. Every model there runs with one ollama run command. One thing to watch: the same list now includes cloud models. Any name ending in -cloud runs on Ollama's servers, so your prompts leave the machine. Handy when a model is too big for your laptop, but it isn't local.

Hugging Face at huggingface.co is where almost every open model is published first, including fine-tuned versions. Look for files ending in .gguf, which is the format laptop runners use. Ollama can pull them directly with ollama run hf.co/ followed by the user and model name from the page. Versions labelled "abliterated" have their refusal training removed. They help when a mainstream model turns down ordinary requests, and they won't push back on anything else either, so the judgement is yours.

Prefer not to use a terminal? The Ollama app has its own chat window, so you can pick a model and type without any commands. LM Studio is another free option with a ChatGPT-style interface, a model browser, and a warning when a model is too big for your machine.

That's it

When Riccardo set this up for me, the moment it clicked was turning off the wifi and watching it keep answering. That's the whole idea. The model is just a file, and your laptop is enough to run it.

You don't need to replace anything. Keep your cloud subscription for the hard problems. Send the private stuff, the quick stuff and the repetitive stuff to the model on your own machine. Start with qwen3.5:4b, ask it something you wouldn't type into ChatGPT, and see how it feels.