Do you actually need a local LLM?

// the appeal

The on-ramp is genuinely one line. Install Ollama (or llama.cpp / LM Studio), pull a model, and it's serving on your machine behind an OpenAI-compatible endpoint — no API key, no per-call bill.

# a model, running locally, in one command $ ollama run llama3 >>> ready — http://localhost:11434

That ease makes "just run it locally" the reflex. The question isn't can you — it's should you.

What local actually buys you

Privacy / data residency. Prompts and documents never leave the machine — no third-party logging. For regulated, proprietary, or air-gapped work, that alone can be the deciding factor.

Offline + no rate limits + no vendor lock-in. It runs with no network, no quota, and no provider that can deprecate your model out from under you.

Marginal cost ≈ $0 — but only at volume. Once the hardware is paid off, each token is basically free. That only beats a hosted API at sustained, heavy throughput; for occasional use, the cloud is cheaper once you count hardware, power, and ops.

The catch

It trails the frontier. The best open-weight models sit a handful of benchmark points behind the best closed models, and the gap is widest on hard reasoning. Local is "good enough" for a lot — not for the top of the difficulty curve.

VRAM is the real gatekeeper. A model's memory ≈ its parameters × bytes-per-weight. That's what decides whether it even loads on your card:

# rough VRAM to run (weights only) 7–8B ~8 GB # fits a consumer GPU 13B ~13 GB 70B ~43 GB # 4-bit; needs 2×24GB or a 64GB-unified Mac

"70–85% of frontier quality" is an estimate, not a spec. The real gap varies by task and model. And "free" ignores hardware, electricity, and ops — plus the KV cache eats extra VRAM that grows with context. See quantization for how 4-bit shrinks the numbers above.

The rule

Don't default either way. Route by the constraint that actually applies:

go local if private data · offline · very high volume use cloud if you need frontier quality · or have no big GPU # common pattern: default local, send the ~10–15% # of hard requests to a frontier API

Local is a tool, not a trophy. Reach for it when privacy, offline, or throughput is the point — and keep the cloud for the jobs where quality is.

Decided to run one locally? The rest of this lane is what that actually costs. Quantization shrinks the weights, the KV cache is the bill the weights do not show you, and active is not what you keep is why a "3B active" download is eighteen gigabytes — move the experts, not the layers if it will not fit. Those are all about MEMORY; speculative decoding is the one about SPEED, and about why the obvious trick for it sometimes runs backwards.

Sources: daily.dev — running LLMs locally · LocalLLM.in — VRAM requirements · open vs closed benchmarks (2026)

One concept a week. Free.

The deeper, copy-paste version of each ToolCall short — in your inbox.

// total: 0.00 · spam: void · unsubscribe: one click