Do you actually need a local LLM?

// the appeal

The on-ramp is genuinely one line. Install Ollama (or llama.cpp / LM Studio), pull a model, and it's serving on your machine behind an OpenAI-compatible endpoint — no API key, no per-call bill.

# a model, running locally, in one command $ ollama run llama3 >>> ready — http://localhost:11434

That ease makes "just run it locally" the reflex. The question isn't can you — it's should you.

What local actually buys you

Privacy / data residency. Prompts and documents never leave the machine — no third-party logging. For regulated, proprietary, or air-gapped work, that alone can be the deciding factor.

Offline + no rate limits + no vendor lock-in. It runs with no network, no quota, and no provider that can deprecate your model out from under you.

Marginal cost ≈ $0 — but only at volume. Once the hardware is paid off, each token is basically free. That only beats a hosted API at sustained, heavy throughput; for occasional use, the cloud is cheaper once you count hardware, power, and ops.

The catch

It trails the frontier. The best open-weight models sit a handful of benchmark points behind the best closed models, and the gap is widest on hard reasoning. Local is "good enough" for a lot — not for the top of the difficulty curve.

VRAM is the real gatekeeper. A model's memory ≈ its parameters × bytes-per-weight. That's what decides whether it even loads on your card:

# rough VRAM to run (weights only) 7–8B ~8 GB # fits a consumer GPU 13B ~13 GB 70B ~43 GB # 4-bit; needs 2×24GB or a 64GB-unified Mac

"70–85% of frontier quality" is an estimate, not a spec. The real gap varies by task and model. And "free" ignores hardware, electricity, and ops — plus the KV cache eats extra VRAM that grows with context. See quantization for how 4-bit shrinks the numbers above.

The rule

Don't default either way. Route by the constraint that actually applies:

go local if private data · offline · very high volume use cloud if you need frontier quality · or have no big GPU # common pattern: default local, send the ~10–15% # of hard requests to a frontier API

Local is a tool, not a trophy. Reach for it when privacy, offline, or throughput is the point — and keep the cloud for the jobs where quality is.

Sources: daily.dev — running LLMs locally · LocalLLM.in — VRAM requirements · open vs closed benchmarks (2026)

One concept a week. Free.

The deeper, copy-paste version of each ToolCall short — in your inbox.

// total: 0.00 · spam: void · unsubscribe: one click