Guide · verified Nov 3, 2026

How to run LLMs locally: runners, models and recommended setups

Open-weight models now run well on a laptop, a big-memory Mac or a gaming GPU. Here is what to install, which models fit your hardware, and how to point an AI agent at a model on your own machine. Every number links to the page we read it from.

Short answer: Install Ollama (terminal) or LM Studio (desktop app), pull a model whose download is comfortably smaller than your free GPU or unified memory, and raise the context window before you hook up a coding agent. Memory is the spec that matters most.

What a local LLM is, and why you'd run one

A local LLM is an open-weight model file (usually GGUF or MLX format) that you download once and run on your own CPU, GPU or Apple silicon through a "runner" app. Nothing goes to a cloud API unless you choose a cloud model.

Why people do it

Privacy. Ollama's FAQ: "We don't see your prompts or data when you run locally." LM Studio says nothing you enter when chatting with a downloaded model leaves your device.
Works offline. LM Studio documents chatting, chatting with documents and its local server as working with no internet once a model is downloaded.
No per-token bill. You pay for hardware and electricity, not API usage. Handy for long agent loops and experiments.
Control. Pick the exact model, quantization and context size, and keep a version pinned.

The trade-offs

Models that fit on consumer hardware are smaller than frontier cloud models, so expect weaker results on hard coding and reasoning tasks.
Memory caps what you can run. A model that spills from GPU to CPU still works but slows down.
Agents need long context, and long context costs extra memory on top of the model.
Some runners also offer cloud models (Ollama does). Turn that off if you need strictly local: Ollama's FAQ documents OLLAMA_NO_CLOUD=1.

Sources: Ollama FAQ · LM Studio: Offline operation

Runners: Ollama vs LM Studio vs llama.cpp vs vLLM vs Jan

A runner downloads the model, loads it onto your hardware and serves it to a chat window or to other apps. All five below expose an OpenAI-style API, which is what lets agents and editors plug into them.

Runner	Best for	What we verified
Ollama	The default pick. Quick CLI and app, huge model library, one-command agent launchers.	macOS, Windows, Linux and Docker installers. Local OpenAI-compatible API at `http://localhost:11434/v1`. GPU support for NVIDIA, AMD Radeon, Apple Metal and Vulkan. `ollama launch` sets up Hermes Agent, OpenCode, Claude Code and more. MIT licensed. docs
LM Studio	The friendliest desktop app: browse, download and chat, then flip on a local server.	macOS (Apple silicon), Windows (x64/ARM) and Linux. Runs GGUF models via llama.cpp and, on Apple silicon, MLX models. OpenAI-compatible endpoints, MCP support, headless mode (llmster). Free for home and work use since July 2025. docs
llama.cpp	Maximum control and portability. The engine many other apps are built on.	Plain C/C++. Apple silicon via Metal, NVIDIA via CUDA, AMD via HIP, plus Vulkan and SYCL. 1.5-bit to 8-bit quantization. CPU+GPU hybrid inference for models bigger than your VRAM. `llama serve` starts an OpenAI-compatible server. MIT licensed. server docs
vLLM	Serving many requests at once on a Linux GPU box: a team server or parallel agent runs.	High-throughput serving library. OpenAI-compatible API server (`vllm serve`). Supports NVIDIA, AMD and Intel GPUs and x86/ARM/PowerPC CPUs, with plugins for other hardware. Apache 2.0. server docs
Jan	An open-source ChatGPT-style desktop app that can mix local and cloud models.	Downloads and runs models from Hugging Face, OpenAI-compatible local server at `localhost:1337`, MCP integration. Apache 2.0. GitHub

Also worth knowing: mlx-lm (Apple's MLX library for running models on Apple silicon) and Hugging Face's ollama run hf.co/… shortcut for running any GGUF repo in Ollama.

Recommended setups

Sizes below are the actual download sizes of each model's default (roughly 4-bit) build on Ollama, read Nov 3, 2026. The download is roughly what the weights take in memory, so leave headroom for the context window and your other apps. No prices or speed numbers here: they change too fast for us to verify.

Tier 1

Everyday laptop or desktop · 16–32 GB RAM

Fits: small and mid models, roughly 3B–14B at 4-bit. With 32 GB, also 20B-class models such as gpt-oss-20b, which OpenAI says runs within 16 GB of memory.

Runner: Ollama or LM Studio. LM Studio recommends 16 GB+ RAM, and on Windows at least 4 GB of dedicated VRAM.

Typical example: a MacBook Air (Apple lists 16, 24 or 32 GB unified memory) or a Windows/Linux laptop with 16–32 GB, running qwen3:8b (5.2 GB), ministral-3:8b (6.0 GB), gemma4:e4b (6.6 GB+) or deepseek-r1:14b (9.0 GB). On 32 GB, gpt-oss:20b (14 GB).

Good for: chat, summaries, quick scripts, autocomplete-style coding help. Sources: LM Studio requirements · MacBook Air specs · gpt-oss

Tier 2

Mac with lots of unified memory · 64–128 GB

Fits: 30B–35B models comfortably, 70B-class models at 4-bit, and at 96 GB+ even gpt-oss-120b (65 GB). On Apple silicon the GPU draws on the same unified memory pool as the CPU, so a big-memory Mac can hold models that would not fit on a single gaming GPU.

Runner: LM Studio (it can run MLX models on Apple silicon) or Ollama (Metal acceleration).

Typical example: a MacBook Pro with M5 Max (configurable to 48, 64 or 128 GB) or a Mac Studio (M5 Max up to 128 GB) running qwen3.6:35b-a3b (23 GB), gemma4:31b (19–20 GB) or llama3.3:70b (43 GB). OpenHands lists "a Mac with Apple Silicon with at least 64GB of unified memory" for quantized Qwen3.6-35B-A3B as its recommended local coding model.

Good for: an all-in-one local coding-agent machine. Sources: MacBook Pro specs · Mac Studio specs · OpenHands local LLMs

Tier 3

Gaming PC with an NVIDIA GPU · 24–32 GB VRAM

Fits: as a rule of thumb, models up to about 20 GB download fit on a 24 GB card with some room left for context; a 32 GB card leaves more headroom for 30B-class models and longer context windows. Bigger models still run by splitting between GPU and system RAM, but slower. Check the split with ollama ps.

Runner: Ollama or LM Studio to start. llama.cpp for fine control of GPU offload. vLLM if you want to serve several agents or users at once on Linux.

Typical example: an RTX 3090 or RTX 4090 (24 GB) or RTX 5090 (32 GB) running qwen3-coder:30b (19 GB), gemma4:31b (19–20 GB), deepseek-r1:32b (20 GB) or gpt-oss:20b (14 GB). OpenHands lists "a recent GPU with at least 24GB of VRAM" for quantized Qwen3.6-35B-A3B.

Good for: putting a gaming GPU you already own to work. Sources: RTX 3090 · RTX 4090 · RTX 5090 · Ollama hardware support

Tier 4

Step-up workstation · 128 GB+ of fast memory

Fits: 70B–120B-class models, including gpt-oss-120b (65 GB on Ollama; OpenAI says it fits a single 80 GB GPU) and qwen3.5:122b (81 GB), with room for long agent contexts.

Runner: vLLM or llama.cpp for serving, Ollama or LM Studio for simplicity.

Typical example: NVIDIA DGX Spark (GB10 Grace Blackwell, 64 GB or 128 GB unified memory) or a Mac Studio with M5 Ultra (96 GB, configurable to 256 GB or 512 GB unified memory). Multi-GPU Linux rigs fit here too, using vLLM's tensor parallelism.

Good for: running a strong open model all day for a small team or a fleet of agents. Sources: DGX Spark · Mac Studio specs · gpt-oss

Rule of thumb: memory vs model size

Weights ≈ parameters × bits ÷ 8. About 2 GB per billion parameters at 16-bit, about 1 GB at 8-bit and about 0.5 GB at 4-bit, then add headroom for the context window (KV cache) and the runner itself.

Gemma 4 (Google's own table)	16-bit	8-bit	4-bit (Q4_0)
Gemma 4 12B	26.7 GB	13.4 GB	6.7 GB
Gemma 4 26B A4B (MoE)	57.7 GB	28.8 GB	14.4 GB
Gemma 4 31B	69.9 GB	34.9 GB	17.5 GB

4-bit is the usual sweet spot for local use. Formats like GGUF Q4_K_M (Ollama's default for many models) trade a little quality for about a quarter of the 16-bit memory.
Mixture-of-Experts models still load every parameter. Google notes all 26B parameters of Gemma 4 26B A4B must be in memory even though about 4B are active per token. MoE buys speed, not memory.
Context costs memory. Ollama defaults to a 4k context under 24 GiB of VRAM (32k at 24–48 GiB, 256k at 48 GiB+) and recommends at least 64k tokens for agents and coding tools, which needs more memory.
Keep it on the GPU. Ollama's docs advise avoiding CPU offload for best performance; ollama ps shows the GPU/CPU split.

Sources: Gemma 4 memory requirements (Google's figures include about 20% loading overhead) · Ollama: Context length · Hugging Face: GGUF

Popular open-weight model families

Family	Who makes it	What to know	Get it
Qwen	Alibaba's Qwen team	Wide size range, from 0.6B (Qwen3) up to MoE giants, plus coder and vision variants. Qwen3 open-weight models are Apache 2.0. Qwen3.6-35B-A3B is OpenHands' recommended local coding model.	Hugging Face · Ollama · docs
Gemma	Google	Gemma 4 ships in E2B, E4B, 12B, 26B A4B (MoE) and 31B, with official 4-bit QAT builds for llama.cpp and LM Studio and built-in function calling. Open weights that permit responsible commercial use.	Google · Ollama
gpt-oss	OpenAI	gpt-oss-20b (21B total, 3.6B active) runs within 16 GB of memory; gpt-oss-120b fits a single 80 GB GPU. Apache 2.0, built for tool use and agentic workflows.	GitHub · Ollama
Llama & Muse	Meta	Llama 3.x models (for example Llama 3.1 8B and Llama 3.3 70B) remain available on Hugging Face and Ollama. Meta's newer open model on Ollama is Muse Glimmer, a 30B model for local agents, listed under Apache 2.0.	Hugging Face · Ollama (Llama 3.3) · Ollama (Muse Glimmer)
Mistral	Mistral AI	Ministral 3 comes in 3B, 8B and 14B with text and vision; Mistral also publishes larger open-weight models.	Mistral docs · Ollama
DeepSeek-R1 distills	DeepSeek	Six dense reasoning models distilled from DeepSeek-R1 onto Qwen and Llama bases: 1.5B, 7B, 8B, 14B, 32B and 70B. MIT licensed (base-model licenses also apply).	GitHub · Ollama
Phi	Microsoft	Small models; Phi-4 is 14B (9.1 GB on Ollama) with a 16K context.	Hugging Face · Ollama

For agents, pick a model tagged for tool calling: Ollama’s tools filter lists them. Sources: Qwen3 README · Gemma 4 overview · DeepSeek-R1 README

Running agents on a local model

An agent harness is the loop around the model: tools, file edits, shell commands and memory. Most harnesses talk to any OpenAI-compatible endpoint, so you point them at Ollama (http://localhost:11434/v1), LM Studio, llama.cpp or vLLM instead of a cloud API. Browse every harness we track in Agent harnesses (open source).

Harness	What it is	Local setup
Hermes Agent	Nous Research's personal agent with persistent memory, auto-generated skills and messaging apps (Telegram, Discord, Slack and more). MIT licensed.	`ollama launch hermes` installs and configures it, or point its custom endpoint at `http://127.0.0.1:11434/v1`. guide
Goose	General-purpose open-source agent: desktop app, CLI and API. Part of the Agentic AI Foundation at the Linux Foundation. Apache 2.0.	Pick Ollama as the provider in Settings or `goose configure`. guide
Aider	AI pair programming in your terminal, built around git.	`aider --model ollama_chat/<model>`; also supports LM Studio. guide
OpenCode	Open-source terminal coding agent that reads your project, edits files and runs commands.	`ollama launch opencode`; needs a 64k+ context window. Also lists LM Studio and llama.cpp as providers. guide
Cline	Coding agent for your editor.	Connects to Ollama or LM Studio. guide
OpenHands	Open-source software-engineering agent platform.	Documents LM Studio as the recommended local server, plus Ollama, vLLM and SGLang. guide

Starter recipe: a local coding agent in 3 commands

# 1. Install Ollama from ollama.com/download, then pull a tool-calling model
ollama pull qwen3.6:35b-a3b        # 23 GB. On 16–32 GB machines try gpt-oss:20b or qwen3:8b
# 2. Give it room to think: agents need 64k+ context
OLLAMA_CONTEXT_LENGTH=64000 ollama serve
# 3. In another terminal, in your project folder
ollama launch opencode             # or: ollama launch hermes

Commands from Ollama: Context length and Ollama: OpenCode. Expect a local model to need more hand-holding than a frontier cloud model on big refactors.

FAQ

Can I run an LLM completely offline?

Yes. Once the model file is downloaded, runners like LM Studio and Ollama run it on your machine without an internet connection. LM Studio documents that chatting, chatting with documents and its local server all work offline; you only need the internet to search for and download models.

How much memory do I need to run a local LLM?

Roughly 0.5 GB per billion parameters at 4-bit, 1 GB at 8-bit and 2 GB at 16-bit, plus headroom for the context window. Google lists Gemma 4 31B at about 17.5 GB at 4-bit, 34.9 GB at 8-bit and 69.9 GB at 16-bit.

Is Ollama or LM Studio better for beginners?

Both are easy. LM Studio is a desktop app with a model browser and chat UI; Ollama is a lightweight background service with a CLI, a desktop app and one-command launchers for agents like Hermes Agent and OpenCode. Both expose an OpenAI-compatible local API.

Can coding agents use a local model?

Yes. Aider, Goose, OpenCode, Cline, OpenHands and Hermes Agent can all connect to Ollama or another OpenAI-compatible local endpoint. Give the model a large context window: Ollama recommends at least 64k tokens for agents and coding tools.

Does a Mixture-of-Experts model need less memory?

Not for loading. Google notes that all 26 billion parameters of Gemma 4 26B A4B must sit in memory even though only about 4 billion are active per token, so plan memory by total parameters. MoE models are mainly faster, not smaller.

Last verified Nov 3, 2026. Facts on this page come only from the linked sources; model sizes and hardware specs change often, so confirm before you buy. Spotted something out of date? Tell us and we'll re-verify.