Local LLM Setup Guide

Local LLM Setup Guide

Running models locally gives you privacy, no per-token cost, and offline capability — at the price of hardware and setup. This is the companion to the Raspberry Pi LLM guide for larger machines: consumer GPUs, beefy CPUs, and Apple Silicon. The good news is that a local model behind an OpenAI-compatible API means most existing tooling “just works.”

Hardware: VRAM is the binding constraint

For GPU inference, the model’s weights must fit in VRAM — that’s the limit, more than raw compute. Quantization shrinks the weights (lower precision) so larger models fit, trading a little quality for a lot of VRAM. Rough guide (varies by quantization):

VRAM	Comfortably runs (quantized)
8 GB	~7–8B models (Q4)
16 GB	~13–14B, or 7–8B at higher precision
24 GB	~32B (Q4), or smaller models fast

CPU-only works for smaller quantized models but is slow (memory-bandwidth-bound); system RAM then matters more than CPU clock.

Ollama — the easy path

Ollama is the simplest way to run, manage, and serve local models. See the community Ollama Guide too.

ollama pull llama3.1          # download a model
ollama run llama3.1           # chat in the terminal
OLLAMA_HOST=0.0.0.0 ollama serve   # expose on the network

It exposes an OpenAI-compatible /v1/chat/completions endpoint, so existing clients can point at it directly.

llama.cpp and quantization

llama.cpp is the engine under many tools (Ollama included), compilable for CUDA, Metal, or CPU. Its GGUF quantization levels trade quality/speed/VRAM:

Quant	Trade-off
Q4_K_M	The common default — good size/quality balance
Q5_K_M	A bit larger, slightly better quality
Q8_0 / F16	Near-full / full precision; large

llama-server provides an HTTP API for direct use.

vLLM — throughput for GPUs

vLLM targets high-throughput GPU serving (its PagedAttention manages the KV cache efficiently for many concurrent requests). It also serves an OpenAI-compatible API. Reach for it over Ollama when you need to serve many users/requests; for single-user local use, Ollama is simpler.

LM Studio — GUI

LM Studio is a desktop GUI for discovering, downloading, and chatting with models, plus a local server mode — the friendliest option for non-technical users on Mac/Windows/Linux.

Choosing a model

Pick by task and what fits your VRAM. The strong open-weight families evolve fast, so treat names as a snapshot and check current releases:

• General/instruction — Llama, Qwen, Mistral, Gemma families.
• Coding — Qwen-Coder, DeepSeek-Coder families.
• Embeddings — e.g. nomic-embed-text (for RAG).

OpenAI-compatible API

The killer feature: Ollama, llama.cpp, vLLM, and LM Studio all expose /v1/chat/completions, so you can point existing tools (LangChain, LlamaIndex, many IDE assistants) at http://localhost:<port>/v1 with a dummy key and they work unchanged — including driving local agents.

Apple Silicon

Macs are excellent local-LLM machines: unified memory means the GPU can use most of system RAM as “VRAM,” and llama.cpp/Ollama use Metal acceleration. An M-series Mac with plenty of unified memory runs larger models than a similarly-priced discrete GPU.

Benchmarking

The metric that matters is tokens/sec (throughput) and time-to-first-token (latency). Performance is usually memory-bandwidth-bound, which is why unified-memory Macs and high-bandwidth GPUs punch above their FLOPS. Measure on your hardware and model; quantization and context length both move the numbers.

Related

• Raspberry Pi LLM Guide — the small-hardware companion
• Ollama Guide — the community’s Ollama walkthrough
• LLM Evaluation Frameworks — checking a local model is good enough for your task
• RAG Patterns — pairing a local model with retrieval