Every cloud LLM call is a small tax: an API key to manage, a per-token bill that grows quietly, and - the part that matters most for client work - your code and prompts leaving your machine. Ollama removes all three. It's a single Go binary that downloads, runs, and serves open models like Llama, Mistral, Qwen, Gemma, and Phi entirely on your own hardware, exposing them through a clean local API. If you've ever wanted "ChatGPT, but it never phones home," this is the shortest path to it.
One Command to a Running Model
The whole experience is deliberately boring in the best way. Install it, then pull and chat with a model in one line:
# Pull and chat with a model interactively
ollama run llama3.2
# Start the background server (OpenAI-compatible API on :11434)
ollama serve
# Grab an embedding model for local search
ollama pull nomic-embed-text
Models are quantized by default, so a capable 7B-class model runs comfortably on a modern laptop, and larger ones scale up if you have the GPU for it.
An API That Speaks OpenAI
The reason Ollama slots into real projects is its API. It serves both its own simple REST endpoint and an OpenAI-compatible one, which means most tools and SDKs that already talk to OpenAI can be pointed at localhost:11434 with a one-line base-URL change. Continue, llm, LiteLLM, and dozens of other tools treat Ollama as a first-class provider.
Modelfiles: Your Own Named Models
A Modelfile lets you bake a system prompt, parameters, and a base model into a reusable named model - the local equivalent of a saved assistant. It's a lightweight way to keep a "Drupal reviewer" or "commit-message writer" around without re-pasting instructions every session.
Rough Edges and Limitations
- Local models aren't frontier models. A quantized 7B-14B model is genuinely useful for drafting, classifying, and summarizing, but it won't match a top hosted model on hard reasoning. Match the model to the task.
- RAM and VRAM are the real limits. Bigger models need real hardware; try to run something too large and it either crawls or won't load.
- First pull is heavy. Model files are gigabytes. The initial download isn't instant, though it's cached forever after.
- Quality varies by model. The library is broad; some community models are excellent and some are rough. Stick to the well-known families until you've tested one.
For anything privacy-sensitive, cost-sensitive, or just offline, Ollama is the tool I reach for first. It turns "run a local model" from a weekend project into a one-liner, and everything else in the local-AI ecosystem tends to assume you have it installed.
Internal link suggestion: link "OpenAI-compatible" to the Onyx or Free Claude Code post, both of which pair naturally with a local Ollama backend.