Ollama Best Practices: Running Local LLMs Efficiently and Reliably

Ollama has become the de facto standard for running large language models locally. It abstracts away the complexity of model quantization, GPU management, and inference engine configuration, letting developers focus on building applications rather than wrestling with dependencies. However, "it just works" doesn't mean "it's already optimal." This course section covers the production-grade best practices that separate a casual Ollama setup from a robust, performant, and maintainable one.

This guide covers five critical areas:

1. Hardware & Resource Management — matching models to your system's capabilities
2. Modelfile Design — crafting reproducible, version-controlled model configurations
3. Performance Optimization — squeezing the most throughput from your hardware
4. Security & Networking — safely exposing Ollama in multi-user or remote contexts
5. Operational Patterns — keeping Ollama healthy in long-running production environments

Understanding the Ollama Architecture

Before diving into best practices, it's essential to understand how Ollama works under the hood. Ollama acts as a local inference server that manages model downloads, quantized weights, and GPU/CPU memory allocation via its bundled llama.cpp backend.

Ollama listens on localhost:11434 by default and exposes a RESTful API compatible with the OpenAI chat completions format. When a model is loaded, Ollama attempts to offload as many GGUF layers as possible to GPU VRAM, falling back to CPU for any remaining layers. This hybrid execution path is the single biggest factor in Ollama's performance on consumer hardware .

Footnotes

1. Ollama GitHub Repository — Architecture and source code for Ollama inference engine. ↩

1. Hardware & Resource Management

1.1 VRAM Budgeting

The most common mistake new Ollama users make is pulling a model that doesn't fit in their GPU's VRAM. When a model exceeds VRAM, Ollama offloads the remaining layers to CPU RAM, creating a severe performance penalty because data must shuttle between GPU and CPU on every token.

The rule of thumb: target a model whose Q4_K_M size is 80–90% of your total VRAM. This leaves headroom for the KV cache, which grows with context length. A model that barely fits in VRAM at short contexts will spill to CPU at long contexts .

1.2 Multi-GPU Considerations

Ollama supports multi-GPU inference natively. When multiple GPUs are detected, Ollama distributes GGUF tensor splits across available devices:

• Same-architecture GPUs: Ollama splits layers evenly. This works well for two matching RTX 3090s or 4090s.
• Mixed GPUs: Ollama still works, but the slower GPU becomes the bottleneck. If you pair an RTX 3090 with a GTX 1060, inference speed is dominated by the GTX 1060's bandwidth.
• Environment variable: Set OLLAMA_NUM_GPU to control how many GPU layers to offload. Setting it to 0 forces CPU-only inference (useful for debugging) .

Footnotes

1. Ollama Model Library — Official model sizes, quantizations, and VRAM requirements. ↩

2. Ollama GPU Documentation — GPU configuration and multi-GPU setup guides. ↩

2. Modelfile Design Best Practices

The Modelfile is Ollama's equivalent of a Dockerfile. It specifies the base model, system prompt, inference parameters, and chat template. Well-structured Modelfiles are essential for reproducibility and collaboration.

2.1 Modelfile Anatomy

1# Modelfile — My Custom Assistant
2FROM llama3.1:8b
3
4# System prompt defines persistent behavior
5SYSTEM """
6You are a senior software engineer who gives concise, accurate answers.
7Use markdown formatting when helpful. If unsure, say so.
8"""
9
10# Inference parameters
11PARAMETER temperature 0.3
12PARAMETER top_p 0.9
13PARAMETER num_ctx 4096
14PARAMETER num_predict 512
15PARAMETER repeat_penalty 1.1
16
17# Template (usually inherited from base model)
18TEMPLATE """{{- if .System }}<|start_header_id|>system<|end_header_id|>
19{{ .System }}<|eot_id|>
20{{- end }}
21<|start_header_id|>user<|end_header_id|>
22{{ .Prompt }}<|eot_id|>
23<|start_header_id|>assistant<|end_header_id|>
24{{ .Response }}<|eot_id|>"""

2.2 Parameter Selection Guide

2.3 Modelfile Anti-Patterns

❌ Don't use overly long system prompts — they consume context window and add latency to every request. ❌ Don't set num_ctx to maximum unless needed — KV cache scales as $O(n^2 \cdot d)$ in attention, meaning double the context can quadruple memory for attention computation. ❌ Don't override the TEMPLATE unless you're using a non-standard base model — incorrect templates cause gibberish output .

✅ Do version-control your Modelfiles in Git alongside your application code. ✅ Do use .ollamaignore or minimal FROM references to keep builds reproducible. ✅ Do document each parameter choice with a comment in the Modelfile.

Footnotes

1. Ollama Modelfile Documentation — Complete Modelfile syntax reference and parameter guide. ↩

3. Performance Optimization

3.1 Quantization Selection

Quantization trades model quality for reduced memory and faster inference. Ollama uses the GGUF format, which supports many quantization schemes. Understanding which to choose is critical:

The sweet spot for most production use cases is Q4_K_M — it preserves ~98% of the model's quality while reducing memory by 70%. Only move to Q5_K_M or Q8_0 when quality degradation is measurably impacting your application .

3.2 Keep-Alive and Model Swapping

By default, Ollama keeps a loaded model in memory for 5 minutes after the last request. This prevents cold-start latency on subsequent requests. You can tune this:

1# Keep model loaded for 30 minutes
2ollama run llama3.1:8b --keep-alive 30m
3
4# Keep model loaded indefinitely
5ollama run llama3.1:8b --keep-alive -1
6
7# Immediately unload after response
8ollama run llama3.1:8b --keep-alive 0

In a multi-model production environment, set OLLAMA_KEEP_ALIVE as an environment variable:

1export OLLAMA_KEEP_ALIVE="15m"

When to increase keep-alive:

• High-traffic production services where cold starts are unacceptable
• Interactive chat applications with 5–15 minute session gaps

When to decrease keep-alive:

• Memory-constrained environments serving multiple models
• Batch processing where you intentionally swap models between jobs

3.3 Concurrent Request Handling

Ollama processes requests sequentially per model. When multiple users send requests simultaneously to the same model, they are queued. To handle concurrent workloads:

• Option A: Deploy multiple Ollama instances behind a load balancer, each loading the same model on separate GPUs.
• Option B: Use the OLLAMA_MAX_LOADED_MODELS environment variable to allow multiple model copies in memory (requires enough VRAM).
• Option C: For M-series Macs, Ollama can use the unified memory architecture; increase OLLAMA_NUM_PARALLEL to process multiple requests in a single batch .

1# Allow 4 parallel request slots (requires sufficient memory)
2export OLLAMA_NUM_PARALLEL=4
3
4# Allow 2 models loaded simultaneously
5export OLLAMA_MAX_LOADED_MODELS=2

Footnotes

1. GGUF Quantization Methods — Technical details on GGUF quantization schemes and quality metrics. ↩

2. Ollama FAQ — Frequently asked questions including concurrency and parallelism. ↩

4. Security & Networking Best Practices

4.1 The Default Bind Address Risk

By default, Ollama binds to 127.0.0.1:11434 — localhost only. Never change this to 0.0.0.0 without additional security measures, as the Ollama API has no built-in authentication. Exposing it on a public network allows anyone to:

• Run arbitrary models (consuming all your resources)
• Send any prompt (potential data exfiltration if prompts contain sensitive data)
• Delete models or modify configurations

4.2 Safe Remote Access Patterns

Pattern 1: Reverse Proxy with Authentication

Use nginx or Caddy as a reverse proxy that adds:

• TLS termination (HTTPS)
• API key authentication via header checks
• Rate limiting (e.g., 10 req/min per key)

Pattern 2: SSH Tunnel

For personal remote access, use an SSH tunnel instead of exposing the port:

1# From your local machine
2ssh -L 11434:localhost:11434 user@remote-server
3
4# Now localhost:11434 on your machine is tunneled to the remote Ollama

Pattern 3: Docker Network Isolation

When running Ollama in Docker alongside your application:

1# docker-compose.yml
2services:
3  ollama:
4    image: ollama/ollama:0.5.7
5    volumes:
6      - ollama_data:/root/.ollama
7    networks:
8      - internal
9    # No published ports — only accessible from internal network
10
11  app:
12    image: my-app
13    networks:
14      - internal
15    environment:
16      - OLLAMA_HOST=http://ollama:11434
17
18networks:
19  internal:
20    internal: true  # No external access

4.3 Environment Variable Security

5. Operational Patterns for Production

5.1 Health Monitoring

Ollama provides a basic health check endpoint:

1# Returns "Ollama is running" if healthy
2curl -s http://localhost:11434/
3
4# Check which models are loaded
5curl -s http://localhost:11434/api/ps
6
7# List all available models
8curl -s http://localhost:11434/api/tags

For production monitoring, wrap these in health-check scripts:

1#!/bin/bash
2# healthcheck.sh
3STATUS=$(curl -s -o /dev/null -w "%{http_code}" http://localhost:11434/)
4if [ "$STATUS" != "200" ]; then
5  echo "Ollama is DOWN — HTTP $STATUS"
6  exit 1
7fi
8
9LOADED=$(curl -s http://localhost:11434/api/ps | jq '.models | length')
10if [ "$LOADED" -eq 0 ]; then
11  echo "WARNING: No models currently loaded"
12fi
13
14echo "Ollama healthy — $LOADED model(s) loaded"

5.2 Logging and Observability

Enable structured logging for production:

1export OLLAMA_DEBUG=false  # Never true in production
2# Ollama logs to stderr by default
3# Redirect to a file with rotation
4ollama serve 2>> /var/log/ollama/ollama.log &

For containerized deployments, Ollama logs to stdout/stderr automatically — capture them with your container orchestration platform's logging driver.

5.3 Model Management Lifecycle

Key practices:

• Always pull models explicitly (ollama pull llama3.1:8b) rather than relying on auto-pull, which can fail silently in production.
• Version-pin models in deployment scripts. Avoid bare tags like latest in production.
• Use ollama rm to clean up unused models periodically to reclaim disk space.
• Back up your custom Modelfiles and any fine-tuned adapters — Ollama stores these in ~/.ollama/models/ .

Footnotes

1. Ollama Docker Guide — Official Docker images and deployment instructions. ↩

6. Common Pitfalls and Debugging

6.1 "The model is really slow"

Check these in order:

1. VRAM spillover: Run nvidia-smi during inference. If VRAM is maxed out, your model is partially on CPU. Switch to a smaller quant or model.
2. Context too long: Large contexts create enormous KV caches. Reduce num_ctx if you don't need it.
3. CPU inference on Mac: On Intel Macs, Ollama runs on CPU only. M-series Macs use the Neural Engine and GPU via Metal — ensure you're on M1+.

6.2 "Model quality is poor"

1. Wrong quantization: Q2_K and Q3_K produce noticeably worse outputs. Upgrade to Q4_K_M or Q5_K_M.
2. Bad parameters: High temperature (0.8+) causes randomness. Lower it to 0.2–0.3 for factual tasks.
3. Missing/incomplete system prompt: Many models rely on a specific system prompt format. Check the model's documentation.

6.3 "Out of memory errors"

1. Set OLLAMA_NUM_GPU to fewer layers, forcing more onto CPU (slower but stable).
2. Reduce num_ctx — this is the most common OOM trigger.
3. Close other GPU applications — VRAM isn't shared well across processes.
4. Use a smaller model — sometimes the simplest solution is correct.