Prometheus in Production

Running Prometheus in production is fundamentally different from running it in development. At scale, you face challenges around high availability, cardinality management, long-term storage, and operational resilience that require deliberate architectural decisions.

Prometheus was designed with a philosophy of locality and simplicity: each Prometheus server scrapes metrics from targets it can reach directly, stores data locally in a TSDB, and evaluates alerts and recording rules independently. This design trades distributed consensus for operational simplicity — but it also means that scaling beyond a single instance requires you to extend the architecture thoughtfully.

The diagram above illustrates a typical production topology: shard by function, send to long-term storage via remote write, and optionally federate for cross-cluster visibility. Each decision in this architecture — sharding strategy, storage backend, HA approach — comes with trade-offs that we'll explore in depth 2.

Footnotes

1. How to Build a Scalable Prometheus Architecture - Logz.io guide on scaling Prometheus with federation and remote write ↩

2. Prometheus Federation Explained: Architecture & Pitfalls - Groundcover deep dive on federation architecture and challenges ↩

Scaling Stages: Where Are You?

Not every organization needs the same Prometheus architecture. Your approach should match your scale:

At the 1–5 million active series mark, a single instance becomes viable only with aggressive optimization — functional sharding (one instance for infrastructure, one for application metrics, one for business metrics) and remote write to object storage for long-term retention .

Footnotes

1. Prometheus Scalability: High Cardinality And How To Fix It - Scaling stages from 100K to 50M+ active series ↩

Cardinality: The Silent Killer

Cardinality is the single most impactful factor on Prometheus performance in production. Each unique combination of label values creates a separate time series, requiring its own chunk in memory and on disk.

Consider a metric http_requests_total with labels instance (3 values), method (5 values), and endpoint (1,000 values):

$\text{Cardinality} = 3 \times 5 \times 1000 = 15{,}000 \text{ series}$

That's 15,000 separate chunks in the TSDB index, each needing memory for in-memory head chunks, index entries in postings and symbols, and disk I/O for compaction and querying. At a scrape interval of 15s, this generates approximately 4 million samples per hour across all series. A real-world optimization at one company reduced their active series from 10M to 877K — a 92% memory reduction (from ~60GB to <5GB) — simply by removing unnecessary labels and metrics 2.

Detecting High Cardinality

Use Prometheus's built-in introspection tools:

Footnotes

1. Understanding and Optimizing Resource Consumption in Prometheus - Palark case study: 92% memory reduction through cardinality optimization ↩

2. How to Manage High Cardinality Metrics in Prometheus and Kubernetes - Grafana Labs guide on cardinality management strategies ↩

Recording Rules: Pre-Computing for Performance

Recording rules are the most underutilized tool in the Prometheus toolbox at scale. They allow you to:

1. Reduce query latency: Dashboards query a pre-computed recording rule instead of running expensive PromQL at render time
2. Reduce cardinality: Aggregate away high-cardinality dimensions (e.g., drop endpoint while keeping job and method)
3. Compose complex expressions: Break multi-stage calculations into named, debuggable intermediates

Naming convention is critical for maintainability. The Prometheus community recommends the pattern:

$\text{level:metric:operations}$

For example: job:http_requests:rate5m, service:availability:ratio_rate5m, cluster:node_cpu:avg_5m

This convention encodes the aggregation level, the source metric, and the operation applied — making it immediately clear what a recording rule produces and how it can be consumed by dashboards and alerts .

Rule Organization Best Practices

Structure your rule files logically to support team ownership and independent deployment:

/etc/prometheus/rules/
├── recording_rules/
│   ├── infrastructure.yml
│   ├── application.yml
│   └── sli_slo.yml
└── alerting_rules/
    ├── critical_alerts.yml
    └── warning_alerts.yml

Always validate rules before deploying using promtool check rules <file>. In Kubernetes environments, use the PrometheusRule custom resource provided by the Prometheus Operator for declarative rule management .

Footnotes

1. Prometheus Recording Rules Documentation - Official Prometheus docs on recording and alerting rule syntax ↩ ↩²

High Availability: Making Prometheus Resilient

Prometheus has no built-in clustering — HA is achieved by running redundant, independent instances. The key design principle is: each replica scrapes the same targets independently and evaluates the same rules independently. This means:

• No shared state: Each replica maintains its own TSDB
• No split-brain risk: There's no consensus protocol to worry about
• Duplicate data is expected: Deduplication happens at the query layer (Thanos/Mimir/Cortex)

External labels are essential for HA deduplication. Configure unique replica labels:

1global:
2  external_labels:
3    cluster: 'us-east-1'
4    replica: 'prom-1'

The LTS backend uses the replica label to identify and deduplicate data from both instances .

Footnotes

1. Prometheus High Availability (HA) - New Relic documentation on HA configuration with external labels ↩

Monitoring Prometheus Itself

A production Prometheus that isn't monitored is a ticking time bomb. Key self-monitoring metrics:

Set alerts on:

• Memory pressure: process_resident_memory_bytes > 0.8 * available_memory
• Remote write backlog: time() - prometheus_remote_storage_queue_highest_sent_timestamp_seconds > 300
• Scrape failures: up == 0 for critical targets for > 5 minutes
• Rule evaluation delays: prometheus_rule_evaluation_duration_seconds{quantile="0.99"} > 10