AdvancedObservability & Monitoring

Production Prometheus Alerting Rules: SLIs/SLOs

Stop alert fatigue by writing actionable, symptom-based Prometheus recording and alerting rules based on Google SRE principles.

PrometheusAlertmanager

Prerequisites

Running Prometheus instance
Node Exporter

Configuration Files

alerts.yaml/etc/prometheus/alerts.yaml

groups:\n- name: node_alerts\n  rules:\n  - alert: HostOutOfDiskSpace\n    expr: (node_filesystem_avail_bytes{mountpoint="/"}  * 100) / node_filesystem_size_bytes{mountpoint="/"} < 10\n    for: 5m\n    labels:\n      severity: warning\n    annotations:\n      summary: "Host out of disk space (instance {{ $labels.instance }})"

Explanation:Triggers a warning if root filesystem free space drops below 10% for 5 minutes.

Verification Steps

Validates syntax of rule files.

$promtool check rules /etc/prometheus/alerts.yaml

Expected OutputSUCCESS: 1 rules found

Production Gotchas

Avoid static CPU threshold alerts (e.g., CPU > 80%). Focus instead on error rates and request latency (symptom-based alerting).

Frequently Asked Questions

What is the "for" clause?

It requires the alert condition to be true for the specified duration before firing, preventing noise from temporary spikes.