Monitoring & Observability — รู้ทุกอย่างที่เกิดขึ้นใน SystemMonitoring & Observability — Knowing Everything That Happens Inside the System

global:
  scrape_interval: 15s        # ดึง metrics ทุก 15 วินาที

scrape_configs:
  # Monitor ตัว Prometheus เอง
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  # Monitor Node (เครื่อง server)
  - job_name: 'node'
    static_configs:
      - targets:
        - 'server1:9100'
        - 'server2:9100'
        - 'server3:9100'

  # Monitor App
  - job_name: 'myapp'
    metrics_path: '/metrics'
    static_configs:
      - targets: ['app:3000']

version: '3.8'
services:
  prometheus:
    image: prom/prometheus:latest
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus_data:/prometheus

  grafana:
    image: grafana/grafana:latest
    ports:
      - "3001:3000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=secret
    volumes:
      - grafana_data:/var/lib/grafana

  node-exporter:
    image: prom/node-exporter:latest
    ports:
      - "9100:9100"
    volumes:
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro

volumes:
  prometheus_data:
  grafana_data:

# รัน monitoring stack
docker compose up -d

# เข้า Prometheus: http://localhost:9090
# เข้า Grafana:    http://localhost:3001 (admin/secret)

# CPU usage เฉลี่ย 5 นาที
100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

# Request rate ต่อวินาที
rate(http_requests_total[5m])

# Error rate (%)
rate(http_requests_total{status=~"5.."}[5m]) / rate(http_requests_total[5m]) * 100

# 95th percentile response time
histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))

# Memory usage (%)
(1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100

// ❌ Bad — ยากต่อการ parse
console.log("User login failed for user123");

// ✅ Good — structured JSON
logger.error({
  event: "login_failed",
  userId: "user123",
  reason: "invalid_password",
  ip: "203.0.113.50",
  timestamp: "2026-03-07T09:00:00Z"
});

# Loki + Promtail Docker Compose
services:
  loki:
    image: grafana/loki:latest
    ports:
      - "3100:3100"

  promtail:
    image: grafana/promtail:latest
    volumes:
      - /var/log:/var/log:ro
      - ./promtail.yml:/etc/promtail/config.yml

// OpenTelemetry ตัวอย่าง (Node.js)
const { trace } = require('@opentelemetry/api');

const tracer = trace.getTracer('my-service');

app.get('/api/checkout', async (req, res) => {
  const span = tracer.startSpan('checkout');

  try {
    // แต่ละขั้นตอนมี child span
    const userSpan = tracer.startSpan('get-user', { parent: span });
    const user = await getUser(req.userId);
    userSpan.end();

    const paySpan = tracer.startSpan('process-payment', { parent: span });
    await processPayment(user, req.amount);
    paySpan.end();

    res.json({ status: 'success' });
  } catch (err) {
    span.setStatus({ code: 2, message: err.message });
    res.status(500).json({ error: err.message });
  } finally {
    span.end();
  }
});

# alert-rules.yml
groups:
  - name: critical
    rules:
      # Server down
      - alert: InstanceDown
        expr: up == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "{{ $labels.instance }} is down!"

      # CPU สูงเกิน 90% นาน 5 นาที
      - alert: HighCPU
        expr: 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 90
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "CPU > 90% on {{ $labels.instance }}"

      # Error rate สูงเกิน 5%
      - alert: HighErrorRate
        expr: rate(http_requests_total{status=~"5.."}[5m]) / rate(http_requests_total[5m]) * 100 > 5
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Error rate > 5% on {{ $labels.job }}"

      # Disk เหลือน้อยกว่า 10%
      - alert: DiskSpaceLow
        expr: (node_filesystem_avail_bytes / node_filesystem_size_bytes) * 100 < 10
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Disk < 10% on {{ $labels.instance }}"

# alertmanager.yml
route:
  receiver: 'team-slack'
  routes:
    - match:
        severity: critical
      receiver: 'pagerduty'     # Critical → ปลุกคนมาแก้!
    - match:
        severity: warning
      receiver: 'team-slack'    # Warning → แจ้งใน Slack

receivers:
  - name: 'team-slack'
    slack_configs:
      - api_url: 'https://hooks.slack.com/services/xxx'
        channel: '#alerts'
        title: '🚨 {{ .GroupLabels.alertname }}'
        text: '{{ .CommonAnnotations.summary }}'

  - name: 'pagerduty'
    pagerduty_configs:
      - service_key: 'your-pagerduty-key'

In the previous post we learned IaC — using Terraform to build infrastructure and Ansible to configure servers automatically.

But once it is deployed to production... how do you know the app is behaving normally? When is the server going to fall over? What error is a user hitting right now? 🔍

This is where Monitoring & Observability comes in — the system that lets us "see" everything happening inside, like a DevOps engineer sitting and staring at a dashboard all day long 🐕‍🦺📊

---

Monitoring vs Observability — What's the Difference?

Many people use the two terms interchangeably, but they are genuinely different — Monitoring is "checking whether the system is still working normally" (Is CPU high? Is the disk full? Is the error rate climbing?), asking questions we already know to ask. Observability is "being able to understand the state of the system from the data it emits" — answering questions we have never thought to ask. It suits complex microservices whose failure modes are impossible to predict:

Monitoring	Observability
Answers "is something wrong?"	Answers "why is something wrong?"
Checks the things we anticipated	Finds the things we never considered
Dashboard + Alerts	Explore + Drill-down + Correlate
CPU is high → send an alert	Why is CPU high? → trace → found a slow query

💡 An easy way to remember it: Monitoring = installing the security cameras | Observability = having a detective who analyses the footage

---

3 Pillars of Observability

A good observability setup needs three kinds of data that complement each other — Metrics tell you "is something wrong" (numbers), Logs tell you "what happened" (events), Traces tell you "where it is slow" (request flow). Missing any one of them makes debugging hard:

---

Pillar 1: Metrics — The Numbers to Watch

Metrics are numbers measured over time (time-series data) — CPU usage 72%, response time 230ms, error rate 0.5%. They suit alerting (fire when a threshold is crossed) and trending (watching long-term patterns). Metrics come in three levels:

System Metrics

System metrics measure the health of the infrastructure — CPU, memory, disk, network. Every server needs them; they are what tells you "when the machine is going to fall over":

• CPU Usage — how hard the machine is working
• Memory Usage — how much RAM is left
• Disk I/O — how fast the disk is reading and writing
• Network I/O — how much bandwidth is being used

Application Metrics (RED Method)

The RED Method (Rate, Errors, Duration) is the set of metrics measured from the user's point of view — how many requests arrive? what percentage of them error? how slowly does it answer? It matters more than system metrics because it tells you whether the user experience is good:

• Rate — requests per second
• Errors — percentage of errors (5xx)
• Duration — response time (p50, p95, p99)

Business Metrics

Business metrics connect technical data to business impact — orders per minute, signup rate, cart abandonment rate. They help answer questions like "the system got 200ms slower, did that hurt sales?":

• Active Users — how many people are using it right now
• Orders/min — orders per minute
• Revenue — revenue in real time

💡 RED Method: use it for services (Rate, Errors, Duration)
USE Method: use it for resources (Utilization, Saturation, Errors)

---

Prometheus + Grafana — The Most Popular Metrics Stack

Prometheus is a metrics database (TSDB) that stores time-series data — it uses a "pull model", meaning Prometheus itself fetches data from its targets (servers, apps) every 15 seconds. Grafana is the dashboard tool that renders those metrics as beautiful graphs. Both are open-source, CNCF graduated projects, and the industry standard:

Configuring Prometheus

The prometheus.yml file:

global:
  scrape_interval: 15s        # pull metrics every 15 seconds

scrape_configs:
  # Monitor Prometheus itself
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  # Monitor the nodes (the server machines)
  - job_name: 'node'
    static_configs:
      - targets:
        - 'server1:9100'
        - 'server2:9100'
        - 'server3:9100'

  # Monitor the app
  - job_name: 'myapp'
    metrics_path: '/metrics'
    static_configs:
      - targets: ['app:3000']

Docker Compose for the Monitoring Stack

The monitoring stack is easy to set up with Docker Compose — Prometheus + Grafana + Node Exporter (system metrics) + cAdvisor (container metrics), ready to use in a few minutes:

version: '3.8'
services:
  prometheus:
    image: prom/prometheus:latest
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus_data:/prometheus

  grafana:
    image: grafana/grafana:latest
    ports:
      - "3001:3000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=secret
    volumes:
      - grafana_data:/var/lib/grafana

  node-exporter:
    image: prom/node-exporter:latest
    ports:
      - "9100:9100"
    volumes:
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro

volumes:
  prometheus_data:
  grafana_data:

# Run the monitoring stack
docker compose up -d

# Open Prometheus: http://localhost:9090
# Open Grafana:    http://localhost:3001 (admin/secret)

PromQL — Querying Data Out of Prometheus

PromQL is Prometheus's query language — you use it to compute the metrics you want, such as the error rate over 5 minutes, 95th percentile latency, or average CPU usage. Learning PromQL is an essential DevOps skill:

# Average CPU usage over 5 minutes
100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

# Request rate per second
rate(http_requests_total[5m])

# Error rate (%)
rate(http_requests_total{status=~"5.."}[5m]) / rate(http_requests_total[5m]) * 100

# 95th percentile response time
histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))

# Memory usage (%)
(1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100

---

Pillar 2: Logs — The Record of Events

Logs are the record of events that happened in the system — "user X logged in successfully", "payment failed: insufficient funds", "database connection timeout". Metrics tell you "there is a problem", but logs tell you "what actually happened" — which matters enormously when debugging.

Structured Logging — Better Than Plain Text!

Instead of logging plain text ("User john logged in"), use structured logging (JSON format) — it makes searching, filtering and aggregating easy. Ask "every error from user X in the last hour" and you get an answer immediately:

// ❌ Bad — hard to parse
console.log("User login failed for user123");

// ✅ Good — structured JSON
logger.error({
  event: "login_failed",
  userId: "user123",
  reason: "invalid_password",
  ip: "203.0.113.50",
  timestamp: "2026-03-07T09:00:00Z"
});

Log Levels

Log levels help separate what matters — production should log INFO and above only (skip DEBUG, its volume is enormous). When a problem appears, turn DEBUG on temporarily:

Level	Use when	Example
ERROR	A real failure has occurred	Database connection failed
WARN	There may be a problem later	Disk usage > 80%
INFO	A normal but significant event	Server started on port 3000
DEBUG	Details for debugging	Query result: 42 rows

ELK Stack — Centralized Logging

Once you have many servers and containers, SSH-ing into each machine to read its log does not work — you need centralized logging that gathers every log in one place. The ELK Stack (Elasticsearch + Logstash + Kibana) is a popular solution:

A lighter alternative: Loki + Promtail + Grafana (you can keep the Grafana you already have!)

# Loki + Promtail Docker Compose
services:
  loki:
    image: grafana/loki:latest
    ports:
      - "3100:3100"

  promtail:
    image: grafana/promtail:latest
    volumes:
      - /var/log:/var/log:ro
      - ./promtail.yml:/etc/promtail/config.yml

---

Pillar 3: Traces — Following a Request

When a single request has to pass through many services (microservices), how do we know where it is slow?

The trace lets us see that Payment Service took 850ms — that is the bottleneck!

Tracing Tools

The main tracing tools — Jaeger (CNCF, open-source, good with Kubernetes), Zipkin (from Twitter, lightweight), OpenTelemetry (the new standard that unifies metrics + logs + traces in one vendor-neutral SDK):

• Jaeger — open-source, from Uber
• Zipkin — open-source, from Twitter
• OpenTelemetry — the new standard (metrics + logs + traces together)
• Datadog APM / New Relic — commercial (easy but expensive)

// OpenTelemetry example (Node.js)
const { trace } = require('@opentelemetry/api');

const tracer = trace.getTracer('my-service');

app.get('/api/checkout', async (req, res) => {
  const span = tracer.startSpan('checkout');

  try {
    // each step gets its own child span
    const userSpan = tracer.startSpan('get-user', { parent: span });
    const user = await getUser(req.userId);
    userSpan.end();

    const paySpan = tracer.startSpan('process-payment', { parent: span });
    await processPayment(user, req.amount);
    paySpan.end();

    res.json({ status: 'success' });
  } catch (err) {
    span.setStatus({ code: 2, message: err.message });
    res.status(500).json({ error: err.message });
  } finally {
    span.end();
  }
});

---

Alerting — Warning You Before It Breaks

However beautiful the dashboard is, it does not help if nobody is looking at it — you need alerts!

Alert Rules (Prometheus)

Alert rules define "when condition X holds for Y minutes → send an alert". The important part is the for duration, which prevents false alarms from momentary spikes, while labels and annotations help the on-call team understand the problem quickly:

# alert-rules.yml
groups:
  - name: critical
    rules:
      # Server down
      - alert: InstanceDown
        expr: up == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "{{ $labels.instance }} is down!"

      # CPU above 90% for 5 minutes
      - alert: HighCPU
        expr: 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 90
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "CPU > 90% on {{ $labels.instance }}"

      # Error rate above 5%
      - alert: HighErrorRate
        expr: rate(http_requests_total{status=~"5.."}[5m]) / rate(http_requests_total[5m]) * 100 > 5
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Error rate > 5% on {{ $labels.job }}"

      # Less than 10% disk remaining
      - alert: DiskSpaceLow
        expr: (node_filesystem_avail_bytes / node_filesystem_size_bytes) * 100 < 10
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Disk < 10% on {{ $labels.instance }}"

Where Do Alerts Go? (Alertmanager)

Alertmanager receives alerts from Prometheus and forwards them to the appropriate channels — Slack, PagerDuty, email, webhook. It supports grouping (merging identical alerts), silencing (muting temporarily) and routing (critical → PagerDuty, warning → Slack):

# alertmanager.yml
route:
  receiver: 'team-slack'
  routes:
    - match:
        severity: critical
      receiver: 'pagerduty'     # Critical → wake someone up to fix it!
    - match:
        severity: warning
      receiver: 'team-slack'    # Warning → post it in Slack

receivers:
  - name: 'team-slack'
    slack_configs:
      - api_url: 'https://hooks.slack.com/services/xxx'
        channel: '#alerts'
        title: '🚨 {{ .GroupLabels.alertname }}'
        text: '{{ .CommonAnnotations.summary }}'

  - name: 'pagerduty'
    pagerduty_configs:
      - service_key: 'your-pagerduty-key'

🐕 Alert Fatigue: do not put an alert on everything! Too many notifications and people start ignoring them → Critical = act immediately | Warning = look at it today | Info = look when you have time

---

Popular Monitoring Stacks

There are a great many monitoring tools — the table below compares the popular stacks, both open-source and commercial, to help you choose according to your needs (budget, scale, complexity):

Stack	Components	Strengths
PLG	Prometheus + Loki + Grafana	Open-source, K8s friendly, all 3 pillars covered
ELK	Elasticsearch + Logstash + Kibana	Excellent log search, enterprise
TIG	Telegraf + InfluxDB + Grafana	Good for IoT and time-series
Datadog	All-in-one SaaS	The easiest, but expensive
New Relic	All-in-one SaaS	Good APM, free tier

---

4 Golden Signals (Google SRE)

The 4 Golden Signals come from the Google SRE Book — if you can only monitor four things, monitor these four; together they cover the whole user experience. Latency tells you about the experience, Traffic tells you about demand, Errors tell you about quality, Saturation tells you about the limit:

1. Latency — how long does it take to answer? (p50, p95, p99)
2. Traffic — how many requests arrive per second?
3. Errors — what percentage of requests are errors? (HTTP 5xx)
4. Saturation — how full is the system? (CPU, memory, disk, queue)

💡 SLO Tip: set an SLO (Service Level Objective) such as "99.9% uptime" or "p95 latency < 200ms", then monitor whether you actually meet it — that measurement is the SLI (Service Level Indicator)

---

Summary

Monitoring & Observability is a must-have, not a nice-to-have — without it you are driving a car with no dashboard, with no idea when the fuel runs out or how hot the engine is. Start with metrics + alerting (Prometheus + Grafana), then add logs and traces as the system grows more complex:

• Monitoring = "is something wrong?" | Observability = "why is something wrong?"
• 3 Pillars = Metrics (numbers) + Logs (records) + Traces (following a request)
• Prometheus + Grafana = the popular metrics stack, queried with PromQL
• Loki / ELK = centralized logging — every server's logs gathered in one place
• Jaeger / OpenTelemetry = distributed tracing, finding the bottleneck in microservices
• Alerting = notifications through Slack/PagerDuty — beware alert fatigue!
• 4 Golden Signals = Latency, Traffic, Errors, Saturation 📊🐕