SRE — Site Reliability Engineering ที่ DevOps ต้องรู้ 🚒SRE — The Site Reliability Engineering Every DevOps Team Needs 🚒

# Prometheus — วัด SLI
# Latency (p99)
histogram_quantile(0.99,
  rate(http_request_duration_seconds_bucket[5m])
)

# Error Rate
sum(rate(http_requests_total{status=~"5.."}[5m]))
/
sum(rate(http_requests_total[5m]))

# Availability (ใน 30 วัน)
1 - (
  sum(rate(http_requests_total{status=~"5.."}[30d]))
  /
  sum(rate(http_requests_total[30d]))
)

# SLO: User API Service
# ──────────────────────
# SLI: Availability
#   Definition: % of non-5xx responses
#   SLO: 99.9% over 30-day rolling window
#   Error Budget: 0.1% = ~43 minutes/month
#
# SLI: Latency
#   Definition: p99 response time
#   SLO: p99 < 300ms for 99% of requests
#
# SLI: Freshness
#   Definition: data staleness
#   SLO: 95% of reads return data < 1 min old
#
# Consequences:
#   If error budget exhausted:
#   → Freeze feature deployments
#   → Focus on reliability improvements
#   → Require SRE approval for any change

# Grafana Dashboard — Error Budget Tracker
# ─────────────────────────────────────────
# Total requests (30d)
total = sum(increase(http_requests_total[30d]))

# Failed requests (30d)
failed = sum(increase(http_requests_total{status=~"5.."}[30d]))

# Error budget (SLO 99.9%)
budget_total = total * 0.001        # 0.1% allowed failures
budget_used  = failed
budget_remaining = (budget_total - budget_used) / budget_total * 100

# Alert เมื่อ budget < 20%
# → Slack notification to team
# → Require SRE review for deploys

# Incident Roles (P1/P2)
# ──────────────────────
# 🎖️ Incident Commander (IC)
#    - ตัดสินใจ, coordinate, สั่งการ
#    - ไม่ต้อง debug เอง
#
# 🔧 Technical Lead
#    - วิเคราะห์ root cause
#    - Execute mitigation
#
# 📢 Communications Lead
#    - อัพเดท status page
#    - แจ้ง stakeholders
#    - เขียน customer comms
#
# 📝 Scribe
#    - จด timeline ทุกอย่าง
#    - Record decisions & actions

# Postmortem Template
# ═══════════════════

## Incident: Production API Outage
## Date: 2026-03-07 14:23-14:35 UTC
## Duration: 12 minutes
## Severity: P1
## Impact: 100% of API requests failed for 12 minutes
##         ~18,000 failed requests, ~2,400 users affected

## Summary
Deploy v2.4.1 contained a memory leak in the user session
handler. Under load, pods exceeded memory limits and were
OOMKilled. The auto-scaler created new pods that also crashed,
causing a cascading failure.

## Root Cause
Memory leak in UserSessionMiddleware — each request allocated
a new 2MB buffer that was never garbage collected due to a
circular reference. Under 500 req/s, memory grew ~1GB/min.

## Timeline
- 14:20 — Deploy v2.4.1 rolled out (3/6 pods)
- 14:23 — Alert: Error rate > 5%
- 14:25 — IC assigned, war room opened
- 14:30 — Identified memory leak in v2.4.1
- 14:32 — Rollback to v2.4.0
- 14:35 — Service restored

## What Went Well
- Alert fired within 3 minutes of impact
- Quick rollback via ArgoCD (2 minutes)
- Clear incident response process followed

## What Went Wrong
- No memory profiling in CI pipeline
- Load test only ran at 100 req/s (prod = 500 req/s)
- Canary deployment was skipped for "small change"

## Action Items
1. [P1] Add memory leak detection to CI — @somchai — Due: Mar 14
2. [P1] Load test at 2x production traffic — @somying — Due: Mar 14
3. [P2] Enforce canary deployment for all changes — @team — Due: Mar 21
4. [P3] Add OOMKill alert (before cascade) — @ops — Due: Mar 28

## Lessons Learned
- "Small changes" can have big impacts — always canary
- Memory profiling should be part of CI, not manual
- Rollback capability (ArgoCD) saved us — invest in it

# วัด Toil — track เวลาที่ใช้
# ──────────────────────────
# Week 1: SRE Team Time Breakdown
#
# Engineering work:  25 hours (50%)  ← target: > 50%
# Toil:              15 hours (30%)  ← target: < 50%
# On-call:            5 hours (10%)
# Training/misc:      5 hours (10%)
#
# Top Toil Items:
# 1. Manual DB backups         — 4h/week → AUTOMATE
# 2. Certificate renewals      — 3h/week → cert-manager
# 3. Capacity request tickets  — 3h/week → self-service
# 4. Log investigation         — 5h/week → better alerting

# PagerDuty Schedule Example
# ──────────────────────────
# Primary:   สมชาย → สมหญิง → สมศักดิ์ (rotate weekly)
# Secondary: สมศักดิ์ → สมชาย → สมหญิง (backup)
#
# Rules:
# - Primary responds within 5 min (P1) / 30 min (P2)
# - If no response in 10 min → escalate to Secondary
# - If no response in 20 min → escalate to Engineering Manager
#
# Compensation:
# - On-call allowance per week
# - Time off after high-severity incidents
# - No more than 25% of time on-call per quarter

# Prometheus Alert Rules
groups:
  - name: sre-alerts
    rules:
      # ✅ ดี: User-facing, actionable
      - alert: HighErrorRate
        expr: |
          sum(rate(http_requests_total{status=~"5.."}[5m]))
          / sum(rate(http_requests_total[5m])) > 0.01
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Error rate > 1% for 5 minutes"
          runbook: "https://wiki.example.com/runbooks/high-error-rate"
          dashboard: "https://grafana.example.com/d/api-overview"

      # ✅ ดี: SLO-based alert
      - alert: ErrorBudgetBurnRate
        expr: |
          error_budget_remaining < 0.2
        labels:
          severity: warning
        annotations:
          summary: "Error budget < 20% remaining this month"
          action: "Consider slowing deployments"

      # ❌ แย่: CPU alert (not user-facing)
      # - alert: HighCPU
      #   expr: cpu_usage > 80
      #   → CPU สูงไม่ได้แปลว่า user ได้รับผลกระทบ!

# Capacity Planning Questions
# ──────────────────────────
# 1. ปัจจุบันใช้ resource เท่าไร? (baseline)
# 2. Growth rate เท่าไร? (trend)
# 3. เมื่อไรจะถึง limit? (forecast)
# 4. ต้องเพิ่มอะไร เมื่อไร? (plan)

# PromQL — Forecast disk full
predict_linear(
  node_filesystem_avail_bytes{mountpoint="/"}[7d],
  30 * 24 * 3600    # predict 30 days ahead
) < 0
# → Alert: "Disk will be full in < 30 days"

# PromQL — Traffic growth trend
rate(http_requests_total[30d])
# → ถ้าโต 20%/month → plan capacity 3 months ahead

# Chaos Experiments ที่ควรทำ
# ──────────────────────────
# 1. Kill random pod       → K8s จะสร้างใหม่มั้ย?
# 2. Increase latency      → App handle timeout ดีมั้ย?
# 3. Fill disk 100%        → Alert ทันมั้ย? App crash มั้ย?
# 4. Kill database primary → Failover ทำงานมั้ย?
# 5. DNS failure           → Retry logic ทำงานมั้ย?
# 6. Network partition     → Split-brain มั้ย?

# LitmusChaos — K8s Chaos Engineering
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: pod-kill-test
spec:
  appinfo:
    appns: production
    applabel: app=myapp
  chaosServiceAccount: litmus-admin
  experiments:
    - name: pod-delete
      spec:
        components:
          env:
            - name: TOTAL_CHAOS_DURATION
              value: '30'          # 30 seconds of chaos
            - name: CHAOS_INTERVAL
              value: '10'          # kill pod every 10 sec
            - name: FORCE
              value: 'true'

In the previous post we covered Cloud Architecture — designing a resilient system on the cloud. But once it is built... who keeps it running 24/7?

The answer is SRE — Site Reliability Engineering — the idea from Google that turns "the system doesn't go down" into a matter of engineering rather than luck 🚒🐕

"SRE is what happens when you ask a software engineer to design an operations team." — Ben Treynor, VP Engineering @ Google (the founder of SRE)

---

🤔 What Is SRE? How Is It Different from DevOps?

SRE (Site Reliability Engineering) is the discipline of applying a software engineering approach to operations problems:

Plenty of people are unsure how SRE differs from DevOps. The table below makes the difference clear — in particular, SRE carries a much more explicit notion of "measure it with numbers" and "cap the toil work" than DevOps, which is mostly about culture and process.

Feature	Traditional Ops	DevOps	SRE
Focus	Keeping the system alive	Culture + CI/CD	Reliability engineering
Approach	Manual, ticket-based	Automate everything	Error budgets + engineering
Goal	The system runs	Deploy fast	Reliable + deploy fast
Toil	Accepted	Chipped away at	Capped at < 50% of the time
Risk	Avoided	Accepted	Calculated with an error budget
On-call	Every problem	Shared	Governed by an explicit policy

The table shows that SRE is "DevOps made concrete" through clearly defined metrics. Instead of saying "the system must be stable", SRE says "the system must have 99.9% uptime" and specifies exactly how that is measured. The error budget idea turns the balance between stability and shipping new features into a question of data rather than of feelings.

💡 An easy way to remember it: DevOps = "the culture" (philosophy) | SRE = "how it is actually done" (implementation) — SRE implements DevOps

---

📊 SLI / SLO / SLA — How Do You Measure Reliability?

The heart of SRE is measuring reliability with numbers, not with feelings:

SLI, SLO and SLA are the core of SRE: they turn "stability" into numbers rather than a feeling or a vague phrase. Understanding how the three differ helps us design systems and measure them effectively.

The diagram shows that the SLI is the measurement collected from the real system, the SLO is the target we set based on that SLI, and the SLA is the agreement with the customer that carries a real consequence (a refund, for instance). The key rule is that the SLO must be stricter than the SLA, because you need a buffer to absorb risk.

The SLIs that matter — "The Four Golden Signals"

Google recommends starting with the "four golden signals", which together cover both the user experience and the health of the system. Having all four gives you a full picture and lets you catch problems quickly.

Signal	What it measures	Example SLI
🕐 Latency	Response time	p50 = 50ms, p99 = 200ms
📊 Traffic	Request volume	1,500 req/sec
❌ Errors	Error rate	0.1% of requests = 5xx
💾 Saturation	How much resource is in use	CPU 70%, Memory 80%

The important detail is that latency must separate successful requests from failed ones, because errors are usually faster (fail fast) and can hide the real problem. Traffic tells you the demand, errors tell you the quality, and saturation tells you the system's limit. All four together give the complete picture.

# Prometheus — measuring SLIs
# Latency (p99)
histogram_quantile(0.99,
  rate(http_request_duration_seconds_bucket[5m])
)

# Error Rate
sum(rate(http_requests_total{status=~"5.."}[5m]))
/
sum(rate(http_requests_total[5m]))

# Availability (over 30 days)
1 - (
  sum(rate(http_requests_total{status=~"5.."}[30d]))
  /
  sum(rate(http_requests_total[30d]))
)

The Prometheus code above shows how to compute the basic SLIs. Using a histogram for latency gives you the flexibility to compute any percentile later. For the error rate, watch out for division by zero and provide a fallback value. Computing availability over a 30-day window gives a longer-term view that is far steadier than looking at a short interval.

An example SLO document

# SLO: User API Service
# ──────────────────────
# SLI: Availability
#   Definition: % of non-5xx responses
#   SLO: 99.9% over 30-day rolling window
#   Error Budget: 0.1% = ~43 minutes/month
#
# SLI: Latency
#   Definition: p99 response time
#   SLO: p99 < 300ms for 99% of requests
#
# SLI: Freshness
#   Definition: data staleness
#   SLO: 95% of reads return data < 1 min old
#
# Consequences:
#   If error budget exhausted:
#   → Freeze feature deployments
#   → Focus on reliability improvements
#   → Require SRE approval for any change

---

💰 Error Budget — A Budget for Getting It Wrong

An error budget is the "budget" a system has for failing — computed from the SLO:

The error budget is a revolutionary idea: it turns the balance between stability and new feature work into arithmetic rather than a fight between Dev and Ops. The table below shows that a higher SLO (99.99%) leaves a very small error budget, which means far more care is needed on every deploy.

SLO	Error budget/month	Error budget/year
99%	7.3 hours	3.65 days
99.9%	43.8 minutes	8.76 hours
99.95%	21.9 minutes	4.38 hours
99.99%	4.38 minutes	52.6 minutes

The table shows that the 0.09% gap between 99.9% and 99.99% makes the error budget ten times smaller. This is exactly why an SLO has to be set to match a business requirement rather than chasing "more nines is always better". The bars below show what tracking an error budget looks like in practice, so the team can see where it stands and decide in time.

The error budget bars above show the situations you are likely to meet. At 65% left (green) you can keep deploying as usual. At 15% left (amber) you need to start being careful and may have to hold back some features. When it is gone (red) you freeze new feature deploys immediately and focus on fixing reliability. This policy gives the team a clear incentive to build a stable system.

💡 The principle: Plenty of error budget left → deploy features fast | Error budget running low → slow down, focus on reliability | Error budget exhausted → feature freeze!

# Grafana Dashboard — Error Budget Tracker
# ─────────────────────────────────────────
# Total requests (30d)
total = sum(increase(http_requests_total[30d]))

# Failed requests (30d)
failed = sum(increase(http_requests_total{status=~"5.."}[30d]))

# Error budget (SLO 99.9%)
budget_total = total * 0.001        # 0.1% allowed failures
budget_used  = failed
budget_remaining = (budget_total - budget_used) / budget_total * 100

# Alert when budget < 20%
# → Slack notification to team
# → Require SRE review for deploys

---

🚨 Incident Management — When the System Goes Down

Severity levels

Clearly defined severity levels let the team prioritize and respond appropriately — not every problem is an "emergency". Having different response times for different levels lets the team spend its energy well and avoid burnout.

Level	Impact	Response Time	Example
P1 Critical	Service completely down	< 5 minutes	Production database down
P2 High	Major feature broken	< 30 minutes	Payments not working
P3 Medium	Minor feature broken	< 4 hours	Search slower than usual
P4 Low	Cosmetic / minor	Next business day	Typo on the error page

The table shows that P1 is serious business (5 minutes) because it affects every user, while P4 can wait because it does not touch functionality. The guiding principle is "impact on the user", not which server broke — a monitoring server going down may only be a P3 if it does not directly affect the user experience.

Incident response flow

The incident response process needs clear steps that are rehearsed regularly, rather than waiting for a real outage and then working out what to do. The flow below puts "mitigation before investigation", because stopping the damage matters more than finding the root cause in those first minutes.

This flow shows how much detecting quickly matters, and that mitigating does not have to be a permanent fix — a rollback or a workaround will do. The important point is that the postmortem is not "optional": it is part of the incident response, and it is what stops the same problem happening again.

An example incident timeline

A good timeline shows the decision-making process and the lessons learned. Recording accurate times makes the postmortem credible, and tracking the action items makes the response to the next incident better. The example below is an incident resolved in 12 minutes, which is very fast.

The timeline shows that having a good deployment tool (ArgoCD) makes the rollback fast — two minutes. Communication with customers happens after the system is stable, not while the team is still firefighting. A timeline recorded like this becomes a key input to the postmortem, where you work out what went well and what needs improving.

Incident commander — the roles

# Incident Roles (P1/P2)
# ──────────────────────
# 🎖️ Incident Commander (IC)
#    - Decides, coordinates, gives direction
#    - Does not debug personally
#
# 🔧 Technical Lead
#    - Analyses the root cause
#    - Executes the mitigation
#
# 📢 Communications Lead
#    - Updates the status page
#    - Informs stakeholders
#    - Writes the customer comms
#
# 📝 Scribe
#    - Records every step of the timeline
#    - Records decisions & actions

---

📝 Blameless Postmortem — Learning from Failure

A postmortem is the document written after an incident — focused on "learning", not "blaming people" (blameless!):

The blameless postmortem is central to SRE culture, because if people fear blame nobody will want to report a problem or write an honest postmortem. Building a culture that learns from failure is what makes a system stronger over the long run. The template below shows the full structure of a good postmortem.

# Postmortem Template
# ═══════════════════

## Incident: Production API Outage
## Date: 2026-03-07 14:23-14:35 UTC
## Duration: 12 minutes
## Severity: P1
## Impact: 100% of API requests failed for 12 minutes
##         ~18,000 failed requests, ~2,400 users affected

## Summary
Deploy v2.4.1 contained a memory leak in the user session
handler. Under load, pods exceeded memory limits and were
OOMKilled. The auto-scaler created new pods that also crashed,
causing a cascading failure.

## Root Cause
Memory leak in UserSessionMiddleware — each request allocated
a new 2MB buffer that was never garbage collected due to a
circular reference. Under 500 req/s, memory grew ~1GB/min.

## Timeline
- 14:20 — Deploy v2.4.1 rolled out (3/6 pods)
- 14:23 — Alert: Error rate > 5%
- 14:25 — IC assigned, war room opened
- 14:30 — Identified memory leak in v2.4.1
- 14:32 — Rollback to v2.4.0
- 14:35 — Service restored

## What Went Well
- Alert fired within 3 minutes of impact
- Quick rollback via ArgoCD (2 minutes)
- Clear incident response process followed

## What Went Wrong
- No memory profiling in CI pipeline
- Load test only ran at 100 req/s (prod = 500 req/s)
- Canary deployment was skipped for "small change"

## Action Items
1. [P1] Add memory leak detection to CI — @somchai — Due: Mar 14
2. [P1] Load test at 2x production traffic — @somying — Due: Mar 14
3. [P2] Enforce canary deployment for all changes — @team — Due: Mar 21
4. [P3] Add OOMKill alert (before cascade) — @ops — Due: Mar 28

## Lessons Learned
- "Small changes" can have big impacts — always canary
- Memory profiling should be part of CI, not manual
- Rollback capability (ArgoCD) saved us — invest in it

The template shows how separating "What Went Well" from "What Went Wrong" produces a balanced view instead of focusing only on the failures. Action items need an owner and a due date — not a vague "we will improve". Tracking those action items through to completion is the part that most often gets overlooked.

🐕 Blameless culture: "Somchai deployed buggy code" → ❌ | "The system has no memory check in the CI pipeline that would have caught this bug" → ✅

---

🔧 Toil Reduction — Cutting Manual Work

Toil = work that is manual, repetitive, automatable, and has no lasting value — Google's rule is that an SRE should spend no more than 50% of their time on toil:

Toil (reduce this)	Engineering (increase this)
Restarting services by hand	Build auto-restart + self-healing
Manually scale servers	Auto-scaling policies
Copy-paste deployment steps	CI/CD pipeline automation
Answering "reset password" tickets	Self-service password reset
Reading logs by eye	Automated anomaly detection
Manual certificate renewal	cert-manager auto-renew

The comparison shows that toil tends to be repetitive, predictable and without lasting value, while engineering work creates leverage — you invest once and keep benefiting. Turning a manual task into automation is not just writing a script: it also means designing the monitoring, the error handling and the long-term maintenance.

# Measuring toil — track where the time goes
# ──────────────────────────────────────────
# Week 1: SRE Team Time Breakdown
#
# Engineering work:  25 hours (50%)  ← target: > 50%
# Toil:              15 hours (30%)  ← target: < 50%
# On-call:            5 hours (10%)
# Training/misc:      5 hours (10%)
#
# Top Toil Items:
# 1. Manual DB backups         — 4h/week → AUTOMATE
# 2. Certificate renewals      — 3h/week → cert-manager
# 3. Capacity request tickets  — 3h/week → self-service
# 4. Log investigation         — 5h/week → better alerting

Tracking time like this gives a clear picture of where the team's hours actually go. The 50% engineering target is not a magic number, but a guideline that stops the team drowning in manual work with no time left to improve the system. Naming the "Top Toil Items" lets you prioritize what to automate by impact.

---

📟 On-Call Best Practices

On-call rotation

# PagerDuty Schedule Example
# ──────────────────────────
# Primary:   Somchai → Somying → Somsak (rotate weekly)
# Secondary: Somsak → Somchai → Somying (backup)
#
# Rules:
# - Primary responds within 5 min (P1) / 30 min (P2)
# - If no response in 10 min → escalate to Secondary
# - If no response in 20 min → escalate to Engineering Manager
#
# Compensation:
# - On-call allowance per week
# - Time off after high-severity incidents
# - No more than 25% of time on-call per quarter

The on-call structure above balances coverage against fairness. A secondary backup takes pressure off the primary and creates a safety net, and capping on-call at 25% of the quarter guards against burnout. Most important is a clear escalation path, so the on-call engineer is never left carrying more than they can handle.

Alerting — a good alert vs a bad one

Designing good alerting is an art of balancing "catch every problem" against "don't interrupt people for nothing". Alert fatigue is the great enemy of SRE: when there are too many alerts, people start ignoring even the real ones. The rule is that an alert must always be actionable.

✅ A good alert	❌ A bad alert
Actionable — something has to be done	Informational — just so you know
User-facing impact	Internal metric spike
"Error rate > 1% for 5 min"	"CPU > 80% for 1 min"
Has a runbook link	Says nothing about what to do
Low alert fatigue	Alerts every 5 minutes (noise!)

The table shows that a good alert is about "impact on the user", not "server metrics" — high CPU does not always mean users are affected. A runbook link lets the on-call engineer respond quickly, especially in the middle of the night or when they are new and do not yet know the system.

# Prometheus Alert Rules
groups:
  - name: sre-alerts
    rules:
      # ✅ Good: user-facing, actionable
      - alert: HighErrorRate
        expr: |
          sum(rate(http_requests_total{status=~"5.."}[5m]))
          / sum(rate(http_requests_total[5m])) > 0.01
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Error rate > 1% for 5 minutes"
          runbook: "https://wiki.example.com/runbooks/high-error-rate"
          dashboard: "https://grafana.example.com/d/api-overview"

      # ✅ Good: SLO-based alert
      - alert: ErrorBudgetBurnRate
        expr: |
          error_budget_remaining < 0.2
        labels:
          severity: warning
        annotations:
          summary: "Error budget < 20% remaining this month"
          action: "Consider slowing deployments"

      # ❌ Bad: CPU alert (not user-facing)
      # - alert: HighCPU
      #   expr: cpu_usage > 80
      #   → High CPU does not mean users are affected!

The Prometheus alert rules above show the difference between a good alert and a bad one. HighErrorRate is about user impact and carries full context (runbook, dashboard). The error budget alert is a proactive warning that heads off a problem before it happens. The CPU alert is commented out because it does not relate directly to the user experience. And note that `for: 5m` is what filters out transient spikes.

---

📈 Capacity Planning

Capacity planning is "guessing the future" in a principled way, instead of waiting for the system to overload and then panic-buying. Forecasting from data lets you plan and procure infrastructure in time — which matters especially in an era when hardware procurement is slow.

# Capacity Planning Questions
# ──────────────────────────
# 1. How much resource are we using now? (baseline)
# 2. What is the growth rate? (trend)
# 3. When do we hit the limit? (forecast)
# 4. What do we add, and when? (plan)

# PromQL — Forecast disk full
predict_linear(
  node_filesystem_avail_bytes{mountpoint="/"}[7d],
  30 * 24 * 3600    # predict 30 days ahead
) < 0
# → Alert: "Disk will be full in < 30 days"

# PromQL — Traffic growth trend
rate(http_requests_total[30d])
# → if it grows 20%/month → plan capacity 3 months ahead

The PromQL above shows how to use predict_linear() to forecast disk usage and traffic growth. Predicting 30 days out gives you enough time to procure new resources — but be careful, because a linear prediction may not suit seasonal traffic or a sudden spike. An alert that fires in advance is what saves you from firefighting late at night.

---

🛡️ Chaos Engineering — Testing for Resilience

Chaos engineering = break your own system before the real one breaks, to find the weak points:

Chaos engineering means breaking your own system deliberately, to find the weak points before they fail at the worst possible moment. Controlled failure teaches us how the system behaves in abnormal conditions and gets the recovery process ready.

# Chaos experiments worth running
# ──────────────────────────
# 1. Kill random pod       → will K8s create a new one?
# 2. Increase latency      → does the app handle timeouts well?
# 3. Fill disk 100%        → does the alert fire in time? does the app crash?
# 4. Kill database primary → does failover work?
# 5. DNS failure           → does the retry logic work?
# 6. Network partition     → do we get split-brain?

# LitmusChaos — K8s Chaos Engineering
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: pod-kill-test
spec:
  appinfo:
    appns: production
    applabel: app=myapp
  chaosServiceAccount: litmus-admin
  experiments:
    - name: pod-delete
      spec:
        components:
          env:
            - name: TOTAL_CHAOS_DURATION
              value: '30'          # 30 seconds of chaos
            - name: CHAOS_INTERVAL
              value: '10'          # kill pod every 10 sec
            - name: FORCE
              value: 'true'

The list of chaos experiments above starts simple (kill a pod) and works up to the complex (network partition). The LitmusChaos configuration tests pod resilience by killing a pod every 10 seconds for 30 seconds. Setting FORCE=true makes it an immediate kill rather than a graceful shutdown, to simulate the real-world failures that are usually unexpected.

💡 The chaos rules: start in staging, then move to production | run it during working hours (with people watching) | always have a rollback plan

---

📋 SRE Checklist — Where Do You Start?

Adopting SRE in an organization has to happen one step at a time, not all at once, because SRE needs both technical implementation and a culture change that takes time. The checklist below is ordered by priority: the foundation (items 1–3) has to be solid before you move on.

1. Define your SLIs — measure latency, error rate, availability
2. Set your SLOs — reasonable targets (they do not have to be 100%!)
3. Track the error budget — a dashboard showing the budget remaining
4. Set up alerting — SLO-based alerts, not CPU/memory alerts
5. Incident process — severity levels, response playbooks, war room
6. Blameless postmortems — write one for every P1/P2, track the action items
7. Reduce toil — measure → automate → measure again
8. On-call policy — rotation, compensation, escalation
9. Capacity planning — forecast growth, plan ahead
10. Chaos engineering — break it before it breaks itself

Starting with SLI/SLO is not because it is easy, but because it is the shared language that lets the team talk to each other. Once the SLO exists, the error budget follows naturally, and alerting gains a clear direction. Blameless postmortems gradually shift the culture, and the final steps optimize the system into something genuinely resilient.

---

Summary

• SRE = Software engineering approach to operations — "SRE implements DevOps"
• SLI/SLO/SLA = measuring reliability with numbers (Indicator → Objective → Agreement)
• Four Golden Signals = Latency, Traffic, Errors, Saturation
• Error Budget = "a budget for getting it wrong" — exhausted = feature freeze!
• Incident Management = Detect → Triage → Mitigate → Resolve → Postmortem
• Blameless Postmortem = blame the system, not the person → build a culture that learns
• Toil < 50% = measure the manual work, then automate it away
• Chaos Engineering = test resilience before it breaks for real 🚒🐕