Observability and Site Reliability

Programme overview

Across this track, Tallybook's month-end outage has been blamed on too few web servers and on a cryptominer. In this course you find the real cause. With a day of per-minute metrics you'll read the four golden signals and see that the database connection pool, not the web tier, ran out; with structured logs you'll find the bulk invoice-sending job that started at 09:40:02; with distributed traces you'll see requests spending seconds waiting for a connection while their queries barely slowed. Then you'll build the reliability practice that prevents a repeat: SLIs and SLOs with error budgets (two mornings used about two and a half months' worth), multi-window burn-rate alerts, a clean-up of pages that were nine-tenths noise, connection pool sizing with Little's law and job isolation, and an error budget policy with toil measured and cut. Every number comes from running the code on the incident's data.

Curriculum

1. 01What Observability Is15 min

   • What observability isWhat observability means, the three kinds of telemetry (metrics, logs and traces), the four golden signals, and a first look at the minute-by-minute record of Tallybook's month-end outage.
2. 02Metrics and Percentiles25 min

   • Metrics and percentilesWhy averages hide what users experience, how percentiles (p50, p95, p99) show it, how to read latency over a day, and which metrics to put on a service's first dashboard.
3. 03Structured Logs25 min

   • Structured logsWhy logs should be structured (JSON) rather than free text, how to filter and count them like data, and how to use them to find the event that started an incident.
4. 04Distributed Traces15 min

   • Distributed tracesRead traces made of spans, rebuild a request's path through the web tier, API, database and email provider, and compare where time goes in normal and incident traffic.
5. 05SLIs and SLOs25 min

   • SLIs and SLOsDefine service level indicators from what users experience, set objectives and error budgets, and measure Tallybook's August against them, including how much of the month's budget two mornings used.
6. 06Burn-Rate Alerts25 min

   • Burn-rate alertsAlert on how fast the error budget is being spent rather than on fixed thresholds, compute burn rates over short and long windows, and see when each alert would have fired on 31 August.
7. 07Alert Quality and On-Call25 min

   • Alert quality and on-callMeasure how useful each alert is (how often it needs action, when it wakes people, how fast it's acknowledged), find the noisy ones, and design an on-call that people can sustain.
8. 08Capacity and Saturation25 min

   • Capacity and saturationUse Little's law to work out how many database connections a workload needs, see exactly when Tallybook's pool ran out, and size the pool and the bulk job so month-end can't do it again.
9. 09Toil and Reliability Work20 min

   • Toil and reliability workMeasure toil (repetitive manual operations work), decide what to automate first, and set the error budget policy and practices that keep reliability work from being crowded out.
10. 10Final Project20 min

    • Final project: Tallybook's reliability reviewPlan your final project, a reliability review that finds the real cause of the month-end outages with metrics, logs and traces, and sets the SLOs, alerts, capacity plan and practices that prevent them.

Assessment and certificate

You are assessed with a final assessment (pass mark 60%) and a final project. Your certificate is awarded when you meet those requirements. Paying for the programme does not award it on its own.