A postmortem is a short write-up after an incident: what happened, why, and what you'll change so it's less likely to happen again. Its purpose is learning. Once it becomes a hunt for someone to blame, people stop being open and the write-up stops being useful.
Why blameless
Blameless still means people are accountable. You assume everyone acted reasonably with what they knew at the time, and you look at the systems and conditions that allowed the failure. People who expect blame leave out details, and those details are what you need to find the real cause.
The sections
- Summary: two or three sentences someone can read in passing.
- Impact: who was affected, how badly and for how long.
- Timeline: what happened and when, from the first alert to resolution.
- Root cause: the underlying condition as well as the trigger.
- What went well and what didn't: detection, response, communication.
- Action items: specific changes, each with an owner and a due date.
Template
- Summary: On [date], [service] was [impact] for [duration]. The cause was [root cause]. We have [main fix] and are tracking [follow-ups].
- Impact: [who/what was affected], [scope], [start] to [end].
- Timeline: [time] first alert. [time] acknowledged. [time] cause identified. [time] fix applied. [time] resolved.
- Root cause: [the underlying condition that made this possible].
- What went well: [e.g. detection was fast]. What didn't: [e.g. the alert was noisy, or the first customer update was late].
- Action items: [change], owner [name], by [date].
Follow through on action items
Action items are what prevent a repeat, and they only work if each one is specific, owned and dated. Pick a few changes you will actually make. Check on them later. Consider publishing the postmortem, or its summary, on your status page or sending it to affected customers.