Rolling Out IncidentOps for a Client in Eight Steps

Clients rarely ask a consultant to “install IncidentOps”. They ask why the last outage took a whole afternoon, or why three managers were shouting in the same chat. The fix is a short, ordered rollout, and this guide lays it out as eight steps you can carry into an engagement. It assumes a client with a few services, an informal on-call habit and no written severity policy.

Throughout, the framing comes from the Ops Atlas entry covering IncidentOps, which treats the name as an informal alias for incident management. Take what is useful and ignore the label if it irritates the client.

Step 1: size the client before you design anything

The Atlas offers a tiered view that saves you from over-engineering. Tier 0 is ticket handling only. Tier 1 adds one rota and one template. Tier 2 brings in a commander, a communications lead and a status page, and is justified when incidents regularly cross team lines or when two or more serious ones land in a month. Tier 3 adds a deputy, liaisons and sub-teams, which makes sense only when more than five responders routinely pile in.

Count the client’s incidents over the last quarter, note how many teams each touched, and pick the tier. Most small clients land on Tier 1 or 2. Write that decision in the first page of your report.

Step 2: write the severity policy in plain English

Use three or four levels. For each, state one observable test, such as “customers cannot pay” or “one internal tool is slow”. Add the declaration criteria that apply to everyone: another team is needed, customers notice, or the problem has survived roughly an hour of effort. The point of writing them down is that nobody should have to argue about whether to raise the alarm at two in the morning.

Step 3: fix the rota maths

This is the step clients skip and later regret. The Atlas works through an example: six engineers covering a round-the-clock rota with two slots each week would carry roughly a third of their time on call, which exceeds the quarter-of-time ceiling cited from Google’s practice. Staffing that rota sustainably needs about eight people. If the client has six, the answer is fewer slots, a follow-the-sun arrangement or a smaller scope, not heroics.

Define the rota in code where you can. If the client uses a paging vendor with a Terraform provider, keep the schedule in a repository so changes get reviewed. One warning from the source belongs in your handover notes: marking a variable as sensitive only hides it from plan output, and the stored state file and raw outputs still reveal the integration routing keys. Use encrypted remote state.

Step 4: hand out role cards

Create one-page cards for the commander, the operations lead, the communications lead and the scribe, plus an optional deputy. State each person’s single job in a sentence. The commander coordinates and does not fix. The operations group is the only one that changes production. The communications lead posts on schedule. Make clear that these are hats worn for the duration of an incident, not job titles.

• Keep each card under 150 words.
• Print them or pin them in the incident channel.
• Add the handover phrase: at a change of shift the outgoing commander asks the newcomer to take over in so many words, then waits for an audible yes.

Step 5: stand up the communication path

Pick a hosted status page and place it on infrastructure that does not share fate with the client’s own platform. Agree the cadence: an initial post within five minutes of the call opening, a scoping update soon after, then fresh updates no later than twenty minutes apart across the opening two hours, and a closing note that states whether any data was lost. Use a working channel for responders and a separate read-only channel for updates, so the room does not swell into a crowd.

Check the tooling calendar while you are at it. The Atlas reports that Opsgenie shuts down on 5 April 2027 and deletes data not migrated by then, so a client still on it needs a migration plan on your roadmap. Also confirm that the paging integration expects an HTTP 202 on event submission and retries with backoff on a 429.

Step 6: run one tabletop exercise

Choose a believable scenario, perhaps a failed release that degrades checkout, and walk the team through it with the cards in hand. Stopwatch three moments: the call to declare, the naming of a commander and the first update going out. The exercise will expose gaps that no document review can. If you need help framing the surrounding engagement, the wider sequence is mapped in our outline of a consulting engagement that runs from first audit to team enablement.

Step 7: tie in security without blurring the lines

When the incident might involve an attacker, the plan changes. The Atlas proposes that the commander keeps coordinating while the security workstream moves into a restricted channel with the response specialists, evidence rules limit what can be restored, and legal approval gates outside statements. Say so in the policy and refer to your wider guidance, for example our notes on weaving security into delivery pipelines. Keep attacker response itself out of scope for this rollout; it deserves its own plan.

Step 8: set the review rhythm and the metrics

Agree that every serious incident gets a review with an owner named before the call ends. Pick a deadline measured in days, keep the action list short, give each action one owner and a date, and agree a target for closing the important ones; a large software vendor, for instance, gives itself roughly one or two months, depending on the service tier. Ask a facilitator with no direct stake in the outage to run it.

For reporting, do not hand the client an average. Explain that repair-time labels mean different things and that skewed durations make means jump. Show distributions for detection, acknowledgement, mitigation and resolution, error-budget minutes used, a trailing pager-load figure and the closure rate of actions.

Three mistakes that sink rollouts

First, buying tooling before agreeing the rules. A paging product cannot decide what counts as serious; people do. Second, training only one hero. If a single person knows the process, the process leaves when they take a holiday. Train several commanders and rotate them. Third, treating the first tabletop as a pass or fail test. The point is to find friction in a safe room, so celebrate each gap you uncover and log it as a task with an owner.

A practical note on documentation: keep the whole policy to a handful of pages, because a thick manual is something nobody opens under pressure. If a rule cannot be recalled from memory after two rehearsals, shorten it until it can.

Thirty, sixty, ninety

The Atlas compresses the rollout into three checkpoints. By day thirty: severity policy, rotas as code, a hosted status page, role cards and one tabletop. By day sixty: four to six trained commanders, action deadlines, a heartbeat alert that proves the alerting path is alive, and the first paging drill. By day ninety: reviews on time, distribution-based reporting, and rehearsals of the handoffs to security response and disaster recovery.

Hand over the checklist, schedule a follow-up at ninety days, and measure whether the first public update now goes out within five minutes. If it does, the client has the habit, and you have a case study you can talk about without naming anyone.

Kabir Malhotra