Releases That Wait on One Engineer: A Deployment Runbook Fix

Here is a pattern that shows up in client after client. Releases go out on Thursday evenings. They work, mostly, because one senior engineer knows the order of operations: which service to drain first, which migration must finish before the new code starts, which flag to flip afterwards. The release has never been written down because it never needed to be. Then that engineer takes a week off, and the first deployment without them takes six hours and ends in a rollback nobody rehearsed. What follows names the underlying fault and the cure, a deployment runbook that makes the release repeatable by whoever is on shift.

The symptoms: how to spot a person-shaped dependency

You rarely hear anyone say “our deployments depend on one individual”. The signs are indirect, and a consultant sees them within a day:

• Release dates quietly move around one person’s calendar.
• Handovers consist of the phrase “just ping Dev, he’ll know”.
• The deployment pipeline exists, but a handful of manual steps sit outside it and live only in memory.
• Junior engineers are never allowed to push the button, not out of policy but out of fear.
• Postmortems end with “we should document this” and nothing happens.

Any two of these together mean the team is carrying a hidden single point of failure. It is not a technology fault. It is an information fault.

The diagnosis: three root causes

1. The pipeline automates the easy 80 percent

Automation tends to cover the steps that are simple to script: build, test, package, ship. The awkward remainder, such as warming a cache, notifying a partner team or checking a dashboard before declaring victory, stays manual because nobody had time to automate it. Those leftover steps are exactly the ones that go wrong when a newcomer improvises.

2. The expert has no reason to write it down

For the person who knows, the procedure is trivial. Writing it feels like paperwork with no benefit, and the day-to-day pressure to ship always beats the pressure to document. The cost of not writing is paid later, by someone else, so the incentive never lines up.

3. Nothing forces the knowledge to be tested

Even where a wiki page exists, it was written once and never exercised. Steps are missing, command names have changed and a screenshot shows an interface that was redesigned last year. A runbook that has never been followed by a stranger is a rumour, not a procedure.

The solution: a deployment runbook built in four moves

Move 1: Capture the real procedure by watching it

Do not ask the expert to write the runbook alone. Have them run a real release while a second engineer records every action in sequence, including the small checks that the expert performs without thinking. Ask “why did you look at that graph?” and write down the answer. The goal is a faithful copy of what actually happens, not an idealised version.

Move 2: Shape it into a standard page

Use the same structure for every runbook so people learn where to look. A reliable skeleton for a deployment page:

1. Scope. What is being released and what is not covered.
2. Readiness check. Conditions that must be true before starting, such as green tests, an approved change and a staffed on-call rota.
3. Release steps. Numbered, each with its expected outcome and an estimate of how long it takes.
4. Verification. The specific signals that confirm the release is healthy, with the places to read them.
5. Rollback. A separate, equally clear sequence, and flag the moment when going back stops being simple.
6. Communication. Who is told at the start, on completion and on failure.

Place the rollback section where nobody can miss it. In a failure, people do not read the whole page; they jump to the exit.

Move 3: Rehearse it with someone who wasn’t there

Choose a low-risk environment and have a different engineer follow the page exactly. Every deviation, hesitation or “oh, you also have to…” is a bug in the document. Fix them on the spot. Run this rehearsal until a newcomer can complete the release and the rollback without asking for help.

Move 4: Move the manual steps into the pipeline where it is sensible

Once the written procedure exists, you can see which steps are good candidates to automate and which should remain human decisions. Writing first and automating second prevents you from encoding a half-understood process. Keep the decision points human, and let machines do the repetitive confirmations.

A worked example: the Thursday evening release

To make this concrete, imagine a client whose weekly release has twelve real steps. Four are fully automated, five are manual but obvious, and three are the hidden ones: waiting for a queue to empty before the schema change, checking that a payment sandbox is reachable, and telling the support team a banner will appear. The first draft of the runbook lists all twelve in order and marks the three hidden ones in bold, because those are the steps a newcomer would otherwise skip.

After one rehearsal, the page gains two more lines. The queue step gets a threshold (“proceed when fewer than a handful of messages remain”), and the support note gets a message template. Nothing dramatic, yet the next release finishes in ninety minutes with a junior engineer driving.

How this fits into a larger engagement

Runbooks are rarely the headline of a consulting project, but they decide whether the benefits last after the consultants leave. A well-run engagement, such as the sequence described in this outline of an engagement running from first assessment to team enablement, ends with the client’s own people able to operate what was built. A page of numbered steps that the team has actually rehearsed is one of the most tangible forms that enablement can take.

The same logic holds for infrastructure moves. In any account of how outside help supports a migration to the cloud, there is a stage where new systems become somebody’s daily responsibility. That is the point at which a runbook pays for itself, because the people operating the platform did not necessarily design it.

Practical rules for keeping the runbook alive

• One owner per page. Not the only person allowed to run it, only the one who keeps it current.
• Update after every release that surprised you. The edit takes five minutes and the next shift benefits for months.
• Put the date of the last successful rehearsal on the page. Readers can judge how far to trust it.
• Link it from the place people work. The release ticket template and the on-call channel topic should both point to the page.
• Keep secrets out. Reference the vault entry, not the password.
• Retire pages that are no longer true. A wrong runbook is more dangerous than none.

Reading beyond your own shop

Teams that treat runbooks seriously usually pick up ideas from outside their own organisation. Data engineers, for instance, have developed their own habits for keeping scheduled pipelines healthy, and TheDataOps.org is one place where those habits are written up for practitioners. Even if you never touch a data warehouse, the shared theme of making recurring work dependable carries over to application releases.

What success looks like

You will know the fix has worked when a few quiet things change. The release no longer waits for a particular person. A junior engineer runs a deployment under supervision and the sky stays where it is. Postmortems produce edits to a page instead of vague promises. And when the senior engineer finally takes a proper holiday, nobody mentions the release schedule at all.

The remedy is not glamorous. It does not need a new platform, a budget or a reorganisation. It needs a week of watching, a page of numbered steps and a rehearsal with a fresh pair of hands. Do that for the three riskiest recurring tasks first and the hidden dependency on one person starts to dissolve.

Kabir Malhotra