When a client asks what separates a calm delivery team from a chaotic one, the honest answer is rarely a tool. It is usually what the team does with its close calls. A release that nearly went out with the wrong configuration, a rollback that worked only because someone remembered an undocumented step: these are gifts, and in DevOps consulting work we see them ignored all the time. What follows are replies to the queries that reach us most about close-call learning.
What exactly is a near miss in a delivery pipeline?
It is any event that could have caused a failed release, an outage or a data problem, but was stopped before it did. A deployment script that pointed at the wrong environment until someone noticed the hostname. A test suite that passed only because a flaky test happened to be skipped. A certificate that was renewed on the final afternoon. No customer saw anything, so no ticket was opened, and that is precisely the problem: the evidence evaporates.
The simplest definition we use with teams is this: if somebody said “phew” out loud, it counts.
Why should we spend time on events where nothing went wrong?
Because the cause of a near miss and the cause of a real failure are usually the same. What differs is luck. If your next outage will arise from a weak spot you already brushed against last month, then studying last month’s scare is a free rehearsal. You get the insight without paying for downtime, apologies or lost trust.
There is a managerial angle too. When you only review failures, the team learns that attention arrives with pain. When you also review close calls, attention becomes routine and nobody needs to be embarrassed to take part.
How do we collect near misses without creating paperwork?
Keep it light enough that someone worn out after a heavy shift will still fill it in. We often suggest a single shared channel or a short form with three prompts.
- What nearly happened?
- What stopped it?
- What would have made the stop automatic?
No severity ratings, no approval chain, no required fields beyond those three. If you find the entries are rare, the cause is almost never laziness. Usually people are unsure whether their story is “big enough”, so say out loud that small stories are the point.
Who should read the reports, and who should not?
Readers should be the people who can change how work is done: team members, platform owners, and the managers who allocate time. Resist forwarding raw entries to executives as a performance indicator. The moment a count of near misses appears on a scorecard, people start trimming the list, and the data stops being honest. Share themes and fixes upward; keep the raw material inside the team.
How often should we review them?
A short fortnightly slot works well for most teams. Thirty minutes is enough: read the entries, group the ones that look alike, and choose at most two follow-up actions. Anything beyond that tends to stall. If a fortnight feels too frequent, move to monthly, but do not let the gap grow longer, because memory fades and the details that make an entry useful disappear.
What does a good follow-up action look like?
A good action is small, owned by one person, and checkable. “Improve release quality” is not an action. “Add a check that fails the pipeline if the target environment name does not match the branch” is. After the check is added, test it by deliberately triggering the situation in a safe environment. A control you have never seen fire is a control you cannot trust.
Be generous with actions that cost minutes rather than weeks. Renaming two similarly named buckets, adding a confirmation prompt to a risky script, or putting the environment name into the terminal prompt are all unglamorous and effective.
Can a near miss be too small to bother with?
Only if it has no lesson attached. A typo caught by a linter is the tool doing its job. A typo that reached the pull request because the linter was switched off for that repository is worth a line. The test is whether the entry points to a gap in your process rather than a gap in somebody’s concentration. When in doubt, log it; you can always drop an entry during review, but you cannot recover one that was never written down.
Do on-call handovers count as a source?
They are one of the richest. Ask the outgoing engineer what they nearly escalated, what they were unsure about and what they quietly worked around. Handover notes are full of half-sentences such as “seemed fine after a restart”. Each of those is a candidate entry, and reading them aloud in the review often shows that three people have worked around the same quirk without ever mentioning it.
Is this something a consultant needs to run for us?
Not necessarily. Many teams can start alone with the form and the fortnightly slot. Outside help is useful when patterns are hard to see from the inside, for instance when every team has a different name for the same problem, or when a transformation programme is moving fast and nobody has time to step back. If you are wondering how an engagement is usually structured from assessment through hand-over, the article laying out a blueprint for a DevOps consulting engagement, beginning with assessment describes the usual phases.
How do we stop people from hiding mistakes?
You stop it by showing, repeatedly, that reports lead to improvements and never to blame. Thank the reporter first. Write entries in the language of the system: “the script accepted an ambiguous argument”, not “an engineer typed the wrong value”. Close the loop by announcing which change came from which entry. After a few cycles, people begin to compete to find the next weak spot, which is the culture you want.
Does this connect to wider enterprise transformation?
Very much. Large change programmes are full of handoffs, and handoffs are where near misses cluster. A practical overview of how structured DevOps support helps organisations become more agile appears in a strategic walkthrough of enterprise agility with DevOps consulting. Reading the two together gives a sense of where close-call reviews fit in the larger picture: they are a cheap feedback loop sitting underneath the bigger changes.
Where did the idea of learning from close calls come from?
The habit has deep roots in safety-conscious industries, and online-service reliability groups have adopted it widely. If you would like to browse writing from that world, an online home for short reliability-engineering articles is a reasonable starting point. Its content centres on keeping services running smoothly, and the attitude it reflects, studying what almost happened, carries across to any delivery team.
What should we do this week?
Pick three small moves and make them real before Friday.
- Create the shared channel or three-prompt form and tell the team it exists.
- Ask each person to post one close call from the past month, however minor.
- Put a thirty-minute review on the calendar and pick the first two follow-ups together.
What results can we realistically expect?
Do not expect fireworks. Expect fewer repeat surprises, a quieter on-call rota and a team that talks about risk without defensiveness. Over a quarter you will probably notice that the same two or three themes keep appearing, such as unclear environment naming, steps that live only in one person’s head, or checks that run too late in the pipeline. Fixing those themes tends to be more valuable than any single large initiative, because you are removing the faults the team actually bumps into.
Be careful about promising numbers to stakeholders. Near-miss programmes are easy to oversell, and the figures that matter, such as repeat incidents and recovery effort, take several months to move. Describe the programme as a steady habit, not a campaign, and let the quieter pager do the persuading.
What if the team says it has no time?
Then start smaller. Choose one pipeline, one channel and one fifteen-minute slot each fortnight. Teams that claim they have no time for near-miss reviews often spend far more hours on repeat firefighting that a single review would have prevented. Frame the slot as a trade: a quarter of an hour now against an evening of recovery later. That framing tends to win over even sceptical engineers, and once the first useful fix lands, the time argument mostly disappears.
If you still meet resistance, ask the sceptic to bring a single example from the last month. There is nearly always one, and talking it through demonstrates the value better than any presentation could.
Best Cardiac Hospitals Near You
Discover top heart hospitals, cardiology centers & cardiac care services by city.
Advanced Heart Care • Trusted Hospitals • Expert Teams
View Best Hospitals