The AutomationOps Audit: Ten Checks Before You Build

Every consulting engagement has a moment where the client says the magic word: “automation”. The pipeline should automate, the failover should automate, the cleanup should automate. Sometimes they are right. Often what they actually need first is an honest look at the automation they already run unattended — and whether anyone could stop it on a bad night. That is the work of an AutomationOps audit, and this piece gives you the checklist we would run before agreeing to build anything new. The reference definition comes from the Ops Atlas page on AutomationOps, which frames the whole subject plainly: automation is a production system, chosen deliberately, made safe on repeats, contained when it misbehaves, and owned until it retires.

The AutomationOps Audit: Ten Checks Before Anyone Builds

Run the checks in order. Each one either passes on paper or opens a conversation the client has been avoiding.

1. Count what exists. List every scheduled job, event trigger and runbook script that acts on production without a human keystroke. Include the feral ones — the cron entry in a departed engineer’s account, the laptop-only job that survives because a terminal window stays open. The gap between the official register and reality is your first finding.
2. Name an owner for each. An automation without a named owner is an orphan with privileges. Orphaned scripts carrying admin rights are how governance debt compounds silently for years.
3. Check identity. Does every job share one superuser credential? That single shared login is the classic failure multiplies-through pattern; each automation deserves its own identity, scoped to what it touches.
4. Grade the danger. Assign every entry a tier: read-only; reversible on one target; reversible across many; or the irreversible, data-destroying class that demands second-person approval, soft delete and staged rollout.
5. Test the rehearsal. Can each job run without effects? Caveat for the checklist: run the dry pass with Ansible and remember that check mode bypasses command and shell steps unless the author flags them, so a green rehearsal may only prove the dangerous steps stayed home.
6. Break the empty list. Feed a target validation a blank list. If it proceeds, it inherits the instincts of Google’s Diskerase, a cleanup tool that read an empty wipe-list as a licence for the whole fleet. Empty must stop the run.
7. Measure the blast radius. Does the job touch one machine at a time, with health checks between cohorts? Or does it fan out to everything at once, canary-less?
8. Find the fuse and the brake. A failure budget halts the run after a set number of failures; a kill switch halts it instantly. Both must exist, per job and globally, and the stop must have been actually pulled — twice yearly is the Atlas’s drill rhythm for the risky tiers.
9. Demand the ledger. Net hours returned (runs, times old manual minutes, minus residual minutes and upkeep), success rate, human fallback rate, incidents the automation itself caused, guard trips, and drill-measured time to stop. One worked reference: at forty runs a month the arithmetic yields a bit over seven and three-quarter hours back, with payback inside five months and 95 of every 100 runs landing clean.
10. Ask where the rules live. If tiers, limits and owners live in a spreadsheet, the audit is half failed. Registry as code — entries versioned with tier, limits, identity and review date, guard tests running in CI — is the standard the audit holds up.

One check sits before all ten, and the audit asks it first: should this exist at all? The Atlas’s decision path runs in four gates. Does the task repeat, and is the underlying procedure genuinely understood — or would automation merely put chaos on a schedule? Must the result be removable when it is wrong; if not, fix the source. Does the step need judgement mid-flight? Then design a human-backed interface instead of a hands-off job. Only when a task clears every gate does the final question arrive — does it pay back in hours, or in the consistency the business actually needs?

Scoring the AutomationOps Audit Against Maturity

Ten checks produce a profile, and the Atlas’s maturity ladder gives the profile a name. Level one, ad hoc: nobody can list what exists; the audit cannot even complete check one. Level two, shared: scripts circulate through a repository, but owners are missing and a single credential opens every door. Most organisations the audit visits sit exactly here — it is the famous stall. Level three adds guards: rehearsals, budgets, stops that work. Level four is governed, where the registry is policy and approvals follow tier. Level five, self-correcting, is where a tripped guard teaches the estate something new. Do not sell a client level five before they have survived level three.

The scorer’s discipline is refusing to confuse volume with maturity. A client running four hundred unattended jobs with no owners is less automated than a client running forty governed ones; they are merely four hundred times more exposed. The audit reports exposure, not effort. And when the conversation turns to what could possibly go wrong, the published incident record carries the argument by itself: AWS’s 2017 outage traced to a mistyped input in a routine command, and Facebook’s 2021 backbone failure to a bug in the audit tool that was supposed to guard the change. Neither was exotic. Both were unattended actions meeting absent controls.

Turning Findings into a Ninety-Day Plan

An audit without a plan is a bill. The Atlas’s 30/60/90 arc converts findings into motion. First month: complete the inventory and register the twenty riskiest entries — not all of them, twenty, because a register nobody finishes is a register nobody starts. Second month: consolidate onto one runner, issue each entry its own identity, and run the first stop drill; the room always goes quiet the first time someone actually pulls it. Third month: promote the rest on evidence, and set the target that at least nine in ten automations have a named owner. Approvals should follow tier from day one — gate everything and approvers learn to click yes, which is how an approval theatre gets people hurt.

A note for the debrief, borrowed from a 1983 paper by Lisanne Bainbridge: the ironies of automation hold that mechanising work does not eliminate human error, it relocates that error to the design stage, where it hides inside assumptions until the worst night of the year. The audit’s deepest question is therefore never “can we automate this?” but “who is designing the error path, and when do they rehearse it?” Manual craft stays rehearsed on purpose; the on-call engineer retains the standing right to stop anything without asking first.

Where the Audit Fits the Wider Engagement

Position the audit early, alongside the assessment phase of an engagement — the same discipline described in this blueprint for a successful consulting engagement. Its findings feed the security workstream naturally, since tier approvals and dual authorization align with what the DevSecOps integration guide already prescribes for pipelines. And when the client asks how quality gates apply to jobs rather than releases, point them at the piece on how automated testing transforms delivery quality — guard tests for automations are the same reflex, pointed at operations. Ten checks, one ladder, ninety days: that is the whole method, and it fits on a single slide, which is exactly how the client will remember it.

Kabir Malhotra