Inside a GitOps Engagement: A Quarter with a Payments Team

Let me introduce you to Sahyadri Payments — not their real name, but everything else in this story is faithful to engagements like it. Forty engineers, three product lines, ten Kubernetes clusters, and a delivery process held together by a CI server that had somehow become the most important employee in the company. They called us in about slow deployments and a tired platform team. By the end of week one we knew the real subject: who is allowed to change production, and through which door. The answer we built together was GitOps — declaring production’s intended state in a reviewed repository and letting an in-cluster agent pull that declaration and enforce it — and the quarter we spent getting there makes a better case study than any slide deck.

Discovery Week: Counting the Keys to Production

Every consulting gig starts with an inventory, and ours produced a number that made the room go quiet: fourteen people, contractors included, held working credentials for production clusters. Some sat in the CI server’s secret store. A few lived in browser password managers. One was a shared account whose creator had left the company two years earlier. Nobody had done anything wrong — the credentials had simply accumulated, the way keys do in an old family house.

While we counted keys, we also mapped flow. A typical release crossed four queues: build, approve, deploy window, and the manual verification afterwards. Median time from merge to production sat around two days, and nobody could explain what any given deployment contained beyond “the latest”. Both numbers became our baseline; whatever we proposed had to beat them measurably, or the client was right to keep their old door.

The second finding was cultural. Changes reached production by push: the pipeline ran, fetched its stored credentials, and applied whatever the pipeline decided. When something broke, engineers fixed it directly and hoped the next push would not undo the fix. The platform team described their jobs as “guarding a door that everyone has a key to”. You do not need to be a security specialist to feel the problem. It is the same one NIST flags for microservices deployment: models that push changes inward, carrying production credentials inside the automation, rate poorly — the trust surface simply grows with every integration.

Why the GitOps Option Won the Scorecard

We shortlisted three ways forward and scored them honestly against the client’s reality: multiple teams, compliance obligations, ten clusters. The first option kept push-based delivery but tightened the credential plumbing. The second moved declared state into a repository with an agent pulling it, while humans approved each sync by hand. The third went further — automated pull and reconciliation, with the agent correcting drift on its own.

Our weighted scorecard gave option three the lead: 4.05 against 3.70 for pull-with-manual-approval and 2.80 for the hardened push setup. The gaps came from exactly the things that hurt Sahyadri — audit evidence, standing credentials, and the cost of humans babysitting syncs across ten clusters. The numbers matter less than the shape of the result: the more teams and clusters you run, the more the pull model pays. A four-person startup with one cluster would score the three options in a near tie, and we said so; there is no point selling enterprise machinery to a lifeboat.

If you want the underlying reasoning laid out more formally than a client scorecard allows, the detailed GitOps reference kept by xOps School covers the pull model, its principles and its counterweights with proper sourcing. We drew on it while drafting the client’s decision record.

The GitOps Rollout: Three Days That Set the Rules

Day zero was about choices on paper, and we forced ourselves to make every one of them explicitly: the store (Git, with room for an OCI registry later), the repository layout, the topology (a central hub with pulling agents, given their sprawl), the secrets approach (encrypted at rest, never plaintext), and a written drift policy per resource type. That last one sounds bureaucratic until the first disagreement about a scaling setting arrives and the policy settles it in a minute.

Day one was deliberately modest: stand up the reconciler and point it at a repository that described the reconciler itself. From that moment the tool renewed its own setup by exactly the route every other change takes — a reviewed merge — so the practice never had to argue with the tooling. Migration then proceeded namespace by namespace, one service family at a time, with automated pruning switched off while confidence built. Pruning is where trust is won or lost; nobody forgives a tool that deletes something on day three. Deletion stayed a human decision until each team had watched the agent work for a fortnight.

Week Nine: When GitOps Met an Incident Head-On

Quarter-long engagements without a setback are usually not telling you something. Ours arrived in week nine, when a production incident and the reconciliation loop met each other head-on. Responders applied an emergency change at the console — the old reflex — and the agent, faithful to the repository, reverted it within minutes. The incident ran longer than it should have, and the postmortem was uncomfortable but productive.

The fix was procedural, not technical: a rehearsed break-glass path. Suspending reconciliation for a given application takes seconds in both major tools, and we drilled it until responders trusted it. The client wrote the rule into their runbook in one sentence — during an active incident, suspend first, fix second, reconcile the repository afterwards — and the loop stopped being an adversary. Since then I have put that drill into every GitOps engagement; it costs half an hour and prevents exactly one bad night.

End of Quarter: What Changed and What It Cost

The quarter closed with standing production credentials retired and the invariants written into an architecture note that every team lead signed. Five lines, worth repeating anywhere this practice is adopted:

• One reviewed road to production — no side doors, no console edits that count.
• No mutable references anywhere in the declaration; a version means one artefact forever.
• Agents hold the least privilege that still lets them reconcile.
• Deletion is guarded, never a side effect of a renamed file.
• Data stays out of Git; the store declares systems, not contents.

Deployments stopped being events. The platform team’s calendar filled with something they had not done in years — a planned rebuild of a cluster from the repository alone, timed quarterly, which turned disaster recovery from a fear into a routine. Key rotation went onto the same calendar.

I will not pretend the quarter was free. Pull requests became the bottleneck on busy days, because every change now queues for a reviewer; that waiting is where the model bills you, and the currency is people’s attention rather than licences. Sahyadri decided the price was fair — their words were “we were paying it anyway, just invisibly, in incidents”. That is usually how the arithmetic lands. If your own engagement map includes clients drowning in credentials and drift, the structure of this story transfers cleanly; our guide to a successful DevOps consulting engagement from assessment to enablement covers the wrapper, and the security threads of this story connect naturally with our guide to integrating security into DevOps pipelines. The rest is patience, one namespace at a time.

Kabir Malhotra