What happens when your cluster quietly stops matching the YAML in your repository, and nobody notices for a week? That question drives almost every GitOps failure. The model sounds airtight on paper: Git holds the desired state, a controller makes the cluster match it, and everything is auditable forever. In practice, teams run into the same handful of problems repeatedly, and most of them have nothing to do with the tools and everything to do with how the workflow around them is designed. This article describes the recurring trouble spots, why they happen, and what actually helps. (GitOps)
Drift Is the Failure Mode Nobody Plans For
Configuration drift is the gap between what Git says should run and what is actually running. It creeps in through the most ordinary actions: someone runs a quick `kubectl edit` during an incident, a hotfix gets applied through a cloud console, or a mutating webhook rewrites a field the reconciler never expected to change.
The problem splits into two frustrating halves. If the reconciler is aggressive, it silently reverts the manual fix, and the on-call engineer who just solved an outage watches their change disappear. If the reconciler only watches Git and never compares live state, drift accumulates invisibly until a deploy reveals that production has been running something else for months. The OpenGitOps principles, published by the CNCF GitOps Working Group, treat continuous reconciliation as a core requirement rather than a nice-to-have, which is precisely where many real-world setups fall short.
Practical fixes that work
- Turn on drift detection and alerting before you turn on automatic self-healing. Visibility first, enforcement second.
- Give on-call a documented path for emergency changes that ends with a commit to Git, not a console session.
- Audit for controllers that own the same fields. Two tools managing one resource is one of the most common causes of perpetual reconcile churn.
Two Controllers Fighting Over the Same Resource
Running Argo CD and Flux side by side, or pairing a Helm operator with an admission webhook that mutates the same object, produces a loop that is genuinely hard to diagnose. Controller A writes a field, Controller B strips it, Controller A writes it back. The logs fill with sync events, CPU climbs, and the actual application looks fine, which makes the problem easy to ignore until it isn’t.
The tell is a resource whose diff never goes clean. If your sync status flips between synced and out-of-sync on a fixed rhythm, you almost certainly have overlapping ownership. The fix is organizational as much as technical: assign each resource to exactly one reconciler, and document that boundary. Teams that skip this step often spend days tuning sync intervals when the real issue is a second controller they forgot was installed.
Secrets Management Remains the Weakest Link
Storing plaintext credentials in Git is the anti-pattern everyone warns about and a surprising number of teams still live with. The usual mitigations are Sealed Secrets, SOPS, the External Secrets Operator, or a dedicated vault. Each solves the storage problem but introduces its own operational edges: rotation timing, sync failures when the upstream secret store is unreachable, and the awkward question of what the reconciler does when a secret it needs simply isn’t there yet.
A pattern that holds up well is referencing secrets rather than embedding them, so Git contains a pointer and the actual value lives in a secret manager. That keeps the repository safe to share and keeps rotation out of the deploy pipeline. The trade-off is a new dependency: if the secret backend is down during a cluster rebuild, your bootstrap stalls. Teams that have lived through this usually keep a break-glass procedure written down somewhere outside Git, which feels like heresy until the first time they need it.
Git as Single Source of Truth Gets Messy at Scale
The clean mental model of one repository describing one cluster falls apart the moment you have twenty clusters and four teams. Per-cluster overlays, app-of-apps structures, and layered Kustomize bases all add indirection, and indirection makes the simplest question hard to answer: what is actually deployed where?
| Setup | What it buys you | Where it hurts |
|---|---|---|
| Single repo, single cluster | Simple, obvious, fast to reason about | Does not scale past a handful of environments |
| Repo per environment | Clear isolation, easy permissions | Duplication drifts between repos |
| Monorepo with overlays | One place to search, shared base configs | Slow CI, tangled ownership, noisy diffs |
| App-of-apps | Composable, bootstraps itself | Deep nesting obscures what runs where |
None of these is wrong. The mistake is picking one without deciding who owns which directory and how a change gets reviewed. In practice, the teams that struggle most are the ones that adopted a structure for technical reasons and never revisited it as the organization grew.
Rollback Is Not as Clean as the Pitch Suggests
The promise is elegant: revert the commit, let the reconciler do its thing, and you’re back. The reality is slower and messier. A Git revert plus reconcile takes minutes, not seconds, and it can replay intermediate states that were never meant to run in production. Worse, some changes simply aren’t reversible by Git at all. Database migrations are the classic example. A schema change applied by a Job does not undo itself when you revert the manifest, and CRD updates can leave the cluster in a state no commit can describe.
What works in practice is separating reversible from irreversible changes. Stateless application config rolls back cleanly through Git. Schema changes, CRD upgrades, and anything that touches persistent data need an explicit forward-fix plan and a tested backup, not a revert. Teams that treat Git revert as a universal undo button eventually learn this the hard way, usually at an inconvenient hour.
The CI and GitOps Boundary Gets Blurred
A subtle but widespread problem is CI pipelines that build an image and then push manifests straight to the cluster, bypassing the reconciler entirely. It works, which is exactly why it spreads. But it quietly reintroduces the drift problem the whole model was meant to eliminate, because now there are two paths into the cluster and only one of them is tracked.
The cleaner division is straightforward: CI builds and tests artifacts and writes the new version reference to Git. The reconciler notices and applies it. That separation is what makes the audit trail meaningful. When a pipeline can push directly, the Git history stops being a reliable record of what happened. If your team is still sorting out where that line should sit, the cloud coverage on 9481.org digs into how deployment pipelines and platform tooling fit together.
Scale Limits Show Up Later Than You Expect
Small clusters hide performance problems. Thousands of Applications, a large monorepo, and a controller doing full comparisons on a short interval will eventually strain both the reconciler and the underlying etcd store. Symptoms include slow syncs, timeouts, and a UI that takes forever to load, none of which look like a capacity issue at first glance.
Mitigations are unglamorous but effective:
- Shard Applications across multiple controller instances.
- Increase sync intervals for stable, low-churn workloads.
- Trim repository size and avoid cloning enormous histories on every sync.
- Watch reconciler queue depth and memory alongside application health.
The trap is treating GitOps as free. Every additional Application adds reconciliation work, and the cost compounds quietly until one day a routine deploy takes ten minutes instead of ten seconds.
Audit and Compliance Gaps
Git history proves who changed a manifest and when. It does not, by itself, prove what was actually running in the cluster at 3 a.m. on a given date. That distinction matters for anyone facing a compliance review or a post-incident analysis. If the reconciler’s logs are not retained, and cluster audit logging is off, you have intent recorded but not outcome.
Retaining reconciler logs alongside cluster audit logs closes most of the gap. It is also worth deciding early how long you keep them, because the retention window is usually the first thing an auditor asks about and the last thing anyone configures.
People Problems Outnumber Tool Problems
Ask anyone who has run GitOps in production for a while and the technical issues start to sound secondary. The harder problems are human. Developers used to direct deploys find the reconciler’s independence confusing. Reviewers approve manifest changes without understanding the rollout impact. Nobody documents which repository owns which namespace, so changes land in the wrong place and get reverted by a controller nobody remembers installing. (ConfigMaps)
The fixes here are boring and they work. Write down the ownership map. Make the emergency-change path explicit and short enough that people actually use it. Review the GitOps structure quarterly, because the org chart changes faster than the repository layout does. For ongoing perspective on how teams handle these transitions, the news section at 9481.org tracks platform and infrastructure developments as they land.
Conclusion
Most GitOps problems are not failures of the tools. They are failures of boundaries: between controllers, between CI and CD, between reversible and irreversible changes, between what Git says and what the cluster actually does. The teams that do well are the ones that treat drift detection as a first-class feature, assign ownership explicitly, and accept that some changes cannot be undone by a revert. Get those boundaries right and the model delivers what it promised. Skip them and you get a very well-documented record of a cluster that quietly stopped matching it. The 9481.org blog covers more of these operational patterns for teams building out platform workflows.
What causes configuration drift in a GitOps setup?
Manual changes are the main cause. Someone edits a resource directly with a command-line tool, applies a hotfix through a cloud console, or lets an admission webhook rewrite a field the reconciler did not expect. Drift also appears when a tool only watches Git and never compares live state, so changes accumulate without anyone seeing them.
Can two GitOps controllers run in the same cluster?
They can, but only if they manage different resources. If both own the same object, they will continuously overwrite each other’s fields, producing sync status that never settles and steadily rising CPU use. The reliable fix is to assign each resource to exactly one reconciler and document that boundary.
Is it safe to store secrets in a Git repository?
Not as plaintext. The common alternatives are Sealed Secrets, SOPS, the External Secrets Operator, or an external vault that the cluster references at runtime. Each adds its own operational edges around rotation and sync failures, so teams should also keep a break-glass procedure documented outside Git.
Why does rolling back a Git commit sometimes fail to restore the previous state?
Git revert only reverses manifest changes. It cannot undo a database migration that already ran, reverse a CRD upgrade, or clean up data written by a Job. Those changes need a forward-fix plan and a tested backup. Stateless application configuration rolls back cleanly; anything touching persistent state usually does not.
How many applications can a GitOps controller handle?
It depends heavily on sync intervals, repository size, and how much comparison work each application requires. Performance problems usually appear gradually as slow syncs, timeouts, and memory growth. Sharding controllers, lengthening intervals for stable workloads, and trimming repository size are the standard mitigations.
Does Git history prove what was running in production?
No. Git history shows who changed a manifest and when, which is intent rather than outcome. Proving what actually ran requires retaining reconciler logs and cluster audit logs for a defined period. That retention window is typically one of the first things a compliance review asks about.
