What happens when the cluster and the Git repository disagree? Every GitOps rollout has to answer that question, and the answer separates calm, boring deployments from 2 a.m. firefights. GitOps treats Git as the single source of truth for infrastructure and application state, with an automated agent constantly reconciling the live system to match. That model holds only when teams follow practices the industry learned over years.
This guide walks through those practices in practical terms. It covers repository structure, pull-based reconciliation, secret handling, drift detection, progressive delivery, and the supply-chain habits that keep a pipeline trustworthy. It also flags the mistakes that show up again and again in real clusters, because knowing what breaks is half the battle.
Start With the Four Principles, Not the Tool
Before picking a tool, it helps to understand what actually makes a workflow GitOps. The vendor-neutral OpenGitOps principles, published by a working group under the Cloud Native Computing Foundation, describe four requirements: the system state is declarative, it is versioned and immutable in Git, changes are pulled automatically by an agent, and software agents continuously reconcile the live state with the desired state. A setup that misses any one of these is really just CI with extra steps.
The declarative part matters more than it sounds. If a team writes imperative scripts that say “create this, then patch that,” the repository describes a sequence of actions rather than a desired end state. Reconciliation then has nothing stable to compare against. Declarative manifests, by contrast, let an agent ask a simple question on every loop: does what exists match what is written? If not, fix it.
A common mistake is adopting a GitOps tool while leaving half the environment managed by hand. Those manual changes become invisible to the reconciler, and the next sync silently wipes them out. The practical rule: if it is not in Git, it does not exist. Anything that must persist belongs in the repository, including cluster add-ons, namespace quotas, and network policies.
Choose Pull-Based Reconciliation Over Push
The second structural decision is how changes reach the cluster. In a push-based model, a CI system holds cluster credentials and applies manifests directly. In a pull-based model, an agent running inside the cluster watches the repository and fetches changes itself.
Pull-based reconciliation is the widely recommended default, and the reason is security. Push-based pipelines require storing long-lived cluster credentials in a CI system, which turns the CI platform into a high-value target. With pull, the cluster reaches out; the CI system never needs the keys. The blast radius of a compromised build server shrinks dramatically.
The trade-off is latency and observability. A pull agent typically polls or listens for webhooks, so changes are not instantaneous unless webhooks are configured. Teams should also watch the agent itself: if reconciliation stalls, deployments stop without any obvious error in the application logs. Alerting on reconciliation failures and sync lag is not optional in a pull-based setup.
Structure Repositories for Environments, Not for Teams
Repository layout sparks more debate than almost any other GitOps topic, and there is no single correct answer. What matters is that the structure makes the desired state of every environment obvious and reviewable.
| Pattern | How it works | Best for |
|---|---|---|
| One repo per environment | Separate repositories for dev, staging, and production | Strong isolation and access control between environments |
| Single repo with overlays | One repository, environment-specific overlays via Kustomize or per-environment Helm values | Small teams that want one review queue and shared base configs |
| One repo per application | Each app owns its manifests, aggregated by a root repository | Large organizations with many autonomous teams |
Whichever pattern a team picks, a few habits pay off. Keep environment differences in thin overlays rather than copy-pasted manifests, because duplicated YAML drifts apart within weeks. Put a root or “app of apps” configuration in place so a single entry point defines what should exist. And keep the repository readable: a reviewer should be able to tell from a pull request exactly which environment changes and why.
Treat Secrets as a First-Class Problem
Storing plaintext secrets in Git is the most common GitOps anti-pattern, and it is a serious one. Once a credential lands in a commit, it lives in history forever unless the repository is rewritten. Rotating the secret helps, but the exposure has already happened.
There are three broad mitigations, and most mature teams combine them:
- Encrypted secrets in Git. Tools encrypt secret values before commit so the repository stores ciphertext, and the cluster decrypts at reconciliation time.
- External secret stores. The repository holds only a reference to a secret held in a dedicated vault or cloud secret manager, and an operator syncs the value into the cluster.
- Sealed or scoped secrets. Secrets are encrypted for a specific cluster or namespace, so a leak in one repository does not grant access to another environment.
The practical tip that saves the most pain: decide the secret strategy before the first production workload, not after. Retrofitting encryption across hundreds of manifests is tedious, and it is exactly the kind of project that gets postponed until an incident forces it.
Detect Drift Continuously and Act on It
Drift is the gap between what Git says should exist and what actually runs. It appears for mundane reasons: someone runs a quick manual patch during an incident, an autoscaler changes a replica count, or a webhook mutates an object after creation.
Continuous reconciliation is what catches drift, but detection alone is not enough. Teams need a policy for what happens next. Some drift is legitimate and should be reflected back into Git. Some is an emergency fix that must be codified before the next sync overwrites it. And some is an unauthorized change that deserves a conversation.
A workable approach is to alert on drift, review it on a schedule, and set sync policies deliberately. Automatic sync keeps environments consistent but can erase manual fixes without warning. Manual sync gives a human a checkpoint but lets environments drift for days. Most teams land on automatic sync for lower environments and a reviewed sync window for production, with drift alerts routed to the owning team.
For teams tracking platform news and tooling changes, following a reliable feed of updates helps, and resources like the news section at 9481.org can be a useful place to keep an eye on shifts in cloud-native practice.
Pair GitOps With Progressive Delivery
GitOps controls what is deployed; it says little about how a change rolls out to users. That is where progressive delivery comes in. Instead of replacing every pod at once, a canary release sends a small slice of traffic to the new version, watches error rates and latency, and promotes or rolls back automatically.
The combination works well because both halves are declarative. The rollout strategy lives in the repository alongside the application manifest, and the reconciler keeps the cluster aligned with it. If a canary fails its health checks, the rollout reverts without a human touching a terminal.
The mistake here is treating progressive delivery as a luxury for large teams. A simple two-step canary with an automated rollback on error rate is not much configuration, and it catches the class of bug that passes unit tests but falls over under real traffic. Start with one service, measure the rollback behavior, and expand from there.
Secure the Supply Chain Behind Every Commit
GitOps makes the repository powerful, which means the integrity of what flows from it deserves attention. If an attacker can merge a malicious image reference, the reconciler will faithfully deploy it. Two practices address this.
First, sign and verify artifacts. Image signing with tools built on Sigstore lets the cluster admit only images whose signatures match an expected identity. Second, adopt provenance standards. The SLSA framework, maintained by the Open Source Security Foundation, defines levels describing how trustworthy a build artifact’s origin is. A reconciler that checks provenance refuses artifacts built by an unknown pipeline.
These controls add friction, and that is the point. The goal is that a change reaching production carries a verifiable chain from commit to build to deployment. Teams often start by signing images in the main branch pipeline and enforcing verification in production only, then expand coverage as the workflow matures.
Build a Review Culture Around the Repository
GitOps turns infrastructure changes into pull requests, and that is a cultural opportunity as much as a technical one. Every change becomes reviewable, attributable, and reversible. Teams that lean into this get more value than teams that treat the repository as a dumping ground.
A few habits make reviews effective. Require at least one reviewer for production changes, and make the diff the center of the conversation. Keep pull requests small, because a 900-line manifest change gets skimmed while a 30-line one gets read. Use commit messages that explain the why, since the what is visible in the diff. And when an incident produces an emergency change, open the pull request to codify it before the next sync window, so the fix does not vanish.
It also helps to treat the repository as documentation. A new engineer should be able to read the structure and understand what runs where. If that is not possible, the layout needs work.
Watch the Operational Details That Bite Later
Some GitOps problems only surface months in, and they are worth planning for early.
Reconciliation loops can become expensive at scale. A cluster with thousands of objects and a short sync interval will hammer the API server. Tune sync intervals, use webhooks where supported, and shard reconciliation across multiple agents if the load warrants it.
Version pinning matters too. Helm charts and container images referenced by a floating tag will change under the team’s feet. Pin to immutable versions, and let a deliberate pull request be the only way a dependency moves.
Finally, keep an eye on the platform layer. GitOps increasingly extends beyond Kubernetes to cloud resources and even edge devices, and the tooling landscape shifts. Following a steady stream of cloud and platform updates helps teams spot when a familiar pattern is being replaced. For deeper dives into tooling comparisons and workflow write-ups, the 9481.org blog collects practical material worth reading alongside official documentation. (Kubernetes Documentation: Overview)
Conclusion
GitOps best practices are less about a specific tool and more about a set of commitments: declarative state, Git as the only source of truth, pull-based reconciliation, deliberate secret handling, continuous drift detection, progressive rollouts, and a verified supply chain. None of these are exotic. What separates teams that thrive is the discipline to apply them consistently and the willingness to fix the small operational details before they become incidents. Start with the four principles, pick a repository structure the team can actually maintain, and let every change flow through review. The result is a deployment process that is boring in the best possible way.
Frequently Asked Questions
What is the difference between GitOps and CI/CD?
CI/CD automates building, testing, and delivering software. GitOps is a specific operating model for the delivery and runtime side, where a repository holds the desired state and an agent continuously reconciles the live system to match it. A team can practice CI/CD without GitOps, and GitOps typically sits alongside a CI pipeline rather than replacing it. (GitOps)
Is pull-based reconciliation always better than push?
It is the widely recommended default because it avoids storing cluster credentials in CI systems, which shrinks the attack surface. Push can still make sense in constrained environments, such as clusters that cannot reach a repository or where an agent cannot run. The trade-off is that pull adds latency and requires monitoring of the agent itself.
How should secrets be handled in a GitOps repository?
Never commit plaintext secrets. Common approaches include encrypting secrets before commit, referencing an external secret manager, or sealing secrets to a specific cluster or namespace. Most mature teams combine methods, and the strategy should be chosen before the first production workload rather than retrofitted later.
What is drift, and why does it matter?
Drift is any difference between the state declared in Git and what is actually running. It matters because undeclared changes are invisible to review, get overwritten by the next sync, and can mask real problems. Continuous reconciliation detects drift, but teams also need a policy for reviewing and resolving it.
Do small teams need progressive delivery?
It helps more than many small teams expect. A simple canary with an automatic rollback on error rate is modest configuration and catches failures that unit tests miss. Starting with one service lets a team learn the rollback behavior before expanding the pattern.
How often should reconciliation run?
There is no universal number. Short intervals catch drift quickly but increase load on the API server, especially in large clusters. Teams should tune intervals to cluster size, use webhooks where supported, and shard reconciliation across agents if a single loop becomes a bottleneck.
